Confidence-Aware Imitation Learning from Demonstrations with Varying Optimality
Authors: Songyuan Zhang
Zhangjie Cao
Dorsa Sadigh
Yanan Sui
Venue: NeurIPS 2021 Poster (2021)
Keywords: Imitation Learning
Learning from Demonstration
Learning from Suboptimal Demonstrations

Abstract: Most existing imitation learning approaches assume the demonstrations are drawn from experts who are optimal, but relaxing this assumption enables us to use a wider range of data. Standard imitation learning may learn a suboptimal policy from demonstrations with varying optimality. Prior works use confidence scores or rankings to capture beneficial information from demonstrations with varying optimality, but they suffer from many limitations, e.g., manually annotated confidence scores or high average optimality of demonstrations. In this paper, we propose a general framework to learn from demonstrations with varying optimality that jointly learns the confidence score and a well-performing policy. Our approach, Confidence-Aware Imitation Learning (CAIL) learns a well-performing policy from confidence-reweighted demonstrations, while using an outer loss to track the performance of our model and to learn the confidence. We provide theoretical guarantees on the convergence of CAIL and evaluate its performance in both simulated and real robot experiments.
Our results show that CAIL significantly outperforms other imitation learning methods from demonstrations with varying optimality. We further show that even without access to any optimal demonstrations, CAIL can still learn a successful policy, and outperforms prior work.

Confidence-Aware Imitation Learning from Demonstrations with Varying Optimality
Songyuan Zhang',, Zhangjie Cao²; Dorsa Sadigh2, Yanan Sui!
National Engineering Lab for Neuromodulation, SAE, Tsinghua University, China *Department of Computer Science, Stanford University, USA
szhang21 2mmmt edu, {caozj, dorsa]@cs. stanford edu, ysu ysui@tsinghua..... edu
Abstract
Most existing imitation learning approaches assume the demonstrations are drawn from experts who are optimal, but relaxing this assumption enables us to use a wider range of data. Standard imitation learning may learn suboptimal policy from demonstrations with varying optimality. Prior works use confidence scores rankings to capture beneficial information from demonstrations with varying opti- mality, but they suffer from many limitations, e.g., manually annotated confidence scores or high average optimality of demonstrations. general framework learn from demonstrations with varying optimality that jointly learns the confidence score and a well- nell-performing policy. Our approach, Confidence-Awars Imitation Learning (CAIL) learns well- pell-performin policy demonstrations, while using an outer loss to track the
this paper, we propose
from weaffidenne-reweighee performance of model and learn the confidence. We provide theoretical guar- antees on the convergence of CAIL and evaluate its performance in both simulated and real robot experiments. Our results show that CAIL significantly outperforms other imitation learning methods from demonstrations with varying optimality. We further show that even without access any optimal demonstrations, CAIL can still learn successful policy, and outperforms prior work.
Introduction
We consider an imitation learning setting that learns well-performing policy from a mixture of demonstrations with varying optimality ranging from random trajectories to optimal demonstrations. As opposed to standard imitation learning. where the demonstrations come from experts and thus are optimal, this benefits from a larger and more diverse source of data. Note that different from setting that the demonstrations are optimal but lack some causal factors [31], in our setting, the demonstrations can be suboptimal. However, this introduces a new set of challenges. First, one needs to select useful demonstrations beyond the optimal ones. We are interested in settings where we do not have sufficient expert demonstrations the mixture so we have to rely on learning from sub-optimal demonstrations that can still be successful at parts of the task. Second, we need to be able to filter the negative effects useless or even malicious demonstrations, e.g., demonstrations that implicitly fail the tasks.
To address the above challenges, we propose to use measure of confidence to indicate the likelihood that demonstration is optimal. A confidence score can provide fine-grained characterization of each demonstration's 's optimality. For example, can differentiate hetween near-optimal demonstrations or adversarial ones. By reweighting demonstrations with confidence score, we can simultaneously learn from useful but sub-optimal demonstrations while avoiding the negative effects of malicious ones. So
"qqual contribution.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

our problem reduces to learning an accurate confidence measure for demonstrations. Previous work learns the confidence from manually annotated demonstrations [30], which are difficult to obtain and might contain
example, conservative and careful demonstrator may assign lower confidence compared to an optimistic demonstrator to the same demonstration. In this paper, we remove restrictive assumptions on the confidence, and propose an approach that automatically learns the confidence score for each demonstration based on evaluation of the outcome of imitation learning. This evaluation often requires access to limited evaluation data.
We propose a new algorithm, Confidence-Awart Imitation Learning (CAIL), to jointly learn a well- performing policy and the confidence for every state-action pair in the demonstrations. Specifically, our method adopts a standard imitation learning algorithm and evaluates its performance update the confidence scores with an evaluation loss, which we refer to as the outer loss. In our implementation, we use a limited amount of ranked demonstrations as our evaluation data for the outer loss. We then update the policy parameters using the loss of the imitation learning algorithm over the demonstrations reweighted by the confidence, which we refer to as the inner loss. Our framework can accommodate any imitation learning algorithm accompanied with an evaluation loss to assess the learned policy.
We optimize for the inner and outer loss using bi-level optimization [5], and prove that our algorithm converges to the optimal confidence assignments under mild assumptions. We further implement the framework using Adversarial Inverse Reinforcement Learning (AIRL) [14] as the underlying imitation learning algorithm along with its corresponding learning loss as our inner loss. We design a ranking loss as the outer loss, which is compatible with the AIRL model and only requires easy-to-acces ranking annotations rather than the exact confidence values.
The main contributions of the paper be summarized as:
We propose a novel framework, Confidence-Awand Imitation Learning (CAIL), that jointly learns confidence scores and a well-performing policy from demonstrations with varying optimality.
We formulate our problem as modified bi-level optimization with a pseudo-update step and prove that the confidence learned by CAIL converges to the optimal confidence in O(1/VT) (T is the number of steps) under some mild assumptions.
We conduct experiments on several simulation and robot environments. Our results suggest that the learned confidence can accurately characterize the optimality of demonstrations, and that the learned policy achieves higher expected return compared to other imitation learning approaches.
2 Related Work
Imitation Learning. The most common approaches for imitation learning are Behavioral Cloning (BC) [20, 4, 23, 22, 3), which treats the problem as a supervised learning problem, and Inverse Reinforcement Learning (IRL), which recovers the reward function from expert demonstrations and finds the optimal policy through reinforcement learning over the learned reward [1. 21, 32). More recently, Generative Adversarial Imitation Learning (GAIL) [18] Iearns the policy by matching the occupancy measure between demonstrations and the policy in an adversarial manner [15]. Adversarial Inverse Reinforcement Learning (AIRL) [14] and some other approaches [13, 17] improve upon GAIL by simultaneously learning the reward function, and the optimal policy. However, these approaches assume that all the demonstrations are expert demonstrations, and cannot learn a well-performing policy when learning from demonstrations with varying optimality.
Learning from Demonstrations with Varying Optimality: ods learn a policy from a sequence of demonstrations annotated with rankings [2, 25, 29, 10]. T-REX learns a reward from the ranking of the demonstrations and learns a policy using reinforcement learning [8]. In our work, we assume access to rankings of a small subset of the demonstrations. The reward function learned from such a small number of rankings by T-REX may have low generalization ability to out of distribution states. D-REX improves T-REX by automatically generating the rankings of demonstrations [9], and SSRR further finds the structure of the reward function [12]. These techniques automatically generate rankings under the assumption that perturbed demonstration will have a lower reward than the original demonstration, which is not necessarily true for random or malicious demonstrations that can be present in our mixture. DPS utilizes partial orders and pairwise comparisons over trajectories to learn and generate new policies [19]. However, requires interactively collecting feedback, which is not feasible in our offline learning setting.
Ranking-based. Ranking- bbasdd meth-

---

Learning from Demonstrations with Varying Optimality: Confidence-based. Confidence-basse methods assume each demonstration or demonstrator holds confidence value indicating their optimality and then reweight the demonstrations based on this value for imitation learning. To learn the confidence, 2IWII. requires access to ground-truth confidence values for the demonstrations to accurately learn confidence predictor [30]. Tangkaratt et al. require that all the actions for demonstration are drawn from the same noisy distribution with sufficiently small variance [27]. IC-
GAIL implicitly learns the confidence score by aligning the occupancy measure of the learned policy with the expert policy, but requires a set of ground-truth labels to estimate the average confidence [30]. Following works relax the assumption of access to the ground-trutt confidence, but still require more optimal demonstrations than non-optinnn non-optimal ones in the dataset [26]. Other works require access to the reward of each demonstration [11]. All of these methods either rely on specific imitation learning algorithm or require strong assumptions on the confidence. To move forward, we propose a general framework jointly learn the confidence and the policy. Our framework is flexible as it can use any imitation learning algorithm as long as there exists a compatible outer loss, i.e., the outer loss can evaluate the quality of the imitation learning model.
Problem Setting
We formulate the problem of learning from demonstrations with varying optimality as Markov decision process (MDP): .M- A, T, R. po: 7). where S is the state space, A is the action space, T : S A x S - [o, 1] is the transition probability, Po the distribution of initial states, R: : S x A + R is the reward function, and y is the discount factor. A policy т S x A + [0, 1] defines a probability distribution over the action space in a given state. The expected return, which 1 ? RR(s,, at)], where t
evaluates the quality of policy, can be defined Th indicates the time step.
Esurppas
We aim to learn policy that imitates the behavior of a demonstrator d following policy ndd who provides set of demonstrations E {Ei,..... EG.. {D} and &a ~ T". Each trajectory is sequence of state- action pairs = {~0, , 8N) ). and the expected return of a trajectory is = Ei= 7 RR(s,, at).
A common assumption in classical imitation learning work is that the demonstrations are drawn from the expert policy n" = n., i.e.. the policy that optimizes the expected return of the MDP . [18, 14). Here, we relax this assumption so that the demonstrations may contain non-expert demonstrations or even failures-die drawn from policies other than π². Given the demonstration set D, we need to assess confidence in each demonstration. To achieve learning confidence over this mixture of demonstrations, we rely on the ability to evaluate the performance of imitation learning. This can be achieved by using an evaluation loss trained on evaluation data, Dg (as shown in Fig. 1). In our implementation, we rely on a small amount of rankings between trajectories as our evaluation data: DE '1, 2 ... m.... To summarize, our framework takes set of demonstrations with varying optimality D as well as a limited amount of evaluation data DE along with an evaluation loss to find well- performnnnn policy. Note that unlike prior work [30], we do not assume that optimal demonstrations always exist in the demonstration set, and CAIL can still extract useful information from D while avoiding negative effects of non-optimal demonstrations.
4 Confidence-Aware Imitation Learning
In our framework, we adopt an imitation learning algorithm with a model Fa parameterized by 0 and a corresponding imitation learning loss Lin. which we refer to as inner loss (as shown in Figure I). We assign each state-action pair a confidence value indicating the likelihood of the state-action pair appearing in the well-performing policy. The confidence can be defined as function mapping from state-action pair to a scalar value 3 S x A + IR. We aim to find the optimal confidence assignments 3* to reweight state-action pairs within the demonstrations. We then conduct imitation learning from the reweighted demonstrations using the inner imitation loss Lin to learn a well well-performing policy. Here, we first define the optimal confidence 3* and describe how to learn it automatically.
Defining the Optimal Confidence. We define the distribution state-action pairs visited by a policy based on the occupancy measure p= S A - IR: (=(8, a) (als)) Ee= 7 P(s s) which can be explained as the un-mormalized distribution of state transitions that an agent encounters when navigating the environment with the policy We can normalize the occupancy measure to form the
3

---

Inmer-level optimizution: Update model
$1 = (Spi "n, 1x
Demonstration Policy x='
E2 = (Sorag, "n, All S4g
Confidence B
Imitution Learning
Inner Loss Lip
Model Fo
Ag. "", ".t S.
Outer-level optimization:
update confulence
Evaluation Data: o.g. Partial runking of trajectories
Outer Loss oont
71, 2 nc 2 ." Mkm
Figure 1: Confidence-Awan Imitation Learning. The demonstrations are shown in the orange box drawn from demonstration policy: E1...., ED x² The confidence learning component and the outer loss are shown in blue. The confidence 3 reweights the distribution of state-action pairs in the demonstration set, and then the imitation learning model Fo learns well-performing policy and new parameters 9 with the confidence-reweiihhee distribution using the inner loss (imitation loss) shown in green. Next iteration, the updated Fo generates trajectories that are then evaluated by the outer loss and potentially other evaluation data (e.g. partial ranking of trajectories) to update confidence.
state-action distribution; (s, - E... E (=,aa Recall that nd is the policy that the demonstrations
(s.a)
are derived from, which can potentially be mixture of different expert, suboptimal, or even malicious policies. We reweight the state-action distribution of the demonstrations derive a new state-action distribution, which corresponds to another policy Nnew: Pr (8, a) A(s, a)p, (s, a). Our goal is to find the optimal confidence 3 that ensures the derived policy Tnew maximizes the expected return:
8* (s, a) arg max 7/m
With such 3 (*, we can conduct imitation learning from the reweighted demonstrations to maximize the expected return with the provided demonstrations.
Learning the Confidence. We will learn an estimate of the confidence score 3 without access to any annotations of the ground-truth values based on optimizing two loss functions: The inner loss and the outer loss. The inner loss L. is accompanied with the imitation learning algorithm encouraging imitation, while the outer loss Lour captures the quality of imitation learning. and thus optimizing finds the confidence value that maximizes the performance of the imitation learning algorithm.
Specifically, we first learn the imitation learning model parameters 0* that minimize the inner loss:
0 (B) = arg min ((s.a)-3(s (in 0,
We note that the inner loss Lin (8. 0, 33 refers to settings where (s, a) is sampled from the distribu- tion A(s, ().)> and hence implicitly depends on B. Thus we need find the optimal ,9*, which can be estimated by minimizing an outer loss four:
Bout arg min (0 (0))).
(3)
This evaluates the performance of the underlying imitation learning algorithm with respect to the reward with limited evaluation data Dr (e.g. limited rankings if we select ranking loss as our choice of Lomis which we will discuss in detail in Sec. 4.3).
4.1 Optimization of Outer and Inner Loss
We design a bi-level optimization process consisting of an inner-level optimization and an outer- level optimization to simultaneously update the confidence 3 and the model parameters A. Within the outer-level optimization, we first pseudo-update the imitation learning parameters to build connection between |8 and the optimized parameters 8' with the current B3. We then update 3 to make the induced O' minimize the outer loss Lout in Eqn. (3). The inner-level optimization is to find the imitation learning model parameters that minimize inner loss Lin with respect to the confidence B. We introduce the details of the optimization below. We use 7 to denote the number of iterations. Note that the losses in this section are all computed based on the expectation over states and actions.

---

Outer-Level Optimization: Updating B. Let 8, be the confidence at time т. Using 3r, we first pseudo-update the imitation learning parameters / using gradient descent. Here pseudo-update means that the update aims to compute the gradients of 3 but does not really change the value of 0. Let % 0- be the current imitation learning model parameters, and we update 6' as:
0/4 02 V (in a; Br),
where js is the learning rate, is the pseudo-updating time step for 0'. We will update O' with respect to the fixed 3r after convergence of Eqn. (4). After updating 0', we now update / using gradient descent with the outer loss Loa from Eqn. (3):
Br+1 B, c (V
(5)
where a is the learning rate for updating B. Intuitively, updating 3 as in Eqn. (5) aims to find the fastest update direction of 0' for decreasing the outer loss Cout Though we compute gradients of gradients for 3 here, 3 is only iteration of training. we only sample a mini-batch of thousands of state-pairs for update. Thus, within each iteration, the total dimension of 3 is small and computing the gradient of gradient is not costly,
nhe-dimension scalar for each state-action pair and within each
Inner-Level Optimization: Updating O. With the updated BT+1, now will update O using gradient descent, where we denote the initialization as 0g = 0,.
(6) With the two updates introduced above (outer and inner optimization), we finish one update iteration with setting 8, to 87+1 using the converged value from Eqn. (5) and 0, 07+1 using the converged value from Eqn. (6).
8++1 0: V a; 0c, B7+1).
After convergence, we set 0,+1
In each iteration of the above optimization- the steps of pseudo-updating and the steps of updating the imitation learning model-multiple gradient steps are required for convergence, meaning that there is a nested loop of gradient descent algorithms. The nested loop costs quadratic time and inefficient especially for deep networks. To further accelerate the optimization, we propose an approximation, which only updates 6 once in the pseudo-updating and the updating steps. Therefore, the new updating rule can be formalized as follows:
0f 0/+1 0, a V aLinn a; 0r, 3r),
8y+1 B, a V 2 f (8/+1)
0,41 0, V (8, Oy B,,, Br41).
4.2 Theoretical Results
We analyze the convergence of the proposed bi-level optimization algorithm for the CAIL. framework and derive the following theorems. We provide the detailed proofs of these theorems in Appendix.
Theorem 1. (Convergence) Suppose the outer loss Lout is Lipschitz-smooth with constant L, the inequality
(8) holds for a constant C 2 in every step T.? and the learning rate satisfies μ. S ", then the outer loss decreases along with each iteration: Lou((9 S (f.), and the equality holds if V VgLnuiBB (87) = O or 07+1 0,-
VeLmm(0141)) Vo, Z (1,,
Remark 1. The inequality in the assumption of Theorem 1 (Eqn. 8) indicates that the directions of the gradients of Law and Lin with respect to 0 should be close. Intuitively only when the two gradient directions align, we can decrease the evaluation loss Lou by updating 9 with Lin-
Theorem ensures that the confidence and the imitation learning parameters monotonically decrease the outer loss. When the gradient of the outer loss with respect to / is zero, 3 converges to the optimal confidence that minimizes the outer loss, i.e., , 3 in Eqn. (1). With the optimal confidence, we can learn a well-performing policy from more useful demonstrations by reweighting them. Thus, the learned imitation model induces lower outer loss (has higher-quality) than the imitation learning model learned from the original demonstrations in the dataset without reweighting.
We remove (s, a) in Cin for notation simplicity.
5

---

Theorem 2. (Convergence Rate) Under the assumptions in Theorem 1, let
9(0, = - VVaCin, a; 0,3)
We assume that Lgw 3)) is Lipschitz-smoott w.E.. B with constant L1, Lin and Low have a- bounded gradients, and the norm V VoVoLn((; (3) is bounded by dv, L is the sipohhit--mmooth constant for Low w.r.t. g(0, 9) as shown in Theorem I. Consider the total training steps as T, we set C2 for some constant C1 where O < C1 S LI and μA 4 for some constant C2. Then: a VT
1
min B & c0 SO ISTST
(10)
VT
and bounded first-order and second-order gradients of Lin and Cou,, which are satisfied for typical Lin and Lout such as the cross-entropy loss of AIRL and the ranking loss in our implementation of CAIL in Section 4.3.
Remark 2. The assumptions of Theorem 2 are
With the bound on the convergence rate, the gradient of the outer loss with respect to 3 is gradually getting close to O, which means that 3 gradually converges to the optimal confidence 3 that minimizes the outer loss if Lea is convex with respect to B.
4.3 An Implementation of CAIL
To implement CAIL, we need to adopt imitation learning algorithm whose imitation loss will be the inner loss. We also need to design an outer loss on the imitation learning algorithm to evaluate the quality of imitation given some evaluation data DE (e.g. partial ranking annotations).
Based on the above considerations, as an instance of the implementation of CAIL, we use Adversarial Inverse Reinforcement Learning (AIRL) [14] as imitation learning model. We use the imitation loss of AIRL as the inner loss, and a ranking loss (based on a partial ranking of trajectories) as the outer loss. AIRL and the ranking loss are compatible since AIRL can induce the reward function from the discriminator within the model, and the ranking loss can penalize the mismatches of the trajectory rankings computed by the induced reward function and the ground-truth rankings from the evaluation data Dg- Furthermore, the implementation only requires the ranking of a subset of demonstrations { m E, i.e.. DE Mg, 2 Te, 2 2 Ti... which is much easier to access than the exact confidence value annotations [6, 19] since confidence not only reflects the rankings of different demonstrations but also how much one demonstration better than the other.
AIRL consists of generator G parameterized by fG as the policy, and a discriminator parameterized by 0p. The generator and the diseriminator are trained in adversarial manner in [15] to match the occupancy measures of the policy and the demonstrations. We write the loss Lin as:
C (s, a; 02,00 ,))
|- log D(8, a)) + (e,a))r |- log(1 D(s,
(11)
( a; (C) = (ss,a)= [log D(s, a) - log(1 - D(s, a))],
(12)
where LD is the inner loss for the discriminator, C is the inner loss for the generator and Tge is the policy derived from the generator. The discriminator D is learned by minimizing the loss DD which aims discriminates the state-action pair (s, a) drawn from g&e and the state-action pair (s, a) drawn from n. The generator parameter d is trained to minimize the loss C which enables the generator to generate state-action pairs that are similar to the state transitions in the demonstrations.
For the outer loss, AIRL approximates the reward function by the discriminator parameters, i.e., R.g. We compute v/c. = Σi' Rigo (se, at) as the expected return of trajectory using the reward Ryp. Then we penalize the mismatches of the rankings derived by Víé. and the ground-trutt rankings:
Lou (0D)
-Σ RK T rle, L[n.. ne,]]
(13)
where I[Me, > nE, is ne. > Me, and otherwise is 1. RK is defined as revised version of the widely-used margin ranking loss with margin as 0:
RK T né, Ns. nE. = {
max((. I[[ee > ne, n),))),
Iné. nTé, > c
max(0, 1 (Ilne. > 1 nk,,) -)))), Inf. rlé Se (14)
4e

---

We revised the original margin ranking loss within a range around the point of ( = 0 to make Lipschitz smooth. If we adopt small enough e, the functionality of the revised marginal ranking loss is close to the original one. In all the experiments, we use 10-5
5 Experiments
In this section, we conduct experiments on the implementation of CAIL in Sec. 4.3, We verify the efficacy of the CAIL in simulated and real-world environments. We report the results on various compositions of demonstrations with varying optimality. The code is available on our website?
We conduct experiments in four environments including two MuJoCo environments (Reacher and Ant) [28] in OpenAI Gym [7], one Franka Panda Arm simulation environment, and one real robot environment with a UR5e robot arm's For each environment, non-optimal demonstrations with different optimality to show the efficacy of CAIL. We investigate the performance with respect to the optimality of demonstrations ranging from failures to near-optimal or optimal demonstrations. We provide the implementation details and more results on the sensitivity of the parameters, and visualize the learned confidence in the supplementary materials.
collect mixture of optimal and
Source of Demonstrations. For MuJoCo environments, following the demonstration collecting method in [30], we train reinforcement learning algorithm and select four intermediate policies as policies with varying optimality and the converged policy as the optimal policy, so that the demon- strations range from worse-than tee-tthan--ndddn ones to near-optimal ones. We draw 20% of demonstrations from each policy. For the RL algorithm, we use SAC [16] for the Reacher environment and PPO [24] for the Ant environment. For the Franka Panda Arm simulation and the real robot environment with URSe, we hand-craft demonstrations with optimality varying continuously from near- ones to unsuccessful ones to approximate the demonstration collecting process from demonstrators with different levels of expertise. We label only 5% of the demonstrated trajectories with rankings since we target realistic settings where only small number of rankings are available for the demonstrations.
Baselines. We compare CAIL. with the most relevant works in our problem setting including: the ofthh-fft standard imitation learning algorithms: GAIL [18], AIRL [14], imitation learning from suboptimal demonstration methods including two confidence-based methods, 2IWIL and IC- GAIL [30]. and three auninn-basedd methods, T-REX [8], D-REX [9]. and SSRR [12). GAIL and AIRL learn directly from the mixture of optimal and non-optimal demonstrations. T-REX needs demonstrations paired with rankings, so provide the same number of rankings as our approach. For D-REX and SSRR, we further generate rankings by disturbing demonstrations as done in their papers. For 2IWIL and IC-GAIL- that need subset of demonstrations labeled with confidence- label the subset of ranked demonstrations with evenly-spaced confidence, i.e., the highest expected return as confidence 1, and the lowest expected return O. This is a reasonable approximation of the confidence score with no prior knowledge available. For a fair comparison, we re-implement 2IWII. with AIRL as its backbone imitation learning method. For the RL algorithm in T-REX, D-REX, and SSRR, we also use PPO. DPS [19] requires interactively collecting demonstrations and the approach in Cao et al. [11] requires the ground truth reward of demonstrations, which are both not implementable under the assumptions in our setting, so we do not include them.
5.1 Results
Reacher and Ant. In the Reacher, the end effector of the arm is supposed to reach final location. Figure 2(a) shows the optimal trajectories of the joint and the end effector in green, which illustrates the policy reaching the location with the minimum energy cost, and the trajectories with lower optimality in red and orange, where the agent just spins around the center and wastes energy without reaching the target. We collect 200 trajectories in total, where each trajectory has 50 interaction steps.
In Ant, the agent has four legs, each with two links and two joints. Its goal is to move in the x-axis direction as fast as possible. Figure 2(b) illustrates the demonstrated trajectories, where green shows the optimal one, and red shows suboptimal trajectories (darker colors show lower optimality). In
Phttp::/sstes.genee http://sseee.gogg...oo///////aal com/view/cal
4 "htts://www.frrnka..
7

---

optimal demonstrations, the agent moves quickly along the x-axis, while in suboptimal ones, it moves slowly to other directions. We collect trajectories with 200,000 interaction steps in total.
As shown in Figure 2(e) and 2(f), CAIL. achieves the highest expected return compared to the other methods and experiences fast convergence. For Reacher, the p-value between CAIL and the closest baseline method, T-REX, is 5.4054 x 10-6 (statistically significant), For Ant, the p-value between CAIL and the closest baseline method, 2IWIL, is 0.1405. CAIL outperforms standard imitation learning methods, GAIL and AIRL, because CAIL selects more useful demonstrations, and avoids the negative influence of harmful demonstrations. We observe that 2IWIL and IC-GAIL do not perform well because neighboring demonstrations in a given ranking are not guaranteed to have the same distance in terms of confidence score and thus the rvenly-saaces confidence values derived from rankings are likely not accurate, All the ranking- ranking-based methods do not perform well. For T-REX, the potential reason can be that the rankings of a subset of demonstrations are not enough learn generalizable reward function covering states. For D-REX and SSRR, the automatically generated rankings can be incorrect since we also have unsuccessful demonstrations which can at times be worse than random actions-and perturbing such demonstrations is not guaranteed to produce demonstrations that imply rankings.
(a) Reacher Illustration
(b) Ant Illustration
(c) Simulation Illustration (d) Real Robot Illustration
4000
Expatel Rotam
440
Experted
20000
1000
-500
-500
-69
1520
-80
-2000
-2000
1500
-2000
Number Interaciion Stepi (1e))
Number of Intersciio Stepe (1e6)
Number of Interctioon (lop)
(c) Reacher Results
(f) Ant. Results
Simulation Results
(h) Real Robot Results
CAIL
2IWIL.
IC-GAII
AIRL
GAIL
TREX
D-REX
SSRR
Figure 2: (a) Reacher, (b) Ant. (c) Simulated Panda Robot Arm, (d) Real UR5e Robot Arm. In (a-d), the green trajectories indicate the optimal demonstrations, while the red and orange trajectories indicate demonstrations with varying optimality. (e-g) The expected return with respect to the number of interaction steps. (h) The expected return of the converged policies for UR5e Robot Arm.
Robot Arm We further conduct experiments in more realistic environments: a simulated Franka Panda Arm and real UR5e robot arm. As shown Figure 2(c) and 2(d), we design a task to let the robot arm pick up a bottle, avoid the obstacle, and put the bottle on a target platform. In the optimal demonstrations in green, the arm takes the shortest path to avoid the obstacle, and puts the bottle on the target, while in suboptimal ones in red (where, similar to before, the brightness of the trajectories indicates their optimality), the arm detours, does not reach the target, and even at times collides with the obstacle. The suboptimal demonstrations represent wide range of optimality from near-optimal ones (small detour) to adversarial ones (colliding). We vary the initial position of the robot end-effector and the goal position within an initial area and goal area respectively. For both simulated and real robot environments, we collect trajectories with 200,000 interaction steps in total.
As shown in Figure 2(g) and 2(h). CAIL outperforms other methods in expected return in both the simulated and real robot environments. For the simulated robot arm environment, the p-value between CAIL. and the closest baseline, 2IWIL, is 0.0974. For the real robot environment, the p-value between CAIL. and the closest baseline, AIRL,, is 0.0209 (statistically significant). In particular, the real robot environment, CAIL achieves a low standard deviation while other methods especially AIRL, IC-GAIL, D-REX and GAIL suffer from an unstable performance. The results demonstrate that
All valuee are computed by the student's t-test and the null hypothesis is the performance of CAIL is equal smaller than the baseline methods.

---

-20
CAIL
Expected Xetm
Exported Retum
2000
2IWIL
-40
Expeetted
IC-GAIL
-50
AIRL
GAIL
-80
2000
T-REX
-100
-100
D-REX
60
40 20
25
50
75
100
10
20
SSRR
Average Reward of Demonstrations
Ratio of Optimal Demoonttations 1%)
Number of Interaction Staps (1e6)
(a) Varying Optimality
(b) Varying Ratio
(c) Pure Suboptimal
Figure 3: (a-b) The Expected Return with respect to different optimality of demonstrations in the Reacher environment, where the different optimality are created by varying the optimality of non- optimal demonstrations, and varying the ratio of optimal demonstrations. (c) Results for learning from only non-optimal demonstrations in the Ant environment.
CAIL can work stably in the real robot environment. We report the success rate- rate that the robot successfully reaches the target within the time limit without colliding with the obstacle- videos of sample policy rollouts the supplementary materials.
Demonstrations with Different Optimality. We show the performance of different methods with demonstrations at different levels of optimality in the Reacher environment. We fix 20% of the total demonstrations to be optimal and make the remaining 80% demonstration drawn from the same suboptimal policy. We vary the optimality of this policy investigate the performance change with respect to different optimalities. Another way to obtain different optimality is to vary the ratio of optimal demonstrations. We show the results of both varying optimality in Figure 3(a) and 3(b) respectively. We observe that CAIL consistently outperforms or performs comparably to other methods with demonstrations at different optimality. Also, CAIL, performs more stably while the baselines suffer from a performance drop at specific optimality levels.
Learning from Only Non-optimal Demonstrations. We verify that CAIL can also learn from solely non-optimal demonstrations without relying on any optimal demonstrations. We remove the optimal demonstrations
the Ant environment and use the remaining demonstrations to conduct imitation learning. As shown in Figure 3(c), CAIL still achieves the best performance among all the methods, which demonstrates that even with all demonstrations being non-optimal, CAIL still can learn useful knowledge from those demonstrations with higher expected return and induce better policy. The highest p-value between CAIL and the closest baseline (2IWIL) is O. 0.0067, which indicates the performance gain is statistically significant. We observe that the performance of AIRL first increases and then decreases. This is because even though the demonstrations are suboptimal, there are potentially optimal state-action transitions leading to the initial high performance. However, at this early training stage, the AIRL model does not converge yet and the model parameters can still change rapidly. After training sufficient number of steps, the AIRL model observes both useful and less useful transitions and converges to the average return of all the demonstrations.
6 Conclusion
Summary. We propose a general learning framework, Confidence-Awart Imitation Learning, for imitation learning from demonstrations with varying optimality. We adopt standard imitation learning algorithms with their corresponding imitation loss (inner loss), and leverage an outer loss to evaluate the quality of the imitation learning model. We simultaneously learn a confidence score over the demonstrations using the outer loss and learn the policy through optimizing the inner loss over the confidenceerrweeihtter reweighted distribution of demonstrations. Our framework is applicable to any imitation learning model with compatible choices of inner and outer losses. We provide theoretical guarantees on the convergence of CAIL and show that the learned policy outperforms baselines on various simulated and real-world environments under demonstrations with varying optimality.
Limitations and Future Work. Although we propose flexible framework to address the problem of imitation learning from demonstrations with varying optimality. our work is limited in a few ways: To learn a well-performing policy, we still require that the dataset consists of demonstrations that encode useful knowledge for policy learning. We also require that the demonstrations and the imitation agent have the same dynamics. In the future, we plan to learn from demonstrations with more failures and relax the assumptions of the demonstrator and imitator having the same dynamics.

---

7 Acknowledgement
We would like thank Tong Xiao for inspiring discussions about the backbone imitation learning algorithms and her help on the experiments. This work is funded by Tsinghua GuoQiang Research Institute, FLI grant RFP2-000, NSF Awards 1941722 and 1849952, and DARPA HiCoN project.
References
[1] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, page 1, 2004.
[2] Riad Akrour, Marc Schoenauer, and Michele Sehag. Preference-based policy learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 12-27. Springer, 2011.
[3] Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469-483, 2009.
[4] Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelli- gence 15, 1995.
[5] Jonathan F Bard. Practical bilevel optimization: algorithms and applications, volume 30. Springer Science & Business Media, 2013.
[6] Erdem BIyık, Dylan P Losey, Malayandi Palan, Nicholas C Landolfi, Gleb Shevchuk, and Dorsa Sadigh. Learning reward functions from diverse sources of human feedback: Optimally integrating demonstrations and preferences. arXiv preprint arXiv:2006. 1001v 2020.
[7] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jic Tang, and Wojciech Zaremba. Openai gym, 2016.
[8] Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Nickum. Extrapolating beyond sub- optimal demonstrations via inverse reinforcement learning from observations. In International Conference Machine Learning, pages 783-792. PMLR, 2019.
[9] Daniel Brown, Wonjoon Goo, and Scott Niekum. Better-than-dmmonstate imitation learning via automatically--aaked demonstrations. In Conference on Robot Learning, pages 330-359. PMLR, 2020.
[10] Benjamin Burchfiel, Carlo Tomasi, and Ronald Parr. Distance minimization for reward learning from scored trajectories. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
[11] Zhangjie Cao and Dorsa Sadigh. Learning from imperfect demonstrations from agents with varying dynamics. IEEE Robotics and Automation Letters (RA-L), 2021.
[12] Letian Chen, Rohan Paleja, and Matthew Gombolay. Learning from suboptimal demonstration via self-supervised reward regression. In Conference on Robot Learning. PMLR, 2020.
[13] Chelsea Finn, Paul Christiano, Pieter Abbeel, and Sergey Levine. A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models. arXiv preprint 16Xiv:1611.03852 2016.
[14] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforcement learning. In International Conference on Learning Representations, 2018.
[15) Ian Goodfellow, Jean Pouget-Abadic Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets, In Advances in neural information processing systems, pages 2672-2680, 2014.
[16) Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference Machine Learning, pages 1861-1870. PMLR, 2018.
10

---

[17] Peter Henderson, Wei-Di Chang. Pierre-Luc Bacon, David Meger, Joelle Pineau, and Doina Precup. Optiongan: Learning joint reward-policy options using generative adversarial inverse reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[18] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in neural information processing systems, pages 4565-4573, 2016.
[19] Ellen R. Novoseller, Yibng Wei, Yanan Sui, Yisong Yue, and J. Burdick. Dueling posterior sampling for preference-based reinforcement learning. In Uncertainty in Artificial Intelligence (UAT), 2020.
[20] Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural computation, 3(1 3()):88 ):88-988 99, 1991.
[21] Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. In IJCAI, volume 7, pages 2596, 2007.
[22] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635,20 2011.
[23] Stefan Schaal. Is imitation learning the route to humanoid robots? Trends in cognitive sciences, (6)22233-24,, 1999.
[24] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06444 2017.
[25] Hiroaki Sugiyama, Toyomi Meguro, and Yasuhiro Minami. Preference-learning based inverse reinforcement learning for dialog control. Thirteenth Annual Conference of the International Speech Communication Association, 2012.
[26] Voot Tangkaratt, Nontawat Charoenphakdee, and Masashi Sugiyama. Robust imitation learning from noisy demonstrations. arXiv preprint arXiv:2010. 10181, 2020.
[27] Voot Tangkaratt, Bo Han, Mohammad Emtiyaz Khan, and Masashi Sugiyama. Variational imitation learning with diverse-quality demonstrations. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 9407-9417, Virtual, 13-18 Jul 2020. PMLR.
[28] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033. IEFE, 2012.
[29] Christian Wirth, Johannes Fürnkranz, and Gerhard Neumann. Model-fres preference- based reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 30, 2016.
[30] Yuch-Hua Wu, Nontawat Charoenphakdee, Han Bao, Voot Tangkaratt, and Masashi Sugiyama. Imitation learning from imperfect demonstration. In International Conference on Machine Learning, pages 681 8818-6,27, 2019.
[31] Junzhe Zhang, Daniel Kumor, and Elias Bareinboim. Causal imitation learning with unobserved confounders. Advances in neural information processing systems, 33, 2020.
[32] Brian D Ziebart, Andrew Maas, J Andrew Bagnell, and Anind Dey. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pages 1433-1438. Chicago, IL, USA, 2008.
11


Reviews:
[{"review_id": "r7FzMSWOKGV", "reviewer": "Reviewer_g7rL", "summary": "* This paper proposes a novel imitation learning algorithm, Confidence-Aware Imitation Learning (CAIL), to learn policy from suboptimal data.\n  * In particular, this proposed algorithm iteratively does two gradient updates:\n    1. learns a function, \\beta, that maps (state, action) to a confidence score using an evaluation dataset (e.g., a limited amount of partial rankings) with an outer loss.\n    2. learn the imitation learning model parameters \\theta by reweighting the full IL dataset (could be suboptimal, without labeled rankings) using \\beta, and optimizing for an inner loss.\n  * The authors proposed a convergence rate of the proposed method as a bi-level optimization problem (Theorem 2).\n\n* The authors conduct a comprehensive evaluation between the proposed algorithm with a set of baseline algorithms.\n  * Domains: MuJoCo reacher, MuJoCo ant, Franka Panda simulation, UR5e real robot.\n  * Baselines:\n    * Standard imitation learning algorithms: GAIL, AIRL.\n    * Confidence-based methods: 2IWIL, ICGAIL.\n    * Ranking-based methods: T-REX, D-REX, SSRR.\n  * Results:\n    * Reacher: CAIL outperformed other methods in expected return.\n      * All comparisons (between CAIL and another method) are statistically significant.\n    * Ant: CAIL outperformed other methods in expected return.\n      * The p-value between CAIL and the closest baseline method, 2IWIL, is 0.1405.\n    * Franka Panda simulation: CAIL outperformed other methods in expected return.\n      * The p-value between CAIL and the closest baseline, 2IWIL, is 0.0974.\n    * UR5e real robot: CAIL outperformed other methods in expected return.\n      * All comparisons (between CAIL and another method) are statistically significant.\n    * Demonstrations with Different Optimality\n      * CAIL consistently outperforms or performs comparably to other methods with demonstrations at different optimality.\n      * CAIL performs more stably while the baselines suffer from a performance drop at speciﬁc optimality levels.\n    * Learning from Only Non-optimal Demonstrations\n      * Ant: CAIL outperformed other methods in expected return.\n        * All comparisons (between CAIL and another method) are statistically significant.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n* The contribution is clear, and evaluation is comprehensive\n  * The authors propose a novel IL algorithm that allows the robot to learn from suboptimal demonstrations. The authors show that the proposed method achieved a higher expected return in the learned policy than a set of baseline methods.\n\n* One potential typo/mistake (in case it is truly a mistake, results might need to be rerun)\n  * Eq. 14: In Eq. 14 2nd case, I think it should be `-\\epsilon`?\n    * I tried to plot the value of `RK[x,x,x,x]` as a function of `1[x>x](x-x)`, but it is not continuous. I think if you write `-\\epsilon` in the 2nd case, it would be continuous?\n\n* The writing is clear. Some comments:\n  * Line 60: `with varying optimality`: when I first read this, I was confused. I thought `optimality` is a binary term, optimal or not. Later, I understood what the authors meant. I think maybe add some explanations or examples in Sec. 1 about this `varying`.\n  * Line 178: `pseudo-update`: it might be very helpful to provide some insights into why this `pseudo-update` is necessary and why we cannot just simply use `\\theta_t` in Eq. 5. My understanding is that it is required in the convergence proof?\n  * Eq. 14: on the right of the 1st case, there is a missing `)`.\n  * Eq. 14: on the left-hand side, the `;` and `,` are both used.\n  * Line 312: `For Ant, the p-value between CAIL and the closest baseline method, 2IWIL, is 0.1405`: It would be helpful if any insights into this insignificance can be provided.\n  * Line 335: `For the simulated robot arm environment, the p-value between CAIL and the closest baseline, 2IWIL, is 0.0974`: It would be helpful if any insights of this insignificance can be provided.\n  * Line 343: `Demonstrations with Different Optimality`: I am not sure in which domain the authors evaluated this.\n\n* I didn't check the proofs in the appendix.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "* The main limitation has been discussed in Sec. Limitations and Future Work.\n\n# Post rebuttal\nThank you for the clarification and additional explanations!\nI have decided to keep my original score.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "DhqylatnHU8", "reviewer": "Reviewer_Tsdv", "summary": "This paper considers the problem of offline imitation learning from demonstrations when these demonstrations may have been generated from a mixture of optimal, suboptimal, and even adversarial (i.e. worse than random) agents. To address this problem, the authors introduce Confidence-Aware Imitation Learning or CAIL, an imitation learning methodology where the policy is optimized jointly with a weight function that dynamically reweighs the \"expert\" demonstrations so that policy more closely imitates the high quality demonstrations. To enable the training of this weight function, CAIL requires, similarly to prior work, some additional supervision in the form of demonstration ranks, namely the quality of a small collection of demonstrations are ranked relative one another. Two theorems show that CAIL converges at a given rate under intuitive assumptions. Finally a collection of experiments show that, for both simulated environments and real-world robotic settings, CAIL outperforms competitive baselines.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Post-rebuttal response\n\nI thank the authors for their helpful response to my, and other reviewer, comments. I found the additional ablation of the size of the ranking dataset to be particularly enlightening. I was positive about this paper before the rebuttal period and have not found any reason to change my opinion, I would be happy to see this paper accepted.\n\n---\n\n## Overall\n\nWhile I do have some small concerns and questions (see below) I'm generally positive about this paper:\n\n* Learning from suboptimal demonstrations is clearly an important problem although the requirement for additional ranking supervision is a bit limiting.\n* CAIL's formulation as a bi-level optimization problem is interesting and the theoretical results are promising.\n* CAIL substantially outperforms other ranking based methods across a number of diverse tasks and is competitive with confidence-based methods.\n\nBaring any surprises or poor responses to my below points, I argue for the acceptance of this paper.\n\n## Clean sensor modalities\n\nGiven B.1.1, it seems that, even for the real robotic experiments, that you use  clean, low-dimensional, inputs and relatively small neural models. My confidence in the generalizability of your experiments would be improved if there were, even preliminary, results for more complex settings (e.g. see any existing task in popular embodied simulators AI2-THOR, Habitat, iGibson, etc). Have you considered any such experiments?\n\n## Ablating the size of the ranking dataset\n\nWhile your Fig. 3 provides some interesting ablations for varying optimality, I am most interested in how the ranking and confidence based methods perform as the amount of additional supervision from ranks/confidence scores is varied. Would you have any additional results for this?\n\n## Line-by-line comments/questions\n\nIn the following I give line-by-line comments and questions.\n\nLine 46\n - I would suggest being more explicit about what is being ranked here. I.e. clarify that you require a dataset of ranked demonstrations.\n\n\nLine 48-50\n - I found this a bit vague on my first reading, while you are technically agnostic to the imitation learning algorithm I would suggest grounding your introduction by describing how you actually achieve this (you hint at this in the next paragraph but I would mention explicitly that you use the reward function learned by AIRL to rank demonstrations and then compare these rankings against your gold labels).\n\n\nLine 117\n - I'm not sure I understand what makes this return \"expected\". What is the expectation taken over when the demonstration is fixed and $\\mathcal{R}$ is a function?\n\n\nLine 127-128\n - In most cases I would expect demonstrations to be labeled with their source, i.e. demonstration $d_j$ came from annotator $i$. This additional label $i$ could be used as an additional feature for $\\beta$ and could act as a strong signal for learning which demonstrations can be safely ignored. Have you considered this setting?\n\n\nLine 177\n - I would suggest formally naming the $\\beta$-functions so that you do not have to use somewhat awkward phrases like \"value of confidence\".\n\n\nLine 178\n - I think it would be worth defining pseudo-update or giving some intution as to why you have named it this (as otherwise one has to wait until L185 before things start to become clear).\n\n\nLine 190-192\n - While taking multiple gradient steps is a problem, it seems to me that the bigger problem is that you're, unless I'm mistaken, taking gradients of gradients of graidents of... This is quadratic in time but exponential in the parameter dimension.\n\n\nLine 202\n - I'd suggest placing the footnote after the comma (i.e. \\tau,\\footnote{...}) as otherwise this looks like an exponent.\n\n\nLine 254\n - You should define the indicator function here rather than waiting to L256.\n\n\nEquation (14)\n - There are mismatched parentheses in the first $|...| > \\epsilon$ statement.\n - Why not bring the negative sign into the difference on the first line of the equation?\n - I suspect there is something wrong with the second line of the equation. Note that the exxpression inside the maximum is always positive (as it's squared) so the max with 0 is unnecessary. This second line also seems to have some strange behavior, it becomes larger when the rankings agree but smaller when they don't?\n\n\nLines 256-259\n - I would appreciate a proof that RK is Lipschitz.\n\n\nLine 310\n - How are you computing these p-values?\n - \"The p-value\" -> \"the p-value\"\n\n\nLine 327\n- \"...where similar to before the...\" -> \"...where, similarly as before, the...\"\n\n\nEquation (8) in the appendix\n- In the second to the last line, $\\mu G$ should be $\\mu C$.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The noted limitation that you \"still require that the dataset consists of demonstrations that encode useful knowledge for policy learning\" is true but is a bit broad. I would prefer your limitations focus on concrete limitations of your study, e.g. (1) what about high-dimensional complex tasks?  (2) how do the assumptions of your proofs fail in practice? (3) how realistic/expensive is it to collect trajectory rankings in real settings? (4) how often do we really have large collections of mixed optimality demonstrations?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "DG8311wsNnw", "reviewer": "Reviewer_1GZR", "summary": "This paper investigates imitation learning under the setting where there are non-interactive state-action demonstrations with varying optimality, and among them a small percentage has trajectory-level ranking annotations. The authors propose an imitation learning framework that can leverage all demonstrations, although some of them are suboptimal or even malicious, and potentially outperform the demonstrators. The idea is to formulate learning a state-action confidence function (which weights demonstrations) as a bi-level optimization problem. This closely ties IRL and ranking matching. The authors prove the convergence rate of the resulting algorithm -- CAIL -- and show empirically both in simulation and in real-word that CAIL outperforms recent algorithms designed for similar settings. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nUsing bi-level optimization to learn the confidence function in IRL seems new. Compared with the two-step approaches, e.g. I2WIL and T-REX, that first learns a confidence function then applies IRL, the bi-level optimization formulation tightly connects the IRL with ranking matching to effectively leverage all demonstrations. Even though bi-level optimization is harder to solve than the two-step formulations, empirically it demonstrates better or on-par performance.\nEncourage results:\n\nIn the experiments, the authors compare CAIL with a range of state-of-the-art algorithms and demonstrate encouraging results, especially in Fig 3.\n\n\nWeakness: \n\nOne of the three claimed main contributions in the paper is proving the convergence rate of the resulting algorithm CAIL. However, the bi-level formulation , i.e., $\\min_\\beta L_{out}(\\theta^*(\\beta))$ where $\\theta^*(\\beta) = \\arg\\min_\\theta L_{in}(\\theta, \\beta)$, and analysis seems to be standard.  If the authors can explain the challenges and novelties in the analysis, that would be great. \n\nRegarding writing, the paper is overall easy to follow. However, I think the framework proposed in Section 4 is a bit abstract and readers may easily get lost. For example, in Section 4, L_out (the ranking loss) appears in Eq. (3), but it’s unclear how L_out is a function of the parameters of any imitation learning algorithm. Therefore, it might be helpful to discuss that L_out relies on an estimate of the reward function in Section 4. \nFurthermore, I found the following points confusing: \n- Eq. (1) and (3) both define beta*. But they are different. For instance, Eq. (3) is sample-based ranking loss. \n- If Eq. (4) and Eq.(6) are applied until convergence, what is the difference between them?\n\nRegarding experiments, in the simulated Franka Panda Arm and real robots experiments, CAIL does not seem to out-perform 2IWIL, which is arguably a simpler approach as during policy optimization the confidence function is fixed. In the Reacher task (Fig 2e) and Ant task (Fig 3c), it’s interesting that the performance of 2IWIL and ARIL decrease with increasing interaction steps; whereas CAIL improves monotonically. I wonder whether authors have intuitions for this phenomenon. If the highest expected return is reported instead of the last policy expected return, how would Fig 3a and Fig 3b look like? In addition, interestingly, in Fig 3c, with pure suboptimal demonstrations, AIRL can still learn a policy that has expected return close to the best policy learned by CAIL with 5e6 iterations steps. It will be helpful if the authors can provide the expected return of the experts. \n\nIt’s also worth noting that the optimal confidence function \\beta* is defined by an empirical risk minimizer (Eq. 13). It’s unclear to me why the bi-level optimization approach is able to learn with a smaller amount of data than 2IWIL and T-REX. \n\nOther comments: \n- p-values are used to show statistical significance in the experiments. What are the null hypotheses and tests? \n- In P4 L 153, \\beta(s,a) is used to define the state-action probability of \\pi_new. Is certain normalization property enforced for function \\beta?\n- In P3 L91, it’s mentioned that the paper focuses on the offline learning setting. However, the instantiation of CAIL (Section 4.3) needs interactions with the environment.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "-M49sLmKMNs", "reviewer": "Reviewer_9kz8", "summary": "The paper proposes Confidence-Aware Imitation Learning (CAIL) that learns from sub-optimal demonstrations collected by different policies. The main idea is to train an imitating policy from demonstrations weighted by an estimated confidence scores, while the confidence scores are simultaneously estimated by using the imitating policy and ranked demonstrations. The paper also presents the convergence proofs of the algorithm. Experiments on simulated and real-world environments indicate that CAIL is more effective at learning from sub-optimal demonstrations when compared to existing algorithms.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In short, the paper presents an improvement over prior works and shows that the new algorithm works empirically well on simulated and real-world problems. However, the weaknesses of the paper are the technical quality and clarity. I currently rate the paper as a weak-rejection unless these weaknesses are addressed. \n\n- Originality:\n\nThe paper presents a new idea to learn from sub-optimal demonstrations. This idea is a combination of two ideas: imitation learning with confidence scores [1] and imitation learning with ranked demonstrations [2]. The paper clearly mention these prior works and differentiate itself that the idea is different mainly because [1,2] did not jointly learn the confidence/ranking and policy. I view this as a fair improvement over the prior works. \n\n- Quality:\n\nThe idea of the framework is sounded. I like that the framework is general and allows changing the loss functions for inner learning and outer learning. The analyses in Section 4.2 are also valuable to show that the framework can learn the optimal policy under some conditions. (I did not carefully check the proofs in the appendix, but the proofs seem to follow a standard convergence proof for smooth objective functions.) However, I could not understand the practical implementation in Section 4.3 and have several questions and comments. \n\n1. In [1], the confidence is constrained to be a probability simplex that is normalized. However, the confidence in CAIL is unnormalized given Figure 1(a) in the appendix, and the upper-bound of the confidence is not defined. This unbounded $\\beta$ should affect the gradient of AIRL loss, i.e., the first term can have a high magnitude from $\\beta$ and dominates to second term in equation 11. How does CAIL handle this issue? \n\n2. The inner loss $\\mathcal{L}_{in}(s,a; \\theta, \\beta)$ is defined as a function of $(s,a)$, but the AIRL loss does not depend on a specific $(s,a)$ because the loss computes the expectation over state-action distributions. How does this affect the theoretical analyses?   \n\n3. Without defining $\\beta$ to be a probability simplex as done in the prior work, the optimal solution in equation 1 is $\\beta(s,a) = \\infty$ which is not meaningful. While this is a minor point since the algorithm does not use equation 1, an important quantity such as the theoretically-optimal confidence should be more meaningful. \n\n4. Figure 1(a) in the appendix plots the confidence of different policies, but $\\beta$ is a function of state-action pair. Could the authors clarify how they compute the confidence in Figure 1(a)? Does the plot show the average of $\\beta(s,a)$ in each trajectory?\n\n- Clarity:\n\nThe writing is mostly clear. Though, there are some parts that could be improved:\n1. The word \"varying optimality\" can be misunderstood as learning from multiple expert policies as considered by [3] where demonstrations in the dataset are optimal w.r.t. different rewards. However, CAIL assumes there is only one reward function and only one optimal policy. I think \"varying optimality\" is too ambiguous. \n\n2. Theorem 2 overclaims that the solution is the optimal confidence. The theorem guarantees only the convergence to a local solution where the gradient is zero. Since it is not assumed that $\\mathcal{L}_{out}$ is convex w.r.t. $\\beta$, the solution is not always the optimal confidence. The claim from Theorem 2 should be revised. \n\n3. There is an inconsistent notation: Equation 2 uses $\\beta(s,a) p_{\\pi^d}(s,a)$ to denote the weighted sampling distribution, but equation 11 uses $\\beta(s,a) \\pi^d$. \n\n4. There is a slight gap between line 146 where the optimal confidence is defined and line 158 where the objective to learn the confidence is defined. That is, are equation 1 and equation 3 equivalent? If they are, it should be shown. If they are not, it should be explained why equation 3 is a good surrogate to equation 1 and how the choice of surrogate (i.e., the outer loss) affect the solution. \n\n- Significance:\n\nThe paper is significant in terms of algorithmic improvement and practical demonstrations.\n\n\n\n[1] Yueh-Hua Wu, Nontawat Charoenphakdee, Han Bao, Voot Tangkaratt, and Masashi Sugiyama. Imitation learning from imperfect demonstration. ICML, 2019\n\n[2] Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond sub optimal demonstrations via inverse reinforcement learning from observations. ICML, 2019.\n\n[3] Karol Hausman, Yevgen Chebotar, Stefan Schaal, Gaurav Sukhatme, and Joseph J. Lim. Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Nets. NeurIPS, 2017.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The main limitation of the proposed framework is that it requires an additional demonstration dataset for evaluation. This requirement is mentioned in the paper, but not explicitly as a limitation in Section 6. I suggest having a sentence in Section 6 to note this limitation. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "Around 4 hours. "}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers imitation learning from demonstrations with varying optimality. The authors propose a framework to jointly learn confidence scores for demonstrations and a well-performing policy.

The reviewers find the research problem interesting. The strengths include: A bi-level optimization formulation of the problem, promising theoretical results, and strong empirical results. There is some concern on the applicability of the proposed method to high-dimensional complex tasks.  However, there is a clear consensus among the reviewers that the paper should be accepted.

It is recommended that the authors consider if there is a connection between this work and Zhang, et al . Causal imitation learning with unobserved confounders, NeuRIPS 2020.

Decision: Accept (Poster)