Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper considers the problem of offline imitation learning from demonstrations when these demonstrations may have been generated from a mixture of optimal, suboptimal, and even adversarial (i.e. worse than random) agents. To address this problem, the authors introduce Confidence-Aware Imitation Learning or CAIL, an imitation learning methodology where the policy is optimized jointly with a weight function that dynamically reweighs the "expert" demonstrations so that policy more closely imitates the high quality demonstrations. To enable the training of this weight function, CAIL requires, similarly to prior work, some additional supervision in the form of demonstration ranks, namely the quality of a small collection of demonstrations are ranked relative one another. Two theorems show that CAIL converges at a given rate under intuitive assumptions. Finally a collection of experiments show that, for both simulated environments and real-world robotic settings, CAIL outperforms competitive baselines.

main_review: ## Post-rebuttal response

I thank the authors for their helpful response to my, and other reviewer, comments. I found the additional ablation of the size of the ranking dataset to be particularly enlightening. I was positive about this paper before the rebuttal period and have not found any reason to change my opinion, I would be happy to see this paper accepted.

---

## Overall

While I do have some small concerns and questions (see below) I'm generally positive about this paper:

* Learning from suboptimal demonstrations is clearly an important problem although the requirement for additional ranking supervision is a bit limiting.
* CAIL's formulation as a bi-level optimization problem is interesting and the theoretical results are promising.
* CAIL substantially outperforms other ranking based methods across a number of diverse tasks and is competitive with confidence-based methods.

Baring any surprises or poor responses to my below points, I argue for the acceptance of this paper.

## Clean sensor modalities

Given B.1.1, it seems that, even for the real robotic experiments, that you use  clean, low-dimensional, inputs and relatively small neural models. My confidence in the generalizability of your experiments would be improved if there were, even preliminary, results for more complex settings (e.g. see any existing task in popular embodied simulators AI2-THOR, Habitat, iGibson, etc). Have you considered any such experiments?

## Ablating the size of the ranking dataset

While your Fig. 3 provides some interesting ablations for varying optimality, I am most interested in how the ranking and confidence based methods perform as the amount of additional supervision from ranks/confidence scores is varied. Would you have any additional results for this?

## Line-by-line comments/questions

In the following I give line-by-line comments and questions.

Line 46
 - I would suggest being more explicit about what is being ranked here. I.e. clarify that you require a dataset of ranked demonstrations.


Line 48-50
 - I found this a bit vague on my first reading, while you are technically agnostic to the imitation learning algorithm I would suggest grounding your introduction by describing how you actually achieve this (you hint at this in the next paragraph but I would mention explicitly that you use the reward function learned by AIRL to rank demonstrations and then compare these rankings against your gold labels).


Line 117
 - I'm not sure I understand what makes this return "expected". What is the expectation taken over when the demonstration is fixed and $\mathcal{R}$ is a function?


Line 127-128
 - In most cases I would expect demonstrations to be labeled with their source, i.e. demonstration $d_j$ came from annotator $i$. This additional label $i$ could be used as an additional feature for $\beta$ and could act as a strong signal for learning which demonstrations can be safely ignored. Have you considered this setting?


Line 177
 - I would suggest formally naming the $\beta$-functions so that you do not have to use somewhat awkward phrases like "value of confidence".


Line 178
 - I think it would be worth defining pseudo-update or giving some intution as to why you have named it this (as otherwise one has to wait until L185 before things start to become clear).


Line 190-192
 - While taking multiple gradient steps is a problem, it seems to me that the bigger problem is that you're, unless I'm mistaken, taking gradients of gradients of graidents of... This is quadratic in time but exponential in the parameter dimension.


Line 202
 - I'd suggest placing the footnote after the comma (i.e. \tau,\footnote{...}) as otherwise this looks like an exponent.


Line 254
 - You should define the indicator function here rather than waiting to L256.


Equation (14)
 - There are mismatched parentheses in the first $|...| > \epsilon$ statement.
 - Why not bring the negative sign into the difference on the first line of the equation?
 - I suspect there is something wrong with the second line of the equation. Note that the exxpression inside the maximum is always positive (as it's squared) so the max with 0 is unnecessary. This second line also seems to have some strange behavior, it becomes larger when the rankings agree but smaller when they don't?


Lines 256-259
 - I would appreciate a proof that RK is Lipschitz.


Line 310
 - How are you computing these p-values?
 - "The p-value" -> "the p-value"


Line 327
- "...where similar to before the..." -> "...where, similarly as before, the..."


Equation (8) in the appendix
- In the second to the last line, $\mu G$ should be $\mu C$.
limitations_and_societal_impact: The noted limitation that you "still require that the dataset consists of demonstrations that encode useful knowledge for policy learning" is true but is a bit broad. I would prefer your limitations focus on concrete limitations of your study, e.g. (1) what about high-dimensional complex tasks?  (2) how do the assumptions of your proofs fail in practice? (3) how realistic/expensive is it to collect trajectory rankings in real settings? (4) how often do we really have large collections of mixed optimality demonstrations?
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: * This paper proposes a novel imitation learning algorithm, Confidence-Aware Imitation Learning (CAIL), to learn policy from suboptimal data.
  * In particular, this proposed algorithm iteratively does two gradient updates:
    1. learns a function, \beta, that maps (state, action) to a confidence score using an evaluation dataset (e.g., a limited amount of partial rankings) with an outer loss.
    2. learn the imitation learning model parameters \theta by reweighting the full IL dataset (could be suboptimal, without labeled rankings) using \beta, and optimizing for an inner loss.
  * The authors proposed a convergence rate of the proposed method as a bi-level optimization problem (Theorem 2).

* The authors conduct a comprehensive evaluation between the proposed algorithm with a set of baseline algorithms.
  * Domains: MuJoCo reacher, MuJoCo ant, Franka Panda simulation, UR5e real robot.
  * Baselines:
    * Standard imitation learning algorithms: GAIL, AIRL.
    * Confidence-based methods: 2IWIL, ICGAIL.
    * Ranking-based methods: T-REX, D-REX, SSRR.
  * Results:
    * Reacher: CAIL outperformed other methods in expected return.
      * All comparisons (between CAIL and another method) are statistically significant.
    * Ant: CAIL outperformed other methods in expected return.
      * The p-value between CAIL and the closest baseline method, 2IWIL, is 0.1405.
    * Franka Panda simulation: CAIL outperformed other methods in expected return.
      * The p-value between CAIL and the closest baseline, 2IWIL, is 0.0974.
    * UR5e real robot: CAIL outperformed other methods in expected return.
      * All comparisons (between CAIL and another method) are statistically significant.
    * Demonstrations with Different Optimality
      * CAIL consistently outperforms or performs comparably to other methods with demonstrations at different optimality.
      * CAIL performs more stably while the baselines suffer from a performance drop at speciﬁc optimality levels.
    * Learning from Only Non-optimal Demonstrations
      * Ant: CAIL outperformed other methods in expected return.
        * All comparisons (between CAIL and another method) are statistically significant.

main_review: 
* The contribution is clear, and evaluation is comprehensive
  * The authors propose a novel IL algorithm that allows the robot to learn from suboptimal demonstrations. The authors show that the proposed method achieved a higher expected return in the learned policy than a set of baseline methods.

* One potential typo/mistake (in case it is truly a mistake, results might need to be rerun)
  * Eq. 14: In Eq. 14 2nd case, I think it should be `-\epsilon`?
    * I tried to plot the value of `RK[x,x,x,x]` as a function of `1[x>x](x-x)`, but it is not continuous. I think if you write `-\epsilon` in the 2nd case, it would be continuous?

* The writing is clear. Some comments:
  * Line 60: `with varying optimality`: when I first read this, I was confused. I thought `optimality` is a binary term, optimal or not. Later, I understood what the authors meant. I think maybe add some explanations or examples in Sec. 1 about this `varying`.
  * Line 178: `pseudo-update`: it might be very helpful to provide some insights into why this `pseudo-update` is necessary and why we cannot just simply use `\theta_t` in Eq. 5. My understanding is that it is required in the convergence proof?
  * Eq. 14: on the right of the 1st case, there is a missing `)`.
  * Eq. 14: on the left-hand side, the `;` and `,` are both used.
  * Line 312: `For Ant, the p-value between CAIL and the closest baseline method, 2IWIL, is 0.1405`: It would be helpful if any insights into this insignificance can be provided.
  * Line 335: `For the simulated robot arm environment, the p-value between CAIL and the closest baseline, 2IWIL, is 0.0974`: It would be helpful if any insights of this insignificance can be provided.
  * Line 343: `Demonstrations with Different Optimality`: I am not sure in which domain the authors evaluated this.

* I didn't check the proofs in the appendix.

limitations_and_societal_impact: * The main limitation has been discussed in Sec. Limitations and Future Work.

# Post rebuttal
Thank you for the clarification and additional explanations!
I have decided to keep my original score.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper investigates imitation learning under the setting where there are non-interactive state-action demonstrations with varying optimality, and among them a small percentage has trajectory-level ranking annotations. The authors propose an imitation learning framework that can leverage all demonstrations, although some of them are suboptimal or even malicious, and potentially outperform the demonstrators. The idea is to formulate learning a state-action confidence function (which weights demonstrations) as a bi-level optimization problem. This closely ties IRL and ranking matching. The authors prove the convergence rate of the resulting algorithm -- CAIL -- and show empirically both in simulation and in real-word that CAIL outperforms recent algorithms designed for similar settings. 

main_review: Strengths:

Using bi-level optimization to learn the confidence function in IRL seems new. Compared with the two-step approaches, e.g. I2WIL and T-REX, that first learns a confidence function then applies IRL, the bi-level optimization formulation tightly connects the IRL with ranking matching to effectively leverage all demonstrations. Even though bi-level optimization is harder to solve than the two-step formulations, empirically it demonstrates better or on-par performance.
Encourage results:

In the experiments, the authors compare CAIL with a range of state-of-the-art algorithms and demonstrate encouraging results, especially in Fig 3.


Weakness: 

One of the three claimed main contributions in the paper is proving the convergence rate of the resulting algorithm CAIL. However, the bi-level formulation , i.e., $\min_\beta L_{out}(\theta^*(\beta))$ where $\theta^*(\beta) = \arg\min_\theta L_{in}(\theta, \beta)$, and analysis seems to be standard.  If the authors can explain the challenges and novelties in the analysis, that would be great. 

Regarding writing, the paper is overall easy to follow. However, I think the framework proposed in Section 4 is a bit abstract and readers may easily get lost. For example, in Section 4, L_out (the ranking loss) appears in Eq. (3), but it’s unclear how L_out is a function of the parameters of any imitation learning algorithm. Therefore, it might be helpful to discuss that L_out relies on an estimate of the reward function in Section 4. 
Furthermore, I found the following points confusing: 
- Eq. (1) and (3) both define beta*. But they are different. For instance, Eq. (3) is sample-based ranking loss. 
- If Eq. (4) and Eq.(6) are applied until convergence, what is the difference between them?

Regarding experiments, in the simulated Franka Panda Arm and real robots experiments, CAIL does not seem to out-perform 2IWIL, which is arguably a simpler approach as during policy optimization the confidence function is fixed. In the Reacher task (Fig 2e) and Ant task (Fig 3c), it’s interesting that the performance of 2IWIL and ARIL decrease with increasing interaction steps; whereas CAIL improves monotonically. I wonder whether authors have intuitions for this phenomenon. If the highest expected return is reported instead of the last policy expected return, how would Fig 3a and Fig 3b look like? In addition, interestingly, in Fig 3c, with pure suboptimal demonstrations, AIRL can still learn a policy that has expected return close to the best policy learned by CAIL with 5e6 iterations steps. It will be helpful if the authors can provide the expected return of the experts. 

It’s also worth noting that the optimal confidence function \beta* is defined by an empirical risk minimizer (Eq. 13). It’s unclear to me why the bi-level optimization approach is able to learn with a smaller amount of data than 2IWIL and T-REX. 

Other comments: 
- p-values are used to show statistical significance in the experiments. What are the null hypotheses and tests? 
- In P4 L 153, \beta(s,a) is used to define the state-action probability of \pi_new. Is certain normalization property enforced for function \beta?
- In P3 L91, it’s mentioned that the paper focuses on the offline learning setting. However, the instantiation of CAIL (Section 4.3) needs interactions with the environment.

limitations_and_societal_impact: yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes Confidence-Aware Imitation Learning (CAIL) that learns from sub-optimal demonstrations collected by different policies. The main idea is to train an imitating policy from demonstrations weighted by an estimated confidence scores, while the confidence scores are simultaneously estimated by using the imitating policy and ranked demonstrations. The paper also presents the convergence proofs of the algorithm. Experiments on simulated and real-world environments indicate that CAIL is more effective at learning from sub-optimal demonstrations when compared to existing algorithms.
main_review: In short, the paper presents an improvement over prior works and shows that the new algorithm works empirically well on simulated and real-world problems. However, the weaknesses of the paper are the technical quality and clarity. I currently rate the paper as a weak-rejection unless these weaknesses are addressed. 

- Originality:

The paper presents a new idea to learn from sub-optimal demonstrations. This idea is a combination of two ideas: imitation learning with confidence scores [1] and imitation learning with ranked demonstrations [2]. The paper clearly mention these prior works and differentiate itself that the idea is different mainly because [1,2] did not jointly learn the confidence/ranking and policy. I view this as a fair improvement over the prior works. 

- Quality:

The idea of the framework is sounded. I like that the framework is general and allows changing the loss functions for inner learning and outer learning. The analyses in Section 4.2 are also valuable to show that the framework can learn the optimal policy under some conditions. (I did not carefully check the proofs in the appendix, but the proofs seem to follow a standard convergence proof for smooth objective functions.) However, I could not understand the practical implementation in Section 4.3 and have several questions and comments. 

1. In [1], the confidence is constrained to be a probability simplex that is normalized. However, the confidence in CAIL is unnormalized given Figure 1(a) in the appendix, and the upper-bound of the confidence is not defined. This unbounded $\beta$ should affect the gradient of AIRL loss, i.e., the first term can have a high magnitude from $\beta$ and dominates to second term in equation 11. How does CAIL handle this issue? 

2. The inner loss $\mathcal{L}_{in}(s,a; \theta, \beta)$ is defined as a function of $(s,a)$, but the AIRL loss does not depend on a specific $(s,a)$ because the loss computes the expectation over state-action distributions. How does this affect the theoretical analyses?   

3. Without defining $\beta$ to be a probability simplex as done in the prior work, the optimal solution in equation 1 is $\beta(s,a) = \infty$ which is not meaningful. While this is a minor point since the algorithm does not use equation 1, an important quantity such as the theoretically-optimal confidence should be more meaningful. 

4. Figure 1(a) in the appendix plots the confidence of different policies, but $\beta$ is a function of state-action pair. Could the authors clarify how they compute the confidence in Figure 1(a)? Does the plot show the average of $\beta(s,a)$ in each trajectory?

- Clarity:

The writing is mostly clear. Though, there are some parts that could be improved:
1. The word "varying optimality" can be misunderstood as learning from multiple expert policies as considered by [3] where demonstrations in the dataset are optimal w.r.t. different rewards. However, CAIL assumes there is only one reward function and only one optimal policy. I think "varying optimality" is too ambiguous. 

2. Theorem 2 overclaims that the solution is the optimal confidence. The theorem guarantees only the convergence to a local solution where the gradient is zero. Since it is not assumed that $\mathcal{L}_{out}$ is convex w.r.t. $\beta$, the solution is not always the optimal confidence. The claim from Theorem 2 should be revised. 

3. There is an inconsistent notation: Equation 2 uses $\beta(s,a) p_{\pi^d}(s,a)$ to denote the weighted sampling distribution, but equation 11 uses $\beta(s,a) \pi^d$. 

4. There is a slight gap between line 146 where the optimal confidence is defined and line 158 where the objective to learn the confidence is defined. That is, are equation 1 and equation 3 equivalent? If they are, it should be shown. If they are not, it should be explained why equation 3 is a good surrogate to equation 1 and how the choice of surrogate (i.e., the outer loss) affect the solution. 

- Significance:

The paper is significant in terms of algorithmic improvement and practical demonstrations.



[1] Yueh-Hua Wu, Nontawat Charoenphakdee, Han Bao, Voot Tangkaratt, and Masashi Sugiyama. Imitation learning from imperfect demonstration. ICML, 2019

[2] Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond sub optimal demonstrations via inverse reinforcement learning from observations. ICML, 2019.

[3] Karol Hausman, Yevgen Chebotar, Stefan Schaal, Gaurav Sukhatme, and Joseph J. Lim. Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Nets. NeurIPS, 2017.

limitations_and_societal_impact: The main limitation of the proposed framework is that it requires an additional demonstration dataset for evaluation. This requirement is mentioned in the paper, but not explicitly as a limitation in Section 6. I suggest having a sentence in Section 6 to note this limitation.
review_title: null

Meta-review: This paper considers imitation learning from demonstrations with varying optimality. The authors propose a framework to jointly learn confidence scores for demonstrations and a well-performing policy.

The reviewers find the research problem interesting. The strengths include: A bi-level optimization formulation of the problem, promising theoretical results, and strong empirical results. There is some concern on the applicability of the proposed method to high-dimensional complex tasks.  However, there is a clear consensus among the reviewers that the paper should be accepted.

It is recommended that the authors consider if there is a connection between this work and Zhang, et al . Causal imitation learning with unobserved confounders, NeuRIPS 2020.

Decision: Accept (Poster)