Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work presents an approach that learns a policy from expert demonstrations data in an offline setting. It is trying to solve an important problem in imitation learning: the expert demonstration may not cover the entire state/action space pretty well, and many existing algorithms require further interaction with the environment. To solve the problem of distribution shift when expert data is not diverse enough, the method proposes to introduce a large number of supplementary demonstrations of various qualities. In this way, the proposed algorithm DemoDICE does not require any on-policy samples. The method adapts the OptiDICE method for its own purpose by using a transformed objective function. The theoretical analysis and empirical experiments demonstrate the improvement of the method over BC and ValueDICE.
main_review: Strength: the paper studies an important problem for imitation learning with only offline data. Several tricks in equation 11, 21 helps to improve the method over baselines. Converting the original minimax problem into a minimization problem helps to stabilize the training. Comparing with BC and ValueDICE demonstrates that the proposed DemoDICE can achieve a better performance in several MuJoCo environments. Especially, it is important to have the results of different number of combinations of expert/imperfect demonstrations. 

Weakness: 
The experiments part may still need further work. 
1. In figure 1, the half cheetah environment results seem to be unfinished. The curves are not converging yet especially for M3. It would be great to see the results when the curves converge. 
2. The authors mentioned that the supplementary demonstrations are imperfect, containing expert or near-expert demonstrations. However, the experiments section used either expert or random demonstrations. It would be great to see the performance of the algorithm when sub-optimal demonstrations are provided instead of totally random trajectories. According to proposal of the paper, the work is to tackle problems where expert demonstrations are not diverse enough. Is it possible to demonstrate how the performance of the proposed method will change when the expert demonstrations are of different coverage of the state/action space? For example, intuitively, when the expert demonstrations cover enough space, the difference between BC, ValueDICE and DemoDICE may not be that significant, while when the expert demonstrations is of very limited coverage,DemoDICE will be much better than BC and ValueDICE.
3. In the results, it seems that the more imperfect demonstrations given the same perfect demonstrations, all methods tend to have worse performance (it appears that M1,M2,M3 are using the same number of expert demos but M3 are given much more bad demos). If that is the case, why include bad demonstrations in the data given that only using good demos can achieve a better performance?
4. In M1 task in Hopper, Walker2d, Ant, the proposed DemoDICE isn't better than the baseline BC with beta=0. Also in the case, where the RB dataset is used, DemoDICE is not better than BC for beta=0 in ant and half cheetah. It is also worse than ValueDICE in Walker2d. Is there further explanation why this happens? 
summary_of_the_review: Overall the paper is of good quality and it studies an important problem. The proposed method formulation makes sense. However, the experiments part need improvement and clarification. 

Post rebuttal: I read the authors' rebuttal and appreciate all the feedbacks. I hope the author could incorporate the suggestions by the reviewers to improve the overall quality of the paper. I raised my rating for the rebuttal and the paper has a nontrivial contribution to the field.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper considers an offline imitation learning (IL) problem with an addition of supplementary imperfect demonstrations. To solve this problem, the paper proposes DemoDICE which regularizes a distribution-matching objective of IL by a KL divergence between the agent distribution and a mixed of expert and imperfect distributions. DemoDICE finds an optimal state-action distribution of this regularized objective by using a dual-program technique similar to that of OptiDICE (Lee et al., 2021) for offline RL with an improvement in terms of stability. Given the optimal state-action distribution, DemoDICE extracts the expert policy by performing weighted behavioral cloning. 
Empirical evaluation on Mujoco tasks with D4RL datasets show that DemoDICE can efficiently and effectively solve the offline IL problem. 

### Contributions
- A new learning problem combining offline IL and IL with imperfect demonstrations. 
- A new model-free offline IL method based on dual-program optimization.

main_review: ### Strengths
The proposed problem is a natural and practical extension of the offline IL problem. This is a nice step towards practical applications of IL. The derivation of the method is also grounded. While the derivation steps are based on existing work, the stability improvement by using a soft-max surrogate and its accompanying proofs are good additions. 
### Weaknesses
1) Related work

It is interesting to see that the final policy learning objective in equation 21 essentially reweighs behavioral cloning (BC) objective based on the advantage function. This result makes DemoDICE closely related to [1] which reweighs GAIL objective based on the advantage function and [2] which learns from imperfect demonstrations by reweighting a BC objective. These recent works should be discussed in the paper. 

2) Experiments

Experiments are conducted sufficiently to demonstrate the effectiveness of DemoDICE on continuous-control benchmarks. Though, there are only two baseline methods (BC and ValueDICE) and more baselines should be evaluated (e.g., the method in [2]).

3) Clarity

The paper is overall well written and well organized. However, there are unclear statements that need further clarification: 

  - In Section 3.3, why cannot $w^\star$ be computed from $\tilde{\nu}^\star$? In my understanding, $\tilde{\nu}$ is a neural network that minimizes equation (18) and the advantage function can be computed directly from it. With these, it should be straightforward to compute $w^\star(s,a) = \exp(\frac{A_{\tilde{\nu}^\star}(s,a)}{1+\alpha}-1)$ on state-action samples drawn from the dataset $D^U$ to reweight the objective.

  - I do not understand the sentence "Based on the weighted BC, we simply extract the policy without training any additional network.". It is indeed  possible to find a non-parametric $\pi$ from equation (21). However, such $\pi$ only defines action probabilities on states observed in the datasets and cannot be used in the environment. At the same time, the appendix does mention about a neural network for the actor. So I do not understand which  "additional network" the above sentence refers to.

  - Proposition 1 requires $\nu$ to be in a large family of functions. Are there restrictions of $\nu$? Lemma 1 and 2 seem to hold for any function $\nu$, so I think Proposition 1 should hold for any $\nu$.

### Questions

1) It is intriguing to see that regularized BC does not work in the experiments but DemoDICE does given that BC is also a distribution-matching approach (without matching the state-marginals). Does the superior performance of DemoDICE come from matching the state-marginals or from the dual-program optimization?

2) In theory, the policy function can be extracted from state-action distribution exactly with $\pi(a|s) = \frac{d^\pi(s,a)}{\int_A d^\pi(s,a) \text{d}a}$. The integration is intractable in general for continuous control tasks. However, for Mujoco tasks the action-space is bounded in $[-1,1]$. Is computing the exact policy function possible in DemoDICE with the knowledge of action-space's bounds?

[1] Yunke Wang, Chang Xu, Bo Du, and Honglak Lee. Learning to Weight Imperfect Demonstrations. ICML, 2021.

[2] Fumihiro Sasaki and Ryota Yamashina. Behavioral Cloning from Noisy Demonstrations. ICLR, 2021.


summary_of_the_review: I overall like the idea and execution in the paper. I rate the paper as acceptance. Still, there are issues that need to be addressed (related work, baseline methods, and clarity). Nonetheless, these issues are minor and should be straightforward to address. 

** Update after rebuttal **
I read the rebuttal and the other reviews. The rebuttal and the revision address my questions and concerns. I think the paper should be accepted.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper considers the problem of imitation learning, and specifically the setting where a small number of expert demonstrations is paired with some amount of suboptimal data. To tackle this setting, the paper proposes to optimize a convex constrained optimization, with linear variables denoting d^pi, linear constraints expressing the MDP transitions, and an objective composed of a linear combination of KL from the expert and KL from the suboptimal data. The paper presents derivations transforming this convex constrained optimization to a practical, unconstrained objective. The final algorithm is composed of three stages: (1) train a discriminator to get a density ratio estimator; (2) use the density ratios as a reward in a separate log-expected-exp objective (similar to REPS or ValueDICE objectives) to learn a different density ratio w; (3) extract a policy from w using weighted max likelihood training on the suboptimal data. The algorithm is evaluated on a variety of simulated robotics (mujoco) environments, and shows favorable results compared to baselines. 
main_review: Strengths:

-- The proposed setting is practical and important.

-- The proposed algorithm is novel (although heavily derives from existing work).

-- The empirical results & baselines are comprehensive.

-- The writing is clear and easy to follow.

Weaknesses/Questions:

-- The 3-stage process in the algorithm is arguably more complex than baselines (e.g., BC is just a single objective, ValueDICE is a single min-max objective) and can present challenges in practice w.r.t. hyperparameter tuning of each stage.

-- The derivations heavily borrow from existing work (e.g., OptiDICE, AlgaeDICE). At times, the derivations seem to be more complicated than they need to be. For example, when introducing the log-expected-exponent, it is presented as a very involved multi-step derivation. I think it could be simplified by just noting that the Fenchel dual of KL when inputs are restricted to the simplex is exactly the log-expected-exponent function; see Section 5.3.1 in https://arxiv.org/abs/2001.01866 (in fact, this existing tutorial includes a lot of derivations similar to DemoDICE).

-- I have missed this in the paper, but I'd be curious to know how DemoDICE performs in the pure imitation learning setting (no imperfect demos) compared to the baselines.

-- You may want to include a reference to https://arxiv.org/abs/2102.13185 which also proposes an IL algorithm that is composed of a 1st stage that does discriminator-based density ratio estimation and then uses that density ratio as a reward for a later DICE-like stage.
summary_of_the_review: Overall the paper is a good submission, and I would recommend accepting it. I hope the authors can incorporate my feedback into the final version.
review_title: null

Meta-review: The paper presents a method for learning sequential decision making policies from a mix of demonstrations of varying quality. The reviewers agree, and I concur, that the method is relevant to the ICLR community. It is non-trivial, the empirical evaluations and theoretical analysis are rigorous, resulting in a novel method that produces near optimal policies from more readily available demonstrations. The authors revised the manuscript to reflect the reviewers' comments.
Decision: Accept (Poster)