Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents a model-based offline imitation leaning algorithm – DMIL. The method jointly trains a policy, a dynamics model, and a discriminator in an iterative algorithm. The policy and the dynamics model are trained with maximum likelihood on the expert data, with further regularization (corrective loss terms) using the discriminator. The discriminator is trained with a GAN-style objective on the expert and rolled-out data. Crucially, the discriminator includes the log-likelihood of the policy and the dynamics model as inputs to facilitate information sharing. An extension (D2MIL) is also presented for when sub-optimal data is available. Experiments on D4RL MuJoCo and Adroit tasks, and a real-world wheeled robotic task show that DMIL compare favorably to the baseline offline imitation methods especially when expert data is limited. 
strengths_and_weaknesses: Improving the sample-efficiency (in terms of expert data size) of offline imitation learning algorithms is important for their practical application. The paper provides an interesting approach to achieve this and backs it up with nice experiments -- for instance, Table 1 shows that DMIL can achieve competitive performance with only 2% of the original data in Hopper and Walker. Another strong point is the inclusion of results on real robot tasks. I feel that the paper has some shortcomings in the current form – presentation of material and clear differentiation from prior work. These are expanded further below.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: My pre-rebuttal rating is a weak-reject since I have some reservations about the presentation, comparison to prior work, and experiments (see issues below). The paper has several nice elements to it and I would be happy to reconsider after discussion with the authors.
issues: 1. Differentiation from prior work: The “cooperative information sharing” for the discriminator was introduced in DWBC (Xu et al.), where they used $\log\pi$ in the discriminator input. DMIL builds upon this idea by introducing a learned dynamics model and augmenting the discriminator input with the model likelihood. The corrective loss terms follow the same derivation as DWBC. Although I believe that the extensions are important, the current presentation in Section 2.2 could be improved to cover DWBC in a bit more detail and clearly outline the new contributions of this paper.


2. Other suggestions regarding presentation:

* &nbsp; It would really help to move the DMIL algorithm box (Appendix B.1) to the main paper. The algorithm box provides a full picture of the approach that is currently difficult to glean from Section 2. 

* &nbsp; I would also recommend that you move the no-$\log f$-no-$\log\pi$ ablation results from the Appendix (Table 5) to the main paper results, and comment on the performance comparison to DMIL. This seems to be an important ablation to justify the “cooperative information sharing” argument.

* &nbsp; Section 2.3: some details could be included here to make the paper self-contained. For instance, the acronym “PU” is not expanded, $\eta$ is not defined, etc.

3. MuJoCo environments – it’s nice to see that for Hopper/Walker/Cheetah, using only about 5% of the expert data recovers the performance of using the full data. However, I wonder if this is largely due to the nature of the environments – they are deterministic and involve repetitive (cyclic) motions, so only a few expert transitions are usually sufficient. On the more complicated tasks, it seems there’s still big gap to the expert’s performance (especially hammer/door). Could the authors give some intuition/ideas for this?

4. Clarifying question: there’s a relaxation that is made in the derivation of equation 6, where $\frac{\partial d}{\partial \log\pi}$ is ignored. In an iterative algorithm, can this term be computed for the current discriminator using the auto-differentiation tools and used for the policy/dynamics gradient?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper introduces an approach to offline imitation learning that extends a dynamic forward model with an adversarial discriminator. The idea is to use the discriminator to estimate the forward model correctness and optimality of the model rollouts at the same time. Additionally, the authors extend the method to incorporate sub-optimal data as well.
strengths_and_weaknesses: # Strengths
* The method is novel. I like the idea of re-evaluating the correctness of the predictions of the forward model using a discriminator. The existing techniques use online interactions (see [1, 2]) instead of a model that is accurate and does not require re-evaluation. Or they use ensembles of forward models, which are computationally expensive (PETS, [3]). Using a discriminator mitigates the issues associated with computational costs of PETS.
* The method is theoretically sound and outperforms on the baselines. In particularly, the method outperforms the baselines in the low-data regime which is considered to be the most challenging for offline learning.

# Weaknesses
* The clarity of writing can be improved. It is not clear how the method is implemented. The paper can be improved by adding a pseudo-code for the algorithm.

[1] Generative adversarial imitation learning, J. Ho, S. Ermon

[2] Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning, I. Kostrikov, K. Agrawal, D. Dwibedi, S. Levine, J. Tompson

[3] Deep reinforcement learning in a handful of trials using probabilistic dynamics models, K. Chua, R. Calandra, R. McAllister, S. Levine
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: In general, I recommend this paper for acceptance. However, the clarity of presentation can be improved. See my comments above.
issues: * Clarity of presentations. See the comments above.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents a new approach for offline imitation learning. The key idea is to learn a dynamics model along with the policy. To do so they use a discriminator that takes into account both dynamics and policy simultaneously. They show that this approach improves performance of agents even when the number of expert demonstrations available is low. They also show on a real robot their methods works better than other baselines.
strengths_and_weaknesses: Strengths
1) The proposed approach has solid performance over different expert dataset sizes. 
2) The proposed method also outperforms many different offline imitation learning algorithms. 
3) One of the baselines proposed by the authors "2 phase BC+d" is a very reasonable baseline that outperforms more complicated algorithms. More details on this baseline would be useful.
4) Real robot experiments showing the effectiveness of their approach.

Weakness
1) Ablation experiments missing the show the importance of each term in the loss function. This will give more insight into why their method is performing so well. It seems only in Fig. 5 when the authors test the robustness of policy with noisy datasets, they have an ablation experiment. 

Minor Weakness
1) "performance degeneration under limited expert data due to covariate shift" - this statement in the abstract is not backed up by any experiment. Performance degeneration is usually due to drift from expert demonstrations' data distribution rather than covariate shift.


robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The proposed approach performs well when the number of demonstrations available are low and there is noise present in the data. The authors have made extensive comparison with existing methods. They have also shown their method works on a real robot. 
issues: 1. Adding ablation experiments with different loss terms and sensitivity of their method to different hyper-parameter settings.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a model-based offline imitation learning method that trains a model, a policy and a discriminator in both a cooperative and adversarial way to improve the robustness of the dynamics model and the the performance of the policy. The method is evaluated in simulated domains with both expert demonstrations and sub-optimal offline data settings and is shown to be superior compared to prior offline imitation learning methods. The authors also include the evaluation on the real-world wheel-legged robot where the method appears to be effective.
strengths_and_weaknesses: **Strengths:**

1. The paper presents a new model-based offline method that trains a discriminator along with the learned policy and dynamics model, unlike prior model-based IL methods. The discriminator takes the dynamics model as an additional input, which is different from the prior discriminator-based IL works [19]. Such a scheme seems to effectively achieve a cooperative yet adversarial game among the discriminator, dynamics model and the policy. Moreover, the method achieves strong results in the simulated D4RL domains with fully expert or human data.

2. The authors include ablations that show the importance of $L_corr$ and the log-likelihood of the dynamics model as inputs to the discriminator, showing the effectiveness of the algorithmic contributions in practice.

**Weakness**: 

1. Despite that the method gives the cooperative yet adversarial game, the technical part of the method appears to be a model-based version of the prior work [19], which is rather straightforward and makes the novelty a bit limited. The derivation of the algorithm is also heavily based on [19].

2. I'm not sure why comparisons to [19] are not included in Table 1 and Figure 19 as the difference between the proposed method and [19] is rather small in Figure 5 and 6. It is important to compare to [19] everywhere as it appears to be the strongest baseline. Moreover, I think it is important to compare to some model-based IL methods such as [35, 36]. While these methods are proposed to be online, they should be easy to be applied in the fully offline setting. I also think DemoDICE [45] should be a good baseline to include as it handles offline IL with suboptimal data.

3. Finally, I wonder how the hyperparameters are tuned. The method introduces lots of hyperparameters and it's known that tuning hyperparameters in offline IL/RL usually requires online rollouts, which kind of defeats the purpose of fully offline training particularly in the case where there are so many hyperparameters to tune.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Based on the Strengths And Weaknesses section, I'm leaning towards a weak reject of the paper but I can change my rating if all the concerns are properly addressed.
issues: 1. Compare to model-based baselines such as [35, 36] and also DemoDICE [45].

2. Include comparisons to [19] in all domains.

3. Clarify hyperparameter selection schemes.
review_title: null

Meta-review: metareview: Paper proposes a method for offline imitation learning by setting up a three party game. The problem is definitely interesting to the CoRL community, the presentation is largely clear. The reviewers are generally enthusiastic about the method and appreciate real-world experiments. However, the main drawback is missing comparisons from [19], pointed out by two reviewers and other clarification questions. 

**Post Rebuttal Update**
Authors satisfactorily compared their method to [19]. Given these evaluations, the presented work is a meaningful contribution to the growing body of work in offline imitation learning. 
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)