Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces an offline imitation learning algorithm. The proposed algorithm consists of two parts: (1) to train a world model using demonstration of any quality and (2) to train a policy using $\lambda$-TD where the reward function is computed as Equation (7) (i.e. intrinsic reward). Finally, the authors empirically show that the proposed method outperforms baselines in offline setting.
strength_and_weaknesses: ## **Strength**

DITTO shows good empirical results.

## **Weakness**
**Algorithm**

- DITTO uses an intrinsic reward function. However, there are no justification to use this reward function. At least there should be some consideration of the ideal case (e.g. in theory, DITTO is equivalent to a distribution matching between current policy and expert policy)

**Experiments**

- In Figure 3. D-BC always achieves near-expert performance when the number of expert trajectories is large enough. Compared to D-BC, DITTO shows slightly better performance. Therefore, it seems that more experiments are needed to show the superiority of DITTO.
- I am not sure if it is a good thing to outperform the expert in (offline) imitation learning. Why DITTO exceeds average expert performance?

**Related Works**

- One of the simplest ways to solve offline imitation learning with learned world model is to use HIDIL [1], which aims to solve an offline imitation learning problem given a misspecified simulator.
- In addition, existing (model-free) offline IL algorithms [2-4] can also be used for policy learning on learned world model.

[1] Jiang, Shengyi, Jingcheng Pang, and Yang Yu. "Offline imitation learning with a misspecified simulator." NeurIPS. 2020.

[2] Kim, Geon-Hyeong, et al. "DemoDICE: Offline imitation learning with supplementary imperfect demonstrations." ICLR. 2022.

[3] Xu, Haoran, et al. "Discriminator-weighted offline imitation learning from suboptimal demonstrations." ICML. 2022.

[4] Ma, Yecheng Jason, et al. "SMODICE: Versatile Offline Imitation Learning via State Occupancy Matching." ICML. 2022.
clarity,_quality,_novelty_and_reproducibility: Using intrinsic rewards as in Equation (7) is interesting and novel.

However, theoretical justification seems necessary.

summary_of_the_review: Solving offline imitation learning is an important, but the proposed algorithm is too empirical.
This paper uses only 5 Atari domains and there is no theoretical analysis.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies imitation learning in partially observable environments. The proposed method is to learn a world model using a dataset of any quality, unroll the agent's policy in the latent space of the world model to simulate the agent's trajectory, and train the policy with a reward function to make the agent's trajectory close to the demonstration trajectory in the latent space.

Specifically, the world model follows the recurrent neural network to predict future latent states, and the world model is trained with ELBO objective, consisting of the KL divergence between the prior and posterior distributions of latent states, and reconstruction error for the raw observation. The agent is trained via the on-policy actor-critic algorithm, where the policy is executed in the latent space of the world model. The dense reward signal penalizes the distance on the latent state densities between the agent's trajectory and the expert demonstration trajectory.

Experiments are conducted on five Atari games and the proposed is compared with behavior cloning, and generative adversarial imitation learning, showing advantages over the baselines in three out of five environments.
strength_and_weaknesses: Strengths:
The method is well-motivated to study the covariate shift issue in imitation learning, which is an important and interesting problem.
The proposed method is technically solid. The learning of the world model and on-policy actor-critic is well executed to make the relatively complicated pipeline work in Atari games. 
The paper writing is mostly clear and easy to follow.

Weaknesses:
The performance improvement in only three out of five Atari games is not very impressive.
The statement "address the problem of covariate shift" in the abstract is not well-supported by the experiments. Regarding the experiments, it is unclear whether the bottleneck for baselines is covariate shift, and how much covariate shift can be addressed by the proposed method. 
The evaluation and analysis can be improved. For example, how does the quality of the dataset affect the quality of the world model, and how does the quality of the world model affect the final performance of imitation learning? I doubt "the quality of the policy used to generate world model training episodes did not affect final performance". See below for more detailed questions and comments. 

clarity,_quality,_novelty_and_reproducibility: Clarity:
The method is described clearly except for several details.
In Equation (5), \hat{M} is undefined before. I guess it is a typo and it should be M?
About the reward design, it is mentioned that "unlike L2, the formulation is independent of the scale of the underlying feature space". However, it is also stated that "maximizing the max-cosine objective is equivalent to minimizing the L2 norm between two vectors". So the relation between your reward formulation and L2 distance is confusing. It will be better to explain this paragraph in the Appendix with the corresponding mathematics formulas and derivations.

Quality:
Because the proposed method uses both datasets of any quality (for world model learning) and expert demonstration data (for policy learning), I'm curious how the proposed method compares with model-based offline RL algorithms, given the same dataset with trajectories of different qualities.

The statement that "the quality of the policy used to generate world model training episodes did not affect final performance" seems overclaiming to me. For example, in Atari games Montezuma's Revenge, the environment is composed of 24 rooms in the first levels. When the quality of datasets for world model learning is moderate, many rooms will not be covered in this dataset. Thus, the world model never sees some rooms during training. With great expert demonstrations covering all these 24 rooms, will the world model fail in the unseen rooms? How does the proposed imitation learning approach perform in these rooms?

It will be great to see more analysis of the proposed method. As for world model learning, how do the number and quality of datasets affect the final performance? As for policy learning, how does the agent training horizon H affect the performance? It seems surprising that the policy can unroll in the world model for H=15 steps to predict future value.

Regarding the reward design for intrinsic reward, could you compare it with some existing reward designs based on latent space distance, such as https://proceedings.neurips.cc/paper/2018/file/35309226eb45ec366ca86a4329a2b7c3-Paper.pdf? The reward in this work is to calculate the step-by-step distance, which might be too restrictive to encourage the agent exactly follow the demo. When the environment is stochasticity (not sure whether the environment is deterministic or stochastic in the current experiment), will this kind of reward design hurt the final performance?

Novelty:
The novelty is okay. Although learning the world model and reward design according to distance in latent space is not novel ideas, it is novel to combine them in the model-based imitation learning setting.

Reproducibility:
Code for experiments is provided in Appendix. So the reproducibility is okay. The main issue is to apply the proposed method further for more environments because there is no ablative study about hyper-parameters and we have no idea how to tune the method in new environments.
summary_of_the_review: Overall, this paper proposed a technically solid method for imitation learning, but the evaluation can be better if conducted in more environments, with more datasets, and more design choices for the reward function.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces DITTO, a method for fully offline imitation learning in POMDPS high dimensional observation spaces. The idea progresses by first learning a dynamics model or 'world model'. Then, an RL agent is trained in this latent space to minimize the divergence to the learned state-action density to the expert state-action density. 
strength_and_weaknesses: Strengths
+ The proposed method is very neat, and explained well. The improvement obtained from an explicit model-based approach is great to see, and encouraging for model-based RL in general.
+ The theory is nice to have, as is the discussion of the fundamental challenges facing imitation learning. 
+ The experimental results seem promising to start with. 

Weaknesses
+ The authors write 'to the best of our knowledge we are the first to study completely offline imitation learning without behavior cloning in high-dimensional observation environments'. In fact, there are several prior works which deal with this setting. They include ValueDice [1], IQ-Learn [2], EDM [3] AvRIL [4], all of which achieve decent performance. There are almost certainly more, of which you can probably identify most by looking at the papers that cite [3] .

  [1] Kostrikov et al, Imitation Learning via Off-Policy Distribution Matching, ICLR 2019

  [2] Garg et al, IQ-Learn: Inverse soft-Q Learning for Imitation, Neurips 2021

  [3] Jarrett et al, Strictly Batch Imitation Learning, Neurips 2020

  [4] Chan et al, Scalable Bayesian Inverse Reinforcement Learning, ICLR 2021

  Several of these methods are non-adversarial and so do not suffer from the (very reasonable) issues raised by the authors with regards to GAIL. 

+ I think I may be misunderstanding, but is there a reason that RL is required to learn the actor in the world model? It seems that if everything is controlled and latent, we can just differentiate through the policy directly (using a Gumbel-softmax) for issues with discrete distributions.
clarity,_quality,_novelty_and_reproducibility: Clarity

The paper is clear enough, but a bit verbose in places (eqns 11-14 could be cut, for instance)

Quality

The paper is relatively high quality, and the work is quite polished. 

Novel 

As far as I am aware, this approach is novel. There does seem to be a bit of similarity to the EDM approach, which might be worth clarifying.

Reproducibility

Code is provided, so the approach seems reproducible.

summary_of_the_review: The paper proposes a new method for explicitly model-based offline imitation learning. The method is exciting and shows promise. However, the experimental results are weak, not evaluating on the state-of-the-art modern offline imitation learning baselines. This is understandable since the authors were not aware of these methods. However, I don't think that a paper which lacks comparison to current benchmarks should be recommended for acceptance. I would be very happy to reconsider if the authors could provide comparisons to the methods mentioned above, and demonstrate how DITTO compares.

Update after rebuttal:

Thanks to the authors for the replies and additional baselines. Based on these, I have increased my recommendation to 6.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an offline imitation learning (IL) method supported by a word model. The world model is obtained by training on the transitions collected in a dataset, without any direct access to the actual environment. A intrinsic reward is also designed that measures the divergence between expert and agent in the trained word model. The approach is well motivated. Empirical results shows that the proposed method outperforms baselines adapted to the world-model, namely GAIL and BC.
strength_and_weaknesses: Strength:
1. The paper is well written and easy to follow. All elements of the proposed model appear well-motivated. The proposed method is sound, and appears a novel combination of various techniques.

2. Empirically, the evaluation shows the proposed method outperforming the baselines appropriately adapted to the world-model setting.

Weakness:
1. The empirical evaluation, while carefully designed, is limited in scope and does not explicitly highlight/explain why the proposed method outperforms the baselines. For instance, my understanding is that the only difference between GAIL and DITTO is that GAIL uses adversarial reward. However, from this singleton comparison it is difficult to assess if adversarial reward is worse than intrinsic reward. Related to this, there are various recent adversarial formulation that outperforms GAIL and they should also be considered in the evaluation to more broadly evaluate the impact of the reward formulation. As an example, PWIL [1] is fairly similar to DITTO in terms of how reward is computed.

2. The quality of the world-model and reward functions are not independently analyzed. In the current experiments, the size of the dataset directly affects both the quality of the world model and of the reward. However, it is important to separate these two effects, as the authors also argued that learning the world model can be done using transitions of any quality. Potential ablation may include separate the datasets used for world model and reward computation to isolate the two effects.

3. The method appears directly applicable to Mujoco environments. It would be nice to see if world model is still beneficial in a vector-based observation/state setting.

Overall, any additional experiments to improve our understanding of the proposed method and why it outperforms existing ones would be beneficial.

[1] Primal Wasserstein Imitation Learning, Dadashi et al. ICLR 2021
clarity,_quality,_novelty_and_reproducibility: The paper is well written and motivated. The proposed method appears to be a novel combination of various existing components. The practical algorithm is described in details with source code included for reproducibility.
summary_of_the_review: The paper proposes a fully offline IL method by training a world model and leverage standard RL for imitation learning. The method is well motivated and shows improved performance over baselines. Additional ablation and experiments are needed to better understand the advantages of the proposed method.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes combining the idea of learning a world model with imitation learning to help mitigate the covariance shift problem in imitation learning. The key idea is to reformulate the imitation learning problem as an on policy optimization problem in the latent world model space. The counterpart framed in this paper to this approach is vanilla behavior cloning and GAIL (generative adversarial imitation learning). The paper does a good job at showing that this approach has promise and can do better than vanilla imitation learning. Their method is compatible with these approaches and augment them in interesting ways as shown in the experiments. 

The key experimental result is to show that the world model formulation leads to better performance on 5 atari games. The writing is reasonable but needs more rigor so others can build on it.

The biggest current limitation of this paper is experimental validation. I would encourage the authors to experiment on more domains so the generality of this approach is demonstrated and validated more carefully. Using world models for imitation learning is an extremely intereting and important topic for reinforcement learning research and this approach has a lot of promise. 
justification_for_why_not_higher_score: Needs more experimental validation, expand and rewrite to make it more rigorous and test on a wider variety of environments.
Decision: Reject