Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This work considers an offline imitation learning setting where both expert and sub-optimal demonstrations are assumed to be known. While the covariate shift issue (the gap between training and test distributions) is a critical problem in offline imitation learning, e.g., behavioral cloning, authors address this issue by training a dynamics model through the sub-optimal demonstration (having state-action-successor state triples) and applying pessimistic penalty (incentivizing visiting the state-action pairs in the expert demonstration). With the theoretical derivations for various settings for dynamics modeling and the assumption that the demonstration does not have to cover the entire state-action space, authors show the theoretical guarantee of their proposed method. Also, the empirical results support their claim by showing that MILO (proposed algorithm) outperforms ValueDICE and BC.
main_review: I’m generally satisfied with the submission due to the authors’ clear writing, various experiments (including ablation studies like the impact of coverage), and rigorously defined mathematical terms. While I believe the submission is sufficient to be accepted, there are minor comments below:
- I wonder why the authors did not check the performance of other recent baselines on offline imitation learning that cite the ValueDICE paper. For example,
  - Jarret et al., “Strictly batch imitation learning by energy-based distribution matching,” (NeurIPS 2020, the reference [28] in the submission)
  - Chan et al., “Scalable Bayesian inverse reinforcement learning,” (ICLR 2021, the reference [13] in the submission)
- The citation style is not consistent with the normal NeurIPS citation style.
- Would you please increase the size of figures and fonts in Figures 2, 3, and Table 1,2?

limitations_and_societal_impact: The authors pointed out the negative biases that are inherently given in expert demonstration, which I believe hasn’t been addressed in the submission. I couldn’t clearly imagine how to perfectly solve this but I believe recent works on using ranking in expert demonstrations might be helpful.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies imitation learning without access to the environment. The agent is provided with two datasets: one containing expert policy state-action pairs and one containing state-action-next state triplets collected from the environment. The authors propose the algorithm MILO that leverages pessimism to mitigate covariate shift and provide performance upper bound in discrete MDPs, kernelized nonlinear regulators, and Gaussian processes. The authors show improved performance achieved by MILO compared to other baselines such as behavioral cloning in MuJoCo control tasks. 
main_review: Originality:
- Theory on non-parametric models in offline RL is new.
- The authors derive bounds depending on the relative condition number w.r.t. expert policy in the KNR setting. In linear function approximation, the presented bound depends on the rank of offline data instead of assuming a full rank covariance. 
- The authors thoroughly discuss related work. 
- Novelty in algorithmic design is limited (e.g. can be considered a natural variant of MOPO in IL utilizing a cost function class).

Quality:
- The claims are supported with both theoretical analysis and empirical studies.
- BC can perform much better than MILO in certain scenarios and it is unclear how one should choose between these two algorithms based on the datasets. For example, when the expert dataset has good coverage but the offline dataset doesn’t, MILO performs worse than BC by avoiding parts of the environment that are not covered in the offline dataset. On the other hand, when the expert dataset is extremely small but the offline dataset is large with good coverage on expert policy, MILO performs well (e.g. the setting considered in the experiments).

Clarity:
- The paper is very well-written and easy to follow.

Significance:
- The message that pessimism only requires partial coverage is already discussed in prior works. 

Questions/comments:
- Below Theorem 3, the authors compare MILO upper bound $O(H/\sqrt{N})$ with BC upper bound $O(H^2/\sqrt{N})$ in [3]. However, the upper bound of BC in [3] is not tight and the comparison is not fair. A tight upper bound of BC is $O(H^2/N)$ given in [47] which is much faster.
- Could you explain the particular choice for characterization of expert policy coverage, e.g. using a ratio in discrete MDP setting? Why didn’t you use alternative measures such as the ones mentioned in Remark 26 in Appendix C.3?

Minor points:
- $b(s,a)$ is referred to as penalty and bonus interchangeably.
- Should cost function class in line 103 be $\mathcal{F} \in \mathcal{S} \times \mathcal{A} \rightarrow [0,1]$?
- A few capitalization errors in the references.

___
After the rebuttal:

I thank the authors for responding to my questions. I am happy with the answers and stick to my score in favor of accepting this paper.  
limitations_and_societal_impact: The authors discuss the limitations and potential negative societal impacts.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors present MILO an offline imitation learning algorithm that uses a learned model of the environment to generate trajectories from arbitrary policies. This allows to reuse similar algorithms than in the online case such as GAIL or Wasserstein GAIL.  In addition, the authors use a bonus (pessimism) to penalize data that is generated in places with high model uncertainty. This can be achieved using an ensemble. They present experimental results as well as an ablation study on OpenAI Gym.  
main_review: I think the paper is still not ready for publication in a venue such as Neurips for the following reasons.

First, the literature may have not been studied properly. Indeed,  the idea of offline imitation or offline inverse reinforcement learning using a set of expert trajectories and a set of non-expert trajectories to add dynamics information is not new :

 - A Cascaded Supervised Learning Approach to Inverse Reinforcement Learning (ECML 2013)

Unfortunately, this paper is not cited.  Also the authors do not cite DART (DART: Noise Injection for Robust Imitation Learning) which is also one way to avoid covariate shift by adding noise when collecting trajectories from the expert.  Finally the idea of framing the IL/IRL problem as a min-max problem dates at least from the 2004 seminal paper of Peter Abbeel (Apprenticeship Learning via Inverse Reinforcement Learning) and could have been cited. 

Second, the experiments section should prove the main point against online methods raised by the authors in the introduction: "online interactions are often costly and prohibitive for real world applications where active trial-and-error exploration in the environment could be unsafe or impossible". Unfortunately, I was not convinced. Indeed, in the experiments section the non-expert data set to train the generative model uses 1 million transitions. Therefore, one possible experiment could be to see how much data an online imitation learning algorithm such as GAIL needs to generate to reach the same performance as MILO and show that MILO is at least one order of magnitude below that. Without this experiment we can't really judge how data-efficient is MILO compared to GAIL. One other interesting experiment could be to see what is the proportion of low-score trajectories generated by GAIL compared to the dataset used to train the generative model. This experiment would tell you how unsafe GAIL is compared to MILO. Those two experiments could prove clearly the point you were making in the introduction. I encourage strongly the authors to run those experiments which will make the paper stronger.

Third, you should have an offline IRL or IL baseline (such as CSI) and not baselines that are artificially weakened such as behaviour cloning baselines with sparse expert data. It is known that this is an adversarial setting for these methods. The same goes for MILO when the non-expert data is random which results on poor results as shown in Table 2. 

Fourth, the analysis section could be moved in the appendix section because it covers cases such as discrete MDPs and GPs that are not related to the practical algorithm presented and used in the experiments. Indeed different instanciations of the general algorithms are presented and this can be confusing for the reader. For instance the algorithm is presented with and without behaviour cloning loss and then the exact cost function is only provided in the appendix. An exact and detailed description of the algorithm used for the experimental section should be done in the main paper.  In addition, the authors should focus more on how could we learn the model \hat{P} on more challenging environments. This is the main bottleneck of this algorithm. This scaling problem seems quite challenging if the states are images or high-dimensional in general. How much data will be needed in that case? Should this model be learned at the latent level?

Overall I think the paper has some strong points like the idea of learning the bonus b with an ensemble of networks and could be published in a top-tier conference. However, there is still a lot of work on the experiment section to show quantitatively what is the advantage of using offline methods versus online methods and on the method section to present more clearly the exact algorithm.

limitations_and_societal_impact: Yes the authors address the limitations and potential negative societal impact of their work adequately.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work proposes a new method for offline imitation learning. The setup is slightly different than usual as it supposes that, additionally to the expert dataset we want to imitate, we have access to a large dataset of non-expert interactions.
The method is the following: 

1) learn a model of the transition kernel from the data.

2) Along with the model, learn an “uncertainty penalty” (the converse of an exploration bonus) that will encourage the agent to stay on the expert support.

3) Train *offline* a min-max model-based policy-optimization with the uncertainty penalty. 

The contributions also include theoretical analysis on the proposed method in simplified cases.
main_review: Thank you for this interesting work. A few questions, particularly on the experimental part:

Methods:

- Why don’t you also use the expert dataset for training the model? I see no interest in leaving it out.

- Have you thought about any connection to the GAIL algorithm? What you call the pessimistic model-based offline IL is, I am not mistaken, an instantiation of the “adversarial imitation learning” framework with trajectories played in the model rather than online, and an additional non-exploration bonus. 

Experiments:

- It is said (both in title and analysis) that the method does not require full coverage of the state space. I would have appreciated to see an experiment with the results of the methods when decreasing the size of $D_o$ (which is really important: 1M samples is not quite a “partial coverage” for these locomotion environments).

- I find it slightly problematic that no presented baselines make use in a principled way of $D_o$ (BC just learns from it like it was expert data). In particular, one natural baseline (feel free to disagree) to me would be to train any offline RL learner on $D_e+D_o$ with the following reward: 1 for any sample in $D_e$, 0 for any sample in $D_o$ (in the spirit of SQIL [1]). In a GAIL-like manner, you could also define the reward as $log(D)$ with D a discriminator between $D_e$ and $D_o$ and train an offline agent on  $D_e+D_o$ with this reward function. 

- It would be great to detail in the appendix how hyperparameter selection was performed. In particular, I would encourage you to check out [2], that encourages not to select hyperparameters on the underlying reward function when performing imitation.

- Can you clarify Appendix E.1? Why did you modify the base environments most people in the community use? I find this quite surprising and I do not feel comfortable with this. According to me, you should either switch back to the widely used environments or justify very clearly why you needed these modifications.

- From Fig.2 it looks like the method is initialized with BC. As far as I can tell, this is not specified in the paper. Otherwise, why is MILO’s curve not starting at random behavior’s performance?


I would increase my score if you can provide clear explanations concerning the experimental setup. Bonus if you can elaborate a bit on the link to GAIL and add these additional baselines.



[1] Reddy, S. et. al., 2019. Sqil: Imitation learning via reinforcement learning with sparse rewards. arXiv preprint arXiv:1905.11108.
[2] Hussenot, Leonard, et al. "Hyperparameter Selection for Imitation Learning." arXiv preprint arXiv:2105.12034 (2021).



# Update after rebuttal

I would like to thank the authors for clearly answering my questions and providing additional experiments/ablation supporting the method's results.

I encourage the authors to include the feedback they provided here in the paper itself.

I would say that the methodology for hyperparameter selection is not following best practices (e.g. taking best of with/without BC init) and I encourage the authors to improve this for future works. Yet, most of my concerns were answered and, although HP selection could be improved, the presented results look valid.

I will thus increase my score to 6. 
limitations_and_societal_impact: No direct societal impact.
review_title: null

Meta-review: The reviewers in most part agree that paper is well-written, is tackling an interesting problem and also the authors have engaged constructively during the reviewer feedback period and addressed the main concerns and questions. Please address the minor points raised by the reviewers in the final version.
Decision: Accept (Poster)