Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the reward extrapolation challenge in offline inverse reinforcement learning. The setting considered is access to a dataset of expert demonstrations as well as an offline dataset, collected by a behavior policy that is not necessarily an expert policy. The authors propose a conservative IRL algorithm called CLARE that integrates conservatism into the learned reward function and exploits learned transitions. They provide some analysis of their approach and show improved empirical performance over prior methods on continuous control tasks.

strength_and_weaknesses: **Strengths:**

- The topic of IRL and issues related to covariate shift and generalization are important and of interest to the community.
- Empirical results show significant improvement over prior inverse offline RL methods.

**Weaknesses:**
- Novelty and contributions are limited. The benefit of combining expert and offline data via pessimism on rewards and model-learning is already introduced in prior work such as Chang et al. 2021. The multiplicative weights are also used by Yu et al. 2021. No new insights are revealed, e.g. the discussion is limited to the well-known covariate shift issue in offline learning, penalizing rewards of less visited state-actions pairs, etc. 
- Theoretical results are very weak. An important issue is that while the paper focuses on issues such as covariate shift and generalization in high-dimensional and/or continuous environments, in several places state-action pair counts are used such as empirical data distributions based on counts (below Eq. (2)) and Assumption 4.1. These are not aligned with the high-dimensional problem considered where typically $|D(s,a)| = 0, 1$. Indeed, Assumption 4.1 can only hold in tabular using concentration inequalities. Furthermore, the statements are not very interpretable and do not give bounds on standard quantities such as suboptimality of learned policy with respect to target policy expressed in terms of sample size, data coverage, and hypothesis class cardinalities. Theory is also not very insightful e.g. Theorem 4.2 is very similar to Theorem 4.4 in Yu et al. 2020.
- I don't think it's meaningful to discuss exploration (e.g. in the abstract, Figure 1, Introduction, etc.) in the context of purely offline learning from datasets. Running iterations on a learned model is not exploration.
- Generally, the paper makes several strong claims that are not supported by either the analysis or experiments. For example, it is not correct to say that optimizing the upper bound leads to balancing exploration-exploitation. Or the findings do not show the algorithm "can effectively capture the expert intention" which is claimed in the paper. The claim in paragraph 3 of Section 3 is not well-supported "it is clear that utilizing a learned dynamics model is indeed beneficial due to its capability of providing broader generalization (compared to model-free counterparts)...". The question of generalization of model-based vs. model-free approaches is still debated and unclear.
- Empirical evaluations can be improved by providing ablation studies such as showing the impact of expert data sample size on performance, the impact of conservative reward weighting, and accessing the impact of offline data on performance. Finally, tests on environments with high-dimensional image inputs on Atari strengthen the work on the empirical aspects.




**References**

Chang, J., Uehara, M., Sreenivas, D., Kidambi, R., & Sun, W. (2021). Mitigating Covariate Shift in Imitation Learning via Offline Data With Partial Coverage. Advances in Neural Information Processing Systems, 34, 965-979.

Yu, T., Kumar, A., Rafailov, R., Rajeswaran, A., Levine, S., & Finn, C. (2021). Combo: Conservative offline model-based policy optimization. Advances in neural information processing systems, 34, 28954-28967.

Yu, T., Thomas, G., Yu, L., Ermon, S., Zou, J. Y., Levine, S., ... & Ma, T. (2020). Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33, 14129-14142.
clarity,_quality,_novelty_and_reproducibility: Clarity is good but the novelty and original contributions are limited. Code is provided.
summary_of_the_review: Despite the improved empirical performance, I did not find algorithmic design and theoretical analysis to be significant. The paper discusses well-known insights and does not provide any new insights.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: CLARE uses a learned dynamics model and conservative policy objective in order to tackle the problem of learning a reward function from a dataset of expert trajectories, i.e. "offline IRL". Their contribution tackles both the reward extrapolation problem in offline IRL as well as improves generalization of the learned policy. The first well-known issue is tackled using the conservative reward update and a notion of safety constrained policy update. Secondly, the generalization to unseen states is handled by learning a dynamics model. Additionally, the authors provide a theoretical framework for balancing the exploitation-exploration tradeoff with a derived upper bound on the return gap between the learned and expert policies. 
strength_and_weaknesses: Strengths:
- Paper is well written - especially in tackling two related, but still distinct issues in offline IRL, it is easy to follow the contributions and their exposition is well done.
- Along the above point, the combination of different techniques to improve overall performance of the offline IRL algorithm is quite a reasonable one, i.e. combinging model-based conservative policy learning into one technique.
- Results are very strong and convincing on known benchmarks
- The pointwise weight parameters is a novel and interesting contribution.

Weaknesses:
- Why is the reward regularizer chosen as r^2. I could not find any further elaboration on this, and the rest of the section on regluarizers discusses the conventional policy regluarization as a KL divergence. Theorem 4.1 states the reasoning behind various reward regularizations, which I could just be overlooking as I am not familiar with the cited works, however, it would be good to hear more about the reasons r^2 was chosen in terms of why it performs well with respect to the datasets used. 
- As far as I can tell, the generalization ability coming from the policy updates under the learned dynamics model is not a novel contribution. I think it would benefit this paper's visibility on the overall contribution if they reframed their approach as being only the conservative reward function learning which is compatible with an off-the-shelf offline RL method rather than a two step update algorithm. 
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written, high quality, and novel. I only have minor concerns about the reproducibility as it involves many moving parts, but as the authors have open-sourced their implementation, I think this should be helpful towards that end. 
summary_of_the_review: Overall, this is a nice paper with empirical and theoretical contributions and I would lean towards acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper addresses the problem of reward extrapolation in inverse reinforcement learning (IRL). The authors propose a modification of the MaxEntIRL to properly address the problem by leveraging an additional dataset collected by a behavioral policy and properly adjusting the loss function, introducing additional terms for assessing the reward function on the samples of this dataset. The loss function depends on a auxiliary function $\beta(s,a)$. A theoretical analysis is presented to show the guarantees of the approach as well as proposing a suitable value of $\beta(s,a)$. An experimental evaluation is presented showing the advantages of the proposed approach over several baselines on multiple Mujoco environments.
strength_and_weaknesses: **Strengths**
- The paper addresses a very important issue of IRL
- The paper provides both empirical and theoretical contributions showing, in both cases, the benefits of the proposed approach

**Weaknesses**
- The rationale behind the loss function of Eq (2) could be explained better
- The "Related work" section should include additional papers (as detailed below)
clarity,_quality,_novelty_and_reproducibility: **Clarity and Quality**
The paper is written in good English and reads well. While the motivation of the paper is properly introduced and the meaning of the loss function of Eq (2) is explained, the connection between the two is not fully straightforward. Specifically, I understand the problem of reward extrapolation, but I cannot fully agree that this problem is addressed in Eq (2). In this sense, I have some questions:
1. Why does the visitation distribution $\tilde{\rho}^D$ consider the union of the datasets $\mathcal{D}_E$ and $\mathcal{D}_B$, when the dataset $\mathcal{D}_E$ is already considered in the distribution $\tilde{\rho}^E$?
2. From an intuitive perspective, the quality of the recovered reward function should depend on the behavioral policy $\pi^B$ used to collect dataset $\mathcal{D}_B$. Does this aspect emerge from the analysis?

**Related Works**
Since the paper considers the setting of the offline IRL, I suggest the authors to include in the "Related work" section the papers that address the "truly batch model-free IRL":

[1] Klein, Edouard, et al. "Inverse reinforcement learning through structured classification." Advances in neural information processing systems 25 (2012).

[2] Pirotta, Matteo, and Marcello Restelli. "Inverse reinforcement learning through policy gradient minimization." Thirtieth AAAI Conference on Artificial Intelligence. 2016.

[3] Ramponi, Giorgia, et al. "Truly batch model-free inverse reinforcement learning about multiple intentions." International Conference on Artificial Intelligence and Statistics. PMLR, 2020.

**Novelty**
Although the approach builds upon a well-known IRL algorithm (MaxEntIRL), the approach shows a notable novelty.

**Reproducibility**
The appendix provides sufficient details to reproduce the results.

summary_of_the_review: Overall, the paper provides a relevant contribution to the IRL literature. The reward extrapolation is a relevant problem in IRL and the approach proposed by the paper is novel and has shown to be effective from both the theoretical and empirical sides. I suggest the authors to expand the explanation of the loss function of Eq (2). My evaluation of the paper is positive.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a model-based reward learning algorithm. The key idea is to smartly balance the use of the expert data and the low-quality data in learning the reward function. Theoretical analysis is provided to justify the weight for each low-quality data. Empirical study also verifies the efficacy of the proposed algorithms.
strength_and_weaknesses: Strength: 
The paper is well written and easy to follow. I in particular like the baseline MOMAX. This is the simplest way to use offline data in the IRL pipeline and comparing with this baseline makes the empirical study more convincing. The proposed algorithm outperforms baselines by a large margin.

Weakness:
1) It is not clear why the y-axis in Figure 2 means. I presume it is the performance of the learned policy, since the algorithm needs to alternatively learn \pi and r. But I do not understand why BC is included, since it is not a inverse RL algorithm at all.
2) Table 1 may benefit from having second order (e.g., variance) and averaged statistics (e.g., expectation) instead of just the highest score
clarity,_quality,_novelty_and_reproducibility: I think the paper is of good quality and clarity and should be easy to reproduce.
I am however not familiar with IRL literature so cannot comment on novelty, as well as the selection of baseline algorithms.
summary_of_the_review: see above
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper addresses the problem of offline inverse reinforcement learning, where learned reward estimates may be biased due to data mismatch. The paper proposes an approach based on conservatism and a careful balance between multi-level tradeoffs. The authors provide theoretical guarantees for the tabular setting and strong empirical results in the MuJoCo domain.

Reviewers highlighted the importance of the addressed problem, interesting approach and strong empirical results. At the same time, reviewers initially raised a number of concerns. Concerns ranged from needs for clarification to larger concerns about the novelty of the work and the significance of the theoretical results.

The authors responded to reviewer concerns during the rebuttal period. The responses have been well received, and clarification questions have been addressed. Remaining questions regarding novelty and the relative importance of the theoretical contribution were discussed among reviewers.

In the camera ready version of the paper, the authors are encouraged to emphasize the strong empirical results that their proposed method achieves. Regarding the theoretical results, the authors are encouraged to carefully discuss limitations, e.g., restriction to the tabular setting. 

Overall, the paper addresses and important problem in an interesting and novel way, and achieves strong empirical results. On this basis, the paper is recommended for acceptance.

Noted typos:
- "continuous environments Jarrett et al. (2020)"
- "state-actions that characterized by larger"

justification_for_why_not_higher_score: The paper proposes a principled approach to an important problem in offline inverse reinforcement learning, taking one step beyond prior work. For a higher score, I would expect a more substantial contribution, such as a larger delta from previous work.
justification_for_why_not_lower_score: A main concern was regarding the theoretical contribution. The reviewers and AC acknowledge that (inverse) RL theory that goes beyond the tabular / more limited setting is difficult and rare. In this context, having even limited theoretical insights is valuable. Rather than rejecting the paper, the authors are encouraged to carefully discuss limitations of their theoretical analysis.
Decision: Accept: poster