Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposes a model-based offline IRL approach based on Bayesian theory of mind where the learner must estimate the demonstrator's reward function and dynamics. The authors show a connection to robust MDPs and demonstrate their algorithm on a grid world and on D4RL benchmarks.
review: Strengths:
+ Empirical results show that the BTOM and RTOM approaches outperform an ML-IRL baseline.
+ Authors derive an approach that scales to high-dimensional tasks.

Weaknesses:
- I do not understand why the authors use a BTOM approach. There is no motivation for this approach and it does not seem to match the experimental domains.

- The authors do not assume that the expert has access to the true parameters of the environment, but all experiments assume this.
- Some of the derivations are hard to follow or not explained.

- The approach seems identical to other approaches except for the addition of the prior. However, as noted later the prior seems hard to justify in the setting of BTOM.

---
Post rebuttal: 
I think the paper is stronger, and I think it is better framed by not using TOM wording (unless the authors were to address cases where the expert does not have a good model of the transition dynamics). However, I still do not see a significant win over ML-IRL and as such I'm not sure when this method would be chosen over existing methods. I think the paper needs to more clearly show why/when a Bayesian approach is better. Some ideas are to add experiments showing where ML-IRL has a significant drop in performance but the proposed Bayesian approach maintains high performance because of the robustness property. Other ideas include showing that the proposed approach is more data or computationally efficient. Prior work has shown that reward learning methods that seek point estimates are less robust than Bayesian robust counterparts (https://arxiv.org/pdf/2106.06499.pdf) and I would like to see something similar for this paper. Bayesian methods are also often useful for active learning and uncertainty quantification, but there is no evidence of this benefit in the current paper. I maintain my recommendation of weak reject.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The empirical results show an improvement; however, the general approach is poorly motivated and not studied in the setting most appropriate for BTOM. It is not clear why the approach works better than other approaches nor whether it will work in cases where the demonstrator does not have access to the true dynamics. The authors only compare against one baseline. Furthermore the mathematical details are often hard to follow.

---
Post rebuttal: see above notes in review. I think the paper needs more convincing results that there is value to the proposed approach as opposed to ML-IRL. Currently the gains seem mostly marginal and given the error bars it's unclear whether they are statistically significant. I maintain my recommendation of weak reject.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper proposes an approach to inverse reinforcement learning that accounts for the theory of mind of the demonstrator. In particular, this accounts for the demonstrator's subjective (and possibly incorrect) estimate of the dynamics. The authors analyze IRL in this framing and show how it has nice connections to robust learning from demonstrations. They provide experiments in a gridworld and on Mujoco benchmarks that demonstrate the approach.
review: # Strengths

## Quality

The theoretical results in the paper are valuable contributions. I really appreciate the connection to robustness that they identify. I also appreciate the discussion of performance guarantees.

## Originality

The key idea in this paper: doing IRL in a way that estimates the expert's subjective beliefs is novel and promising.

## Clarity

The paper does a good job of setting up and proving the theoretical results.

## Significance

Depending on how useful these methods are in practical situations, this paper has the potential for high impact. It leverages a nice insight into IRL, supports that with solid theory, and demonstrates some initial empirical results. 

# Weaknesses

## Quality

The experiments in this paper are not particularly strong. The results are primarily a comparison of two different ToM approaches, with less comparison of ToM approaches to other approaches. It is possible I have missed something with the experiments, but it was hard for me to identify the benefits of ToM in comparison to other IRL approaches (despite the claim in l.209). In either case, comparing to more baseline methods (clearly grounded in the related literature) would help. 

Beyond this issue, the experiments are not super convincing as they are in simulation for a gridworld and Mujoco. I'm not sure how easy it would be to fix this, but a complex evaluation domain would help the paper. 

Finally, a few related work pieces to include: https://arxiv.org/abs/1901.08654 estimates reward where the observed behavior has a subjective estimate of reward; https://dl.acm.org/doi/10.5555/3327345.3327381 deals with negotiating agents who have different dynamics estimates

## Clarity

It wasn't very clear how one could replicate the experiments or methods. Consider including pseudocode for the method and more details about, e.g., the baseline method. Furthermore, I found the analysis of the results (lines 254-258) incomplete. I recommend adding some more discussion of how and why performance differs across the conditions.

A few specific lines I found unclear:
  - l.116-117] Please say more about the distinction with (3); 
  - [l.197] Please add more English description of the implications of the theorem, I had trouble parsing the math into an intuitive understanding
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I'm somewhat torn about this paper. I think the idea is quite promising, but the presentation and experiments have substantial room for improvement. With the paper as it is, I weakly recommend rejection. However, I think there are several points that can be addressed that improve my opinion of the paper. 

Update after discussion/rebuttal: I’m happy with the authors response and changes. I still think that there is substantial room for improvement on the empirical work, but I value the combination of theoretical results and empirical validation. I’ve increased my score to recommend acceptance.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper introduces a model-based inverse reinforcement learning approach for offline datasets, in which a dynamics model and reward function are estimated simultaneously. The method is motivated by a Bayesian Theory of Mind framework and approaches the problem from a bayesian inference perspective (MAP estimate). Two variants are proposed, BTOM and RTOM, where BTOM solves the full solution and RTOM solves a reduced version under the assumption that the real and estimated dynamics are sufficiently close. Experiments in Gridworld and MuJoCo are performed in which the proposed methods are compared to a prior method, ML-IRL, and show performance improvements in some scenarios.
review: Strengths:
* Using Bayesian Theory of Mind as a motivating framework for model-based inverse reinforcement learning is an interesting perspective. The mapping of the dynamics model to belief and reward to desire is intuitive on its face and differs from that in prior work (although I also feel that this is a bit of aweakness, see below).

* The derivations appear to be correct to my eye, and the connection between the proposed method and prior robust model-based offline RL methods is a nice observation. Intuitively this connection makes sense and provides context for the methods.

Weaknesses:
* While an interesting perspective, I also feel that there is an over-emphasis on the Bayesian Theory of Mind aspect to a point where it detracts from the paper. Although the dynamics model and reward can be mapped to the beliefs and desires of an agent, this seems to me to simply be a motivation for construction a Bayesian posterior (in Eq. 5) as opposed to a full theory of mind model. For example, this work appears to assume that the expert's model of the environment dynamics is stationary, i.e., maps to a fixed belief, as opposed to rigorous theory of mind models which often consider dynamic beliefs in partially observable (POMDP) environments.

* The relationship to prior methods is not entirely clear. Line 44 indicates that the proposed algorithms are extensions of prior works to continuous spaces, but this connection is never touched on again. The loss function defined in Eq. 7 is clearly similar to that defined in [18] (plus an entropy regularizer), so is the primary contribution the sample-based approximation scheme introduced in Sec. 3.1? Does the proposed Bayesian setup contribute to this, or could the sample-based approximations also have been directly applied to the loss functions defined in prior work?

* The experimental results are not very comprehensive, as the proposed methods are only compared to a single prior method (ML-IRL). Given that connections are drawn to robust methods (specifically RAMBO), I would have expected a comparison to at least this method if not others. In addition, the results seem somewhat ambiguous. In some scenarios BTOM performs best, while others RTOM best, and others ML-IRL. Some analysis as to why this is the case would be helpful, e.g., in what scenarios is RTOM/ML-IRL expected to perform better than BTOM.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: As the paper currently stands, I would lean towards a weak reject due to the lack of baselines in the experimental results and the lack of clarity in the methodology. I am open to revising my recommendation though given any additional updates in the rebuttal as I think the paper has a lot of potential.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper proposes an offline inverse reinforcement learning (IRL) framework in which both the reward and transition functions are learned simultaneously from observations in an offline manner. The proposed framework is built on the Bayesian Theory of Mind perspective under which expert demonstrations implicitly convey their beliefs about the environment dynamics. Using this idea, the authors propose a class of priors that encode how accurate the learning agent believes the expert model of the environment is and later show how variations in this prior affect the learner agent's performance. Two solutions (naive and robot) to this new formulation are also proposed and evaluated in simulation.

review: The paper addresses an important challenge in IRL: how to mitigate compounding errors in the traditional two-step approaches where dynamic estimates are obtained prior to reward and policy learning. The paper is well-written and the idea of considering IRL under the Bayesian Theory of Mind framework is well-motivated and interesting.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I think the core idea is interesting and applicable to varied domains such as robotics and psychology. However, I would like to see a more thorough analysis and/or discussion of how BTOM and RTOM performance changes when there is a discrepancy between the learning agent's belief and the accuracy of the expert's internal dynamics model.
review_title: null

Meta-review: Reviewers appreciated the novelty of the proposed approach to IRL, the inclusion of theoretical results, and the thorough response in the rebuttal. The additional results with more baselines were very much appreciated, and were felt to make the paper stronger, although some doubts with regard to a clear performance win over ML-IRL remained (and should be addressed).
Decision: Accept (Poster)