Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new reward pseudometric called Dynamics Aware Reward Distance (DARD) which uses approximate transition model of the environment to compare reward functions while being indifferent to reward shaping. Previous work Equivalent Policy Invariant Comparison (EPIC) addresses the same problem, but it uses all possible state transitions feasible or not. The authors show that this can cause problems for reward functions that are technically equivalent in the feasible state transition state, but unequal on the infeasible transitions (which should not matter). The authors evaluate the method in "bouncing balls" (a discrete navigation task) and "reacher"  (robotic manipulation) environments. The results show that the proposed distance is more accurate compared to EPIC (distance is zero for "FEASIBILITY", 2 reward functions that are technically different but practically the same) and the distances are more aligned with the learned policies scores.
main_review: The idea of comparing reward function without training policies is very appealing for Reinforcement Learning. The authors take a previous method (EPIC) and improve it by making it consider only feasible state transitions. As a results the distance between reward function become more aligned with the trained policies scores.

The proposed idea is techincally sound and well presented. It is an incremental change based on a previous method, but the authors explain in which possible scenarios it would provide benefits. The evaluation is scritly oriented to validate this claim. The authors compare few reward functions crafted for these scenarios. In my opinion, it is hard to draw (or support) the conclusion from 5 reward functions. It would be very beneficial if the authors could provide a scenario where the DARD compares hundreds (or thousands) of arbitrarily chosen reward functions, and based on its results the authors could train a policy that outperforms the other. This is actually related to a high level comment that I have. Reading the paper, it was not easy for me to understand how the proposed method can be used in a large scale reward designing process. I think that the theme of "reward designing based on DARD for a very complex (or unknown) problem" could make the paper more impactful. 

The method assumes access to the dynamics model, and the authors state that this can be approximated (or learned) as well. I believe that an example based on learned dynamics model would make the paper much better.

summary_of_the_review: The paper proposes an incremental method that improves over past work for to ignore the infeasible state transitions. The idea is incremental, but has a good potential. The evaluation could be improved to show the impact of the given method. I think the paper can also use a higher level use-case of reward comparison where the proposed method is used to design a reward function to learn a problem that is otherwise unsuccessful.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper considers a reward comparison method in RL, where the goal is to evaluate how close an alternative reward function is to the ground truth reward function in terms of how well the set of optimal policies for each reward function is aligned. The proposed method (DARD) builds upon EPIC [Gleave et al.] and addresses an issue of EPIC by taking into account the actual dynamics of the environment when canonicalizing/transforming reward function as opposed to considering all possible (s, a, r') triples in EPIC. This allows DARD to be more robust to out-of-distribution transitions compared to EPIC while still being invariant to reward shaping like EPIC. The empirical results on Bouncing Balls and Reacher environments show that DARD can represent the reward distance more reliably EPIC does.
main_review: * Pros
  * The paper clearly identifies an issue with the prior work (EPIC) and proposes an appropriate solution.
  * The paper is very well-written. 
  * The experiments are well-designed. 
* Cons
  * The empirical results could be more comprehensive.
  * The proposed method is a bit incremental to EPIC. 

---

* The paper motivates the problem well by clearly identifying the issue with the prior work (EPIC) and proposes a reasonable solution that takes into account the actual dynamics of the environment. 
* The experiments (various manually designed / learned reward functions) are carefully designed to verify the hypotheses of the paper. I also appreciate the authors for acknowledging the limitation of the proposed method with smooth reward functions.
* The presentation of the paper is excellent. The problem is introduced nicely with appropriate preliminaries, and the main idea is illustrated with a good motivating example (Figure 1).
* Although the empirical results are good, it could be more comprehensive if the paper showed how the amount of the data affects the results, because the proposed method seems a bit more sensitive to the the amount of transitions in the dataset, because this is how the proposed method taking into account the dynamics. For example, it could be interesting to see how the results change as the amount of data available is varied. 
* Although the idea of considering dynamics for reward comparison in EPIC is novel, the overall method seems a little incremental because the method heavily builds upon EPIC in that it uses a similar canonicalization step followed by the same Pearson distance measure.
summary_of_the_review: Although the overall method is still quite similar to EPIC, this paper proposes a reasonable solution to overcome a clear limitation of EPIC. In addition, the experiments are quite well-designed, and the paper is very well-written. 


---
Update after the rebuttal: Thank you for adding experiments with learned transition models. The results look good. Since my major concern is addressed, I increased my score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new reward pseudometric (DARD), which is invariant to reward shaping and computationally efficient. Assuming access to the transition model of the environment, DARD could avoid the error due to samples out of the training distribution. The numerical results in simulated physical domains show that DARD outperforms previous methods. 
main_review: Strengths:
1. DARD does not relies on the input datasets, and therefore avoids unrelied reward values due to inconsistent samples.
2. The numerical results show that DARD is more powerful compared to EPIC. In particular, the experiments demonstrate that samples "out of distribution" would harm the performance of EPIC significantly.


Weaknesses:
1.  Compared to EPIC, DARD needs access to the transition model $T$ when computing the reward function, which may fail in many practical cases. Even assuming the transition model $T$ is accessable, DARD requires additional samples, which may worse the sample efficiency. 
2.  The theorectical intuition of DARD is not clear. In intuition, the samples "out of distribution" leads to inaccuracy. Then the questions are that: how to define a sample "out of distribution" and how it harms the performance of EPIC? There are many measurements to describe the inconsistency between the offline distribution and online distribution and I think it is possible to derive some theorectical guarantee using these arguments.

summary_of_the_review: Overall, the paper is well written. My current score is 5 and I look forward to further discussion with the authors.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a novel method for comparing reward functions without policy optimization, the Dynamics-Aware Reward Distance. DARD improves on the state-of-the-art EPIC distance by using an approximate transition model to evaluate reward functions on transitions close to the training distribution, while EPIC evaluates on arbitrary transitions (which can be infeasible). The authors prove that DARD is also invariant to reward shaping, and experimentally demonstrate on the Bouncing Balls and Reacher environments that DARD is a better predictor of policy return than EPIC and works for coverage distributions induced by either random or expert actions. 
main_review: This paper makes progress on the important problem of comparing reward functions by addressing a significant shortcoming of the EPIC distance. The paper is clearly written and well-motivated. The theoretical results seem sound based on a quick skim of the proofs. The method is evaluated empirically on a wide range of reward models, and claims H1-H3 seem well-supported by the results. 

The advantage of DARD over EPIC comes at the cost of assuming knowledge of the transition dynamics, which is a limitation for scaling the method to more complex settings where an environment simulator may not be available. It would be great to see how well DARD works for learned transition models as suggested in the conclusion, though I understand this is out of scope for this paper.

Here are some suggestions for improving the paper:
* The fact that DARD does not perform well on environments like PointMaze where nearby transitions always have similar reward values is an important limitation, and I would suggest adding the PointMaze experiment to the main text of the paper. 

* Define a feasibility reward model for the Reacher environment as well, to demonstrate that this shortcoming of EPIC is not specific to one environment.

* To further support hypothesis H3 that DARD is effective across diverse coverage distributions, it may be useful to also test it on a coverage distribution induced by a non-random suboptimal policy. 

* It would be good to see the two methods compared on more than two environments, e.g. other MuJoCo environments. 

* Clarify the difference between $D_\rho$ and $D_p$, which seem to both be defined as the Pearson distance


summary_of_the_review: This paper makes progress on the important problem of comparing reward functions by introducing a new reward distance that addresses a significant shortcoming of the EPIC distance. The results are significant and novel, and I would recommend accepting the paper.
review_title: null

Meta-review: The paper proposes a new pseudometric, DARD, for comparing reward functions that avoid policy optimization. DARD builds on a recent work by Gleave et al. 2020 where the pseudometric EPIC was proposed. In contrast to EPIC, DARD operates on an approximate transition model and evaluates reward functions only on transitions close to their training distribution. Empirical experiments in different domains demonstrate the effectiveness of the proposed pseudometric. The reviewers acknowledged the importance of the studied problem setting and generally appreciated the results. I want to thank the authors for their detailed responses that helped in answering some of the reviewers' questions and increased their overall assessment of the paper. At the end of the discussion phase, there was a clear consensus that the paper should be accepted. The reviewers have provided detailed feedback in their reviews, and we strongly encourage the authors to incorporate this feedback when preparing a revised version of the paper.
Decision: Accept (Spotlight)