Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper analyzes and evaluates several different measures for evaluating reward learning. The authors provide a theoretical worst-case analysis showing limitations of common measures and provide empirical tests across several domains. 

---
I thank the reviewers for their responses. I still struggle with the novelty of this paper as it relates to well known theoretical and empirical work. However, I can see that there may be benefit to explicitly discussing these issues explicitly so I am raising my score to a weak reject.
strengths_and_weaknesses: Strengths
- interesting problem area
- good empirical study of several reward learning measures
- includes both theoretical and empirical analysis

Weaknesses
- not focused on robotics
- theory seems overly simple (focused on simple 2 feature examples) and it is unclear how useful the worst-cast analysis is
- missing related work
- not broad enough to qualify as a survey
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations section not present
summary_of_recommendation: The authors study an interesting machine learning problem. However, I do not think there is enough depth or breadth for CoRL and while there is a strong connection to robot learning, the paper seems better suited for a machine learning conference. The analysis seems overly simple and does not seem to contribute to the field---many of the claims made are commonly understood in the field. While I appreciate the effort to survey measures for reward learning, the paper provides insufficient breadth for a survey and does not have as much novelty as claimed. There are several important missing works and the ideas of reward measures being uninformative and the importance of testing in transfer domains do not appear to be significant contributions given prior work.
issues: It is unusual to have a reward function that maps directly from trajectories to rewards. Most work considers R(s) or R(s,a).

It is well known that just comparing reward parameters is not a good way to evaluate a learned reward. Indeed, much work has studied and analyzed the problem of which rewards lead to the same optimal policy [1] and which rewards make a given policy optimal [2]. 

[1] Brown et al. "Value alignment verification." International Conference on Machine Learning. 2021.

[2] Ng and Russell. "Algorithms for inverse reinforcement learning." ICML. 2000.

Also, no mention is made of the common practice to test learned rewards in transfer domains:
- Levine et al. "Nonlinear inverse reinforcement learning with gaussian processes." Advances in neural information processing systems 24 (2011).
- Fu, Justin, Katie Luo, and Sergey Levine. "Learning Robust Rewards with Adverserial Inverse Reinforcement Learning." International Conference on Learning Representations. 2018.

Recent work has looked at characterizing learned reward functions:
- Michaud, Eric J., Adam Gleave, and Stuart Russell. "Understanding learned reward functions." arXiv preprint arXiv:2012.05862 (2020).
and also, importantly, at evaluating learned rewards using metrics other than alignment or mean squared error:
- Gleave, Adam, et al. "Quantifying Differences in Reward Functions." International Conference on Learning Representations. 2020.

The paper mentions other measures in 2.3, but does not evaluate them. If this is designed to be an analysis and survey paper, then I think these should be included for study.

In table 1, R^rel is defined but I do not find any definition of R^abs

Line 117: [1] and [2] above provide ways of knowing whether the robot will solve the task, even if w' != w^user

Line 160: Shouldn't this be -N + 1? [looking at the values in line 155]

Presentation recommendations: It would be beneficial to show the different domains being studied. Based on the descriptions in the paper it is hard to tell how complex or interesting these tasks are.

No limitations section! This was explicitly asked for in the author instructions. The authors need to address limitations of their paper and analysis.

End of section 6 has some interesting ideas, but they seem very difficult and it would be nice to provide some ideas for how they could be accomplished. How would the robot modify its reward estimate. Also, future work direction (1) seems like it is already done in the literature. Many works on active learning select training instances.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper brings up some of limitations underlying existing methodologies for evaluating reward learning algorithms in HRI. After classifying existing evaluation methodologies into two different classes, the authors demonstrate practical issues through empirical and theoretical analysis over a set of planning benchmarks.
strengths_and_weaknesses: *Strengths*

-I enjoyed reading this paper. It is very well written and easy to follow. I liked the balance between exemplary discussions, theoretical work, and empirical work. Well done!

-The paper brings up some of the limitations of existing reward learning approaches. The point is interesting and timely. While there is a very rich literature in this space with major claims and cool demonstrations, it looks like the evaluation methodologies followed are often fundamentally flawed. Specifically, it looks like the measures that many researchers tend to use to evaluate how well a robot has adapted to a user's preferences are not reliable indicators.

Weaknesses
-The paper does not discuss related work but rather lists a large set of references. Most of the references are coming from 1-2 groups. If this aspires to serve as a survey as the authors state, I would recommend a richer and deeper discussion of the literature within the narrative of the introduction.

-This seems to be in part a "statement" paper with relatively limited novelty. The main point is definitely intriguing and interesting but the key insight is relatively straightforward and the final discussion is quite limited. It would have been more interesting/impactful (to me) to expand upon the closing discussion on what comes next and what paths might be viable. I understand that the space limitations might not allow that but in its current form, I feel like the paper is missing an insightful discussion.

-The authors discuss this briefly but it seems to me that one fundamental limitation of this whole space of approaches is the assumption that w^{user} is somehow available when in the whole field of HRI this is most likely not the case. The included analysis (as well as the reviewed approaches) makes use of that assumption. Perhaps it would be valuable to expand a bit on that.

-The limitations statement, a required component for this year's conference is missing.

*Minor*

Line 32: “These observation…" --> These observations 

Explicitly define R_abs

Make the scale of axes in Fig. 1 equal 

Line 207: “five categories” --> four categories

Clearly name the planning problems, perhaps use bold to define Dubins, Driver, Server in text so that the correspondence from text to table 3 is clear.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations section not present
summary_of_recommendation: Overall, this paper makes an interesting and intriguing point about the rigor of evaluation methodologies in the field of reward learning for HRI. It is well written and makes nice use of examples, theoretical and empirical work to support the main observation. While interesting, the main point is still relatively straightforward and I would have appreciated a more elaborate discussion on what comes next. I am generally favorable if such a discussion is included.
issues: -How were the five problem instances in Sec. 5.1 decided? How much they vary and how representative are they?

-Is there a reason why the server experiment is not featured in Fig. 2? Is it not interesting?

-Include a deeper discussion of the literature and a more insightful closure at the end of the paper. See my notes above.

-Add a limitations statement.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work serves to point out that the current methods of evaluating reward function algorithms do not appear to be sound. In particular, the paper shows both parameter and reward based comparisons to ground truth reward functions can fail to give a reasonable sense of how good a given solution is, and by proxy, how effective the associated algorithm is. Both theoretical and empirical arguments are provided to support their case. 

strengths_and_weaknesses: Strengths:
- The paper demonstrates a fault with the evaluation methodology employed by the community. In my opinion, it does a good job in showing the existence of the issue, which is perhaps tacitly known but, to my knowledge, had yet to be formally investigated. 
- Both theoretical and empirical evidence / arguments are presented, which makes it clear the problem exists across multiple settings. 
- A well-written paper that is easy-to-follow (up to the points below)

Weaknesses:
- Although the paper clearly shows a problem exists, it is not clear how severe or widespread the issue is, especially with more complex robot tasks. The theoretical results are based on very simple constructed examples, and the numerical results are for relatively simple domains. The paper would be more impactful if the authors can discuss the implications of their work, e.g., if previous comparisons in the cited works are invalidated? 
- The multi-scenario setting in Sec 3.2 points to mis-modeling of the problem; the path length and jerk are not invariant to the goal location? Is it that the rewards are transferable from train to test, but the relative difference between accumulated rewards is not preserved? 
- On a related note, the paper falls short of analyzing why and when the reward/parameter measures fail to be meaningful. I believe the paper will be more impactful if the authors can provide a discussion around the factors affecting the veracity of these measures. Alternatively, this limitation should be discussed in the paper. 
- The paper lacks a limitations section.
- The paper has several typos, e.g., 
   - line 115: “they do consider” -> “they do *not* consider”
   - line 162: “now upper bound” -> “no upper bound”
   - line 180: “now” -> “no”
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations section not present
summary_of_recommendation: Overall, I believe the paper provides interesting results that will make the community re-evaluate our we should compare algorithms for reward learning. There are some minor issues with the work, but I think the results are interesting enough to warrant publication. A major weakness of the paper is that it does not clearly identify the limitations of the provided theoretical and empirical analysis. 

issues: Please see the weaknesses listed above and I hope the authors can modify the paper to provide a more clear statement of the limitations surrounding their analysis.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper is an unusual contribution for a robotics conference. It does not directly contribute to advancing understanding of robot control or learning. Rather it presents a critical analysis of the measures used to assess the quality of policies learned via HRI driven reward learning.  The key disadvantages of the two main classes of performance metrics (Parametric similarity and Relative Reward) are demonstrated with a simple example, theoretical mathematical abstraction, and numerical simulation on several examples. 
Central to the paper are two proofs: (1) that even when the alignment of the learned weight vector is arbitrarily close to 1, there is not upper bound on the difference between the learned reward and the optimal reward(according to user preferences). (2)no lower bound exists on the relative reward of the test data. These two theorems are further demonstrated with evaluation of randomly sampled weight vectors in 2 examples.

strengths_and_weaknesses: #### Strengths
- The motivating example in section 3 provides a clear explanation of the way in which the two classes of measures might fail.
- The proofs and numerical results support the central contribution that typical measures do not provide guaruntees of the actual performance of the learner.
- Overall ideas are well-communicated.

#### Weaknesses
- There’s no robots? This seems like it may not be the best fit for CoRL. The criticisms of reward learning could be applied to many different learning problems. The paper does not make a convincing case as to why it is critical to robot learning in particular.
- The paper lacks a discussion of what aspects of the measures make them fail. It just states that they do. The discussion poses very generic questions.
- Given that there are these possible worst-case scenarios, the paper could talk about how existing works (the first 15 references) avoid these worst-case scenarios or if there are published instances in the sim2real literature where this might be part of the issues.


robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations section not present
summary_of_recommendation: Overall the paper is well-written and presents an interesting perspective. This perspective could be valuable to the greater community in analyzing why learners fail despite high performance expectations based the chose performance measure. The paper does lack some discussion of either why these performance measures fail, more specific requirements for 'good' measures, or how existing measures might be modified or combined to avoid some of these pitfalls. 
The main issue is that the work includes neither a robot nor any learning, and the manuscript does not make a strong case for why this is critical to the robot learning community specifically as opposed to machine learning community in general. 
issues: #### Major Issues
-  The paper needs some explanation of how prior works have avoided the pitfalls presented by the paper.
-  More discussion of either the specific attributes of the measures that lead to failure in the given cases or how one might approach a reward learning problem to avoid the 'worst-case' scenarios is needed.

#### Minor Issues
- Line 33: the wording says high(but not optimal) but before explaining the measure I think it makes more sense to say that the parameters values are very similar (but not identical)
- Line 162: now -> no
- Line 180: now -> no
review_title: null

Meta-review: metareview: Post discussion update
------------------------

The authors completed a rebuttal that addressed all of the reviewers' critiques, and also uploaded a revised and improved manuscript. The most significant improvement seems to be the addition of a small related work section in the introduction and a limitation plus future work section at the end of the paper. However, since the final paper has an 8 page limit, the current 9-page manuscript will need to be shortened. 

The clarifications seemed to sway two of the reviewers toward increasing their ratings. Generally, it's clear that although this is a non-standard paper, it raises interesting and potentially provocative points that would be great for the CORL community to discuss. Therefore, I am recommending acceptance.



Original review
----------------


This paper addresses the HRI challenge of measuring how well a robot has learned a user's reward function. It describes two main types of measures---parameter-based and reward-based---and provides both theoretical and empirical examples of the shortcomings of each. 


Strengths
* This seems like the type of paper that would be useful to assign in an HRI course or give to a new grad student, because it clearly explains the different measures and provides a comparison of them.

* Reviewers noted that the issues brought up by this paper are important and very relevant to current approaches. The paper is timely.

* Reviewers agreed that paper balances well between theory and empirical examples. 

* The paper was well written and easy to understand.


Weaknesses

* The main critique is that reviewers feel that the paper doesn't go deeply enough into the discussion of what to do with these findings. How should robotics researchers use this information to design better measures? In what cases are these measures likely to fail, or to be accurate? Section 6 poses such questions, but it would add substantially to the novelty of the paper if there was some suggestion of where to go from here.  

* Although the paper bills itself as a survey, reviewers noted that it doesn't provide much of a discussion of related work. Reviewers also provided additional citations that address some of the issues brought up regarding current measures of performance. 

* A couple of reviewers questioned the connection to robot learning in particular.

* There is no limitations statement, which is a required component.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)