Do we use the Right Measure? Challenges in Evaluating Reward Learning Algorithms
Authors: Nils Wilde
Javier Alonso-Mora
Venue: CoRL 2022 Poster (2022)
Keywords: Human Robot Interaction
Reward Learning

Abstract: Reward learning is a highly active area of research in human-robot interaction (HRI), allowing a broad range of users to specify complex robot behaviour. Experiments with simulated user input play a major role in the development and evaluation of reward learning algorithms due to the availability of a ground truth. In this paper, we review measures for evaluating reward learning algorithms used in HRI, most of which fall into two classes. In a theoretical worst case analysis and several examples, we show that both classes of measures can fail to effectively indicate how good the learned robot behaviour is. Thus, our work contributes to the characterization of sim-to-real gaps of reward learning in HRI.

Do we use the Right Measure?
Challenges in Evaluating Reward Learning
Algorithms
Nils Wilde
Javier Alonso-Mora
Department of Cognitive Robotics Delft University of Technology Netherlands n. w.wile@tuuelff.nl
Department of Cognitive Robotics
Delft University of Technology Netherlands w.wide@tudelft.nn
Abstract: Reward learning is a highly active area of research in human-robot interaction (HRI), allowing broad range of users to specify complex robot be- haviour. Experiments with simulated user input play a major role the devel- opment and evaluation of reward learning algorithms due to the availability of ground truth. In this paper, we review measures for evaluating reward learning algorithms used in HRI, most of which fall into two classes. In a theoretical worst case analysis and several examples, we show that both classes of measures can fail effectively indicate how good the learned robot behaviour is. Thus, our work contributes to the characterization of sim-to-reat gaps of reward learning in HRI.
Keywords: Human Robot Interaction, Reward Learning
Introduction
Researchers in human-robot interaction (HRI) study how robot behaviour can be adapted to the end- users' preferences. To achieve this, intelligent autonomous robotic systems often do not receive explicit instructions, but instead optimize reward function that encapsulates how the robot should accomplish its tasks. Defining such reward functions challenging. Therefore, interactive learning frameworks have been designed that allow broader range of users to transfer their preferences into a reward function [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]. In those frameworks, different modes of interaction have been studied, including pair-wise comparisons of trajectories [2, 3, 4, 5, 6, 7, 8, 15], rankings of trajectories [16, 17], and feedback [18, 11, 13, 19], among others.
The development and validation of efficient algorithms for reward learning requires numerical ex- periments where the ground truth reward function is known. Since the reward function of real users is in general unknown, this usually involves simulated users, i.e., sampling potential user reward functions and then generating user input following a behaviour model. To evaluate algorithm per- formance various measures are used, most of which fall into two classes: parameter-bass reward-based measures. The former capture how close the learned parameters of the user's reward function are to the ground truth parameters, while the second class considers the reward that the user assigns to the learned solution.
and
We show that both classes have significant shortcomings: parameter-based measures do not give direct indication about the reward collected, i.e., how good the learned behaviour is in specific instances, while reward based measures do not necessarily translate well between related problem instances, e.g., between training and test environments. These observations shed light on limitations of evaluation methods for reward learning algorithms in HRI: i) Even similar (but not identical) parameters do not guarantee that the learned robot behaviour close to optimal. ii) Reward-based measures might indicate strong performance in training environments, yet the performance in test environments can be much poorer, even when the environments are relatively similar. Thus, both classes of measures do not effectively indicate how good the learned robot behaviour is in general.
Related Work: Parameter- and reward-based measures are used extensively in reward learning in HRI [2, 4, 5, 20, 21, 22, 23, 24, 25, 26]. Yet, their respective shortcomings are not widely discussed.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

To address the transfer problem between training and testing, it is common in machine learning to use several testing instances [27]. This practise is also applied in reinforcement learning (RL) and inverse reinforcement leraning (IRI.) [28, 29]. The authors of Fu et al. [29] use an adversarial RI. framework to obtain a reward function that yields robust behaviour test instances. The relation of inaccurate parameters and optimality of the corresponding solution leads to the observation that even inaccurate reward functions can yield an optimal RL-policy [29, 30, 31]. Similarly, Wilde et al. [9] discretize the parameter space of graph-based planner into subsets called equivalence regions such that all parameters from the same subset yield the same path. Brown et al. [32] recently proposed value alignment for Markov Decision Processes (MDP) to capture if the robot behaviour corresponds to a user's preference, avoiding the pitfalls of parameter- based measures. Gleave et al. [31] propose a measure for evaluating reward functions in RL without requiring the costly computation of policy. In contrast to these works, our paper does not focus on a specific problem domain such as RI. or graph search. Instead, study shortcomings of learning measures used in HRI, describe their theoretical worst-case and show different characteristics in numerical experiments.
Contributions: In this paper, we provide a review of measures used in HRI for how well a robot has learned user's reward function, followed by an extensive analysis of their limitations. In particular, we study exemplary cases highlighting the shortcomings of the two main reward learning measures. We then provide theoretical analysis, showing that both measures can be arbitrarily bad indicators for the actual performance. In numerical experiments for three different robot planning problems, we highlight that the issue is prevalent various domains.
1.1 Reward Learning Formalism
Let I be the instance of robot planning problem, defined by robot's state and action space, an initial state, a set of goal states, and a collection of constraints. In HRI, reward functions are often used describe how well robot trajectory fits user's preferences. Thus, we consider reward function R taking a robot trajectory T as an input and assigning a real value. Usually, R assumed to be a weighted sum of features [33, 23, 2]. Let be robot trajectory. and (T()) [oi (T) ... Φn (T) be some predefined feature functions. Then the reward of T is
R(T, w uaer)
w uaer (T)),
where waeer is weight vector, expressing how a particular values these features. Without loss of generality, we assume the weights to be positive. The objective is that a robot executes trajectories of maximum reward, i.e., computes
T arg max R(T, w user)
T
for any instance 1 of the planning problem. However, in many applications, it is difficult to obtain the weights qeer from a particular user. Reward learning is the problem of interactively learning these weights in order to solve (2). This can be informally stated as follows:
Problem 1 (Reward Learning). Given is user with hidden preference wheer some mode of user interaction M, and K iterations for querying the user. Using the interaction, the robot needs to learn the best possible estimate w' of the user's weight vector u
Review of Reward Learning Measures
Problem 1 is ambiguous about what defines good estimate w'. In simulation experiments ground truth user weights uaer are available, allowing for direct comparison to optimal. The HRI com- munity uses different measures, which can divided into two primary categories: reward-based and parameter-based. We will briefly review the most common measures from the literature, followed by a discussion about the fundamental difference of reward-basse and parameter-based measures.
2.1 Reward-Based Measures:
Reward- measures directly evaluate estimated weights un' based on how well the corresponding optimal trajectory 'I' solves the principal problem in (2).

---

The most direct reward-based measure is to consider the reward R(T', 200000 _ some trajectory T' collects for user weights user This measure has been used in [20, 21, 22], among many others.
Returned Reward:
Relative Reward: To measure closeness to optimal. the Relative Reward [15, 26] takes the ratio of the reward wusse assigns to the learned solution T' over the reward assigned to T i.e.
R(T', user)
R (w', 2 (Ber)
33)
R((Tuee wusez)
In case the reward function takes negative values, the ratio is inverted. When rewards can be positive and negative, a normalization is needed.
Regret: In some cases, it might be favourable to use a negative measure, i.e. where O corresponds to the best achievable value. This is capture by the Relative Regret [5], which is Reg*s1 R (w', w wusee). Alternative to the ratio, the Absolute Regret is formulated as a difference [26, 23, 25], ie.. Regabs R(Tuser, wubeer)= R(T', wuser) This is convex function of waer [26].
2.2 Parameter-Basee Measures:
This group measures also requires ground truth weights wueer, and measures the similarity of the estimated weights w' and the optimal weights wueer
The alignment measure [2, 4, 34] captures how similar the learned weights are to the user weights by measuring the angle between the vectors:
Alignment:
w
w
user
alw w
(4)
wuser
Mean squared weight error: Similar to alignment, other works use the mean squared error (MSE) of estimated weights and user weights [23, 24).
2.3 Other Measures
For completeness we also briefly review a third category used learning from choice frameworks, which we label as predictive measures. There are two measures in this category, log-likelihood [16, 3, 15], and prediction accuracy [3]. Both measures describe how well a learned probability distribution over weights allows for predicting how user would choose between two trajectories. These measures not require the ground truth weights, but only user's response to a validation set consisting of choice queries, which makes them suitable for evaluating user studies. However, prediction measures are indirect and depend on the validation set.
2.4 Comparison of Mcasures
Parameter-be based and reward- based measures take different perspectives on describing how well some estimate w' describes user reward function with weights uaer Reward- basedd measures consider the robot trajectory that a planner policy returns when optimizing for weights w', and compute the reward assigned it by the user weer This directly captures how well the robot able to solve problem (2). However, this statement only relates to one specific problem instance. For instance, a robot might learn a reward function for how to carry dishes around the kitchen, where the features trajectory length and risk of collision. When trained for specific planning problem in a specific kitchen, the reward-based measures then describe how well the robot able to solve that task, but give no direct information about how well the robot would operate in different kitchen.
In contrast, parameter-basel measures describe how well the reward function is estimated. These measures are universal, i.e., independent of the instance. When capturing uaer accurately, a robot would be able to compute Tueer in any problem instance with the same features. Yet, parameter- based measures also have major drawback: they do not consider the robot's planner, i.e., the mapping from weights to optimal trajectory. Thus, there might be no relation to the collected reward and thus no information about how well the robot will solve (2), unless w' user.

---

0.0
0.5
-0.5
y,005
-025
11.00
-22
30
0.5
2.3
2.2 23
10
1.8
3.5
23
DOG
8.00s
(a) Dubins instance 1.
(b) Dubins instance
Figure 1: Different Dubins paths with three features. Green shows user's optimal trajectory Tueer blue and orange two different estimates TA and T Grey shows other Dubins paths for different radii.
(a) Instance 1, Fig. la.
(b) Instance 2, Fig. 1b.
R
R
Reghh
MSE
R
Re
Regth
MSE
uyter
-5.84
1
.01
upte
-3.39
1
.01
w
55.9
.98
.08
.98
.01
w
-3.56
.95
.17
.8
.01
6.36
w
-3.39
Table 1: Estimation measures for the Dubins example.
3 Exemplary Fail-Cases
Using a simple motion planning task we now illustrate the weaknesses of parameter and reward based measures. A robot is navigating in 2D space with constant velocity following Dubins paths, where the radius can chosen within some bounds. There are two features: path length, and integral square jerk (IS jerk) the first captures the time-efficiency while the second measures discomfort or risk (Note: to obtain maximization problem, we take the negative of these values).
Single-Scenario Example: We arbitrarily choose user with preference user = [.5, 5). and two estimates w = [.4, .6). and " [.6, .4.. We illustrate the corresponding optimal trajectories in Figure la and summarize the resulting measures in Table (aa). The parameter-based measures evaluate 4 and w as equally good estimates w uaer However, w collects more reward than B making it the better solution. This observation demonstrates that the alignment metric is unaware of the sensitivity of the optimization to find the optimal trajectory given some w. Thus, alignment can fail capture how well estimated weights can allow the robot compute trajectories that fit a user's preference.
Multi-Scenario Example: We now study the where robot has perform multiple similar tasks. Hence, we consider a second planning instance Figure 1b with a different goal location. This serves test case, while practise, a robot might have obtained these estimates from reward learning in the first instance. A good estimate of user should then also achieve high reward in the case. The measure values are shown in Table 1(b). Naturally, the alignment and MSE yield the same value as earlier since they only depend on the weights, not on the plamning instance or trajectory. However, the reward measures show that, while w" was poorer estimate than w in the first example, the places are now reversed. Hence, good values of reward-based based metric do not necessarily translate between instances, egg., from training to test cases.
the same user and estimates A and w before. In
4 Theoretical Analysis
In addition to the previous motion planning tasks with a Dubins model, we present more rigorous theoretical results. For brevity, the proofs can be found in the supplementary material.

---

Reward
Reward
-8-
0.50 Fstimate #/
0.75
0.00
0.25
0.50
0.75
1.00
Estimate
(a) Rewards for Dubins instance 1.
(b) Rewards Dubins instance 2.
Figure 2: Example reward functions Tapeser (w') for the Dubins problem from Figure 1. In each plot, we pick 5 different uherr uniformly spaced, and plot the respective rewards Yygus (w)". The markers on each line show the location of the optimal estimate, i.e., where w w ueee
4.1 Worst-Case Study
First, we show that for any non-optimal alignment MSE error the collected reward under the true weights wser can be arbitrarily far from optimum.
Theorem 1 (Unbounded Reward Difference). Let w be user weight, and w' be an esti- mate, where the alignment R(Tuser w R(T', w uBer)r is unbounded.
S a(w', wueer) < 1 for some < 1. The difference in reward
Theorem shows that even when the alignment is arbitrarily close to 1, it does not allow for claims on how much reward is collected, compared to optimal. Following the same proof the result extends to the second parameter-based measure, the MSE. Next, study the multi-scenario case to show that the reward-based measures do not translate from test training scenarios. Let ]Train be training and /Tees test instance. For weights w', wueer we use Ret1 R (w', wu to denote the relative reward collected in /Train and similarly Reest (w', waeer) for the test instance /asst
Theorem (Unbounded Test Error). Let uneer be a user weight, and w' be an estimate. Further, let [Taan be a training instance, where the relative reward Arrein (w', wxeer) taking values in [ô, 1] for some < 1. There exist test instances / Test where the relative reward frrel (w', weeer has no tighter lower bound than 0.
The two theorems show that in a worst case the observed shortcomings can cause both classes of measures to be arbitrarily poor indicators of the actual performance.
4.2 Non-linear Characteristics Reward Functions
Given the exemplary cases and the worst-case analysis, we briefly characterize the underlying relationship of weights and rewards. Using the convention that T' is the optimal trajectory for weights w' we can write the collected reward when optimizing for estimate w' rew w')
wurer_ ((T"). This is linear function of w for any fixed w'. Yet, maybe surprisingly. for any fixed wser is in general non-linear function of the estimate w'. That is, the features ((T)) are often not linearly dependent on the weights w'. This is related to the sensitivity of optimization problems small changes in w' can lead the same solution. Indeed, if trajectories are computed with a discrete planner, then almost all trajectories outputted by the planner are optimal solu- tion to not just a single weight, but set of weights. Thus, the function rewerer w') ) piece-wise constant over w' [9]. Yet, this phenomenon also applies in continuous space. For instance, when planning around an obstacle, multiple weights can lead to the tightest feasible trajectory that takes the detour around the obstacle; only for high enough weight minimizing trajectory length, the planner will switch to the other side of the obstacle. Different weights leading to the same optimal solution is also related to reward-ambignin in RL [30, 29].
We illustrate the non-ling con-linearity of uaw w') in Figure using the planning examples from Sec- tion 3, where we fix several values for user Since the problem is only two-dimensional, we can set We = 1 - wi, allowing us to plot uee. w') over the scalar values of w/. We make two key observations: i) r(w user, w') is non-linear in un and and can exhibit jumps, which causes the shortcomings of parameter-based measures. ii) reur w') shares very little similarities between the two instances, resulting in the transfer problem between training and testing.
5

---

Instance
Instance
Instance
Inxtance
Instance
Inctance
Instance
velaatveRewar
Aliggrent
Aligenntt
(a) Dubins experiment.
(b) Driver experiment.
Figure 3: Examples for the relationship between alignment and relative reward. In each plot, the relative reward values have been normalized.
Dubins
Driver
Server
Min
Mm
Md
SD
Min
Min/MMx
Pears.
.41
.43
.28
-33/ .96
.45
.50
.32
-.39/
.96
.45
.46
.18
-.06/ .87
Spear.
.5
.56
.29
-36/
.97
.56
.59
.23
-.17/
.97
.42
.42
.18
-006/ .85
Table 3: Correlations between alignment and relative reward. Correlation values: Pears. Pearson correlation coefficient, Spear.- Spearman Rank coefficient. Statistics: Mn-mean, Md median, SD standard deviation.
Numerical Results
To validate that the described fallacies of the measures do only arise in constructed examples, we conduct extensive numerical experiments with randomly generated instances and user weights.
Robot planning problems: We consider three different robot experiments: i) mobile navigation problem using Dubin's car similar to Figure 1 but with three features: trajectory length, jerk and the closeness to humans present in the environment. ii) The driver simulation previously used in [2, 4, 5, 15, 32], among others. This problem has four features: keeping speed, closeness another vehicle, staying in lane, and orientation on the road. iii) A manipulator robot serving drinks. The experiment has eight features, describing the choice of drink well how the robot moves plate and stove. Trajectories were generated on real-world system by the authors [15].
Generation of user weights and estimates For each experiment, generate set Ω of 100 uniformly random weights vectors in [0, 1]" (where is the number of features). User weights user are picked uniformly random from Q. We do conduct any reward learning. Instead, all weights w' e Ω are considered as estimates of user,, which could have been the result of learning.
For each pair (weeer w²) we compute different learning measures. We characterize the relationship between measures using the Pearson correlation and Spearman rank coefficients. The Pearson coefficient describes linear correlation strong linear relationship would be the best possible result. The Spearman coefficient measures monotonicity.
Dependent Measures:

---

Dubins
Driver
Server
Min, Max
Mn
Md
SD
Min, Max
Min, Max
Pears.
.44
.47
.23
-5/.97
.48
.47
.25
-,57/.90
.80
.91
.26
-.21/1.00
Spear.
.55
.43
.66
-55//97
.76
.83
.19
-007/95
.80
.93
.28
-.17/1.00
Table 4: Correlations between relative reward in training and testing.
5.1 Analysis of Alignment- Reward Relation
First, we investigate how alignment relates to solving the principal problem in equation (2). For any pair (unuee,, w') we compute alignment and relative reward'. Figure shows exemplary results for 5 users, across 5 different problem instances. Overall, the relationship between alignment and relative reward varies drastically between instances as well as between users. The results fall into five categories:
1. Approximately Linear: In some instances alignment is a good indication of the collected reward. Examples are Dubins: User 5, Instances 1 and 5, as well Driver: Users 2 and 4, Instances 2-5.
2. Underestimates / Simple Instances: In some cases, most samples with alignments >> .5 collect close to optimal reward. Thus, the different sampled solutions are similar to optimal solutions for these users, resulting in easy problem instances. Here, the alignment is drastic underestimate of the actually collected reward. Examples are Dubins: Users 1 and Inssttac 3, Driver: User 3 and 5, all instances expect 3.
3. Overestimates: Some plots show monotone trend that becomes steeper for higher align- ments. In most such cases the relative reward is small even for high alignment values. Thus, the collected reward is overestimated. In case of Dubins, User 2, Instance I or 2, the align- ment values of &J .95 correspond to less than .25 relative reward, i.e. highly suboptimal solutions. Other examples are Dubins: User 4 Instances 1, 2 and 5.
4. Unstructured: In various instances, alignment and reward have or at most a weak cor- relation, i.e., alignment gives almost no reliable information on the collected reward.
We repeated the experiments for 40 random seeds for ? yielding 1000 user-instance pairs, numerical results are shown in Table 3. For all three planning problems the Pearson correlations are weak and have high standard deviations. This strongly supports the observations from the plots: Across instances and users there is no consistent linear trend between alignment and reward. The Spearman coefficients slightly higher for Dubins and Driver, yet not high enough indicate strong monotonic relationship. In summary, the experiments show that alignment is not a reliable indicator on how much reward is collected, i.e., how well some estimate w' solves the principal problem (2).
5.2 Analysis of Reward between Training and Testing
Next, study the relation of relative reward between training and test instances. We fix one in- stance as a training instance [Taaia, and compute the relative reward with respect to weer for all w' Ω. For the same samples, we compute the relative reward in a set of test instances. Figure 4 shows exemplary results: Each row corresponds to fixed random user, and each column is different instance with the first column being the training, and the others the test instances.
We observe a large variability in the relationship between the relative rewards collected in training and testing, with similar archetypes of correlations in the first experiment. For Driver we notice that in Instances 3 and 5 almost all samples are close to optimal for user 1-4, showing that these in- stances trivial for some, but not all users. Again, we repeated the experiment for 40 random seeds and report the statistic in Table 4. For the Dubins and Driver experiment, we observe only weak cor- relations the reward collected in training is not reliable indicator for the reward collected in other instances. However, the Server task exhibits strong Pearson and Spearman correlations. Indeed, for
'The relative value is favourable over returned reward or absolute regret since it takes values in the unit in- terval. Further, we relative reward instead of relative regret such that the optimum is at 1, akin to alignment.
7

---

Instance
Instance
Imstance
Itatance
Itstance
Instance
00
60
61
Ret att Reward TTxtinng
melative Reward -Testing
so-
cm
on-
140
to
in
385
Make
03
Relative Reward - Training
Relative Reward Trainiin
(a) Dubins experiment.
(b) Driver experiment.
Figure 4: Examples for the relationship between relative reward in Training and Testing. In each plot, the relative reward values have been normalized.
many instances, the relationship follows strong linear trend allowing for better predictions (exam- ple plots can be found in the supplementary material). Nonetheless, the high standard deviations and very small min values reveal that this does not extend to all instances and users.
In summary, we observe that the relative reward does not necessarily translate from training to test instances. Thus, high rewards in training do not necessarily yield good performance in test instances.
6 Discussion
Summary: In this paper, we reviewed measures for evaluating reward learning algorithms when ground truth weights wuser are available, and characterized the two main classes of measures: parameter-based and reward-based Through i) illustrative examples, ii) theoretical worst-case anal- ysis, and iii) experiments with various robot planning problems, we have shown that both classes are not reliable indicators for how well the robot has adapted to user's preferences.
Limitations: While analysis describes theoretical worst-cases for both classes of measures, does not derive characteristics of feature functions under which the shortcomings are more severe, or where tighter worst-case bound could be derived. Further, our work is mostly concerned with identifying and describing these issues, but provide no detailed study of how prevalent the they are in published work. Moreover, the paper relies on relatively simple planning problems for theoretical studies, leaving it ambiguous how severe the issues are in more complex or real-world settings.
Our analysis raises the question about better way evaluate reward learning algorithms. The transfer problem can be addressed by using several test instances for reward-based measures, which is common in machine learning. This approach could be adapted actively or adversarially selecting test instances similar to [29]. Unlike random test instances this could yield a bound on the error any test instance, i.e., soften the negative result of Theorem 2. In Section 4.2 we showed that the relation from weights to features and thus rewards for a fixed user is non- linear. Yet, this depends on the choice of features. Future work could investigate if features can be modified to mitigate the two shortcomings described in this paper. However, it is unclear if there exists principal way, if this would require creating praxu-features for individual instance. Finally, post-procet post-processing estimates could make them more robust towards the observed fallacies. Even when collecting high reward in training, the estimated parameters are likely not accurate. This allows us to modify the estimate by solving bi-objecitve optimization problem: the first objective that the new estimate collects as much reward as the initial estimate in the training. The second objective then solves a max-min robust optimization problem for the test instance.
Future Work:

---

Acknowledgments
Acknowledgements: This research is partially supported by the European Union's Horizon 2020 research and innovation program under Grant 101017008.
References
[1] Jeon, Milli, and A. D. Dragan. Reward-rational (implicit) choice: A unifying formalism for reward learning. In Advances in Neural Information Processing Systems (NIPS), Dec. 2020.
[2] D. Sadigh, A. D. Dragan, S. S. Sastry, and S. A. Seshia. Active preference-based learning of reward functions. In Proceedings of Robotics: Science and Systems (RSS), July 2017.
[3] E. Biyik, N. Huynh, M. J. Kochenderfer, and D. Sadigh. Active preference-based gaussian process regression for reward learning. In Proceedings of Robotics: Science and Systems (RSS), July 2020.
[4] E. Biyik,, M. Palan, N. C. Landolfi, D. P. Losey, and D. Sadigh. Asking easy questions: A user- friendly approach to active reward learning. In Proceedings of the 3rd Conference on Robot Learning (CoRL), 2019.
[5] N. Wilde, D. Kulić, and S. L.. Smith. Active preference learning using maximum regret. In 2020 IEEF/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10952-10959, 2020.
[6] C. Basu, M. Singhal, and A. D. Dragan. Learning from richer human guidance: Augment- ing comparison-based learning with feature queries. In Proceedings of the 2018 ACM/IEEE International Conference on Human-Rokee Interaction, pages 132-140, 2018.
[7] R. Holladay, S. Javdani, A. Dragan, and S. Srinivasa. Active comparison based learning incor- porating user uncertainty and noise. In RSS Workshop on Model Learming for Human-Robot Communication, 2016.
[8] K. Li, M. Tucker, E. Biyik, E. Novoseller, J. W. Burdick, Y. Sui, D. Sadigh, Y. Yue, and A. D. Ames. Roial: Region of interest active learning for characterizing exoskeleton gait preference landscapes. In International Conference on Robotics and Automation (ICRA), May 2021.
[9] N. Wilde, D. Kulić, and S. L. Smith. Bayesian active learning for collaborative task specifi- cation using equivalence regions. IEEE Robotics and Automation Letters (RA-L), 162): 1698, Apr. 2019. ISSN 2377-3766.
[10] N. Wilde, A. Blidaru, S. L. Smith, and D. Kulić. Improving user specifications for robot behavior through active preference learning: Framework and evaluation. The International Journal of Robotics Research (IJRR), 39(6):651-65) 2020.
[11] A. Bajcsy, D. P. Losey, M. K. O'Malley, and A. D. Dragan. Learning robot objectives from physical human interaction. In Proceedings of the 1st Conference Robot Learning (CoRL), pages 217-226. PMLR, 2017.
[12] M. Palan, N. C. Landolfi, G. Shevchuk, and D. Sadigh. Learning reward functions by integrat- ing human demonstrations and preferences. In I Proceedings of Robotics: Science and Systems (RSS), June 2019.
[13] M. Li, A. Canberk, D. P. Losey, and D. Sadigh. Learning human objectives from sequences of physical corrections. International Conference on Robotics and Automation (ICRA). IEEE, 2021.
[14] M. Cakmak, S. S. Srinivasa, M. K. Lee, J. Forlizzi, and S. Kiesler. Human preferences for robot-human hand-over configurations. In 2011 IEEE/RSJ International Conference on Intel- ligent Robots and Systems, pages 1986-1993. IEEE, 2011.
[15] N. Wilde, F. Biyik, D. Sadigh, and S. Smith. Learning reward functions from scale feedback. In Proceedings of the 5th Annual Conference Robot Learning, pages 353-362. PMLR, 2021.

---

[16] V. Myers, E. Biyik, N. Anari, and D. Sadigh. Learning multimodal rewards from rankings, In Proceedings of the Sth Conference on Robot Learning (CoRL). Nov. 2021.
[17] D. Brown, W. Goo, P. Nagarajan, and S. Niekum. Extrapolating beyond suboptimal demon- strations via inverse reinforcement learning from observations. In International Conference Machine Learning, pages 783-792. PMLR, 2019.
[18] M. Kollmitz, T. Koller, J. Boedecker, and W. Burgard. Learning human-awar robot navigation from physical interaction via inverse reinforcement learning. In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1025-11031. IEEE, 2020.
[19] Y. Cui and S. Niekum. Active reward learning from critiques. In International Conference on Robotics and Automation (ICRA), pages 6907-6914, IEEE, 2018.
[20] D. S. Brown, W. Goo, and S. Niekum. senter-taa-ddmonnttaae imitation learning via automatically-raakedd demonstrations. In Proceedings of the 4th Conference on Robot Learning (CoRL., pages 330-359. PMLR, 2020.
[21] F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems (NIPS), pages 4299-4307, 2017.
[22] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pages 1126-1135. PMLR, 2017.
[23] D. Haddield-Menell S. J. Russell, P. Abbeel, and A. Dragan. Cooperative inverse reinforcement learning. Advances in Neural Information Processing Systems (NIPS), 29, 2016.
[24] V. Myers, E. Biyik, N. Anari, and D. Sadigh. Learning multimodal rewards from rankings. In Proceedings of the 5th Annual Conference on Robot Learning, pages 342-352. PMLR, 2021.
[25] D. S. Brown, Y. Cui, and S. Nickum. Risk-aware active inverse reinforcement learning. In Proceedings of the 2nd Conference on Robot Learning (CoRL), pages 362-372. PMLR, 2018.
[26] N. Wilde, A. Sadeghi, and S. L. Smith. Learning submodular objectives for team environmental monitoring. IEEE Robotics and Automation Letters, 7(2):960-967, 2021.
[27] M. Mohri, A. Rostamizadeh, and A. Talwalkar. Foundations of machine learning. MIT press, 2018.
[28] S. Levine, Popovic, and V. Koltun. Nonlinear inverse reinforcement learning with gaussian processes, Advances in Neural Information Processing Systems (NIPS), 24, 2011.
[29] J. Fu, K. Luo, and S. Levine. Learning robust rewards with adverserial inverse reinforcement learning. In International Conference on Learning Representations, 2018.
[30] A. Y. Ng, D. Harada, and S. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In International Conference on Machine Learning (ICML), volume 99, pages 278-287, 1999.
[31] A. Gleave, M. Dennis, S. Legg, S. Russell, and J. Leike. Quantifying differences in reward functions. In International Conference on Learning Representations (ICLR), 2021.
[32] D. S. Brown, J. Schneider, A. Dragan, and S. Niekum. Value alignment verification. In Inter- national Conference on Machine Learning, pages 1105-1115. PMLR, 2021.
[33] P. Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Pro- ceedings of the twenty-first international conference on Machine learning, page 1. ACM, 2004.
[34] E. Biyik, D. P. Losey, M. Palan, N. C. Landolfi, G. Shevchuk, and D. Sadigh. Learning reward functions from diverse sources of human feedback: Optimally integrating demonstrations and preferences. The International Journal of Robotics Research (IJRR), 41(1):45-67, 2022.
10


Reviews:
[{"review_id": "dBXf7DLRccC", "reviewer": "Reviewer_ktP2", "summary": "This paper analyzes and evaluates several different measures for evaluating reward learning. The authors provide a theoretical worst-case analysis showing limitations of common measures and provide empirical tests across several domains. \n\n---\nI thank the reviewers for their responses. I still struggle with the novelty of this paper as it relates to well known theoretical and empirical work. However, I can see that there may be benefit to explicitly discussing these issues explicitly so I am raising my score to a weak reject.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- interesting problem area\n- good empirical study of several reward learning measures\n- includes both theoretical and empirical analysis\n\nWeaknesses\n- not focused on robotics\n- theory seems overly simple (focused on simple 2 feature examples) and it is unclear how useful the worst-cast analysis is\n- missing related work\n- not broad enough to qualify as a survey", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The authors study an interesting machine learning problem. However, I do not think there is enough depth or breadth for CoRL and while there is a strong connection to robot learning, the paper seems better suited for a machine learning conference. The analysis seems overly simple and does not seem to contribute to the field---many of the claims made are commonly understood in the field. While I appreciate the effort to survey measures for reward learning, the paper provides insufficient breadth for a survey and does not have as much novelty as claimed. There are several important missing works and the ideas of reward measures being uninformative and the importance of testing in transfer domains do not appear to be significant contributions given prior work.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations section not present", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "It is unusual to have a reward function that maps directly from trajectories to rewards. Most work considers R(s) or R(s,a).\n\nIt is well known that just comparing reward parameters is not a good way to evaluate a learned reward. Indeed, much work has studied and analyzed the problem of which rewards lead to the same optimal policy [1] and which rewards make a given policy optimal [2]. \n\n[1] Brown et al. \"Value alignment verification.\" International Conference on Machine Learning. 2021.\n\n[2] Ng and Russell. \"Algorithms for inverse reinforcement learning.\" ICML. 2000.\n\nAlso, no mention is made of the common practice to test learned rewards in transfer domains:\n- Levine et al. \"Nonlinear inverse reinforcement learning with gaussian processes.\" Advances in neural information processing systems 24 (2011).\n- Fu, Justin, Katie Luo, and Sergey Levine. \"Learning Robust Rewards with Adverserial Inverse Reinforcement Learning.\" International Conference on Learning Representations. 2018.\n\nRecent work has looked at characterizing learned reward functions:\n- Michaud, Eric J., Adam Gleave, and Stuart Russell. \"Understanding learned reward functions.\" arXiv preprint arXiv:2012.05862 (2020).\nand also, importantly, at evaluating learned rewards using metrics other than alignment or mean squared error:\n- Gleave, Adam, et al. \"Quantifying Differences in Reward Functions.\" International Conference on Learning Representations. 2020.\n\nThe paper mentions other measures in 2.3, but does not evaluate them. If this is designed to be an analysis and survey paper, then I think these should be included for study.\n\nIn table 1, R^rel is defined but I do not find any definition of R^abs\n\nLine 117: [1] and [2] above provide ways of knowing whether the robot will solve the task, even if w' != w^user\n\nLine 160: Shouldn't this be -N + 1? [looking at the values in line 155]\n\nPresentation recommendations: It would be beneficial to show the different domains being studied. Based on the descriptions in the paper it is hard to tell how complex or interesting these tasks are.\n\nNo limitations section! This was explicitly asked for in the author instructions. The authors need to address limitations of their paper and analysis.\n\nEnd of section 6 has some interesting ideas, but they seem very difficult and it would be nice to provide some ideas for how they could be accomplished. How would the robot modify its reward estimate. Also, future work direction (1) seems like it is already done in the literature. Many works on active learning select training instances."}}, {"review_id": "P4rRbDnNQ7Im", "reviewer": "Reviewer_GMPv", "summary": "This paper brings up some of limitations underlying existing methodologies for evaluating reward learning algorithms in HRI. After classifying existing evaluation methodologies into two different classes, the authors demonstrate practical issues through empirical and theoretical analysis over a set of planning benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*\n\n-I enjoyed reading this paper. It is very well written and easy to follow. I liked the balance between exemplary discussions, theoretical work, and empirical work. Well done!\n\n-The paper brings up some of the limitations of existing reward learning approaches. The point is interesting and timely. While there is a very rich literature in this space with major claims and cool demonstrations, it looks like the evaluation methodologies followed are often fundamentally flawed. Specifically, it looks like the measures that many researchers tend to use to evaluate how well a robot has adapted to a user's preferences are not reliable indicators.\n\nWeaknesses\n-The paper does not discuss related work but rather lists a large set of references. Most of the references are coming from 1-2 groups. If this aspires to serve as a survey as the authors state, I would recommend a richer and deeper discussion of the literature within the narrative of the introduction.\n\n-This seems to be in part a \"statement\" paper with relatively limited novelty. The main point is definitely intriguing and interesting but the key insight is relatively straightforward and the final discussion is quite limited. It would have been more interesting/impactful (to me) to expand upon the closing discussion on what comes next and what paths might be viable. I understand that the space limitations might not allow that but in its current form, I feel like the paper is missing an insightful discussion.\n\n-The authors discuss this briefly but it seems to me that one fundamental limitation of this whole space of approaches is the assumption that w^{user} is somehow available when in the whole field of HRI this is most likely not the case. The included analysis (as well as the reviewed approaches) makes use of that assumption. Perhaps it would be valuable to expand a bit on that.\n\n-The limitations statement, a required component for this year's conference is missing.\n\n*Minor*\n\nLine 32: “These observation…\" --> These observations \n\nExplicitly define R_abs\n\nMake the scale of axes in Fig. 1 equal \n\nLine 207: “five categories” --> four categories\n\nClearly name the planning problems, perhaps use bold to define Dubins, Driver, Server in text so that the correspondence from text to table 3 is clear.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, this paper makes an interesting and intriguing point about the rigor of evaluation methodologies in the field of reward learning for HRI. It is well written and makes nice use of examples, theoretical and empirical work to support the main observation. While interesting, the main point is still relatively straightforward and I would have appreciated a more elaborate discussion on what comes next. I am generally favorable if such a discussion is included.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations section not present", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "-How were the five problem instances in Sec. 5.1 decided? How much they vary and how representative are they?\n\n-Is there a reason why the server experiment is not featured in Fig. 2? Is it not interesting?\n\n-Include a deeper discussion of the literature and a more insightful closure at the end of the paper. See my notes above.\n\n-Add a limitations statement.\n"}}, {"review_id": "8tt0WRjPqIf", "reviewer": "Reviewer_PQx1", "summary": "The paper is an unusual contribution for a robotics conference. It does not directly contribute to advancing understanding of robot control or learning. Rather it presents a critical analysis of the measures used to assess the quality of policies learned via HRI driven reward learning.  The key disadvantages of the two main classes of performance metrics (Parametric similarity and Relative Reward) are demonstrated with a simple example, theoretical mathematical abstraction, and numerical simulation on several examples. \nCentral to the paper are two proofs: (1) that even when the alignment of the learned weight vector is arbitrarily close to 1, there is not upper bound on the difference between the learned reward and the optimal reward(according to user preferences). (2)no lower bound exists on the relative reward of the test data. These two theorems are further demonstrated with evaluation of randomly sampled weight vectors in 2 examples.\n", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "#### Strengths\n- The motivating example in section 3 provides a clear explanation of the way in which the two classes of measures might fail.\n- The proofs and numerical results support the central contribution that typical measures do not provide guaruntees of the actual performance of the learner.\n- Overall ideas are well-communicated.\n\n#### Weaknesses\n- There’s no robots? This seems like it may not be the best fit for CoRL. The criticisms of reward learning could be applied to many different learning problems. The paper does not make a convincing case as to why it is critical to robot learning in particular.\n- The paper lacks a discussion of what aspects of the measures make them fail. It just states that they do. The discussion poses very generic questions.\n- Given that there are these possible worst-case scenarios, the paper could talk about how existing works (the first 15 references) avoid these worst-case scenarios or if there are published instances in the sim2real literature where this might be part of the issues.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall the paper is well-written and presents an interesting perspective. This perspective could be valuable to the greater community in analyzing why learners fail despite high performance expectations based the chose performance measure. The paper does lack some discussion of either why these performance measures fail, more specific requirements for 'good' measures, or how existing measures might be modified or combined to avoid some of these pitfalls. \nThe main issue is that the work includes neither a robot nor any learning, and the manuscript does not make a strong case for why this is critical to the robot learning community specifically as opposed to machine learning community in general. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations section not present", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "#### Major Issues\n-  The paper needs some explanation of how prior works have avoided the pitfalls presented by the paper.\n-  More discussion of either the specific attributes of the measures that lead to failure in the given cases or how one might approach a reward learning problem to avoid the 'worst-case' scenarios is needed.\n\n#### Minor Issues\n- Line 33: the wording says high(but not optimal) but before explaining the measure I think it makes more sense to say that the parameters values are very similar (but not identical)\n- Line 162: now -> no\n- Line 180: now -> no\n\n"}}, {"review_id": "8CjPzA2rtGP", "reviewer": "Reviewer_LhcT", "summary": "This work serves to point out that the current methods of evaluating reward function algorithms do not appear to be sound. In particular, the paper shows both parameter and reward based comparisons to ground truth reward functions can fail to give a reasonable sense of how good a given solution is, and by proxy, how effective the associated algorithm is. Both theoretical and empirical arguments are provided to support their case. \n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper demonstrates a fault with the evaluation methodology employed by the community. In my opinion, it does a good job in showing the existence of the issue, which is perhaps tacitly known but, to my knowledge, had yet to be formally investigated. \n- Both theoretical and empirical evidence / arguments are presented, which makes it clear the problem exists across multiple settings. \n- A well-written paper that is easy-to-follow (up to the points below)\n\nWeaknesses:\n- Although the paper clearly shows a problem exists, it is not clear how severe or widespread the issue is, especially with more complex robot tasks. The theoretical results are based on very simple constructed examples, and the numerical results are for relatively simple domains. The paper would be more impactful if the authors can discuss the implications of their work, e.g., if previous comparisons in the cited works are invalidated? \n- The multi-scenario setting in Sec 3.2 points to mis-modeling of the problem; the path length and jerk are not invariant to the goal location? Is it that the rewards are transferable from train to test, but the relative difference between accumulated rewards is not preserved? \n- On a related note, the paper falls short of analyzing why and when the reward/parameter measures fail to be meaningful. I believe the paper will be more impactful if the authors can provide a discussion around the factors affecting the veracity of these measures. Alternatively, this limitation should be discussed in the paper. \n- The paper lacks a limitations section.\n- The paper has several typos, e.g., \n   - line 115: “they do consider” -> “they do *not* consider”\n   - line 162: “now upper bound” -> “no upper bound”\n   - line 180: “now” -> “no”", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, I believe the paper provides interesting results that will make the community re-evaluate our we should compare algorithms for reward learning. There are some minor issues with the work, but I think the results are interesting enough to warrant publication. A major weakness of the paper is that it does not clearly identify the limitations of the provided theoretical and empirical analysis. \n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations section not present", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Please see the weaknesses listed above and I hope the authors can modify the paper to provide a more clear statement of the limitations surrounding their analysis. ", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)