Assisted Robust Reward Design
Authors: Jerry Zhi-Yang He
Anca Dragan
Venue: CoRL2021 Poster (2021)
Keywords: Reward Design
Safety
Human-in-the-loop
Active Learning

Abstract: Real-world robotic tasks require complex reward functions. When we define the problem the robot needs to solve, we pretend that a designer specifies this complex reward exactly, and it is set in stone from then on. In practice, however, reward design is an iterative process: the designer chooses a reward, eventually encounters an ``````''edge-case'' environment where the reward incentivizes the wrong behavior, revises the reward, and repeats. What would it mean to rethink robotics problems to formally account for this iterative nature of reward design? We propose that the robot not take the specified reward for granted, but rather have uncertainty about it, and account for the future design iterations as future evidence. We contribute an Assisted Reward Design method that speeds up the design process by anticipating and influencing this future evidence: rather than letting the designer eventually encounter failure cases and revise the reward then, the method actively exposes the designer to such environments during the development phase. We test this method in an autonomous driving task and find that it more quickly improves the car's behavior in held-out environments by proposing environments that are ``''edge cases'' for the current reward.

Assisted Robust Reward Design
Jerry Zhi-Yang He Electrical Engineering and Computer Science
Anca D. Dragan Electrical Engineering and Computer Science University of California, Berkeley anca@berkeley. edu
University of California Berkeley hzyjerryyberkeley edu
Abstract: Real-world robotic tasks require complex reward functions. When we define the problem the robot needs to solve, we pretend that a designer specifies this complex reward exactly, and it is set in stone from then on. In practice, however, reward design is an iterative process: the designer chooses reward, eventually encounters an "edge-case" environment where the reward incentivizes the wrong behavior, revises the reward, and repeats. What would it mean to rethink robotics problems to formally account for this iterative nature of reward design? We propose that the robot not take the specified reward for granted, but rather have uncertainty about it, and account for the future design iterations future evidence. We contribute an Assisted Reward Design method that speeds up the design process by anticipating and influencing this future evidence: rather than letting the designer eventually encounter failure cases and revise the reward then, the method actively exposes the designer to such environments during the development phase. We test this method in a simplified autonomous driving task and find that it more quickly improves the car's 's behavior in held-out environments by proposing environments that are "edge cases" for the current reward.
Keywords: Reward Design, Safety, Human-in human-in-the-loon in-the-lopp
Introduction
One popular way to design robotic agents is to program them to maximize the cumulative reward in their environments. While we have made great progress in solving this optimization problem, often delegate finding a good reward function to the human designer. In reality, reward design is an iterative and challenging process. Human designers spend tremendous effort iterating the reward function as they test the robot in more and more environments. What is more, once deployed, reward functions can still be incorrect edge-case environments, leading failures.
Take, for instance, autonomous cars. They have to correctly balance safety, efficiency, comfort, and abiding by the law. A human designer starts by looking at some representative traffic scenarios and specifying a reward function that leads to desired behaviors each of them. But working well in this set of environments is not enough the reward function has to incentivize the right behavior in any environment the car will encounter in its lifetime!. The engineer will test-drive the car in both simulation and in the real world. Almost inevitably, optimizing for the specified reward will lead to some undesirable behavior in some further environments. The engineer will then revise the initial reward and repeat the process. If they are lucky, they will have converged function by deployment time. But for many systems, this is a never ending process: the car will encounter some edge-cases in the real world that they did not foresee an unseen road layout or traffic situation - and the engineer revises the reward again on the new environment.
suitable reward
In this work, we explore what it would mean for the robot explicitly account for this iterative nature of reward design. Rather than treating the current reward as set in stone, we envision a system in which the robot works together with the designer and helps them design the right reward function, as Fig. 1: the robot takes the reward function specified by the designer, and proposes a new environment for the designer investigate where the reward function might fail. To capture this formally, we first enable the robot to explicitly account for the uncertainty in the reward function
While there are challenges building world model, planning algorithms, etc., for this paper, we assume success in that and focus on designing the reward function.
5th Conference on Robot Learning (CoRI. 2021), I London, UK.

---

P(Wlw",M,
P(w"i, t0 Ma
00
48 00
W
Mi+1
M2
M2
M
M
Figure 1: In the Assisted Reward Design process. the designer specifies a proxy reward i on a s of environments MI....r The robot agent takes the current proxy, infers about the true reward w" and queries the designer with a new environment where the proxy fails. The designer then revises her proxy work on this new environment. itself: every reward revision does not define the reward function fully, but is instead an evidence about it. Previous work on Inverse Reward Design [1] incorporates this uncertainty into the reward function with the goal of making the robot plan more conservatively in new environments. work, rather than passively updating the robot's estimate of the reward, we argue that it is the robot's s job to actively help the engineer design more robust reward functions, with the goal of performing well at deployment time. We refer to this as Assisted Reward Design. The key insight is that:
this
By accounting for the iterative nature of reward design, the robot does not just passively estimate the reward it receives but proactively inffuences the future evidence it can gather from the designer.
Our contributions are three-fold:
1. Formalizing reward design as an iterative process. We formalize the problem of iterative reward design as a Meta MDP with hidden state. Under this formulation, the robot can and should account for the future iterations of the reward as fulure evidence. The robot can then make these pieces of evidence more informative by exposing the designer to environments where revisions are most likely needed. Rather than letting the designer eventually encounter edge-case environments, our method proposes candidate environments and asks for reward designs, inserting itself in the design process loop, in Fig. 1. This has two implications. In cases where the designer would converge to a good reward before deployment time, our method speeds up the iterations by focusing on informative environments. Perhaps more importantly, in cases where the designer might still otherwise have the wrong reward when deploying the system, our method has the potential to expose the edge-case environments that cause failures, prompting fixes in the reward ahead of time.
Providing an approximate algorithm for our method. It is intractable to compute the reward uncertainty given the nigh-dimensional nature of the reward space and the environment space. We propose an approximate algorithm based on particle filtering to enable efficient computation.
3. Analyzing Assisted Reward Design in simplified autonomous car domain. We evaluate our method's practicality and effectiveness in designing reward functions for autonomous cars, using the open-source environment proposed by Sadigh et al. [2]. We conduct experiments with both simulated reward designers and robotics experts. We find that our method uncovers more difficult test scenarios and speeds up the reward design process. We also showcase that it proposes interesting environments that tend to be edge cases where the current reward estimate fails.
Overall, this paper takes a step towards making our robotic systems understand reward specification as iterative process. In the end, would like robotic systems not just to optimize what design for them but help us design better and more robust reward functions. Even though admittedly more work is needed put these ideas in real- weal-world systems, especially in light of the need for a simulator or learned model of environments, we are excited to see our method assist in difficult reward design tasks and produce edge-case environments without any hand-coded heuristics.
2 Related Work
Designing robots work reliably in an open world remains challenging problem. Our work bridges three lines of work towards this goal: teaching robots the correct behaviors, enabling robots to actively learn from humans, and ensuring safety and robustness when deploying robots.
Learning Rewards from Diverse Human Input Given the difficulty of specifying reward functions, researchers have explored wide variety of different methods to enable robots to learn rewards instead

---

from human input, especially demonstrations [3, 5, 6]. Using this idea, recent works have explored learning from diverse set of modalities. These include language instructions [7. 8], numerical feedback [9, 10], comparisons [11, 12], advice [13], facial expressions [14], collaborative behavior [15], or corrections [16, 17]. In contrast, our work builds on [1], which uses the specified reward function itself as the human input to learn from. We generalize this idea to an iterative, active process.
Active Reward Learning Our work an instance of active reward learning. Prior work has focused on actively querying for comparisons [18, 19]. demonstrations [20, 21, 22. 23], or reward labels [24]. Such queries are typically for trajectories or states in a given environment (MDP). Recent work has proposed active learning to synthesizes the environment itself to gain more information about the reward (either the initial configuration and behavior of other agents in the scene, such as [12], or the entire environment [24]). We also propose to actively query new environments, but the human feedback is a proxy reward designed to work only in that environment. This interesting for two reasons. First, we show that it leads to the robot proposing edge cases to the designer where the previous reward is likely to fail, thus aiding in revising the reward. Second, we argue that the same active reward learning ideas from the literature can be applied to the reward engineering process when we design a reward function, we should not just have the robot go optimize it blindly. We should instead think about it as an iterative process through the lens of active reward learning.
Reward Shaping Many prior works have also focused on recovering reward functions for suboptimal planners, given a ground truth reward function that is difficult to optimize, e.g. due to sparsity or non-differentiability [25, 26]. Note that in many real-world problems, such as autonomous cars, the ground truth reward is not obvious to write down. Assisted Reward Design is different from the reward shaping problem in that we do not assume access to ground-truth reward function, but seek techniques that leverage proxy rewards that are the designer's 's best guesses.
Safety in Real-World Robots Robots deployed in the open-world need handle large set of possible environment configurations, be it a busy road for an autonomous car or a messy kitchen for a home robot. Given this challenge, there has been rising number of work on safeguarding robot behaviors by generating edge cases. Koren et al. [27] and O' D'ellly et al. [28] studied using black-box optimizers to perform adaptive testing and generate failure cases. [29] trains a generative model of environments from datasets. Uesato et al. [30] generates edge cases by learning a failure predictor. These methods require defining hard constraints that the robot should not violate. Similarly, reward shaping work has looked at how to incorporate such constraints into the reward [31].
In contrast, we focus on settings where such hard constraints are incomplete or difficult specify. Our method does not assume access to such constraints. We show that our method can learn to develop edge-case scenarios that implicitly violate constraints without having to specify them.
3 Problem Statement
We first describe the Unassisted Reward Design problem works, then we formulate the Assisted Reward Design problem and introduce our method.
3.1 The Unassisted Reward Design Problem
Problem Setup Let an "environment" M be a Markov Decision Process without the reward function. At development time, we assume access to a large set of such environments, Maevel- We assume this is a large set that the designer cannot exhaustively test. This can be, for instance, all the environments in a several-million-iile autonomous driving dataset. Or, it can be a parameteric set of possible environment configurations, such as placements of objects, roads, and other agents. Finally, it can also be a generative model of the world. The robot will run at deployment time in a different set of environments Maeploy. which we do not have access to during development time."
We further assume access to a space of reward functions parameterized by w E W these can be linear weights on pre-defined features or weights in neural network mapping raw input to scalar reward. We denote by w' weights of the ground truth reward function, which induces the desired behavior in Maevel and Masplay We denote by & the trajectory (or policy, but for this work, we consider deterministic MDPs with set initial states) optimal to the cumulative reward induced by w
"Our method works best when we can induce a vast set Maesel that includes all possible scenarios in Maeplay- Otherwise, if Maeploy is allowed to be drastically different, and if Maevel is not enough to design a robust reward function, this exposes the robot unanticipated failures after deployment.
3

---

random proposals
infogain proposals
Development (274k)
Deploy (91k)
Dificully
(a)
(b)
(c)
(d)
Figure 2: Left: (a) The driving environment with autonomous vehicle (yellow) and human-driven vehicles (red). (b)(e)d) highlights selected features: obstacle distance, vehicle distance, progress. Right: the distribution of Maevel (blue) and Macploy (pink) based on difficulty metric.
in M. Éw.M arg m & R (f: Our assumption is that there exists w such that Éwr,M .M is the desired behavior for any M E Maevel and any E Maeploy- We do not have access to w".
The Process of Unassisted Reward Design The designer takes a subset of environments M S Maevel and specify reward function (via parameters w)) that leads to good behavior in those environments. In a one-shot design, they would select a set M, assume it is representative of Maeploy. design w for that M, and deploy the system with w (never changing the reward again). However, reality, this is an iterative process. They start with an Mo, design a Wo, and eventually encounter during their testing new environment M' on which optimizing wo does not lead desirable behavior. Then, they would augment their set to Mi My U {M') and re-design the reward: Mo - Wo - M₁ - Wi - This might continue into deployment the reward at the time of deployment still fails on Macploy- Implicitly, at every step along the way, the robot treats the current w; as equivalent to w*.
3.2 The Assisted Reward Design Problem
We formulate the Assisted Reward Design Problem by introducing meta-MDP problem, where the meta-agent receives proxy rewards as observations from the designer and acts by proposing new environments. The goal is to minimize regret of the reward design over deployment Maeploy-
Meta-States, Meta-Actions, Observations, Transitions,3 In the Assisted Reward Design problem, the notion of "state" is set M of environments this is what changes over time and is directly observable- - along with the hidden state w". The meta-agent starts at state (Mo. w") (environments either chosen by the designer be representative, or simply the empty set). It transitions to the next state by "acting", i.e. adding one more environment M, to the set: M = M; U Mj. Every time it enters a state (M++,, w"), receives an observation Wi+1 about w* from the designer. For the meta-ages ageet to assist the designer, it needs to treat the observed reward w; not the same as w*, but as a proxy that emulates w* only on the current Mj. It can then interpret as evidence about w* and use all past evidence to obtain belief of w". To assist the designer, the key step is to account for (and thus influence) the future evidence order to get a better reward estimate.
Observation Model. Reward designers tend to select proxy rewards that work well in training environments. With this intuition, previous work [1] introduced an observation model that computes the probability of observing proxy reward w under w and designer's 's training environment set M:
(1) with & controlling how close to optimal we assume the person to be. This distribution informs the meta-agent that it is receiving proxy rewards from an approximately optimal designer who looks at the current M. We can then leverage this model infer a belief over w*: bm(w").
Pdesign (W/w", .M) x exp /MM (fR,,(
The objective of Assisted Reward Design is to achieve high reward at deployment time. What the meta-agent controls are a sequence of actions, i.e. a sequence of environments My it can propose. These lead to observations about w*. At deployment time, the meta-agent uses its belief' to generate optimal trajectories and accumulates reward according to w", We want to find the set of environments My such that when given to the designer, these environments induce a belief that leads to the best performance in Mdeploy:
Objective.
max M, EM MMe Bw-bby (w*) Rw (E (Ew.M)
(2)
FFrr the rest of this paper, all our references "state" and "action" stand for "meta state" and "meta action".

---

where Myr (w") is the meta-agent's belief at deployment time, based on the evidence gathered from the states Ma... My: : fw.M = m (f: is the optimal trajectory.
As currently stated, this objective is impossible to optimize directly because the meta-agent does not know Maepley- In what follows, introduce solution Assisted Reward Design based on the heuristic objective of identifying w" - if the meta-agent uses its actions to disambiguate what w' should be, then it can achieve optimum performance on the objective in Eq. 2 because it can plan optimally to w". We first introduce the Maximal Information objective. Then we show how the meta-agent can track a belief and perform future belief updates in order to propose new environments.
4 Assisted Reward Design via Active Info-Gathering.
Maximal Information. We invert the probability model of Eq. to obtain belief over w": P(w"|w, M). Our key insight is that we can reduce the belief uncertainty by proposing future environments to influence the designer. In other words, we would like to find environments that can disambiguate the reward as much as possible - if we can identify w" exactly, the meta-agent attains zero regret at deployment. Thus, we introduce an approximate strategy for the meta-agent: proposing environments that provide the most information gain over its belief distribution, commonly done in active learning [32, 33] and robot active exploration [34]. For computational tractability, we do a one-step look-ahead only, although the meta-agent could benefit from reasoning over multiple future iterations. Given an action M (MDP), we compute its mutual information with w:
f(M++1) I[:: H[[ Wo.l
(3)
Bw ( 1 PPoagg ( 1i-1 [w", MM-))) [[[[[Moith Woa+l]
Here H[[ wo:ll is the entropy of posterior distribution of Pi(w), and Wi+1 is a new observation sampled from the updated designer model Pdesign( (M;+1. After observing Wit1, we can compute the conditional entropy H. [| wo.++] using methods from Sec. 4.1. In our imple- mentation, we set W = w" to make Eq. more tractable and find that it empirically works well. Next we introduce two update rules to compute conditional entropy joint and independent mode.
4.1 Updating Belief's over n
Joint reward design. At iteration i+1 1 of Assisted Reward Design, the meta-agent takes an action Mi+1 (proposes an environment), and receives an observation Wi+1 reward that works over all environments M = M2 [M++]] Assuming that W:t1 is independent from previous observations, we can use 1+11 to update the belief over w using Eq. 1:
(i) or Pi(w) Paesign (W:sign(W::w.... M))
with Po the meta-agent's prior (uniform in our experiments). We take a sequential-importanee resampling particle filtering approach to approximate Pi+1: we sample N, particles from P;, compute importance weights based on Eq. 1 for each particle, and resample using the importance weights. In order compute the normalized Pdesign. we approximate the normalizer in Eq. I via samples.
Independent reward design. We also allow designers to provide an observation w; based solely on the new environment M, Prior work [35] finds that this model puts less burden the designer. Under this assumption, our belief update simplifies to relying on a single new environment:
(w/, xz P((w) ( Pdssinn Ww (w)
(5)
4.2 Adding New Features
Reward engineering often requires adding new features to the environment. Our method compatible with the designer adding features. See appendix for more details.
5
Experiments with Simulated Designers
We evaluate Assisted Reward Design in simplified autonomous driving task. In this section, hypothesize a ground truth reward function and simulate designer behaviors. Note that our method does not require such unique ground truth w" - this is merely used here for evaluation. In Sec. 6, we conduct an expert study where we do not have the ground truth.
5.1 Experiment Setup
Driving task. As illustrated Fig. 2, the autonomous car to merge to the leftmost lane. Since there may be other human vehicles (simplified as constant speed) and obstacles, the autonomous car
5

---

Independent Mode
Joint Mode
Postertor Regnet
Retative baok
Posterfor Regnet
Kelative regret
betteee
0.20
0.25
1.25
0.15
1.00
0.1
1..0
003
-
0.25
0.10
intggnn
Intogann
Inlogin
condom
dificutt
Infogain
random
difficlt
Figure 3: Experiment with simulated designers. Left: independent mode, right: joint mode. (a)(c) shows regret of the posterior mean compared to true reward under different acquisition functions. (b)() shows the efficacy of the proposed environment. The "box" denotes 5th to 95th percentile.
Maximal Information
Difficult
Random
Car Crash Unsmooth Aggressive Crash
Speeding Good
Likert Questions
['Trajectory] shows good driving behaviors.
Q2: ['Trajectory] show undesirable driving behaviors.
Figure 4: Visualization of the top proposed environments based on each acquisition function.
needs to decide whether overtake slow down. Our reward funetion based Sadigh et al. [2] a weighted sum of six features, including vehicle distance, progress, control effort, etc. Please refer to supplementary material for their details. A good reward function should correctly balance these terms and reflect our desired behaviors in all different situations. To obtain Mäevel. we parametrize environments the initial placements of obstacles and other vehicles.
Environment Difficulty. Successful merging tends to more difficult when human vehicles and obstacles closer to the autonomous vehicle. We environment: difficulty = Σ, 1/ dnuman vehicle Σ, 1/duuffic cone,- We visualize the distribution of Maevel and Macploy in Fig. 2 using this metric. Note that both Maevel and Mácploy have "long-tail "til" distribution of events events that are rare to encounter daily, but could be highly useful for the reward design. An ideal acquisition function in should be able identify these events.
simple metric to describe each
Baselines. We compare our Maximal Information acquisition function with two baselines that represent common design paradigms where the designer selects (1) the most difficult tasks ranked by M E Maevel (2) random environments from Maevel-
5.2 Experiment Results
Evaluation. A good Assisted Reward Design method should help the reward designer quickly recover high-quality reward functions. To evaluate this, we specify one set of ground truth reward w At every iteration, we simulate the designer using w" under both independent mode and joint mode as described in Sec. 4.1. Please refer to the appendix for experiment details. We plot the performance of posterior mean and the proxy reward under different methods Fig. To mitigate the effect of the choice of initial environments, and find that our active method outperforms the proxy reward w, as well as the random and difficult baseline in both joint and independent modes (p < 0.05).
experiment on six random initial environments

---

Our active method (orange) outperforms both the random baseline (grey) and the difficulty heuristic (blue) as shown in Fig. This means that certain environments, despite being rated "easy" by heuristics, turn out useful for the reward design. These environments are often edge-cases that subtly trigger undesirable behaviors- - in Fig. a rather empty road with two human vehicles triggered the autonomous vehicle to crash. Next we further investigate this.
Edge-Case Nature of Proposed Environments To study why our method performs better than the baselines, we measure the quality of the proposed environments using: r(Mnex) [Regret(W::'' .M) the ratio of regret on next environment versus average regret in deployment. Higher r(Mnex)) means the next envi- ronment better uncovers the overall regret of the current posterior. As shown in Fig. 3, our active method proposes environments of highest relative regret. This suggests that to find the most useful environment for Assisted Reward Design, simply relying on heuristic environment ranking is not enough. It is more effective to utilize all proxy rewards and their induced uncertainty over w
Be-P(wee")) Reegeet MMex)) / (BBaa-P(ww))BMM-AA
6 Experiments on Real Reward Design
We conducted study with 14 robotics experts* on the simplified autonomous driving tasks, where we ask them to design reward functions to generate their preferred driving behaviors.
Posteric Vinistia
Edge priposed loko
wwlgggn miggaa prosy
2.5
dan mandon pracy
2000
autcatt althell prosy
(tating Whigher (eneete))
1.25
1.30
6.1 Experiment Setup
1.25
We give the experts 10 minutes to familiarize with the features. We then let them design re- ward functions over four consecutive iterations, under three different methods and three different random seeds. This gives a total of 36 individ- ual reward designs. We use counterbalancing to randomize the sequence of different methods. All our experiments are conducted over Zoom.
1,00
075
0.30
Intogain
randon
diticall
Figure 5: Experiment with robotic experts. The left shows the violation under different acquisition func- tions. The right compares the relative value of proposed environments.
Violation. Since we do not have the ground truth reward function, cannot measure posterior regret as in Sec. 5. For evaluation purposes only, we introduce seven violation criteria to measure the quality of our reward design, including collision, driving off track, stopping, etc. Please refer to the appendix for detailed definitions. We use these criteria as unit tests on the behavior to capture unwanted behaviours. We then count the total number of violations of each trajectory. Better reward functions should induce lower violation counts. Although we could explicitly incorporate these tests as hard constraints in the "reward" function [31], we keep the reward features oblivious to them. This is for two main reasons. First, in the real world, constraint violations are acceptable in some edge-case scenarios: slamming on the brake, despite uncomfortable, is justifiable the sight of traffic accidents. Such trade-off's are difficult to specify and it is our goal to exploit potential edge-cases. Second, violation tests only capture subset of unwanted behaviors. Tailgating, for instance, does not break the tests. Yet, it is undesirable in most cases for the autonomous vehicle. In Sec. 6.2 we provide qualitative examples from the experiment to illustrate such undesirable behaviors.
6.2 Experiment Results
Evaluation. We perform Assisted Reward Design where experts answer the queries stemming from our algorithm, as well as the baselines. We visualize the result in Fig. left. Maximal Information outperforms the other acquisition functions in causing fewest average violations Macploy- This shows that Assisted Reward Design can help us tune a reward function that fails fewer tests as a measurable proxy for the good driving behaviors in the real world.
Edge-Case Nature of Proposed Environments. To measure the quality of the proposed environ- ments at each iteration, we ask the experts to watch the trajectories that their proxy designs generate on the proposed tasks. We then use the Likert scale questions in Fig. 4 and ask the users to rate how much they agree with the statement on a scale of 1-7. We then compute the normalized score Q1 Q2 as measure how good the behavior is. A lower score indicates better behavior. We use the inverse of the score to indicate edge-caseness in Fig. 5. Our maximal information method outperforms the heuristic difficulty metric in finding edge-case environments that contain more violations.
*from the authors' institution
7

---

1000000
Figure 6: Selected environments proposed by Maximal Information with the highest rating by users. The robot plans a trajectory using previously designed WMAP. Normalized Likert scale rating is shown on the bottom right of each scenario. Higher rating means more likely to be an edge case. Notice that these trajectorics failure cases but do violate the pre-defined difficulty metrics such collision.
Qualitative Analysis of Proposed Environments We randomly sample an initial environment and query designer. We then use maximal information, heuristic difficulty, and random acquisition function to propose the next environments. We visualize the trajectory of MAP estimate WMP each of these new environments in Sec. 6. Notice that interestingly, our method tends to find environments where the current MAP estimate fäils, even though we do not explicitly optimize for finding failure cases. It turns out that these environments help reward designers effectively narrow down on the true reward. In comparison, heuristic difficulty finds environments with dense interactions. Yet, they do not induce failure in the resulting trajectories likely because such difficulties have been addressed by past reward designs. Heuristic difficulty tends to generate repetitive environments.
Can Violation Count Replace the Maximal Information? Sec. 6.2 uses violation counts to evaluate trajectories. One may ask the question: can we simply use the violation count as an acquisition function to find edge-case environments? In other words, environments where the design leads to high collision, off-track counts, etc. should be proposed as edge cases. On surface level, this seems to be a reasonable approach. In fact, this is commonly used industry [36]. However, we argue that setting violation counts as the acquisition function limits the scope that can identify edge cases. We highlight this in Fig. 6, where we visualize selected environments proposed by Maximal Information. They exhibit subtle failures and lead to the highest Likert-scall ratings despite obeying all violation criteria. Interestingly in the second, fourth, and seventh examples, the vehicle narrowly avoids the collision. Such behaviors found by Maximal Information are hard to capture with hand- designed violation metrics. By acting on the edge of violations, the algorithm pushes the reward designer better specify her "decision boundaries". As result, Maximal Information can propose more diverse edge cases and gather designer preference more quickly.
Discussion
We contribute Assisted Reward Design, an active reward learning method that generates environments and queries for reward functions as user inputs. We find experimentally that our method exposes edge cases environments where the current proxy reward leads to high regret. Experiments with robotic experts confirmed that these environments are edge cases that lead undesirable behaviors.
Our method is applicable to model predictive control problems such as drone, indoor robot navigation, motion planning for manipulation [35] where we have known dynamics, as well as RL problems given the environment simulator or learned model. Given that naively applying RL in the inner loop of reward inference is intractable, it remains an open question how to efficiently combine with RL.
There are several limitations to our approach. Firstly, our environment proposal algorithm could be more efficient. One may explore synthesis and continuous optimization over the environment parameters. Secondly, we need of formal analysis and guarantee of generating edge cases. Thirdly, method assumes access to distribution of environments. Obtaining such realistic distributions [37, 38, 29] for simulation remain an open problem for the community.
8

---

Acknowledgments
We thank Dylan Hadfield-Menell, Ellis Ratner and Jessy Lin for their feedbacks on this work. This project is supported by National Science Foundation NRI, Air Force Office of Scientific Research (AFOSR), the Office of Naval Research (ONR-YIP).
References
[1] D. Medelll S. Milli, P. Abbeel, S. J. Russell, and A. Dragan. Inverse reward design. In Advances in neural information processing systems, pages
2017.
[2] D. Sadigh, S. Sastry, S. A. Seshia, and A. D. Dragan. Planning for autonomous cars that leverage effects human actions. In Proceedings of Robotics: Science and Systems, 2016.
[3] Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning, 2004.
[4] A. Y. Ng and S. J. Russell. Algorithms for inverse reinforcement learning. In Pruceeding of the 17th International Conference Machine Learning. 2000.
[5] B. D.D. Ziebart, L. J.M. Bagnell, and A. K. Dey. Maximum entropy inverse learning. 2008
[6] N. D. Ratliff, J. A. Bagnell, and M. A. Zinkevich. Maximum margin planning. In Proceedings of (ICML) International Conference on Machine Learning. 2006.
[7] E. C. Williams, N. Gopalan, M. Rhee, and S. Tellex. Learning to parse natural language to grounded reward functions with weak supervision. In 2018 IEEE International Conference Robotics and Automation (ICRA), 2018
[8] J. Fu, A. Korattikara, S. Levine, and S. Guadarrama. From language to goals: Inverse rein- forcement learning for vision-based instruction following. CoRR, 2019.
[9] M. Bloem and Bambos. Infinite time horizon maximum causal entropy inverse reinforcement learning. 2014.
[10] W. B. Knox and P. Stone. Tamer: Training agent manually via evaluative reinforcement, In 2008 7th IEEE International Conference on Development and Learning, 2008
[11] P. F. Christiano. J. Leike, T. Brown, M. Martic, S. LLgg, and D. Amodei. Deep reinforcement learning from human preferences. 2017.
[12] D. Sadigh, A. D. Dragan, S. Sastry, and S. A. Seshia. Active preference-bas based learning of reward functions. In Proceedings of Robotics: Science and Systems, 2017.
[13] Odom and S. Natarajan. Active advice seeking for inverse reinforcement learning. Interna- tional Foundation for Autonomous Agents and Multiagent Systems, 2015.
[14] Y. Cui, Q. Zhang, A. Allievi, P. Stone, S. Niekum, and W. B. Knox. The empathic framework for task learning from implicit human feedback. CoRR, 2020.
[15] D. Hadnield-Menell, A. Dragan, P. Abbeel, and S. Russell. Cooperative inverse reinforcement learning. CoRR, 2016.
[16] A. Jain, Sharma, Joachims, and Saxena. Learning preferences for manipulation tasks from online coactive feedback. CoRR, 2015.
[17] A. Bajesy, D. P. Losey, M. K. O'Malley, and A. D. Dragan. Learning robot objectives from physical human interaction. Conference on Robot Learning (CoRL), 2017.
[18] E. Biyik and D. Sadigh. Batch active preference-bate based learning of reward functions. PMLR, 2018.
[19] M. Tucker, E. Novoseller, C. Kann, Y. Sui, Yue, W. Burdick, and A. D. Ames. Preference- based learning for exoskeleton gait optimization. 2020.

---

[20] D. S. Brown, Y. Cui, and S. Niekum. Risk-aware active inverse reinforcement learning, 2019.
[21] C. Daniel, O. Kroemer, M. Viering, J. Metz, and J. Peters. Active reward learning with a novel acquisition function. Autonomous Robots, 2015.
[22] C. Daniel, M. Viering, J. Metz, O. Kroemer, and Peters. Active reward learning. In Proceedings of Robotics: Science and Systems, 2014.
[23] M. Lopes, F. Melo, and L.. Montesano. Active learning for reward estimation in inverse reinforcement learning. In Join! European Conference on Machine Learning and Knowledge Discovery Databases, pages 31-46. Springer, 2009.
[24] S. Reddy, A. D. Dragan, S. Levine, Legg, and J. Leike. Learning human objectives by evaluating hypothetical behavior, 2019.
[25] S. Singh, R. L. Lewis, and A. G. Barto. Where do rewards come from. 2010.
[26] M. Riedmiller, R. Hafner, T. Lampe, M. Neunert, J. Degrave, T. Wiele, V. Mnih, N. Heess, and
J. T. Springenberg. Learning by playing solving sparse reward tasks from seratch. 2018.
[27] M. Koren, S. Alsaif, R. Lee, and M. Kochenderfer. Adaptive stress testing for autonomous vehicles. 2018.
[28] M. O'Kely,, A. Sinha, H. Namkoong, Duchi, and R. Tedrake. Scalable end-to-end autonomous vehicle testing via rare-event simulation. CoRR, 2018.
[29] G. Izatt and R. Tedrake. Generative modeling of environments with scene grammars and variational inference. 2020 IEEE International Conference on Robotics and Automation (ICRA), 2020.
[30] J. Uesato, A. Kumar, C. Szepesvari, T. Erez, A. Ruderman, K. Anderson, N. Heess, P. Kohli, et al. Rigorous agent evaluation: An adversarial approach to uncover catastrophic failures. 2018.
[31] Achiam, D. Held, A. Tamar, and P. Abbeel. Constrained policy optimization. 2017.
[32] N. Houlsby, F. Huszár, Z. Ghahramani, and M. Lengyel. Bayesian active learning for classifi- cation and preference learning. 2011.
[33] Y. Gal, R. Islam, and Z. Ghahramani. Deep bayesian active learning with image data. 2017.
[34] W. Burgard, D. Fox, and S. Thrun. Active mobile robot localization. Citeseer, 1997.
[35] E. Ratner, D. Hadfield-Menell, and A. D. Dragan. Simplifying reward design through divide- and-conquer. 2018.
[36] R. Tedrake. Tri taking on the hard problems in manipulation research toward making human- assist robots reliable and robust. 2019.
[37] A. Dosovitskiy, G. Ros, F. Codevilla, Lopez, and V. Koltun. Carla: An open urban driving simulator. PMLR, 2017.
[38] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. CoRR, 2019.
[39] T. Schaul, D. Horgan, K. Gregor, and D. Silver. Universal value function approximators. In Proceedings of the 32nd International Conference Machine Learning, 2015.
[40] Bradbury, R. Frostig, P. Hawkins, M. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman- and Q. Zhang. JAX: composable transforma- tions of Python+NumPy programs, 2018.
[41] Moritz, R. Nishihara, S. Wang, A. Tumanov, Liaw, E. Liang, M. Elibol, Z. Yang, W. Paul, M. I. Jordan, et al. Ray: A distributed framework for emerging (AI) applications. 2018.
10

---

A Algorithm
Algorithm. We now present our main algorithm for Assisted Reward Design. At every iteration i, use the Maximal Information acquisition functions to select the next environment Mi+1 and query the reward designer. Note that the environment space Máevel is continuous and high dimensional and it is intractable exhaustively search through it. We thus uniform sampling to select candidate set Meand E Maevel- Note that one can employ different heuristics for selecting such candidate sets, and leave the heuristic design to future work.
Representing Belief Distribution. We use particles to represent Pi(w w*): at each step, we sample N, particles from (w = w"), compute importance weights based on Eq. for each particle, and resample using the importance weights.
Algorithm 1 Assisted Reward Design via Info-Gathering
Require prior Po(w), Maevel, Neand, Np. initial training environments Mo Initialize posterior Po(w = ==)== Po(w)
=
for 00.., do
W;- P (user(W[w" [/", MM) ) { Querr the designer on M; } Compute posterior P:+1(W w") using Eq. or Eq. 5
Sample Neand candidate environments Meand E Maevel
for M E Mcand do Compute f(M) end
Select M aaw.aaaa f(M)
M1+1 M2 U MM{(Ma) 11++
end
Complexity. Our algorithm is bounded by the number of particles Np for representing posterior distribution and the number of candidates Neand- At every iteration, the algorithm needs solve for each particle in each candidate environment, which leads to total of O(N -Naad) planning problems order to compute belief update. This the main speed limit. While one can potentially speed up by learning fast planners [39]. we leave this to future work. In our experiments, we implement the environment as in Sec. B such that the dynamics and reward function are both vectorizable. We then concatenate the reward functions of O(Np Neand) problems and compute batch forward planning using gradient-based planner. Note that this is not feasible for general planning problems with non-differentiable dynamics reward functions.
Adding New Features. When the designer adds in new feature Φk+1 on environment Mn, the new proxy reward has d+1 dimensions while the previous proxies have d dimensions. We can proxies
still perform reward inference on all of the d+ dimensions, as long as we incorporate received so far. The meta-agent would then have to revisit all the reward designs it's gotten so far and recompute posterior over the new augmented space. To do so, we invert Eq. using the formula from [1]:
exp (BW (i) 2(()) =
[ exp (Bw (i. $.)
P(w = w Wins M x P(w)
2
with P(w) being the prior. We use MCMC to infer the distribution. During inference, every sample w is of d+1 dimensions, and & are d I dimensions, computed from all previous proxy rewards (of or i+1 dimensions) on the existing tasks. To compute the normalizer, we integrate over i in the d+1 dimensional space.
B Driving Environment Implementation
This section provides the details of the driving environment used in the paper. We introduce definitions environment dynamics, feature and reward functions, the environment distributions, and how we implement the environment efficiently for trajectory optimization.

---

Environment Features
Feature Name
Raw Feature
Transformation Pfull
Meaning
Onow
Speed
v
-(praw ggoal)
How much the vehicle deviates from the goal speed.
Control
Msteers Mace]
The control effort by the vehicle.
Lane
x
Nor Egoal clame)
How much the vehicle deviates from target lane center.
Car
[x,y]
Ei N(Oraw Kear deari
How much the vehicle is driving close to other vehicles.
Obstacle
[x,y]
2, Noraw - Tobs dobs
How much the vehicle is driving
close to the obstacles.
F'nce
x
Kfence left &raw
How much the vechile is outside the fence
Brww - Kfence right]+
Table 1: Environment Feature Table
Dynamics We represent the forward dynamics of the MDP as xy41 f(x,, where we have the full knowledge of x. To model system dynamics, we use simple point-mass vehicle model with holonomic constraint. Let Xear = [x,y, [x.yy0,v]. T. where x. are the corrdinates of the vehicle, heading, and is the speed. We let = [M Mace] be the control input, where Mster is the steering input and Mace acceleration. We also use a as the friction coefficient. The model for a single vehicle is:
the
[ ÿ [v (v)(((())))) V. (in
Uster
Mace a' v]
(6)
We model human vehicles as moving forward at constant speed and traffic cones as static objects.
Feature and Reward Functions. In the following Table we introduce the environment features & the driving environment. Each feature composes of oraw, subset of the current state x, and control M,, and non-linear transformation that when maximized, it induces the desired behaviour that feature, such as moving to the target lane. These environment features are are differentiable and that we can characterize the desired driving behavior via a linear weighed sum +b
draw- The nonlinear transformation is designed such
Violations. Here we provide in Table 2 the definition of environment constraint functions used in the experiment section to evaluate driving quality. Each constraint is boolean function that returns true or false for one timestep. and we compute the final violation count for each trajectory by summing over constraints over the time horizon. These constraint functions are not used for optimization, but as evaluation criterion for the case study experiment. Lower violation counts correspond to better driving behavior.
Environment Constraints
Feature Name Overspeed
Definition
Meaning
v Vmax
The vehicle is driving over the maximal al-
lowed speed.
Underspeed
Vmin
The vehicle is driving below the minimum
speed on highway (i.e. backing up).
Uncomfortable
[ oo |max
The vehicle applying too much force that it's uncomfortable (i.e. accerlating too much or jerky).
Collision
F Near
S d min
The vehicle crashes into the other vehicles.
Crash Object
x Nobj ill
5 dmin
The vehicle crashes into the obstacles.
Offtrack
x Xfence left or
The vehicle drives off the left or the right fence.
Xfence right
Wronglane
x - Xlane left
S diane
The vehicle drives drives onto the wrong lane while merging.
Table Environment Constraints Table
Planning Speed. We implement the dynamics and reward function using JAX [40] and leverage the JIT compilation to speed up running time. We use shooting method and perform gradient-based planning using the Adam optimizer for 200 steps. We plan at horizon of 10 timesteps and replan

---

every 5 timesteps. When planning for a single environment, we can generate 2.57 trajectories per second. Because we can vectorize the planning problem and plan for multiple trajectories once, we can achieve 75 trajectories per second, by planning for 500 trajectories in batch. Computations are benchmarked on the c2-standard-4 instance on Google Cloud. We use batch planning for computing the belief distribution and environment proposals, as discussed in Sec. A as the main bottleneck of our algorithm.
Environment Distribution. We use a simple method to define the distribution of environments Maesign, Macploy. There are two human vehicles and two traffic cones positioned on the three-lane highway. We assume that the autonomous vehicle starts from the center of the scene (x = 0, y = 0), and sample the starting position of the other vehicles and obstacles. The other vehicles can start anywhere in Xmin car S Near max cars min car S S Ymax car and the obstacles can be initialized anywhere in can start anywhere in Xmin obs E Nobss S Xmax obs- Ymin ohs Σ y S Ymax ohs. We filter out situations where the other vehicles or obstacles are initialized to be colliding with the main vehicle.
Note that this very coarsely designed distribution showcase our method. We have several limitations. For instance, we do not exclude the environments where the other vehicles runs into obstacles. These are relatively unlikely environments, and in principle we can define more realistic distributions by more careful environment engineering or by loading real world driving datasets.
The difference between Maevel and Macploy is that in Macploy, define a tighter feasible range of Xear centered around the autonomous vehicle. This results in a shifted distribution of pseudo crcudo-dificulty metric with long-tail events. After discretization, Maevel has 274k environments and Mies has 91k environments. Given these large number of possible environments, it is impossible for reward designers enumerate them manually.
C Experiment Details
Optimal Planning For the autonomous driving task, we compute trajectories of 10 timesteps. We use finite-horizon optimal planning with regard to given rewards. We plan at a horizon of 10 timesteps, and replan every 5 timesteps.
Reward Design We use rationality factor B = πa to simulate noisy designer, and B in the inverse model. This is because we find empirically that the proxy reward quickly approaches the ground truth reward when we have multiple proxy environments. We thus divide the rationality factor B it by the number of proxy environments to maintain its noisiness. To compute the posterior probability in Eq. 4 or Eq. 5, we need to approximate the normalizing constant. We follow the approach in [1] by sampling w. In the section 4.2 of [1], they find empirically that it helped to include the candidate sample w in the normalizing sum. This requires planning with w and computing its feature sum in the MCMC inner loop, which largely slows down the inference. Instead, we include the candidate w. but multiplies it with proxy w's feature sum the normalizing constant.
We then compare three methods (random, difficulty, maximal information) in iterative fashions. We use the same initial environment for the three methods and perform 9 iterations. We aggregate results over five random seeds.
Environment Proposal. During environment proposal, we use N₂ 500 particles on Neand 64 environments. We implement a vectorized car dynamic model using JAX [40] and Ray [41] for batch planning. Our environment proposal step takes 4 minutes on c2-standard-4 instance on Google Cloud.
Evaluation. To examine the quality of our inferred posterior of the reward function, we need to evaluate the posterior in new driving environments. We thus sample 500 environments uniformly from Macploy as the deployment set for evaluation. We compute the regret, note that because of the different placement of vehicles and objects, different environment have different maximum and minimum reward they can produce. Thus it is unfair to compute the absolute regret Fmax Tw. We thus compute the relative regret, by first taking the worst cases reward Tmin in each environment. Then the relative regret is regret,, = (Fmax - T ( (Tmax max - T minn) (rw" - /r - Fmin).
3


Reviews:
[{"review_id": "dQbMH4pgL_Q", "reviewer": "Reviewer_5qJa", "summary": "This paper formalizes the reward design problem for robotic tasks as an iterative process. A human engineer can specify a reward signal for the robot, and the agent can propose environments that highlight potential unexpected consequences of optimizing that reward signal, enabling the human engineer to iteratively change the reward function until the desired behavior is consistent across environments. This paper also proposes an approximate algorithm for tractably estimating the reward function based on a particle-filter approach for representing uncertainty about the true reward function. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Treating the reward design process as an explicit problem that the robot agent can help guide in is very interesting, and addresses the difficult problem of creating reward signals for autonomous robots in complex environments. This is a very interesting paper and idea to tackle.\n- The assisted reward design problem is well-formulated and easy to follow, and the observation model for incorporating the human engineers reward function into what the robot thinks the actual reward function is principled and based on previous work in Inverse Reward Design. \n- The approximate strategy for proposing environments that provide the most information gain over the belief distribution over reward functions is interesting and is a good solution to the formulated problem.\n\nWeakness:\n- In the related work, the authors claim: \"Note that in many real-world problems, such as autonomous cars, there are dangerous situations where no ground-truth rewards exist\". The authors should clarify this statement, since it's not clear if this implies that the authors believe RL is not applicable to these kind of tasks.\n- The references should be addressed and properly presented (certain works are missing years [22],[25] and many are missing where they were published)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper presents an interesting and useful formulation of reward specification as an iterative process and proposes a tractable method for aiding users through the reward design process. This paper is easy to read and presents an interesting approach, and I recommend it be accepted. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "The authors should fix the references section so that it is consistent and accurate with what publications were cited. ", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {}}, {"review_id": "aZfWEX3B8Mu", "reviewer": "Reviewer_J4qS", "summary": "This paper studies the problem of reward design for tasks where there is no inherent notion of the correct reward. The expert/user is expected to provide the reward function as weighted combination of pre-defined features such that it would result in the learner following desired trajectory. The learner maintains an epistemic uncertainty about the true reward function and models it as a belief over candidate weights. The learner then proposes a new environment to elicit a new input from the user based on the principle of maximum information gain from a predefined distribution (or collection) of domains, and this process continues iteratively. The primary claims of the authors are as follows:\n\n1. An active learning approach leads to better identification of `misbehaving' environments than just considering the most difficult environments from the development distribution, or by selecting domains at random. This is borne out very well from simulation experiments and user studies. \n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths :**\n\n- Reward design is an important yet under-addressed issue for RL: Reward functions for many real world tasks are seldom set in stone, and are notoriously hard to formalize. There is a strong need to consider approaches that reason about the epistemic uncertainty of the true task specifications (in this case Markov rewards, but the issue extends to many task specification formalisms). This work tackles this important problem, and has promising results albeit in a toy domain.\n\n- Active Learning Formulation through information gain heuristic: The paper provides strong evidence that information gain is a useful heuristic for active learning within Markov reward domains. It complements prior research evidence that indicate towards information gain being a useful heuristic. This paper instantiates an info-gain based active learning approach to direct reward elicitation that to my knowledge is novel.\n\n- Intuitive results: The active learning approach identifies domains that actively try to 'break' the current reward function. This is what an active learner would be expected to perform.\n\n**Methodological Comments:**\n\n- It seems that the burden of designer shifts from specifying the right reward distribution to capturing the right distribution over domains. It appears that in this work, the authors assume access to a dataset of domains, and sample from that dataset. But for domains where data over environments is sparse, this approach would require a generative model over the domains. It would be nice to see an example where the problem distribution is described as a generative model.\n\n- The authors have used particle filtering based techniques, and seem to assume a unimodal distribution over users reward function feedback model. I wonder if that would be true in all cases. If the `true' reward function is non-unique, would the particle filtering approach still recover all the modes? Would the potentially conficting feedback throw the learner off?\n\n- This method relies on active learning queries that necessarily result in failure during deployment. Such a technique would me most amenable to domains where simulators are readily available. However active learning can also be valuable in domains where lack of simulators mean new policies need to be learnt on the fly during deployment. Can the info-gain heuristic be modified to generate executions that result in low performance  but not outright failure? (Note this is not a weakness, just a question to the authors)\n\n- There are many heuristics for active learning (uncertainty sampling, max model change, variance reduction, query-by-committee etc.) [1], have the authors considered other heuristics, but selected information gain? If not it would be interesting future work to explore the alignment of these heuristics within the Markov reward domain considered by the authors. \n\n[1] - Settles, Burr. \"Active learning literature survey.\" (2009).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I would recommend the acceptance of this paper. This is an under-explored problem, and to my knowledge a novel approach to the problem of reward design.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "- I would request the authors to better distinguish their work from prior work in active reward learning. I understand that the work is novel, but given there is significant research out there on active learning for rewards and more generally for task specifications, the novelty of the work should be made more explicit.\n\n- I would also urge the authors to include characterization of the problem domain distribution, and efficient inference over it as yet another open challenge "}}, {"review_id": "Rd5pXI59O8h", "reviewer": "Reviewer_pnb1", "summary": "The paper presents assisted reward design method where the reward is a weighted function of different components and the goal is to find the best reward function that is optimal for all different possible scenarios. The context is similar to active learning, where the user provides parameters for the reward function, while the algorithm tries to find set of scenarios that is provided to the user. This is an iterative process and the algorithm tries to make the user find the reward function that is optimal for all possible scenarios (even the ones not provided to the user by the algorithm). The authors show their results on an autonomous driving simulation where the car is trying to switch to the left lane with existence of different cars and obstacles. The results show that experimenting with 12 experts, the algorithm finds edge cases for different ", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written and it does a good job at covering basics and relevant publications. The main strength of the paper is that they take a different approach to the reward design. \nAt times, I found the paper to be hard to understand due to the terms used. Maybe it is because of my background with reinforcement learning, but the paper usesthe terms \"reward design\", \"state\", \"action\" for:\n - cost function that is used to calculate the behavior\n - set of environments\n - adding environment to the set\nwhich makes it very confusing as these terms are commonly used in reinforcement learning, but here there is no learning of a policy in the defined setting. Also, the authors use the term \"robot\" to define an agent that selects the set of environments,which is very uncommon (also not sure why the authors preferred the term robot for it). In my opinion the paper would be significantly more clear if the users drop the terms \"robot\", \"state\", \"reward design\". If I understand the concepts correctly, it presents an algorithm that selects set of scenarios such that if a person would optimize weights of a cost function based on these scenarios, behaviors based on that cost function will perform good in the set of environments defined by parameters.\n\nThe selected environment (autonomous driving) contains a single scenario of \"autonomous driving\" by looking at moving to the left lane. It provides unlimited amount of scenarios due to environment parameters such as obstacle or other cars. The other cars are driven with fixed speed, and they don't react to the car that changes lane using this cost function. Considering this, I think that presenting the domain as \"autonomous driving\" is an overstatement. \n\nThe experimentation is done with \"11 robotics expert\" to design the parameters of the reward function. I was wondering if being a robotics expert is needed, or any regular person who drives a car maybe could use this framework to optimize the cost function. This is mainly because the goal of the algorithm presented in this paper is to assist the user in designing the weights.\n\nAlso, I was wondering if any blackbox algorithm or optimization could be used to optimize these weights instead of using a robotics expert. This would allow, thousands or millions of iterations between the algorithm and the designer, and would be fully automated.\n\nThe reward design problem is actually very interesting within the reinforcement learning context. If the authors change the problem so that the agent (or robot) is the car, and it behaves based on some observations (i.e. lidar or features), we would use reinforcement learning to train the car. In this case, the problem of reward design would become \"reinforcement learning while also finding the optimal reward function\" and be comparable to other works in the literature such as intelligent environment design or learning to learn. This would also make the problem more applicable to other real world domains.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper touches an interesting problem of designing the reward function with human in the loop. On the other hand, the framing of the terms are very confusing (i.e. robot, state, action). The authors define environment selection algorithm as the robot, but I do not think that this definition would be applicable to problems in robotics. The tested domain is a very simple scenario (switching to lane), despite its potential difficulty due to unlimited number of scenarios with different parameters, I don't think it is representative of the problem of autonomous driving. \nThe paper has an interesting direction and good potential in the context of active learning, but I don't think that it fits the robotics framework that the authors try to use. I would also expect it to be evaluated on more complex problems where learning might be needed using the optimized / designed reward function.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "1: Not very relevant to the robot learning community.", "issues": "I think the main issues are the terminology, the fit to the robotics domains and the evaluation of the method. Please see the comments above  for details.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "LseN4GgQjFh", "reviewer": "Reviewer_oemY", "summary": "This paper formulates reward design as an iterative procedure between a learning agent and a designer. It assumes the existence of a ground-truth reward function for a distribution of environments. The agent treats the reward definition from the designer as an evidence and tries to form a belief over the ground-truth reward. Solution: at any given iteration, the agent proposes an environment to incentivize the designer to specify a reward which provides maximal information about the true reward. For example, in autonomous driving, the agent could present a scenario where the agent trajectory brings it dangerously close to other cars. The designer would have to carefully craft a reward function, that helps the agent quickly improve its belief over the ground-truth reward. Experiments with simulated and human designers on a toy environment show that assisted reward design gets closer to ground-truth reward when compared against baselines which propose the most difficult environments or random environments.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n1. **Novelty**: The assisted reward design setup is novel, to the best of my knowledge. It is a meaningful problem because reward design is indeed an iterative procedure in practice. Thus, it makes sense to assist a designer by actively querying rewards for information maximizing scenarios.\n2. **Human Evaluation**: The experiments on simulated and human designers are positive and consistent. This validates the problem setup and method well.\n3. **Qualitative Results**: I appreciate the analyses and examples explored by the paper to demonstrate convincingly how maximal information gain is helping the reward design to be efficient.\n\n# Weaknesses\n1. **Major assumptions**\n\nThe paper makes major assumptions in the problem setup, which are usually unrealistic.\n\n- *Environment*: A known distribution of environments is assumed which can be sampled from repeatedly. Thus, the setup does not work for a single environment/MDP, which is most often the case in reinforcement learning.\nEven if we wish to somehow convert a single environment to a distribution,\nthis is only possible in simulation, and only when when the environment configurations can be parameterized.  \nThe paper mentions large datasets to be valid as well, but (a) it is not feasible to relabel the entire dataset every time the designer changes the reward function, and (b) it is not possible to find the optimal trajectory from a static dataset without active environment interactions, when a new reward function is given.\n- *Ground-Truth Reward*: The paper assumes that there is a ground-truth reward function governing the true behavior on the entire distribution of environments. This usually does not hold. For instance, if I am training an agent to walk on diverse terrains, there is no concept of a ground-truth reward. A reward specifying agent stability and distance covered could be too unspecified that it doesn't help training, and a reward too detailed would become specific to each terrain so it is no longer common for all environments. This brings me to the next assumption...\n- *Parameterized Reward*: The reward function should be defined on pre-defined features and the designer's job is limited to changing the weights. This is limiting because most iterative reward engineering (in my experience) involves figuring out some edge cases or special scenarios. The fix usually requires extra dimensions to be added to the reward or defining conditional statements. \n- *World Model & Planning algorithms*: The paper acknowledges that access to perfect world models and efficient planning algorithms can be challenging. I just want to reiterate that training RL agents on every particle sample representing a reward on a large variety of environments for every iteration will blow up in complexity.\n\n\n2. **Writing**: The paper is well-written as a whole, but there are parts in the method section which become hard to follow. I understand it is difficult to compress all the information in 8 pages, but still it would assist the reader if more details about the method are present in the paper. For example:\n- intuition behind Eq. (1).\n- how eq. (4) and (5) are used to update beliefs.\n- moving the algorithm from appendix to the main paper.\n- where and how to use world models and RL policy in the method.\n- in figure 1, the captions or figure should describe what w, w~ and M represent.\n\n\n3. **Evaluation**: The paper only provides results on a proof-of-concept toy environment. It would be nice to see how assisted reward design scales to more complex control problems to really demonstrate how this can be a valuable procedure to the community in the near-future. I do not consider this a major weakness, since the concept seems novel enough to me. But, could the authors at least suggest some other environments where this proposed setup is already feasible to use, so we can imagine a bridge to realistic problems?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Despite the assumptions and toy-evaluation, I believe the concept presented in this paper is novel and potentially valuable in the future. The paper includes human designer evaluation which aptly shows the correctness and benefits of this method. After a few improvements to writing, I recommend this paper to be accepted on these grounds.\n\n===== Response After Rebuttal (No Change) =====\nThank you for your response. I am satisfied with it and keep my recommendation of weak accept. Please accommodate any leftover writing and clarity changes as pointed by other reviewers.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "1. I hope the authors can justify the assumptions of the problem setup and respond to their limitations.\n2. The authors can add more details to the writing to make it more approachable for the reader.\n3. Some explanation on how to extend this setup to realistic scenarios is required, especially what are the next immediate environments to consider?\n\nDetails for each of these issues are listed under weaknesses above."}}]

Author rebuttal:

Meta-review: This paper proposes an actively learning approach for iterative reward design in reinforcement learning. All reviewers acknowledge that the problem is interesting and under-explored. This paper has presented novel solutions to this important problem, makes good technical contributions, and has promising results albeit in a toy domain. After the revision, reviewers' original concerns about use-of-terminologies, and strong assumptions were addressed. Please accommodate remaining writing and clarity changes, and explicit statements contrasting with prior work, as pointed by reviewers, in the camera-ready version of the paper.


Decision: Accept (Poster)