Information Directed Reward Learning for Reinforcement Learning
Authors: David Lindner
Matteo Turchetta
Sebastian Tschiatschek
Kamil Ciosek
Andreas Krause
Venue: NeurIPS 2021 Poster (2021)
Keywords: reward learning
reinforcement learning
active learning
preference learning
human feedback

Abstract: For many reinforcement learning (RL) applications, specifying a reward is difficult. In this paper, we consider an RL setting where the agent can obtain information about the reward only by querying an expert that can, for example, evaluate individual states or provide binary preferences over trajectories. From such expensive feedback, we aim to learn a model of the reward function that allows standard RL algorithms to achieve high expected return with as few expert queries as possible. For this purpose, we propose Information Directed Reward Learning (IDRL), which uses a Bayesian model of the reward function and selects queries that maximize the information gain about the difference in return between potentially optimal policies. In contrast to prior active reward learning methods designed for specific types of queries, IDRL naturally accommodates different query types. Moreover, by shifting the focus from reducing the reward approximation error to improving the policy induced by the reward model, it achieves similar or better performance with significantly fewer queries. We support our findings with extensive evaluations  in multiple environments and with different types of queries.

Information Directed Reward Learning for Reinforcement Learning
David Lindner
Matteo Turchetta
Department of Computer Science ETH Zurich
Department of Computer Science
ETH Zurich
david. wiw.eer@inf.etieee...ee.e
matteo turchetta@inf wwweettttaiffeeh....... ch
Sebastian Tschiatschek
Kamil Ciosek
Department of Computer Science
Spotify
University of Vienna
kamilcdspotifyy.cm com
sebastian. www.aattthhhhsthhhhhcuitee.eee ac.at
Andreas Krause
Department of Computer Science
ETH Zurich krauseaGethz.co
Abstract
For many reinforcement learning (RL) applications, specifying reward is difficult. This paper considers an RL. setting where the agent obtains information about the reward only by querying expert that can, for example, evaluate individual states or provide binary preferences over trajectories. From such expensive feedback, we aim to learn model of the reward that allows standard RI. algorithms achieve high expected returns with as few expert queries as possible. To this end, we propose Information Directed Reward Learning (IDRL), which uses a Bayesian model of the reward and selects queries that maximize the information gain about the difference in return between plausibly optimal policies. contrast prior active reward learning methods designed for specific types of queries, IDRL naturally accommodates different query types. Moreover, it achieves similar or better performance with significantly fewer queries by shifting the focus from reducing the reward approximation error to improving the policy induced by the reward model. We support our findings with extensive evaluations in multiple environments and with different query types.
Introduction
Reinforcement learning (RL; Sutton and Barto, 2018) casts the problem of learning to perform complex tasks by interacting with environment agent aims to maximize its expected cumulative reward. Despite the remarkable successes of RL (e.g., Mnih et al., 2015; Silver et al., 2016), specifying reward functions that capture complex tasks is still an open problem. A promising approach is to learn a reward function from human feedback (e.g., Christiano al., 2017). However, since human feedback is expensive, active reward learning aims to minimize the number of queries. Prior work often focuses on approximating the reward function uniformly well. However, this may be aligned with the original goal of RL: finding an optimal policy, as Figure shows. Moreover, prior work is often tailored to
optimization problem where the learning
Work done while at Microsoft Research Cambridge.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

T
n22
((T)) i ((6)
((T2) - ((() +
((TI) -C((T)
F(
T(((m) G(T2), (g:0))
argmax gEt 0.0,,) IDRL objective
T = 4
Figure The robot wants collect food for human. can only move timesteps gridworld. cannot pass through the black walls, and collecting more food is always better. The robot does not know the human's preferences. but it can ask for food ratings, Common active learning methods aim to learn the reward uniformly well, and would query all items similarly often. In contrast, IDRL considers only the two plausibly optimal policies Tı and æ2. Since both policies collect cherry, and do not collect the pear, the robot only needs to learn about the apple and the com. IDRL can solve the task with 2 queries instead of 4.
specific types of queries, such as comparisons of trajectories (e.g., Sadigh al., 2017) or numerical evaluations of trajectories (e.g., Daniel et al., 2015), limiting its applicability.
Contributions. We propose Information Directed Reward Learning (IDRL), a general active reward learning approach for learning model of the reward function from expensive feedback with the goal of finding a good policy rather than uniformly reducing the model's error. IDRL can use arbitrary Bayesian reward models and arbitrary types of queries (Section 4), making it more general than existing methods. We describe an exact and efficient implementation of IDRL using Gaussian process (GP) reward models (Section 5) and different types of queries, and an approximation of IDRL that uses a deep neural network reward model and a state-of-the-att policy gradient algorithm to learn from comparison queries (Section 6). We evaluate IDRL extensively in simulated environments (Section 7), including a driving task and high-dimensional continuous control tasks in the MuJoCo simulator, and show that both implementations significantly outperform prior methods.
Related work
Reward Learning for RL. Several works aim to directly learn policies rather than reward functions from expert feedback in the form of numerical evaluations (Knox and Stone, 2009; MacGlashan al., 2017) or comparisons (Regan and Boutilier, 2009; Fürnkranz et al., 2012), and some work also explores active query selection (Akrour et al., 2012; Wilson et al., 2012). However, learning difficult to combine different types of feedback, and policies tend to generalize poorly between environments. Ng and Russell (2000) argue that reward functions are a more robust representation of desired behavior than policies. Inverse reinforcement learning (IRL) aims to learn reward model from expert demonstrations (Abbeel and Ng. 2004). Reward models can also be learned from comparisons two or more different behaviors (Wirth et al., 2017), other kinds of feedback (Jeon et al., 2020). While reward models can successfully learn sard-to-specify control and game-playing tasks (Christiano et al., 2017; Ibarz et al., 2018), most work uses simple heuristics to select queries to make. In contrast, active reward learning aims to select the most informative queries in a principled way.
policies directly from feedback has several downsides:
Active reward learning. For linear reward functions, Sadigh al. (2017) ask the expert compare trajectories synthesized to maximize the volume removed from a hypothesis space. Bıyık et al. (2020b) argue that maximizing information gain leads to better sample efficiency and queries that are easier to answer than volume removal. Bıyık et al. (2020a) generalize maximizing information gain to non-linear reward functions using a GP model. We also use an information gain objective to select queries; however, our approach focuses on finding an optimal policy instead of uniformly reducing the error of the reward model. With similar motivation, Wilde et al. (2020) aim to capture how informative query is for distinguishing policies. However, their method is limited to comparisons between potentially optimal policies. Daniel et al. (2015) also introduce an acquisition function to measure how informative query is for learning good policy. However, their setting restricted to observing the cumulative reward of a trajectory, and their acquisition function computationally expensive. Table gives overview of how method compares to this prior work.

---

Non-linear
Single-state Queries
Trajectory Queries
Numerical Queries
Comparison Queries
Considers Env. Dynamies
Rewards
Sadigh et al. (2017)
X
x
X
Biyik al. (2020b)
X
x
x
XXX X
Brynk al. (2020a)
X
X
Daniel et al. (2015)
V
x1
/
X1
Wilde et al. (2020)
X
X
X
IDRL (ours)
The original authors do not consider this setting, hut we provide an extension to their method in Section and Appendix D.
Table 1: In contrast to most prior work on active reward learning for RI., IDRL can handle non-linear reward functions and different query types, in particular numerical evaluations and comparisons of individual states and (partial) trajectories. Further, IDRL takes the environment dynamics into account to achieve better sample efficiency (ef. Figure 1).
Bayesian optimization. Bayesian optimization (BO) aims to maximize an aopensive-to-evaluuee function by learning a Bayesian model of the function and selecting informative queries (Mockus et al., 1978). We face a related but significantly harder problem: we aim find an optimal policy in RL but only indirectly obtain information about the value of policies. Our problem has striking connections to variants of the partial monitoring problems (Rustichini, 1999) and transductive linear bandits (Fiez et al., 2019). We explore this connection detail in Appendix C.
bandit problem (Bubeck and Cesa-BBanchi 2012), in particular
3 Background and problem setting
Markov decision process. Markov decision processes (MDPs; Puterman, 2014) model sequential
decision-making problems in dynamical systems. An MDP (S, A, P,r,po:3 consists of a state space S, an action space A, transition function P, reward function r, an initial state distribution po. and a discount factor 7 E [o, 1). an MDP, the agent starts in state 8p ~ po(s) and, when taking action de. transitions from state 8t to state st+1 with probability P(s++1|s;,a, az). The agent affects the environment through actions determined by a policy π (aclss)) indicating the probability of taking action a in state St. The agent's goal is to find policy that maximizes the expected discounted return G(TT) = E 1Eioo where rg is the reward obtained at time t.
Information gain. Intuitively, the information gain between two random variables measures the amount of information that can be obtained about one of them by observing the other. Formally, for two random variables X and Y with marginal distributions px, Py and joint distribution P(X.Y) the information gain (or mutual information) [XX, Y) DxL (p(x,)) Py), where (x is the KL-divergence. Given third random variable Z, conditional information gain is defined as T(X; Y = =) DRI. 22
pY|z=z).
Problem setting. We focus on MDPs where the reward function is not readily available. Instead, the agent can query an expert for information about the reward. In iteration i, the agent makes query & to the expert, and receives a response yi- For example, q: could ask the expert to compare two trajectories or judge a single trajectory, and could indicate which of the two trajectories is better or provide the return of single trajectory. We assume that the agent can interact with the environment cheaply, but queries to the expert are expensive, and hence the agent has to find a policy æ that maximizes the expected return G(r)) using as few queries as possible.
Our reward learning approach. We approach this problem by learning a model of the reward function, i.c., model that predicts the reward of a given state, 2 and computing a policy that maximizes the return induced by the model. Importantly, we want to learn a reward model such that the induced optimal policy achieves a high return under the true reward function. Note that any RL algorithm can be used to find the policy. Hence, the problem reduces to selecting a model for the reward function and deciding which queries to make. Our key insight is that queries that help most to find a good policy might differ from those that uniformly reduce the model's uncertainty.
Our approach is also applicable to reward functions that depend on state aate-action pairs or transitions. We focus on state-dependent reward functions for simplicity of exposition.
3

---

Algorithm 1 Information Directed Reward Learning (IDRL). The algorithm requires set of candidate queries Qe Bayesian model of the reward function, and an RL algorithm that returns policy given reward function. ((TT) the belief about the expected return of policy x, induced by the reward model P)) and f is the belief about the reward function.
1: D < f); II initialize candidate policies; initialize reward model with prior distribution P(f)
2: while not converged do
3: Select query:
4:
1T. 22 argmax, r'Ell (()(T)) DD
5:
9º argmax 9 I(((T) - (TT)): (q. ()
6:
Make query and update reward model:
7:
y" < Response to query q"
8:
P (q"""")) x D.
D. Update belief about the reward >- Add observation to dataset
9: D + DU [('","")
10: Optionally update candidate policies II
11: end while
12: f <- mean estimate of the reward model; #* < (F(
13: return R*
4 The Information Directed Reward Learning acquisition function
This section introduces Information Directed Reward Learning (IDRL) for a general Bayesian model of the reward and discusses how to select queries 9r making no assumptions on their form the responses yi-
Reward model. To select informative queries, we need to quantify uncertainty; hence, we use a Bayesian model of the reward function. From Bayesian perspective, it is important to distinguish between the agent's belief about quantity and its "actual" value that unknown us. We denote the belief about the reward in state s with f(s) and its actual value with r(s).
Query selection. To select informative queries, we have to consider that the responses might only give indirect information about the set of optimal policies. For example, assume the agent can ask the expert to quantify the reward of individual states. These rewards provide information about the expected return of policy but may yield no information about the set of optimal policies. For example, a state is visited similarly often by every plausibly optimal policy, knowing its reward does not help decide between the policies (e.g., the cherry in Figure 1). Therefore, any approach that only aims to reduce the uncertainty of the reward model may waste expensive queries that do not help find an optimal policy.
Intuitively, we want to instead select queries that help identify the optimal policy. the language of information theory, we want to maximize the information gain of a query about the identity of the optimal policy. More formally, if D = {(q1. y1...... (qt, yt)} is a dataset of past queries and responses, let us denote with P(*" (D) the agent's belief about the optimal policy, induced by our belief about the reward function f. Also, let Q. be a set candidate queries the agent can make. Then, one way to formalize this intuition is to select queries g-' E argmax,e [ (q. D)) where denotes the information
ÿ is the agent's belief about the response it will get to query q. gain. Unfortunately, this objective has two undesirable properties. First, the agent has to keep track of a distribution over all possible policies to compute it, which is intractable in general. Second, reducing uncertainty about the optimal policy only matters long there are significant differences in the return of plausibly optimal policies. For example, if the agent identifies a set of plausibly optimal policies with similar returns, care less about identifying exactly which policy is optimal, compared to when such policies have very different returns.
To address the first challenge,
obtain finite set of candidate policies II, that are plausibly optimal according to our Bayesian reward model. To address the second challenge, we select the most informative query for distinguishing policies in terms of their value.
Let us first discuss how select queries, assuming a set of plausibly optimal policies II, to be available. We can exploit the fact that the belief about the reward function & induces a belief about the expected return of policy E 1l. denoted as ((T). Concretely, the expected return

---

of a policy can be computed as the scalar product G(T) (f r), where f is vector of the (discounted) expected state-visitation frequencies of policy and r is a vector of rewards of the corresponding states. We can estimate f from trajectories sampled using policy n, and then determine ((T) from
Given (TT), IDRL proceeds in two steps. It first selects two policies that maximize the model's uncertainty about the difference their expected returns:
TI: 72 E argmax (rg -(())))
x,
where H is the entropy of the belicf conditioned on past queries. To gather information about the distinction between Tı and π2, IDRL then selects queries that maximize the information gain about the difference in expected return between иT and n2:
q" argmax (rgmax/(((())) -( (( (q.0)lD). qEQ:
Note that this is not the same as jointly maximizing the information gain about ((mI) and (ma)). Equation (2) prefers queries that help distinguish πj and Tī over queries that help to determine the exact value of G(T1) and (x2)) Assuming set of optimal policies is contained in TT., reducing the uncertainty about the difference in returns within II will help to identify an optimal policy quickly. In particular, if there is no remaining uncertainty about the differences in return, we clearly identify an optimal policy.
Let us now discuss how to obtain a set of candidate policies II.,. For IDRL to select informative queries, II, has to reflect the agent's current belief about optimal policies. IDRL uses Thompson sampling (TS, Thompson, 1933) a flexible way to create IIe We implement TS by repeatedly sampling reward function from the posterior reward model and finding an approximately optimal policy for this sampled reward function. This approximates sampling from the posterior distribution over optimal policies. Since TS demanding, in our experiments, we investigate two effective alternatives to alleviate its computational burden: (1) update II, in regular intervals, rather than at every step, and (2) we start from the candidates computed in previous steps rather than starting the policy optimization from scratch.
Algorithm shows the full IDRL algorithm. In each iteration, IDRL identifies two plausibly optimal policies with high uncertainty about their difference in return and then aims to reduce this uncertainty. We can stop the algorithm after a fixed number of queries, or by checking a convergence criterion, and return policy म* that is optimized for the current reward model.
Note that IDRL is agnostic to how the candidate queries 2, are generated. Different applications might require different approaches to generating Q.. In our experiments, for example, we consider: using all possible queries in small environments, choosing states or trajectories to query from rollouts of the currently optimal policy ø*, selecting queries from rollouts of the candidate policies, and selecting queries from trajectories a pre-defined explorations policy. Importantly, all of these, and others, are compatible with IDRL.
5 An exact and efficient implementation of IDRL for GP reward models
Here, we describe one concrete implementation of IDRL using a Gaussian process (GP, Rasmussen and Williams, 2006) reward model and linear query types. These choices allow us to compute equations (1) and (2) exactly and efficiently.
Reward model. We model the reward function GP with (w.l.ogg.) zero-mean prior distribution f(s) ~ (P s')) using kernel & which measures the similarity of states.
Query selection. We first show how compute equations (1) and (2) if the posterior belief about the reward function is Gaussian. Then, we discuss a family of practically relevant query types that satisfy this assumption. We provide proofs for all results in Appendix A.
5

---

Proposition 5.1. If f(s) D is a GP. then P()(#) - G(T') D) is Gaussian and:
argmax ((T(T) - G(=))D) =
argmax Var(G - -
x.T'ell:
argmax ((G - ((T)): (q.0))) argmin Var(G(m) D( {(200))
q.
qEQ.
We can compute both variances analytically, enabling exact implementation of equations (1) and (2).
Query types. To apply this result, we need f(s) be GP, which not the case for general observations (g. Yt). . If the queries are individual states, i.e., 9; = 8; and y; = =s(s;,, the problem is standard GP regression, and [D)
GP (Rasmussen and Williams, 2006). More generally, similar statement holds if the observations are linear combinations of rewards.
Definition 5.1. We call 02- ===6 (=, C) linear reward query, if it consists of states S = {sI...... 8N) and linear weights C = {(1,-... ,NN, and the response to query q linear combination of rewards - ンドー (r((s)) + e, with Gaussian noise 8 ~> (0,
Proposition 5.2. Let q be linear reward query. If the prior belief about the reward f(s) is GP, then the posterior belief about the reward f(s)((g. y) is also a GP.
Linear reward queries result in a particularly efficient implementation of IDRL. course, IDRL with a GP model could be extended non-linear observations using approximate inference. However, it turns out that many commonly used query types can be modeled as linear reward queries, including the return of trajectories or comparisons of trajectories (see Appendix B).
scalable Deep RL approximation of IDRL
GP models provide a convenient way to implement IDRL exactly. But, can IDRL also used if we can not model the reward function as a GP? Moreover, can we scale it to large environments on the scale of typical Deep RL applications?
To address these questions, we propose a second implementation of IDRL using a deep neural network (DNN) reward model. To scale IDRL to large Deep RI scenarios, we integrate into policy optimization algorithm, similar to Christiano et al. (2017).² In our experiments, we focus on comparison queries, but it is straightforward extend the algorithm to other query types.
Reward model. To model the reward function, we use adaptive basis function regression with DNNs, similar Snoek al. (2015). Concretely, we train DNN from comparisons of short clips of the agents behavior using the Bradley-Tery model and lz-regularizatio. We then treat the learned representation as basis function and the final layer of the DNN as a maximum a posterioni (MAP) estimate of the parameters of a Bayesian logistic regression model. Finally, we approximate the full posterior using Laplace approximation.
Query selection. Because of the Laplace approximation, the posterior distribution of f(s) is Gaussian, and we can compute equations (1) and (2) the same way we did for a GP reward model.
Query types. Similar to Christiano al. (2017), consider queries q: = (o) that compare two segments of trajectories of and o?. where the user responds with their preference y E {-1, 1}. Candidate policies. In large environments, it is infeasible to train new policies from scratch during the Thompson sampling step. To avoid this, we maintain a fixed set of policies that we update regularly, instead of training new policies from scratch whenever we receive new samples.
Candidate queries. We generate candidate queries by rolling out the current policy optimized for the mean estimate of the reward model, as well the candidate policies and uniformly sampling pairs of segments from the resulting trajectories.
Full algorithm. We use a policy gradient algorithm train a policy for the current reward model, and the candidate policies. Similar to Christiano et al. (2017), the agent queries comparisons following fixed schedule in which the number of samples is proportional to 4. where is the number of policy training steps, i.c., we provide more samples early during training and less later on. For more details, including full pseudocode for the Deep RL algorithm, see Appendix E.3.
We provide detailed comparison between our setup and Christiano al. (2017) in Appendix E.3.

---

7 Experiments
We empirically test IDRL in several environments, ranging from gridworlds to complex continuous control tasks, and for several different query types, including numerical evaluations and comparisons of trajectories. Our evaluation covers most scenarios existing in the literature and shows that IDRL attains comparable superior performance methods designed for specific scenarios.
In experiments, the agent's s queries are answered with simulated feedback based on an underlying true reward function unknown to the agent. We usually evaluate the regret of policy trained
using the reward model, i... G(*²)) - G(T)) for an optimal policy x². If we do not know x², approximate it with policy trained on the true reward function.
We first validate that GP-based IDRL improves sample efficiency in simple gridworld environments for numerical and comparison queries (Section 7.2). Next, we consider the most common setup in the literature, that is, learning from comparisons of trajectories, and compare GP-based IDRL against alternative approaches in a driving simulator, proposed in prior work (Section 7.3). Then, we study another natural feedback type: ratings of clips of the agent's behavior. In this setting, we demonstrate how GP-based IDRL can be scaled up to bigger environments in the MuloCo simulator (Section 7.4). Finally, we further demonstrate scalability by considering the Deep RI. implementation of IDRL to learn standard MuJoCo tasks from comparisons of clips of trajectories, similar to Christiano et al. (2017) (Section 7.5).
For each environment, we choose our setup to be close to prior work to promote a fair comparison. This leads some design choices, such as the RL solver or the query types, to differ between environments. As a side effect, this highlights IDRL's generality. Appendices D and E describe the experimental setup more detail, and we provide code to reproduce all experiments."
7.1 Baselines
We consider five baselines: (i) Uniform sumpling selects queries from Oe with equal probability. (ii) Information gain on the reward (IGR) selects queries that maximize information gain about the reward [g, ;); AD). For GP model, this equivalent to maximizing Varlj]D, g). Biyrk et al. (2020b) use IGR to learn rewards from comparisons of trajectories; however, it can be extended to other query types. (iii) Expected improvement on the reward (EIR) maximizes the improvement in the value of query compared to the best observation so far, in expectation, and is a common acquisition function in BO (Mockus et al., 1978). EIR can not be applied to comparison queries. (iv) Expected policy divergence (EPD) is an active reward learning method introduced by Daniel et al. (2015), which makes queries that maximally change the current policy. Since EPD updates the policy for each potential observation, it is prohibitively expensive for large Q.. While EPD was introduced to query the return of trajectories, we extend it to other query types (cf. Appendix D). (v) Maximum regret (MR) is acquisition function proposed by Wilde al. (2020). It assumes access to set of candidate reward functions and corresponding optimal policies. MR compares policies that perform well according to one reward function but poorly according to a different one. It can only be used with comparisons of full trajectories. We also tested expected volume removal (EVR, Sadigh al., 2017) for comparison queries; however, we found to get stuck often, which confirms the findings of Bryık et al. (2020b). Note that IGR and EIR reduce uncertainty uniformly over the state space, while EPD and MR consider the environment dynamics.
7.2 Can IDRL improve sample efficiency by considering the environment dynamics?
We first validate our hypothesis that IDRL improves sample efficiency in small toy environments. Here, we highlight experiments in a set of Gridworlds similar to Figure 1. Appendix F presents two additional toy environments that isolate specific reasons why IDRL outperforms the baselines.
Setup. We consider 10 x 10 Gridworlds with randomly placed walls and objects with different rewards. The agent has to find the object with the largest reward. We consider queries about the reward of individual states, i.c. qs s; E S and y/ r(sz), and comparison queries with 4s = (811, 8:2) and y/i E (1,,1). The candidate queries Q. either consist of all states or all
'https://githani cow.david htam/ttavid-Linder///dda
7

---

pairs of states. We use GP-based IDRL, with a kernel that encodes which objects are the same, and which are different. All experiments run for less than I hour on a single CPU.
Results. Figure 2 shows the regret of policy trained on the reward model after different numbers of queries. IDRL finds better policies than the base- lines with a limited number of queries because focuses on regions of the state space relevant for finding the optimal policy. As shown in Figure I, this improves sample efficiency over methods that uni- formly reduce uncertainty, such as IGR and EIR. IDRL also outperforms EPD, because EPD's goal of selecting queries that maximally change the cur- rent policy is also misaligned with the goal of finding an optimal policy. We investigate EPD's specific failure modes in Appendix F.
Reward of States
Comparisons States
regret
20-
10-
0
5 10 15
60 120 150 240 300 number of queries
number of queries
Figure 2: Regret of a policy trained in randomly generated gridworlds ber queries. The queries are either asking about the reward of individual states, or comparisons be- tween two states. The plot compares IDRL (=)) to EPD (=x), IGR (==) and uniform sampling For numerical queries, we also compare to EIR + which is not applicable com- parisons. MR is not applicable to single-state queries. The plots show mean and standard er- over 30 random seeds.
function of the num-
7.3 Can IDRL learn from comparisons of trajectories using GP reward model?
Most prior work studies reward learning from compar- isons of trajectories. To evaluate IDRL in this setting. we consider the 2-dimensional, continuous Driver environment by Sadigh et al. (2017).
Setup. In Driver, the agent controls car on a high- way with another car driving on fixed trajectory (cf. Figure 3a). For each experiment, we randomly sample an underlying (linear) reward function to describe the desired driving behavior. We use code by Sadigh et al. (2017) to simulate and solve the environment, but we adapt it to our setting. In contrast to Sadigh et al. (2017), we do not synthesize queries. Instead, we sample fixed set of 200 reward functions from Gaussian prior distribution. We then optimize a policy for each of these reward functions, and, similarly to Wilde et al. (2020), consider all pairs of policies as potential queries. Moreover, we assume linear observation model (sce Appendix B), whereas Sadigh et al. (2017) and Wilde et al. (2020) choose different non-lincar observation models. Each experiment runs for less than 24 hours on single CPU.
Results. Figure 3a shows the regret curves for the learned policy and the cosine similarity for the learned reward function weights. IDRL outperforms the baselines and finds a better policy with fewer queries. However, the difference to pure information gain is small in this simple environment.
7.4 Can IDRL with GP model be scaled to bigger environments?
To demonstrate that GP-based IDRL scales to larger en- vironments, we use the MuJoCo simulator (Todorov al., 2012), which provides challenging environments com- monly used as benchmarks for RL. However, its stan- dard locomotion tasks are very easy to learn for a GP model because the reward directly proportional to the agent's ss velocity in x-direction. Instead, we propose task where the reward function is harder to learn.
Swimmer- Corridor
Ant- Corridor
Uniform Sampling
11.8 0.9
15
IGR
13.3
17.2 +7.2000 0.5
EIR
12.0
17.5 0.8
IDRL (20 updates)
2.4
2.2 2.2 0.8
IDRL updates)
2.7
5
IDRL (2 updates)
5.1 0.6
8+1
IDRL (1 update)
7.6 0.8
12:
Setup. In our Corridor environments (Figure 3b), robot (Swimmer, or Ant) has to move forward and stop at goal position. The simulated ex- pert rates trajectory clips according to a reward function that is proportional to the velocity in the direction of the goal. This reward function linear in a set of features of the state, described in Appendix D.3.5.
Results comparing IDRL for dif- ferent update frequencies of the candidate policies the Corridor environments. The table shows the estimated regret of a policy trained using 20 queries about the reward function.
Table 2:
8

---

Legend:
Uniform Sampling
Expected Inprovement Maximum Regret
Information Gain Reward (IGR)
Expected Policy Divergence (EPD)
IDRL (ours)
estimnted regnt
0000 Mea-Scallarits
Figure 3: Results in the (a) Driver and (b) Swimmer- Correior environments (shown on the left). We show the regret of policy trained on the reward model compared to policy trained on the truc reward func- tion a function of the number of queries (middle plot). We report the cosine sim- ilarity between the learned and the true reward function (right plot). The plots show mean and standard error 30 dom seeds. IDRL finds significantly better policies, while not necessarily learning an overall more accurate model of the reward function. A similar plot for Ant- Corridor can he found in Appendix F.
200
300
400
500
300 400 500 number quaries
100
200
mumber
(a) Driver (comparison of trajectories)
orimated rngret
cle- fee-iimberitt
0.75
10
12.301
30
number queties
mumber of queties
(b) Swimmer- Coimier-Corridor (evaluation of trajectory clips)
We use augmented random search (Mania et al., 2018) as RL algorithm. For the Swinmer-Corridor we learn linear policy, and for the Ant-Corridor learn hierarchical policy top of pre-trained poli- cies moving in four different directions. To generate candidate queries, we use a fixed, noisy exploration policy that moves along the whole corridor. Unfortunately, EPD is too expensive to evaluate in this environment and MR is not suited to this kind of queries.
Results. Figure 3b shows that IDRL needs significantly fewer queries to find good policy than any of the baselines. IDRL adapts its queries to the policies that the current reward model induces: initially samples clips in which the robot moves close to its starting position and shifts its focus to other regions as the reward model improves, and the learned policy starts to move. In contrast, the baselines make queries in the whole reachable space similarly often, and, therefore, waste queries in regions that are not directly relevant for improving the policy.
The computationally most expensive part of this implementation of IDRL is updating the candidate policies in each iteration. Updating them less often reduces the computational cost the expense of potentially reducing the sample efficiency. Table 2 studies this trade- off and shows that IDRL outperforms the baselines even when the policies are updated only once at the beginning of training. In this extreme case, we reduce IDRL's runtime and from about 40 hours
from about 40 hours to about 20 hours in Serimmer-Corridae about 10 hours Ant-Corridor. This shows the benefits are larger when solving the RL problem more expensive. Nonetheless, the baseline algorithms are still faster, and run for only 2 hours. This because they not require the additional inference steps necessary to optimize Equation (2). These results indicate that IDRL using full Thompson sampling to generate candidate policies can trade-off computational cost and sample efficiency, which allows to be applied to large environments.
7.5 Can IDRL be scaled to a Deep RL setting?
Finally, consider the Deep RL implementation of IDRL from Section 6, using the Soft Actor- algorithm (SAC; Haarnoja et al., 2018). We test it on standard MuJoCo locomotion tasks, which harder to learn with DNN than with GP model because the former encodes less prior information.
Setup. We consider suite of standard tasks in MuJoCo implemented in OpenAI Gym (Brockman et al., 2016): HalfCheetah-v3, Walker2d-v3, Hopper-v3, Ant-v3, Swimmer-v3, InvertedPendulumm InvertedDoubieeenduuuum
Reacher-v2. Similar to Christiano al. (2017), we modify some environments to remove the termination conditions. Our environments differ slightly from Christiano al., for the details see Appendix E.3. Our evaluation metric a normalized score, averaged over all environments. A score of 0 corresponds to random policy and score of 100 is the performance of a policy trained on the true reward function. We provide results for the individual environments in Appendix F. Since IDRL tracks the candidate policies, it generates the candidate queries rolling

---

out the currently optimal policy and the candidate policies. However, the baselines do not have access to the candidate policies, and therefore consider a smaller set of potential queries. For fair comparison, we perform an ablation where IDRL does not consider the candidate policies to generate candidate queries. Since IDRL maintains 3 (additional) candidate policies, it roughly 4 times slower (about 80 hours on single GPU) than the baselines (about 20 hours on a single GPU).
Results. Figure shows that IDRL on average learns good policies significantly faster than the base- lines. The individual results in each environment (in Appendix F) are more nuanced. IDRL clearly outperforms the baselines some environments (e.g., Hopper-v3)) performs comparable in other envi- ronments (e.g. Walker2d-v3), and performs worse than uniform sampling in few environments (e.g., HalfCheetah--33) Also while mostly using the can- didate policy rollouts improves the performance of IDRL, this is not always the case (e.g., in Swimmer-v3 the ablation performs better). This indicates that much of the variance might be caused by which queries are considered, which could be improved by using other exploration strategies than the candi- date policies generate candidate queries. Crucially, these experiments demonstrate that IDRL is scalable to nigh-dimensional, complex tasks, while still im- proving sample efficiency over existing methods for such tasks.
siverage scure
40
20
0
0.00
0.25
0.50
0.75
1.00
timestep
x10*
Figure Normalized score policies learned from 1400 (synthetic) comparisons of clips of the agent's behavior, averaged over all MuJoCo en- vironments (higher is better). We show the mean and standard error of the score averaged over random seeds per environment. The plot compares IDRI. (= to IGR (==) and uniform sampling as well not the candidate policies generate addi- tional candidate queries (=*=). EPD is too ex- pensive and MR is not suited to this kind of queries.
ablation of IDRL that does
8 Conclusion
We studied the problem of actively learning reward function models using as few expert queries as possible. We introduced Information Directed Reward Learning (IDRL), novel information information-theoreti -thorretic algorithm that focuses on learning a good policy rather than attaining low approximation eтror of the reward and that, differently from most prior methods, works with multiple types of feedback. We show it needs significantly fewer queries than prior methods and that it scales to complex environments.
Limitations and future work. The main practical limitation of IDRL is its computational cost. We demonstrated how to scale IDRL to complex environments, increasing the runtime by only a constant factor. While IDRL still more demanding than most existing algorithms, it preferable in situations where better sample efficiency is more important than low computational cost.
Our problem setup also has some conceptual limitations. We assume that interactions with the environment are cheap, which is not the case in many applications. Future work could aim to achieve low sample complexity in terms of environment interactions as well as reward queries. Moreover, we assume that the goal of RL is to learn a good policy in a single environment, which does not consider the problem of generalizing to other environment. In fact, being designed learn good policy in a single environment, IDRL might not be best for learning a reward model that generalizes well. To address this, future versions of IDRL could aim to learn a reward model that leads to good policies over a distribution of environments instead of a single environment.
Overall, consider IDRL an addition to the set of existing active reward learning algorithms rather than a replacement of existing methods.
Broader impact. IDRL improves the sample efficiency of learning reward models, which is step towards making RL a viable solution for real-world problems. RL systems can be used in various ways, and they could cause risks from malicious actors (Brundage et al., 2018). However, overall, learning reward models is likely to help in making RL more robust and safe (Leike et al., 2018).
Improving the sample efficiency of learning reward models, is crucial for making RL more useful. By addressing this problem, IDRL takes step towards making RL a more viable solution for real-world problems.
10

---

Acknowledgements
This research was supported through the Microsoft Swiss Joint Research Center. We thank Johannes Kirschner and Jonas Rothfuss for valuable feedback on earlier version of this paper, and Nils Wilde for valuable comments about the Maximum Regret approach to reward learning.
References
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of International Conference on Machine Learning (ICML). 2004.
Riad Akrour, Marc Schoenauer, and Michèle Sebag. April: Active preference soaming-baseed
reinforcement learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 2012.
Christopher M Bishop. Pattern recognition and machine learning. Springer, 2006.
Erdem Bıyık, Nicolas Huynh, Mykel J. Kochenderfer, and Dorsa Sadigh. Active sifference-basee Gaussian process regression for reward learning. In Proceedings of Robotics: Science and Systems (RSS), July 2020a.
Erdem Bıyık, Malayandi Palan, Nicholas C Landolfi. Dylan P Losey, and Dorsa Sadigh. Asking easy questions: user-friendly approach active reward learning. In Conference on Robot Learning (CoRL), 2020b.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. OpenAl Gym. arXiv arXiv:1606.01540 2016.
Miles Brundage, Shahar Avin, Jack Clark, Helen Toner, Peter Eckersley, Ben Garfinkel, Allan Dafoe, Paul Scharre, Thomas Zeitzoff, Bobby Filar, et al. The malicious use of artificial intelligence: Forecasting, prevention, and mitigation. arXiv: 100..008220728 2018
Sébastien Bubeck and Nicolo Cesa- Bianchi. Regret analysis of stochastic and nonstochastic multi- armed bandit problems. Foundations and Trends@ in Machine Learning, 5(1):1-122, 2012.
Sébastien Bubeck, Rémi Munos, and Gilles Stoltz. Pure exploration in multi-armed bandits problems. In International Conference on Algorithmic Learning Theory, pages 23-37. Springer, 2009.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, 2017.
Thomas M Cover and Joy A Thomas. Elements of information theory. John Wiley & Sons, 2006.
Christian Daniel, Oliver Kroemer, Malte Viering, Jan Metz, and Jan Peters. Active reward learning with novel acquisition function. Autonomous Robots, 3905):399445,, 2015.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In Proceedings of International Conference Machine Learning (ICML), 2016.
Tanner Fiez, Lalit Jain, Kevin G Jamieson, and Lillian Ratifff Sequential experimental design for transductive linear bandits. In Advances in Neural Information Processing Systems, 2019.
Johannes Fürnkranz, Eyke Hüllermeier, Weiwei Cheng, and Sang-Hyeun Park. Preference- based reinforcement learning: a formal framework and a policy iteration algorithm. Machine learning, 89 (1-2):123-155) 156, 2012.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maxi- mum entropy deep reinforcement learning with stochastic actor. Proceedings of International Conference Machine Learning (ICML), 2018.

---

Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations Atari In Advances in Neural Information Processing Systems, 2018.
Hong Jun Jeon, Smitha Milli, and Anca Dragan. Reward-rational (implicit) choice: A unifying formalism for reward learning. In Advances in Neural Information Processing Systems, 2020.
Johannes Kirschner, Tor Lattimore, and Andreas Krause. Information directed sampling for linear partial monitoring, In Conference Learning Theory, 2020.
W Bradley Knox and Peter Stone. Interactively shaping agents via human reinforcement: The TAMER framework. In International Conference on Knowledge Capture, 2009.
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: research direction. arXiv:/811.0788 2018.
James MacGlashan, Mark Ho, Robert Loftin, Bei Peng, Guan Wang. David I. Roberts, Matthew Taylor, and Michael I. Littman. Interactive learning from policy-dependent human feedback. In Proceedings of International Conference on Machine Learning (ICML), 2017.
Horia Mania, Aurelia Guy, and Benjamin Recht. Simple random search of static linear policies is competitive for reinforcement learning. In Advances in Neural Information Processing Systems, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518, 2015.
Jonas Moekus, Vytautas Tiesis, and Antanas Zilinskas. The application of Bayesian methods for seeking the extremum. Towards Global Optimization, 2, 1978.
Andrew Ng and Stuart Russell. Algorithms for inverse reinforcement learning. Proceedings of International Conference Machine Learning (ICML), 2000.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John
Wiley & Sons, 2014.
Antonin Raffin, Ashley Hill, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, and Noah Dormann. Stable baselines3. https://github. wtt.////AAR/////taalbbaaaaaallll 2019.
Carl Edward Rasmussen and Christopher K. Williams. Gaussian Processes for Machine Learning. The MIT Press, 2006.
Kevin Regan and Craig Boutilier. Regret-based reward elicitation for Markov decision processes. In Uncertainty in Artificial Intelligence, 2009.
Daniel Russo and Benjamin Van Roy. Learning to optimize via information-direected sampling. In Advances Neural Information Processing Systems, 2014.
Aldo Rustichini. Minimizing regret: The general case. Games and Economic Behavior, 29(1-2): 224-243 1999.
Dorsa Sadigh, Anca D Dragan, Shankar Sastry, and Sanjit A Seshia. Active preference-base based learning of reward functions. Proceedings of Roboties: Science and Systems (RSS), 2017.
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature. 529(7587):4844435 2016.
12

---

Jasper Snoek, Oren Rippel. Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Mostofa Patwary, Mr Prabhat, and Ryan Adams. Scalable Bayesian optimization using deep neural networks. In Proceedings of International Conference on Machine Learning (ICML), 2015.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
William R Thompson. On the likelihood that unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4) 294, 1933.
Emanuel Todorov, Tom Frez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012.
Larry Wasserman. All of statistics: concise course in statistical inference. Springer Science & Business Media, 2004.
Nils Wilde, Dana Kulic, and Stephen Smith. Active preference learning using maximum regret. In International Conference Intelligent Robots and Systems (IROS), 2020.
Aaron Wilson, Alan Fern, and Prasad Tadepalli. A Bayesian approach for policy learning from trajectory preference queries. Advances in Neural Information Processing Systems, 2012.
Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. A survey of preference- based reinforcement learning methods. The Journal of Machine Learning Research, 18(1):4945- 4990, 2017.
13


Reviews:
[{"review_id": "zKLxJ4aoCCR", "reviewer": "Reviewer_2bor", "summary": "The authors propose a Bayesian method for learning the reward function in an RL setting.  They study the setting in which the algorithm can query experts.  The form these queries can take is quite general.  They analyse their algorithm theoretically and empirically.  Their key insight is that queries should be focused on enabling the reward model to be used to find the best policy possible (as efficiently as possible), rather than decreasing the reward model’s uncertainty in general.\n", "questions": "", "limitations": "", "rating": 9, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "--BEGIN UPDATE--\n\nAfter reading the other reviews and the author responses, I maintain my score; this is an outstanding paper.  I think the improvements the authors have promised will make it even stronger.\n\n--END UPDATE--\n\n\nThe amount of details included about the experimental setup is outstanding (main paper and Sections D-F) and the additional experiments/theory in the appendix are very nice.  The experiments have outstanding breadth and depth and the results in Section 7.3 in particular were interesting and well-thought-out.\n\nSmall issues/edits:\n\n-\\mathcal D is first used about 5 lines before it is defined, it would be easier to read if the definition was moved before the first use.\n\n-There are some definitions that can only be inferred or are stated informally, and should be more formally defined.  For example, the belief about the optimal policy, y hat and the set within which it exists, G hat (see next issue below; I’m not sure which “G” “G hat” estimates, a very literal reading makes me think it estimates the first definition, but the context makes me think it estimates the second definition), etc.\n\n-The definition of G(pi) is overloaded; this is confusing and should not be done.  (This maybe contributes to the confusion in the point above.)  Edit: On my second read-through, it occurs to me that the second “definition” of G(pi) (on line 128) is actually not a definition, and that that notation denotes the dot product.  This would explain a lot of my confusion above. Two suggestions if this is the case: 1) use \\coloneqq for definitions (for example, the definition of G in Section 3) to clearly distinguish definitions from equalities, and 2) tell the reader on line 128 that that notation denotes the dot product (at first, I assumed it was a vector or a tuple or a contention of vectors, or something like that; in other words I thought that G was being overloaded, and the new definition included f^pi and r rather than just their scalar dot product).\n\n-Line 8 of algorithm 1 (I think it’s the update to the belief about the reward function) is a bit confusing to me at a high level; high-level intuition/clarification in the pseudocode or in the relevant text (the second to last paragraph of Section 4) would be nice.\n\n-The error bars of Figure 2 are not specified (I assume they are standard error based on the other figures).\n\n-Please specify the number of trials (random seeds, or “runs”) used for Figure 2.  (This would be a major issue, except that the bars, which I assume are standard error, see above, seem to indicate that there is a reasonable amount of statistical significance in this experiment.)\n\n-The caption of Figure 6 in the appendix has a dead reference (to a section or a figure perhaps).\n\nMajor issue: Line 114: p^* is completely undefined. “I” (the mutual information?) is also undefined. Because of these undefined symbols, I’m not following the next parts (e.g., the “two undesirable properties”) as well as I would like.  Related minor suggestion: Assuming the latter is mutual information, perhaps there should be a little background on this; I suspect readers are more likely to know the basics of MDPs than the basics of information theory (and the authors have rightly included MDP basics).  This can be a short appendix section if there’s no room in the main paper.\n\nQuestion/suggestion: Throughout the paper, you assume unlimited and cheap access to the environment and its transition function (minus the reward function), correct?  There’s nothing with this problem setting, but it might be worth explicitly mentioning this point in Section 1 or 3.\n\nIn summary, I strongly recommend an accept.  This recommendation is contingent on the authors addressing the (easily-fixed) issues I am concerned about.  However, my confidence in my assessment is low; so my assessment may change depending on the feedback of/discussion with the other reviewers.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "~6-7"}, "extra_text": {}}, {"review_id": "rnhyZcElErg", "reviewer": "Reviewer_nVyN", "summary": "Information Directed Reward Learning (IDRL) actively selects queries that are most informative at distinguishing policies in terms of their value. By contrast, prior work has typically tried to maximize information gained about the reward function itself -- but some parts of the state space may be unreachable, or never visited by any plausible optimal policy, and so learning about them is irrelevant for task performance. IDRL has a pleasing theoretical motivation, although practical implementations of it require considerable use of approximations and heuristics. Additionally, IDRL is compatible with a broad variety of reward learning queries (e.g. preference comparisons, labeling trajectories with returns) whereas most prior work focuses on a single feedback modality. Experimental results show significant improvements in sample efficiency from use of IDRL relative to a variety of baselines.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper tackles an important problem, sample-efficient reward learning, with a novel approach that has some theoretical justification and promising empirical results. The presentation is in general very clear, and the experiments seem to be carefully executed: e.g. I was pleased to see the environment termination conditions were removed in section 7.5, which is a common pitfall for work in this area.\n\nMy main hesitation lies in the scalability of the approach. The GP approach, while having pleasing theoretical properties, seems intractable to scale without significant hand-designed feature engineering -- e.g. section 7.4 required pre-training fixed policies for Ant, and there was a relatively simple mapping from observations to reward. The deep RL approach seems potentially more scalable, but the approximations used are so substantial that the method loses much of its theoretical justification. The empirical performance of deep IDRL goes some way to allaying this concern, but the improvements are modest and inconsistent between environments, on occasion even doing worse than random sampling.\n\nOverall I vote to accept the paper. In particular, the key idea of choosing queries based on how this affects the return achieved seems like an insight that will prove useful in the long-term. I am unsure if the IDRL method itself will see adoption, but it is certainly plausible, especially as Bayesian NNs improve the deep IDRL method may be able to scale.\n\n# Originality\n\nModerately original. The algorithm is new, but operates in the existing (but understudied) paradigm of active reward learning. The key idea of focusing on information gain on policy-relevant factors rather than reward itself has been considered in other contexts before (e.g. Russo and Van Roy, 2014 which is cited in the appendix). But as far as I know it has not been applied to reward learning in general sequential decision making problems before, and the focus on difference in return between policies is new (although I'm not convinced it's quite the right objective).\n\n# Quality\n\nThe paper generally seemed to be carefully executed. I did not check the proofs in the appendix but all claims and derivations seemed plausible. Details of the experimental setup were for the most part clear, the baselines were reasonable and when a direct comparison could not be made the authors highlighted this limitation.\n\n# Clarity\n\nWhile the paper was in general easy to read, there were a few parts that seem confusing or ambiguous:\n  - Use of the phrase \"potential reward function\" at line 248 and elsewhere is ambiguous. I think you mean something like a candidate reward function. But it could also be misunderstood to mean a reward function based on potential shaping. I would suggest rephrasing.\n  - It is unclear what the reward function in section 7.4 actually is -- can you add more detail or least reference a relevant section in the appendix? In particular, I am not sure what is meant by a \"one dimensional grid\". Furthermore, you say it is \"different across the grid\" -- different in what way?\n  - You say IDRL supports multiple query types, but it's unclear to me exactly how flexible it is. Could you choose between queries of different types (e.g. preference comparisons vs reward labels) on the basis of informativeness? If so, why does no experiment do this? Multi-modal feedback seems particularly promising to me, as some feedback sources (e.g. demonstrations) are likely to be more sample-efficient at the start of training when uncertainty everywhere is high, whereas other feedback sources (e.g. preference comparison) are likely to be more informative towards the end of training when it is necessary to disambiguate specific areas. This could be a key advantage of IDRL that I don't see any mention of currently.\n\nThere were also a few more minor points, which I discuss at the end of the review.\n\n# Significance\n\nThe problem domain is important. I find it unlikely that IDRL will be of direct use to practitioners today due to the computational cost and complexity of implementation -- it may often be cheaper to just gather more data -- but I expect this to improve over time.\n\n-----\n# Minor typos etc\n\n- Figure 1: overall I really liked this figure. However I initially didn't parse the square around the pear as being an additional wall, and so was confused why there were only two plausible optimal policies. It didn't take long to figure out, but it might be worth being explicit in the caption: e.g. “It can only move 4 timesteps in the gridworld, *cannot move through walls (in black)* and collecting more food is always better.”\n\n- Line 72: “(MDPs, Puterman, 2014)” is confusing – is MDP an author? Use a semicolon instead to separate: (MDPs; Puterman, 2014)”.\n\n- Line 79: $r_t$ is not defined. Although clear from context, this is still worth defining, e.g. you could add it in the sentence starting “In an MDP, ...”\n\n- Line 114: what is $\\hat{p}^*$? I think this is meant to be $\\hat{\\pi}^*$?\n\n- References: some capitalization issues, likely due to missing {} around words to preserve capitalization in your .bib file. Line 396: “Openai gym”, line 423: “atari”. Please proofread this section.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "One significant limitation of the work that is unfortunately not discussed is that it may harm generalization relative to other active learning criteria, such as maximizing information gain over the reward. You highlight generalization as a significant advantage of reward learning (line 44 onwards): \"policies tend to generalize poorly between environments\" while \"reward functions are a more robust representation of desired behavior\". But if you seek only information that lets you discriminate between policy return, you are liable to lose much of this generalization benefit. Indeed, consider Figure 1: IDRL specifically avoids querying about the pear. If at deployment the wall around the pear is removed, an IDRL agent will be highly uncertain what to do, whereas random sampling or better yet EVR would have learned about the pear.\n\nThere's a fundamental tradeoff here, and I think IDRL may well be the right choice for many applications, but it seems disingenuous to simultaneously claim the benefits of reward learning in terms of generalization and systematically avoid gathering information that would help generalize.  I did find it interesting that the cosine similarity (Figure 3) nonetheless was higher or comparable to existing method, but I would need to see comparison of learned to ground-truth rewards in more environments to be convinced, or explicit test of transfer to new environments.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "kbDYCWBEcO", "reviewer": "Reviewer_vsGF", "summary": "This paper proposes a new active reward learning method called IDRL, designed to select the most informative query for identifying an optimal policy among a set of plausibly optimal policies. First, IDRL selects two candidate policies that maximize the entropy of the difference in expected returns given past queries. Then IDRL chooses queries that reduce the uncertainty of the return difference most among a set of candidate queries. Updating the reward model using the selected queries rather than queries maximizing the information gain about the reward stimulates identifying the optimal policy. Experimental results show that IDRL requires fewer queries than the considered baselines and works with different query types.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "(1) Originality and significance: \\\nThis work changes the main focus of active reward learning from achieving a low estimation error of the reward to directly improving the policy which the reward model induces. This approach can be a new direction for further research in the active reward learning area.\n\n(2) Quality: \\\nThis paper is supported by theoretical background on Gaussian distribution and GP. IDRL scales to Deep RL setting with approximation the posterior as Gaussian using Laplace approximation, making the algorithm tractable (including computation of entropy and information gain).\n\nHere are some related questions. \\\nQ1. Can you show any statistical test for the results in Figures 3 and 4? \\\nQ2. What is the performance tendency on the number of candidate policies (although too many candidates cannot be used due to the computational cost)?\n\n(3) Clarity: \\\nHere are some questions for clarification. \\\nQ3. In Section 7.5, \"it(IDRL) generates the candidate queries rolling out the currently optimal policy and the candidate policies(356-357).\" On the other hand, Algorithm 5 of Appendix E.1 uses the currently optimal policy only (line 23).\" Does the author say that the former is IDRL and the latter  'IDRL without candidate policy rollouts'?\\\nQ4. In Algorithm 5 of Appendix E.1, does line 25 mean that the Hessian is calculated after (virtually) updating DNN with $q_i$ and its predicted response $y_i$? \\\nQ5. In Section 7, \"the agent's queries are answered with simulated feedback based on an underlying true reward function(219).\" Could you clarify how 'simulated feedback(expert)' runs (or is implemented) in detail?\n\n\n---------------- Post Rebuttal --------------------\n\nI read the authors' response carefully and appreciate the detailed explanations. I am inclined to accept this paper.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed the limitations and potential societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "O3IS8QA563h", "reviewer": "Reviewer_kbe1", "summary": "The authors introduce a query selection strategy for reward learning in reinforcement learning. This strategy can use different query types and is not restricted to pairwise trajectory queries, as many other approaches. Additionally, the strategy is also considering the environment dynamics. The papers main contribution is a method, that reduces the number of required queries, besides the aforementioned aspects. The authors also show, that the approach is also applicable to Deep RL techniques.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The algorithm is based on well known entropy-based learning approaches, but combines this idea with policy awareness. This is an interesting idea and a relevant step forward, despite the fact that the elements themselves are not novel. As this algorithms builds on principles with well established theory, it can be considered sound and is clearly described. Related work is sufficiently covered, differences mentioned and the experimental section is also mostly well structured and understandable. \n\nThe novel sections (wrt. \"Improvements Made\") are a clear improvement of the paper, but could be improved. As example, the algorithm 5 from the appendix is relevant for understanding the DNN setup and should be moved to the main paper, or at least, explained in more detail. A more substantial limitation is the discussion of the results from the Deep RL setting (7.5). From the appendix, it is clearly visible that IDRL is subject to failure modes (e.g. Swimmer-v3) or premature convergence (Ant-v3, InvertedDoublePendulum-v2). The same is true for most other approaches, but potential reasons are not really discussed. The section (together with 7.4) is sufficient for fulfilling the claim, that it is computationally possible to scale the approach to higher dimensional problems, but performance remains in question. \n\nThe rest of the evaluation was performed on simpler domains, enabling the authors to disregard the scalability and exploration issues of the RL approach (by fully recomputing the candidate set multiple times). This is very interesting, as it allows to specifically determine the effect of the query selection strategy, which is the papers main contribution. It should also be positively considered, that they compare against a broad range of baseline methods. However, the capability of dealing with non-linear reward queries was not evaluated, but it is clear, that it is theoretically possible.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors addressed several limitations, but there is quite a bit room for improvement. This mostly relates to section 7.5 (as stated in the main review). It is especially interesting if the mentioned problems are induced by the DNN approach,  the limited candidate update rate or the higher dimensionality. Multiple ideas come to mind: Using scalable, exact GP approaches [Exact Gaussian Processes on a Million Data Points, Wang, 2019], Kernel Neural Networks with GPs [Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness, Liu, 2020] or running SAC from scratch often. However, i acknowledge, that this would introduce substantial runtime demands, making it an interesting study for future work.\nLastly, evaluations concerning the non-linear query capabilities should be added", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new active reward learning algorithm, where where the agent is required to reason about unknown reward function by querying an expert. The main idea is to select queries that maximize the information gain about the difference in return between policies with high uncertainties of their return difference. The results show that the proposed method can achieve similar or better performances with fewer queries and can work with different types of queries unlike the prior work.

All of the reviewers appreciated that the proposed objective that focuses on improving policy as opposed to reducing reward model error, which has been done in the prior approaches, is novel and technical sound. The paper also presents both a theoretically exact implementation and a scalable approximation for Deep RL. The results look solid in that the proposed method is much more query-efficient and can work with more types of queries compared to the baseline approaches. Overall, this is a solid and neat paper. Therefore, I recommend to accept the paper.
Decision: Accept (Poster)