Learning Multimodal Rewards from Rankings
Authors: Vivek Myers
Erdem Biyik
Nima Anari
Dorsa Sadigh
Venue: CoRL2021 Oral (2021)
Keywords: HRI
reward learning
multi-modality
rankings
active learning

Abstract: Learning from human feedback has shown to be a useful approach in acquiring robot reward functions. However, expert feedback is often assumed to be drawn from an underlying unimodal reward function. This assumption does not always hold including in settings where multiple experts provide data or when a single expert provides data for different tasks---we thus go beyond learning a unimodal reward and focus on learning a multimodal reward function. We formulate the multimodal reward learning as a mixture learning problem and develop a novel ranking-based learning approach, where the experts are only required to rank a given set of trajectories. Furthermore, as access to interaction data is often expensive in robotics, we develop an active querying approach to accelerate the learning process. We conduct experiments and user studies using a multi-task variant of OpenAI's LunarLander and a real Fetch robot, where we collect data from multiple users with different preferences. The results suggest that our approach can efficiently learn multimodal reward functions, and improve data-efficiency over benchmark methods that we adapt to our learning problem.

Learning Multimodal Rewards from Rankings
Vivek Myers' i
Erdem Bryak' t Nima Anari i Dorsa Sadigh ist
Department of Computer Science, Stanford University Department of Electrical Engineering, Stanford University
(vmyers, ebiyik @stanford. ( edu, {anari, , 0cs stanford. edu
Learning from human feedback has shown to be a useful approach in acquiring robot reward functions. However, expert feedback is often assumed be drawn from an underlying unimodal reward function. This assumption does not always hold including in settings where multiple experts provide data or when a single expert provides data for different tasks-we thus go beyond learning a unimodal reward and focus on learning a multimodal reward function. We formu- late the multimodal reward learning mixture learning problem and develop a novel ranking- -baed learning approach, where the experts are only required to rank given set of trajectories. Furthermore, access interaction data is often expensive in robotics, we develop an active querying approach to accelerate the learning process. We conduct experiments and user studies using multi-task vari- ant of OpenAl's LunarLander and a real Fetch robot, where we collect data from multiple users with different preferences. The results suggest that our approach can efficiently learn multimodal reward functions, and improve data-efficiency over benchmark methods that we adapt our learning problem.
A Abstract:
Keywords: HRI, reward learning, multi multi-modality, rankings, active learning
Introduction
Learning a reward function from different sources of human data is a fundamental problem in robot learning. In recent years, there has been large body of work that learns reward functions using various forms human input, such as expert demonstrations [1], suboptimal demonstrations [2, 3, 4], pair- wise comparisons [5, 6]. physical corrections [7, 8J. rankings [9], and trajectory assessments [10]. These works focus on learning unimodal reward function that models human pref- erences on a target task. However, this unimodality assump- tion does not always hold: human preferences are usually more complex and need to be captured via a multimodal representa- tion. Further, even if the preferences of human are truly uni- modal, we often use mixture of data from multiple humans, which can be difffiuult to disentangle, leading multimodality.
1
mch
51 3 3
ビ 33
Figure I: Fetch robot putting a banana on one of the three shelves. The two users have different preferences. and so they provide different rankings to the robot. The robot needs to be able to model multimodal reward functions for successfully achieving the task.
As an example, consider a robot placing banana on one of the three shelves (see Fig. 1). The middle shelf is often used for fruits, but it has no room left and if the robot tries to put the banana there, it may cause other fruits to fall. The top shelf has some space but has been used for cooked meals. The bottom shelf has lot of free space, but is usually used only for toys. In such a scenario, people may have very different preferences about what the robot should do. If we try to learn a unimodal reward using data collected from multiple people, the robot is likely to fail in the task, because the data will include inconsistent preferences.
One solution is of course to label the different modes in the data. For example, one could separate the data based on the preferred shelf, and learn different reward functions for each shelf. However, this separation is not always straightforward. For example in a driving dataset, it is unclear what should be labeled as aggressive or timid driving. Clustering the data based on the human who provided
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

the data is also not viable as it will introduce data -ata-innfficieney issues, and perhaps more importandly humans are not always unimodal: a usually timid driver can drive more aggressively when in hurry.
These examples motivate us to develop methods that can learn multimodal reward functions us- ing datasets that not specifically labeled with the modes. To this end, previous work proposed learning from demonstrations to learn multimodal policies [11, 12] or reward functions with multi- ple possible intentions [13, 14]. However, learning from expert demonstrations is often extremely challenging in robotics as providing demonstrations on a robot with high degrees of freedom is non- trivial [5. 15], and humans have difficulty giving demonstrations that align with their preferences due to their cognitive biases [16, 17]. Thus, it is desirable to have methods that learn from other more reliable sources of data. For instance, humans can reliably compare two different trajectories, enabling a robot to learn from pairwise comparisons [5, 9, 18].
While learning from pairwise comparisons provides rich source of data for learning reward func- tions, the theoretical results by Zhao et al. [19] imply that extending the existing comparison-basee reward learning techniques multimodal reward functions is not possible, i.e. failure cases can be constructed, where pairwise comparisons are not sufficient for identifying different modes of a mul- timodal function. Our insight is that it is possible learn a multimodal reward function by going beyond pairwise comparisons and instead using rankings.
To achieve this, we formulate multimodal reward learning mixture learning problem. As data is a very expensive resource in robotics, we further develop an active querying method that aims to ask the human users the most informative queries. Our contributions three-fold:
We develop a method that uses rankings from humans to learn multimodal reward functions. We develop an active querying method to improve data-efficiency by asking the most informative ranking queries.
We conduct extensive experiments and user studies with OpenAl's LunarLander and real Fetch robot to test our learning and querying methods in comparison to baselines.
2 Related Work
Reward Learning in Robotics Learning reward functions from human feedback is a fundamental problem in robot learning. Ng and Russell [20] and Abbeel and Ng [21] introduce the problem of learning from demonstrations in the space of robotics. Later works focus on improving inverse reinforcement learning algorithms by reducing the ambiguity in learned rewards [1, 22].
Due to the difficulty in providing expert demonstrations in robotics, recent works attempt to learn reward functions using other forms of human feedback, such as physical corrections [7, 8] and pair- wise comparisons where a human user compares the quality of two robot trajectories [23, 24, 25, 26]. Later works extend these algorithms for better time and data-efficiency [27, 28, 29]. Though there have been works to extend this framework to reward functions modeled with Gaussian processes to capture nonlinearities [6, 30, 31J, the underlying reward function has always been unimodal.
Preference-based Learning. Outside of robotics, preference-basee learning, where data are in the form of comparisons, selections out of a set, or rankings, has attracted attention due to the ease of collecting data and its reliability. Prior works have studied this in the classification [32], bandits [33], and reinforcement learning settings [34].
In problem, we have continuous hypothesis space of reward functions. In such cases, it is common to model human comparisons or rankings with a computational model. Bradley-Terry is one such model for pairwise comparisons [35], which easily extended to the queries where the human chooses the best of multiple items. Known as multinomial logits (MNL) [36], this model has been widely used for human preferences in many fields [37, 38, 39] including robotics [25, 27, 18].
To extend these models to rankings, Plackett-Luce [40, 41] and Mallows models [42, 43, 44, 45] are commonly employed. In this paper, we use the Plackett-Luce model as it is a natural extension of MNL, which is widely used in robotics with great success. We formalize this model in Section 3.
Even though there has been much research in this domain, all works we mentioned here focus on the unimodal case, and do not work with the multimodal preferences.
Learning Mixture Models from Rankings. One way to model multimodal reward functions is through mixture models, where the data is assumed come from different individual models with some unknown probabilities. To this end, previous works consider mixtures of MNLs [46, 47],
2

---

Plackett- models [19], and Mallows models [48]. Other works adopt different methods to model multimodality, such as by assuming latent state dynamics that transition between different modes [49, 50] or by learning the different modes from labeled datasets [51, 52]. To avoid these modeling assumptions, we focus on directly learning the mixture model.
While Zhao et al. [19] have theoretically studied the mixture of Plackett-Luce choice models, which also informs our algorithm in terms of the query sizes, they only focus on learning the rewards of a discrete set of items. To the best of our knowledge, our paper is the first work that deals with a continuous hypothesis space under a mixture of Plackett-Luce models. Furthermore, we propose an active querying strategy for this mixture model to improve data-efficiency for learning, which is crucial in data- mata-hungry applications such as robotics.
Problem Formulation
Setup. We consider uully-observable deterministic dynamical system. A trajectory in this system is a series of states and actions, i.e., { = (8o, 40. ,8,, arr). The set of feasible trajectories is E.
We assume there is set of M individual reward functions that are possibly different, each of which encodes some preference between the trajectories in E. For the rest of the formalism, we refer to each individual reward function an expert for the clarity of the presentation.
Following the common linearity assumption reward learning [1, 5, 28], we assume each preference be modeled as linear reward function over known fixed feature space D, the reward associated with a trajectory & with respect to the m expert is Rm = w (E). where wm is the unknown vector of weights. Across the expert population, there exists some unknown distribution over the reward parameters, corresponding to the ratio of the data provided by the experts. We represent this distribution with mixing coefficients I'm such that Σm= O'm 1. We will then learn both the unknown reward functions [wm 1m=1 and the mixing coefficients {am Jm using ranking queries made the M experts. This setup generalizes [5], which studied unimodal rewards.
Ranking Model. We define a ranking query to be a set of the form Q = {EI, {k} } for a fixed query size K. The response to ranking query is ranking over the items contained therein, of the form x= .E....... :<ax),
), where 01 is the index of the expert's ' top choice, 02 is the second top choice, and so on. While it is known which expert provided the response to the query, we know the prior that a response comes from expert m with some unknown probability Om. i.e., Pr(R = Rm) = Omm Going back our banana placing example, ranking query of K robot trajectories is generated by the algorithm, and user-whose identity is unknown to the algorithm-responde to this query.
We then capture how human experts respond to these ranking queries by modeling a ranking dis- tribution through an iterative process using Luce's 's choice axiom [53]. In this process, the experts repeatedly select their top choice Onn. with probability distribution generated with the softmax rule to generate a ranking from the order items were selected:
6 (5n))
Pr (.) = Ea, R = Rm
Ej= Bon Ben(Ea,)
In the following iterations, the experts select their top choice among the remaining trajectories:
e.im
Pr (T: = Eo, Ii,,.... , Ti- Rm)
E/_ e Rmm(Ea,))
This is known the Plackett-Luce ranking model 411. Together with the prior over experts Obn the resulting distribution over rankings x ~ X is a mixture of Plackett-Luce models with mixing coefficients O.m and weights proportional to e (E))
Hence, the ranking distribution first selects the reward function R,, with probability Omn and then selects trajectories from Q sequentially with probability proportional to the exponent of their reward, i.e., RR, among the remaining trajectories until none is left, generating ranking of the trajectories.
So given knowledge of the true reward function weights Wm and mixing coefficients am, we have the following joint mass over observations r from a query Q:
M
K
e m (55a))
Pr(x IQ]= =
(2)
L (5
m=1
3

---

Objective. Our goal is to design series of adaptive queries Q(9) to optimally learn the reward weights Wm and corresponding mixing coefficients am upon observing the query responses x*)) We constrain all queries consist of fixed number of elements K.
Active Learning of Multimodal Rewards from Rankings
In this section, we first start with presenting our learning framework. We then discuss how we can improve data-efficiency, and propose an active querying approach.
4.1 Learning from Rankings
To learn the reward weights wms and mixing coefficients am, we adopt a Bayesian learning approach. For this, we maintain a posterior over the parameters wm and amm Denoting the distribution over the parameters a, and W; as O, this posterior can be written as
Pr(O) Q+), 2011, "
2 .....
Pr
(Q(), 211), Q(2)) xx()
(
=Pr(O) II Pr Q Q ,0)) ",.,,,,,Q(I))) ((t"),,((-1)) _) Pr(())
II Pr 20 10,
(3)
where we use the conditional independence of ranking queries 21(1) given and the conditional independence of the Q!t) on O given Q x,(,, ..... Q((-1) (1(11 in the last equation. To be able to compute this posterior, we assume some prior distribution over the reward weights and the mixing coefficients, which is system- deeeedeet and may come from domain knowledge, and use Eq. (2) to calculate the likelihood terms. For example, in our simulations and user studies, we adopted a Gaussian prior wi (0, 7) and a uniform prior a Unif(A where AM-1 the unit M - 1 simplex. Learning this posterior distribution in Eq. (3), one can compute maximum likelihood estimate (MLE) or expectation as the predicted reward weights and mixing coefficients.
Equation (3) implies the queries made to the experts, Q''',, affect how well the posterior will be learned. Assuming a limited budget of queries, which is often the case in many real-world applications, including robotics, that the responses would give the highest amount of information about the reward weights and the mixing coefficients. However, this is NP-hard, even in the unimodal case with pairwise comparisons [54]. We therefore resort to greedy optimization techniques to develop our active learning approach.
would ideally find an optimal adaptive sequence of queries such
4.2
Active Querying via Information Gain
A query Q is desirable if observing its value x yields high information about the underlying model parameters, Om and Wm- Therefore, we propose using an information gain objective to adaptively select the most informative query at each querying step, generalizing the approach of [5].
Assume at a fixed timestep we have made past query observations D = (Q()) =0) desired query is then Q' = arg maxe O Q, D) where [(::-) denotes mutual information.
f==1" The
Equivalently, denoting the joint distribution over and 0- {(lm, Wm } m== conditioned on Q and D as P(X, @ Q, D), we see
M
Eø BIp Pr
Q,0")
log The details of this derivation are presented in Appendix A.
Q' arg arg min Q
E ((X,0
(4)
Pr[X = x Q.
4.3 Overall Algorithm
To efficiently solve the optimization in Eq. (4), we first note that we should use Monte Carlo approximation since the expectations are taken over a continuous variable and a discrete variable X over an intractably large set of K! alternatives. To perform this Monte Carlo integration, we require samples from the posterior Pr(X, θ IQ,
Our key insight is that we can obtain joint samples from both posteriors by first sampling from θ ~
Pr(O D) and then sampling x ~ Pr(X Q, = Ô) since θ 1 Q DD and X 1 D Q, O. We perform the sampling x ~ Pr(x Q, 0 = Q) efficiently using Eq. (2). In general, exact sampling from the posterior Pr(O D) is intractable. However, we note Eq. (3) can be directly evaluated (using Eq. (2)) and gives Pr(O) D) up to proportionality constant factor.
With this unnormalized posterior of Eq. (3), in Appendix B to generate samples from the posterior Pr(0) D).
the Netropolis-Hasting algorithm as described

---

We see our optimization problem simplifies to finding, for N fixed samples 0, ~ Pr(O) D) and corresponding samples: Iig
Pr Pr( QQ,0,)
2(Q;,,0) log Pr a Q,0;))
log Pr|m; Q.0.1 Q' arg min L(Q; 0). (5)
-
We solve this optimization using simulated annealing [55] (see Ap- pendix C). Algorithm 1 goes over the pseudocode of our approach, and discuss the hyperparameters in
Algorithm Active Querying via Information Gain
Require: Observations D
1: [0: {0:) ~Pr(O) D)) w/ Eq. (3) via Metropolis-
2: procedure QUARQU)R
our experiments in Appendix D.
3: Vi,I; Pr( Q,0:)
4.4 Analysis
4: return L(Q;
Eq. (5)
5: end procedure
We start the analysis by stating the bounds on the required number of trajectories in each ranking query achieve generic identifiabiliyy.
6: Q + MINIMIZE (EVALQQUE( Simulated annealing
7: select query Q
Plackett-Laue Luce model over E is generically identifiable if for any sets of parameters O and O₂ inducing the same distribution over the responses all queries of size K on E, the mixing coefficients of i and (₂ are the same and the induced rewards R., (E) are identical across E\t up a constant additive scaling factor.
Theorem (Zhao et al. [19]). mixture of M Plackett- models with query size and | K
generically identifiable if M S. K-2
This statement follows directly from [19]. which proves above bound assuming that each query to the Plackett-Luce mixture is a full ranking over the set of items (i.e. | K). However, the assumption [ = K untenable the active learning context, as prevents any active query selection. To apply this result for our active learning algorithms, we relax the condition to | 2 K.
Corollary 1.1. A mixture of M Boackett-Luce models with query size K is generically identifiable if < K- K-2
We prove Corollary 1.1 in Appendix E. In our context, generic identifiability implies if the human response is modelled by a Plackett-Luce mixture, our Algorithm I will be able recover its true parameters (up to constant additive factor for the rewards) in the limit of infinite queries.
Remark 1. Greedy selection of queries maxinizing information gain in Eq. (4) is not necessarily within a constant factor of optimality.
Appendix E.2 justifies Remark 1. In fact, greedy optimization of information gain for adaptive active learning can be significantly worse than constant factor of optimality in pathological settings [56]. Despite its lack of theoretical guarantees, information gain is commonly used effective approach in adaptive active learning [29, 57, 58]. Although other approaches like volume removal satisfy adaptive submodularity [25], they fail settings with noisy observations by selecting high-noise low-information queries, and in practice achieve far worse performance than information gain.
Experiments
Having presented our learning and active querying algorithms, we now evaluate their per- formance in comparison with other alternatives. We start with describing the two tasks we ex- perimented with:
LunarLander. We used 1000 trajectories in
(e) Lending the londing Figure 2: The LunarL.ander environment is visualized with the two tasks. Sample trajectories associated with these tasks shown.
(b) Slaying in the air long prpesiible
OpenAl Gym's LunarLander environment [59] shown in Fig. 2 (see Appendix G.1 for details on how they were generated).
Fetch Robot. We generated 351 distinct trajectories of the Fetch robot [60] putting the banana on the shelves shown in Fig. 1 (see Appendix G.2 for details).
5.1 Methods
We compare our active querying via information gain (IG) discussed in Algorithm with two base- lines: A simple benchmark for active learning is rundom query selection without replacement. We also benchmark against volume removal (VR), common objective for active learning of robot re- ward functions [25]. See Appendix F for the details of these two baselines.
5

---

5.2 Metrics
We want evaluate both the active querying and the learning performance. The former requires metrics that assess the quality of the algorithm's 's selected queries D = information they provide on the model parameters O. We use two such metrics: mean squared error (MSE) and log-likeliben oogliiklihoodd Since both active and non-active methods are expected to reach the same performance with a large number of queries, we look at the area under the curve (AUC) of these two metrics over number of queries. To evaluate the learning performance, we quantify the success of robot, which learned a multimodal reward, via the learned policy rewards on the actual task.
} in terms of the
MSE. Suppose we know the human is truly modeled by O* adhering to the assumed model class of Section 3. Given a set of observations D, we compute MLE estimate θ of the model parameters using Eq. (2). The MSE then the squared error between Q and O- (see Appendix H.I).
While this metric cannot be evaluated with real humans, we can use this metric with synthetic human models (model with known parameters (*) in simulation. A lower MSE score means the selected queries D allow us to better learn a multimodal Plackett-Luce model close to the true model O"
Log-Likelihodd. The log-likelihood metric measures the log- lngllikelihodd of the response to a random query given the past observations D. If the past observations are informative, the true response to random query Q will in expectation be more likely, meaning the log-likelihood metric will be greater. See Appendix H.2 for details on how we compute this metric.
Learned Policy Reward. We take the MLE estimate of each reward weights vector and train a DQN policy using them [61].' We then run these learned policies on the actual environment with the corresponding true reward functions (see Appendix H.3) to obtain the learned policy rewards.
5.3 Results
Multimodal Learning is Necessary. We first compare unimodal and multimodal models show the insufficiency of unimodal rewards when the data come from a mixture. To leave out any possible bias due to active querying, we make this comparison using random querying.
We let the true reward function have M = 2 modes and set a query size of K = 6 items for identi- fiability as Section 4.4 suggests, and for acquiring high information from each query. We simulate 100 pairs of experts whose reward weights wm and the mixing coefficients am are sampled from the prior Pr( Having these simulated experts respond to 15 queries, we report the MSE in Fig. 3. LunarLander
Fetch Robot
The unimodal reward model causes an unstably increasing MSE. This is mostly due to the out- liers where the reward weights w1 and w/ are far away from each other and the unimodal reward fails to learn any of them. We therefore also plot the median values and quartiles in Fig. 3. While the bimodal reward model learned using our proposed approach decreases the MSE over time, the unimodal model has a roughly con- stant MSE, which suggests it is unable to learn when the data come from mixture.
60
80
Unimndal (mssn)
60
--Unimodal (medion)
Miettuee (mean)
MSE
40
Mixxtue (medias)
20
20
10
# Queries
of Queries
Figure 3: Unimodal and bimodal reward learning mod- els are compared under MSE. Both mean and median values (over 100 runs) are shown. Shaded regions show the first and the third quartiles.
We present an additional unimodal learning baseline evaluated on the user study data in Appendix K.
Active Querying with Information Gain Dfta-Efficient. We next compare our information- gain-based active querying approach with the other baselines. For this, we use the same experiment setup as above with M = 2 reward function modes and ranking queries of size K = 6, and simulate 75 pairs of human experts. We present the results in terms of MSE in Fig. 4. In LunarLander, the information gain objective significantly outperforms both random querying and volume removal in terms of the AUC MSE (p- 0.005, paired- sample t-test). Notably, volume removal performs even worse than the random querying method, which might be due to the known issues of volume removal optimization as briefly discussed in Appendix F.2. On the other hand, the difference is not statisti- cally significant in the Fetch Robot experiment, which might be due to the small trajectory dataset,
As we are using real Fetch robot for our experiments and it would be infeasible and unsafe to train DQN on Fetch, so this metric is limited to our simulations, i.e., LunarLander in our experiments.

---

online form. Afterwards, or because almost all trajectories in the dataset minimize or maximize some of the trajectory fea- tures, accelerating and simplifying learning under the linear reward assumption. See Appendix G.2 for details about the trajectory features and how we generated the trajectory dataset.
We further analyze the querying methods in this multimodal setting under the log-likelihood metric in Fig. 4. Information gain significantly outperforms random querying and volume re- moval in both experiments with respect to the AUC log-likee likelihood (p < 0.005). With respect to the final log-likelihne likelihood, information gain re- duces the amount of required data in LunarLan- der by about 35% compared to random query- ing and about 60% to volume removal. Sim-
LunarLander
Fetch Robot
Mixture- IG
12
20
Mixture
Mixture Rendom
MSE
10
15
10
Lag- Lelllbol
-9
ilarly in the Fetch Robot, the improvement is approximately 25% over both baselines.
-12
10
10
Appendix presents two additional experi- ments: one which clearly shows the effective- ness of our approach for learning a mixture of more than two reward functions (specifically, M 5))), and one which studies the robustness against misspecified M.
# of Quertiss
# of Quartes
Figure 4: Different querying methods are compared with the (top) MSE and (bottom) log- likelihood metrics (mean-sse over 75 runs).
Information Gain Leads to Better Learning. Having seen the superior predictive performance of the reward learned via information gain optimization, we next assess its performance in the actual environment. As random querying outperforms volume removal in terms of log likelihood and MSE as in Fig. 4, we compare the information gain with random querying.
For this, we run the multimodal reward learning with 75 pairs of randomly generated reward weights (M = 2 and K 6). For each of the 150 individual reward functions, we compute the learned policy rewards. Fig. 5 shows the results. While the standard errors in the plots seem high, this is mostly because optimal trajectories for different reward weights differ substantially in terms of rewards, which causes an irreducible variance. However, since the underlying true rewards are the same between the information gain and random querying methods, we ran the paired sample t- test between the results and observed statistical significance (p < 0.05). This means although the learned policy rewards between different runs differ substantially, the reward function learned via the information gain method leads to better task performance compared to random querying.
6 User Studies
Mixture- Random
*
We now empirically analyze the performance of our algorithm with two online user stud- ies.? We again used the LunarLander and Fetch Robot environments. We provide summary and a vidco of the user studies and their re- sults at htttps://sstts..eee .owgoooooo......oeieeee.o.oe com/view/ mulltiodaal--eeeen rewarddalrreewar--eaannnnnn
Mixture-
0.5
1.5
Reward
Figure 5: Information gain and random querying meth- ods are compared with the learned policy rewards (meanaise over 75 runs which correspond to 150 ran- domly generated reward weights) in LunarLander.
Experimental Setup. For LunarLander, subjects were presented with either of the following in- structions at every ranking query: "Land softly on the landing pad" or "Stay in the air as long as possible". We randomized these instructions such that users get one of them with 0.6 and the other with 0.4 probability. We kept the presented instructions hidden from the learning algorithms so that they need to learn a multimodal reward without knowing which mode each ranking belongs to.
For the Fetch robot environment, we recorded the 351 trajectories on the real robot as short video clips so that the experiment can be conducted online under the pandemic regulations. Human sub- jects participated in the experiment as groups of two to test learning from multiple users. Each participant was instructed that the robot needs to put the banana in one of the shelves and different shelves have different conditions (the same as in our running example, see Fig. 1, Appendix I.1).
After emphasizing there is no one correct choice and it only depends on their preferences, we asked each participant to indicate their preferences between the shelves
We have IRB approval from research compliance office under the protocol number IRB-52441.
7

---

each group of two subjects responded to 30 ranking queries in total where each query consisted of 6 trajectories. We selected who responds each query randomly, with probabilities 0.6 and 0.4.
Appendix 1.2 presents details on the user interface used in our experiments.
Independent Variables. We varied the querying algorithm: active with information gain and ran- dom querying. We excluded the volume removal method to reduce the experiment completion time for the subjects, as it already performed worse than random querying in our simulations.
Procedure. We conducted the experiments as a within-subjece study. We recruited 24 participants (ages 19- - 56; 9 female, 15 male) for LunarLander and 26 participants (ages 19- 56; 11 female, 15 male) for the Fetch robot. Each subject in the LunarLander, and each group of two subjects in the Fetch robot experiment responded to 40 ranking queries; 15 with each algorithm and 10 random queries for evaluation at the end. The order of the first 30 queries was randomized to prevent bias.
Dependent Measures. Learning the multimodal reward functions via the 15 rankings collected by each algorithm, we measured the log-likelihood of the final 10 rankings collected for evaluation.
Hypotheses. With LunarLander and Fetch robot, we test the following hypotheses respectively:
H1. Querying the participants, who are trying teach two different tasks, actively with information gain will lead to faster learning than random querying.
H2. While learning from two people with different preferences, active querying with information gain will lead to faster learning than random querying.
Results. Figure 6 visualizes how log-likelihood of the evaluation queries changes over the course of learning by both algorithms. Active querying with information gain leads to sig-
LunarLander
Fetch Robot
-70
-50
Log-Liioeibooed
-90
-90
-90
-100
nificantly faster learning compared to random querying in LunarLander. Indeed, the differ- ence in AUC log-likelible likelihood is statistically sig- nificant (p < 0,05). Furthermore, the active querying method enabled reaching the final per- formance of random querying after only 9 or 10 queries, for around 35% reduction in the amount of data needed, supporting H1.
-100
-120
Mixture IG Mixiure Racdom
0
10
15
0
5
10
15
#
Queries
# of Queries
Figure 6: User study results (meantise over 21 users for LunarLander and 13 groups for Fetch Robot).
As the robot experiments have a an easier task with a small number of variables between the trajec- tories, both querying methods converge to similar performances by the end of 15 queries. However, active querying accelerates learning in the early stages- difference in AUC log-likelihood is again statistically significant (p < 0.05). Looking at the final performance with random querying, improvement in data efficiency is about 10%, supporting H2.
7 Conclusion
Summary. This work presents a novel approach for learning multimodal reward functions. We for- mulated the problem as mixture learning problem solved using ranking queries that are answered by experts. We further developed an active querying method that maximizes information gain to improve the quality of ranking queries made to the experts. The results suggest our model learns multimodal reward functions, with data efficiency improved by our new active querying method.
Limitations and Future Work. Our model for learning multimodal rewards requires knowing the number of different modes (experts tasks) M in advance. This might be difficult in some settings. For example, when several experts belonging to different clusters, e.g.. timid and aggressive drivers, provide data, it might be difficult to know the number of clusters in advance. However, simple approach that fits the multimodal reward under various M could reveal the true number of underlying modes. Another challenge is that learning a mixture reward model may contribute to the reward ambiguity problem in inverse reinforcement learning: each individual reward may have its own ambiguity. Future work should investigate the practical implications of this. In addition, theoretical results assert, to guarantee the reliable learning of a multimodal reward with M modes, ranking queries should consist of K queries such that M S KK= Jt. While this is manageable by multiple pairwise comparisons for each query or an iterative process where the expert selects the top item, it might consume too much time for large M. Thus, an interesting future direction is to investigate how to incorporate multiple forms of expert feedback, e.g., demonstrations in addition to rankings, to pretrain and reduce the required interaction time with humans.
8

---

Acknowledgments
The authors would like to acknowledge funding by NSF grants # #1849952 and #1941722, FLI grant RFP2-000, and DARPA.
References
[1] B. D. Ziebart, L. Maas, A. Bagnell, and Dey. Maximum entropy inverse reinforeement learning. In Aaai, volume 8, pages 1433- 1438, 2008.
[2] Z. Cao and D. Sadigh. Learning from imperfect demonstrations with varying dynamics. IEEE Robotics and Automation Letters, 2021.
[3] D. H. Grollman and A. Billard, Donut as Learning from failed demonstrations, ICRA, 2011.
[4] Y.-H. Wu, N. Charoenphakdee. H. Bao, V. Tangkaratt, and M. Sugiyama. Imitation learning from imper- fect demonstration. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 18-866.77 PMLR, June 2019.
[5] E. BIyIk, Losey, M. Palan, N. Landolfi, G. Shevchuk, and Sadigh. Learning reward functions from diverse sources of human feedback: Optimally integrating demonstrations and preferences. The International Journal of Robotics Research (IJRR), 2021.
[6] E. Biyik, N. Huynh, M. J. Kochenderfer, and D. Sadigh. Active preference-based gaussian process re- gression for reward learning. In Proceedings of Robotics: Science and Systems (RSS), July 2020.
[7] A. Bajesy, D. P. Losey, M. K. O'Malley, and A. D. Dragan. Learning from physical human corrections, one feature a time. In Proceedings of the 2018 ACM/IEEE International Conference on Human--Robo Interaction, pages 141-149, 2018.
[8] M. Li, A. Canherk, D. P. Losey, and D. Sadigh. Learning human objectives from sequences of physical conrections. In International Conference on Robotics and Automation (ICRA). 5 2021.
[9] D. Brown, W. Goo, P. Nagarajan, and S. Niekum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. In International Conference Machine Learning, pages 783-792. PMLR, 2019.
[10] A. Shah and J. Shah. Interactive robot training for non-markov tasks, arXiv preprint arX1v:2003.02232, 2020.
[11] K. Hausman, Y. Chebotar, S. Schaal, G. Sukhatme, and Lim. Multi-modal imitation learning from unstructured demonstrations using generative adversarial nets. 3/st Annual Conference on Neural Information Processing Systems (NIPS 2017), pages 1236-1246, 2018.
[12] C. Fei, B. Wang, Y. Zhuang, 7., Zhang, J. Hao, H. Zhang, X. Ji, and W. Liu. Triple-gail: A multi- modal imitation learning framework with generative adversarial nets. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 2929-2935 July 2020.
[13] M. Babes, V. N. Marivate, K. Subramanian, and M. L. Littman. Apprenticeship learning about multiple intentions. In ICML, 2011.
[14] G. Ramponi, A. Likmeta, A. M. Metelli, A. Tirinzoni, and M. Restolli. Truly batch model-free inverse reinforcement learning about multiple intentions. In International Conference on Artificial Intelligence and Statistics, pages 2359-2369. PMLR, 2020.
[15] B. Akgun, M. Cakmak, K. Jiang, and A. L. Thomaz. Keyframe- -basd learning from demonstration. International Journal of Social Robotics, 4(4):343-355, 2012.
[16] C. Basu, Q Yang, D. Hungerman, M. Singhal, and A. D. Dragan. Do you want your autonomous drive like you? In 2017 12th ACM/IEEE International Conference Human-Robot Interaction (HRI, pages 417 417-425, 2017.
[17] M. Kwon, E. Biyik, A. Talati, K. Bhasin, D. Losey, and D. Sadigh. When humans aren't optimal: Robots that collaborate with risk-aware humans. In ACM/IEEE International Conference on Human- Robot Interaction (HRI), March 2020.
[18] C. Basu, M. Singhal, and A. D. Dragan. Learning from richer human guidance: Augmenting comparison- based learning with feature queries. In Proceedings of the 2018 ACM/IEEE International Conference on Human- Robot Interaction, pages 132-140, 2018
[19] Z. Zhao, P. Piech, and L. Xia. Learning mixtures of plackett-luce models. In International Conference on Machine Learning, pages 2906-2914, PMLR, 2016.
[20] A. Y. Ng and S. Russell. Algorithms for inverse reinforcement learning. In in Proc. 17th International Conf. Machine Learning, 2000.

---

[21] P. Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference Machine learning, page 1, 2004.
[22] N. D. Ratliff. J. A. Bagnell, and M. A. Zinkevich. Maximum margin planning. In Proceedings of the 23rd 2006.
international conference Machine learning, pages
[23] M. Cakmak, S. S. Srinivasa, M. K. Lee, J. Forlizzi, and S. Kiesler. Human preferences for robot human hand-over configurations. In 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 1986 1996-1993,2222 2011.
[24] R. Akrour, M. Schoenauer, and M. Sebag. April: Active preference Seaaning-based reinforcement learn- ing. In Joint European Conference on Machine Learning and Knowledge Discovery Databases, pages 116-131. Springer, 2012.
[25] D. Sadigh, A. D. Dragan, S. S. Sastry, and A. Seshia. Active preference-based learning of reward functions. In Proceedings of Robotics: Science and Systems (RSS), July 2017.
[26] P. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In NIPS, 2017.
[27] E. Biyik and D. Sadigh. Batch active preference-based learning of reward functions. In Proceedings of the 2nd Conference on Robot Learning (CoRL.), volume of Proceedings of Machine Learning Research, pages 519-528. PMLR, October 2018
[28] N. Wilde, D. Kulić, and Smith. Active preference learning using maximum regret. 2020 IEEE/RSJ International Conference on Intelligent Robots and Syxtems (IROS), pages 10952-10959, 2020.
[29] E. Biyik, M. Palan, N. C. Landolfi, D. Losey, and D. Sadigh. Asking easy questions: user-friendly approach to active reward learning. In Proceedingy of the 3rd Conference on Robot Learning (CoRL), October 2019.
[30] M. Tucker, E. Novoseller, C. Kann, Y. Sui, Y. Yue, J. W. Burdick, and A. D. Ames. Preference-basee learning for exoskeleton gait optimization. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 2351-2357, 2020.
[31] K. Li, M. Tucker, F. BIyik, E. Novoseller, J. W. Burdick, Y. Sui, D. Sadigh, Y. Yue, and A. D. Ames. Roial: Region of interest active learning for characterizing exoskeleton gait preference landscapes. arXiv preprint arXiv:20/ 1.048 2020.
[32] L. Chen, H. Hassani, and A. Karbasi. Near-optimal active learning of halfspaces via query synthesis in the noisy setting. In Proceedings of the AAAI Conference Artificial Intelligence, volume 31, 2017.
[33] R. Busa-Fekete and E. Hüllermeier. A survey of preference-based online learning with bandit algorithms.
In International Conference Algorithmic Learning Theory, pages 18-39. Springer, 2014.
[34] C. Wirth, R. Akrour, G. Neumann, J. Fürnkranz, et al. A survey of preference-based reinforcement learning methods. Journal of Machine Learning Research, 18(13)):1444 -6,, 2017.
[35] X. Chen, P. N. Bennett, K. Collins Collins- Thompson, and E. Horvitz. Pairwise ranking aggregation in a crowd- sourced setting. In Proceedings of the sixth ACM international conference Web search and data mining, pages 193-202, 2013.
[36] X. Chen, Y. Li, and J. Mao. A nearly instance optimal algorithm for top-k ranking under the multinomial logit model. In Proceedings the Twenty-Ninth Annual ACM-SIAM Symposium on Discrete Algorithms, pages 2504-2522, SIAM, 2018
[37] M. Ben-Akiva and S. Lerman. Discrete choice analysis: theory and application to travel demand. Transportation Studies, 2018
[38] R. C. Wilson and A. G. Collins. Ten simple rules for the computational modeling of behavioral data. Elife, 8:e49547,201
[39] F. Baysk, D. A. Lazar, D. Sadigh, and R. Pedarsani. The green choice: Learning and influencing human decisions on shared roads. In 2019 IEFE 58th Conference on Decision and Control (CDC), pages 347- 354, 2019.
[40] L. Maystre and M. Grossglauser Fast and accurate inference of plackett-luce models. In Proceedings of the 28th International Conference on Neural Information Processing Systems- 1, pages 172-180, 2015.
[41] C. Archambeau and F. Caron. Plackett-luce regression: a new bayesian model for polychotomous data. In Proceedings of the Twenty- Eighth Conference on Uncertainty in Artificial Intelligence, pages 84 92, 2012.
[42] T. Lu and C. Boutilier. Learning mallows models with pairwise preferences. In Proceedingy of the 28th International Conference on International Conference on Machine Learning, pages 145-152, 2011.
10

---

[43] V. Vitelli, Ø. Sørensen, M. Crispino, A. Frigessi Di Rattalma, and E. Arjas. Probabilistic preference learning with the mallows rank model. Journal of Machine Learning Research, 18(158):1-49, 2018.
[44] T. Lu and C. Boutilier. Effective sampling and learning for mallows models with pairwise-preffrence data, Journal of Machine Learning Research, 15(117)396334009 2014. 2014.
[45] R. Busa- Busa-Fekete, E. Hüllermeier, and B. Szörényi. Preference- rank elicitation using statistical mod- els: The case of mallows. In International Conference on Machine Learning, pages 1071-1079, PMLR, 2014.
[46] P. De Blasi, L. F. James, J. W. Lau, et al. Bayesian nonparametric estimation and consistency of mixed multinomial logit choice models. Bernoulli, 16(3):679-704, 2010.
[47] F. Chicrichetti, R. Kumar, and A. Tomkins. Learning a mixture of two multinomial logits. In International Conference on Machine Learning, pages 961 1699999,. PMLR, 2018.
[48] A. Liu and Moitra. Efficiently learning mixtures of mallows models. 2018 IEEE 59th Annual Symposium on Foundations of Computer Science (FOCS), pages 627-638- 2018.
[49] J. Morton and M. J. Kachenderfer. Simultancous policy learning and latent state inference for imitat- ing driver behavior. 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), pages 1-6, 2017.
[50] C. Basu, Biyik, He, M. Singhal, and D. Sadigh. Active learning of reward dynamics from hicrarchi- cal queries. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems. 2019.
[51] Z. Cao. E. Biyik,
Wang. A. Raventos, A. Gaidon, G. Rosman, and D. Sadigh. Reinforcement learning based control of imitative policies for near-accident driving. In Proceedings of Robotics: Science and Systems (RSS), July 2020.
[52] A. H. Qureshi, J. Johnson, Y. Qin, T. Henderson, B. Boots, and M. C. Yip. Composing task-agnostic policies with deep reinforcement learning. In International Conference on Learning Reprexentations, 2019.
[53] R. D. Luce. Individual choice behavior: A theoretical analysis. Courier Corporation, 2012.
[54] N. Ailon. An active learning algorithm for ranking from pairwise preferences with almost optimal query complexity. Journal of Machine Learning Research, 13(1), 2012.
[55] D. Bertsimas, J. Tsitsiklis, et al. Simulated annealing. Statistical science, 8(1):10-15, 1993.
[56] D. Golovin, A. Krause, and D. Ray. Near-optimal bayesian active learning with noisy observations. In Proceedings of the 23rd International Conference Neural Information Processing Systems- Systems-Volume Volume 1, pages 766-77 (66-777,22 2010.
[57] N. Houlshy, F. Huszár, Z. Ghahramani, and M. Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint / 2011.
[58] A. X. Zheng, Rish, and A. Beygelzimer. Efficient test selection in active diagnosis via entropy approx- imation. In Proceedings of the Twenty-Firs Conference on Uncertainty in Artificial Intelligence, pages 675-682, 2005.
[59] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv preprint 1r0ivv:606.005440 2016.
[60] Wise, M. Ferguson, D. King, F. Diehr, and D. Dymesich, Fetch and freight: Standard platforms for service robot applications. In Workshop on autonomous mobile service robots, 2016.
[61] V. Mnih, K. Kavukcuoglu. D. Silver, A. Graves. Antonoglou. D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602. 2013.
[62] S. Chib and E. Greenberg. Understanding the petropolis-hasting algorithm. The american statistician, 4944)33273335 1995.
[63] D. Golovin and A. Krause. Adaptive submodularity: Theory and applications in active learning and stochastic optimization. Journal of Artificial Intelligence Research, 42:427 486, 2011.
11


Reviews:
[{"review_id": "hanFCa1bZfL", "reviewer": "Reviewer_zn9H", "summary": "Paper proposes to learn multimodal reward functions where the datasets comprises trajectories generated with different intentions or goals. The proposed active learning method leverages rankings that are provided by the experts to better distinguish between the different modes. Theoretical justifications for the approach is given and the method worked well in the LunarLander and Fetch Robot experiments with faster convergence and lower MSE scores. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well-organized and clearly conveys the method and results. Using rankings to improve multiple-intention learning is new to my knowledge and is relevant to the robotics community, especially those working in LbD and/or IRL. The problem setting is realistic since datasets may not have goals/intentions explicitly labelled. The experiments and user study appear well-executed and show the approach is effective compared to volume removal and random query selection methods. \n\nAlthough I'm positive about the paper, it does miss out on related work on multiple intention IRL [Babes-Vroman, 2011; Ramponi et al, 2020] where a similar mixture model was proposed. The key difference is that rankings are used in the learning process but these prior works should be discussed and compared against.  \n\nIn Eqn (3), the term $Pr (x^{(t)} | \\Theta, Q^{(t)})$ should be expanded and clarified. Since the queries depend on the previous rounds due to the inference, it's unclear whether (3) remains valid under the active inference setting? Would the estimated posterior be biased? \n\n### References\n- Monica Babes-Vroman, Vukosi Marivate, Kaushik Subramanian, and Michael L. Littman. Apprenticeship Learning About Multiple Intentions. In Proceedings of the 28th International Conference on Machine Learning, ICML ’11 \n- Giorgia Ramponi, Amarildo Likmeta, Alberto Maria Metelli, Andrea Tirinzoni, and Marcello Restelli. Truly Batch Model-Free Inverse Reinforcement Learning about Multiple Intentions. In International Conference on Artiﬁcial Intelligence and Statistics, 2020\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, the paper makes good technical contributions to the relevant problem of learning from a trajectory dataset with multiple intentions. I vote for acceptance. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Please see above regarding related work on MI-IRL and potential bias. ", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "aYS0n79_3x3", "reviewer": "Reviewer_94gh", "summary": "This paper proposes a method for multi-modal reward learning by formulating the problem as learning the parameters of a continuous mixture distribution based on ranked query responses by humans.\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Paper Strengths**\nWriting: This paper is very well written, with good flow throughout the introduction and a clear explanation of the method. Furthermore, the experiments section is formatted well; the jumps from one subsection to the next made sense and the order in which results were presented makes the experiments feel intuitive.\n\nTechnical Contribution: The method is novel AFAIK, and it is a significant technical contribution with sensible design choices along the way. \n\nExperiments: The experiments are comprehensive as they measure MSE, log likelihood, and perform real-world human studies on both a simulated environment and on a real robot. Furthermore, they justify the necessity of the design choices made for the method.\n\nResults: The authors’ method shows decent to solid improvements over baselines across all of the experiments, thus demonstrating the superiority of the method.\n\n**Paper Weaknesses**\nExperiments: Although the simulated human feedback experiments demonstrate the need for multi-modal instead of unimodal reward models, results on user studies would still be elucidating. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is a solid accept; experiments are robust and the method is a solid contribution. \n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "See weaknesses section.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {}}, {"review_id": "HlwGq3-6fpk", "reviewer": "Reviewer_4NNL", "summary": "This paper addresses the problem of learning multimodal reward functions from interaction with a human teacher. The main contribution is the use of rankings, rather than just pairwise comparisons, to pose more informative queries to a teacher. Additionally, the paper presents an information-gain method for selecting optimal queries to pose to the teacher.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Writing is clear and concise\n* Ability to model conflicting feedback from teachers\n* Use of human-provided data in the evaluation\n* Information gain-based approach to query selection results in improved performance over baseline\n\nWeaknesses:\n* Multimodal approach is not well justified. More discussion is needed regarding the cases where a multimodal approach is preferable over learning multiple, unimodal models. \n* Evaluation focuses mainly on testing the querying approach, rather than demonstrating the benefit of a multimodal approach.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper addresses the problem of learning multimodal reward functions from interaction with a human teacher. The main contribution is the use of rankings, rather than just pairwise comparisons, to pose more informative queries to a teacher. Additionally, the paper presents an information-gain method for selecting optimal queries to pose to the teacher.\n\nThe experiments described in this paper make use of data provide by real users in a user study, and the results indicate that the information gain method produces better results in a simulated lunar lander task and in a sorting task on a Fetch robot. \n\nOverall, the goal of learning a multimodal model is an interesting technical challenge. However, the motivation for it is not yet clear. What is the benefit of learning a single model that is a mixture of multiple teachers' conflicting feedback? Rather than learning multiple models that reflect these preferences separately? The evaluation also focuses on the benefit of using an information gain approach to selecting queries, but does not provide a compelling justification for why a multimodal approach is ideal in the first place.\n\nThe implications of using a multimodal model should also be addressed. What makes multiple experts' reward weights compatible enough that they can be represented using a single model? And at what point *should* they be represented using multiple models? How would this approach scale to M > 2 modes?\n\nOverall, this paper addresses a very interesting technical problem by modeling teachers' conflicting preferences in a single model. While the evaluation provides support for the proposed querying method, it does not yet fully justify why/when a multimodal approach should be used.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "* What is the benefit of learning a single model that is a mixture of multiple teachers' conflicting feedback? Rather than learning multiple models that reflect these preferences separately? \n* What makes multiple experts' reward weights compatible enough that they can be represented using a single model? And at what point *should* they be represented using multiple models? \n* How would this approach scale to M > 2 modes?"}}, {"review_id": "DiIf7DsEq6k", "reviewer": "Reviewer_H6SU", "summary": "This paper addresses the problem of learning from multi-model ranking feedback from humans. Given that different people may have different preferences over a learning agent’s behavior, it is important to explicitly model this effect to capture the underlying data distribution. This paper also proposes an active learning algorithm that proposes ranking queries to maximize information gain.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "To effectively and efficiently learn from human feedback is an important problem in robot learning. Learning from multiple humans adds additional challenges. This work proposes to explicitly learn multi-modal ranking models when we know there are different modes in the data. Experiments in simulation and with real human users demonstrate the effectiveness of the proposed active learning method. \n\nOne major weakness of the proposed method, as the authors also acknowledge, is that it requires knowing the number of different modes, which is unlikely to be true for many real applications. It is desirable to see results from simulation experiments to show how robust or sensitive the proposed algorithm is to a noisy estimation of the number of modes. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The problem setting is important, the proposed algorithm is novel and the experiments are convincing. However, the limitations of the algorithm is not well-tested. I recommend accepting the paper but will not argue against rejecting it. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See weaknesses. More ablation experiments are needed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: **Update after rebuttal**
I thank the authors for their thorough response, revising the paper based on feedback and clarifying some misunderstandings. My concerns have been addressed. I recommend accept

**initial meta-review**
**summary**
This work addresses the problem of learning reward functions from demonstrations that may have been generated from varying reward functions. It proposes an approach to leverage rankings over demonstrations to learn multi-modal reward functions. Furthermore, an active querying framework is introduced that aims to reduce the amount of demonstrations required.

**Strengths**

Learning reward functions from multiple different demonstrations is an important problem in the robotics community

A novel methodology of learning from rankings of demonstrations is presented, together with active query-ing approach that addresses scalability concerns

The approach is evaluated in simulated settings, as well as on demonstrations collected on real fetch robot.

**Weaknesses**

I agree with the reviewer asking for more motivation on why it’s important to learn multi-modal reward functions.  Additionally, I would expect additional baselines. For instance, a simple baseline could have been to just pick the top ranked demonstration and learn a uni-modal reward from that. This would have told us whether there is any benefit from learning multi-modal reward functions. Your current comparison (learning unimodal rewards from conflicting demonstrations vs learning multi-modal reward from conflicting demos) does not really answer the question of “are multi-modal rewards needed”. Without this additional baseline the impact of the proposed approach cannot really be measured.

The related work section should more clearly create connections between what has existed before, and how this work is extended. For instance currently the manuscript does not very clearly state which type of approach towards learning reward function weights is used, and what work your method is based on. Only the linear reward function assumption is stated, however various IRL styles use this assumption. After some searching, it seems like for instance this could be a good reference to clearly state as setting some foundations of the work in this manuscript: https://arxiv.org/pdf/2006.14091.pdf) . This lack of clearly stating what overall IRL approach is used makes it a) harder to understand the overall approach quickly b) harder to appreciate the progress made. Finally some other relevant related work should be included as pointed out by one of the reviewers.

The mixture of reward functions probably makes the issue of ambiguous reward functions worse? If yes it would be great to see some discussion on this topic, as it is typically a major concern of this type of IRL approach.

Decision: Accept (Oral)