Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper proposes to learn from a novel type of human feedback in the form of scale-based preferences, i.e. allow a human teacher to indicate how much they prefer one trajectory query over another using a slider bar. Such form of feedback provides more nuanced information compared to binary/ternary choice-based feedback as used in existing preference-based reward learning methods. The authors formulate the problem of learning from scale feedback under the preference learning framework, develop a probabilistic model to account for noisy user feedback and propose two active learning methods that generate queries to maximize information gain and max regret respectively. Experiment in a simulated driving domain demonstrates the advantage of using scale feedback over choice feedback. A user study with a robot performing a shelving task is also conducted to compare learning from scale-based feedback vs choice-based feedback with human inputs. The user study shows that, while scale feedback enables faster learning than choice feedback, users indicated that choice feedback is easier to use.   

strengths_and_weaknesses: Learning from human feedback is an important research topic that is relevant to many research areas in robot learning. Finding new modalities that a human could provide feedback enriches the ways users could communicate with robots. This paper proposes a novel type of human feedback that an agent could learn from and demonstrate its advantage over existing approaches. The authors also develop a probabilistic model to account for the noisiness in human feedback. The paper is well-written with clarity and the experimental results clearly shows the proposed method’s advantage over the choice-based baseline. 


For studying methods that interact with human users, it is very important to also control/measure the human user’s experience. While the authors did a very good job at asking how the users would rank the easiness of the two approaches, it would be useful to also see some objective measures such as how much time it actually took for the user to answer these preference questions on average.
One other small critique I have about the user study design is that the robot’s trajectories are all pre-computed, which is ok for the purpose of the study but does not apply to real applications. It would be nice if the authors could also comment on how to address this limitation if this method were to be deployed on real systems.

summary_of_recommendation: I highly recommend the acceptance of this paper. This paper proposes to learn from a novel type of human feedback with active learning algorithms that improves efficiency over existing methods. The writing and presentation of the work is well-executed.  
issues: The author should also comment on practical limitations of the method, and potential ways of extending this method to real-time interactive learning systems.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: Update: I sincerely appreciate the authors' thorough and thoughtful responses as well as updates to the research/paper itself. I have updated my review accordingly.

======
This paper presents a framework for learning user reward functions for humanoid robotic tasks (and possibly other more general robotic applications) by using scale feedback in lieu of binary or ternary choices. The paper presents the framework in terms of a noiseless user model as well as a noisy user model, and it also provides theoretical results on error bounds for some of the framework. The paper presents an algorithm for query generation, and then evaluates the efficacy of the algorithm relative to several baselines, across two studies.  The paper is clearly written and structured well. 


strengths_and_weaknesses: The paper focuses on an interesting problem of understanding user preferences and then being able to formally encode this information in terms of learning a reward function. The paper is structured well, the prose is clear.

My main suggestion has to do with scope, and I am not sure if this is "fair" given that I would essentially be asking the authors to change their approach entirely. I question whether a slider bar (scale response) evaluating two different trajectories is even the right way to think about evaluating/understanding/learning/encoding user preferences. I have several questions about this in my detailed comments below. Reducing everything down to a binary choice does not seem like a great idea, as the paper points out well. However, turning this choice into a sliding scale does not seem to enhance thing much, and as the users note, this does not seem to be very expressive. While the results suggest that the method does indeed converge on a more accurate reward function, I think one of the promises (and challenges) of explicitly including humans involves the now-overused terms like "explainability" and "transparency" and "trust" and things of this ilk. If the method is really just about learning rewards, then I suppose this paper does well enough. Again, I recognize that this assessment might not be fair and I am trying to be helpful here, I promise. What is the purpose of bringing the human in the loop? Can the authors make this more clear, given the relatively simple task that was used for evaluation/experimentation? To me, the big topics involving humans and automation have to do with the buzzwords I mention above, not necessarily understanding user preferences for a task that (in my opinion) is amenable to just writing down a reward function in a straightforward way.

Finally, I am not sure I agree with the conclusions about the efficacy of the approach. I might say it works in the sense that it is not worse than binary/ternary feedback, but that feels like a bit of a strawman argument.
summary_of_recommendation: Hopefully it is clear from my comments below that I am concerned about the approach and whether this results in a significant enough contribution to the robotics community. There are no major technical flaws that I can see, and like I said above, I think the paper is written well enough. I encourage the authors to think about the fact that (as they state) these problems are multi-objective, and have many qualitatively different classes of solutions. 
issues: I am not entirely clear what the first paragraph (lines 14-18) have to do with the paper's focus and contributions. Yes, robots need to trade off between goals and have competing objectives, but I believe the paper is about understanding user preferences. While these concepts are ultimately related in any interactive robotic environment, I do not understand this paragraph's purpose. 

Regarding scale feedback as a general notion, I have one general concern and question. Wouldn't it be more benefiical to compare more than 2 trajectories, as opposed to having a sliding scale between a binary choice? This latter approach still seems very simplistic and might not help a human think about all the amazing possibilities that a robot could identify. 

Line 87, delete comma after "setting. (I think)

How senstive is the system's performance to the annotations on the slider bar? There are 3 annotations in the approach; what if there were 5? And are they anchored in any type of way, such as what is typically done in polling and the like?

Can the authors provide more justification for the modeling assumptions in definition 4? In particular, why is user noise zero-mean Guassian? And why is it stationary? Why is this stationarity independent of what the actual choice psi is? For example, I imagine the choice 0 might generally be less certain than 1 or -1. Thoughts here? 

Proposition 1 is well-reasoned. However, this seems like a bit of a "strawman". Of course if you give more options to non-noisy agent, any algorithm should do better both in the average case and worst case. The authors seem to imply this with the point that adding choices removes volume from the weight set. But what happens if the human is noisy?

Line 204 states that max regret "hand minimizes" the maximum regret. Is this a typo, or a term I am unfamiliar with?

Line 238-239: missing word: "...make most progress *during* the first 10 iterations"?

Figure 5(b) is very interesting. The authors discuss this a bit in the body text, but I am not sure it is helpful. Can the authors at least propose a framework that would help (in future work) to support or invalidate H3? Is it as simple as doing more experiments? Or does the design of experiments actually need to be changed?

I do not agree with the conclusion, in line 297, that the data partially supports H5. It either leads one to reject H5, or partially reject it, but not partially support. Furthermore, what does "expressiveness" even mean? Was this explained tot he user during the questioning?

Regarding the discussion about expressiveness: returing to my earlier comment, I think the important factor to the user is the expressiveness of the *trajectory itself*. That is, do the presented trajectories really represent the tradeoffs I care about? This is why I had the earlier question about presenting multiple "classes" of trajectories. What is a user really expressing when they say they prefer one trajectory over another? There are so many different objectives (as the authors acknowledge), and collapsing things down to a single dimension - which a sliding bar does, no matter how small epsilon becomes - severely limits the user's ability to tell the machine about his/her preferences. 

In short, I am skeptical about whether the slider even fundamentally allows the system to "ask the right question" to the users. Going from a binary (or 3-nary) choice to an infinite slider is fine (apparently, from the data) if one wants to accurately estimate a user's optimal policy. Therefore, who really cares about expressiveness? In the end, how much does a user's opinion of the interface matter, as long as the user can stay engaged to the point that a model can be learned?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper introduces the use of scale feedback to learn user reward functions expressed as a probabilistic model. The author empirically evaluate the advantages of their method in simulation and in a real robot experiment.
strengths_and_weaknesses: Strength
======
- the proposed method is intuitive and sound;
- the lower bound on the worst case error is a nice result;
- experimental evaluation is well described and informative.

Weaknesses
==========
- I find the technical description of the method a bit hard to follow;
summary_of_recommendation: I like how the paper motivates the approach and provides a solid theoretically sound algorithmic implementation. I am not an expert in this field of study, but I find the validity of this method quite intuitive. In fact, I find it weird that it has not been proposed in the past. I am not knowledgable of the related literature, so I will trust the opinion of other reviewers about the originality of this work. It’s also difficult for me to judge the impact of this work in the robotics community, but I definitely think it could be interesting to a group of researchers. Overall, I am convinced of the validity of this work thanks to the accurate mathematical formalization of the problem and the intuitive motivation of the proposed method, although I am not sure about its impact and originality.
issues: I would appreciate if the authors can help me understanding the impact and originality of their work. For example, why do you think this has not been proposed in the past? What are the major problems that other researchers may have encountered?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: I have updated my estimate of the technical quality from fair to good in response to further user studies and more detailed analysis of the data. Overall, I have upgraded my judgment from Strong reject to weak accept in response to this further evidence. I urge the authors to perform further experiments in order to confirm or reject their understanding of the discrepancy between log-likelihood and user ratings, and I personally still find it more likely that there is some problem in the log-likelihood evaluation. However, the collection of evidence is interesting enough that it ought to be published. The original review is below. 

===

Choice feedback has been successful as an alternative way to specify the desired behavior of an RL agent. The key feature of choice feedback is that it is easier to correctly produce compared to optimal demonstrations or a manually specified reward function. However, choice feedback is easy to produce in part because it is an inherently low information communication mechanism: each binary choice by definition provides at most one bit of information.

Humans can sometimes provide more information than this single bit when comparing two trajectories. For example, in prior literature there are methods which present the human with a third option to indicate that two trajectories are roughly as good as each other. This paper takes this insight to it's natural conclusion by allowing the user to specify a scalar strength indicating precisely how much they prefer one trajectory to another, including not at all. Whereas most (perfect, noiseless) choice feedback implicitly imposes a halfspace constraint on the reward function space (under known linear reward features), the paper shows that a scalar opinion reduces the set of possible reward values to a hyperplane in the reward space (reducing the total dimension of the feasible rewards by one). Humans are never actually capable of providing noiseless feedback, so the paper also develops a natural Gaussian noise model of the probability of a human reporting a certain preference strength under a given ground truth reward function, which they use to infer the true reward function in a Bayesian fashion.

Human feedback elicitation research often operates by picking a feedback modality to elicit from the human, a Bayesian model of how that feedback is produced, and then finally a criterion with which to determine the optimal query to ask at every timestep. This paper is not attempting to innovate on the latter, and so evaluates their method using simulated and real human data under two such criteria, maximizing expected information gain and minimizing the maximum regret, as well as a baseline where completely random questions are asked independent of the current belief distribution. Experiments are performed using all of these query generation methods under a binary choice feedback modality and the new scalar feedback modality in the simulated case, and a smaller subset of these experiments are performed on human data.
strengths_and_weaknesses: Strengths:
1. The idea to elicit scalar preferences from a human is a natural extension to existing successful ideas in choice feedback, and frankly I wish I had done these experiments myself.
2. The paper is clearly written with excellent figures.
3. The appendix includes a large number of supporting simulated human experiments.

Weaknesses:
1. The experimental results of the paper do not support the conclusions stated in the abstract and introduction.
2. There is an important missing experiment from the human study.
3. The practical choice of $\sigma$ is an important complication of the method is not well discussed.
summary_of_recommendation: As the paper stands the experiments can only support the conclusion that the Gaussian noise model under scale feedback is not so incorrect as to be disastrous. There is no robust evidence to suggest that scale feedback is superior to choice feedback on human data. The main issue is that no choice experiment was performed using a state of the art active query criteria (e.g. infogain). This makes it impossible to make like to like comparisons between scale and choice feedback while using the techniques either are likely to be used with. The results reported from the experiments that were performed support neither the claim that humans find scalar feedback more expressive nor the claim that the learned reward function under scalar feedback better fits their preferences on the experimental task. In fact, on the latter measure, the evidence suggests that humans will likely prefer the reward function learned under choice feedback, and they prefer choice feedback when comparing like query generation to like. As the likelihood models are different for the scalar and choice feedback, log likelihood cannot be compared between feedback types, and so I cannot see it as an indication of learning good reward functions. In order to accept this paper I would need to see the missing infogain + choice human experiment and either

a) results indicating that scalar feedback robustly outperforms choice feedback

b) if scalar feedback is not always better, a discussion of when/for what problems it is, or

c) a more frank portrayal of the results, perhaps by saying that you can get to the same (or a slightly worse) reward function in many fewer queries to the human, or whatever conclusion the results of the final experiment support.
issues: How exactly was the choice of $\sigma=0.35$ made in the user study (as mentioned on line 255)? Shove this in the appendix if you have to, but this is an important part of your experimental method. Did you try using a per-user sigma value? In my experience different users will behave wildly differently in these kinds of settings. But perhaps there simply wasn't enough data per user to determine a meaningful $\sigma$ value.

On line 287, tell me exactly what the p-value is. Given how small all of the other p values are, I'm not inclined to trust a "tending towards significance" conclusion. You can say that the study is underpowered and H4 is inconclusive as you do with H3 (for which I would also like a p-value for the hypothesis that you're exactly wrong). Alternatively, if you think the current results really do suggest there's something there that shouldn't be ignored, you can do a Bayesian analysis and report a (presumably) moderate but not overwhelming likelihood ratio, which would say what I imagine you're trying to say with "$p \approx 0.05$" but in a principled way.

The choice of experiments in the user study is very confusing, and results in a weak baseline that makes it impossible to be certain that scale queries are robustly better than soft choice queries. In the prior work on the subject (I'm mostly familiar with Biyik and Sadigh's work) the rough ranking of the efficacy of active query generation criteria is infogain > regret > random. In some sense random queries are the obvious thing to do, and serve as a baseline for all other active methods. Why then is the only soft choice query baseline in the user study using questions generated randomly, instead of one of these better criteria? Due to this choice, the only apples-to-apples comparison we can make is between Scale - Random and Soft Choice - Random, leaving completely open the possibility that soft choice might outperform scale when both are using infogain generate queries. Even if this is not the case (i.e. scale infogain queries are the best), it is interesting to know how relatively important the active reward criteria is relative to the form of query. If soft choice infogain outperforms scale random queries, then we know it is most important to use infogain queries (or possibly some other non-random criteria), and that scale queries are a robust improvement, but perhaps a minor one. It is hard to tell from the figures in the paper (I would appreciate a table of the exact final alignment, relative reward, and log likelihood values of all runs in the appendix), but this is plausibly true in the simulated experiments as well. Alternatively, we mind find that scale random outperforms soft choice infogain, and that the choice of query type is more important. As it stands, there are many fundamental questions about your method that we cannot answer due to the choice of a weak baseline.

This may just be my familiarity with the area, but Proposition 1, while informative, also seems essentially trivial. It would significantly improve the paper if there were a similar theoretical result for the noisy case, even a noisy case where you have to assume "too much" about the noise model in order to get a result.

Several plots in figures 10 and 11-13 make it appear as if choice queries are approaching the performance of scale queries. If this is the case scale queries may be much better for requiring less data, but it would be interesting to see where this saturation point is. Based on the runs where the two lines do meet, and due to your theory, it seems unlikely that choice ever surpasses scale, but this would be valuable information to have in future comparisons of the information value and human difficulty of different styles of query.

There are a couple of places in the introduction where I agree with the spirit of the claim but disagree in the specifics:

Line 51: "While demonstrations are usually most informative" Demonstrations are often not more informative than preferences. In particular, optimal demonstrations are often good along all features, and so it can be in principle impossible to disambiguate between different relative weights between pairs of features which are highly correlated in optimal trajectories. To steal the example from your video, if we only see examples of passing the beverage that are fast and don't break anything on the table, it is in principle impossible to tell if the speed or the safety is the more important factor here.

Line 58: "it also only provides one bit of information" choice queries provide AT MOST one bit of new information, they often provide much less, especially if the query being asked is essentially redundant considering prior queries. I personally find this is often the case when working with active preference elicitation algorithms. This only strengthens your point.

The following are grammatical or formatting issues that I expect can be fixed easily and are primarily for the author's use:

Line 162: Please write out all of the dot products explicitly as equalities or inequalities. At the very least, place commas as separators between the terms, the spacing alone makes it hard to tell.

Line 240: "the simulation showcase that" -> "the simulation showcases that"

Figure 5: The lines below and above the figures are distracting
review_title: null

Meta-review: Overall, the reviewers found that the paper addressed an important problem and was clearly written. The proposed method is a reasonable and intuitive solution to the problem and was supported by empirical evidence consisting of a user study. 

The paper would be improved by more discussion about the impact of the approach, for example, is using a slider to select preferences the right approach or could there be potentially better solutions? What are limitations of the approach? Furthermore, the approach was evaluated with pre-recorded robotic trajectories, and so discussion of how this might scale to more difficult settings would also help make the impact more clear. Finally, there were some concerns about the user study and the conclusions drawn in the paper, and so further evidence or discussion supporting the claims would improve the paper. 

Decision:
Thank you to the authors for taking time to address the reviewer's concerns. Overall, I already found the paper to be interesting and believe the paper will be greatly improved with the new updates and experiments.

Decision: Accept (Poster)