Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper addresses the problem of learning end-to-end reward functions from preferences. Rather than rely on hand-defined feature representations, the proposed method learns a reward function directly from high-dimensional input. To avoid requiring excessive training data, the proposed method leverages meta-learning to learn over multiple tasks and quickly adapt to a new one using a tractable number of preference queries.

strengths_and_weaknesses: Strengths:
+ The approach is novel and described clearly. 
+ The results back up the intended claims of the paper: the proposed method results in significantly (20x) fewer queries than current baselines.
+ The proposed method is impactful in making meta-learning approaches more tractable for interactive robot systems by making use of a well-studied interaction type (i.e., preferences).

Weaknesses:
- There is very little information provided about the user study that aims to validate whether this system is practical. For example, how many queries were posed to participants? How many participants were there? What was their background and experience with robots? How much time did the interactions take? What subjective measures were collected about how "practical" the interaction was? How were preferences presented to users? I understand that there may not be room for all of this information in the main paper, but it should be at least present in the appendix.

- Figures 2 and 3 present compelling results about the system's ability to converge during training. However, these graphs would be more interesting with a different x-axis. Rather than report performance w.r.t. # of update steps (with a fixed number of preference queries), it would be more interesting to see performance w.r.t. # of preference queries (with a fixed number of update steps). This would provide more granular information about how many queries are really needed by the system, and how much "faster" it learns compared to baseline (where faster is measured w.r.t. user engagement, rather than convergence speed).

- The network architecture also appears to be missing from the main paper and the appendix. 

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall, this is an interesting paper that presents techniques that will be useful to other researchers in interactive robot learning. The paper would be made even more impactful by adjusting the graphs to use a different x-axis (highlighting the relationship between # of queries and performance) and including necessary details on the approach and user study.

issues: I would love to see alternative graphs for Figures 2 and 3 that demonstrate the relationship between # of queries and performance, if you have them available. If this is not possible, any other discussion about this relationship would be useful.

Please do list the requested details about the technical approach and user study design.

Additional clarification:
"For each environment, we reduce the total feedback budget by a factor of 20 in comparison to the maximum value used in PEBBLE, but keep all other hyper-parameters the same."
Does this mean that the numbers in the Figure 2 graph titles are the number of queries used by the proposed method, but not PEBBLE?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The overall idea of the paper is:
1. Use an existing dataset to meta learn a reward function that can quickly adapt to a few comparison tuples $(\sigma_1, \sigma_2, y)$. The reward function is used in the preference prediction, and the underlying meta learning method is MAML.
2. Given a new task, as well as a few comparison tuples, adapt the reward function, then perform RL. Specifically, the comparison tuples are provided every $K$ iterations, where the most "uncertain" segments are selected as queries for human.
strengths_and_weaknesses: ***Strengths***:
1. To my knowledge, the overall idea is novel and straightforward: meta-learn the preference is not only reasonable, but potentially important for applying preference learning to real-world problem.
2. The "active learning" part also makes sense to me.
3. The overall presentation is clear and easy to follow.

***Weaknesses***:
1. It might be good to see an ablation over $K$ or essentially how the model performs with different shots. 
2. It will also be good to see the difference of picking the $(\sigma_1, \sigma_2)$ that maximizes $std(P(\sigma_1 > \sigma_2))$ versus for instance randomly selecting two $\sigma$s. Basically how important it is to have this particular selecting scheme.

robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I think overall adding the meta-learning part to preference learning is a novel and important step. This is the main reason I recommend acceptance of this paper.
issues: Please clarify the points I mentioned in the weakness section or provide further empirical evidence.
In general, I think adding more ablations on important design choices of the algorithm can much improve the paper. Minor point: Fig.2 will look better if its resolution is higher.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper introduces a few-shot preference learning framework via meta learning. A task in this setting is represented as solving for the optimal policy of an MDP with known transition but unknown reward function. 

To learn the reward function, the model assumes access to a collection of preference indications over pair of state-reward trajectories. Each indication is 0/1 signaling whether a segment of state-reward trajectories is preferable to another. The reward function can then be learned via optimizing Eq. (1) which is the cross-entropy objective corresponding to the Bradley-Terry model of paired comparison.  

To combat the scarcity of data in new task, the paper assumes access to preference data of prior (similar) tasks. A direct application of MAML on the above cross-entropy objective function thus enables learning a base parameterization for the reward function, which can be fine-tuned to new tasks with little preference data. 
strengths_and_weaknesses: Strength:

The paper is generally well-written. The technical exposition is easy to follow.
Clear pseudocode of the proposed algorithm is also included which is nice.
Experiments with both simulation & hardware are presented, which has a strong robotic focus

Weakness:

The paper appears to be a direct application of MAML on an existing preference learning model but it is not quite clear what are the non-trivialities in doing so?

From a learning perspective, MAML is a model-agnostic recipe that can be applied to any loss functions that are optimizable via gradient descent/ascent so suppose that preference data is available for a range of relevant tasks, I tend to think that if there is nothing else, then the application here is straight-forward. 

The extensive experiment of course requires significant effort but it does not seem to add much insights to the literature as the empirical questions it addressed are pretty much the same set of questions that were raised and reasonably addressed in MAML (as a general principle to pre-training) and its follow-up variants with various applications (robotic applications included).

I do notice an interesting point (4.2) about few-shot learning with human feedback. This might be a new aspect that has not been explored much but I do not see how this is connected to the technical formulation. In fact, the narrative in 4.2 is mostly about the potential noisiness of human feedback and engineering feature to allow human to skip on hard queries. 

It might be more interesting to develop learning policies that seek human feedback in an optimal way, e.g. how to structure queries that are less ambiguous to human & given limited interaction budget, what are the most informative queries; and whether such proactive query strategies would reduce the sample complexity & etc.         

--

These are my current understanding of this paper (from a learning perspective) but please do correct me if I miss anything important. I will be happy to discuss.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper seems to be a direct application of MAML on an existing preference learning model. From what was presented in the main text, I unfortunately do not see much new insights (as MAML is a generic model-agnostic recipe for pre-training). 

The application with real robotic scenario is nice & interesting but on its own, it is only a re-confirmation to the broad applicability of MAML with no new insight. With this, I believe this paper does not quite meet the novelty bar for publication.
issues: I think the main issue with this paper is that it has not highlighted the non-trivialities that require new learning capabilities to be innovated to enable the application of MAML. It would be good if the authors can focus on this while preparing the rebuttal.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper studies preference-based RL in a setting where the agent is also given to an offline dataset of preference/reward data from other tasks. The main idea is to use the offline dataset to meta-learn the preference model, which is then adapted on the target task. The motivation is that such meta-learning should significantly decrease the number of human queries required to solve the target task. Empirically, the proposed method outperforms 2 baselines (vanilla finetuning, no access to offline data) on 6/6 tasks. Additional experiments show that the method can work from real human feedback, and that the learned policy can be adapted to work on a physical robot.
strengths_and_weaknesses: Strengths
* The paper is generally well written.
* To the best of my knowledge, the proposed method is novel, and seems to address an important limitation with preference-based learning.
* The results show strong gains relative to the prior method (PEBBLE).

Weaknesses
* My main concern is with the central premise of the paper: that preference-based RL is an effective way to learn tasks. While there is a large and growing body of work that studies this problem setting, I am unaware of convincing prior work that shows that it is easier/cheaper for humans to provide preferences than other sorts of supervision. For example, in the DM Control tasks described in L251, specifying a goal (or, equivalently, reward function) requires just 2 numbers. I would expect that specifying the goal location wouldn't take more than a few seconds, of similar complexity to specifying a preference comparison. Similarly, doing back-of-the-envelope calculations for the results in Fig 2: The tasks can all be described by a hand + object goal state (3 + 3 dimensions, sometimes less). If we want 1/16 resolution, we need 4 bits / coordinate, for a total of 24 bits of supervision. In contrast, the proposed method uses at least 10k / 20 = 500 bits of supervision (21x).
* I found the presentation of the results in Fig 2 a bit confusing because the "20x" improvement isn't visualized anywhere. If possible, it'd be great to make a plot where one axis directly shows the 20x improvement (e.g., plot reward vs number of queries)
* The real-world results are a bit lackluster because, if I understand correctly, they aren't actually obtained by querying human users with real robot images but rather are obtained through sim2real. While this isn't an issue for most robotics papers, it does seems like a missed opportunity for a paper studying how robots should learn from humans in the real world.
* A number of minor writing comments (e.g., claims without citations attached). See below.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall, this paper is strong in many respects: it presents a novel method, is well written, and shows compelling empirical results. The paper does a great job delivering on the stated title ("few-shot preference learning for human-in-the-loop RL"). My main concern with the paper is that I'm not convinced that preference based learning is an effective method for soliciting human feedback. If the task can be described as reaching a simple goal state, it seems easier to ask the human to provide that goal state than to provide O(100s-1000s) preferences. While this paper doesn't purport to show that preferences are more efficient to provide than reward functions or goal states, I think that studying this question is critical for demonstrating the *relevance* of work; without this, it seems like this paper is a great contribution to an irrelevant problem. Concretely, there are a number of ways the paper might study the effectiveness of different sorts of supervision. One easy method would be to quantify how many seconds it takes a human to provide a goal state versus a preference, assuming that a reasonable UI is used for both. If the paper can show that preference-based learning is more effective than alternatives (e.g., manually-specifying the reward function or goal state), I would advocate for accepting the paper.

------------------------
**Update after rebuttal** -- The discussion period clarified and resolved most of my concerns about the paper. The additional experiments seem to provide good evidence for the claim that the proposed method can tackle tasks that are hard to define in terms of rewards or success states. I now vote for accepting the paper.
issues: Larger issues:
* Many of the claims in the introduction require evidence or citations (see below)
* L160 - L164 -- I found this a bit confusing because it seems like providing a meta-learning dataset of preferences would be very, very expensive. Also, this seems a bit disingenuous, since none of the experiments actually involve meta-learning on real human preference data.
* L195 "paradigm shift" -- This seems to be overclaiming a bit.
* L249 -- L262 -- Why is a different environment used for these environments. This seems a bit suspicious, especially since the new DMC environments appear easier.
* L264 -- L268 -- Do humans ever actually provide preference feedback on real robot experience? The "sim2real" note seems to imply otherwise. This seems important to clarify.


Writing comments
* L12 "Contrary to ..." -- I found this explanation very helpful!
* "20X" --> "$20\times$"
* "Deep Reinforcement Learning" --> "deep reinforcement learning"
* "Markov Decision Process" --> "Markov decision process"
* "inordinate", "galvanized", "unduly" -- awkward word choice
* L28 -- I found the discussion about door opening with sparse rewards a bit disingenuous, as many prior works can learn such tasks with sparse rewards.
* L34 "aim to capture" -- Cite.
* L37 "their environment" -- Cite.
* L41 "learned reward functions are ..." -- Cite. (This claim seems surprising to me)
* L46 "seems hopeless" -- Cite, or provide evidence.
* L48 "... or linear reward models" -- Cite.
* L49 "necessitate expressive reward models." -- Cite.
* L53 "is far to" --> "is far too"
* L83 broken reference
* L86 "they are harder for users to provide" -- Cite.
* Fig 1 "it" -- Ambiguous pronoun reference
* L90 "These assumptions ... " -- Cite.
* L107 "classification" -- Is the implicit assumption here that reward learning is equivalent to classification?
* L119 -- What is $\mathcal{P}$?
* L123 "only that it ..." -- At this point, I was confused about what the prior data actually contained. Was it rewards, preferences, demos? (This is answered a few paragraphs later)
* L127 "multiple tasks" -- I was confused by this, as Fig 1 only seems to show adaptation to a single task.
* L129 -- The placement of $\pi_{new}$ makes it seem like $\pi$ is a task, not a policy.
* L133 "[50]" -- Does this 2016 paper really discuss meta-RL?
* L133 "model reward" --> "model the reward"
* L135 -- Why use trajectory segments instead of individual transitions?
* L177 "prior works" --> "prior work" (or, provide more citations)
* L210 "difficult ... as they are sequential" -- This seems a bit disingenuous, as most of these tasks can be solved in 10 - 20 steps.
* L261 "prior guides the agents" -- It'd be great to provide evidence for this claim.
* L445 -- L452 -- These citations have the wrong indent.
review_title: null

Meta-review: metareview: There is consistent agreement from all 4 reviewers that this is a well-motivated and well-written paper with contributions that could provide clear impact in robot learning. The proposed meta-learning approach is novel and straightforward with experimental validation both in simulation and on real hardware.
confidence: 5: The area chair is absolutely certain
Decision: Accept (Poster)