Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes an approach to incorporate preference feedback for meta-RL, as opposed to assuming access to dense environment rewards. This takes the form of querying the oracle for ranking two trajectories, and the method leverages ideas from information theory (berlekamp volume) to deal with noise in oracle feedback. The method is evaluated on 6 gym meta-RL benchmark tasks. 

strengths_and_weaknesses: Significance - 

-> The paper addresses an important limitation for practical application of meta-RL, i.e the assumption of dense environment rewards. Querying an oracle (eg: human user) is much more feasible. However, in the included experiments, the number of samples required for meta-training is on the order of a million, and the number of preference queries per step is around 10, putting the total number of queries required on the order of 10 million, so training with this scheme would still be impractical for deployment. It is nonetheless a step in the right direction, since dense rewards are very difficult to specify in the real world for arbitrary tasks, while ranking trajectories can still be done. Further since current approaches to meta-RL don’t commonly use preference feedback, this paper will be valuable to the community in helping to start work in this direction. 

Originality - 
 
-> The method proposed involves bringing preference based learning to meta-RL. The authors leverage ideas from information theory (Berlekamp volume) to account for a noisy oracle, and add this to standard context based meta-RL. This formulation seems novel and while the performance benefit over a simpler greedy binary search baseline is marginal, this paper might inspire subsequent work that borrows from similar ideas. 

-> The paper shows advantages of using non-parametric latents (instead of using an inference network like [1]) for the binary feedback problem, as even a random-query strategy can outperform a baseline based on [1]. 

-> The related work adequately discusses context based meta-RL and preference based learning. 

[1] : Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables (Rakelly et al.)


Clarity - 

-> The paper is clearly written, well motivated and easy to follow, and contains ample details (algorithm boxes, hyperparameter details) to aid in reproducibility. 

Quality - 

-> The preference based meta-RL formulation seems sound. While the experiments only include gym domains these are standard meta-Rl benchmarks. The authors also include ablations where increase in oracle noise leads to worse performance, as expected. 


questions: -> Can this approach actually be used to train agents with a human oracle (without using the mujoco simulator to rank trajectories?). Experiments using human feedback like in section 5.4 of [2] will help demonstrate the efficacy and practicality of the proposed approach. 

-> Does the approach scale to more complex meta-learning benchmarks, such as meta-world?


[2] : Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training (Lee et al.)
 

limitations: Limitations are adequately discussed
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper studies the setting of meta-reinforcement learning, but with the constraint that rewards are not observable at adaptation time. Rather, the algorithm must infer the task by querying a noisy oracle (e.g. a human) which indicates its preferences by ranking given pairs of trajectories. The goal is to adapt to the test task with few queries, to minimize burden on the human.

The authors draw a connection between preference-based adaptation and Rényi-Ulam’s game, an information-theoretic game in which one player tries to guess what the other player is thinking, but they can only communicate via yes/no questions over a noisy channel. Based on this, they propose to use a classical uncertainty quantification tool known as Berlekamp’s volume in order to choose which queries to propose to the oracle. Specifically, they choose the query that maximizes the worst-case uncertainty reduction.

Experimental results indicate that the use of Berlekamp’s volume provides improvement over baselines such as using greedy queries which do not consider the possibility of noise in the oracle’s responses, as well as modifications of PEARL to the preference-based adaptation setting.
strengths_and_weaknesses: Strengths: 
* The setting of adaptation without dense rewards is useful because it meta-RL more practically applicable.
* The connection between preference-based adaptation in meta-RL and Rényu-Ulam’s game is novel to my knowledge, and brings information-theoretic tools (specifically, the Berlekamp volume) to bear on the design of the algorithm. This is a useful contribution.
* The experiments demonstrate improvement due to use of the BVol-based query proposal.
* The paper is well-written, being organized, clear, and generally well-motivated.

Weaknesses:
* The paper would benefit from a bit more explanation of the formula for Berlekamp’s volume, since most readers will not be familiar with it.
* Another useful comparison would be ordinary PEARL using the dense rewards. (I guess you do not even have to re-run it; just use their published results.) This essentially serves as an (unattainable) upper bound on the performance we could expect with a PEARL-like adaptation algorithm, and the gap would show how much is lost by using the preference-based feedback.
questions: The query trajectories in Eq. (6) are selected from “the experience buffer”. Based on the fact that you have to subsample this buffer (presumably for computational purposes), I assume this buffer includes trajectories from meta-training time. Is it reasonable to assume that those trajectories will be available at adaptation time?
limitations: The authors have addressed limitations, namely that their method cannot completely eliminate the effect of a noisy oracle (which is to be expected).
I agree with the authors that the risk of negative societal impacts is negligible. If anything, the impact will be positive because it improves the ability of humans to communicate their preferences to deployed AI agents.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work considers a new few-shot meta-RL setting, where the agent does not observe rewards during its few shots on a new task, but instead submits pairs of trajectories to a preference oracle, which returns which of the two trajectories is preferred (i.e., would achieve higher returns). The idea is that querying a preference oracle may require less supervision than obtaining reward labels on these trajectories. Then, to solve this setting, this work proposes a new method for task inference, where a preference predictor is learned to imitate the preference oracle, and then the task is inferred by performing an error-tolerant form of binary search with the preference predictor: the idea is to find a task under which the preference predictor most agrees with the preference oracle. This work evaluates the proposed method on standard MuJoCo meta-RL tasks, where only the reward function changes between tasks, and finds that the proposed method outperforms vanilla PEARL (which is not designed to handle this setting). Furthermore, the experiments show that adding error tolerance to the binary search procedure is important when the oracle includes noise.
strengths_and_weaknesses: # Originality
This work considers a novel setting, where rewards are not observed in the few shot episodes of meta-RL, and instead, only preferences between pairs of trajectories are given. In addition, and as a result of this novel setting, this work offers a new connection with Renyi-Ulam's games and the notion of Berlekamp's volume, and is overall quite original. However, it is worth noting that the introduction slightly overclaims: Line 48 states that "reward-free few-shot policy adaptation" has not been explored in prior work, while [1] does reward-free few-shot policy adaptation, though in quite a different way.


# Quality
- My main concern about the quality is that this work does not obviously validate its claims about feedback efficiency. This work states in several places that it aims to design an algorithm that meets two goals (1) feedback efficiency, and (2) error tolerance. The first goal seems particularly important, since that is a key motivation behind the setting: if it actually requires more supervision than reward labels to provide, then it's unclear why we should consider a preference-based setting. However, the experiments neither test whether the proposed method provides feedback efficiency compared to other methods in this setting, nor whether the preference-based setting requires less supervision than a reward-based setting. I believe that both are critical.
- Another concern that I have with the experiments is that the experiments consider the case where $K_E$ is correctly specified. However, in practice, it seems unlikely that the noise level of the black-box preference oracle is known a priori. It therefore also seems important to investigate how performance changes if $K_E$ does not match the true noise level.
- A final concern with the experiments is the set of baselines. Since this work considers a new setting, it is natural that many prior works do not directly apply to the setting, and therefore finding appropriate baselines is tricky. However, while PEARL is an important meta-RL approach, there is a rich set of other meta-RL methods (e.g., RL^2 [2], VariBAD [3]), and it would be helpful to understand whether other existing meta-RL methods also work well in this setting. I note that it's unclear to me how to directly adapt these methods to the new setting in this work, so I do not hold this against this work, but it would also be nice to compare with other meta-RL algorithms beyond PEARL.

# Clarity
This work is generally understandable, although the presentation is confusing in some places and requires reading ahead and re-reading. I've noted a few areas below:
- Lines 163-177 discuss how the preference predictor helps transform the setting to Renyi-Ulam's Game by transforming binary preferences into subset inclusion. However, how the preference predictor achieves this is not clear until later in the binary search discussion. In the introduction to the preference predictor, it's only clear that the preference predictor is learned to match the black-box preference oracle.
- In addition, while the connection to Renyi-Ulam's Game is interesting, introducing it while introducing the main ideas of the algorithm makes it more challenging to understand what's going on. Perhaps first introducing the main ideas of the algorithm, and later showing how they connect to Renyi-Ulam's Game would be easier to understand.
- The notation can be quite difficult to parse in some places, with many superscripts. It would be nice if this could be simplified in some areas, if possible.

# Significance
- My main concern with this work's significance is how practically relevant the setting is. As I alluded to in my section on quality, it's unclear to me that the preference-based setting indeed requires less supervision than providing reward labels. In fact, in the domains included in the experiments, it seems like providing reward labels requires the same amount of supervision. Computing the binary preference seems to require the same computations as the reward labels, e.g., computing the distance from the goal, direction, or desired velocity. I would appreciate if the authors could comment on settings where it is clear that preference-based meta-RL is preferably to the standard setting.

[1] Explore then Execute: Adapting without Rewards via Factorized Meta-Reinforcement Learning. Evan Zheran Liu, Aditi Raghunathan, Percy Liang, Chelsea Finn. https://openreview.net/pdf?id=La1QuucFt8-

[2] RL2: Fast Reinforcement Learning via Slow Reinforcement Learning. Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, Pieter Abbeel. https://arxiv.org/abs/1611.02779

[3] VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning. Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, Shimon Whiteson. https://openreview.net/forum?id=Hkl9JlBYvr
questions: I've condensed the main questions in order of most important to least important I have from my review above here:
- When is preference-based meta-RL practically better than the standard meta-RL setting (and vice versa)? How does the amount of supervision compare between preference-based meta-RL and standard meta-RL?
- How do the various methods compare in terms of feedback efficiency? How does performance change as a factor of $K$?
- Are there reasonable ways to compare with other existing meta-RL algorithms?
- What happens when $K_E$ is mis-specified?
limitations: This work could benefit from more discussion about the limitations of the proposed new setting. Additionally, this work restricts itself to the setting where only the rewards change between tasks, while existing meta-RL works often also consider changing dynamics. It would be useful to see some discussion about whether this restriction is fundamental to the proposed method or setting, or if it can be relaxed in some form.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work studies meta-RL with preference-based reward learning at meta-test time. The authors interpret this problem as classical problem formulation called Rényi-Ulam’s game and design a query selection method based on the technique of Berlekamp’s volume. The authors verified the effectiveness of proposed methods on standard RL benchmarks.
strengths_and_weaknesses: # Strength

* Interesting research problem: Meta-RL with preference-based learning is a very interesting research problem, which can be useful for utilizing RL in challenging domains. 

* Clear writing: the paper is well-written and easy to follow. 

* Interesting connection to Rényi-Ulam’s game & technique of Berlekamp’s volume to generate queries.

# Weakness

* Lack of real human experiments

* Lack of comparison with several baselines

# Overall

The paper is well-written and the proposed method sounds reasonable. However, more experiments are required to make this work more interesting. My recommendation is "boardline accept".
questions: * There are various uncertainty-based sampling for improving preference-based RL. For example, one can utilize the uncertainty from reward ensembles. It would be nice if the authors could include more baselines in the experiments.

* Even though experiments on locomotion tasks are interesting, it would also be nice to include robotics tasks. I think Meta-world (https://github.com/meta-world/meta-world.github.io) can be a good candidate

* Som quantitative results which visualize the selected queries from the proposed method and baseline can be interesting to understand the effects of ANOLE.

* Other noisy feedback. In addition to iid noise (flipping with the probability of epsilon), experiments on other noisy feedback can be interesting. For example, authors can consider the Boltzman preference model. 
limitations: I'd like to recommend including real-human experiments. Because real humans can show various irrationality and have an inductive bias in decision making, the trends on real humans can be different from synthetic preferences. Therefore, it would be nice if the authors could include real human experiments.
review_title: null

Meta-review: metareview: **Strengths**: This paper introduces an interesting new problem setting (meta-RL for preference-based adaptation) that is of practical relevance, along with a sensible new approach that makes progress on addressing the problem. 

**Weaknesses**: After the author discussion period, there are two remaining concerns:
* Additional experiments, particularly human experiments and more complex tasks.
* Further discussion / clarification / support for when preference-based feedback is preferable to other forms of supervision, e.g., sparse rewards.

Overall, the reviewers and AC agree that this paper makes a worthy contribution to NeurIPS, despite the weaknesses. Nonetheless, we expect that human experiments in particular would help with both of the weaknesses and increase the impact of the paper, so we especially encourage the authors to work on such experiments before the camera ready version.
confidence: Certain
Decision: Accept