Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper introduces the meta-reward-net (MRN) method for preferenced-based reinforcement learning (PbRL). MRN learns a reward from human preferences over trajectory pairs by combining a supervised preference-based loss with a bi-level optimization loss that optimizes the reward based on the Q-function's ability to reflect human preferences. Experiments show that MRN outperforms state-of-the-art baselines in PbRL. Furthermore, MRN improves the efficiency of using feedback labels. Lastly, the paper analyzes the source MRN's gains as coming from a more accurate Q-function. 
strengths_and_weaknesses: Strengths:
* The methodology is novel and well motived for the problem. Updating the reward based on the Q-function's performance is a sensible way to provide more reward supervision as this will update the reward to be better suited for the task.
* MRN has the novel insight of introducing bi-level optimization into PbRL. Bi-level optimization for PbRL could open future research into using other metrics in the outer loss.
* MRN consistently outperforms state-of-the-art prior work in all the considered benchmarks. Furthermore, the benchmarks are standard in the field and a fair assessment of MRN and baselines.
* Figure 4 in Section 5.3 demonstrates that MRN is preferred over baselines when less feedback is available.
* Analysis of the Q-function accuracy in Section 5.3 helps analyze the source of performance gains. MRN learns a more accurate Q-function, which enables MRN to perform better with less feedback than baselines.
* The paper is well written and clear. 

Weaknesses:
* Can MRN outperform baselines even with more feedback? In Figure 4 h,d, where methods receive 2k feedback labels, all methods perform similarly. Or can MRN help PbRL scale to more complex tasks better than baselines, even with more feedback? Since the amount of feedback greatly impacts performance, how was the amount of feedback selected for the experiments in Figures 2 and 3?
* There is still a large gap between MRN and the performance of the expert with the ground truth reward. What limitations of MRN, and PbRL in general, prevent it from reaching the performance of the expert?

questions: * The paper only considers a single pseudo update of the Q-function when computing the outer-loop loss. Did the work consider multiple psuedo updates? This could result in greater efficiency due to more accurate meta-gradient updates.
* What is the impact of the weighting of the two losses in MRN? In MRN, is the bi-level loss and the auxiliarly loss weighted the same? While, Section D analyzes different update frequencies, is the magnitude of these losses also important? 
limitations: The paper analyzes the limitations of MRN through the feedback comparison experiments in Section 5.3. These results show that MRN is limited to working best relative to baselines when there is little available avaiable feedback. However, a more explicit discussion of the limitations of the approach would be valuable.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper propose MRN, a feedback-efficient PbRL framework that incorporates bi-level optimization for both reward and policy learning. The key idea is to incorporate a new auxiliary task into standard PbRL framework, through utilizing the performance of the Q-function to predict preference label. Multiple experiments are conduct to verify MRN's superiority over various PbRL baselines in terms of high feedback efficiency and cumulative return.
strengths_and_weaknesses: Originality:
Although MRN is the first for PbRL, its key ideas have been proposed and widely used in computer vision. 

Quality：
The central idea of this paper is straightforward. I have two key doubts as follows。（1）Is it reasonable to utilize $Q_{\theta}(s_{0}^{0}, a_{0}^{0})$ to represent the return of the segment $\sigma^{0}$ in Eq. (6)? $Q_{\theta}(s_{0}^{0}, a_{0}^{0})$ represents the cumulative return of continuing executing $\pi$ after selecting action $a_0^0$ at state $s_0^0$. However, the behavior performed by $\pi$ may not be consistent with segment $\sigma^{0}$ at subsequent state $s_{t}^{0}$, where $t>0$. (2) Although the experiment is relatively sufficient, there is still a lack of rational explanation, why using the Q value to give the pseudo-label will bring about the high feedback efficiency?

Clarity: 
The main content is clear, but there is a lack of rational explanation for some key points. For example, it is unclear why using the Q value to give the pseudo-label will bring about the high feedback efficiency?

Significance:
The work is significant for improving feedback efficiency in PbRL, but I doubt the soundness of the main idea.
questions: （1）Is it reasonable to utilize $Q_{\theta}(s_{0}^{0}, a_{0}^{0})$ to represent the return of the segment $\sigma^{0}$ in Eq. (6)? $Q_{\theta}(s_{0}^{0}, a_{0}^{0})$ represents the cumulative return of continuing executing $\pi$ after selecting action $a_0$ at state $s_0$. However, the behavior performed by $\pi$ may not be consistent with segment $\sigma^{0}$ at subsequent state $s_{t}^{0}, where $t>0$. 
(2) Although the experiment is relatively sufficient, there is still a lack of rational explanation, why using the Q value to give the pseudo-label will bring about the high feedback efficiency?
(3) Why MRN performs far less than SAC on almost all experiments？
(4) Why does MRN perform only marginally better than other baselines in the DMControl environment?
limitations: This paper lacks to address the limitations of their work. （1）Since reward learning and policy learning are performed iteratively, it is not clear whether the convergence of the algorithm can be guaranteed. （2）The extent to which the performance of reward learning depends on the quality of human feedback.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a novel algorithm called MRN for preference-based reinforcement learning, which models the preference order directly from the Q-value function instead of per-step rewards.

To realize this framework, this paper develops a meta-gradient-based method to learn reward functions from the supervision upon Q-values.

The experiment results show that MRN significantly outperforms baselines in terms of both sample efficiency and final policy quality.
strengths_and_weaknesses: The paper is well organized. The notations are clear and intuitive

The proposed algorithm is novel and the experiment results are solid.
questions: 1. In section 5.3, what is the ground-truth Q-value? The policy preference is defined by binary comparison signals. How can the scale of $\hat r_\psi$ match the scale of unknown ground-truth rewards?

2. The introduction of bi-level optimization to preference-based RL is novel. However, the related work section can benefit from prior works on meta-gradient-based reward learning [1-2].

[1] Zhang et al. "On Learning Intrinsic Rewards for Policy Gradient Methods" NeurIPS 2018.

[2] Zhang et al. "What can learned intrinsic rewards capture?" ICML 2020.
limitations: No potential negative societal impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work introduces a meta-learning based interpretation of preference-based reinforcement learning. The proposed method MRN utilises bi-level optimisation to learn reward functions using expert preferences, a method introduced in [23]. The novelty of this work is in the implicit conditioning of the reward learning process on the values obtained from the Q function. The work is evaluated thoroughly in simulation and compared to reasonable baselines. The discussed results show the faster and more efficient (in terms of required expert labels) learning process of MRN. Overall, my opinion of this work is generally positive. 

However, I do have a few concerns and suggestions. The strongest reservations I have are that there is **no sufficient motivation** for using the proposed approach over alternative meta-learning based reward learning solutions. Furthermore, at this point it seems like the proposed solution has a **fairly moderate impact** (more details below). I believe that addressing these concerns and the rest of my suggestions listed below can make this work stronger and can therefore change my score.
strengths_and_weaknesses: ### Strengths
- A successful application of bi-level optimisation in the context of preference-based RL
- An overall thorough evaluation and nice ablations
- A fairly clear description of the algorithm

### Weaknesses
- lacks motivation for the use of this approach as opposed to alternatives (e.g. IRL)
- at times lacks clarity of expression
questions: ### Additional motivation needed
- This work should draw clearer boundaries between pbRL and alternative reward learning methods and more generally alternative learning from human feedback methods. Right now, it is not very clear how and why this is different to IRL, for example, unless the reader is very familiar with [23]. This should not be the case, instead the paper should be really readable on its own and less dependent on prior work
    - Ideally, this should be done by proposing a stronger motivation for why pbRL (and more specifically meta-pbRL) as opposed to using something alternative (like IRL or meta-IRL).
    - It would be great to have a more exhaustive review of other meta-learning approaches from human feedback. A non-exhaustive list would include [i, ii, iii, iv], for example.

[i] Yu, L., Yu, T., Finn, C. and Ermon, S., 2019. Meta-inverse reinforcement learning with probabilistic context variables. NeurIPS 2019

[ii] Zhou, A., Jang, E., Kappler, D., Herzog, A., Khansari, M., Wohlhart, P., Bai, Y., Kalakrishnan, M., Levine, S. and Finn, C., 2019. Watch, try, learn: Meta-learning from demonstrations and reward. ICLR 2020

[iii] Finn, Chelsea, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. "One-shot visual imitation learning via meta-learning." CoRL 2017

[iv] Das, N., Bechtle, S., Davchev, T., Jayaraman, D., Rai, A. and Meier, F., 2020. Model-based inverse reinforcement learning from visual demonstrations. CoRL 2020

### Improving the Clarity of Expression
- The paper seems to refer to implicit conditioning on the predicted values from the Q function for the reward learning process as feedback but this is generally confusing since feedback has a very particular meaning in the context of robotics and robot learning. Specifically, **feedback refers to the mechanisms used for measuring the response over a specific variable** and this is not what is meant here. Similarly, the sentence '_[24] utilizes two kinds of feedback, initializes with imitation learning policy, and further improves the performance of the policy  with feedback from humans_' is confusing. Bootstrapping a policy does not mean the method uses demonstrations as feedback, similarly, using human feedback means actively correcting the system online, similarly to what Dagger-like methods do. Therefore, statements like this can be made a bit crisper. Similarly, other references to '_feedback_' in the paper seem equally confusing. An additional example is '_MRN outperforms prior methods in the case of little feedback_ ' - learning from a few labels does not mean there is a little feedback. I'd strongly encourage the authors to be clearer in the proposed expressions as to improve clarity.
- The authors mention that their '_intuition is that the optimization of the reward function needs both supervised learning and the feedback from the Q-function_' but the authors also conduct thorough experimentation ablating the use of those terms in appendix. Referring the readers to the relevant experiments would be great here too.
- It is overall great that the proposed work is doing better than the alternative solutions. However, it is not really clear why the performance of Preference PPO is so poor on the DMControl tasks. In particular, when comparing the reported results in Figure 3 with the original Preference-based paper [23] on those environments, it is really strange that Preference PPO does so poorly. It may be a good idea to clearly report the number of labels used to learn the rewards in the figures themselves. It took me a while to realise that there is a significant difference in the number of demos used in this work and the reported results in [23]. Adding a bit more discussion on this front can also be beneficial.
- Figure 4 is great, but it is really not clear what the upper-bound is. Therefore, it may be useful to add the achieved results with the ground truth reward using SAC. This should be easy, unless I am mistaken, those results already exist and are reported in Figures 2 and 3. Furthermore, it will be useful if the rows clearly indicated the names of the environments, similar to what is done in Figures 2 and 3.
- Conclusion summarises that the work is evaluated on robotic tasks but it should be more explicit, it is evaluated on robotic simulated tasks which is very different in practice.
- The paper also states that '_To our knowledge, we do not see any potential negative social influences of our work_' but the carbon footprint of the ran experiments and the required future runs for building on top of this (or any model-free approach) has a fairly negative impact. As a result, it may be better acknowledging this as opposed to stating that you do not see any negative social influences. Alternatively, just take out this sentence.
- The paper has a typo, namely '_we the mean squared error between_'' should be '_we use.._'
limitations: This paper does not list the existing limitations of the proposed approach. I would really like to see a comprehensive list and a discussion of the existing limitations. 

- A good example is that although it learns better from a few labels, it does not seem to solve the tasks with those few labels. This (along with the lacking discussion pro/con IRL) is a main reason for me to feel that the proposed solution has a fairly moderate impact. Perhaps a discussion on this topic would be great.
- The method also seems very sensitive to the number of meta updates. 
- It also seems (from Fig. 2) that the learning process is not very stable and that some seeds completely failed to solve the tasks.
- Additionally, it is not clear how to scale this work to learning visual observations as it may require infeasible number of labels. 

Discussing all those and potentially other limitations can be very useful.
review_title: null

Meta-review: metareview: The reviewers carefully analyzed this work and agreed that the topics investigated in this paper are important and relevant to the field. Overall, the reviewers had a generally positive impression of this paper. One reviewer argued that the paper introduces a novel technique for learning a reward function from human preferences. The reviewer acknowledged that the method outperforms SOTA baselines, that it is well-motivated, and that it introduces important insights. As the main weakness, this reviewer pointed out that the paper analyses of the method's limitations show that it works best w.r.t. baselines when there is little available feedback. They encourage the authors to include a more explicit discussion on limitations such as this. Another reviewer had a less favorable view of this work and argued that the key ideas in this paper have been proposed and widely used in computer vision. They had two main technical questions, to which the authors responded. The reviewer, however, remained concerned about whether, e.g., using Q-values could introduce bias and if the method lacked sufficient theoretical analyses. Two other reviewers had more positive views of this paper. One of them argued that the paper introduced a novel algorithm and that its experiments showed that it outperforms baselines both in sample efficiency and final policy quality. The authors responded to the technical questions made by this reviewer, and the reviewer said they were satisfied with the authors' rebuttal. Finally, a fourth reviewer also acknowledged that this method is novel and that it was thoroughly evaluated in simulation and compared to reasonable baselines, where it was shown to be faster/more efficient. This reviewer initially thought there was insufficient motivation for using this approach over alternative meta-learning techniques—in which case the paper's impact could be moderate. However, the authors addressed the reviewer's concerns/questions in detail and the reviewer seemed satisfied with the rebuttal, changing their score to Accept. Overall, thus, it seems like most reviewers were positively impressed with the quality of this work. They look forward to an updated version of the paper that addresses the suggestions mentioned in their reviews and during the discussion phase.
confidence: Less certain
Decision: Accept