Meta-Reward-Net: Implicitly Differentiable Reward Learning for Preference-based Reinforcement Learning
Authors: Runze Liu
Fengshuo Bai
Yali Du
Yaodong Yang
Venue: NeurIPS 2022 Accept (2022)
Keywords: preference-based reinforcement learning
human-in-the-loop reinforcement learning
deep reinforcement learning
bi-level optimization

Abstract: Setting up a well-designed reward function has been challenging for many reinforcement learning applications. Preference-based reinforcement learning (PbRL) provides a new framework that avoids reward engineering by leveraging human preferences (i.e., preferring apples over oranges) as the reward signal. Therefore, improving the efficacy of data usage for preference data becomes critical. In this work, we propose Meta-Reward-Net (MRN), a data-efficient PbRL framework that incorporates bi-level optimization for both reward and policy learning. The key idea of MRN is to adopt the performance of the Q-function as the learning target. Based on this, MRN learns the Q-function and the policy in the inner level while updating the reward function adaptively according to the performance of the Q-function on the preference data in the outer level. Our experiments on robotic simulated manipulation tasks and locomotion tasks demonstrate that MRN outperforms prior methods in the case of few preference labels and significantly improves data efficiency, achieving state-of-the-art in preference-based RL. Ablation studies further demonstrate that MRN learns a more accurate Q-function compared to prior work and shows obvious advantages when only a small amount of human feedback is available. The source code and videos of this project are released at https://sites.google.com/view/meta-reward-net.

Meta-Reward-Nett Implicitly Differentiable Reward Learning for Preference-based Reinforcement Learning
Runze Liu Fengshuo Bai", Yali Du Yaodong Yang 1,5.1
Institute for AI, Peking University, 2Shandong University Institute of Automation, Chinese Academy of Science "King's s College London, 'Beijing Institute for General AI
t: Corresponding to yali du@kcl ac.......... uk, yaodong. yaodong-yang@pkur edu. cn
Abstract
Setting up well-designed reward function has been challenging for many rein- forcement learning applications. Preference-basee reinforcement learning (PRRL. provides new framework that avoids reward engineering by leveraging human preferences (i.e., preferring apples over oranges) as the reward signal. Therefore, improving the efficacy of data usage for preference data becomes critical. In this work, we propose Perta-Reward-Ne (MRN), a data-efficient PbRL framework that incorporates bi-level optimization for both reward and policy learning. The key idea MRN is to adopt the performance of the Q-function as the learning target. Based this, MRN learns the Q-function and the policy in the inner level while updating the reward function adaptively according to the performance of the Q-function the preference data in the outer level. Our experiments robotic simulated manipulation tasks and locomotion tasks demonstrate that MRN outper- forms prior methods in the case of few preference labels and significantly improves data efficiency, achieving state-of-the-art in preference-based RL. Ablation studies further demonstrate that MRN learns more accurate Q-function compared prior work and shows obvious advantages when only a small amount of human feedback is available. The source code and videos of this project are released https://siten httpp::///ites.gool...... www.wiiwewettttaaeeeeeeeeeeeeeee
Introduction
In recent years, reinforcement learning has achieved great success in solving complex sequential decision-making tasks, such gaming AI [ 4 5 autonomous driving 6 7 robotic manipu- lation order dispatching [I. population biology I quantitative finance 12, automation system B
etc. For common decision making tasks, the goal of the agent is maximize the cumulative reward. However, one central challenge to reinforcement learning is how to design reward functions. On the one hand, the quality of the designed reward function largely depends the problem solver's understanding of task objective, operation logic, and related background knowledge. Even excellent engineers still need plenty of time try different methods in complex RL tasks. On the other hand, there is problem that the agent might hack the reward function. In policy learning. the agent utilizes the defect of the reward function maximize the cumulative reward instead of solving the expected task. Besides, in human-involved. scenarios, the objective of the agent is to maximize happiness of humans, making it hard specify reward function.
Work done as a research intern at Peking University.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

s, a,
Ppp
implicit derivative
L (p)) (meeets((())) 0
(Qo(s,a)
(s,(s,
cnv
dB
dvp
①
policy
policy
human preferences
T (als)
improvement
replay buffer
evaluation
a)
Figure/ 1: Framework of Peta-Reward-Nel ! Trajectories are sampled by interacting with the environment and reward labeled by Fg. (2) Transitions are sampled from the replay buffer and are relabeled by the up-to-date Fe for optimizing the policy and the Q-function. ( The performance of the Q-function on the preference data is evaluated to provide implicit derivative for reward learning.
Previous work has provided some ideas to avoid directly constructing reward functions, such as imitation learning [T]. Although imitation learning has an excellent performance some tasks, its performance is difficult to surpass human level. Preference-be based reinforcement learning is more flexible and convenient alternative method. A human expert can easily give preference to a trajectory pair of the agent, which implies the desired behaviors, that is to say, the goal of the task. In preference- based RI., the reward function is learned through preferences given by the human teacher trajectory pairs. The feedback from humans guides the agent achieve specified goals learn desired behaviors. Recent work 177 19 shows that providing sufficient feedback achieves better performance under this setting.
However, preference data queried from human experts is expensive. In reality, there is likely to be only small amount of data available. With the limitation of the amount of feedback, previous methods perform poorly or even do not work. Meanwhile, recent work on semi-supervised learning takes success in computer vision (20), pseudo labels can be adjusted according to the performance of the student and further improve student's 's performance. Inspired by this, we consider utilizing the performance from the Q-function the preference labels for reward learning via bi-level optimization. By considering the implicit differentiation in reward learning, the reward function is aware of the accuracy of the Q-function, which is beneficial for the learning of the Q-function and the policy.
In this work, we focus on the efficiency of feedback in the learning of preference-basee RL. We use bi- level optimization method for reward learning which contains two loops. In the inner loop, update the Q-function and the policy via the reward function, while optimizing the reward function according to the performance of the Q-function the preference data in the outer loop. Our experiments demonstrate that our method considerably improves feedback efficiency. Besides, the results of further evaluation show that MRN exceeds other methods by large margin when few preference labels are available and learns a more accurate Q-function.
In summary, the main contributions of our work are three-fold. First, we propose new preference- based RL algorithm learning, Second, we show that MRN substantially improves the feedback efficiency and outperforms preference-based RL baselines on variety of robotic simulated manipulation tasks from Meta- world 21 and locomotion tasks from DeepMind Control Suite (DMControl) [2 2x Last, we demonstrate that benefiting from bi-level optimization, the advantage becomes obvious compared to PEBBLE when only few preference labels provided. We also show that considering the performance of the Q-function in reward learning is beneficial for agent learning, leading to a more accurate Q Q-function and a better policy.
which utilizes bi-level optimization methods in reward
2 Related Work
Reinforcement Learning, Reinforcement learning has gradually become an effective and powerful method solve complex sequential decision-making problems. In recent years, much prior work has proven the capacity of reinforcement learning. Reinforcement learning applications include

---

Preference-based Reinforcement Learning. In standard reinforcement learning framework, a finite Markov decision process (MDP) can be presented as tuple of (S,A, R, P,7), which consists of state space S, action space A, transition function, reward function, and discount factor. P(s'|s, [s, a) represents stochastic dynamics of the environment, which is the probability of selecting action to video games [1 2 3
robot control [24, manipulation 114 151. board games 4 autonomous
driving [ and so on. In addition, there are many applications of reinforcement learning in the
fields of computer vision 25 natural language processing [26]. recommendation system 27 28 and game theory [2 30 the framework of reinforcement learning, an agent obtains data from interaction with the environment to optimize policy to maximize the expected return. In this process, the reward function plays a crucial role. Our method differs in that we do not assume that there is reward function from engineering. Instead, we use the preferences of humans to guide the agent towards desired behaviors.
Preference-based Reinforcement Learning. Prior work has successfully trained the agent to complete specific tasks or achieve goals through the feedback from the teacher. [31 provides a general learning framework of preference-basee
reinforcement learning. B2 utilizes expert demonstrations and human preference feedback, initializes with imitation learning policy, and further improves the performance of the policy. In this work, we mainly focus on feedback efficiency in the learning of
57 considered learning reward function from the most informative data to be consistent with the human preferences. Recently, several feedback-efficient preference-basee RL algorithms have been proposed. PEBBLE [17] combines unsupervised pre-training and the technique of relabeling experience to improve feedback efficiency. SURF [ learns the reward function by semi-supervised learning and data augmentation. RUNE 19 facilitates exploration via reward uncertainty to reduce the number of preference labels. However, these methods only focus on guiding reward learning through supervised loss between human preferences and estimated preferences. We have different approach in that in addition utilizing the supervised loss, we also consider the performance of the Q-function on the labeled preference dataset assist reward learning, thus beneficial for agent learning. By introducing this extra signal, MRN gets more task-related information from human preferences to improve feedback efficiency remarkably.
preference-based RI. Much previous work K5
36
Bi-level Optimization. computer vision, there are several bi-level optimization algorithms that achieve great success. Meta- Sett-Weiggtt-et 3S provides weights for samples from an unbalanced dataset in a bi-level manner, while Meta Label Correction 33 view this problem as a label correction problem. Meta Pscudo Labels 220 combines Pseudo Labels methods and bi-level optimization to generate high quality pscudo labels. In reinforcement learning, LIRPG [00 learns parametric intrinsic rewards and combines them with extrinsic rewards improve the performance of policy-gradies
based learning methods. PR2 41 and GR2 42 propose to conduct multi-level recursive reasoning for multi-agent interactions. Bi-level method has also been studied for multi-agent coordination 4%] LUR [41 learns an intrinsic reward function for each agent to achieve better cooperation in multi- agent reinforcement learning. CAIL
proposes to re-weight demonstrations of different optimality 16 proposes scalable meta-gradient framework to learn useful intrinsic reward functions capture a rich form of knowledge across over multiple episodes. Neural auto- curricula 47 learns to create an autonomous and diverse agenda 4% 49
5
in imitation learning.
for solving two-player zero-sum games. Our method uses a similar bi-level method for reward learning. To the best of our knowledge, we are the first to introduce bi-level optimization into preference-basee RL.
Meta-learning from Demonstrations. Previous research has many effects meta-leariing from demonstrations. [50) presents a new framework, PEMIRL, which enables meta-learning of rewards by leveraging unstructured demonstrations from multi-task. [5]] presents a meta-learning method for one-shot imitation, which combine meta-learning with imitation learning. Specifically, this approach enables the agent learn how learn more efficiently and makes it acquire new skills from a single demonstration. [52) proposes a meta-learning algorithm that can utilize a small amount of trial experience to learn new behaviors illustrated by a single demonstration. (53) introduces an inverse reinforcement learning algorithm. which optimizes cost functions by differentiating at the inner optimization step. Unlike prior work, our method focuses on preference data rather than demonstrations, which is usually more expensive.
3 Preliminaries
3

---

transit to s' in given state s. R(s, a) represents the reward obtained by selecting an action a in given state s. The policy {{a]s) a mapping from state space to action space. The objective of the agent is to collect trajectories from interaction with the environment to maximize the expected return.
In the general preference-based RL from 3T, there neering and a reward function estimator F should be learned to be consistent with preferences from the human expert. Specifically, segment a is sequence of states and actions which is (8t+1, d2+1, ... St44k; dg+k). Human expert provides a preference on given two segments (o o²) and y is the distribution over {0, 1). y {11 0), (0, 1), (0.5, 0.5)). Following the Bradley-Terry model [54] a preference predictor constructed by the reward function estimate F is formulated as:
reward function from reward engi-
exp E, (s).
Polo > -o'l= exp E,
(1)
8.. d)) exp E, Fa)
where oº > o denotes o is more consistent with the expectations of human experts compared with o². The reward function learning can be solved by minimizing the cross-entropy loss between predictions from preference predictors and human preferences.
((0) log Pola" o'l + v(1) log Polot > o4)] Ssupervised (v) (o0,a M) -D
We refer to this objective as supervised loss in the following sections. By optimizing the reward function using this loss, segments that are more in line with human preferences obtain higher cumulative reward.
Soft Actor-Critic. SAC [5S] is an off-policy algorithm based on the maximum entropy RL, which encourages the agent to explore the environment by acting as randomly as possible. SAC consists of soft Q-function (s(s, a) with parameters and policy To (>(als) with parameters Φ. Q-function with parameters 0 is defined as the expectation of return:
T
Qo(5c, ar) = 20
Σ
'
St = 8c. Ar = az
(3)
where 7 E [0, 1] is a discount factor.
The parameters f of soft Q-function are trained by minimizing the soft Bellman residual:
(Q E.,~8 (Qo(se, ar) - -t -V((()+))
where (st) Ea,oo QQ ar) a log (arls:) Ti (st, at, Sc+1; Te) is the transition at time step a a learnable temperature parameter that controls the item of entropy, are parameters of the target soft Q-function, and B is replay buffer. After the updating of the Q-function, policy ns is updated by minimizing the loss:
(Ø) Ber~Barn (a log (a, Qo(s(, at)
(5) By performing policy evaluation and policy improvement alternately, SAC trains an agent with excellent and stable performance. In this work, consider using SAC as our backbone reinforcement learning algorithm.
Meta-Reward-NNe
In this section, formally present Peta--Rearrd-Ne.. which includes two key components, opti- mizing the reward function based on the performance the Q-function in outer loop and learning the agent in the inner loop. In the following, first provide new perspective that the Q-function can be used to compute preference labels, then define the objective of MRN and formulate a bi-level optimization problem.
4.1 The Objective
The probability that segment " is preferred is proportional to the exponential return of it. Motivated by this, use (o ag) and Q0(s), ab) to respectively measure of the return of o0 and of since

---

the Q-value equal to the expectation of segment return. Therefore, the probability that σº is preferred is computed through the Q-function:
exp Qo( al))
Polo" > o'!
(6)
exp Qu( a + exp Qu(s), att
Given human preference labels the performance of the Q-function is measured by the cross-entropy loss between preference predictions computed by and ground-trut labels:
Lemeta (emet((((b) (o",al (0)) log Perw)la > o²) u(1) log Pacw)lo's > o'1].
(7)
where ((()) denotes the updating of 9 depends the reward provided by Pip.
The core of MRN is making reward learning be aware of the Q-function, which means that the optimization of the reward function takes the performance of current Q-function into consideration. Based on this idea, we formulate the objective using 7 as a measurement of the Q-function. The objective of MRN is to minimize the loss of Qo on labeled preference dataset and the Q-function is trained by minimizing the Bellman residual. The overall objective is formulated as:
min 4.0
Lmeaa (O()))
(8)
((+) arg min JQ. v))
By formulating MRN as a bi-level optimization algorithm, this allows the reward function to provide rewards that beneficial for improving the performance of the Q-function, which further leads to better policy TT.
4.2 Bi-level Optimization
The objective in ( is solved by the following bi-level optimization algorithm: A is optimized by the reward estimation from Fgz in inner loop while updated according to the performance of the Q-function on the labeled data in outer loop.
Building Connection between and th. To improve the performance of Q- function, we formulate the outer loop updating as optimizing the loss of 0 on the labeled preference data with respect vi However, we can directly optimize this objective since the objective is independent of sb. So the first step is to build a connection between 0 and vb. Sample a mini- batch state-action pairs from the replay buffer and use them query current reward function with parameters (R) to obtain reward estimation Pip (s), where denotes the current iteration step. Then becomes:
Pseudo Updating:
(Q = E,,-B (Qo a.) Fu (st, a.) - - ()
(9)
At each bi-level optimization step, first pseudo update the parameters of the Q-function. Pseudo updating means that we do not directly perform the updating on the Q-function, but update the parameters of copy of the current Q-function by minimizing O with learning rate
((k)
g(k) a VoJo(0, Mppcs
(10)
where (kk) denotes the updated copy of ((*) By performing (O), the connection between ((k) and 2)(0)) is built.
Outer Loop: Optimizing v to Improve the Performance of Qo on Labeled Data. After building connections through pseudo updating, the copy of Q-function with parameters ((k) is tested on labeled preference data. The predicted preference label (x(x) is computed by the Q-function using ( where denotes segment pair (o σ¹) We use implicit differentiation in our method. The objective of outer loop is formulated in 7 and the implicit derivative of the outer loss with respect to v is calculated using the chain rule:
Imela
(K)
Cmmer (.mmmaa((()))) (0(1)) ((v)
Ve
h Vur azi 1900
(11)
where 20 V (0(+)) VoQ(si, az; g(() and full derivation can be found in Appendix
B
formulates the implicit derivative from the fuucciio and this can be done easily using automatic
5

---

differentiation in Pytorch [56]. Since pseudo updating is performed and the connection between 0 and d is built, updated improve the performance of the Q-function by minimizing the cross-entropy between preference labels from Qo and ground-truth labels:
yitk = =b((
Imeis
(k)
(12)
(00
where B is the learning rate of the outer loop.
Inner Loop: Optimizing 0 and Φ. In the inner loop, the objective is the same as and 5 in the training of SAC. To calculate the new reward (s at), use the same batch of state-action pairs in pseudo updating to query the updated reward function. However, do not need the connection in the inner loop since the connection is used for outer level optimization. With newly obtained reward estimation, we update Q-function Qo by minimizing with learning rate a:
g(k+1) = g(k) V Jo))(o))
(13)
and update policy n by minimizing S with learning rate
(k+1) = ((*)
Va.Ja (0))(())
(14)
Auxiliary Loss. In addition to optimizing the loss in the outer loop, the reward function is augmented with a supervised loss generally used in preference-based RL, which is formulated in 2 Our intuition is that the optimization of the reward function needs both supervised learning and awareness of the performance of the O-function, and neither of them can be removed. On the one hand, supervised loss is necessary for it ensures the reward estimation is consistent with human preferences. On the other hand, the outer loss is beneficial because it improves the performance of the Q-function, leading more accurate Q-function and finally better policy. Additional experiments are conducted to discuss the relation between the two losses Appendix
The full procedure of our method is detailed in Appendix A Before reward learning, we first initialize the policy and replay buffer with unsupervised exploration, which is proposed in PEBBLE [7 and can be found in Appendix D We use off off-policy RL algorithm SAC to collect transitions and save them in the replay buffer. The Q-function and the policy is optimized each training step.
4.3 Algorithm Convergence
For theoretical analysis, we provide convergence guarantee of Meta-Rewardd-ee. Theorem demon- strates the convergence rate of the outer loss, while shows the convergence of the inner loss. The detailed proofs of two theorems are presented in Appendix C
Theorem Assume the outer loss Lumus Lipschitz smooth with constant L, and the gradient of Emeta and JQ is bounded by p. Let Fe be twice differential, with its gradient and Hessian respectively bounded by 8 and B. For some ei > 0, suppose the learning rate of the inner updating aa min 1, }, where < T. For some 02 > O, suppose the learning rute of the outer updating
32 = m- ( 3, where VT 2 L. E k= 3n S oo and Σx=1 $2 S xc. Mela-Reward-Ne can
achieve:
V Emeta (0 (b6))))
min E
S
(15)
1SKST
VT
Theorem 2. Under the conditions in Theorem 7 Meta-Rewarde Reward-Net can achieve:
voJo(00); ( [|] = O.
lim E R-200
(16)
Experiments
In this section, our method evaluated on a variety of robotic simulated manipulation tasks from
Meta-world [21] and locomotion tasks from DeepMind Control Suite (DMControl) [2 23 The tasks used in our experiments are shown in Appendix which are the same as the tasks used in SURF.

---

5.1
Setup
Baselines. Reward-basse SAC and three state-of-the-ard preference-based RL algorithms are used for comparison:
SAC is considered as the upper bound algorithm since the agent is provided with ground-truth reward function, which is not the case in preference-based RI.. SAC is evaluated in our experiments because is the backbone RI. algorithm PEBBLE.
SAC 551
the method is re-implementaaion using PPO 57 It uses an ensemble of reward functions and disagreement sampling for querying.
Preference PPO :
PEBBLE 17: the method is preference-based RI, method with unsupervised exploration and reward relabeling.
SURF [8]
the method combines temporal data augmentation and pseudo labels in semi- supervised learning, which is the state-of-the-art algorithm preference-based RL.
Rewa-Reeard--Net (MRN): the proposed method, which is aware the performance of the Q-function in reward learning through bi-level optimization.
Implementation Details. For all methods, we use unsupervised exploration proposed in PEB- BLE [17]. For the sampling of queries, disagreement- based sampling is used for all preference- RL methods, following the setting in B1 An ensemble of three reward functions is used and the reward output is computed by averaging output of three reward functions. To systematically evaluate the performance and speed the training process, following the setting in PEBBLE [17] and SURF 18, we consider a script teacher that can always provide the ground-trute -ruth preference label of segment pair. Concretely, this is implemented by comparing the ground-truth return of each segment, but the reward is not accessible to the agent under the setting of preference- based RL.
For the implementation of SAC, Preference PPO and PEBBLE, we use the publicly released repository of B-Pref (58] in our experiments F In their implementation, Preference PPO re-implemented using on-policy algorithm PPO. SURF is also implemented using their released code f For SAC, PEBBLE and SURF, the hyperparameters and network architectures we use are the same them (e.g., number of network layers, learning rate, frequency of supervised reward learning). For the amount of human's 's preference feedback, we set 100 for Walker, Cheetah, Button Press and Window Open, 700 for Quadruped, 1000 for Door Open and Drawer Open, 4000 for Sweep Into, and 10000 for Hammer.
Our method is implemented by using PEBBLE as the backbone. We use bi-level updating frequency N 5000 for Cheetah, Hammer, Button Press, Drawer Open and Window Open, N 1000 for Walker, N = 3000 for Quadruped, and N 10000 for Door Open and Sweep Into.
For each task, run all algorithms independently for ten times and report the average with standard deviation. Tasks of Meta-world are measured on success rate while the tasks of DMControl are measured ground- truth episode return. The experiments
single machine with one NVIDIA RTX 2080 Ti GPU. Details on hyperparameters, network architectures can be found in Appendix H
5.2 Results
Meta- Tasks. Examples of the six continuous control tasks from Meta-worff are shown in Appendix These tasks are selected for our experiments, including robotic simulated manipulation skills of various difficulty. The details of the tasks can be found in Appendix E
Figure shows the training curves of MRN and the baselines on the Meta-world tasks. In this figure, SAC achieves the best performance in each task by using the ground-truti reward function. Since little human feedback is provided, we observe that there gap between all preference-baste methods and the best performance, but MRN still exceeds the preference-buse based RL baselines by large margin. These results demonstrate that MRN considerably improves the performance when only few preference labels are available. We also notice that Preference PPO does not work in most
RL
http:://gitaab.ccm/lllleeeeeeee /ttps///ggthubb.com//l/lll/////
7

---

tasks. The reason is that Preference PPO works well when the amount of preference labels is large. Once the number of labels reduces, the performance will have a significant drop.
DMControl Tasks. For DMControl, three locomotion tasks in Appendix Hrre used for evaluation, including Walker, Cheetah and Quadruped. These tasks encourage the agent providing the agent with the reward that is positively correlated with agent's 's velocity. However, the agent not accessible to the ground-truth reward function, and all preference-based RL methods are provided with human preference labels. Similar to tasks of Meta-world, we only provide the baselines and MRN with few but the same amount of preference labels.
forward by
Figure 3 shows the results of five methods on DMControl tasks. SURF achieves almost the same return PEBBLE, while MRN shows obvious advantages compared with bascline methods. The results show that MRN performs well with few preference labels and improves feedback efficiency.
We remark that MRN can be regarded being aware of the performance of the Q-function through bi-level optimization based on PEBBLE. Comparing the results of MRN and PEBBLE Figure 2 and Figure B find that using bi-level optimization significantly improve performance when few preference labels are provided.
SAC (upper bound)
PrefPPO
PEBBLE
PEBBLE SURF
PEBBLE MRN
100
100
100
Success Rate (%)
Success Rate (%)
75
75
75
Success Rate
50
80
0.25
0.50
0.75
1,00
0.00
0.25
1,00
5
Environment Steps
x10"
Environment Steps
x10"
Environment Steps x10"
(a) Hammer (focdback=10000)
(b) Door Open (fcedback=1000)
(c) Button Press (feedback=100)
100
100
Success Rate (%)
Success Rate (%)
Succees
50
50
25
25
25
0.00
0.25
0.50
0.75
1,00
0.00
0.25
0.50
0
0.75
1,00
0.00
0.25
0,50
0,75
1,00
Environment Steps x10"
Environment Steps x10"
Environment Steps x10"
(d) Sweep Into (feedhack=4000)
(c) Drawer Open (feedhack=1000)
(f) Window Open (feedhack=100)
Figure Training curves on six continuous control tasks from Meta-world The solid line and shaded regions respectively denote mean and standard deviation of success rate, across ten runs.
5.3 Ablation Study
Number of Human's Feedback. Figure 2 and show that our method outperforms Preference PPO, PEBBLE and SURF under relatively small amounts of human's 's feedback. To further analyze the performance of different algorithms with different number of preference labels. Extensive experiments are conducted on Walker and Door Open with varying amounts of feedback: { {100, 400, 1000, 2000). The results in Figure suggest that our method performs better when the number of human's 's feedback small. As the number increases, the performance of three algorithms becomes closer. Intuitively, this phenomenon could be considered that feedback information becomes more sufficient, the performance gap caused by insufficient preference labels disappears.
A Accuracy of Q-function. To compare the quality of Q-function trained by MRN and baselines, we use the mean squared error between ground-truth Q-values and the output of Q-function. Each method is evaluated the same ten trajectories by caleulating the MSE of the output of the Q-function

---

SAC (upper bound)
PrefPPO
PEBBLE
PEBBLE SURF
PEBBLE MRN
1000
1000
1000
Return
Episode Return
Episode Return
600
500
500
250
250
250
0
300
0.25
0.50
0.75
1:00
0.00
0 25
0.50
0.75
0.00
0
0.50
0.
1.00
Environment Steps x10"
Environment Steps x10"
Environment Steps x10"
(a) Walker (fecdback=100)
(b) Cheetah (feedhack=100)
(c) Quadruped (fecdback=700)
Figure 3: Training curves on three continuous control tasks from DMControl. The solid line and shaded regions respectively denote mean and standard deviation of success rate, across ten runs.
SAC (uppor bound)
PEBBLE
PEBBLE SURF
PEBBLE + MRN
1000
1000
1000
1000
750
Walker
Epicde Ketarn
Epscdd
500
800
Ep
3:00
Eprcdd
500
200
250
200
250
0.00
0.25
0.00
0.75
100
8.00
0.25
0.50
0.75
1.00
0.00
0.25
0.00
0.75
1.00
8.00
0.25
0.50
0.75
1.00
Erwinonment Steps x100
Environment
xx10
Eridronment Sieps
xx10"
Environment Steps x110
(a) feedback=100
(b) feedback=400
(c) feedback=1000
(d) feedback=2000
100
100
100
100
Door Open
Success Rate (%)
Success Rate (h)
Succets Hate
50
60
Sucossa
80
60
0.00
0.25
0.00
0.75
1.00
0.00
0.25
0.50
0.75
1000
0.00
0.25
0.60
0.75
0.00
0.25
0.50
0.75 1.00
Environment Steps x100
Environment Steps x110
Enviroment Steps x110
Environment Steps x10"
fcedback=100
foedback=400
foedback=1000
(h) fecdback=2000
Figure 4: Training curves Walker (first row) and Door Open (second row), measured by the ground-truth reward and success rate, respectively. The solid line and shaded regions respectively denote mean and standard deviation, across five runs.
and the ground-truth Q-values. We report mean and standard deviation across ten runs in Table SAC is presented as the upper bound of the quality of Q-function and MRN achieves the lowest MSE among three methods. MRN, reward learning additionally takes the performance of the Q-function into consideration. So the reward provided by the reward function is not only consistent with the ground-truth preference labels but also suitable for the performance of the current Q-function. Therefore, the Q-function learned by MRN is more accurate compared to the baselines.
Table Mean squared error of learned Q-function across ten runs.
Task/Method
PEBBLE
PEBBLE+SURF
PEBBLE+MRN
SAC (upper bound)
Walker
0.12 + 0.04
0.11 +: 0.04
0.10 + 0.02
0.07 + 0.02
Door Open
0.52 + 0.05
0.60 0.32
0.48 + 0.06
0.45 + 0.01

---

Conclusion
In this work, we propose Meta-Rewarde a novel feedback-efficient preference-bast RL method. By incorporating bi-level optimization for reward and policy learning, demonstrate MRN outper- forms prior methods when a small amount of human feedback is available and considerably improves the feedback efficiency the baselines by large margin when there is few preference labels. From empirical results and analysis, we conclude that the efficiency of feedback in our method mainly benefits from: Firstly, our method learns a more accurate Q-function by aligning it with human preferences. Secondly, our method learns the Q-function and policy in the inner loop and optimizes reward function according the performance of Q-function on the preference data in the outer loop. By this way, MRN success- fully establishes
variety of robotic simulated tasks. In particular,
method exceeds
efficient mode of information transmission, which can extract more information.
The potential negative social impact includes the carbon footprint of the experiments and future work based on MRN. We hope our method can provide inspiration for future work and encourage preference- -base reinforcement learning to be better extended to practical applications.
Limitations. There several limitations of MRN as follows. First, MRN relies on qualities of human feedback. Also, MRN cannot discriminate between good and bad actions within one trajectory. Last, representation learning on visual input requires additional parameters and thus requires more data. While this is not the focus this work, we consider this as future works.
Acknowledgements
Yaodong Yang is Sponsored by CAAI-Huawei MindSpore and CCF-Tencent Open Research Fund.
References
[1] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013.
[2] Oriol Vinyals, Igor Babuschkin, Wojeiech M. Czamecki,, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander Sasha Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom Le Paine, Çaglar Gülçehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McK- inney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in starcraft II using multi-agent reinforcement learning. Nature, 575(7782)350-335555 -544 2019.
[3] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Christopher Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki. Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learning. CoRR, 0bs/1912.00660, 2019.
[4] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. Nature, 52947587):4844889 2016.
[5] P Peng, Q Yuan, Y Wen, Y Yang, Z Tang, H Long, and J Wang. Multiagent bidirectionally- coordinated nets for learning to play starcraft combat games. arXiv preprint arXiv:1703. 10069, 2017.
[6] Sen Wang, Daoyuan Jia, and Xinshuo Weng. Deep reinforcement learning for autonomous driving. arXiv preprint arXiv:1811.11322 2018.
10

---

[7] Ming Zhou, Jun Luo, Julian Villella, Yaodong Yang. David Rusu, Jiayu Miao, Weinan Zhang, Montgomery Alban, Iman Fadakar, Zheng Chen, al. Smarts: Scalable multi-agent rein- forcement learning training school for autonomous driving. arXiv preprint arXiv:2010.09776. 2020.
[8] Yuanpei Chen, Yaodong Yang. Tianhao Wu, Shengjie Wang, Xidong Feng, Jiechuang Jiang, Stephen Marcus McAleer, Hao Dong, Zongging Lu, and Song-Chun Zhu. Towards human-level bimanual dexterous manipulation with reinforcement learning. arXiv preprint arXiv:2206. 08686, 2022.
[9] Yiran Geng, Boshi An, Haoran Geng, Yuanpei Chen, Yaodong Yang, and Hao Dong. End-to-end affordance learning for robotic manipulation. arXiv preprint arXiv:2209. 020229999999111 /2941, 2022.
[10] Minne Li, Zhiwei Qin, Yan Jiao, Yaodong Yang, Jun Wang, Chenxi Wang, Guobin Wu, and Jieping Ye. Efficient ridesharing order dispatching with mean field multi-agent reinforcement learning. In The World Wide Web Conference ( (WWW), page 983-994. Association for Computing Machinery, 2019.
[11] Yaodong Yang. Lantao Yu, Yiwei Bai, Ying Wen, Weinan Zhang, and Jun Wang. A study of ai population dynamics with million-agent reinforcement learning. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS), page 2133 2135. International Foundation for Autonomous Agents and Multiagent Systems, 2018.
[12] Xiao- Yang Liu, Hongyang Yang, Jiechao Gao, and Christina Dan Wang. Finrl: deep rein- forcement learning framework to automate trading in quantitative finance. In International Conference on Al in Finance (ICAIF), pages 1:1-1:9. ACM, 2021.
[13] Chris Gamble and Jim Gao. Safety-first ai for autonomous data centre cooling and industrial control. In DeepMind. 2018.
[14] Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. CoRR, abs/ (bs//886.111 10293, 2018
[15] Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learn- ing for robotic manipulation with asynchronous off-policy updates. In IEEE International Conference Robotics and Automation (ICRA), pages 3389-3396, 2017.
[16] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 29. Curran Associates, Inc., 2016.
[17] Kimin Lee, Laura M Smith, and Pieter Abbeel. Pebble: forcement learning via relabeling experience and unsupervised pre-training. In International Conference on Machine Learning (ICML), volume 139, pages 6152-6163 2021.
interactive rein-
[18] Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. SURF: Semi-supervised reward learning with data augmentation for feedback-efficin preference- based reinforcement learning. In International Conference Learning Representations (ICLR), 2022.
[19] Xinran Liang, Katherine Shu, Kimin Lee, and Pieter Abbeel. Reward uncertainty for explo- ration in preference- based reinforcement learning. In International Conference on Learning Representations (ICLR), 2022.
[20] Hieu Pham, Zihang Dai, Qizhe Xie, and Quoc Le. Meta pseudo labels. IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 11557-11568, 2021.
[21] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. Conference on Robot Learning ( (CoRL., volume 100, pages 1094-1100 100. PMIR, 2020.
11

---

[22] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprini /801000690 2018.
[23] Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm_control: Software and tasks for continuous control. Software Impacts, 6: 1000022, 2020.
[24] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2016.
[25] A. V. Bernstein and E. V. Burnaev. Reinforcement learning in computer vision. In International Conference on Machine Vision (ICMV), volume 10696, pages 458 464. International Society for Optics and Photonics, SPIE, 2018.
[26] Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv:1709.000 2017.
[27] Lixin Zou, Long Xia, Zhuoye Ding, Jiaxing Song, Weidong Liu, and Dawei Yin. Reinforcement learning to optimize long-term user engagement in recommender systems. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), page 2810-2818. Association for Computing Machinery, 2019.
[28] Guanjie Zheng, Fuzheng Zhang, Zihan Zheng, Yang Xiang. Nicholas Jing Yuan, Xing Xie, and Zhenhui Li. Dr:: A deep reinforcement learning framework for news recommendation. In World Wide Web Conference (WWW), page 167-176. International World Wide Web Conferences Steering Committee, 2018
[29] Yaodong Yang and Jun Wang. An overview of auett reinforcement learning from game theoretical perspective. arXiv preprint arXiv:2011.0058, 2020.
[30] Xiaotie Deng, Yuhao Li, David Henry Mguni, Jun Wang, and Yaodong Yang. On the complexity of computing markov perfect equilibrium in general-sum stochastic games. arXiv preprint arXiv:2109.01795 2021.
[31] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems (NeurIPS), volume 30. Curran Associates, Inc., 2017.
[32] Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving. Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. In Advances in Neural Information Processing Systems (NeurIPS), volume 31. Curran Associates, Inc., 2018.
[33] Dorsa Sadigh, Anca Dragan, Shankar Sastry, and Sanjit Seshia. Active preference-based learning of reward functions. In Robotics: Science and Systems (RSS), 2017.
[34] Erdem Biyik and Dorsa Sadigh. Batch active preference- based learning of reward functions. In Conference Robot Learning (CoRL), volume 87, pages
PMLR, 2018.
[35] Erdem Biyik, Kenneth Wang, Nima Anari, and Dorsa Sadigh. Batch active learning using determinantal point processes, CoRR, abs/1906.07975. 2019.
[36] Malayandi Palan, Gleb Shevchuk, Nicholas Charles Landolfi, and Dorsa Sadigh. Learning reward functions by integrating human demonstrations and preferences. In Robotics: Science and Systems (RSS), 2019.
[37] Erdem Biyik, Nicolas Huynh, Mykel Kochenderfer, and Dorsa Sadigh. Active Preference- Gaussian Process Regression for Reward Learning. In Robotics: Science and Systems (RSS), 2020.
[38] Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta- neight-net: Learning an explicit mapping for sample weighting. In Advances in Neural Information Processing Systems (NeurIPS), volume 32. Curran Associates, Inc., 2019.
12

---

[39] Guoqing Zheng, Ahmed Hassan Awadallah, and Susan T. Dumais. Meta label correction for noisy label learning. In Association for the Advancement of Artificial Intelligence (AAAI), pages 10053-11611, 2021.
[40] Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. In Advances Neural Information Processing Systems (NeurIPS), volume 31. Curran Associates, Inc., 2018.
[41] Ying Wen, Yaodong Yang, Rui Luo, Jun Wang, and Wei Pan. Probabilistic recursive reasoning for multi-agent reinforcement learning. In International Conference on Learning Representa- tions (ICLR), 2019.
[42] Ying Wen, Yaodong Yang, and Jun Wang. Modelling bounded rationality in multi- interac- tions by generalized recursive reasoning. In Proceedings of the Twenty-Nine International Joint Conference on Artificial Intelligence (IJCAI), pages 414-421. International Joint Conferences on Artiticial Intelligence Organization, 7 2020. Main track.
[43] Haifeng Zhang. Weizhe Chen, Zeren Huang, Minne Li, Yaodong Yang, Weinan Zhang, and Jun Wang. Bi-level actor-critic for multi-agent coordination. Proceedings of the American Association for Artificial Intelligence (AAAI), 34(05))7325-3322 Apr. 2020.
[44] Yali Du, Lei Han, Meng Fang. Ji Liu, Tianhong Dai, and Dacheng Tao. Lirr: Learning individual intrinsic reward in multi-agent reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 32. Curran Associates, Inc., 2019.
[45] Songyuan Zhang, Zhangjie Cao, Dorsa Sadigh, and Yanan Sui. Confidence-awee imitation learning from demonstrations with varying optimality. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pages 12340-12350. Curran Associates, Inc., 2021.
[46] Zeyu Zheng, Junhyuk Oh, Matteo Hessel, Zhongwen Xu, Manuel Kroiss, Hado Van Hasselt, David Silver, and Satinder Singh. What can learned intrinsic rewards capture? In International Conference Machine Learning (ICML), volume 119, pages 11436- 1446. PMLR, 2020.
[47] Xidong Feng, Oliver Slumbers, Ziyu Wan, Bo Liu, Stephen McAleer, Ying Wen, Jun Wang, and Yaodong Yang. Neural auto-curricula in two-player zero-sum games. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pages 3504-3517. Curran Associates, Inc., 2021.
[48] Nicolas -ieve,, Yaodong Yang, Oliver Slumbers, David H Mguni, Ying Wen, and Jun Wang. Modelling behavioural diversity for learning in open open-ended games. In Proceedings of the 38th International Conference on Machine Learning (ICML), volume 139, pages 8514-8524. PMLR, 18-24 Jul 2021.
[49] Xiangyu Liu, Hangtian Jia, Ying Wen, Yujing Hu, Yingfeng Chen, Changjie Fan, ZHIPENG HU, and Yaodong Yang. Towards unifying behavioral and response diversity for open-ended learning in zero-sum games. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pages 941-952. Curran Associates, Inc., 2021.
[50] Lantao Yu, Tianhe Yu, Chelsea Finn, and Stefano Ermon. Meta-inverse reinforcement learning with probabilistic context variables. In Advances in Neural Information Processing Systems (NeurIPS), volume 32. Curran Associates, Inc., 2019.
[51] Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot visual imitation learning via meta-learning. In Conference on Robot Learning (CoRL), volume 78, pages 357-368. PMLR, 2017.
[52] Allan Zhou, Eric Jang, Daniel Kappler, Alex Herzog, Mohi Khansari, Paul Wohlhart, Yunfei Bai, Mrinal Kalakrishnan, Sergey Levine, and Chelsea Finn. Watch, try, learn: Meta-learning from demonstrations and rewards. In International Conference on Learning Representations (ICLR), 2020.
[53] Neha Das, Sarah Bechtle, Todor Davchev, Dinesh Jayaraman, Akshara Rai, and Franziska Meier. Model-based inverse reinforcement learning from visual demonstrations. In Conference on Robot Learning (CoRL), volume 155, pages 1930-1942. PMLR, 2021.
13

---

[54] Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-333 -422 1952.
[55] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning (ICML), volume 80, pages 1861-1870, 2018.
[56] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
[57] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.0634/ 2017.
(58) Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel. B-pref: Benchmarking preference- based reinforcement learning. In Neural Information Processing Systems Track Datasets and Benchmarks (NeurIPS), volume 1, 2021.
[59] Yu Nesterov. Introductory lectures on convex programming, 1998.
[60] Julien Mairal. Stochastic majorization-miniiizz algorithms for large-scale optimization, In Advances Neural Information Processing Systems (NeurIPS), volume 26. Curran Associates, Inc., 2013.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See abstract, Section aand Section E
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
See
Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We have read the ethics and make sure our work conforms to them.
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? and Appendix
[Yes] See Section
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] See abstract and implementation details in Section 5.1
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See implementation details in Section and Appendix E
(c) Did you report error bars (e.g., with respect the random seed after running ex- periments multiple times)? [Yes] See Figure Figure
Figure and Table in
Section 5.2 and Section 5%
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Section 5.1
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Our code refers to the source code published by B-Pref and SURF, and their papers are cited in our paper. See Section
5
(b) Did you mention the license of the assets? [Yes] All referenced code is open source and we notice that B-Pref provides MIT license and mujoco provides Apache License. As for DMControl and Meta-world, you references for license details.
14

---

(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] We release the link of our source code abstract.
(d) Did you discuss whether and how consent was obtained from people whose data you're 'r using/curating? [Yes] MIT license permits to copy and modify the source code of B-Pref. We also get the permission of the author of SURF.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "hzcKJ73BPwN", "reviewer": "Reviewer_5HHg", "summary": "This paper propose MRN, a feedback-efficient PbRL framework that incorporates bi-level optimization for both reward and policy learning. The key idea is to incorporate a new auxiliary task into standard PbRL framework, through utilizing the performance of the Q-function to predict preference label. Multiple experiments are conduct to verify MRN's superiority over various PbRL baselines in terms of high feedback efficiency and cumulative return.", "questions": "（1）Is it reasonable to utilize $Q_{\\theta}(s_{0}^{0}, a_{0}^{0})$ to represent the return of the segment $\\sigma^{0}$ in Eq. (6)? $Q_{\\theta}(s_{0}^{0}, a_{0}^{0})$ represents the cumulative return of continuing executing $\\pi$ after selecting action $a_0$ at state $s_0$. However, the behavior performed by $\\pi$ may not be consistent with segment $\\sigma^{0}$ at subsequent state $s_{t}^{0}, where $t>0$. \n(2) Although the experiment is relatively sufficient, there is still a lack of rational explanation, why using the Q value to give the pseudo-label will bring about the high feedback efficiency?\n(3) Why MRN performs far less than SAC on almost all experiments？\n(4) Why does MRN perform only marginally better than other baselines in the DMControl environment?", "limitations": "This paper lacks to address the limitations of their work. （1）Since reward learning and policy learning are performed iteratively, it is not clear whether the convergence of the algorithm can be guaranteed. （2）The extent to which the performance of reward learning depends on the quality of human feedback.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality:\nAlthough MRN is the first for PbRL, its key ideas have been proposed and widely used in computer vision. \n\nQuality：\nThe central idea of this paper is straightforward. I have two key doubts as follows。（1）Is it reasonable to utilize $Q_{\\theta}(s_{0}^{0}, a_{0}^{0})$ to represent the return of the segment $\\sigma^{0}$ in Eq. (6)? $Q_{\\theta}(s_{0}^{0}, a_{0}^{0})$ represents the cumulative return of continuing executing $\\pi$ after selecting action $a_0^0$ at state $s_0^0$. However, the behavior performed by $\\pi$ may not be consistent with segment $\\sigma^{0}$ at subsequent state $s_{t}^{0}$, where $t>0$. (2) Although the experiment is relatively sufficient, there is still a lack of rational explanation, why using the Q value to give the pseudo-label will bring about the high feedback efficiency?\n\nClarity: \nThe main content is clear, but there is a lack of rational explanation for some key points. For example, it is unclear why using the Q value to give the pseudo-label will bring about the high feedback efficiency?\n\nSignificance:\nThe work is significant for improving feedback efficiency in PbRL, but I doubt the soundness of the main idea.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "N3nNEaH8AHJ", "reviewer": "Reviewer_gArn", "summary": "This work introduces a meta-learning based interpretation of preference-based reinforcement learning. The proposed method MRN utilises bi-level optimisation to learn reward functions using expert preferences, a method introduced in [23]. The novelty of this work is in the implicit conditioning of the reward learning process on the values obtained from the Q function. The work is evaluated thoroughly in simulation and compared to reasonable baselines. The discussed results show the faster and more efficient (in terms of required expert labels) learning process of MRN. Overall, my opinion of this work is generally positive. \n\nHowever, I do have a few concerns and suggestions. The strongest reservations I have are that there is **no sufficient motivation** for using the proposed approach over alternative meta-learning based reward learning solutions. Furthermore, at this point it seems like the proposed solution has a **fairly moderate impact** (more details below). I believe that addressing these concerns and the rest of my suggestions listed below can make this work stronger and can therefore change my score.", "questions": "### Additional motivation needed\n- This work should draw clearer boundaries between pbRL and alternative reward learning methods and more generally alternative learning from human feedback methods. Right now, it is not very clear how and why this is different to IRL, for example, unless the reader is very familiar with [23]. This should not be the case, instead the paper should be really readable on its own and less dependent on prior work\n    - Ideally, this should be done by proposing a stronger motivation for why pbRL (and more specifically meta-pbRL) as opposed to using something alternative (like IRL or meta-IRL).\n    - It would be great to have a more exhaustive review of other meta-learning approaches from human feedback. A non-exhaustive list would include [i, ii, iii, iv], for example.\n\n[i] Yu, L., Yu, T., Finn, C. and Ermon, S., 2019. Meta-inverse reinforcement learning with probabilistic context variables. NeurIPS 2019\n\n[ii] Zhou, A., Jang, E., Kappler, D., Herzog, A., Khansari, M., Wohlhart, P., Bai, Y., Kalakrishnan, M., Levine, S. and Finn, C., 2019. Watch, try, learn: Meta-learning from demonstrations and reward. ICLR 2020\n\n[iii] Finn, Chelsea, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. \"One-shot visual imitation learning via meta-learning.\" CoRL 2017\n\n[iv] Das, N., Bechtle, S., Davchev, T., Jayaraman, D., Rai, A. and Meier, F., 2020. Model-based inverse reinforcement learning from visual demonstrations. CoRL 2020\n\n### Improving the Clarity of Expression\n- The paper seems to refer to implicit conditioning on the predicted values from the Q function for the reward learning process as feedback but this is generally confusing since feedback has a very particular meaning in the context of robotics and robot learning. Specifically, **feedback refers to the mechanisms used for measuring the response over a specific variable** and this is not what is meant here. Similarly, the sentence '_[24] utilizes two kinds of feedback, initializes with imitation learning policy, and further improves the performance of the policy  with feedback from humans_' is confusing. Bootstrapping a policy does not mean the method uses demonstrations as feedback, similarly, using human feedback means actively correcting the system online, similarly to what Dagger-like methods do. Therefore, statements like this can be made a bit crisper. Similarly, other references to '_feedback_' in the paper seem equally confusing. An additional example is '_MRN outperforms prior methods in the case of little feedback_ ' - learning from a few labels does not mean there is a little feedback. I'd strongly encourage the authors to be clearer in the proposed expressions as to improve clarity.\n- The authors mention that their '_intuition is that the optimization of the reward function needs both supervised learning and the feedback from the Q-function_' but the authors also conduct thorough experimentation ablating the use of those terms in appendix. Referring the readers to the relevant experiments would be great here too.\n- It is overall great that the proposed work is doing better than the alternative solutions. However, it is not really clear why the performance of Preference PPO is so poor on the DMControl tasks. In particular, when comparing the reported results in Figure 3 with the original Preference-based paper [23] on those environments, it is really strange that Preference PPO does so poorly. It may be a good idea to clearly report the number of labels used to learn the rewards in the figures themselves. It took me a while to realise that there is a significant difference in the number of demos used in this work and the reported results in [23]. Adding a bit more discussion on this front can also be beneficial.\n- Figure 4 is great, but it is really not clear what the upper-bound is. Therefore, it may be useful to add the achieved results with the ground truth reward using SAC. This should be easy, unless I am mistaken, those results already exist and are reported in Figures 2 and 3. Furthermore, it will be useful if the rows clearly indicated the names of the environments, similar to what is done in Figures 2 and 3.\n- Conclusion summarises that the work is evaluated on robotic tasks but it should be more explicit, it is evaluated on robotic simulated tasks which is very different in practice.\n- The paper also states that '_To our knowledge, we do not see any potential negative social influences of our work_' but the carbon footprint of the ran experiments and the required future runs for building on top of this (or any model-free approach) has a fairly negative impact. As a result, it may be better acknowledging this as opposed to stating that you do not see any negative social influences. Alternatively, just take out this sentence.\n- The paper has a typo, namely '_we the mean squared error between_'' should be '_we use.._'", "limitations": "This paper does not list the existing limitations of the proposed approach. I would really like to see a comprehensive list and a discussion of the existing limitations. \n\n- A good example is that although it learns better from a few labels, it does not seem to solve the tasks with those few labels. This (along with the lacking discussion pro/con IRL) is a main reason for me to feel that the proposed solution has a fairly moderate impact. Perhaps a discussion on this topic would be great.\n- The method also seems very sensitive to the number of meta updates. \n- It also seems (from Fig. 2) that the learning process is not very stable and that some seeds completely failed to solve the tasks.\n- Additionally, it is not clear how to scale this work to learning visual observations as it may require infeasible number of labels. \n\nDiscussing all those and potentially other limitations can be very useful.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- A successful application of bi-level optimisation in the context of preference-based RL\n- An overall thorough evaluation and nice ablations\n- A fairly clear description of the algorithm\n\n### Weaknesses\n- lacks motivation for the use of this approach as opposed to alternatives (e.g. IRL)\n- at times lacks clarity of expression", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "J43ao_e-UQ3", "reviewer": "Reviewer_tWWp", "summary": "The paper introduces the meta-reward-net (MRN) method for preferenced-based reinforcement learning (PbRL). MRN learns a reward from human preferences over trajectory pairs by combining a supervised preference-based loss with a bi-level optimization loss that optimizes the reward based on the Q-function's ability to reflect human preferences. Experiments show that MRN outperforms state-of-the-art baselines in PbRL. Furthermore, MRN improves the efficiency of using feedback labels. Lastly, the paper analyzes the source MRN's gains as coming from a more accurate Q-function. ", "questions": "* The paper only considers a single pseudo update of the Q-function when computing the outer-loop loss. Did the work consider multiple psuedo updates? This could result in greater efficiency due to more accurate meta-gradient updates.\n* What is the impact of the weighting of the two losses in MRN? In MRN, is the bi-level loss and the auxiliarly loss weighted the same? While, Section D analyzes different update frequencies, is the magnitude of these losses also important? ", "limitations": "The paper analyzes the limitations of MRN through the feedback comparison experiments in Section 5.3. These results show that MRN is limited to working best relative to baselines when there is little available avaiable feedback. However, a more explicit discussion of the limitations of the approach would be valuable.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The methodology is novel and well motived for the problem. Updating the reward based on the Q-function's performance is a sensible way to provide more reward supervision as this will update the reward to be better suited for the task.\n* MRN has the novel insight of introducing bi-level optimization into PbRL. Bi-level optimization for PbRL could open future research into using other metrics in the outer loss.\n* MRN consistently outperforms state-of-the-art prior work in all the considered benchmarks. Furthermore, the benchmarks are standard in the field and a fair assessment of MRN and baselines.\n* Figure 4 in Section 5.3 demonstrates that MRN is preferred over baselines when less feedback is available.\n* Analysis of the Q-function accuracy in Section 5.3 helps analyze the source of performance gains. MRN learns a more accurate Q-function, which enables MRN to perform better with less feedback than baselines.\n* The paper is well written and clear. \n\nWeaknesses:\n* Can MRN outperform baselines even with more feedback? In Figure 4 h,d, where methods receive 2k feedback labels, all methods perform similarly. Or can MRN help PbRL scale to more complex tasks better than baselines, even with more feedback? Since the amount of feedback greatly impacts performance, how was the amount of feedback selected for the experiments in Figures 2 and 3?\n* There is still a large gap between MRN and the performance of the expert with the ground truth reward. What limitations of MRN, and PbRL in general, prevent it from reaching the performance of the expert?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "0sbqXrT8bm1", "reviewer": "Reviewer_hg7U", "summary": "This paper proposes a novel algorithm called MRN for preference-based reinforcement learning, which models the preference order directly from the Q-value function instead of per-step rewards.\n\nTo realize this framework, this paper develops a meta-gradient-based method to learn reward functions from the supervision upon Q-values.\n\nThe experiment results show that MRN significantly outperforms baselines in terms of both sample efficiency and final policy quality.", "questions": "1. In section 5.3, what is the ground-truth Q-value? The policy preference is defined by binary comparison signals. How can the scale of $\\hat r_\\psi$ match the scale of unknown ground-truth rewards?\n\n2. The introduction of bi-level optimization to preference-based RL is novel. However, the related work section can benefit from prior works on meta-gradient-based reward learning [1-2].\n\n[1] Zhang et al. \"On Learning Intrinsic Rewards for Policy Gradient Methods\" NeurIPS 2018.\n\n[2] Zhang et al. \"What can learned intrinsic rewards capture?\" ICML 2020.", "limitations": "No potential negative societal impact.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well organized. The notations are clear and intuitive\n\nThe proposed algorithm is novel and the experiment results are solid.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept