SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning
Authors: Jongjin Park
Younggyo Seo
Jinwoo Shin
Honglak Lee
Pieter Abbeel
Kimin Lee
Venue: ICLR 2022 Poster (2022)
Keywords: preference-based reinforcement learning
human-in-the-loop reinforcement learning
deep reinforcement learning
semi-supervised learning

Abstract: Preference-based reinforcement learning (RL) has shown potential for teaching agents to perform the target tasks without a costly, pre-defined reward function by learning the reward with a supervisor’s preference between the two agent behaviors. However, preference-based learning often requires a large amount of human feedback, making it difficult to apply this approach to various applications. This data-efficiency problem, on the other hand, has been typically addressed by using unlabeled samples or data augmentation techniques in the context of supervised learning. Motivated by the recent success of these approaches, we present SURF, a semi-supervised reward learning framework that utilizes a large amount of unlabeled samples with data augmentation. In order to leverage unlabeled samples for reward learning, we infer pseudo-labels of the unlabeled samples based on the confidence of the preference predictor. To further improve the label-efficiency of reward learning, we introduce a new data augmentation that temporally crops consecutive subsequences from the original behaviors. Our experiments demonstrate that our approach significantly improves the feedback-efficiency of the state-of-the-art preference-based method on a variety of locomotion and robotic manipulation tasks.

Published as a conference paper at ICLR 2022
SURF: SEMI-SUPERVISED REWARD LEARNING WITH DATA AUGMENTATION FOR FEEDBACK-EFTICIENT PREFERENCE-BASER REINFORCEMENT LEARNING
Jongjin Park Younggyo Seo Jinwoo Shin- Honglak Lee2.4 Pieter Abbeet Kimin Leea
KAIST University of Michigan 'UC Berkeley LL.G AT Research
ABSTRACT
Preference-based reinforcement learning (RI.) has shown potential for teaching agents to perform the target tasks without costly, pre-defined reward function by learning the reward with supervisor's preference between the two agent behav- iors. However, preference-basee learning often requires a large amount of human feedback, making it difficult to apply this approach to various applications. This data-efficiency problem, on the other hand, has been typically addressed by using unlabeled samples or data augmentation techniques in the context of supervised learning. Motivated by the recent success of these approaches, we present SURF, semi-supervised reward learning framework that utilizes large amount of un- labeled samples with data augmentation. In order to leverage unlabeled samples for reward learning, we infer pseudo-labels of the unlabeled samples based on the confidence of the preference predictor. To further improve the label-efficiency of reward learning, we introduce new data augmentation that temporally crops con- secutive subsequences from the original behaviors. Our experiments demonstrate that our approach significantly improves the feedback-efficienccy of the state-of- the-art preference-based method on a variety of locomotion and robotic manipu- lation tasks.
INTRODUCTION
Reward function plays a crucial role reinforcement learning (RL) to convey complex objectives to agents. For various applications, where we can design informative reward function, RL with deep neural networks has been used solve a variety of sequential decision-making problems, including board games (Silver et al., 2017; 2018), video games (Mnih et al., 2015; Berner et al., 2019; Vinyals et al., 2019), autonomous control (Schulman et al., 2015; Bellemare et al., 2020), and robotic ma- nipulation (Kober & Peters, 2011; Kober al., 2013; Kalashnikov et al., 2018; Andrychowicz al., 2020). However, there are several issues in reward engineering. First, designing a suitable reward complex. For example, defining
function requires more human effort the tasks become reward function for book summarization (Wu et al., 2021) is non-trivial because it is hard to quan- tify the quality of summarization in scale value. Also, it has been observed that RL agents could achieve high returns by discovering undesirable shortcuts if the hand-engineered reward does fully specify the desired task (Amodei et al., 2016; Hadfield-Menell thermore, there are various domains, where single ground-truth function does not exist, and thus personalization is required by modeling different reward functions based on the user's preference.
al., 2017: Lee et al., 2021a). Fur-
Preference- RL (Akrour et al., 2011; Christiano et al., 2017; Ibarz et al., 2018; Lee et al., 2021a) provides an attractive alternative to avoid reward engineering. Instead of assuming hand- engineered reward function, (human) teacher provides preferences between the two agent behav- iors, and an agent learns how to show the desired behavior by learning a reward function, which is consistent with the teacher's preferences. Recent progress of preference-based RL has shown that the teacher can guide the agent to perform novel behaviors (Christiano et al., 2017: Stiennon et al., 2020: Wu et al., 2021), and mitigate the effects of reward exploitation (Lee et al., 2021a). However, existing preference- based approaches often suffer from expensive labeling costs, and this makes hard to apply preference-basee RL to various applications.

---

Published as a conference paper at ICLR 2022
Meanwhile, recent state-of-the- system in computer vision, the label-efficiency problem has been successfully addressed through supervisee learning (SSL) approaches (Berthelot et al., 2019; 2020: Sohn et al., 2020; Chen et al., 2020b). By leveraging unlabeled dataset, SSL methods have improved the performance with low cost. Data augmentation also plays a significant role in im- proving the performance of supervised learning methods (Cubuk et al., 2018; 2019). By using mul- tiple augmented views of the same data as input, the performance has been improved by learning innteentaatonnnvvriann representations.
Inspired by the impact of semi-supervised learning and data augmentation, we present SURF: a Seri-sUpervised Reward learning with data augmentation for Feedback-efficien preference-basee RL. To be specific, SURF consists of the following key ingredients:
(a) Pseudo-labeling (Lee, 2013; Sohn al., 2020): We leverage unlabeled data by utilizing the artificial labels generated by learned preference predictor, which makes the reward function produce confident prediction (see Figure la). We remark that such SSL approach is par- ticularly attractive in our setup as an unlimited number of unlabeled data can be obtained with additional cost, i.e., from past experiences stored in the buffer.
(b) Temporal cropping augmentation: We generate slightly shifted or resized behaviors, which are expected to have the same preferences from teacher, and utilize them for reward learning (see Figure lb). Our data augmentation technique enhances the feedback-effcienccy by enforcing consistencies (Xie et al., 2019; Berthelot al., 2020; Sohn al., 2020) to the reward function.
We remark that SURF is not naïve application of these two techniques, but novel combination of semi-supervised learning and the proposed data augmentation, which has not been considered or evaluated in the context of the preference-based RI..
Our experiments demonstrate that SURF significantly improves the preference-based RL method (Lee et al., 2021a) on complex locomotion and robotic manipulation tasks from DeepMind Control Suite (Tassa et al., 2018; 2020) and Meta-world (Yu et al., 2020), in terms of feedback- efficiency. In particular, our framework could make RL agents achieve 1000 of success rate on complex robotic manipulation task using only a few hundred preference queries, while its baseline method only achieves ~50% of the success rate under the same condition (see Figure 2). Further- more, we show that SURF can improve the performance of preference- RL algorithms when operate on high-dimensional and partially-observable inputs.
RELATED WORK
Preference-based RL. In the preference-basee RI. framework, a (human) supervisor provides pref- erences between the two agent behaviors and the agent uses this feedback to perform the task (Chris- tiano et al., 2017; Ibarz et al., 2018; Leike et al., 2018; Stiennon et al., 2020; Wu et al., 2021; Lee et al., 2021 10:)a; Since this approach is only feasible if the feedback is practical for a human to pro- vide, several strategies have been studied in the literature. Ibarz et al. (2018) initialized the agent's policy with imitation learning from the expert demonstrations, while Lee et al. (2021a) utilized un- supervised pre-training for policy initialization. Several sampling schemes (Sadigh al., 2017; Biyik& & Sadigh, 2018; Biyik et al., 2020) to select informative queries also have been adopted for improving the -fffcickefficiennyy. Our approach differs in that we utilize unlabeled samples for reward learning, and also provide a novel data augmentation technique for the agent behaviors.
Data augmentation for RL. In the context of RL, data augmentation has been widely investigated for improving data-efficiene, (Srinivas et al., 2020; Yarats et al., 2021), or RL generalization (Cobbe al., 2019; Lee et al., 2019). For example, RAD (Laskin et al., 2020) demonstrated that data augmentation, such as random crop, can improve both data-efficiency and generalization of RI. algorithms. While these methods are known setup, they have not been tested for learning rewards. To the best of our knowledge, present the first data augmentation method specially designed for learning reward function.
beneficial to learn policy the standard RI.
Semi-supervised learning. The goal of semi-supervised learning (SSL) is to leveraging unlabeled samples to improve a model's performance when the amount of labeled samples are limited. In an attempt to leverage the information in the unlabeled dataset, number of techniques have been proposed, e.g., entropy minimization (Grandvalet & Bengio, 2004; Lee, 2013) and consistency reg- ularization (Sajjadi et al., 2016; Miyato et al., 2018; Xie et al., 2019; Sohn et al., 2020). Recently,
2

---

Published as a conference paper at ICLR 2022
Po o
Preference predictor
Ps
Cross eutropy
V (1,0)
(a) Pseudo-labeling
(b) Temporal cropping
Figure 1: Overview of SURF. (a) We leverage unlabeled experiences by generating pseudo-labels from the preference predictor Ps: in (1). To mitigate the negative effects from this semi-supervised learning, only utilize pscudo-labels when the confidence of the predictor is higher than threshold т. (b) Given two segments (a", ()), we generate augmented segments (6", ,ā") by cropping the subsequence from each segment.
the combination of these two approaches have shown state-of-the-ar performance in benchmarks, e.g., MixMatch (Berthelot et al., 2019), and ReMix (Berthelot et al., 2020), when used with advanced data augmentation techniques (Zhang et al., 2018; Cubuk et al., 2019). Specifically, Fix- Match (Sohn et al., 2020) revisits pseudo-labeling technique and demonstrates that joint usage of pseudo-labele and consistency regularization achieves remarkable performance due to its simplicity.
PRELIMINARIES
Reinforcement learning (RL) is a framework where agent interacts with environment in dis- crete time (Sutton & Barto, 2018). At each timestep t, the agent receives state S, from the en- vironment and chooses action a, based its policy л (ar[s:). In conventional RL framework, the environment gives reward r(s:, ar) and the agent transitions to the next state S/+1- The return R, -k=0 ( ac+*) is defined discounted cumulative sum of the reward with discount factor 7 [0, 1). The goal of the agent is to Iearn policy that maximizes the expected return.
Preference-based reinforcement learning. In this paper, we consider a preference-based RL framework, which does not assume the existence of hand-engineered reward. Instead, (human) teacher provides preferences between the agent's behaviors and the agent uses this feedback to per- form the task (Christiano et al., 2017; Ibarz et al., 2018; Leike et al., 2018; Stiennon et al., 2020; Lee al., 2021a;b; Wu al., 2021) by learning reward function, which is consistent with the observed preferences.
We formulate reward learning problem as supervised learning problem (Wilson al., 2012; Christiano et al., 2017). Formally, segment is sequence of observations and actions {Sk, ....... (SK+ H-11 a --I)). Given a pair of segments (o", oo",a)). ). teacher gives a feedback indicating which segment preferred, i.e., y {0, 1, 0.5), where 1 indicates o! > σ indicates σº > o¹. and 0.5 implies an equally preferable case. Each feedback is stored in a dataset D as a triple (a", o², y). Then, model preference predictor using the reward function F.e following the Bradley-Terry model (Bradley Terry, 1952):
Polo > =09= Eie(o.1) Ex (s).
(I)
where o¹ rd denotes the event that segment è preferable segment j. The underlying assump- tion of this model is that the teacher's probability of preferring segment depends exponentially on the accumulated sum of the reward over the segment. The reward model trained through super- vised learning with teacher's preferences. Specifically, given a dataset of preferences D, the reward function is updated by minimizing the binary cross-entropy loss:
C
E lo"_
[Revard
(o)) [1 y) log Polo > o'1 ylog Pylo > d9
(o".0"
The reward function P.r is usually optimized only using labels from real human, which are expensive to obtain in practice. Instead, we propose a simple yet effective method based on semi-supervised learning and data augmentation to improve the feedback-efficienge of preference-hased leaming.
3

---

Published as a conference paper at ICLR 2022
Algorithm SURF
Require: Hyperparameters: unlabeled batch ratio μ., threshold parameter т, and loss weight
Require: Set of collected labeled data D;, and unlabeled data D.,
1: for each gradient step do
2: Sample labeled batch {(o ,7, () Tima ~ Di Ju
3: Sample unlabeled batch {(o2, G.) Jj=1 ~ Da
4: // DATA AUGMENTATION FOR LABELED DATA
5: for i in 1... B do
6: (o ot) TDA((o), ()) in Algorithm end for
7:
8:
AND DATA AUGMENTATION FOR UNLABELED DATA // 1... pB do
9:
for j in
Predict pseudo-labels (((a u. o2)
11:
(o G1) TDA((OL, ob) in Algorithm
end for
13:
Optimize _ (3) with respect to ø
14:
end for
SURF
In this section, we present SURF: Semi-sUpervised Reward learning with data augmentation for Feedback-efficient preference-based RL, that can be used in conjunction with any existing preference-bart RL methods. Our main idea is to leverage a large number of unlabeled samples collected from environments for reward learning, by inferring pseudo-labels. To further increase the effective number of training samples, we propose a new data augmentation that temporally crops subsequence of the agent behaviors. The full procedure of our unified framework in Algorithm (See Figure 1 for the overview of our method).
4.1 SEMI-SUPERVISED REWARD LEARNING
To improve the feedback efficiency, we propose semi-supervised learning (SSL. method for lever- aging unlaheled experiences in the buffer for reward learning. In addition to a labeled dataset D, (of o/ ()) J==1 we utilize an unlabeled dataset D, = {o o (1)) J==1 to optimize the reward model rg. Specifically, we generate the artificial labels ÿ by pseudo-labelie Sohn et al., 2020) for the unlabeled dataset D,. We infer preference for unlabeled segment pair (o (,) as a class with higher probability as follows:
(Lee, 2013;
[0, if Polo > o.l > 0.5
(ou onh
otherwise.
By generating labels from the prediction model, we can obtain free supervision for optimizing our reward model. However, pseudo-labels from low-confidence predictions can be inaccurate, and such noisy feedback can significantly degrade the peformance of preference-based learning (Lee et al., 202 To filter out inaccurate pseudo-labels, we only use unlabeled samples for training when the confidence of the predictor is higher than pre-defined threshold (Rosenberg et al., 2005). Then the reward model
rw is optimized by minimizing the following objective:
SSL
I (of of wo (o 10 D-
Rovard of - (o o 1) I(Pplo > r)). (3)
RRsard
where A= arg max V() is index of the preferred segment from the pseudo-label, À is hyperparameter that balances the losses, and is a confidence threshold. Training with the pseudo- labels encourages the model output more confident predictions on unlabeled samples. This can be seen as form of entropy minimization (Grandvalet & Bengio, 2004), which is essential to the success of recent SSL methods (Berthelot et al., 2019; 2020). The entropy minimization can improve the reward learning by forcing the preference predictor to be low-entropy (i.e., high-confidence) on
'The unlabeled dataset D2 is not constrained to fixed size since one can collect those unlabeled samples flexibly by sampling arbitrary pairs of experiences from the buffer.
4

---

Published as a conference paper at ICLR 2022
Algorithm 2 TDA: Temporal data augmentation for reward learning
Require: Minimum and maximum length Hain and Hras, respectively, for cropping
Require: Pair of segments (o".a") with length Hf
I: o" =(s8) ab) (s
2: o² = {(s), at) (s) 1. a
3: Sample H' from range of Hain Hanx]
4: Sample ko, k1 from range of [0, H H'
5: // RANDOMLY CROP A SEQUENCE WITH LENGTH IT'
6: σ" +- [(sk a8 ).).... (sk H1' -1 k
7: Ø² + {(s), ak, ).... (Sh, 4 +
8: Return (o", )
unlabeled samples. During training, we sample a larger minibatch of unlabeled samples than labeled ones by factor of y following (Sohn et al., 2020), since unlabeled samples with low confidence dropped within minibatch.
4.2 TEMPORAL DATA AUGMENTATION FOR REWARD LEARNING
To further improve the feedback-efficiency in preference-based RL, we propose a new data aug- mentation technique specially designed for reward learning. Specifically, for given two segments and preference (a", .y), we generate augmented segments (Φ), T y) by cropping the subse- quence from each segment (see Algorithm for more details).? Then, we utilize augmented sam- ples ( at) to optimize the cross-entropy loss in (3). The intuition behind the augmentation is that for a given pair of behavior clips, the human teacher may keep their relative preferences for slightly shifted or resized versions of them. In the context of SSL, data augmentation is also related to consistency regularization (Xie et al., 2019; Sohn et al., 2020) approaches that train the model to output similar predictions augmented versions of the same sample. Namely, this temporal cropping method enables our framework can also enjoy the benefits of consistency regularization.
5 EXPERIMENTS
We design experiments investigate the following:
How does SURF improve the existing preference-based RL method in terms of feedback effi- ciency?
What is the contribution of each of the proposed components in SURF?
How does the number of queries affect the performance of SURF?
Is temporal cropping better than existing state-based data augmentation methods in terms of feedback efficiency?
Can SURF improve the performance of preference-based RL methods when we operate on high-dimensional and partially observable inputs?
5.1
SETUPS
We evaluate SURF on several complex robotic manipulation and locomotion tasks from Meta- world (Yu et al., 2020) and DeepMind Control Suite (DMControl; Tassa et al. 2018; 2020), respec- tively. Similar to prior works (Christiano et al., 2017; Lee al., 2021a;b), in order to systemically evaluate the performance, we consider scripted teacher that provides preferences between two tra- jectory segments the agent according to the underlying reward function." Since preferences of the scripted teacher exactly reflects ground truth reward of the environment, one can evaluate the algorithms quantitatively by measuring the true return.
"The length of the cropped segment generated randomly because the preference predictor uses the accumulated sum of reward over time. While utilizing preferences from the human teacher is ideal, this makes hard to evaluate algorithms quan- titatively and quickly.
the hatch hut the for segment pairs,
5

---

Published as a conference paper at ICLR 2022
ground
gro
100
100
100
Success Rate (%)
75
75
75
Success Rate
50
50
50
Success
25
0
0.0
0.5
1.0
1.5
2.0
0.0
0.25
0.5
0.75
1.0
0.0
0.25
0.5
0.75
1.0
Environment Steps (x10%)
Environment Steps (x10%)
Environment Steps (x10*)
Hammer
(b) Door Open
(c) Button Press
SAC witt ground truth reward
SAC with ground truth reward FE2 SLE (loddlack=6000) SURF FEB BLE (codlacc=66000
SAC with ground truth reward FEBSLE [1oedback-400) + SUMF FEBSLE (oecbbcck4400
PEBBLE (ccdbacc=1000)) SUMF
PEBBLE Micdbacc--0000
100
100
100
Success Rate (96)
Success Rate (%)
Success (%)
50
50
50
25
25
25
0.0
0.25
0.5
0.75
1.0
0.0
0.25
0.5
0.75
1.0
0.0
0.1
0.2
0.3
0.4 0.5
Environment Steps (x109)
Environment Steps (x109)
Environment Steps (x10%)
(d) Sweep Into
(c) Drawer Open
Window Open
Figure 2:
Learning curves on robotic manipulation tasks as measured on the success rate. The solid line and shaded regions represent the mean and standard deviation, respectively, across five runs.
We remark that SURF can be combined with any preference-base based RL algorithms by replacing the reward learning procedure of its backbone method. In our experiments, we choose state-of-the- art approach, PEBBLE (Lee et al., 2021a), as our backbone algorithm. Since PEBBLE utilizes SAC (Haarnoja et al., 2018) algorithm to learn the policy, we also compare to SAC using the ground truth reward directly, as upper bound of PEBBLE and method. We note that our goal is not to outperform SAC, but rather to perform closely using as few preference queries as possible.
Implementation details of SURF, For all experiments, we use the same hyperparameters used by the original SAC and PEBBLE algorithms, such as learning rate of neural networks and frequency of the feedback session. For query selection strategy, we use the disagreement baaegreeeeet-beeed sampling scheme, which selects queries with high uncertainty, i.e., ensemble disagreement (see Appendix B for more details). At each feedback session, we sample unlabeled samples as 10 times of labeled ones by uniform sampling scheme, unless otherwise noted. Although we only use such amount of unlabeled samples for time-efficient training, we note that one can utilize much more unlabeled samples as needed. For the hyperparameters of SURF, we fix the loss weight À 1, and unlabeled batch ratio 14 for all experiments, and use threshold parameter r 0.999 for Window Open, Sweep Into, Cheetah tasks, and r = 0.99 for the others. We provide more experimental details in Appendix B.
Extension visual control tasks. To further demonstrate the effectiveness of our method, we also provide experimental results on visual control tasks, where each observation is an image of 84 x 84 x 3. Specifically, we choose DrQ-v2 (Yarats et al., 2022), a satt--ofthhe-at pixel-based RL. approach on DMControl, with state-based inputs, we also compare to DrQ-v2 using ground truth reward as an upper bound.
backbone algorithm for PEBBLE and SURF. Similar experiments
5.2 BENCHMARK TASKS WITH SCRIPTED TEACHERS
Meta-world experiments. Meta-world consists of 50 robotic manipulation tasks, which are de- signed for learning diverse manipulation skills. We consider six tasks from Meta-world, to investi- gate how SURF improves a preference- learning method on a range of complex robotic ma- nipulation tasks (see Figure 7 in Appendix B). Figure 2 shows the learning curves of SAC, PEBBLE

---

Published as a conference paper at ICLR 2022
ground reward fTze:backk-10 (teedbach-100)
SAC with ground buth FOMArO
ground
PEBBLE (fecsbadk-100
(lectbadk-100)
1,000
1,000
1,000
Episode Return
750
Episade Return
750
Episode Return
750
500
500
500
250
0
0.1
0.3
0.4
0.5
0.0
0.25
0.5
0.75
1.0
0.25
0.5
0.75
1.0
Environment Steps (x109)
Environment Steps (x110f)
Environment Steps (x10F)
(a) Walker
(b) Cheetah
(c) Quadruped
Learning curves on locomotion tasks as measured on the ground truth reward. The solid line and shaded regions represent the mean and standard deviation, respectively, across five runs.
Figure 3:
and SURF (which combined with PEBBLE) on the manipulation tasks. In each task, PEBBLE and SURF utilize the same number of preference queries for fair comparison. As shown in the figure, SURF significantly improves the performance of PEBBLE given the same number of feedback on all tasks considered, and matches the performance of SAC using the ground truth reward on four tasks. For example, we find that when using 400 preference queries, SURF (red) reaches the same performance as SAC (green) while PEBBLE (blue) far behind SAC on Window Open task. We also observe that SURF achieves similar performance to PEBBLE with much less labels. For example, to achieve comparable performance to SAC on Window Open task, PEBBLE needs 2,500 queries (reported in (Lee et al., 2021a)), requiring about 6 times more queries than SURF. These results demonstrate that SURF significantly reduces the feedback required to solve complex tasks.
DMControl experiments. For locomotion tasks, we choose three complex environments from DMControl: Walker-walk, Cheetah-run, and Quadruped-walk. Figure 3 shows the learning curves of the algorithms with same number of queries. We find that using budget of 100 or 1,000 queries (which takes only few human minutes), SURF (red) could significantly improve the performance of PEBBLE (blue). These results again demonstrate that that SURF improves the feedback-efffciency of preference- based RL methods on a variety of complex tasks.
5.3 ABLATION STUDY
Component analysis. To evaluate the effect of each technique in SURF individually, we incremen- tally apply semi-supervised learning (SSL) and temporal cropping (TC) to our backbone algorithm, PEBBLE. Figure 4a shows the learning curves of SURF on Walker-walk task with 100 queries. We observe that leveraging unlabeled samples via pseudo-labeling (green) significantly improves PEBBLE, in terms of both sample-effciency and asymptotic performance, while standard PEBBLE (blue) suffers from lack of supervision. In addition, both supervised (blue) and semi-supervised (green) reward learning are further improved by additionally utilizing temporal cropping (purple and red, respectively). This implies that our augmentation method improves label-efficiency by gen- erating diverse behaviors share the same labels. Also, the results show that the key components of SURF are both effective, and their combination is essential to our method's 's success. We also provide extensive ablation studies on Meta-world, which show similar tendencies in Appendix C.
Effects of query size. To investigate how the number of queries affects the performance of SURF, we evaluate the performance of SURF with varying number of queries N E (50. 100, 200, 400). As shown in Figure 4b, SURF (solid lines) consistently improves the performance of PEBBLE (dot- ted lines) across a wide range of query sizes. The gain from SURF becomes more significant in the extreme label-scarce scenarios, i.c., N E [50, 1003.
Comparison to other augmentation for state-based inputs. To demonstrate that temporal crop- ping can enduce significant improvements for reward learning, we compare our method to other augmentation methods for state- base- inputs. We consider random amplitude scaling (RAS) and adding Gaussian noise (GN) proposed in Laskin et al. (2020) as our baselines. RAS multiplies an uniform random variable to the state, i.e., $ S 2, where Unif[a, 1). and GN adds

---

Published as a conference paper at ICLR 2022
PESELE SURP (SSL:
Poeet dhockarke 6R0
-2144
Ierppoon Cropp
PEIRLE SURT (SS: TC:
FOOSLE floctauee330
-2377
Nokio
PERBLE SURT (SSL: X, O)
PokelE (hoclarie S2)
+214
FEBOLE
PEBBLE
750
500
Episode Return
Episode Return
750
Episode Return
500
500
250
250
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.4 0.5
Environment Steps (x10%)
Environment Steps (x10*)
Environment Steps (x10*)
(a) Contributions of each component
(b) Query size
(c) Effects of data augmentation
Figure 4: Ablation study on Walker-walk. (a) Contribution of each technique in SURF, i.e., semi- supervised learning (SSL) and temporal cropping (TC). (b) Effects of query size. (c) Comparison of augmentation methods. The results show the mean and standard deviation averaged over five runs.
FEBRAL SUP f1=11 11
FEBBLE + SURF (U..(:
PEBBLE SURF
PERBLE SURR (um
PERBLE SURP (1
PEBALE SURF 0.5)
PESBLE
SURP 4)
PEBBLE SUNFIt
PEBBLE SURF
1,000
PESELE
SUMP ju- 71
1,000
PEBBLE SSUR
1,000
PEDBLE SURT (A-
Episade Retur
750
Episode Return
750
Episade Return
750
500
250
250
250
0
0
0.0
0.1
0.2
0.3
0.4 0.5
0.0
0.1
0.2
0.4
0.5
0.0
0.1
0.2
0.3 0.4 0.5
Envirenment Steps (x10%)
Environment Steps (x10%)
Environment Steps (x10%)
(a) Unlabeled batch ratio μ
(b) Threshold parameter
(c) Loss weight À
Figure 5: Hyperparameter analysis on Walker-walk using 100 preference queries. The results show the mean and standard deviation averaged over five runs.
a multivariate Gaussian random variable z to the state, i.e., s = s z, where z N I). As proposed in Laskin et al. (2020), we apply these methods consistently along the time dimen- sion, and choose the parameters for RAS as a = 0.8, 3 = 1.2. Specifically, for a given segment obtain the augmented sample ē by perturbing each
= ({(s), ..., (Sk+ a state along the segment, i.c., δ = ({(Sk, a (S a H+))-))}) In Figure 4c, we plot the learning curves of PEBBLE with various data augmentations on Walker-walk task with 100 queries. We observe that RAS improves the performance of PEBBLE, but temporal cropping still outper- forms these two methods. GN degrades the performance, possibly due the noisy inputs, Since RAS is an orthogonal approach to augment state-based inputs, one can integrate them with our method to further improve the performance. This may be an interesting future direction for address- ing
in preference- based RL (see Appendix C).
Effects of hyperparameters of SURF. We investigate how the hyperparameters of SURF affect the performance of prference-based RL. In Figure 5, we plot the learning curve of SURF with different set of hyperparameters: (a) unlabeled batch ratio μı E (1, 2, 4, 7). (b) threshold parameter E (0.95, 0.97, 0.99, 0.999]. and (c) loss weight X E (0.1, 1,2). respectively. First, we observe that SURF is quite robust on μa, but the performance slightly drops with large batch size 7. We expect that this is because large batch size makes the reward model overfit to unlabeled data. We also observe that SURF is also robust on the threshold т, except for the smallest value of 0.95. Because there are only two classes in tasks, the optimal threshold could larger than the value typically used in previous SSL methods (Sohn et al., 2020), i.c., 0.95. In the case of the loss weight X, tuning this parameter brings more improvements than other hyperparameters. Although
8

---

Published as a conference paper at ICLR 2022
Dro-2 with ground Eruth roward
prc-22 wit ground ruth spward
PEESLE [ood SURP
FESELE [[cosbast=1201 SUM
PEESLE Hoodoad-1009)
FESBLE (fcolbass-1000
1,000
1,000
1,000
Episode Return
750
Episade Return
750
Episode Return
750
500
500
500
250
250
250
0.75
1.0
0.0
0.25
Number Frames (x10%)
Number (x10f)
Number Frames (x10*)
(a) Walker
(b) Cheetah
Quadruped
Figure 6: Learning curves on locomotion tasks with pixel-based inputs as measured on the ground truth reward. The solid line and shaded regions represent the mean and standard deviation, respec- tively, across five runs.
we use a simple choice, i.., À 1, in our experiments, more tuning À would further improve the performance of our method.
5.4 EXPERIMENTS ON VISUAL CONTROL TASKS
Figure shows the learning curve DrQ-v2, PEBBLE, and SURF with the same number of queries. We observe that SURF (red) significantly improves the performance of PEBBLE (blue). In particu- lar, SURF achieves comparable performance to DrQ-v2 (green) with ground truth reward in Walker- walk, only using budget of 200 queries. These results demonstrate that SURF could also improve the performance with image observations. We remark that our temporal cropping augmentation can be combined with any existing image augmentation methods, which would be interesting future direction to explore.
DISCUSSION
In this work, we present SURF, a semi-supervised reward learning algorithm with data augmentation for preference-based RL. First, in order utilize unlimited number of unlabeled data, utilize pseudo-labeling on confident samples. Also, to enforce consistencies to the reward function, we propose new data augmentation method called temporal cropping. Our experiments demonstrate that SURF significantly improves feedback-efficienccy of current state-of-the-al method on a variety of complex robotic manipulation and locomotion tasks. We believe that SURF scale up deep RL to more diverse and challenging domains by making preference- based learning more tractable.
An interesting future direction is to extend state-based inputs partially-observable or high- dimensional inputs, e.g., pixels. One can expect that representation learning based on unlabeled samples and data augmentation (Chen et al., 2020a; Grill et al., 2020) crucial handle such in- puts. We think that our investigations on leveraging unlabeled samples and data augmentation would be useful in representation learning for preference-based RL.
Ethics statement. Preference-basee RI can align RL agents with the teacher's preferences, which enables us to apply RI. to diverse problems and obtain strong AI. However, there could possible negative impacts if malicious user corrupts the preferences to teach the agent harmful behav- iors. Since we have proposed a method that makes preference-basee efficiently, method may reduce the efforts for teaching not only the desirable behaviors, but also such bad behaviors. For this reason, in addition to developing algorithms for better performance and efficiency, it is also important consider safe adaptation in the real world.
RL algorithms more feedback-
Reproducibility statement. We describe the implementation details of SURF in Appendix B, and also provide source code in the supplementary material.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS AND DISCLOSURE OF FUNDING
This work was supported by Samsung Electronics Co., Ltd 10201211108110-001 OpenPhilosophy, and Institute of Information Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0- 2019-0000075, Artificial Intelligence Graduate School Program (KAIST)). We would like thank Junsu Kim and anonymous reviewers for providing helpful feedbacks and suggestions in improving our paper.
REFERENCES
Riad Akrour, Mare Schoenauer, and Michele Sebag. Preference- policy learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 2011.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Con- crete problems in ai safety. arXiv preprint arXiv: (606.06565, 2016.
OpenAl: Marcin Andrychowicz, Bowen Baker, Maciek Chociej. Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
Mare G Bellemare, Salvatore Candido, Pablo Samuel Castro, Jun Gong, Marlos Machado, Sub- hodeep Moitra, Sameera S Ponda, and Ziyu Wang. Autonomous navigation of stratospheric bal- loons using reinforcement learning. Nature, 588(7836):77-82, 2020.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung. Przemystaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, al. Dota with large scale deep reinforcement learning. arXiv preprint 1912.069)) 2019.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. In Advances in Neural Information Processing Systems, 2019.
David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution matching and augmenta- tion anchoring. In International Conference on Learning Representations, 2020.
Erdem Biyik and Dorsa Sadigh. Batch active preference-based learning of reward functions. In Conference on Robot Learning, 2018
Erdem Biyik, Nicolas Huynh, Mykel J Kochenderfer, and Dorsa Sadigh. Active preference-based gaussian process regression for reward learning. In Robotics: Science and Systems, 2020.
Ralph Allan Bradley and Milton E Terry. Rank analysis incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self- supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006.1000 10029, 2020b.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, 2017.
Karl Cobbe, Oleg Klimov, Chris Hesse, Tachoon Kim, and John Schulman. Quantifying generaliza- tion in reinforcement learning. International Conference on Machine Learning, 2019.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment: Learning augmentation policies from data. arXiv preprint 1805v:1805.09550 2018.
10

---

Published as a conference paper at ICLR 2022
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical data aug- mentation with no separate search. arXiv preprint 1rXiv:1909.1377 2019.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In Ad- vances Neural Information Processing Systems, 2004.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- on Machine Learning, 2018.
Dylan Hadfield-Menel, Smitha Milli, Pieter Abbeel, Stuart Russell, and Anca Dragan. Inverse reward design. In Advances Neural Information Processing Systems, 2017.
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. In Advances in Neural Information Processing Systems, 2018.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. Conference on Robot Learning. 2018.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference on Learning Representations, 2015.
Jens Kober and Jan Peters. Policy search for motor primitives in robotics. Machine learning, 84 (1-2):171-203, 2011.
Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(11):1238-1) 2013.
Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Rein- forcement learning with augmented data. In Advances in Neural Information Processing Systems, 2020.
Dong-Hye Hyyn Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. ICML Workshop, 2013.
Kimin Lee, Kibok Lee, Jinwoo Shin, and Honglak Lee. Network randomization: A simple technique for generalization deep reinforcement learning. arXiv preprint arXiv:1910.05300 2019.
Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. International Conference on Machine Learning, 2021a.
Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel. B-pref: Benchmarking preference- based reinforcement learning. In Thirty-fifth Conference Neural Information Processing Syy- tems Datasets and Benchmarks Track (Round 1), 2021b. URL httptttps///peeeeeeeeee nett/ forum? 1awtummdd-ps9mmmmemmm
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: research direction. arXiv preprint arXiv:1811.078 2018.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and superiseeee learning IEEE transactions on pattern analysis and machine intelligence, 1979,1997111993, 2018.

---

Published as a conference paper at ICLR 2022
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforeement learning, Nature, 518(7540):529, 2015.
Pierre- Yvess Oudeyer, Frdric Kaplan, and Verena V Hafner. Intrinsic motivation systems for au- tonomous mental development. IEEE transactions on evolutionary computation, 11(2):6552866, 2007.
Chuck Rosenberg, Martial Hebert, and Henry Schneiderman. Semi-supervised self-training of ob- ject detection models. In 2005 Seventh IEEE Workshops on Applications of Computer Vision (WACVMOTION'05) Volume 1, volume 1, pp. 29-36, 2005. doi: 10.1 A0.1000//C 107.
Dorsa Sadigh, Anca D Dragan, Shankar Sastry, and Sanjit A Seshia. Active preference-based learn- ing of reward functions. In Robotics: Science and Systems, 2017.
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor- mations and perturbations for deep sumi-supervised learning. In Advances in Neural Information Processing Systems, 2016.
Jürgen Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2010). IEEE Transactions on Autonomous Menial Development, 2(3):230-24) 24700022244220 2010.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz, Trust region policy optimization. In International Conference on Machine Learning. 2015.
David Silver, Julian Schrittwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 35047676):354, 2017.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Artthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through 1144, 2018
Science, 362(6419)) 1140
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in Neural Information Processing Systems, 2020.
Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representa- tions for reinforcement learning. In International Conference on Machine Learning, 2020.
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. arXiv preprint arXiv:2009.00 0255, 2020.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT Press, 2018.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, al. Deepmind control suite. arXiv preprint arXiv:1801.00660 2018.
Yuval Tassa, Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, and Nicolas Heess. dm_control: Software and tasks for continuous control. arXiv preprint arXiv:2006. 12983, 2020.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In International Conference on Intelligent Robots and Systems, 2012.
Oriol Vinyals, Igor Babuschkin, Wojeiech Czarnecki, Michael Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 375(7782):350-354 2019.
Aaron Wilson Alan Fern, and Prasad Tadepalli. bayesian approach for policy learning from trajectory preference queries. In Advances in Neural Information Processing Systems, 2012.
12

---

Published as a conference paper at ICLR 2022
Jeff Wu, Long Ouyang, Daniel M Ziegler, Nissan Stiennon, Ryan Lowe, Jan Leike, and Paul Chris- tiano. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109. 10862, 2021.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. Unsupervised data augmentation for consistency training. arXiv preprini arXiv:1904 12848, 2019.
Denis Yarats, llya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference Learning Representa- lions, 2021.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous trol: Improved data-augmented reinforcement learning. In ICLR, 2022.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, 2020.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
13

---

Published as a conference paper at ICLR 2022
A PEBBLE ALGORITHM
state-of-the-art preference-bane based RL. algorithm, PEBBLE (Lee et al., 2021a), consists two main unsupervised pre-training and relabeling experiences. To collect diverse experience, PEBBLE pre-trains the policy by using intrinsic motivation (Oudeyer et al., 2007; Schmidhuber, 2010) in the beginning training. Specifically, PEBBLE optimizes the policy to maximize the state entropy H(s) -Ex-p(s) [log p(s) to efficiently explore the environment. Then PEBBLE learns the policy by using the state-of-the-art off ooflcy RL algorithm, SAC (Haarnoja et al., 2018). Since the learning process of off-policy algorithms with non-stationary reward function can be unstable, PEBBLE E stabilizes the learning process by relabeling all experiences the buffer when the reward model is updated.
components:
B EXPERIMENTAL DETAILS
(a) Hammer
(b) Door Open
(c) Button Press
(d) Sweep Into (c) Drawer Open (f) Window Open
Figure 7: Rendered images of robotic manipulation tasks from Meta-world. Our goal is learning various locomotion and manipulation skills using preferences from a teacher.
Training details. We choose PEBBLE (Lee et al., 2021a) as backbone algorithm of SURF, and use the hyperparameters in Table I for both PEBBLE and our method. For the reward model, we use three-layer MLP with 256 hidden units and leaky ReLU activation. Following the implementation of PEBBLE (Lee et al., 20)1a), we use an ensemble of three reward models and bound the output [- using tanh function. Each model trained by minimizing the cross-entropy loss using ADAM optimizer (Kingma & Ba, 2015) with the learning rate of 0.0003. For semi-supervised learning and data augmentation of SURF, we use hyperparameters in Table 2.
Sampling schemes. In preference- RL methods. informative query sampling (Biyik & Sadigh, 2018: Biyik al., 2020; Sadigh al., 2017) has been adopted for improving the feedback-effiiieney. For all experiments of PEBBLE and SURF, we use the disagreement- based sampling (Christiano al., 2017) to choose queries for labeling: we first uniformly sample the initial batch of segments, and select Nquery pairs of segments' with high uncertainty based on the variance across ensemble of
preference predictors {Px [a- o ]"m Note that we use uniform sampling scheme for unlabeled samples, because the number of unlabeled samples are not limited. At each feedback session, we sample unlabeled samples as 10 times of labeled ones if the maximum budget of feedback equal or larger than 1,000, and otherwise we sample unlabeled samples as 100 times of labeled ones.
Table 1: Hyperparameters of PEBBLE.
Hyperparameter
Value
Hyperparameter
Value
Initial temperature Length of segment
0.1
Hidden units per each layer
1021 (DMControl) 256 (Meta-world)
50
# of layers
2 (DMControl)) 3 (Meta-world)
Learning rate
0.0003 world)
Batch Size
1024 (DMControl), (Mela-world) Adam (Kinggma Ba, 2015)
0.0005 (Walker, Cheetah)
Optimizer
0.0001 (Quadruped)
Critic target update freq (f1.82) Frequency of feedback
2
Critic EMA
0.005
(0.9. 0.999)
Discount 5
0.99
5000 (Meta-world)
Maximum budget /
1000/100, /100, 100/10 (DMControl)
20000 (Walker, Cheetah)
queries per session
10000/50, 4000/20 (Meta-worid)
30000 (Quadruped)
2000/25, 400/ (Meta-world)
# of ensemble models No..
# of pre-truining steps
10000
"We select 10% of the initial batch.
14

---

Published as a conference paper at ICLR 2022
Table 2: Hyperparameters of SURF with state-based inputs.
Hyperparaeeter
Value
Unlabeled batch rutio Threshold
4
0.999 (Window Open, Sweep Into. Cheetah)
0.99 (others)
Loss weight
Min/Max length cropped segment Mates Haul
53]
Segment length before cropping
60
Hyperparameters of DrQ-v2. We use the same encoder architecture and hyperparameters as in DrQ-v2 (Yarats et al., 2022). The full list of hyperparameters of DrQ-v2 is presented in Table 3. For semi-supervised learning and data augmentation of SURF with pixel-based inputs, we use hyperpa- rameters in Table 4.
Table 3: A set of hyperparameters used in our experiments.
Parameter
Setting 10"
Replay buffer capacity
Action repeat
Seed frames
4000
Exploration steps
n--tepp returns
(Walker), (Cheetah Quadruped)
Mini-hatch
512 (Walker), 256 (Cheetah Quadruped) 0.99
Discount 7
Optimizer
Adam
Leariing rate
10-4
Agent update frequency
Critic Q-function soft soft-update rate
0.01
Features dim.
50
Hidden dim.
1024
Exploration stddev. clip
0.3
Walker: linear(1.0, 0.1, 100000)
Exploration stddev. schedule
Cheetah / Quadruped: linear(1.0) 0.1, 500000)
Table Hyperparameters of SURF with pixel-based inputs.
Hyperparameter
Value
Unlabeled batch ratio μ
Threshold
0.99
Loss weight X
(Cheetah), 0.1 (others)
Min/Max length cropped segment [Hamn Howr]
[45, 55) (Chectah). [48,52] (others)
Segment length hefore cropping
60 (Cheetah). 54 (oihers)
Implementation. We implement SURF using the publicly released implementation repository the PEBBLE algorithm (attps://gt) thub. com. pokaxpoka/3 Pref) with full list of hyper- parameters in Table 1. For visual control tasks, we implement PEBBLE and SURF using the official code of DrQ-v2 (https: / com/ facebook research /ct/drgv/// 2 with hyperparam- eters in Table 3. Note that DMControl environment depends on the MuJoCo simulator (Todorov al., 2012), which is a commercial software. We follow the standard evaluation protocol for the locomotion tasks from DMControl. For robotic manipulation tasks from Meta-worid, we measure the task success rate as defined by the authors. For each run of experiments, we utilize one Nvidia RTX 2080 Ti GPU and 4 CPU cores for training.
C ADDITIONAL EXPERIMENTAL RESULTS
Ablation study on Meta-world. We provide additional experimental results for component analysis on Meta-world (Yu et al., 2020). To evaluate the effect of each technique in SURF individually, we incrementally apply semi-supervised learning (SSL.) and temporal cropping (TC) our backbone
15

---

Published as a conference paper at ICLR 2022
algorithm, PEBBLE. Figure 8a and 8b show the learning curves of SURF on Window Open with 400 queries and Hammer with 10,000 queries, respectively. We observe that both cumissupeeeieed learning (green) and data augmentation (purple) improve the baseline of PEBBLE (blue). Also, applying both of them further improves the performance (red). This shows that the key components of SURF are both effective.
Applying temporal cropping with other augmentations. In Section 5.3, we compared our method random amplitude scaling (RAS) and adding Gaussian noise (GN) proposed in Laskin al. (2020). To investigate if applying RAS or GN with the temporal cropping (TC) further improve the performance, we provide experimental results for the joint usage of the augmentations. In Figure 9a, we plot the learning curves of PEBBLE with various data augmentations on Walker-walk with 100 queries. We observe that the naive combination of two augmentations, i.e., RAS + TC and GN + TC, do not further improve the performance. Investigating how to combine several data augmentation methods for reward learning would be an interesting future direction.
Effects of augmentation intensity. To investigate how does augmentation intensity affects the RL performance, we provide additional experimental results. In Walker-walk with 100 queries, we apply the temporal cropping to PEBBLE, with varying cropping range, i.e., (Huaz Hain)/2, /2, from an array of [2, 5, 10, 20). Figure 9b shows that temporal cropping consistently improves the performance of the PEBBLE. Although the performance with large cropping range of 20 slightly underperforms the performance with 10, these results show that our augmentation method is quite robust the choice of hyperparameters.
O,
(SSL:
PEBBLE SURF (SSL O, TC: X)
PEBBLE SURF (SSL: O. TC: XI
PEBBLE SURF (SSL: X, TC: O)
PEBBLE SURF (SSL: X, TC: O)
PEBBLE
PEBBLE
100
100
Success Rate (%)
Success Rate (%)
75
75
50
50
25
25
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.5
1.0
1.5
2.0
Environment Steps (x10*)
Environment Steps (x10%)
(a) Ablation study on Window Open
(b) Ablation study Hammer
Figure 8: Contribution each technique in SURF, i.e., semi-supervised learning (SSL. and temporal cropping (TC), in (a) Window Open, and (b) Hammer. The results show the mean and standard deviation averaged over five runs.
16

---

Published as a conference paper at ICLR 2022
PEBBLE GN
PEBBLE (crop 201
PEBBLE + TC
(crop
(crop
PEBBLE GN
PESBLE (crop PEBBLE
PEBBLE
750
Episode Return
500
Episode Return
250
250
0
0.0
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0.5
Environment Steps (x10%)
Environment Steps (x10%)
(a) Joint usage of the augmentations
(h) Effect of augmentation intensity
Figure 9: Analysis of the augmentation methods on Walker-walk task with 100 queries. (a) Com- parison of the augmentation methods, i.c., applying random amplitude scaling (RAS) or Gaussian noise (GN) with temporal cropping (TC). (b) Comparison of the performance of PEBBLE with vary- ing augmentation intensity, i.e., cropping range. The results show the mean and standard deviation averaged over five runs.
17


Reviews:
[{"review_id": "dVycTQqtO7s", "reviewer": "Reviewer_JjVp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors use a preference-based reinforcement learning method, which is based on a surrogate reward model. The surrogate reward model is learned by phrasing the preference learning problem as classification over pairwise comparisons. The semi-supervised learning approach uses the preference classifier for deriving high-confidence pseudo labels, that can be used as additional, supervised input data.\nBoth are well known and established principles, making the proposed method a sound contribution. However, as the bridge between classification and PBRL, as well as classification based SSL, are already well known, this is a straight forward application of known methods to a new problem domain, limiting the novelty. The explanation and formalization of this section is good and clear to understand, with one small exception: Two different preference notations are used: (0,1) and >. The first notation is also used for introducing an equivalence relation, which is not used otherwise. This should be unified to the more common > notation (for further preference relation symbols, see https://www.jmlr.org/papers/volume18/16-634/16-634.pdf). An additional, potential improvement is possible by using the tau threshold for defining a confidence cutoff between preference and equivalence relation. However, both are minor issues and do not impact clarity.\n\nSecond, the authors introduce a novel data augmentation scheme, by selecting random subsequences out of the observed trajectories, which is also clearly understandable. This method has the implicit assumption, that the reason (states) for the observed preference are still part of the shorter sequence. Especially, in sparse reward domains, this is not a given.\n\nThe main evaluation is good, and sufficiently diverse. Advantages of the proposed method are clearly visible. Adding some non-robotics (potentially high dimensional) domains could improve the paper further, but this was also clearly stated by the authors. The given details (including appendix), should be sufficient for recreating the experiment setup.\n\nThe ablation study is a welcome addition and the authors covered all relevant aspects, meaning not just the two parts of the introduced method, but also several other factors (and hyperparameters) that can potentially have a substantial impact on the results. However, the ablation study is limited to a single domain, limiting the generality of the derived statements. Especially, i consider the advantage of TDA not sufficiently supported, considering that walker-walk is not a sparse reward domain.", "summary_of_the_paper": "The authors introduce a method for combining semi-supervised learning with preference learning for reinforcement learning. The method uses pseudo labels, derived from a surrogate preference model. Furthermore, a data augmentation scheme for this setting is introduced, which is based on cropped sequences. Evaluation is performed on a range of robotics tasks, including ablation studies for several hyperparameters and the two, distinct parts of the introduced method.", "summary_of_the_review": "The proposed method is sound and interesting, but of limited novelty. The support for the full method is good, but without an extended ablation study, the support for TDA is not sufficient. The method is likely significant within the area of PBRL, but but not for a broad audience. The paper is clearly written and can be easily understood. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AgYZdJWVI3s", "reviewer": "Reviewer_cS2w", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper aims to tackle an important problem in reinforcement learning, i.e., how to learn a policy without accessing a fine-grained reward function. More specifically, this paper focuses on preference-based RL that uses trajectory-wise comparison queries to guide the learning procedure. The paper is well organized and easy to follow. The proposed method is motivated by the success of data augmentation in the supervised learning area. It is interesting to see a connection between reinforcement learning and supervised learning.\n\nSome questions:\n\n- As shown in figure 6(b), the pseudo-labeling technique requires the hyper-parameter \\tau to be a very large value. Why do we need to train our reward model on these sample pairs with extremely high confidence? Please correct me if I misunderstand anything. The loss computed by these high-confidence samples should be very small. Why it can provide a significant improvement on the final performance?\n\n- RAS and GN presented in Figure 6(c) should be orthogonal to Temporal Cropping. I am curious whether we can further improve the performance by applying RAS and/or GN upon SURF.\n\n- The ablation study is only conducted in one specific environment. It would be helpful to repeat these experiments in other tasks.\n\nMinor comments:\n\n- In section 3, regarding the notations of $\\mathcal{L}^{CE}$, the terms $y(0)$ and $y(1)$ have not been defined.\n\n- Algorithm 1 (TDA) is presented in page 4, which is far from section 4.2. It might be better to reverse the order of Algorithm 1 and 2 for presentation.\n", "summary_of_the_paper": "This paper proposes two data-augmentation techniques to improve the query efficiency of preference-based RL. (1) Pseudo-labeling leverages unlabeled data by using high-confidence predictions as labels. (2) Temporal cropping augmentation generates imaginary comparisons by cropping trajectories.\n\nThe proposed augmentation can significantly improve the query efficiency of preference-based RL in a variety of benchmark tasks, including DMControl and Meta-World. The performance of SURF is comparable to dense-reward SAC while only accessing a few expert queries.", "summary_of_the_review": "My recommendation score is marginally above the borderline. I like this paper overall but the discussions of ablation study need to be enriched.\n\nThe methodology introduced by this paper is inspired by related studies in supervised learning. I think it is quite important to understand why and how these techniques benefit RL algorithms.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5vCbjzgBecC", "reviewer": "Reviewer_qrsj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\ns1) The paper proposes a semi-supervised reward learning pipeline aiming to reduce reward engineering efforts, which is an important topic.\n\ns2) Extensive evaluation experiments are conducted to demonstrate the proposed approach.\n\nWeaknesses:\n\nw1) In Fig.1(b), the figure of temporal cropping seems incorrect. According to the description of algorithm 1, the red box should be within the blue box. In the second line, the first image in the red box is outside the blue box.\n\nw2) In Sec 3. “Preference-based reinforcement learning,” “Then, we model a preference predictor using the reward function $\\hat{r}_{\\psi}$ following the Bradley-Terry model.” It’s not clear to the reviewer how the reward function is learned. “Specifically, given a dataset of preferences D, the reward function is updated by minimizing the binary cross-entropy loss.” Does the reward function have extra constraints? Assume $\\psi$ is the reward function that satisfies the Eqn (1). It seems that there are infinite equivalent reward functions ($\\psi$ + arbitrary constant number) if the length of segments (H) is the same.   \n\nw3) In Sec 4.1, the same question as the above question 2) w.r.t. the learning process of the reward function through the loss of Eqn(3). If multiple seeds are used, do these reward functions generate similar results on the samples? \n\nw4) In Sec 4.2, “​​The intuition behind the augmentation is that for a given pair of behavior clips, the human teacher may keep their relative preferences for slightly shifted or resized versions of them.”  The assumption is too strong and does not generally hold. It will actually strengthen the paper if more analyses are provided. What would happen in situations when the assumption does not hold?\n\n", "summary_of_the_paper": "This work introduces a semi-supervised reward learning approach to reduce the efforts of reward engineering, which contains two key components. The first is to produce artificial labels for these unlabeled samples leveraging the pseudo-labeling and the learned reference predictor. The second component is to crop consecutive subsequences for data augmentation. The proposed approach is tested on Meta-world and DMControl suites, and the results show that it significantly improves the performances.\n", "summary_of_the_review": "This work introduces a semi-supervised reward learning approach to reduce the efforts of reward engineering. However, the technical novelty is limited. Some assumptions made by this work are too strong and may not hold. Some important descriptions (e.g., the reward function w2 and w3 in Main Review) are missing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The topic of learning reward functions from preferences and how to do this efficiently is of high interest to the ML/RL community. All reviewers appreciate the suggested technical approach and the thorough evaluations that demonstrate clear improvements. While the technical novelty of the paper is not entirely compelling, all reviewers recommend acceptance of the paper.
Decision: Accept (Poster)