Beyond Reward: Offline Preference-guided Policy Optimization
Authors: Yachen Kang
Diyuan Shi
Jinxin Liu
Li He
Donglin Wang
Venue: Submitted to ICLR 2023 (2023)
Keywords: offline reinforcement learning
preference-based reinforcement learning
hindsight information matching
preference-guided policy optimization

Abstract: In this work, we study offline preference-based reinforcement learning (PbRL), which relaxes the two fundamental supervisory signals in standard reinforcement learning (online accessible transition dynamics and rewards). In other words, the agent is provided with fixed offline trajectory transitions and human preferences between pairs of trajectories. Due to the orthogonality property of rewards and dynamics, one common practice is combining prior PbRL-based reward learning objectives with off-the-shelf offline RL algorithms to bridge preference modeling and offline learning. However, such two isolated optimizations require learning a separate reward function and thus place an information bottleneck on reward learning (the bridge). As an alternative, we propose offline preference-guided policy optimization (OPPO), an end-to-end offline PbRL formulation, which jointly learns to model the preference (for finding the optimal task policy) and the offline data (for eliminating OOD). In particular, OPPO introduces an offline hindsight information matching objective and a preference modeling objective. Then, iterating the two objectives over, we can directly extract a well-performing decision policy, avoiding a separate reward learning. We empirically show that OPPO can effectively model the offline preference and outperform prior competing baselines (including the offline RL algorithms performed over the true reward function).

Under review as a conference paper at ICLR 2023
BEYOND REWARD: OFFLINE PREFERENCE-GUIDED POLICY OPTIMIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
In this work, we study offline preference- based reinforcement learning (PbRL). which relaxes two fundamental supervisory signals in standard reinforcement learning (online accessible transition dynamics and rewards). In other words, the agent is provided with fixed offline trajectory transitions and human preferences between pairs of trajectories. Due to the orthogonality property of rewards and dynamics, one common practice is combining prior PbRL-based reward learning objectives with off-the-shell offline RL algorithms to bridge preference modeling and offline learning. However, such two isolated optimizations require learning separate reward function and thus place an information bottleneck reward learning (the bridge). As an alternative, we propose offline preference-guide icy optimization (OPPO), end-to-end offiin PbRL formulation, which jointly learns to model the preference (for finding the optimal task policy) and the offline data (for eliminating OOD). In particular, OPPO introduces an offline hindsight information matching objective and preference modeling objective. Then, by it- erating the two objectives over, we can directly extract a perfoorrnonnigg decision policy, avoiding separate reward learning. We empirically show that OPPO can effectively model the offline preference and outperform prior competing baselines (including the offline RL algorithms performed over the true reward function).
pol-
INTRODUCTION
Deep reinforcement learning (RL.) presents flexible framework for learning task-oriented behav- iors (Kohl& Stone 2004
Kober & Peters| 2008, Kober al. 2013 Silver al. 2017 Kalashnikov et al. 2018 Vinyals et al. 2019), where the "task" is often expressed as maximizing the cumulative reward sum trajectories generated by rollouting the learning policy the corresponding environ- ment. However, the above RL formulation implies two indispensable prerequisites for the training of decision policy: 1) an interactable environment and 2) a pre-specified reward function. Unfortu- nately, 1) online interactions with the environment
be costly and unsafe (Mihatsch & Neuneier 2002 Hans et al. 2008; Garcia & Fernández) 2015), and 2) designing suitable reward function of- requires expensive human effort, while the heuristic rewards often used are sometimes incapable of conveying the true intention MoadficldMennee et al. 2017).
To relax these requirements, previous works have either 1) focused on the offline RL formulation (the online rollout is inaccessible) (Fujimoto et al., 2019), where the learner has access to fixed of- fline trajectories along with a reward signal for each transition (or along with limited expert demon- strations), or 2) considered the (online) preference-ber based RL (PbRL) formulation (Christiano et al., 2017 Brynk& & Sadigh 2018 Sadigh et al.) 2017 Biyik et al. 2020 Lee et al.) 2021a), where mes- sages about the task objective are passed to the learner through preferences of a (human) annotator between two trajectories rather than rewards for each transition. To further progress in this setting,
propose relaxing both requirements and advocating for offline PbRL.
In the offline PbRL setting with access to an offline dataset and labeled preferences between the offline trajectories, it is straightforward to combine previous (online) PbRL methods and off-the- shelf offline RL algorithms (Shin & Brown 202I). As shown in Fig (left), we can first use the Bradley-Terry model (Bradley & Terry 1952) to model the preference label and supervisedly learn a reward function (normally adopted in prior PbRL methods), and then train the policy with any offline RI. algorithm on the transitions relabeled via the learned reward function. Intuitively, such

---

Under review as a conference paper at ICLR 2023
Labeled offline data
Reward function
Labeled offline data
Policy
Policy
Supervised Learning
Offline RL
OPPO
methods
(preference)
(preference)
A flow diagram of previous offline PbRL algorithms (left) and our OPPO algorithm (right). Previous works require learning a separate reward function for modeling human preferences using the Bradley-Terry model. In contrast, our OPPO directly learns the policy network.
Figure 1:
two-step procedure allows preference modeling via a separate reward function. Fundamentally, however, learning a separate reward function that explains expert preferences does not directly in- struct the policy how to act optimally. As PbRL tasks are defined by preference labels, the goal is to learn the most preferred trajectory by the annotator rather than to maximize the cumulative dis- counted proxy rewards of the policy rollouts. More specifically, when encountering complex tasks such as non-Markovian tasks, conveying information from preferences to the policy via the scalar rewards creats bottleneck in policy improvement. On the other hand, if the learned reward func- tion is miscalibrated, an isolated policy optimization may learn to exploit loopholes in the relabeled rewards, resulting in unwanted behaviors. Then why must we learn a reward function considering possibly being not able to directly yield optimal actions?
To this end, we propose offline enteerence-guidee policy optimization (OPPO), an end-to-end formu- lation that jointly models offline preferences and learns the optimal decision policy without learning a separate reward function (as shown Fig |
right). Specifically, we explicitly introduce a hind- sight information encoder with which we further design offline hindsight information matching objective and a preference modeling objective. Via optimizing the two objectives iteratively, we derive a contextual policy ([ass, z) model the offline data and concurrently optimize optimal contextual variable z to model the preference. In this way, the focus OPPÓ is on learning a high- task-related information and evaluating policies in it. Then, arrive at the optimal policy by having the contextual policy π r(als, 2*) conditioned on the learned optimal z².
dimensional z-space capturing
In summary, our contributions include 1) OPPO: a simple, stable and end-to-end offline PbRL method that avoids learning a separate reward function, 2) instance of preference-based hind- sight information matching objective and novel preference modeling objective over the contextual, and extensive experiments to show and analyze the outstanding performance of OPPO against previous competitive baselines.
RELATED WORK
Preference-based RL. sreference-basee RL is also known as learning from human feedback. Several works have successfully utilized feedback from real humans to train RL agents (Arumugam lt al. 2019; Christiano et al. [0177, , [barz et al., 2018: Knox & Stone, 2009: Lee et al.] 202 Warnell et al. 2017). Christiano al. (2017) scaled preference-based reinforcement learning to utilize modern deep learning techniques, and Ibarz et al.) (2018) improved the efficiency of this method by introducing additional forms of feedback such as demonstrations. Recently, Lee et al. (2021b) proposed feedback-efficien RL algorithm by utilizing off-policy learning and pre-training. Park al. (2022) used eseudo-labeling to utilize unlabeled segments and proposed a novel data augmentation method called temporal cropping.
To mitigate the impact of distribution shifts in offline RL, prior algorithms (a) con- strain the action space (Fujimoto et al., 2019; Kumar et al.] 2019a Siegel et al. 2020), (b) incor- porate value pessimism (Fujimoto al.) 2019 Kumar al. 2020), and (c) introduce pessimism into learned dynamics models (Kidambi et al. 2020; Yu et al.) 2020). Another line of work ex- plored learning wide behavior distribution from the offline dataset by learning task-agnostic set
Offline RL.
2

---

Under review as a conference paper at ICLR 2023
of skills, either with likelihood- based approaches (Ajay al. 2020: Campos al. 2020 Pertsch eet al.
2020; Singh et al.] 2020) or by maximizing mutual information (Eysenbach et al. 2018 Lu et al. 2020; Sharma et al. 2019). Shin & Brown (202)) tried to solve offline PbRL by simply combining previous (online) PhRL. methods and off-the-shelf offline RI. algorithms.
Supervised learning in RL. Some prior methods for reinforcement learning bear more resem- blance to static supervised learning, such as Q-learning (Watkins) 1989 Mnih et al. 2013) and behavior cloning. In these cases, the resulting agent's performance is positively correlated to the quality of data used for training. On the other hand, [Srivastava et al. (2019) and Kumar et al. (2019)) studied "upside-dowe supervised loss conditioned on target return. Ghosh et al. (2019) extended prior UDRL methods to perform goal reaching by taking the goal state as the condition, and Paster et al. (2000) further used an LSTT for goal-conditioned online RI. settings. [Cen al. (2021) and et al. (202T) solved the problem via sequence modeling. Sequence modeling enables to model behaviors without access to the reward, in a similar style to language (Radford et al. 2018) and images (Chen et al.) 2020). In contrast to both supervised RL and UDRL, the purpose of our method is to search for the optimal solution supervised by binary preference signal in the offline setting. Our method can not only work with sub-optimal demonstrations but also reveal optimal behaviors without injecting human priors about the optimal demonstration.
reinforcement learning (UDRL), seeking to model behaviors via a
3 PRELIMINARIES
We consider reinforcement learning (RI.) in Markov decision process (MDP) described by tu- plc (S,A, P.P.. 7), where St E S, a E A, and r, = r(sr, ax) denote the state, action, and reward at timestep t, P(s Ss, at) denotes the transition dynamics, (s()) denotes the initial state distribution, and 7 1) denotes the discount factor. At each timestep f, the agent receives a state St from the environment and chooses an action a, based on the policy (Taa)ss) (st). In the stan- dard RL framework, the environment returns reward ra and the agent transitions to the next state 8141- The expected return (x) = Eyys(()) Ek=o 7 ( (t++)) is defined as the expectation of discounted cumulative rewards, where T (Sp, ag $1, a....... So Po()).. a, π (aa/s,)) and 81+1 P(s a.). The agent's goal is to learn policy that maximizes the expected return.
3.1 OFFLINE PREFERENCE- BASSED REINFORCEMENT LEARNING
In this work, assume fully offline setting in which the agent cannot conduct online rollouts (over the MDP) during training but is provided with a static fixed dataset. The static dataset, D = (7" ......., NNJ. consists of some pre-collected trajectories, where each trajectory 7' consists of contiguous sequence of states and actions: r' := {s). ag si.... J. Such an offline setting is more challenging than the standard (online) setting as removes the ability to explore the environment and collect additional feedback. Unlike imitation learning, we do not assume that the dataset comes from a single expert policy. Instead, the dataset D may contain trajectories collected by some sub- optimal or even random behavior policies.
Further, the standard offline RL assumes the existence of reward signals for each state-action pair in D. However, in the offline seeference-based RL (PbRL) framework, we do not assume such rewards are accessible. Instead, the agent can access offline preferences (between some pairs of trajectories (r',7") that are labeled by expert (human) annotator. Specifically, the annotator gives feedback indicating which trajectory is preferred, i.e., y E {0, 1, 0.5), where 0 indicates 7² > 7 (the event that trajectory r' is preferable to trajectory r²), 1 indicates T > r' (r- is preferable to 7'), and 0.5 implies an equally preferable case. Each feedback is stored in a labeled offline dataset DD= as a triple (r', r', .). Given these preferences. the goal of PbRL is to find a policy a[a, (s:) that maximizes the expected return Jr.. under the hypothetical reward function (st ax) consistent with human preferences. To enable this, previous works learn a reward function (st, as) and use the Bradley- Terry model (Bradley & Terry function:
1952) model the human preference, expressed here as logistic
P|r > r = logistic Σ (w at) - Σ (w(s(,a())
(1)
3

---

Under review as a conference paper at ICLR 2023
where (s/; ai) ~', (s}, (s),aa) rd. Intuitively. this can be interpreted the assumption that the probability of preferring a trajectory depends exponentially on the cumulative reward over the trajec- tory labeled by an underlying reward function. The reward function is then updated by minimizing the following cross-entropy loss:
[11 y) log P(r > T²] + y log P(r > --1]
- -Er)
With the learned reward function ri used label each transition in the dataset, we can adopt an off-the-shelf offline RL algorithm to enable policy learning.
3.2 HINDSIGHT INFORMATION MATCHING
Beyond the typical iterative (offline) RL framework, information matching (IM) (Furuta et al. 2021 has been recently studied as an alternative problem specification in (offline) RL. The objective IM in RL is to learn contextual policy rtaas, z) whose trajectory rollouts satisfy some (pre-defined) desired information statistics value z:
min Bg-p(( (f,, [(xx))]
(3)
where p(z) is prior, π (Fa) denotes the trajectory distribution generated by rolling out a(als, z) in the environment. [(T) is a function capturing the statistical information of a trajectory such as
the distribution statistics of state and reward, like mean, variance (Wainwright et al. 2008). and / a loss function.
On one hand, if we set p(z) as a prior distribution, optimizing Eq corresponds performing unsupervised (online) RL to learn a set of skills (Eysenbach et al. 2018: (Sharma et al. 2019). On the other hand, if we set p(z) as statistical information of a given pof-po trajectory (or state- action) distribution D(T) (or D(s, a)), Eq corresponds to an objective for hindsight information matching in (offline) RL. For example, HER (Andrychowicz et al. 2017) and return-conditioned RL (upside-down RL (Srivastava et al.] [019 Kumar et al. [201bb] Chen et al., 2021;] Janner al., 2021I) use the above concept of hindsight: specifying any trajectory T in the dataset as the hindsight target and setting the information in 3 as T(r). Then, we provide the ((-)-driven hindsight information matching (HIM) objective:
min ((r) (((((T).((T)))
(4)
where z : I(T). In HER, we can set [(T) as the final state in trajectory т; in ceward-con
RL, we set I(r) as the return of trajectory т. Thus, we can use the hindsight information z := I(r) to provide supervision for training the contextual policy π ralss, z). However, in offline settings, sampling Ta from (Tx) is not accessible. Thus, we are required to model the environment transition dynamics besides I(: d-r-vnriven hindsight information modeling. That is to say, we need to model the trajectory itself, i.e., min, (r) [e (T, T2)]. Then, we provide the overall offline HIM objective:
min
[e (T(r,, I(T=)) + (T,
(5)
To give an intuitive understanding of the above objective, here we provide simple example: consid- cring hindsight /(-) being the return of trajectory, optimizing E (I(7), (Tx)) ensures the generated To will reach the same return as T /- 1-1(2 (z).. However, in the offline setting, we must ensure the gen- erated stay in support of the offline data, eliminating the out-of -itt-fff-itribution (OOD) issues. Thus we minimize é (T. Tu) approximately. In implementation, directly optimizing é (T.Tz) r2) enough to ensure the hindsight information is matched, e.g., (T(r), T(Tx)) < e. Here, we explicitly formalizes the E (T(r), ), T(Tx)) term with particular emphasis on the requisite of hindsight information matching objective and meanwhile highlight the difference, see Section 4 between the above HIM objective (taking 1(-) as prior) and our OPPO formulation (requiring learning (o)))).
By optimizing Eq 5 we can obtain contextual policy (aass, z). In the evaluation phase, the optimal policy Tals, 2²) can be specified by conditioning the policy on a selected target z". For example, Decision Transformer (Chen et al. 2022)) takes the desired performance as the target z (c.gg,, specify maximum possible return to generate expert behavior); RvS-G (Emmons et al. 202T takes the goal state the target z"
4

---

Under review as a conference paper at ICLR 2023
Labeled
offline data
lT())
π (a, 2z 1,
: Require Updating
z
PHM
LHM
sample (d.,d.y)
y (1- y (1-
LP
Figure 2: OPPO first maps offline trajectories (both positive 7 and negative r) to latent space via the hindsight information extractor Ig. It then optimizes the offline HIM objective LHM-- Fi- nally, belief of the optimal hindsight information z" is updated to model the human preference with objective LPM. Meanwhile, the preference modeling loss also regularizes the learning of the hindsight information extractor Ia.
OPPO: OFFLINE PREFERENCE-GUIDE POLICY OPTIMIZATION
In this section, we present method, OPPO (offline preference-guided policy optimization), that adopts the hindsight information matching (HIM) objective Section 3.2 to model an offline con- textual policy x (aass s, z). and introduces triplet loss model the human preference as well the optimal contextual variable z". At testing, we condition the policy on the optimal z and thus con- duct rollout with (a|s, z²). In principle, OPPO be paired with any PbRL settings, including both online and offlin.. the scope of our analysis and experiments, however, we focus on the offline setting decouple exploration difficulties in online RL.
4.1 HIM-DRIVEN POLICY OPTIMIZATION
As described in Section 3.1 to directly implement the off-the-shelf offline RL algorithms, previous works in PbRL explicitly learn reward function with Eq (as shown Figft) As an alternative to such a two-step approach, we seek to learn the policy directly from the preference-labeled offline dataset (as shown in right). Inspired by the offline HIM objective Section(.22 we propose to
learn contextual policy 7 (a|s, z) in the offline PbRL setting. Assuming la is (learnable) network that encodes the hindsight information in PbRL, we formulate the following objective:
min PHM == E, ((=)((r)raaa((a)) (o + E (TTTT)) R:
(6)
where z [T)... Note that Eq 6 is different instantiation of Eq where learn the hindsight information extractor Ia(-) in the PRBL setting, while previous (offline) RL algorithms normally set I(-) be a prior (Chen et al. 2021 Emmons et al. 2021). Such an encoder-decoder structure is now similar with Bi-directional Decision Transformer (BDT) proposed by (Furuta et al. 2021] for offline imitation learning. However, since expert demonstrations are missing the PbRL. setting, in we propose to use the preference labels in DD, to extract hindsight information.
Section 4.2
4.2 PREFERENCE MODELING
To make the hindsight information (e(T) in 6 match the preference information in the (labeled) dataset D we construct the following preference modeling objective inspired by the contrastive loss metric learning:
min (m ((z)zzz))))))))))))()) l(z', l(z',z )2)
x
where zt and z represent the embedding of the preferable (positive) trajectory Io(yr) + +(1++++1111 (111))) y)r
and that of the less preferable (negative) trajectory Ig(yr' the idea of using regret for modeling preference (Knox et al., [022]] Chen et al. (022), our basic as- sumption of designing the objective in Eq 7 that humans normally conduct two-level comparisons
r)r) respectively. Closing to
5

---

Under review as a conference paper at ICLR 2023
Algorithm 1 OPPO: Offline Preference-guider Policy Optimization
Require: Dataset D := }r} and labeled dataset D> := {(T', ((7',,',00 ,,,)), where r' E D and 7 E D. Return: 7 z) and zz.
1: Initialize policy network m(aas, z), hindsight information extractor To : -> z, and the optimal contextual embedding z".
2: while not converged do
3: Sample batch of trajectories from D: {T}B D.
4: Update π (aass, z) and Io(-) with sampled {T) using LHIM-
5: Sample batch of preferences from D, {(T', -²,, ..yy)) Dr.
6: Update Io))) and the optimal x with sampled {(T', (T',,',,)) ,y)}n using LPM-
7: end while
before giving preferences between two trajectories T²): 1) separately judging the similarity be- tween trajectory r' and the hypothetical optimal trajectory T², i.c. - -((z', zz), and the similarity between trajectory 7 and the hypothetical optimal one T", -E(2", 2²),, and 2) judging the difference of the above two similarities (-f(z", z²) vs. -((z", 2) and setting the trajectory with the higher similarity as the preferred one. Hence, optimizing Eq guarantees finding the optimal embedding that is more similar to xt and less similar
To clarify, z is the coresponding contextual in- formation for r", whereas 7" will always be preferred comparing to any offlin trajectories in the datasets.
In practice to robustify the preference modeling, we optimize the following objective using the triplet loss in place of the objective Eq
min LPM (r ())) max(((z" (axx(((""zzt)>> l(zz ,2) + margin, 0)]
xx xe
It is worth mentioning that the posterior of the optimal embedding zt and the hindsight information extractor I.))) are updated alternatively ensure learning stability. A better estimate of the optimal embedding helps the encoder to extract features to which the human labeler pay more attention, while better hindsight information encoder, on the other hand, accelerates the search process for the optimal trajectory in the high-level embedding space. In this way. the loss function for the encoder consists of two parts: 1) a hindsight information matching loss in a supervised style in oq and 2) a triplet loss as in Eq g to better incorporate the binary supervision provided by the preference-labeled dataset.
In summary, OPPO learns a contextual policy r(als, z), context (hindsight information) encoder To(T), and the optimal context variable, z", for the optimal trajectory r" Algorithm [ details the training of OPPO. The entire process is summarized follows: 1) we sample batch of trajectories from the dataset D, 2) in Line 4, use Eq(ó] (the hindsight information matching loss) to update (als, z) and To(.) based offline trajectory by the extractor, the policy is able to reconstruct it. 3) Then sample a batch of preferences from the labeled dataset D=, and finally, 4) in Line 6, update Ia(-) and z based on the sampled { T2, y)}n using Eq making the optimal embedding zt near to the more preferred trajectory 2 and meanwhile further away from the less preferred trajectory
sampled trajectories. Consequently, given the extracted out of an
Compared with previous PbRL. works (first learning a reward function with Eq and then learn- ing offline policy with off-the-shelf offline RL algorithms), OPPO Iearns the optimal (offline) policy (T (a(ass z=) directly and thus avoids the potential information bottleneck caused by the limited infor- mation capacity of scalar reward assignment. Compared with the HIMM-bsse offline RL algorithms (e.g., Decision Transformer (Chen et al. (021) and RvS-G (Emmons et al., (02T)), at the testing phase, OPPO does not need to manually specify the target contextual variable for the rollout policy o (a)s,
5 EXPERIMENTS
In this section, we evaluate and compare OPPO to other baselines in the offline PbRL setting. A cen- tral premise behind the design of OPPO is that the learned hindsight information encoder Is(-) can capture preferences over different trajectories, as described by E Our experiments are therefore designed to answer the following questions:

---

Under review as a conference paper at ICLR 2023
1) Does OPPO truly capture these types of preferences? In other words, does the learned z-space (encoded by the learned Io(-)) align with given preferences?
2) Can the learned optimal contextual policy π a[ass, z²) obtain better performance than π ( conditioned on all the other contextual variables z E {Io(T)(T D)?
3) Can OPPO also achieve competitive performance against other offline (PbRL) algorithms?
4) What benefits, if any, do we gain from designing the end-to-end offline preference learning, i.e. iteratively conducting offline data modeling ( and preference modeling 8
To answer the above questions, we evaluate OPPO on the continuous control tasks from the D4RL benchmark (Fu et al., 2020). Specifically, we choose Hopper, Walker and Halfcheetah three base environments, with medium, medium-replay, medium-expert as dataset for each environment. To build our labeled offline preference dataset D-- {(r', ,'. v),)), following prior PbRL bench- mark (Lee et al.) (021a), we use the following stochastic preference model (to simulate a human annotator and) to label preference between trajectories
H
H
y P(r - 3) logistic (
where factors y and are used to model myopic and rational behaviors (of human annotator) re- spectively and are both set to in our experiment.
5.1 DOES THE LEARNED Z-SPACE ALIGN WITH GIVEN PREFERENCES?
In this subsection, we probe that OPPO can enable well-aligned preferences over the z-space en- coded by the learned Ip. We first sample random trajectories from the offline dataset D, and encode them with the learned la. and utilize t-SNE (van der Maaten & Hinton 2008) as a tool to visualize the encoded z, shown in Fig B The learned optimal z' is marked with orange dot. Besides, also mark the (embedding of) optimal trajectory D4RL expert dataset, generated by the learned online optimal policy, with red dot (z**).
According to Eq g embeddings near the actual optimal 2"* in z-space means they are more prefer- able implied by the preference label. Comparing the sampled trajectories (embeddings), find OPPO successfully captures the preference. As illustrated in Fig 3 the trajectories (embeddings) that are near =* often have high returns (points with a deeper color). Further, we observe that our learned optimal z² constantly stays close to actual optimal z which suggests that our learned preserves near-optimal behaviors. Thus, gives justification that OPPO can make meaningful preference modeling.
Medium-Replay
Medium
Medium-Expert
1.0
1.0
0.8
return
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.0
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.0
Figure 3: We utilize t-SNE to visualize the z-space learned in Hopper environment, encoded with a well-trained Ia(.), including the embedding of random trajectories in D. our learned x ("orange dot") and the actual optimal x* ("red dot"), embedding of the best trajectory/policy learned with online reinforcement learning method. Color of the points represent the normalized return of the corresponding trajectory T.
5.2
CAN N (a 2) CONDITIONED THE LEARNED x ENABLE BETTER PERFORMANCE?
Fig 3
shows that our learned Io(+) can produce a well-aligned embedding z-space exhibiting ef- fective preference modeling across (embeddings of) trajectories. More importantly, embeddings'

---

Under review as a conference paper at ICLR 2023
Table 1:
Comparison of (normalized) performance when rollouting the contextual policy π (als,.) conditioned on different embeddings (z", Zhigh and Zlow).
Environment
Dataset
2
Zhigh
Zlow
Medium-Expert Expert
108.0 + 5.1
94.2 + 24.3
79.1 + 28.8
Hopper
Medium
86.3 + 3.2
55.8 + 7.9
51.6 13.8
Redium-Replay
88.9 + 2.3
78.6 26.3
26.6 = 15.2
Meedium-Exped
105.0 = 2.4
106.5 = 9.1
93.4 =
Walker
Medium
85.0 + 2.9
64.9 24.9
72.6 = 10.6
Medium-Replas
71.7 + 4.4
55.7 + 24.8
6.8 + 1.7
Medium-ENee Expert
89.6 + 0.8
48.3 14.4
42.6 = 2.6
Halfcheetah
Medium
43.4 + 0.2
42.5 + 3,9
42.4 + 3.2
Medium-
39.8 + 0.2
35.6 + 8.5
33.9 9.2
Sum
717.7
581.9
448.9
preference property should be preserved when we condition the embedding on the learned con- textual policy (a)ss-). In other words, Io(-) transfers the preference relationship from (7', r²) to (f(z', 2²), f(x', 2")); further, rollouting the contextual policy π Talss, )), (T2i, Tax) should similar preserve the above preference relationship.
To show that, we compare the performance of rollouts by the contextual policy x ralss, conditioned on different embeddings. In Table [ choose three contextual embeddings: z", Zhigh (embed- ding of the trajectory with the highest return in D), and 2low (embedding of the trajectory with the lowest return in D) and provide respective rollout performances (averaged cover that the contextual ( z) conditioned on with high (or low) return (of corresponding trajectory
3 seeds). We dis-
1g (z)) obtains an actual high (or low) return when rollouting this policy in the environment, e.g., π (ajs, Zhigh) performs better than (als, Zlow) (thus preserving the hindsight pref- erence relationship). Further, when conditioned on the learned optimal z", π (aass, z²) produces the best performance over that conditioned on all other offline embeddings. Notice that our learned optimal π (als,, z²) performs better than the contextual policy T (als, Zhigh). This result implies that trajectory of our optimal policy better than any trajectories the offline dataset.
5.3 CAN OPPO ACHIEVE COMPETITIVE PERFORMANCE ON OFFLINE (PBRL) BENCHMARK?
We have shown that OPPO produces near-optimal embedding z", and the learned contextual pol- icy π (ajs, can preserve the hindsight preference. This subsection investigates whether the optimal policy π (als, z² can achieve competitive performance on the offline (PBRL) benchmark. For com- parison, we introduce three offline (PbRL) baselines: 1) DT++: performing Decision Transformer with ground-truth reward function, and the results are run by us; 2) DT+ performing Decision Transformer with learned reward function (using Eq ; CQL+r: performing CQL with ground- truth reward function, reported from the original paper; 4) BC: performing bahavior cloning on the
dataset, the results are reported from (Cen et al. 2021:
Table 2: Performance comparison between OPPO and offline (PbRI.) baselines (DT+r, DT+r and CQL+r) in D4RL Gym-Mujoco tasks, where results are reported over seeds.
Environment
Dataset
Ours
DT+r
DT+ra
COL+r
BC
Medium- Expert
108.0 1 5.1
111.0 1 0.5
95,6 1: 27.3
111.0
79.6
Hopper
Medium
86.3 + 3.2
76.6 + 3.9
73.3 3.0
58.0
63.9
Medium-Ree Replay
88.9 2.3
87.8 + 4.7
72.5 22.2
48.6
27.6
Medium-Expor
105.0 + 2.4
109.2 + 0.3
109.7 + 0.1
98.7
36.6
Walker
Medium
85.0 2.9
80.9 3.1
81.1 2.1
79.2
77.3
Medium Medium-Replay
71.7 -
3.16
80.4 + 4.4
26.7
36.9
Medium -xppert
89.6
86.8 1.3
88.4 0.7
62.4
59.9
HalfCheetah
Medium
43.4 1 0.2
43.4 0.1
43.2 1 0.2
44.4
43.1
Medium-Rere Replay
39.8 + 0.2
39,2 + 0.3
38.8 + 0.3
46,2
4.3
Sum
717.7
714.5
683.0
575.2
429.2
8

---

Under review as a conference paper at ICLR 2023
In Table 2 provide the comparison results. 1) OPPO has retained comparable performance against the Decision Transformer trained using true rewards. OPPO is a PbRL approach requiring only (human) preferences, which have more flexible and straightforward form of supervision in the real world. 2) Although DT+r also shows competitive results in these benchmarks, such a method needs a target of return-to-go go determined by the human prior. Our method, in contrast, avoids the need of such prior target by searching across the z-space. We argue that this searching method brings advantages as rewards are usually hard to obtain in real-world RL applications. And preferences are the only information easily accessible for training and deploying an RL method.
5.4 Do WE GET ANY BENEFITS BY ITERATING OFFLINE HIM AND PREFERENCE MODELING?
Here, we conduct ablation study to analyze the benefit iterating LHIM and LPM (for updating 1g). Firstly, we removed Io from 0Lrs/00 and only left the optimal embedding z" to be updated in Eq Then, we continue visualize the embedding z- space for this ablation setting (OPPO-a), and we show the t-SNE visualization 4 By compar- ing 4 to Fig we can see that the preference relationship in the embedding space (learned with OPPO-a) is all shuffled. In less expressive z-space, it is challenging to model the preference and find the optimal z". Further, as shown in Table 3 the comparison results of medium-replay tasks confirm that such ablation does cause performance degradation.
Table 3: Ablation study
Task
OPPO
OPPO-a
Hopper
88.9 = 2.3
78.3 + 7.1
Walker
71.7 4.4
66.3 + 1.6
HalfCheetah
39.8 + 0.2
39.6 + 0.1
Sum
200.4
184.2
Medium-Repi Replay
Medium
Medium-Expert
1.0
1.0
1.0-
0.8
0.8
0.8
Normalizze return
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
D.2-
0.0
0.0
0.0:
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.0
Figure 4: t-SNE visualization of the embedding space learned with OPPO-a in Hopper environment.
CONCLUSION
This paper introduces offline preference-guided policy optimization (OPPO), an end-to-end offline PbRL method. Unlike the previous PbRL approaches that learn policy from a pseudo reward fune- tion (thus, learning separate reward function is prerequisite), OPPO directly optimizes the policy a high-level embedding space. To enable that, we suggest learning a hindsight information en- coder network and using it to design offline hindsight information matching (HIM) objective and a preference modeling objective. Empirically, we show iterating the above two objectives can pro- duce meaningful and preference-aligned embeddings. Moreover, conditioned the learned optimal embedding, our HiM-based contextual policy can achieve competitive performance on standard of- fline (PbRL) tasks.
Through the visualization, we demonstrate that the z-space learned by the encoder is informative and visually interpretable. Besides, the ablation study proves that preference-guided embedding space could improve task performance asymptotically by a non-neglectable margin. Moreover, OPPO can find a contextual variable represent the embedding of the optimal trajectory, where the resulting trajectory is better than any offline trajectory in the dataset. Last but not least, in the offline setting with environment interaction disabled, our method can acquire the optimal behaviors using binary preference labels between sub-optimal trajectories. As shown in the experiment results, OPPO achieves a competitive performance over DT trained using either true rewards or pseudo rewards.
Preference-hasee relahelled rewards only participate in training phase. During the evaluation phase of DT+ra, we pass the target return-o--oo value in the original DT paper.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Anurag Ajay. Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. Opal: Offline prim- itive discovery for accelerating offline reinforcement learning. arXiv preprint arXiv:2010. 2020.
Marcin Andrychowicz. Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Picter Abbeel, and Wojciech Zaremba. Hindsight experience re- play. Advances in neural information processing systems, 30, 2017.
Dilip Arumugam. Jun Ki Lee, Sophie Saskin, and Michael Littman. Deep reinforcement learning from policy-dependent human feedback. arXiv: Learning, 2019.
Erdem Biyik, Nicolas Huynh, Mykel Kochenderfer, and Dorsa Sadigh. Active preference-based
Gaussian process regression for reward learning. In Robotics: Science and Systems XVI. Robotics:
Science and Systems Foundation, July 2020. doi: 10. 10.156607tss.22200vv...m
URL https:
//doi.org//i 15607/rs3 2020. xvi. 041
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324, December 1952. ISSN 0006-3444. doi: 10. 2307/2334029, URL httttts:/// /ttp:://do....gg13//// / wtt.:/.........02//222222222
Erdem Biyık and Dorsa Sadigh. Batch active preference-based learning of reward functions. arXiv: Learning. 2018.
Víctor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giró-i Nieto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills, In Interna- tional Conference on Machine Learning, pp. 1317-1327. PMLR, 2020.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 14:15084-15099 2021.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. In International conference on machine learning, pp. 1691- 1703. PMLR, 2020.
Xiaoyu Chen, Han Zhong, Zhuoran Yang, Zhaoran Wang. and Liwei Wang. Human-ine numan-in--hh--lon loop: Provably efficient preference-baset reinforcement learning with general function approximation. arXiv preprint arXiv:2205. 11140, 2022.
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. arXiv: Machine Learning, 2017.
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline rl via supervised learning? arXiv preprint arXiv:21 1075),
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv: N0NNvv/800200070 2018.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without PMLR, 2019.
exploration. In International conference on machine learning, pp.
Hiroki Furuta, Yutaka Matsuo, and Shixiang Shane Gu. Generalized decision transformer for offline hindsight information matching. arXiv preprint arXiv:2111. 2021.
Javier Garcia and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.
Dibya Ghosh, Abhishek Gupta, Justin Fu, Ashwin Reddy, Coline Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals without reinforcement learning. 2019.
10

---

Under review as a conference paper at ICLR 2023
Dylan Hadfield-Menell Smitha Milli, Pieter Abbeel, Stuart J Russell, and Anca Dragan. Inverse reward design. Advances in neural information processing systems, 30, 2017.
Alexander Hans, Daniel SchneegaB, Anton Maximilian Schäfer, and Steffen Udluft. Safe explo- ration for reinforcement learning. In ESANN, pp. 148-148. Citeseer, 2008.
Borja Iharz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in Atari. In Neural Information Processing Systems, 2018.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34:1273-1286, 2021.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. QT-opt: Scalable deep reinforcement learning for vision-based robotíc manipulation. In Computer Vision and Puttern Recognition, 2018.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. Advances in neural information processing systems, 33: 18810-2182,,, 2020.
W. Bradley Knox and Peter Stone. Interactively shaping agents via human reinforcement. In Pro- ceedings of the fifth international conference on Knowledge capture K-CAP '09. ACM Press, 2009. doi: 10.145/1597735.55977788 URL 1597738
W Bradley Knox, Stephane Hatgis-Kessell, Serena Booth, Scott Niekum, Peter Stone, and Alessan- dro Allievi. Models of human preference for learning reward functions. arXiv preprint arXiv:2206.02233. 2022.
Jens Kober and Jan Peters. Policy search for motor primitives in robotics. In Neural Information Processing Systems, 2008
Jens Kober, J. Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(1 1274,, August 2013. ISSN 0278- 3649, 1741-3176. doi: 10.117700273364911447777
URL
11777
0278364913495721
N. Kohl and Stone. Policy gradient reinforcement learning for fast quadrupedal locomotion. In IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004. IEEE, 2004. doi: 10.1 10009//obhot2204133345555555 URL httpps://oo....... http://d...rgg/0.11111111
robot 2004. 1307456
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q- learning via bootstrapping emrrr reduction. Advances in Neural Information Processing Systems, 32, 2019a.
Aviral Kumar, Xue Bin Peng. and Sergey Levine. Reward-conditioeed policies. arXiv preprint arXiv:1912.13465. 2019b.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 13:1179-11 1191, 2020.
Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel. B-pref: Benchmarking preference-based reinforcement learning. 2021a.
Kimin Lee, Laura M. Smith, and Pieter Abbeel. PEBBLE: Feedback-efficient interactive reinforce- ment learning via relabeling experience and unsupervised pre-training. In International Confer- ence on Machine Learning, 2021b.
Ilya Loshchilov and Frank Hutter. arXiv: 171 2017.
Decoupled weight decay regularization. arXiv preprint
11

---

Under review as a conference paper at ICLR 2023
Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Reset-free lifelong learning with skill- space planning. arXiv preprint arXiv:2012.03548, 2020.
Oliver Mihatsch and Ralph Neuneier. Risk-sensitive reinforcement learning. Machine learning, 49 (2):267-290) 2002.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.500,, 2013.
Jongjin Park. Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. Surf: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning. arXiv preprint arXiv:2033 10050, 2022.
Keiran Paster, Sheila A Mcllraith, and Jimmy Ba. Planning from pixels using inverse dynamics models. arXiv preprint arXiv:2012.02419, 2020.
Karl Pertsch, Youngwoon Lee, and Joseph J Lim. Accelerating reinforcement learning with learned skill priors. arXiv preprint arXiv:2010.11944 2020.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language under- standing by generative pre-training. 2018.
Dorsa Sadigh, Anca Dragan, Shankar Sastry, and Sanjit Seshia. Active preference-based learning of reward functions. In Robotics: Science and Systems XIII. Robotics: Science and Systems Foun- dation, July 2017. doi:
URL https: org/ 156077
2os..217..0 xiii.iiiii.. 11111.0050 053
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. arXiv preprint arXiv:1907.01657, 2019.
Daniel Shin and Daniel S Brown. Offline preference-based appreticechp learning, arXiv preprint arXiv:2107.09255 2021.
Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Ne- unert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.08396 2020.
David Silver, Julian Schritwieserr, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen. Timothy Lillicrap. Fan Hui, Laurent Sifre, George van den Driessche. Thore Graepel, and Demis Hassabis. Mastering the game of go without human knowledge. Nature, 550(7676):544359) October 2017. ISSN doi: 10. 10.1038/mature24270. URL https::/ddr org. /10. 1038/
0028-0836, 1476-4687,
nature24270
Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey Levine. Parrot: Data-driven behavioral priors for reinforcement learning. arXiv preprint urXiv:2001..1000 2020.
Rupesh Kumar Srivastava, Pranav Shyam, Filipe Mutz, Wojeiech Jaskowski, and Jürgen Schmidhu- ber. Training agents using upside-down reinforcement learning. arXiv preprint arXiv: 1rXiv:1912.0287, 2019.
Laurens der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal Ma- chine Learning Research, 986):2559-2605, 2008. URL http: jmlr www.oor///aa.... /ttgg//pperss//////
vvarderraateeneea.r html
Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John Agapiou, Max Jaderberg, Alexander S. Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L. Paine, Caglar Gulcehre, Ziyu Wang. Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünnshh, Katrina McKinney, Oliver Smith, Tom
12

---

Under review as a conference paper at ICLR 2023
Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in StarCraft using multi-agent reinforcement learning. Nature, 575(7782): 350-354, October 2019. ISSN 0028-0836. 1476-4687. doi: 10.1033/541188-019-- 1724-zZ. URL
ht.pss:/dooi org/10 C. 1038/ / & 4 41886 01 019-171122 1724-z
Martin J Wainwright, Michael Jordan, et al. Graphical models, exponential families, and variational inference. Foundations and Trends@ in Machine Learning. 1(1-2):1-305, 2008.
Garrett Warnell, Nicholas R. Waytowich, Vernon J. Lawhern and Peter Stone. Deep TAMER: Interactive agent shaping in high-dimensional state spaces. arXiv: Artificial Intelligence, 2017.
Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. 1989.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129-14142, 2020.
13

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
A.1 IMPLEMENTATION DETAILS
Codebase. Our code based on Decision Transformer: https::/ contuul.oomkzl11 We provide our source code in the supplementary material.
Recision-trans
OpenAI Gym. We choose the OpenAl Gym continuous control tasks from the D4RL. bench-
mark (uu al. 2020). The different dataset settings described below.
Medium: million timesteps generated by "medium" policy that achieves approximately one-third the score of an expert policy.
Medium- Replay:: the replay buffer of an agent trained the performance of a medium policy (approximately 25k-400k timesteps in our environments).
Medium-Expert million timesteps generated by the medium policy concatenated with million timesteps generated by an expert policy.
For details of these enviromments and datasets, please refer D4RL for more information.
Training Objectives In our experiment, we consolidate & in Eq 6 as MSE Loss and in Eq as Euclidean Distance. this case, we model z as a point in the z z- space and the similarity measure l is L2 distance. Besides, there is alternative option to model z point sampled from learned distribution the z space, where a measurement between two distributions such as the KL divergence.
Also, we add a normalization loss Laoem to constraint the L2 norm of all embeddings generated by hindsight information extractor Ip.
Liotal == LHIM + aLpM + 3L
(10)
During Offline HIM phase, we weighted sum all these 3 losses as in Eq 10 (with ratios listed in Table 4 and perform backpropagation, while in Preference Modeling phase, only LPm is computed and hackpropagated.
Hyperparameter
Value
0.25 for sireheeeta--meiim expertt
0.5 for others
8
0.05 for haffcheetah-mediin-exprrr
0.1 for others
Table 4: Hyperparameters of coefficients of combined losses during Offline HIM.
Architecture & Implementation Details The architecture overview of OPPO is shown in Fig OPPO models the hindsight information extractor To as an encoder network To : 7 - 2, we use the BERT architecture. And similar to Chen al. (2021). T (a|s, z), which predicts future actions via autoregressive modeling.
the GPT architecture to model
Our hyperparameters on all tasks are shown below in Table 5 and Table Models were trained for 10s gradient steps using the AdamW optimizer Loshchilov & Hutter (017 following PyTorch defaults.
Hyperparameters
Computational resources. The experiments were run on computational cluster with 20x GeForce RTX 2080 Ti, and 4x NVIDIA Tesla V100 32GB for 20 days.
A.2 ADDITIONAL RESULTS
More visualization results on z-space. We further show the t-sne results of OPPO and the cor- responding ablation study in the setting described in Section 5.1 and Section 5.4 in walker and
14

---

Under review as a conference paper at ICLR 2023
Hyperparameter
Value
Number of dimensions
for halfcheetah
16 for others
Amount of feedback
50k
Type of optimizer
AdamW
Learning rate
10-2 2 for halfcheetah- haliihhetah-mmmduu--eepprrr
10 for others
Weight decay Margin
10
Table 5: Hyperparameters of zz searching for OpenAl Gym experiments.
Hyperparameter
Value
Number of layers
3
Number of attention heads
2 for encoder transformer
1 for decision transformer
Embedding dimension
128
Nonlinearity function
ReLU
Batch size
64
context length K
20
Dropout
0.1
Learning rate
10-4
Grad norm clip
0.25
Weight decay
10 10-4
Learning rate decay
Linear warmup for first 10 training steps
Table 6: Hyperparameters of Transformer for OpenAl Gym experiments.
halfcheetah environments. By comparing Fig to 5 we discover that the structure of z-space significantly collapses in eight out of nine environments (except for halfcheetah medium-replay) More specifically, we can no longer recognize the distribution pattern and clusters emerged in Fig while such observation is in line with conclusion in the main text.
However, also worth noting that the performance of OPPO-a in D4RL benchmark, is not hin- dered much by this uninformative z-space, shown in Table We attribute this to the effectiveness of preference modelling phase, where our method still able to find a meaningful z less expressive z-space.
This is also justified from t-SNE(Fig as there our learned z (orange dot) locates just in the point of deep color.
Environment
Dataset
OPPO(Ours)
OPPO(as)
Medium-Experr
107.8 1.0
103.5 + 4.4
Hopper
Medium
67.4 3.5
69.2 +: 7.4
Medium-Replay
88.2 1.7
78.3 + 7.1
Medium-Expert
106.8 + 2.8
108.8 + 1.0
Walker
Medium
81.6 2.8
80.7 + 1.5
Medium-Replay
70.6 = 5.1
66.3 + 1.6
Medium Expert
89.6 = 0.8
90.1 + 1.4
HalfCheetah
Medium
43.4 0.2
43.4 + 0.2
Medium-Replay
39.8 = 0.2
39.6 + 0.1
Sum
717.7
679.8
Table 7: Ablation study
15

---

Under review as a conference paper at ICLR 2023
Medium-Reppie
Medium
Medium-Exppe
1.0
1.0
1.0
1.0
0.8
0.8
0.8
Normalized retur
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.50
0.25
0.50
0.50
0.75
1.00
0.0
(a) walker
Medium-Replay
Medium
Medium-Expert
1.0
1.0
1.0
1.0
0.8
Normalizer maddalzzdd return
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.0
(b) halfcheetah
Figure 5: t-SNE of OPPO in walker and halfcheetah environments, including the embedding of random trajectories in D, our learned z (orange dot) and the actual optimal z" (red dot), embedding of the best trajectory/policy learned with online RL method. Color of the points represents the normalized return the corresponding trajectory T.
Redium-RRepas
Medium
Medium-Expor Expert
1.0
1.0
1.0
1.0
0.8
Ladalized retur
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.0
(a) walker
Medium-Replay
Medium
Medium-Exper
1.0
0.8
0.8
0.8
0.8
0.6
0.6
0.6.
Normalized
0.4
0.2
0.0
0.0:
0.0:
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
(b) halfcheetah
Figure 6: t-SNE visualization of the embedding space learned with OPPO-a in walker and halfchee- tah environments.
Ablation of feedback amount For Hopper task, we evaluate the impact of different amounts of preference labels on the performance of OPPO as shown in Tab More specifically, OPPO evaluated using the labels amount from 50k, 3k, 1k, 500. on the dataset from Medium-Exper Expert, Medium, Medium Replay. As illustrated in the table, OPPO performs the best when given 50k preference labels and achives total normalized of 283.1 among the three datasets. When
16

---

Under review as a conference paper at ICLR 2023
Dataset
50k
3k
1k
500
Medium- Expertt
108.0 5.1
92.1 + 9.2
102.9 + 3.2
104.9
Medium
86.3 = 3.2
73.5 + 14.8
90.8 + 2.0
77.5
12.8
Medium-Replay
88.9 2.3
66.2 + 23.3
60.4 + 3.0
68.5 + 22.8
283.1
231.8
254.2
250.9
Table 8: Ablation study
the feedback amount decreases 3k, the performance decreases at the same time. However, the performance stablizes at around 250 for further reduces to 1k and 500. Therefore, OPPO is robust to the changes in the amount of feedback used for training.
17


Reviews:
[{"review_id": "mwvuOUJtor", "reviewer": "Reviewer_LFVP", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors introduce a novel approach for Preference-based Reinforcement Learning in an offline setting. In contrast to other methods, they do not learn an explicit reward function, but use information matching against a latent representation of an (approximate) optimal trajectory. The latent representation is learn via a transformer approach. The algorithm is evaluated on three benchmark domains, including analyzation of three specific aspects of the approach. Namely, an qualitative evaluation of the embedding space in terms of distance to an optimal trajectory, under a simplified learning scheme and effects of using different points in the latent space as conditioner.", "summary_of_the_review": "The paper is very good and the clarity issue are not relevant for acceptance, but could improve the paper further. Only the evaluation is a relevant concern, but is still extensive enough to warrant a recommendation.\n\nIn fact, the reviewer believes, that the method is even more capable than mentioned. In theory, the method should be applicable to preference signals, based on a non-Markovian expert evaluations. This is an important issue for real world PbRL, because real humans cannot be assumed to conform to the Markov property. Therefore a discussion of this assumption (or even evaluation), would be quite interesting.\n", "strength_and_weaknesses": "The paper is very well written and clear to understand, with one, small exception (see clarity). The approach itself mainly builds on existing ideas, but goes beyond a simple, straight-forward combination. Furthermore, it is nicely aligned with the current State-of-the-Art and represents a meaningful step forward. The relation to related work is clearly state. The evaluation is ok, but could be improved. Mostly, because only one of the 4 competitors (Tbl.2) is a true PbRL algorithm. Furthermore, 3 domains (with 3 variants each) is ok, but leaves room for substantial improvement. Especially, non-locomotion tasks should be considered. Additionally, the required number of preference queries should be evaluated, as this is usually a limiting factor in PbRL (due to costly, human involvement). The qualitative ablation studies are a good addition to the evaluation. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "P9rWWLm3l1g", "reviewer": "Reviewer_uVj7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents an approach for learning to solve task from offline data using preference feedback and no reward function. Instead of learning a reward function, a latent embedding is learned that encode information about what it means to complete the target task. The latent embedding, z, is learned from preferences over the offline data used to train the policy. The latent embedding z and the policy are learned separately, where there during policy learning and inference, next actions are conditioned on both z and the current observation. Policy training is conducted following Hindsight Information Matching. Experiments are conducted in the Hopper, Walker, and Half cheetah locomotion tasks from D4RL with offline data coming from the replay buffer of a medium quality policy, rollouts from a medium quality policy, and rollouts from a medium-expert quality policy. To evaluate the quality of the learned latent embedding z*, a t-SNE plot is used to visualize how similar z* is a an ideal trajectory. Across conditions, z* is close to the ideal trajectory. The performance of policy rollouts using the \"optimal\" z embedding, a z embedding from a trajectory with large returns, and a z embedding with low returns are compared. Across conditions rollouts from the \"optimal\" z embedding out perform the z embedding from the trajectories with high and low returns. Policy rollout performance is also compared against other offline RL approaches (decision transformer with the ground-truth reward, decision transformer with a learned reward, CQL with the ground-truth reward, and behavior cloning). In 1/3 of the conditions, the proposed methods out perform the baselines. Finally, the impact of iterating between the Hindsight Information Matching objective and policy learning is evaluated via t-SNE projections and visualization. In all but the medium condition, the learned z* and the ideal trajectory are further apart than in the first experiment. ", "summary_of_the_review": "In summary, the paper presents an interesting idea that seems of benefit to pursue. However, given that OPPO did not consistently outperform the baselines, better motivation for why the method should be used instead needs to be provided. The benefits of OPPO over the baseline need to made more clear. Additionally, there are further experiments that would be helpful to see to fully understand the benefits and robustness of the method. My recommendation for the paper is therefore based upon the need for more experiments and the lack of consistent improvements in performance relative to baselines. ", "strength_and_weaknesses": "Strengths:\n- The idea is really interesting. This is probably one of the first works I have seen that attempts to learn a policy from preferences in the absence of a reward function. Moving away from reward functions seems like it could have benefits and improve policy learning. \n- The work is well grounded in the literature and authors do a good job of contextualizing where the method fits and how it is different from what has been done.\n- A good amount of detail has been given to specify how OPPO works meaning the method and the results should be reasonable to reproduce. \n\nWeakness:\n- The proposed method does not consistently outperform the baselines. Clear benefits of the method are not well motivated, especially given the results. For example, the authors motivate their approach by stating that learning reward functions from preferences may result in a reward function that is not correctly calibrated to the preferences. The authors do not follow up with specific scenarios where or under which the reward functions are not well calibrated. It would have been nice to see clear cases where mis-calibration occurs when learning reward functions and then results demonstrating that OPPO is well calibrated in the identified scenarios. \n- The authors only present results using a synthetic preference labeller that gives perfect preference feedback with respect to the ground truth reward. However, humans are not likely to provide such high quality labels. As the quality of the offline dataset impacts rollout performance, it is likely that preference label quality will also impact rollout performance. It has been shown in the BPref (Lee et al. 2021) paper that labeller quality does impact policy performance.\n- I did not see a discussion about the impact of the amount of preference feedback on rollout performance. In online PbRL methods where the reward function is learned, the amount of preference feedback greatly impacts policy performance. It would have been great to see results on this and for there to have been a discussion.\n- The authors claim that their method is guaranteed to learn the optimal latent embedding z, however no proof that optimality is guaranteed is provided. The experiments (i.e. t-SNE plots) are not enough evidence to conclude optimality is guaranteed. \n- Prior online PbRL papers have included results from human experiments where actual humans provide the preference feedback. It would have been nice to see results with humans providing the preference feedback.\n- The authors do not appear to have compared against Shin & Brown 2021, who they call out as a very similar method that learns a reward function from preferences over an offline dataset and then use offline RL methods to learn a policy. It may have been present in the baselines, but it wasn't clear to me.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "EmT3z3Gp3D", "reviewer": "Reviewer_kZ4s", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper looks at the problem of offline RL from human preferences. Instead of first learning a reward model from human preferences and then training with offline RL, this paper learns a context embedding from trajectories, and then learns a policy that is conditioned on the context. Then, to optimize for the human preferences, it learns the optimal context, which can be fed into the policy. \n", "summary_of_the_review": "I think this is a nice idea, and it should be useful for the community. ", "strength_and_weaknesses": "\nGenerally I like this research direction, and I think that learning context from offline data can be useful for other tasks as well, such as learning options\n Unfortunately, it doesn’t look like the method convincingly outperforms existing baselines. However, it does perform competitively, and I think that the fact that it doesn’t learn a reward model is a nice aspect.\nI like the experiments showing that the learned optimal context aligns with preferences. \nA question: what is the main difference between learning a reward model and learning a context + the optimal context? It seems almost the same thing. Could they be thought of as the same thing in a broader unified perspective?  Why would we expect OPPO to perform better than learning a reward model? \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2TlTkFBU9m", "reviewer": "Reviewer_JqyW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies preference-based offline reinforcement learning where we only have access to the preferences over offline trajectories. Compared to the standard offline RL setting, PbRL doesn't assume the reward function is available, which is more realistic. To solve the problem, the authors propose an iterative algorithm, which optimizes two objectives iteratively and can directly output a policy, instead of first learning a reward function from the preferences and then training a policy on top of it. Empirically, the authors compare their algorithm with several baselines on the D4RL benchmark and demonstrate improved performance. ", "summary_of_the_review": "Overall, I think the problem studied in this paper is very interesting and close to reality. However, I am not really convinced with the motivation of the proposed algorithm, and why we need to do so as in this paper. In addition, the writing of this paper needs to be improved. There are too many vague/ambiguous and informal arguments. The authors should make explainations more formally, and provide either theoretical justifications or empirical results to backup their arguments. \n", "strength_and_weaknesses": "Strength:\n- The problem setup in this paper is more realistic than most of the offline RL papers, where the reward is always available for each state-action pair. \n\n- The proposed algorithm is new and can greatly improve the performance over baseline methods on the D4RL benchmark.\n\nWeakness:\n- The motivation of OPPO is unclear to me. Especially, why optimizing two (heuristic) objectives iteratively is a better choice than solving two separated optimizations, i.e., first, learn a reward function based on the preference, and then fit a policy. Can you provide a more rigorous justification for it? \n\n- In addition, I also noticed that DT+$r_{\\psi}$ performs quite close to (only slightly worse than) OPPO, but it performs slightly better than DT+r (ground-truth reward). This is quite surprising, can you explain why this is the case? Also, it seems the OPPO has little advantage over the two-stage algorithm, DT+$r_{\\psi}$. \n\n- The paper is not easy to follow, it has too many vague, informal and unsupported arguments. E.g., \n\n> \"OPPO learns the optimal (offline) policy (π(a|s, z∗)) directly and thus avoids the potential information bottleneck caused by separately learning a reward function\"  what do you mean by information bottleneck?\n\n> \"A better estimate of the optimal embedding provides hints for the encoder to extract more valuable features, while a better hindsight information encoder, on the other hand, accelerates the search process for the optimal trajectory in the high-level embedding space\":  what do you mean by \"hints\"? Can you provide evidence for supporting this argument?\n\n> \"It is worth mentioning that the posterior of the optimal embedding $z^\\star$: why call it a posterior? Shouldn't $z^\\star$ just be a point?\n\n- What is $\\ell$? I did not see the definition of it.\n\n- Is $\\pi(\\cdot|s,z)$ a parameteric model?\n\n- In general, offline RL relies on pessimism to overcome the distribution shift. It seems that OPPO doesn't need pessimism to perform well. If so, can you explain why it doesn't need?\n\n- Can you provide a comparison on the *Random* datasets, e.g., Hopper-Random, Walker-Random, and Halfcheetah-Random? Also, why is the Ant dataset missing?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject