Unsupervised Behavior Extraction via Random Intent Priors
Authors: Hao Hu
Yiqin Yang
Jianing Ye
Ziqing Mai
Chongjie Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: offline RL
reward-free
behavior extraction
TL;DR: We propose a simple approach using random reward to extract behavior from offline reward-free dataset.

Abstract: Reward-free data is abundant and contains rich prior knowledge of human behaviors, but it is not well exploited by offline reinforcement learning (RL) algorithms. In this paper, we propose UBER, an unsupervised approach to extract useful behaviors from offline reward-free datasets via diversified rewards. UBER assigns different pseudo-rewards sampled from a given prior distribution to different agents to extract a diverse set of behaviors, and reuse them as candidate policies to facilitate the learning of new tasks. Perhaps surprisingly, we show that rewards generated from random neural networks are sufficient to extract diverse and useful behaviors, some even close to expert ones. We provide both empirical and theoretical evidences to justify the use of random priors for the reward function. Experiments on multiple benchmarks showcase UBER's ability to learn effective and diverse behavior sets that enhance sample efficiency for online RL, outperforming existing baselines. By reducing reliance on human supervision, UBER broadens the applicability of RL to real-world scenarios with abundant reward-free data.

Unsupervised Behavior Extraction via Random Intent Priors
Hao Hu'; Yiqin Yang"; Jianing Ye', Ziqing Mai', Chongjie Zhang? Institute for Interdisciplinary Information Sciences, Tsinghua University
Department of Automation, Tsinghua University
DDepartment of Computer Science & Engineering, Washington University in St. Louis Chuh22, yangyiqi19, yejn21, maizq21) t edu. cn chongjie@uustl.co
Abstract
Reward-frea data is abundant and contains rich prior knowledge of human behav- iors, but it is not well exploited by offline reinforcement learning (RL) algorithms. In this paper, we propose UBER, an unsupervised approach to extract useful be- haviors from offline reward-free datasets via diversified rewards. UBER assigns different pseudo-rewards sampled from a given prior distribution to different agents extract diverse set of behaviors, and reuse them candidate policies facili- tate the learning of new tasks. Perhaps surprisingly. we show that rewards generated from random neural networks are sufficient to extract diverse and useful behaviors, some even close to expert ones. We provide both empirical and theoretical evidence to justify the use of random priors for the reward function. Experiments on multiple benchmarks showcase UBER's 's ability to learn effective and diverse behavior sets that enhance sample efficiency for online RL, outperforming existing baselines. By reducing reliance on human supervision, UBER broadens the applicability of RL to real-world scenarios with abundant reward-free data.
Introduction
Self-supervised learning has made substantial advances in various areas like computer vision and natural language processing (OpenAl 2023 Caron al. 2021 Wang al. 2020) since it leverages large-scale data without the need of human supervision. Offline reinforcement learning has emerged
as promising framework for learning sequential policies from pre-collected datasets (Kumar et al.)
2020: Fujimoto & Gu 2021 Ma et al., 2021 Yang et al. 2021 Hu et al. (022). but it is not able to directly take advantage of unsupervised data, as such data lacks reward signal for learning and often comes from different tasks and different scenarios. Nonetheless, reward-free data like experience from human players and corpora of human conversations is abundant while containing rich behavioral information, and incorporating them into reinforcement learning has huge potential to help improve data efficiency and achieve better generalization.
How can we effectively utilize the behavioral information in unsupervised offline data for rapid online learning? online settings, EEyenbach et al. (2018): Sharma et al. (2019) investigate the extraction of diverse skills from reward-free online environments by maximizing diversity objectives. Meanwhile, offline settings, a rich body of literature exists that focuses on leveraging the dynamic information from reward-free datasets (Yu et al. 2022 Hu et al.) 2023). However, the former approach is not directly applicable to offline datasets, while the latter approach overlooks the hehavioral information
in the dataset. Ajay et al. 22020 Singh et al. (2020) employ generative models extract behaviors
'qqual contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Phase Offline Behavior Extraction
Phase Online Policy Reuse
Random Intentions
Offline Algorithm
Rehaviors Fpi
Poliey Expamion
offiine butfer
xp
Figure I: The framework of UBER. The procedure consists of two phases. In the first offline phase, we assign different reward functions to different agent to extract diverse and useful behaviors from the offline dataset. In the second phase. reuse previous behavior by adding them the candidate policy set to accelerate online learning for the new task.
in the offline dataset. However, using a behavior-cloning objective is conservative in nature and has limited ability to go beyond the dataset to enrich diversity.
To bridge this gap, propose Unsupervised Behavior Extraction via Random Intent Priors (UBER), a simple and novel approach that utilizes random intent priors to extract useful behaviors from offline reward-free datasets and reuse them for new tasks, as illustrated in Figure [ Specifically, our method samples the parameters of the reward function from prior distribution to generate different intents for differentt agents. These pseudo-rewards encourage the agent go beyond the dataset and exhibit diverse behaviors. It also encourages the behavior to be useful since each behavior is the optimal policy accomplish some given intent. The acquired behaviors are then applied in conjunction with policy expansion (Zhang et al., 2023) policy reuse (Zhang et al. (2022) techniques to accelerate online task learning. Surprisingly, we observe that random intent prior sufficient to produce a wide range of useful behaviors, some of which even approach expert performance levels. We provide theoretical justifications and empirical evidence for using random intent priors Section 4 and
respectively.
Our experiments across multiple benchmarks demonstrate UBER's proficiency in learning behavior libraries that enhance sample efficiency for existing DRL algorithms, outperforming current baselines. By diminishing the reliance on human-desse designed rewards, UBER expands the potential applicability of reinforcement learning to real-world scenarios.
In summary, this paper makes the following contributions: (1) We propose UBER, novel unsu- pervised RL approach that samples diverse reward functions from intent priors to extract diverse behaviors from reward-free datasets; (2) We demonstrate both theoretically and empirically that random rewards are sufficient to generate diverse and useful behaviors within offline datasets; (3) Our experiments on various tasks show that UBER outperforms existing unsupervised methods and enhances sample efficiency for existing RL algorithms.
1.1 Related Works
Unsupervised Offline RL. Yu et al. 2021 considers reusing data from other tasks but assumes an oracle reward function for the new tasks. Yu et al. (2022) and [Hu al. (2023) utilize reward-free data for improving offline learning but assume labeled offline dataset is available for learning a reward function. Ye et al. (2022); Ghosh et al. (2023) consider the setting where reward and action
are absent. We focus on the reward-free setting and leverage offline data to accelerate online learning.
Unsupervised Behavior Extraction. Many recent algorithms have been proposed for intrinsic behavioral learning without reward. Popular methods includes prediction methods (Burda et al.
2018, Pathak et al. 2017 2019), maximal entropy- based methods (Campos al.) [2021: Liu &
Abbeel 2021b Mutti et al. 2020; Seo et al. 2001 Yarats et al./ (0211). and maximal mutual
information-basee methods (Eysenbach et al.) 2018 Hansen et al. 2019 Liu & Abbeel] 202
Sharma et al. 2019). However, they require online environment to achieve the diversity objective.
Ajay et al. (2020): Singh et (202)) employ generative models extract behaviors in the offline
dataset but have limited ability to enrich diversity.

---

Offline-to-onliee learning and policy reuse. provides a promising paradigm improve offline further learned policies. [Nar al. (2020) is among the first to propose a direct solution to offlne-to-online RI. (Lee et al. 2022) proposes to use a balanced replay buffer and ensembled networks to smooth the offline-to-on transition; /hang et al. (023) proposes to use the expanded policy for offline-to-online RI. B..ll et al. (2023) proposes to reuse the off-policy algorithm with modifications like ensembles and layer normalization, Zhang et al. (2022) proposes a critic-guided approach to reuse previously learned policies. Zhang et al. (2023) proposes a similar and simpler way to reuse previous policies.
RL has been popular recently since
Imitation Learning Our method is also related to imitation learning (Hussein et al. 2017). Popular imitation learning methods include behavior cloning methods PPomerleau (1988) and inverse RI. methods Ho & Ermon (2016); Xiao et al. (2019): Dadashi et al. (2020). However, they assume the demonstration in the dataset be near optimal, while our method works well with datasets of mediocre quality.
2 Preliminaries
2.1 Episodic Reinforcement Learning
We consider finite-horizon episodic Markov Decision Processes (MDPs), defined by the tuple (S,A, A. H, P.r), where S is state space, A is an action space, H is the horizon and (Ph) h= (rr) are the transition function and reward function, respectively.
A policy л = {= specifies a decision decision-making strategy in which the agent chooses its actions based the current state, i.e., aj TT(. sn). The value function V S - R is defined as the sum of future rewards starting at state and step E [H], and similarly, the Q-value function, i.c.
(s) E, Σ (st, at) s/= = s). (s) Ex Σ (n az) 84 =, on a) (1)
where the expectation with respect to the trajectory r induced by policy x. We define the Bellman operator as
(Brf) = a) f(s')]
(2)
for any SS IR and [H. The optimal Q-function Q" optimal value function V and optimal policy x² are related by the Bellman optimality equation
(s) = max a), Qh a) = (IB, (ss, (- argmaax Bann (% (s, a).
(3)
aEA
We define the suboptimality, as the performance difference the optimal policy x and the current policy nı given the initial state That is
SubOpt( s) = VT - (s)
2.2 Linear Function Approximation
To derive concrete bound, we consider the linear MDP (Jin et al., 2020 transition kernel and expected reward function are linear with respect feature map.
2021 as follows, where the
Definition 2.1 (Linear MDP). MDP(S, A, H, P. is a linear MDP with feature map Φ S x A + Rdd if for any h E [H), there exist d unknown (signed) measures 1h = (P ,....,h ((d))) over S and an unknown vector 24 Rdd such that for any (s, S A, we have
(. 8, a) (Ø(s, a), pn()), Without loss generality, we assume (lo(s, a)ll max (S)I [|= & vd for all h E (H).
Th a) = (ø(s, a), zn).
S for all (8,a) E S A. and
When emphasizing the dependence over the reward parameter we also use ra a, =). V (x,z) =) and V 2) to denote the reward function, the policy value and the optimal value under parameter 2, respectively.
3

---

3 Method
How can we effectively leverage abundant offline reward-free data for improved performance in online tasks? Despite the absence of explicit supervision signals in such datasets, they may contain valuable behaviors with multiple modes. These behaviors can originate from human expertise or the successful execution of previous tasks. To extract these behaviors, we assume that each behavior corresponds to specific intentions, which are instantiated by corresponding reward functions. Then, by employing standard offline RL algorithm, we can extract desired behavior modes from the offline dataset using their associated reward functions. Such behavior can be reused for online fine-tuning with methods such as policy expansion (Zhang et al. 2023) or policy reuse (Zhang et al. 2022).
Then, the question becomes how specify possible intentions for the dataset. Surprisingly, found that employing a random prior over the intentions can extract diverse and useful behaviors effectively in practice, which form the offline part of our method. The overall pipeline illustrated in Figure [ Initially, we utilize random intention prior to generating diverse and random rewards, facilitating the learning of various modes of behavior. Subsequently, standard offline algorithms are applied to extract behaviors with generated reward functions. Once diverse set of behaviors is collected, we integrate them with policy reuse techniques perform online fine-tuning.
Formally, we define Z as the space of intentions, where each intention 2 E induces a reward function r= S A + TR that defines the objective for agent. Let the prior over intentions be B E (2). In our case, we let 2 be the weight space W of the neural network and / be a distribution the weight W of the neural network Jw and sample N reward functions from independently. Specifically, we have
rs a) -w (s, we ~ 3, Vi N.
We choose 3 to be standard initialization distribution for neural networks (Glorot & Bengio 2010 He et al. 2015). We choose TD3+BC (Fujimoto & Gu (002)) as our backbone offline algorithm, but our framework is general and compatible with any existing offline RL algorithms. Then, the loss function for the offline phase can be written as
eritic (0t) (s.a.r..))) (Tr Ro Qu, (s)
(5)
Coftinn (dx)) = - ( [AQo, (8, no, (+)) - (To, (+) - a)
(6)
where = (5') the smoothed target policy and À is the weight for behavior regularization. We adopt a discount factor y even for the finite horizon problem commonly used in practice (Fujimoto (et al. 2018).
2023 to reuse As for the online phase, we adopt the policy expansion framework Zhang et al. policies. We first construct an expanded policy set ग = [ Tox Tw. where { 1.v weights of extracted behaviors from offline phase and Tw is new randomly initialized policy. We then use the critic as the soft policy selector, which generates a distribution over candidate policies as follows:
are the
exp (a (s)) Wi [1, N + +1}. Σ, exp Exexp(e (a ((+,,,())))
Ps [
where a is the temperature. We sample a policy i {1..... .N..NNN + 1} according to P3 at each time step and follows to collect online data. The policy Tu and Qa-value network undergo continuous training using the online RL loss, such as TD3 (Fujimoto et al. 2018)):
Conline (8) E(s,a,,"')=D (rr (Qr(s',a) Qo(s,
(8)
conline (w) - Ee (Qo(s, T. (s)))
The overall algorithm summarized in Algorithm and 2 We highlight elements important to our approach in purple.

---

Algorithm Phase 1: Offline Behavior Extraction
1: Require: Behavior size N, offline reward-free dataset Doff prior intention distribution
2: Initialize parameters of N 'ndeeeedentt offiine agents {Qa,, T 1
3: for = L............ do
4: Sample a reward function from priors ~ 3
5: Reannotate Doff Dr with reward Ts:
6: for each training iteration do
7:
Sample random minibatch {r,} R
Dr
8: Calculate Lertte
(0t) as in Equation (5 and update 8:
9: Calculate Loffine (Ø;) as Equation 6 and update Φ;
10: end for
11:
end for
Return (TO
Algorithm 2 Phase Online Policy Reuse
1: Require: {गT J==1 offline dataset Doff, update-to-de to-- ratio G
2: Initialize online agents Qi, The and replay buffer Don
3: Construct expanded policy set मे T
Téy Tw)
4: for each episode do
5: Obtain initial state 31 from the environment
6: for step = 1.... ....,, , do
7:
Construct P= According to Equation 7
8:
Pick an policy to act ग. ~ Pr. 0g ~ (e(-)st) Store transition (st, at Tt, St+1) in Don
9:
for g = .... G do Calculate Conline(0) (6) as in Equation 8 and update 0 12: end for
10:
11:
13: Calculate Conline (w)) in Equation (9) and update w
14: end for
end for
4 Theoretical Analysis
How sound is the random intent approach for behavior extraction? This section aims to establish theoretical characterization of proposed method. We first consider the completeness of our method. We show that any behavior can be formulated as the optimal behavior under some given intent, and any behavior set that is covered by the offline dataset be learned effectively given the corresponding intent set. Then we consider the coverage of random rewards. We show that with reasonable number of random reward functions, the true reward function can be approximately represented by the linear combination the random reward functions with a high probability. Hence method is robust to the randomness in the intent sampling process.
4.1 Completeness of the Intent Method for Behavior Extraction
We first investigate the completeness of intent-bar based methods for behavior extraction. Formally, we have the following proposition.
Proposition 4.1. For any behavior n, there exists an intent z with reward function r(-."2) such that is the optimal policy under r)), That is, for any {m Th - A(A), there exists 2 E Z such that
V. (8, 2) (8, V( ES
Proof. Please refer to Appendix A.1 for detailed proof.
Proposition 4.1 indicates that any behavior can be explained as achieving some intent. This intuitive since relabel the trajectory generated by any policy successfully reaching the final state in
5

---

we can assign an intent to any policy TT. However, unclear whether we can learn effectively from offline datasets given an intent set Z. This is guaranteed by the following theorem.
hindsight (Andrychowicz et al. (017). With Proposition 4.1
Theorem 4.2. Consider linear MDP as defined in Definition 2.1
With an offline dataset D with size N, and the PEVI algorithm (Jin et al. 2021). the suboptimality of learning from an intent z E 2 satisfies
SubOpt(7;; = (V
(10)
N
for sufficiently large N, where + log( (1/6) e is an absolute constant and
x Em
c max sup be[H] |=||=1 x. E.p
with Ing. = a a) 11 Ep. = Ep, ((s),)))(()
Proof. Please refer to Appendix A.2 for detailed proof.
C represents the maximum ratio between the density of empirical state-action distribution p and the density d,, induced from the policy Tg. Theorem 4.2 states that, for any behavior, πy that is well-covered by the offline dataset D, we can use standard offline RL algorithm to extract it effectively, in the sense that learned behavior ßे has small suboptimality under reward function r(;,;2) Compared to standard offline result (Jin et al. (2021). Theorem i.2 only worse off by factor of log (21, which enables us learn multiple behaviors from single dataset effectively, as long as the desired behavior mode is well contained in the dataset.
4.2 Coverage of Random Rewards
This section investigates the coverage property of random rewards. Intuitively, using random rewards may suffer from high variance, and it is possible that all sampled intents lead to undesired behaviors. Theorem (43 shows that such cases rarely happen since with a reasonable number of random reward functions, the linear combination of them can well cover the true reward function with high probability.
Theorem 4.3. Assume the reward function r(s, a) admits RKHS represention (ss a) with [1bss,, a))l < k almost surely. Then with N = CoVM log(18 (og1/8/82' / random reward func- tions (ra (rr) the linear combination of the set of random reward functions F(x, a) can approximate the true reward function with error
[(e,a)~ o( (18/8))
with probability 1 8, where M is the size of the offline dataset D, co and e₁ are universal constants and p the distribution that generates the offline dataset D.
Proof. Please refer Appendix A.3 for detailed proof.
indicates that RL algorithms are insensitive the randomness in the reward function used for learning, as long as we are using reasonable number of such functions. This phenomenon is more significant offline algorithms since they are conservative nature avoid over-generalization. This partly explains why using random rewards is sufficient for behavior extraction, and can even generate policies comparable to oracle rewards. Such phenomenon also observed prior
Theorem 4.3
works (Shin et al. 2023 Li al. 2023)

---

Dxpper--spert-v
expert.v0
expertvv
hosper-modium-o
Epkozt Betum
1000
Eppode Retum
Eybodi
12
USck (Dwm)
TB3
BC MEX
TB3
0.0
2.16
0.29 Millon Staps
1025
6:30
0.10
0125
(120
0.20
0.30
0.30
0.55
125
0.50
0155
0.20
0.25
0.33
Millon Stops
Milloo Stops
Millon Stene
100
Wal medium-umm
I6C
walker2d malker2d-med med expert-vd
mediche
UBER (Dun)
USeR (Dun)
USeR (Oun))
Estand Ratur
BC PEX
Eptande Return
BC BC-FEX
BC PEX
TD3
TD3
TOO
301
to
Eplonde
12
55
0.31
0155
Million
Miliio
Figure 2: Comparison between UBER and basclines in the online phase in the Mujoco domain. We adopt datasets of various quality for offline training. We adopt a normalized score metric averaged with five random seeds.
walker2d- -ilier/d-medium--0
walker2d-expertt-0
80
USER (Ours)
50-
UBER (Ours)
100
UBER (Ours)
Epicade Return
020
OPAL
UDS
00
OPAL PEX
UDS-PEX
Epicode Return
UDS
OPAL-PEX
40-
PARROI-PEE PEX
Episode
40-
PARROT-PEX
50
PARROT-PEX
0.15
0.20
3.25
0.30
0.15
0.20
0.30
0.10
0.15
0.20
0.25
0.30
Million
Steps
Steps
Millian Steps
Figure 3: Comparison between UBER and baselines including unsupervised behavior extraction (PARROR OPAL) and data-sharing methods (UDS). The result is averaged with five random seeds. Our method outperforms these baselines by leveraging the diverse behavior set.
Experiments
Our evaluation studies UBER pre-training mechanism reward-free data, focusing the following questions: (1) Can we learn useful and diverse behaviors with UBER? (2) Can the learned behaviors from one task using UBER be reused for various downstream tasks? (3) How effective is each component UBER?
To answer the questions above, conduct experiments on the standard D4RL benchmark (Fu et al. 2020) and the multi-task benchmark World (Yu et al. 2020), which encompasses variety of dataset settings and tasks. We adopt the normalized score metric proposed by the D4RL benchmark, and all experiments are averaged over five random seeds. Please refer to Appendix for more experimental details.
5.1 Experiment Setting
We extract offline behavior policies from multi-domains D4RL benchmarks, including locomotion and navigation tasks. Specifically, The locomotion tasks feature online data with varying levels of expertise. The navigation task requires composing parts of sub-optimal trajectories to form more optimal policies for reaching goals Muloco Ant robot.
Baselines. We compare UBER with baselines with various behavior extraction methods. We
compare our method with BC-PEX, which uses a behavior-cloning objective and other unsuper- vised behavior extraction methods (Yang et al., 2022), including OPAL (Ajay et al. 2020) and PARROT (Singh et al. 2020). We also compare our method with a strong ptilinn-to-onllee method, RLPD Ball et al. 2023) and unsupervised data sharing method, UDS (Yu et al. 2022). Specifi-
7

---

hopper-medium
walker2d-medium
halfche
Uoba
USCA DATKSET
UsbH
0.8
SICAETT
2.
2-4
DATASET
Frequency
Frequency
6.6
S.C.
0.2
22
0.4
0.0
1320
1500
2300
3000
1920
2005
4030
1200
2000
5036
4000
2000
Retur
Return
Retur
hopper-medium-exDOe
Selkkerd--edium-eee
BC
BC
Bc
LBC
Leen
Usen
DATYSET
Frequency
Frequency
Frequency
0.0
0.5
0.0
0.0
0.0
505
1000
1300
3000
2100 3320
3100
1000
2005
3:00
0003
3000
2050
6003
5003
0300
13000
32000
Return
Return
Retur
Figure 4: The return distribution the dataset (DATASET) and different behaviors generated from random intent priors (UBER) and behavior cloning (BC). UBER can generate diverse behavior set that span return distribution wider than the original dataset.
500k step scores
RLPD (w true reward)
BC-PEX
UBER (Ours)
RLPD (w true reward)
umaze umaze-diverse
0.0+10.0
99.
97.9=0.3
99.9 99.911.11
0.0+00.0
99.9
98.8=0.5
medium-play
0.0+1..0
0.0+0.00
94.0
98.7++0.9
medium-diverse
0.0+-0.0
0.0+00.0
96.5-1.5 1.5
1..
large-play
0.0+-0.0
0.0+00.0
83.3 7.7
94.8+
large-diverse
0.0:10.0
0.010.00
88.7=1.5
93.
100k step scores RLPD (w) true reward)
BC-PEX
UBER (Ours)
RLPD (w true reward)
umaze
0.0+10.0
95.8++1.4
90.7=1.8
82.5++0.2
umaze-diverse
0.0+00.0
95.5++3.7
94.5=1.1
83.3++0.3
medium-play 0.0:10.0
0,0+0.0
0,040.0
91.9-2.9
79.5
medium-diverse
0.040.00
1.4
1..
large-play
0.0.1..0
0.0.0..0
62.8 2.6
62.1
large-diverse
0.0+0..0
0.0+0.00
82.7=2.8
61.444.44
Table 1: Comparison between UBER and baselines in the online phase in the Antmaze domain at 100k and 500k environment steps. Pure online methods without an offline dataset fail completely due the difficulty of exploration.
cally, BC-PEX first extracts behavior from the offline dataset based on behavior cloning and then conducts online policy reuse with PEX (Zhang et al., 2023) following the schedule of Algorithm 2 As for OPAL-PEX and PARROT-PEX. we extract behavior based on the VAE and FLOWW models, and then conduct the same online policy reuse pipeline. We set the reward of the offline dataset to zero to implement UDS-PEX As for the pure online baseline, we use RLPD without prior offline data, named RLPD (no prior) as our baseline.
5.2 Experimental Results
Answer of Question 1: To show that UBER can generate diverse behaviors, we plot the return distribution of behaviors generated by UBER, as well as the original return distribution from the dataset. The experimental results are summarized in Figure We also provide the entropy of the return distribution Table 3 for a numerical comparison. We can see that (1) UBER can extract policies that perform better than the dataset, especially when the dataset does not have high high-quality

---

pueh-uoll-2-esper
1000
USER
Bater
Edlode Rouur
10:00
Eddock Rluer
DC-PEX
(Dwn)
2000
PFX
FEX
PFX
Edlode
28.00
T0O
"DS
2000
1030
1ea0
1000
Milion Steps
Millin Stops
Milloon Steps
Millin Stgs
pick place wall :2 replay
pish bock x2 reg
push wall 1a2l-1222 roplay
shelf place v2 replay
1000
(0um)
(Dam)
(Dmr))
Eplode Retum
300
Eguadd Roum
Experie Beeum
usen (0un)
Milice
Million
Million
Figure 5: Experimental results on the meta-world based on two types of offline datasets, expert and replay. All experiment results were averaged over five random seeds. Our method achieves better or comparable results than the baselines consistently.
hopper expertiv0
hopper
expert (PEX)
Eplood Rature
203-
Uner (CUP)
Epphock Return
100
(CUP)
(PEX)
LMFR (oup)
200
(CUr))
0.30
0.35
3.30
0.25
01:00
0.00
0.15
6.30
0.35
0,30
0.10
0.35
0.30
0.35
0.50
0.00
0.15
0.50
0:35
0.30
Millon Stees
Millon
Steps
Miliion
Steps
Million Staps
Figure 6: Ablation study for the online policy reuse module. Using policy expansion (PEX) for downstream tasks is better than critic-guidee policy reuse (CUP) in general.
data; (2) UBER can learn a diverse set of behaviors, spanning a wider distribution than the original dataset. We hypothesize that diversity is one of the keys that UBER outperform behavior cloning-based methods.
Further, we conduct experiments to test if the set of random intents can cover the true intent. We calculate the correlation of N 256 random rewards with the true reward and measure the linear projection error. The results in Table 2 indicate that random intents have a high correlation with true intent, and linear combinations of random rewards can approximate the true reward function quite well.
To test whether UBER can learn useful behaviors for downstream tasks, we compare UBER with BC-PEX, AVG-PEX, and TD3 in the online phase in the Mujoco domains. The experimental results Figure 2 show that our method achieves superior performance than baselines. By leveraging the behavior library generated from UBER, the agent can learn much faster than learning from scratch, as well as simple behavior cloning methods. Further, the results in Figure 3 show that UBER also performs better than behavior extraction and data sharing baselines in most tasks. This is because prior methods extract behaviors a behavior-cloning manner, which lacks diversity and leads to degraded performance for downstream tasks.
In addition, to UBER's generality across various offline and online methods on various domains, we also conduct experiments on Antmaze tasks. We use IQL (Kostrikov et al., (021] and RLPD (Ball al., (2023) as the backbone of offline and online algorithms. The experimental results in Table show that our method achieves stronger performance than baselines. RLPD relies heavily on offline data and RLPD without true reward has zero return on all tasks. Differently, UBER extracts a useful behavior policy set for the online phase, achieving similar sample efficiency the oracle method that has reward label for the offline dataset. We also provide the learning curve for the antmaze environment in Figure I1

---

hopper v0
hopper- mwdimm-deeemmm expert-vO
hopper expert.vl
walker
Epkazt Beuu
Eplodde Retumn
Renum
Epeode Retum
UBEk (0am)
AX0 FIX
100
TB3
55
0.0
2.16
0.29
0.25
2.10
0.16
0.333
1:35
0.30
0.30
0.55
1:30
125
0.50
0,55
0.20
0.26
0.33
Millon Steps
Millio Stops
Million Stops
Mi lon Steoe
Figure 7: Ablation study for the random intent priors module. Using average reward for policies learning can lead to nearly optimal performances on the original task. Nevertheless, our method outperforms such a baseline (AVG-PEX) consistently, showing the importance of behavioral diversity.
Answer of Question 2: To test whether the behavior learned from UBER can benefit multiple downstream tasks, we conducted the multi-task experiment on Meta-Worid (Yu et al. (020), which requires learning various robot skills to perform various manipulation tasks. We first extract behaviors with random intent priors from the selected tasks and then use the learned behaviors for various downstream tasks. The experimental results in Appendix show that the prior behaviors of UBER can be successfully transferred across multi-tasks than baselines. Please refer to Appendix B for the experimental details and results.
Answer of Question 3: To understand what contributes to the performance of UBER, we perform ablation studies for each component of UBER. We first replace the policy expansion module with critic-guided policy reuse (CUP: Zhang al., 2022). to investigate the effect of various online policy reuse modules. The experimental results in Figure show that UBER+PEX generally outperforms UBER+CUP, We hypothesize that CUP includes hard reuse method, which requires the optimized policy to be close to the reused policies during training, while PEX is a soft method that only uses the pre-trained behaviors to collect data. This makes PEX more suitable to our setting since UBER may generate undesired behaviors.
Then, we ablate the random intent priors module. We use the average reward in the offline datasets to learn offline policies and then follow the same online policy reuse process in Algorithm 2 Using average reward serves as strong baseline since it may generate near-optimal performance as observed in Shin et al. (2023). We name the average reward-based offline behavior extraction method as AVG- PEX. The experimental results in Figure 7 show that while average reward-based offline optimization can extract some useful behaviors and accelerate the downstream tasks, using random intent priors performs better due to the diversity of the behavior set.
6 Conclusion
This paper presents Unsupervised Behavior Extraction via Random Intent Priors (UBER), a novel approach to enhance reinforcement learning using unsupervised data. UBER leverages random intent priors to extract diverse and beneficial behaviors from offline, reward-free datasets. Our theorem justifies the seemingly simple approach, and our experiments validate UBER's s effectiveness in generating high-quality behavior libraries, outperforming existing baselines, and improving sample efficiency for deep reinforcement learning algorithms.
UBER unlocks the usage of abundant reward-free datasets, paving the way for more practical applications of RL. UBER focuses learning behaviors, which is orthogonal representation learning. It is an interesting future direction to further boost online sample efficiency by combining both approaches. Consuming large unsupervised datasets one of the keys to developing generalist and powerful agents. We hope the principles and techniques encapsulated in UBER can inspire further research and development unsupervised reinforcement learning.
7 Acknowledegement
This work is supported in part by Science and Technology Innovation 2030 "New Generation Artificial Intelligence" Major Project (No. 2018AAA0100904) and the National Natural Science Foundation of China (62176135).
10

---

References
Ajay, A., Kumar, A., Agrawal, P., Levine, S., and Nachum, O. Opal: Offline primitive discovery for accelerating offline reinforcement learning. arXiv preprint arXiv:2010. 13611, 2020.
Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J.,
Pieter Abbeel, O., and 7aremba, W. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
Ball, P. J., Smith, L., Kostrikov, I., and Levine, S. Efficient online reinforcement learning with offline data. arXiv preprint urXiv:2302.02944 2023.
Burda, Y., Edwards, H., Pathak, D., Storkey, A., Darrell, T., and Efros, A. A. Large-scale study of curiosity-driven learning. arXiv preprini arXiv: 1808. 04355, 2018.
Cai. Q.. Yang. Z., Jin, C., and Wang. Z. Provably efficient exploration in policy optimization. In International Conference on Machine Learning. pp. 1283-1294. PMLR, 2020.
Campos, V., Sprechmann, P., Hansen, S., Barreto, A., Kapturowski, S., Vitvitskyi, A., Badia, A. P., and Blundell, C. Beyond fine-tuning: Transferring behavior in reinforcement learning. arXiv preprint arXiv:2 102. 13515, 2021.
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J. Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650 2021.
Chua, K., Calandra, R., McAllister, R., and Levine, S. Deep reinforcement learning in handful of trials using probabilistic dynamics models. Advances neural information processing systems, 31, 2018.
Dadashi, R., Hussenot, L., Geist, M., and Pietquin, O. Primal wasserstein imitation learning. arXiv preprint arXiv:2006.0466 04678, 2020.
Degrave, J., Felici, F., Buchli, J., Neunert, M., Tracey, B., Carpanese, F., Ewalds, T., Hafner, R. Abdolmaleki, A., de Las Casas, D., et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897):4144419 2022.
Eysenbach, B., Gupta, A., Ibarz, J., and Levine, S. Diversity is all you need: Learning skills without reward function. arXiv preprint 1802. 2018.
Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072// 2020.
Fujimoto, S. and Gu, S. S. A minimalist approach to offline reinforcement learning, Advances in neural information processing systems, 34:20132-20145, 2021.
Fujimoto, S., Hoof, H., and Meger, D. Addressing function approximation error in actor-critic
methods. In International conference machine learning, pp. 1587-1596. PMLR, 2018.
Ghosh, D., Ajay, A., Agrawal, P., and Levine, S. Offline rl policies should be trained to be adaptive.
In International Conference on Machine Learning, pp. 7513-7530. PMLR, 2022.
Ghosh, D., Bhateja, C., and Levine, S. Reinforcement learning from passive data via latent intentions. arXiv preprini arXiv:2304.04770 2023.
Glorot, X. and Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
Hansen, S., Dabney, W., Barreto, A., Van Wiele, T., Warde-Farley, D., and Mnih, V. Fast task inference with variational intrinsic successor features. arXiv preprint
2019.
He, K., Zhang, X., Ren, S., and Sun, J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.

---

Ho, J. and Ermon, S. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
Hu, H., Yang, Y., Zhao, Q. and Zhang, C. On the role of discount factor in offline reinforcement learning. International Conference on Machine Learning, pp. 9072-9098. PMLR, 2022.
Hu, H., Yang, Y., Zhao, Q. and Zhang, C. The provable benefits of unsupervised data sharing for offline reinforcement learning. arXiv preprint arXiv:2302.13491 2023.
Hussein, A., Gaber, M. M., Elyan, E., and Jayne, C. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):1-35, 2017.
Jin, C., Yang, Z., Wang, Z., and Jordan, M. I. Provably efficient reinforcement learning with linear function approximation. In Conference Learning Theory, pp. 2137-2143. PMLR, 2020.
Jin, Y., Yang, Z., and Wang, Z. Is pessimism provably efficient for offline rl? In International Conference Machine Learning, pp. 5084-5096. PMLR, 2021.
Kostrikov, I., Nair, A., and Levine. S. Offline reinforcement learning with implicit q-learning. arXiv preprint ar 1rXiv:2100.06166 2021.
Kumar, A., Zhou, A., Tucker, G., and Levine, S. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 18:1179-1191 1191, 2020.
Lee, S., Seo, Y., Lee, K., Abbeel, P., and Shin, Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning, pp. 1702-1712. PMLR, 2022.
Li, A., Misra, D., Kolobov, A., and Cheng, C.-A. Survival instinct in offline reinforcement learning. arXiv preprint arXiv:2306.03286 2023.
Liu, H. and Abbeel, P. Aps: Active pretraining with successor features. In International Conference on Machine Learning, pp. 6736-6747. PMLR, 202 la.
Liu, H. and Abbeel, P. Behavior from the void: Unsupervised active pre-training. Advances in Neural Information Processing Systems, 34:18459 008459-18733, 18473, 2021b.
Lu, C., Ball, P. J., Rudner, T. G., Parker-Holded J., Osborne, M. A., and Teh, Y. W. Challenges and opportunities in offline reinforcement learning from visual observations. arXiv preprint arXiv:2206.04779 2022.
Lu, X. and Van Roy, B. Information-thhoreete confidence bounds for reinforcement learning, Advances in Neural Information Processing Systems, 32, 2019.
Ma, X., Yang. Y., Hu, H., Liu, Q., Yang. Zhang, C., Zhao, Q., and Liang, B. Offline reinforcement learning with value- based episodic memory. arXiv preprint arXiv:2110.09796, 2021.
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. Playing atari with deep reinforcement learning, arXiv preprint arXiv:1312.5602, 2013.
Mutti, M., Pratissoli, L., and Restelli, M. A policy gradient method for task-agnostic exploration. In 4th Lifelong Machine Learning Workshop a ICML 2020, 2020.
Nair, A., Gupta, A., Dalal, M., and Levine, S. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020.
OpenAl. Gpt-4 technical report, 2023.
Osband, I. and Van Roy, B. Why is posterior sampling better than optimism for reinforcement learning? In International conference machine learning, pp. 2701-2710. PMLR, 2017.
Osband, I., Blundell, C., Pritzel, A., and Van Roy, B. Deep exploration via bootstrapped dqn. Advances in neural information processing systems, 29, 2016.
12

---

Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. arXiv preprint
2022.
Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp.
PMLR, 2017.
Pathak, D., Gandhi, D., and Gupta, A. Self-supervised exploration via disagreement. In International conference on machine learning, pp. 5062-5071. PMLR, 2019.
Pomerleau, D. A. Alvinn: An autonomous land vehicle in neural network. Advances in neural
information processing systems, 1, 1988.
Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J., Todorov, E., and Levine, S. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprinl arXiv: / 7709. 10087, 2017.
Rashidinejad, P., Zhu, B., Ma, C., Jiao, J., and Russell, S. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems, 11716. 2021.
Rudi, A. and Rosasco, L. Generalization properties of learning with random features. Advances in neural information processing systems, 30, 2017.
Russo, D. and Van Roy, B. Learning to optimize via posterior sampling. Mathematics of Operations Research, (214)):2221112)) 1243, 2014.
Seo, Y., Chen, L., Shin, J., Lee, H., Abbeel, P., and Lee, K. State entropy maximization with random encoders for efficient exploration. In International Conference on Machine Learning, pp. 9454-9454. PMLR, 2021.
Sharma, A., Gu, S., Levine, S., Kumar, V., and Hausman, K. Dynamics-awar unsupervised discovery of skills, arXiv preprint arXiv: 1FXX::1977.01557 2019.
Shin, D., Dragan, A. D., and Brown, D. S. Benchmarks and algorithms for offline preference-based reward learning. arXiv preprint arXiv:2301.01333 2023.
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, Panneershelvam, V., Lanetot, M., al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484-489, 2016.
Singh, A., Liu, H., Zhou, G., Yu, A., Rhinehart, N., and Levine, S. Parrot: Data-driven behavioral priors for reinforcement learning. arXiv preprint arXiv:2011.10022 2020.
Song. Y., Zhou, Y., Sekhari, A., Bagnell, J. A., Krishnamurthy, A., and Sun, W. Hybrid rl: Using both offline and online data can make rl efficient. arXiv preprint arXiv:2210.06718, 2022.
Uehara, M. and Sun, W. Pessimistic model-based offline reinforcement learning under partial coverage. arXiv preprint arXiv:2 2021.
Wang, Y., Zhang, J., Kan, M., Shan, S.. and Chen, X. Self-supervised equivariant attention mechanism for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12275-12284, 2020.
Xiao, H., Herman, M., Wagner, J., 7iesche, S., Etesami, J., and Linh, T. H. Wasserstein adversarial imitation learning. arXiv preprint arXiv: /906/0901111100 13, 2019.
Xie, T., Jiang, N., Wang. H.. Xiong, C., and Bai, Y. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Advances in neural information processing systems, 34: 17395-27407 2021.
Xie, T., Foster, D. J., Bai, Y., Jiang, N., and Kakade, M. The role of coverage online reinforcement learning, arXiv preprint 24.107220. 2022.
13

---

Xiong, W., Zhong, H., Shi, C., Shen, C., Wang. L., and Zhang. T. Nearly minimax optimal offline reinforcement learning with linear function approximation: Single-agent mdp and markov game. arXiv preprini arXiv:2205. /5512, 2022.
Yang, L. and Wang, M. Reinforcement learning in feature space: Matrix bandit, kernels, and regret bound. In International Conference on Machine Learning, pp. 10746-10756. PMLR, 2020.
Yang, Y., Ma, X., Li, C.,, Zheng, Z., Zhang, Q. Huang, G., Yang. J., and Zhao, Q. Believe what you Implicit constraint approach for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34: 24:10299-10312 2021.
Yang. Y., Hu, H., Li, W., Li, S., Yang, J., Zhao, Q. and Zhang, C. Flow to control: Offline reinforcement learning with lossless primitive discovery. arXiv preprint 0rXiv:2212.01105,22000
Yarats, D., Fergus, R., Lazaric, A., and Pinto, L. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2 2021a. 2021a.
Yarats, D., Fergus, R., Lazaric, A., and Pinto, L. Reinforcement learning with prototypical represen- tations. In International Conference on Machine Learning, pp. 11920-11931. PMLR, 2021b.
Ye, W., Zhang, Y., Abbeel, P., and Gao, Y. Become a proficient player with limited data through watching pure videos. In The Eleventh International Conference Learning Representations, 2022.
Yu, Т., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1 100. PMLR, 2020.
Yu, T., Kumar, A., Chebotar, Y., Hausman, K., Levine, S., and Finn, C. Conservative data sharing for multi-task offline reinforcement learning. Advances in Neural Information Processing Systems, 34: 11501-11516, 2021.
Yu, T., Kumar, A., Chebotar, Y., Hausman, K., Finn, C.,, and Levine, S. How to leverage unlabeled data offline reinforcement learning. In International Conference on Machine Learning, pp. 25611 1-2665.. PMLR, 2022.
Zhang, H., Xu, W., and Yu, H. Policy expansion for bridging offline-to-onlie reinforcement learning. arXiv preprint arXiv:2302.00933, 2023.
Zhang. J., Li, S., and Zhang, C. Cup: Critic-guided policy reuse. arXiv preprint 21000:2210.08153, 2022.
14

---

A Missing Proofs
A.1 Proof of Proposition 4.1
Proposition A.1 (Proposition 4.1 restated). For any behavior π. there exists an intent with reward function r,,)),) such that the optimal policy under r(y, 2). That is, for any л h=11: Th.
(A), there exists 2 E Z such that
Vi, Moreover, if π is deterministic, then there exists 2 E Z such that π is the unique optimal policy under r(),,)), the sense that for all optimal policy nH (+1s) (-(ss) dd (s) > O.
(s, a)s, E S Ax [H]
Proof. For any policy ı and any reward function rh (n 2), we have
V (8h 2)
Σ
Emr((s ra(s,a, z)d7 (s, dddaa
(11)
I=h
where di (s, a, si.) is the visitation density at step for ı starting from 8t.
Let Th as, = 1(3s, E S,
S,di (s a, sn)) > 0), then we have
Vr
H - h Vn,max Vi[ (-)=.
(12)
is deterministic, then (s, a) is a one-hot distribution at any given s. So any n' such that there exists Sh. (-(s)))) # 7/ (-1sh) and dz (sh) > O, we have
f rm rr(Ss: a, 2) (sh, a)da < 1,
Then have V (sh, 2) max which means n/ optimal. Therefore, the optimal policy is unique.
Proposition h. shows that any deterministic policy can be uniquely determined by some intent 2 so that it is extractable via standard RL.
A.2 Proof of Theorem 4.2
Theorem A.2 (Theorem
restated.). Consider linear MDP as defined in Definition 2.1 With an
offline dataset D with size N, and the PEVI algorithm (Jin et al. 2021), the suboptimality of learning from an intent E Z with size 2 satisfies
H3,
SubOpt(#; S 400
(13)
N
with probability 5 for sufficiently large N, where = log absolute constant and
4dN|
is logarithmic c is an
x Da, n2
c max sup
he H |
x. Epp
with
Enghh (ls.a)dd,, a)o(s, JI Ep, = ((s,
Proof. Using the fact that any linear reward function still linear MDP, we can reuse the proof for standard offline RL.
Let 8' |2j and following Jin et 2021 we have the result immediately with a union bound.
Note that the original bound Jin et al. 2021 scales as (dd)) but can be improved to (dd)) by finer analysis without changing the algorithm (Xiong et al. 2022).
15

---

A.3 Proof of Theorem 4.3
Proof. Each random reward function can be seen as one dimension of random feature for linear regression. Following Theorem in Rudi & Rosasco (0117), and note that the true reward function has zero error, have that the linear estimator Tridge generated by ridge regression satisfy
((a.))-((Piddell - r(s,a))² B(sxa)~p
Σ wridge r:(s, a) -r(s,a) sc (18/0)/
Noting that
Σ warr(ss r(s,aa) -r(s,))
Σ mr r: ()o.)))rraaa) r(s,a))
min (ssaamp
- B(sra)pp
we have result immediately.
Experiments on Multi-task Transfer
We evaluate method Meta-World (Yu al. 2020), popular reinforcement learning benchmark composed of multiple robot manipulation tasks. These tasks are correlated (performed by the same Sawyer robot arm) while being distinct (interacting with different objectives and having different reward functions). Following Zhang et al. (2022), we use datasets from three representative tasks: Reach, Push, and Pick-Place, shown in Figure SarreBaan and we choose as the target tasks.
Answer of Question 2:
Push-Wall Wall, Pick-Place aall, Dush-Back, and
Pick-Placc-v2
Push-v2
Reach-v2
Figure 8: Visualization of three representative tasks in the metaworld.
Two types offline datasets were considered: an expert policy dataset and replay dataset generated by TD3. The expert dataset comprised O.1M data points obtained from expert demonstrations, while the replay dataset consisted of 0.3M data points sampled from 3M TD3 experiences.
Our experimental results, depicted in Figure 9 reveal that UBER outperforms the baselines in most tasks. This underscores UBER's ability effectively repurpose learned behaviors from various tasks for different downstream applications.
16

---

C
Additional Experiments
Correlation and projection error of random rewards. To investigate whether random rewards has good coverage and high correlation with the true reward, we calculate the maximum correlation between random rewards and the true reward, as well as the projection error of the true reward on the linear combination of random rewards on Mujoco tasks, as shown in Table 2
Tasks
Max Correlation
Min Correlation
Projection Error
hopper-medium-vel
0.569
-0.568
0.016
0,498
-0.540
0,015
hopper-expert-vd
0.423
-0.415
0.011
halfcheetah- seathheetah-mdiium--vte medium-vv
0.569
-0.568
0.016
halfcheetah- medium expertt-v
0.605
-0.589
0.047
halfcheetah- halfcheetab--expert-v0
0.370
-0.461
0.021
walker2d-medium--v0
0.475
-0.582
0.046
walker22d-medium-exprree walker2dl-xpennt-v0)
0.495
-0.472
0.042
0.358
-0.503
0.019
Table 2:
Minimum and maximum correlation and linear projection error for the true reward function using random reward functions on various tasks. The projection error e is defined as c = ||| - -ill /||l where f is the best approximation using linear combination of random rewards. Note that random reward functions (represented by neural networks) based on (s, a) inputs rather than entirely random ones, which have near-zero correlation with the true reward and 40% projection error.
Entropy of return distribution. To numerically show that UBER can generate a diverse behavior set beyond the original dataset, we calculate the entropy of the return distribution for each task, as shown in Table
Methods
halfcheetah-m halfcheetah-mer halfcheetah-e hopper-m
hopper-me hopper-e
BC
0
DATASET
1.21
1.47
0.86
0.71
1.97
0.82
UBER
2.01
1.92
2.69
2.45
2.32
1.11
Methods
walker2d-me walker2d-m
walker2d-e antmaze-md antmaze-mr
BC
0
0
0
DATASET
1.44
1.88
2.43
0.63
0.29
UBER
2.67
2.61
0.41
1.77
1.69
Table Entropy of the return distribution for different methods. Our method generate behavior set that has higher entropy in the return distribution than the original dataset consistently across various tasks.
Distribution of behaviors. To show that UBER can generate diverse behaviors, we plot the distribution of behaviors generated by UBER, well as the original distribution from the dataset. The experimental results are summarized in Figure TO We can see that when there are expert behaviors, most of the behaviors generated by UBER successfully lock the optimal policy. When there are multiple modes of behavior like medium -ummeeeert, UBER can learn both behaviors, matching the distribution of the dataset. When there are no expert behaviors, UBER can learn diverse behaviors, some of which can achieve near-optimal performance. We hypothesize that diversity is one of the keys that UBER can outperform behavior cloning-basee methods.
Learning curve the Antmaze environment. Here we provide detailed learning curve for the Antmaze experiment to demonstrate the efficiency of UBER. We can see from Figure 1T that UBER learns a good strategy in hard exploration tasks like med and large, indicating UBER's strong ability learn useful behaviors.
17

---

Ablation study for the online policy reuse module. We first replace the policy expansion module with CUP (Zhang et al., 2022). UBER (CUP) to investigate the effect of various online policy reuse modules. The experimental results in Figure T2 show that UBER+PEX generally outperforms UBER+CUP. We hypothesize that CUP includes hard reuse method, which requires the optimized policy to be close to the reused policies during training, while PEX is soft method that only uses the pre-trained behaviors to collect data. This makes PEX more suitable to our setting since UBER may generate undesired behaviors.
Complete comparison with baselines. Here we provide a complete comparison with several baselines, including OPAL, PARROT and UDS, as shown in Figure B Our method outperform these baselines due to the ability to extract diverse behaviors.
Ablation study for the random intent priors module. we conduct an ablation study for the random intent priors module. We use the average reward the offline datasets to learn offline policies and then follow the same online policy reuse process in Algorithm 2 Using average reward serves as a strong baseline since may generate near-optimal performance as observed in Shin et al. (2023). We name the average reward-based offline behavior extraction AVG-PEX. The experimental results in Figure |4] show that while average reward-based offline optimization can extract some useful behaviors and accelerate the downstream tasks, using random intent priors performs better due the diversity of the behavior set.
18

---

bucckppace-wall----epere
push-back-v2-expertt
4000
Episode Return
3000
Episode Return
3000
UBER (Ours)
UBER (Ours)
2000
BC-PEX
2000
BC-PEX
TD3
TD3
1000
1000
0
3
Million Steps
Million Steps
Mhelf-place-Pee -2-ace-2-exxprtt
4000
UBER (Ours)
Episode Return
Episode Return
3000
BC-PEX
3000
UBER (Ours)
TD3
BC.PEX
2000
2000
TD3
1000
1000
0
2
3
Million Steps
Million Steps
pick paccppla-wwlll-2--rplaaa
push- -arbbbaac--2-repaay
4000
UBER (Ours)
Episode Return
3000-
Episode Return
BC-PEX
3000
UBER (Ours) BC-PEX
TD3
2000
2000
TD3
1000
1000
01
2
0.0
0.2
0.4
0.6
0.8
1.0
Million Steps
Million Steps
push-wall-v2-replay
shelf- shelf-place-v2-reppaay
4000
UBER (Ours)
UBER (Ours)
Episode Return
BC-PEX
Episode Return
3000
BC-PEX
3000
TD3
TD3
2000
2000
1000
0.0
0.2
0.4
0.6
0.8
1.0
Million Steps
Million Steps
Figure 9: Complete experimental results on the meta-world based on two types of offline datasets, expert and replay. All experiment results were conducted over five random seeds.
19

---

hopper-medium
Soopee--medumme eeppet
hopper-expert
1.0
1.0
3.6
Frequency
Frequency
0.4
0.4
2002
1200
2002
2500
2502
Return
walker2d-medium
wallker2d-mediu-eexeer
walker2d-expert
1.0
USSS DAI6SE1
JBcn
LBIR D41A6ET
6.8
Frequency
Frequency
Trequency
%
5.2
02
32
2006
3002
1002
2300
1000
5052
3000
3500
4002
5350
Return
Return
Retur
halfcheetah- medium
haaficheetahmmediumeeees medium-exxertttttt
haffchectah-expert
BC
BC
BC LAFR
LOFR
DATASET
Frequency
Frequency
Requeecy
0.
5.)
0.0
00
1000
2000
3000
4020
5000
2200
4002
0000
8200
20002
12500
2000
4000
6020
2000
10030
12000
Retur
Retum
Return
antmaze-mediummpplg
antmaze-medlum-diieers
1.5-
1.0
0.-
DNTAGET
6.8
DATKETT
Fre-quennce Segueencc
0.5
0.5
0.4
1.4
0.2
Return
Return
Figure 10: Distribution of random intent priors, datasets, and behavior cloning policies. We restrict y-axis to a small range to make the distribution clearer.
20

---

medium vlay
antmaze large play
100
100
UBER (0ars)
USER (Oun)
URER (Oun)
BC-PEX
Epront
113
Eplauds
50
100
Exhaads
113
prine
prio)
Oracee
Crock
Orocle
0.1
0.2
6.3
0.0
0.1
0.3 Milion Steps
0.2
0.0
0.1
0.3 Million Steps
0.2
0.0
6.5
Millin
ontmacee umnzze diverse-v2
antmace dimn-ue--mdium-iiveeeeee
antmace large diverse-v2
100
Egtoode fietum
75-
(Ounn)
Epiands Ratum
754
URFR (Dur))
Eptoode Retum
URFR (Ours
UC PEX
BC PEX
60
30
30
RLFD (no priee
RUPD (no prioe)
RIPD (1o prior)
25
0.0
0:22
114
0.0
0.3
0.4
005
0.0
0.4
115
Million Stops
Million Steps
Million Stape
Figure 11: Comparison between UBER and baselines in the online phase the Antmaze domain. Oracle is the performance of RLPD at 500k step. We adopt normalized score metric averaged with three random seeds.
hopper-medium--v
hogpe:mmddim-eepprttt0
hopper-expert-v0
Episode Return
100
Episode Return
100+
UBER (CUP)
UBER (PEX)
Episode Retur
100
50-
50
25
UBER (PEK)
UBER (PEX)
UBER (CUP)
UBER (CUP)
04
0.10
0.15
0.20
0.25
0.30
0.15
0.20
0.25
0.30
0.10
0.15
0.20
0.25
0.30
Steps
Million Steps
Steps
waller24-medium--0
wa wellee/2-mmedium-exxert--
walker2d-expert-v0
(PEX)
S0
(PEX)
(PEX)
Episade Retur
60
UBER (CUP)
Egisode Return
UBER (CUP)
Episode Return
UBER (CUP)
100
20
403
20
0-
0-
11:20
0.25
0.10
0.20
0:25
0.30
Millian Steps
Million Steps
Million Steps
halfcheetah- halfcheetah-medium-e v0
halfcheetah hetechhetthh-meiim----e-
halfcheetah salfiheetah-expert-vi expert-vv
80
60
100
(PEX)
Epicade Return
Epioode Return
Episode Return
UBER (CUP)
(PEX)
(PEX)
UBER (CUP)
UBER (CUP)
20
204
0-
0.300
0.2r5
0.30
0.15
0.55
0.215
0.30
Million Steps
Million Steps
Million Steps
Figure 12: Complete result for the ablation study for the online policy reuse module. Using policy expansion (PEX) for downstream tasks better than critic-guided policy reuse (CUP) in general
21

---

medium
wallirrd medium v0
walkerld medium v
ker2d vo
UBER (Qua)
UEER (Qua)
1004
LBSR (Cun)
Epiece Retuss
-2
GRAL PEX
UES-PEX
Epiuace Return
GRAL
JCS-PEX
Epieace Retue
73
OPAL
LOS-PEX
32
[Dan]
PAMIEET
PABIOTT-PK
PARIUTTPE
OPN
0.30
8.15
3.20
0.35
0.30
0.23
8.15
0:20
0.25
0.30
0.3
0.15
0.20
0.35
0.30
0.00
3.16
00:30
0.33
1.00
Steps
Million
Steps
Steps
hopper meduum expert +0
hooper expert vo
halfcheetah medium v0
halfcherthh medium export e0
Eplacce Retum
Eppocci Returs
230+
Ephacis Retur
Elaooe Betm
50-
USE
UBER (0am)
UBCR
uns PPX
UBS SFX
OPN PTX
OPWI PFX
09W PTX
PARROT
PARROOT
PARSOTT
1.0
0.35
0.20 Million Stops
2,66
0.35
0.00
0.20
Miliin Staps
Millio Steps
Milloon Stops
Figure 13: Complete result for the comparison between UBER and baselines including unsupervised behavior extraction and data-sharing method. We adopt normalized score metric averaged with five random seeds.
hopper- medium-v0
hopper medium medium expert-vO
hopper-expert-K6
Episode Return
100-
Episode Return
100
AVG PEX
UBER (Ours)
Episode Return
100
70-
UBER (Ours)
TD3
75
UBER (Ows)
30-
ma
AVG PEX
50
50
Tn3
MVG PEX
25
25
04
0.10
0.15
0.20
0.25
0.30
0.10
0.15
0.20
0.25
0.30
0.10
0.15
0.20
0.25
0.30
Million Steps
Million Steps
M Million Steps
walker2d-medium-v0
wallerrdd-mddimm-oopptty
walker2d-cxxort-v0
100
UBER (Oum)
UBER (Qum)
100
UBER (Ours)
Episode Return
25-
AVC-PEX
Episode Return
75-
AVG-PE
Epizode Return
AVC-PEX
76
50-
26
0-
0.10
0.15
0.200
0.25
0.30
0.10
0.15
0.20
0.25
0.30
0.10
0.15
0.20
0.25
0.30
Million Steps
Million Steps
Million Steps
halfcheetah medium
halfcheetah medium e0
halfcheetah expert:
100
Egisode Return
Egisode Return
Episode Return
50
40
30-
UBER (Ours)
UBER (Ours) AVG-PEX
UBER (Ours)
AVG-PEX
RVG-PEX
TD3
TDa
TDS
0.10
0.15
1.20
0.25
0.50
0.10
0.15
0.20
0.25
0.30
0.10
0.15
0.20
0.25
0.30
Millian
Steps
Million
Steps
Million Steps
Figure 14: Complete result for the ablation study for the random intent priors module. Using average reward for policies learning can lead nearly optimal performances on the original task. Neverthe- less, our method outperforms such a baseline (AVG-PEX) consistently, showing the importance of behavioral diversity.
22

---

D Experimental Details
Experimental Setting. For BC-PEX, we first extract behavior from the offline dataset based on the Behavior Cloning, then conduct online policy expansion based on Algorithm 2 For Avg-PEX, we first use the average reward for the offline optimization, then conduct online policy expansion based on Algorithm 2 For RLPD, we load the offline dataset with the true reward value for the prior data. For RLPD (no prior), we do not load offline-dataset into buffer.
Slyper-parameter,. For the Mujoco and metaworld tasks, we adopt the TD3+BC and TD3 as the backbone of offline and online algorithms. For the Antmaze tasks, we adopt the IQL and RLPD as the backbone of offline and online algorithms. We outline the hyper-parametee used by UBER in Table (4] Table and 6
Hyperparameter
Value
Optimizer
Adam
Critic learning rate
3e-4
Actor learning rate
3e-4
Mini-batch size
256
Discount factor
0.99
Target update rate
5e-3
Policy noise
0.2
Policy noise clipping
(-0.5, 0.5)
TD3+BC parameter
2.5
Architecture
Value
Critic hidden dim
256
Critic hidden layers
2
Critic activation function
ReLU
Actor hidden dim
256
Actor hidden layers
2
Actor activation function
ReI.U
Random Net hidden dim
256
Random Net hidden layers
2
Random Net activation function
ReLU
UBER Parameters
Value
Random reward dim n
256
PEX temperature
10
Table 4: flyper-parameten sheet of UBER in Mujoco tasks.
Baselines Implementation. We adopt the author-provided implementations from GitHub for TD3 TD3+BC IQL RLPD and DrQ-v2 All experiments are conducted on the same experimental setup, single GeForce RTX 3090 GPU and an Intel Core i7-6700k CPU at 4.00GHz.
https///tttabb.oom/ffjjjin//g
httpe///gtthbl.coo//ffhjjnn.11
http://ghthabcccm//ooerrrr///ppp
http://gttabocco//ceeoorrreee//
23

---

Hyperparameter
Value
Optimizer
Adam
Critic learning rate
3e-4
Actor learning rate
3e-4
Mini-batch size
256
Discount factor
0.99
Target update rate
5e-3
Policy noise
0.2
Policy noise clipping
(-0.5, 0.5)
TD3+BC parameter a
2.5
Architecture
Value
Critic hidden dim
256
Critic hidden layers
2
Critic activation function
ReLU
Actor hidden dim
256
Actor hidden layers
2
Actor activation function
ReLU
Random Net hidden dim
256
Random Net hidden layers
2
Random Net activation function
ReLU
UBER Parameters
Value
Random reward dim n
100
PEX temperature
10
Table 5: Hyper-parameters sheet of UBER in metaworld tasks.
Hyperparameter
Value
Optimizer
Adam
Critic learning rate
3e-4
Actor learning rate
3e-4
Mini-batch size
256
Discount factor
0,99
Target update rate
5e-3
IQL, parameter
0.9
RLPD parameter G
20
Ensemble Size
10
Architecture
Value
Critic hidden dim
256
Critic hidden layers
Critic activation function
ReLU
Actor hidden dim
256
Actor hidden layers
Actor activation function
ReLU
Random Net hidden dim
256
Random Net hidden layers
Random Net activation function
ReLU
UBER Parameters
Value
Random reward dim n
256
PEX temperature
10
Table 6: plaameeparameters sheet of UBER in Antmaze tasks.


Reviews:
[{"review_id": "wOk01gpgQu", "reviewer": "Reviewer_MUZS", "summary": "The paper studies the usage of unsupervised (reward-free) data to help RL, which could extract useful behaviors from offline reward-free datasets. The proposed method is called UBER, which generates random intent priors and trains the agents based on them. The procedure generates diverse behaviors which in turn helps RL learn in a sample efficient way.", "questions": "Can the authors justify the novelty part when it compares to the unsupervised RL literature?", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The proposed method is clearly motivated and presented. The illustration figure and the algorithm description are easy to read;\n\n+ The proposed method has theoretical support and proofs;\n\n+ In the empirical experiments, the proposed method is much more sample efficient than baselines due to it learns useful behaviors to help the actual task.", "weaknesses": "I feel that the \"unsupervised RL to learn a set of diverse behavior so that when the actual reward function applies, the RL agents can learn much faster\" idea appeared in the literature. And the paper seems innovative in the setting part where the unsupervised RL happens in an offline RL fashion. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for taking the time to review our manuscript and for providing insightful feedback. We appreciate the opportunity to clarify our contributions and address your concerns.\n\n**W1 \\& Q1: Justify the novelty part when it compares to the unsupervised RL literature.**\n\n**A for W1 \\& Q1:**\n\nThe novelty of our work comes from three perspectives: a new setting, a simple yet effective algorithm, and its theoretical justification.\n\nFirst, we propose a new setting: leveraging unsupervised or reward-free offline datasets to accelerate online learning for novel tasks. This helps reduce exploration costs and enable data reuse from other tasks (e.g., we can discard reward labels in the dataset from other tasks). It also enables learning from large datasets, which has the potential for behavior emergence, which is impossible for online RL due to its low efficiency. Traditional unsupervised RL methods require an online interaction environment [1,2] or oracle reward functions [3,4]. \n\nSecondly, we propose a simple yet effective method using random intentions to extract diverse and useful behaviors. \nPrevious work on online skill extraction relies on an exploration objective (like information gain) to derive diverse skills, which makes them less applicable in offline scenarios. Directly leveraging these methods with an offline dataset could introduce significant extrapolation errors. Previous techniques that employ offline dataset reuse, such as the ones presented in [3,4], predominantly utilize behavior cloning for skill extraction over extended temporal scales. While effective when the dataset closely matches the test environment, these methods might not consistently generate diverse behaviors, which are imperative for learning novel tasks.\n\nThirdly, we provide a theoretical argument for why using random reward functions for behavior extraction is sufficient and effective. We use offline RL theory and random feature theory to show the robustness and coverability in the reward space of random rewards, which justifies using random intentions.\n\n\nWe genuinely hope that our explanation clarifies the distinctiveness and potential of our work. We are eager to engage in further discourse if needed.\n\nSincerely,\n\nThe Authors\n\n\nReferences\n\n[1] Eysenbach, Benjamin, et al. \"Diversity is all you need: Learning skills without a reward function.\" arXiv preprint arXiv:1802.06070 (2018).\n\n[2] Sharma, Archit, et al. \"Dynamics-aware unsupervised discovery of skills.\" arXiv preprint arXiv:1907.01657 (2019).\n\n[3] Ajay, Anurag, et al. \"Opal: Offline primitive discovery for accelerating offline reinforcement learning.\" arXiv preprint arXiv:2010.13611 (2020).\n\n[4] Singh, Avi, et al. \"Parrot: Data-driven behavioral priors for reinforcement learning.\" arXiv preprint arXiv:2011.10024 (2020)."}, {"review_id": "tIaqhZKwpe", "reviewer": "Reviewer_wm6d", "summary": "The authors propose unsupervised behavior extraction via random intent priors (UBER), an unsupervised method for extracting and learning behaviors from an offline dataset for downstream tasks. Assuming the situations where there are no reward labels for the transitions in the offline dataset, they suggest using a set of random *intentions* and thus intention-induced random reward functions to train a set of policies. For the online downstream task learning, they employ the policy expansion scheme and reuse the learned policies along with a newly learning policy based on the output of the critic. The authors also provide theoretical results on the existence of the corresponding intention given an arbitrary behavior, a bound on the suboptimality of the policy that is trained with some intention in a linear MDP setting, and the robustness of the random reward functions. Empirically, they present the comparison of performance and analyses in MuJoCo, AntMaze, and Meta-World environments.", "questions": "- Could you provide analyses to back up the claim that the set of random *intentions* is covering the set of true intentions?\n- Regardless of my concern, I suggest the authors to check out the concurrent work [1].", "limitations": "Please take a look at the Weaknesses and Questions sections above.", "rating": 4, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "- The proposed method shows good empirical performance in general compared to the baselines on the downstream tasks in MuJoCo, AntMaze, and Meta-World.\n- The manuscript is easy to follow and clearly structured. It is also equipped with appropriate conceptual figures, which can help readers' understanding.\n- Learning from unlabeled offline behavior data is an important topic in RL, given that much more unlabeled behavior data is available compared to labeled data in the real world.", "weaknesses": "- My current major concern is that the use of random intentions doesn't seem well-justified (and motivated) to me given the current state of this submission. While Fig.4 suggests that UBER's learned behaviors cover the distribution of *behaviors* from the original offline dataset in terms of their resulting returns, it may not necessarily mean that the random intentions or the random reward functions used for the training cover/match (or are highly correlated with) the true intentions. There is a fair possibility that the pessimism in the offline RL training is playing an important role in matching the offline dataset's behaviors, which is also suggested by a concurrent work [1]. I imagine that the whole *intention* space might be too large to be covered with $N=100$ or $N=256$, but it may not matter for the online phase in practice as the policy selection is performed at every time step.\n- Some writing or editing issues\n  - A missing reference at L117.\n  - The definition (or notations) of the loss functions in Algorithms 1 and 2.\n  - I believe calling RLPD an *oracle* because of the use of the offline dataset with the true rewards for its training can be misleading.\n  - Behavior size $N$ (from the main manuscript) vs Random reward dim $n$ (from the appendix)?\n\n[1] LI, Anqi, et al. Survival Instinct in Offline Reinforcement Learning. *arXiv preprint arXiv:2306.03286*, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your constructive feedback. We've provided additional experimental results and explanations to address your concerns, and we hope the following clarifications shed light on the raised points.\n\n**Q1 \\& W1.1: The motivation and justification of using random intentions and their alignment with true intentions.**\n\n**Response for Q1 \\& W1.1:** We employ random rewards as a simple yet effective mechanism to distill diverse and useful behaviors from offline data.\n\nIn online settings, behavior extraction typically uses exploration objectives, like information gain, to produce diverse behaviors. Such objectives could be better-suited for offline settings, as they can cause extrapolation errors and result in value explosion. On the other hand, existing offline methods learn temporally-extended skills using behavior cloning, but they may not produce enough behavioral diversity. Lack of diversity leads to suboptimal performance, especially when the downstream task differs from the provided dataset.\n\n\nTo substantiate our claim that random intentions effectively cover true intentions, we highlight the following:\n\n**(Experiments)** We calculate the correlation of $N=256$ random rewards with the true reward and measure the linear projection error. Our results (Table 1 in General Response) indicate that random intentions can have a high correlation with true intentions, and linear combinations of random rewards can approximate the true reward function quite well. Note that we use random reward functions (represented by neural networks) based on $(s, a)$ inputs rather than entirely random ones, which have a near-zero correlation with the true reward and $40\\%$ projection error.\n\n\n**(Theory)** Drawing parallels to the random features theory, we consider each random reward function as a unique dimension of a random feature, enabling us to leverage the random feature theory to affirm our random intention coverage. A detailed elaboration is available in the general response.\n\n**Q2 \\& W1.2: Comparing our findings with concurrent work [1] and the role of pessimism in aligning offline behaviors.**\n\n**Response for Q2 \\& W1.2:** Concurrent work [1] indeed resonates with some of our observations, especially Theorem 4.2. While the concurrent study emphasizes the robustness of offline algorithms to reward variations, our findings suggest that they can adeptly learn tasks with various intentions $z$ as long as the optimal policy is represented in the dataset. So, Theorem 4.2 is also a kind of \"robustness over the reward\" statement for pessimistic offline algorithms. This robustness helps smooth the random reward function and makes an implicit trade-off between usefulness and diversity. This is also observed empirically since a reasonable number of behaviors have nontrivial or near-optimal performance.\n\n\nHowever, this robustness alone doesn't encapsulate the efficacy of our approach. For instance, behaviors learned with random intentions are diverse and have significant entropy in their return distributions (Figure 4). It would be contradictory if the reward function's robustness were the only driving factor. Additionally, our method consistently outperforms behavior cloning techniques, even when behavior cloning leads to near-optimal policy (e.g., in expert datasets). This further underscores the importance of behavioral diversity over mere reward robustness.\n\n**W2: Concerns about the quality of writing.**\n\n**Response for W2:** Thank you for pointing these out. We've rectified the issues in our revised manuscript to enhance clarity and readability.\n\nWe deeply appreciate your thorough feedback and sincerely hope our clarifications address your concerns. Any further feedback and discussions are highly appreciated.\n\nBest regards,\n\nThe Authors"}, {"review_id": "roZsbTG9zE", "reviewer": "Reviewer_VsN9", "summary": "This paper tackles the problem of unsupervised behavior extraction from reward-free offline data. The main idea is to pre-train multiple policies with random rewards. UBER consists of two phases. It first trains $N$ ($100$ or $256$) policies with random rewards with an offline RL algorithm (TD3+BC), and in the subsequent online phase, it continues training the N policies plus a newly initialized policy with a soft policy selector based on the Q functions. The authors show that the behaviors learned by UBER are helpful to solve downstream tasks in standard offline RL benchmarks.", "questions": "Based on the weaknesses section above, my two biggest questions are:\n- How does UBER compare to previous unsupervised behavior extraction and/or data-sharing methods? I do not expect comparisons with all the above methods, but it would be nicer if the authors could provide empirical comparisons with some of the methods in these categories (or discussions about why UBER is very different from them).\n- Why is using **random** reward functions a good idea when extracting behaviors? If this is for purely empirical reasons, I'm fine with that (though it would have required more thorough empirical evaluations in diverse environments), but at least the theorems in the paper do not seem to provide an answer to this important question.", "limitations": "The paper lacks discussions about the limitations of UBER. One limitation I can imagine is that this random reward strategy may fail in more complex environments and thus may not be scalable (though addressing this limitation may be out of the scope of this work and it does not affect my score).", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The proposed method seems novel to me and is relatively easy to implement.\n- Despite the simplicity, the behaviors learned by random rewards seem helpful in various downstream tasks in the D4RL benchmark.\n- The paper contains several analyses including ablation studies.\n- The paper is well-written and easy to understand.", "weaknesses": "- The theoretical results do not seem to justify the use of **random** rewards. Theorem 4.2 states a general convergence result in offline RL, and Theorem 4.3 states that if two reward functions are similar, the corresponding optimal value functions are also similar. I'm not convinced how these theorems support the effectiveness of *random* reward functions. Figure 8 in Appendix A.3 states that if random reward functions sufficiently cover the reward function space, any task reward function can be approximated by the closest random reward function. However, it is unclear as to how many random reward functions are needed to enjoy this benefit. For example, we may need exponentially many random reward functions (i.e., $O((1/\\epsilon)^{|S||A|})$) to cover the entire reward function space. I would have expected a complexity analysis similar to Theorem 3.1 in Chen et al. [1].\n- The reason why random rewards lead to diverse behaviors in Figure 1 may heavily depend on the (strict) early termination condition in Hopper (and the same for Walker2d). How do the behaviors from random reward functions look like in HalfCheetah and AntMaze, which do not have early termination conditions? Could the authors provide videos and/or plots similar to Figure 4 in these environments?\n- The paper lacks discussions/comparisons with prior offline unsupervised behavior (or behavioral prior) extraction methods (e.g., OPAL [2], SPiRL [3], and PARROT [4]) and prior unsupervised data sharing methods (e.g., UDS [5] and PDS [6]).\n\nTypos and minor comments\n- L117: Missing citation.\n- OPAL [2] is cited but not mentioned in the manuscript.\n- What does PEX stand for?\n\n[1] Chen et al., Self-Supervised Reinforcement Learning that Transfers using Random Features, 2023.\n\n[2] Ajay et al., OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning, 2021.\n\n[3] Pertsch et al., Accelerating Reinforcement Learning with Learned Skill Priors, 2020.\n\n[4] Singh et al., Parrot: Data-Driven Behavioral Priors for Reinforcement Learning, 2021.\n\n[5] Yu et al., How to Leverage Unlabeled Data in Offline Reinforcement Learning, 2022.\n\n[6] Hu et al., The Provable Benefit of Unsupervised Data Sharing for Offline Reinforcement Learning, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your constructive feedback. We have provided additional experimental results and explanations to address your concerns, and we hope the following clarifications shed light on the raised points.\n\n**W1: The theoretical results do not seem to justify the use of random rewards.**\n\n**A for W1:** \n\n**(Theorem 4.2)**\nThe key focus of Theorem 4.2 is that the offline performance is robust to intention $z$, as long as the dataset covers the corresponding policy $\\pi^*_z$. \nConcurrent work [7] also finds the robustness of the reward function for pessimism algorithms, which aligns with our findings. \nThis robust result allows us to learn diverse behaviors from a *single* dataset, while prior works either learn one policy from one dataset [5,6] or resort to online interactive environments [8].\n\n**(Theorem 4.3)**\nWe agree with the Reviewer that Theorem 4.3 can be coarse in showing coverage, and we thank the Reviewer for the reference.\nWe do not need to cover the whole state-action space but only the dataset distribution. Then, we can use the random feature theory for random intention coverage. Specifically, we can show that $\\tilde{O}(\\sqrt{M})$ random intentions are enough to cover the true intention, where $M$ is the size of the dataset. Please see the general response for more details. This requires ~1000 random intentions for the dataset of size $M=1e6$, which aligns well with our practice with $N=256$.\n\n**W2: How do the behaviors from random reward functions look like in HalfCheetah and AntMaze?**\n\n**A for W2:** We provide similar plots for Halfcheetah and Antmaze tasks in the attached PDF file as Figure 4 in our submission. \nWe can see that UBER is encouraging diverse behaviors regardless of strict terminating conditions. This means that the diversity does not (only) come from terminating at different timesteps but from diverse intentions.\n\n**W3 \\& Q1: Lack discussions/comparisons with previous unsupervised behavior extraction and data-sharing method.**\n\n**A for W3 \\& Q1:** As suggested, we compare with the offline unsupervised behavior extraction methods and unsupervised data sharing methods, including OPAL, PARROT, and UDS. The experimental results in Figure~1 in General Response show that UBER performs better than these baselines in most tasks.\nThe result is not surprising because prior methods extract behaviors in a behavior cloning manner, which lacks diversity and leads to degraded performance for downstream tasks, especially when the downstream tasks differ from the dataset.\n\n**Q2: Why is using random reward functions a good idea when extracting behaviors?**\n\n**A for Q2:** \n\nThe motivation for using random rewards is to provide a simple way to extract *diverse* yet useful behaviors. Previous online behavior extraction methods use an explorative objective (e.g., information gain) to acquire diverse behaviors, which is not applicable in the offline setting since it will lead to extrapolation errors and value explosion. Previous offline methods [1,2] learn temporal-extended skills in a behavior-cloning manner, which lacks diversity and leads to degraded performance for downstream tasks.\n\nThen, we justify the use of random rewards empirically and theoretically:\n\nEmpirically, using random reward functions can be justified from two aspects: the coverage of the true reward and the diversity of the behavior set with strong empirical performance.\n\n**1. Coverage:**\n\n - To show that the set of random intentions does cover the set of true intentions, we calculate the correlation with the true reward as well as linear projection error for each task. \n\n - The experimental results in Table 1 in General Response show that random intentions do have a high correlation with the true intention, and a linear combination of random rewards can well cover the true reward function. Note that we are using the random reward **functions** with $(s,a)$ as input rather than completely random. The latter will lead to near zero correlation with the true intention and about $40\\%$ projection error.\n\n**2. Diversity:**\n\n- Figure 4 in our submission clearly shows that UBER is encouraging diversity since it has a wider span of distributions over the returns. To show this more explicitly, we further calculate the entropy of the return distribution, as shown in Table 2 in the General Response. \n\nTheoretically, using random reward functions is justified from two perspectives: robustness and coverage. \n\n**1. Robustness:**\n\n- For robustness, as stated in the answer of W1, Theorem 4.2 in our work and concurrent work [3] both show that pessimism leads to robustness over rewards. This allows us to smooth out the small fluctuations in the reward function and enables learning diverse behaviors from one dataset.\n\n**2. Coverage:**\n\n- For coverage, as stated in our answer to W1, we can show that a reasonable number of random reward functions is enough to cover the true reward. Note that we do not need to cover the whole state-action space, but the support of the dataset and $\\tilde{O}(\\sqrt{M})$ random functions is sufficient, where $M$ is the size of the dataset.\n\n**L1:  The applicability of the proposed method for complex rewards.**\n\n**A for L1:** We thank the Reviewer for pointing this out, and we will discuss the limitation in the updated manuscript.\n\nThanks again for the detailed and valuable comments. We sincerely hope our response can clear your concerns and look forward to more discussions.\n\nBest,\n\nThe Authors\n\n[1] Ajay, Anurag, et al. \"Opal: Offline primitive discovery for accelerating offline reinforcement learning.\" arXiv preprint arXiv:2010.13611 (2020).\n\n[2] Singh, Avi, et al. \"Parrot: Data-driven behavioral priors for reinforcement learning.\" arXiv preprint arXiv:2011.10024 (2020).\n\n[3] LI, Anqi, et al. Survival Instinct in Offline Reinforcement Learning. arXiv preprint arXiv:2306.03286, 2023."}, {"review_id": "8BAMEqx0Dk", "reviewer": "Reviewer_822o", "summary": "The paper studies a setting where there is an offline trajectory dataset with no reward information and the goal is to extract effective behaviors from the offline data such that they can be re-used during a separate online phase to accelerate online learning. To extract effective behaviors from the offline data, the authors propose to use an offline RL algorithm (TD3+BC) to pre-train on the offline dataset with random reward functions, resulting a policy for each random reward function. Then, during the online phase, a new discrete-action policy is initialized and being optimized to select the set of pre-trained policies obtained from the offline phase using a standard online RL (TD3). The paper also provides theoretical argument for why using random reward functions for behavior extraction is sufficient and effective. Empirically, the proposed method is able to outperform existing methods on D4RL AntMaze tasks and Locomotion tasks. ", "questions": "- L190 -- the authors mention that there are also visual tasks but I could not find them in the paper. ", "limitations": "See the first point in the weakness section.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of extracting behavior prior and learning a selection policy online has been explored in some prior works (e.g., [1]). The authors should definitely discuss how this is related to the proposed approach here. Despite that, the use of random reward network for extracting the behaviors is novel (along with theoretical analyses that justify the idea)\n- Empirical results (especially on AntMazes) are strong, suggesting that the proposed method is effective at extracting behaviors that sufficient for accelerating online learning.\n- The paper is well-written and easy to follow.\n\n[1] Singh, Avi, et al. \"Parrot: Data-driven behavioral priors for reinforcement learning.\" arXiv preprint arXiv:2011.10024 (2020).", "weaknesses": "- For any behaviors that are not covered in the offline dataset, the proposed method would not be able to capture them well. If the online task requires new unseen behaviors, the learning might fail completely. This limitation should be addressed/discussed in more details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe appreciate the Reviewer for finding our work novel, effective and well-written. We provide clarification to the points the Reviewer raised as follows.\n\n**S1: Discussion and comparison with previous behavior extraction methods.**\n\n**A for S1:** Previous behavior extraction methods can be divided into two categories:\n\n- **Online Skill Extraction**: Notable methods in this category [1, 2] are proposed in the presence of an online interactive environment. Their reliance on an exploration objective (e.g., information gain) to learn diverse skills makes them less applicable in offline scenarios. Directly leveraging these methods with an offline dataset could introduce significant extrapolation errors.\n\n- **Offline Hierarchical RL**: Techniques that employ offline dataset reuse, such as the ones presented in [3, 4], predominantly utilize behavior cloning for skill extraction over extended temporal scales. While effective when the dataset closely matches the test environment, these methods might not consistently generate diverse behaviors, which are imperative for learning novel tasks.\n\nIn contrast, our work focuses on **offline unsupervised RL**, where agents extract diverse behaviors in the reward-free offline dataset. \nWe conducted additional experiments to compare with prior offline unsupervised behavior extraction methods, OPAL and PARROT. Specifically, \n\n- We use the VAE model consistent with OPAL to extract behavioral policy and reuse it based on PEX during the online phase. We name this measure OPAL-PEX.\n\n- We use the Flow model consistent with PARROT to extract behavioral policy and reuse it based on PEX during the online phase. We name this measure PARROT-PEX.\n\n- In addition, we set the reward of the dataset to 0 and then learn the offline behavioral policy. Next, reuse it based on PEX during the online phase. We name this measure as UDS-PEX.\n\nThe experimental results in Figure~1 in General Response show that UBER performs better than these baselines in most tasks.\nIt is unsurprising because prior methods extract behaviors in a behavior-cloning manner, which lacks diversity and leads to degraded performance for downstream tasks.\n\n\n**W1: The method would fail if the online task requires new unseen behaviors.**\n\n**A for W1:** \nIt's worth noting that UBER includes a randomly initialized and learnable policy in the behavior set, so it will not completely fail when the online task requires new behaviors. If so, the performance will degrade to pure online learning but not fail. We thank the Reviewer for pointing this out, and we will discuss the limitation in the updated manuscript.\n\n\n\n**Q1: The visual tasks.**\n\n**A for Q1:** These are typos and should be multi-task (i.e., meta-world) rather than visual tasks, where we have multiple datasets and downstream tasks. Please see Appendix B for more details of the experiments on multi-task settings.\n\nThanks again for your supportive comments and suggestions.\nWe sincerely hope that our response has addressed your concerns. Any further feedback and discussions are highly appreciated.\n\n\nBest,\n\nThe Authors\n\n\nReferences\n\n[1] Eysenbach, Benjamin, et al. \"Diversity is all you need: Learning skills without a reward function.\" arXiv preprint arXiv:1802.06070 (2018).\n\n[2] Sharma, Archit, et al. \"Dynamics-aware unsupervised discovery of skills.\" arXiv preprint arXiv:1907.01657 (2019).\n\n[3] Ajay, Anurag, et al. \"Opal: Offline primitive discovery for accelerating offline reinforcement learning.\" arXiv preprint arXiv:2010.13611 (2020).\n\n[4] Singh, Avi, et al. \"Parrot: Data-driven behavioral priors for reinforcement learning.\" arXiv preprint arXiv:2011.10024 (2020)."}, {"review_id": "01CuIX20Uf", "reviewer": "Reviewer_XkfL", "summary": "The authors propose UBER, a method for learning a collection of behavior policies from offline experience data lacking reward labels and ultimately adapting these behaviors in an online setting. UBER generates a collection of randomly-initialized reward models and trains a policy on the offline data using each. During online adaptation, UBER jointly learns an action-value function and policy, selecting between the pre-trained policies to collect data using the critic. In several simulated offline-to-online settings, UBER performs similarly or better than some existing algorithms.", "questions": "What is the y axis in Figure 4?\n\nWhy is RLPD an oracle? It seems to consistently outperform UBER in some settings.\n\nIs any exploration bonus/epsilon sampling used for TD3 and RLPD? It is surprising that they consistently achieve exactly zero return.\n\nMaybe I’m missing something, but in Figure 4, UBER isn’t clearly learning more diversity? For example, in walker2d, it actually seems like the dataset returns have more mass in the high-reward areas?\n\nWhy is the reward-free (vs action-free and reward-free) setting the one we should be interested in? Justifying the problem setting would be helpful.", "limitations": "Discussion of when the various assumptions (offline-only vs having online exploration available, reward-free vs reward-free and action-free) are realistic/application would be very helpful.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper studies an important and relevant problem, offline-to-online RL, and describes a simple but interesting approach to pre-training useful behavior policies for online exploration. The results are fairly promising, showing that UBER outperforms existing approaches to online adaptation after pre-training on online data. The experiments cover a decent collection of environments.", "weaknesses": "While I appreciate an attempt to rigorously understand the proposed method, the interpretations in the theoretical sections seem a bit too generous to me. For example, the suboptimality bound in Eq. 8 seems very loose (proportional to $\\sqrt d^3H^3$), requiring a massive number of samples for fairly modestly state spaces and horizons. Further, the condition in Proposition 4.3 that the infinity norm of the difference of the reward functions is small actually seems very strong to me, and it seems fairly intuitive that the values of the optimal policies under each wouldn't change much. There is no attempt to show experimentally that this norm is small for the actual random reward functions used in practice, so I'm unconvinced this proposition explains anything about the algorithm.\n\nI'm not intimately familiar enough with the area to know all relevant related work, but it seems like additional baseline methods could be included to really show that the proposed method is effective. For example, why not compare with AWAC?\n\nThe paper is overall a bit difficult to follow. In particular, some key related work (RLPD, CUP) and experimental setup (what does “using average reward to learn offline policies” in 5.2 mean?) is not explained. Figure captions are generally too short; for example, the Fig 4 caption says \"distribution of random intent priors, datasets, and behavior cloning policies.\" The y axis is \"Statistics\", with a maximum value of 100. Is this a histogram? Figures 6 and 7 are difficult to unpack because the captions are uninformative, and because the text doesn't adequately explain PEX and CUP.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\n\nWe appreciate the reviewer's valuable feedback.\n\n\n**W1: Concerns about the generosity of the theorems.**\n\n- Theorem 4.2\n\nRecent advanced analysis [1] allows us to refine the suboptimality bound of Theorem 4.2 to be $\\tilde{O}(\\sqrt{d^2H^3/N})$ without algorithmic adjustments. Then, the performance bound is (nearly) minimax optimal and can not be further improved. Also, it has a nontrivial performance bound. For Mujoco tasks, \n$V_{max}\\sim H$, $d\\sim 10$, $H\\sim 1000$, $N\\sim 1e6$, which leads to a guaranteed $68\\%$  performance.\n\nNote that the main focus of Theorem 4.2 is not on sample complexity but on **robustness**. Theorem 4.2 allows us to learn an optimal policy for **any** intention $z$ as long as the corresponding policy $\\pi^*_z$ is covered by the dataset. This indicates that offline algorithms are robust to reward functions, which allows us to learn diverse behaviors from one *single* dataset.\n\n- Proposition 4.3 \n\nWe agree with the reviewer that Proposition 4.3 can be coarse. To give a finer analysis of how the random rewards functions can cover the true reward function, we refine our theory and conduct further experiments. \n\nWe resort to the random feature theory [2] for the theory part. Specifically, we can show that $\\tilde{O}(\\sqrt{M})$ random intentions are enough to cover the true intention, where $M$ is the size of the dataset. Please see the general response for more details.\n\nFor the empirical part, to show that the set of random intentions does cover the set of true intentions, we calculate the correlation with the true reward and linear projection error for $N=256$ random rewards for each task.\n\nThe results are shown in Table 1 in our general response. Random intentions do have a high correlation with the true intention, and a linear combination of random rewards can cover the true reward function.\n\n\n**W2: Additional baselines and comparison with AWAC.**\n\n**A for W2:** As suggested, we compare with the offline unsupervised behavior extraction methods (OPAL [10] and PARROT [11]) and unsupervised data sharing methods (UDS [6]).\n\nThe experimental results in Figure 1 in the General Response show that UBER performs better than these baselines in most tasks.\nIt is unsurprising because prior methods extract behaviors in a behavior-cloning manner, which lacks diversity and leads to degraded performance for downstream tasks.\n\n**W3: Clear description.**\n\n**A for W3:** We appreciate the detailed and valuable comments.\nIn the next revision, we will improve the presentation of the paper by providing a detailed explanation of the experimental setup, other related works (PEX, CUP, and RLPD), and figure captions.\n\n**Q1: Unclear y-axis:**\n\n**A for Q1:** The y-axis represents the unnormalized frequency of each return range. We have updated the visuals to present normalized return distribution for clarity. Please refer to the attached PDF.\n\n**Q2: Why is RLPD an oracle?**\n\n**A for Q2:** RLPD is a SOTA offline-to-online method that **uses the reward information** in the offline dataset, while our method focuses on the unsupervised setting where the reward information is unavailable.\n\n\n**Q3: Exploration bonuses in TD3 and RLPD:**\n\n**A for Q3:** Neither TD3 nor RLPD uses exploration bonuses. However, for environments like Antmaze, common exploration strategies fall short due to the environment's complexity and the sparsity of reward signals. Here, offline datasets are imperative for meaningful results.\n\n**Q4: In Figure 4, UBER isn't clearly learning more diversity?**\n\n**A for Q4:** To provide a clearer view of UBER promoting diversity, we have calculated the entropy of the return distribution, as detailed in Table 2 in our General Response. UBER prominently encourages diversity across all tasks, with the sole exception being walker2d-expert. We hypothesize that expert data dominate the dataset and the task has low sensitivity to reward variations [3]. Also, in this case, the lack of diversity is not a big problem since the optimal behavior is already in the behavior set.\n\n\n**Q5: Why is the reward-free (vs action-free and reward-free) setting the one we should be interested in?**\n\n**A for Q5:**\n\n- Reward-free settings naturally appear in many real-world problems. 1. In real-world problems like robotic tasks and NLP tasks [4,5], reward labels are expensive to get, while action labels are relatively cheap. 2. The setting also appears in data-sharing problems between different tasks [6,7]. We can remove the reward label for other tasks and reuse them as reward-free data for new tasks.\n\n\n- Action labels are usually cheap while being crucial for efficient learning.\n   Their absence hinders our ability to estimate transition models or behaviors, necessitating either additional assumptions [8] or a considerable data volume [9].\n  \n\n**L1: Limitations**\n\n**A for L1:** We thank the reviewer for pointing this out, and we will add discussion for the limitation in the updated manuscript.\n\n\nThanks again for the valuable comments.\nWe hope our response has clarified your concerns.\nWe are looking forward to further feedback and discussions.\n\nBest,\n\nThe Authors\n\nReferences\n\n[1] Xiong, Wei, et al. \"Nearly minimax optimal offline reinforcement learning with linear function approximation: Single-agent mdp and Markov game.\" arXiv preprint arXiv:2205.15512 (2022).\n\n[2] Rudi, Alessandro, and Lorenzo Rosasco. \"Generalization properties of learning with random features.\" Advances in neural information processing systems 30 (2017).\n\n[3] Li, Anqi, et al. \"Survival Instinct in Offline Reinforcement Learning.\" arXiv preprint arXiv:2306.03286 (2023).\n\n[4] Ouyang, Long, et al. \"Training language models to follow instructions with human feedback.\" Advances in Neural Information Processing Systems 35 (2022): 27730-27744.\n\n[5] Christiano, Paul F., et al. \"Deep reinforcement learning from human preferences.\" Advances in neural information processing systems 30 (2017)."}]

Author rebuttal:
Dear Reviewers,


We thank all the reviewers for their constructive feedback and valuable insights. We are encouraged to learn that many found our work "novel," "interesting," "fairly promising," and "well-written." We genuinely appreciate these positive remarks.

We acknowledge the concerns raised about the lack of comprehensive theoretical results, experimental data, and the clarity of our algorithm's explanation. In response:

- We have included enhanced theoretical results.
- Our supplementary section now showcases expanded experimental results, comparing UBRE with additional unsupervised behavior extraction and data-sharing techniques.
- We've also provided a more in-depth explanation of UBER tailored to each reviewer's feedback.

We sincerely hope that these updates and clarifications will address the reviewers' concerns. More discussions and suggestions for further improving the paper are welcomed!

### New Theoretical Results

To give a stronger theoretical guarantee for using random intentions, we resort to random feature theory for a better characterization of the coverage property of random functions. Especially, we have the following theorem.

**Theorem.** Assume the reward function admits a RKHS represention $\psi(s,a)$ with $|\psi(s,a)|\leq \kappa$ almost surely. Then with $N=c_0 \sqrt{M}\log(18\sqrt{M}\kappa^2/\delta)$ random reward functions, the linear combination of the set of random reward functions can approximate the true reward function with error 

$$
\epsilon \leq c_1 \log^2(18/\delta)/\sqrt{M},
$$

with probability $1-\delta$.

The key of the proof is noticing that a random reward function can be seen as one random feature for linear regression. We will update the manuscript for the theorem and its full proof. The theorem above allows us to use $N=\tilde{O}(\sqrt{M})$ random rewards to well cover the true reward function, where $M$ is the size of the dataset.
We hope the theorem above can solve the concerns on the theoretical justification of using random intentions.

### Additional Experiments:

- Correlation ratio and linear projection error on various tasks:

| Task | Max Correlation | Min Correlation | Projection Error |
|-----|------|------|-----|
|hopper-medium-v0 | 0.569 |-0.568|0.016 |
|hopper-medium-expert-v0 |0.498 |-0.540|0.015 |
|hopper-expert-v0 | 0.423 |-0.415 | 0.011 |
|halfcheetah-medium-v0 |0.569 |-0.568|0.016 |
|halfcheetah-medium-expert-v0 |0.605 |-0.589|0.047 |
|halfcheetah-expert-v0 |0.370|-0.461|0.021 |
|walker2d-medium-v0 |0.475 |-0.582|0.046 |
|walker2d-medium-expert-v0 |0.495 |-0.472|0.042 |
|walker2d-expert-v0 |0.358 |-0.503|0.019 |

Table 1. Minimum and maximum correlation and linear projection error for the true reward function using random reward functions on various tasks. The projection error $\epsilon$ is defined as $\epsilon = ||r-\hat{r}||/||r||$ , where $\hat{r}$ is the best approximation using linear combination of random rewards.

- Distribution of random intent priors, datasets, and behavior cloning policies:

| Method\Dataset | halfcheetah-medium | halfcheetah-medium-expert | halfcheetah-expert | hopper-medium          | hopper-medium-expert | hopper-expert |
|----------------|--------------------|---------------------------|--------------------|------------------------|----------------------|---------------|
| BC             | 0                  | 0                         | 0                  | 0                      | 0                    | 0             |
| Dataset        | 1.21               | 1.47                      | 0.86               | 0.71                   | 1.97                 | 0.82          |
| UBER           | 2.01               | 1.92                      | 2.69               | 2.45                   | 2.32                 | 1.11          |

| Method\Dataset | walker2d-medium    | walker2d-medium-expert    | walker2d-expert    | antmaze-medium-diverse | antmaze-medium-play  |               |
|----------------|--------------------|---------------------------|--------------------|------------------------|----------------------|---------------|
| BC             | 0                  | 0                         | 0                  | 0                      | 0                    |               |
| Dataset        | 1.44               | 1.88                      | 2.43               | 0.63                   | 0.29                 |               |
| UBER           | 2.67               | 2.61                      | 0.41               | 1.77                   | 1.69                 |               |

Table 2. The entropy of the return distribution for each method on different tasks.

Meta-review: The paper introduces a method called UBER, which addresses the challenge of learning useful behaviors from offline data lacking reward labels. UBER achieves this by pre-training multiple policies with random reward functions in an offline phase. In the subsequent online phase, it further optimizes these policies alongside a newly initialized policy. The authors demonstrate that UBER's learned behaviors are effective for downstream tasks in various environments. 

**strengths**

* Extensive experiments
* Simplicity of the proposed method
* Theoretical analysis on connection between using random rewards and extracting behaviors

**weaknesses/suggestions**

* The clarity of writing needs improvement (in terms of explaining prior work, experimental results, and more clearly captioning figures with informative captions)
* The applicability of the proposed method for complex rewards

I think the paper makes a nice contribution that the community will find valuable. However, I encourage the authors to think carefully about how to reflect the comments or resolve the questions from reviewers in the camera ready version.
Decision: Accept (poster)