SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hao Hu $^{1}$ , Yiqin Yang $^{2}$ , Jianing Ye $^{1}$ , Ziqing Mai $^{1}$ , Chongjie Zhang $^{3}$
$^{2}$ Department of Automation, Tsinghua University
$^{3}$ Department of Computer Science & Engineering, Washington University in St. Louis
{huh22, yangyiqi19, yejn21, maizq21}@mails.tsinghua.edu.cn
Reward-free data is abundant and contains rich prior knowledge of human behaviors, but it is not well exploited by offline reinforcement learning (RL) algorithms. In this paper, we propose UBER, an unsupervised approach to extract useful behaviors from offline reward-free datasets via diversified rewards. UBER assigns different pseudo-rewards sampled from a given prior distribution to different agents to extract a diverse set of behaviors, and reuse them as candidate policies to facilitate the learning of new tasks. Perhaps surprisingly, we show that rewards generated from random neural networks are sufficient to extract diverse and useful behaviors, some even close to expert ones. We provide both empirical and theoretical evidence to justify the use of random priors for the reward function. Experiments on multiple benchmarks showcase UBER's ability to learn effective and diverse behavior sets that enhance sample efficiency for online RL, outperforming existing baselines. By reducing reliance on human supervision, UBER broadens the applicability of RL to real-world scenarios with abundant reward-free data.
Self-supervised learning has made substantial advances in various areas like computer vision and natural language processing (OpenAI, 2023; Caron et al., 2021; Wang et al., 2020) since it leverages large-scale data without the need of human supervision. Offline reinforcement learning has emerged as a promising framework for learning sequential policies from pre-collected datasets (Kumar et al., 2020; Fujimoto & Gu, 2021; Ma et al., 2021; Yang et al., 2021; Hu et al., 2022), but it is not able to directly take advantage of unsupervised data, as such data lacks reward signal for learning and often comes from different tasks and different scenarios. Nonetheless, reward-free data like experience from human players and corpora of human conversations is abundant while containing rich behavioral information, and incorporating them into reinforcement learning has a huge potential to help improve data efficiency and achieve better generalization.
How can we effectively utilize the behavioral information in unsupervised offline data for rapid online learning? In online settings, Eysenbach et al. (2018); Sharma et al. (2019) investigate the extraction of diverse skills from reward-free online environments by maximizing diversity objectives. Meanwhile, in offline settings, a rich body of literature exists that focuses on leveraging the dynamic information from reward-free datasets (Yu et al., 2022; Hu et al., 2023). However, the former approach is not directly applicable to offline datasets, while the latter approach overlooks the behavioral information in the dataset. Ajay et al. (2020); Singh et al. (2020) employ generative models to extract behaviors
Figure 1: The framework of UBER. The procedure consists of two phases. In the first offline phase, we assign different reward functions to different agents to extract diverse and useful behaviors from the offline dataset. In the second phase, we reuse previous behavior by adding them to the candidate policy set to accelerate online learning for the new task.
in the offline dataset. However, using a behavior-cloning objective is conservative in nature and has a limited ability to go beyond the dataset to enrich diversity.
To bridge this gap, we propose Unsupervised Behavior Extraction via Random Intent Priors (UBER), a simple and novel approach that utilizes random intent priors to extract useful behaviors from offline reward-free datasets and reuse them for new tasks, as illustrated in Figure 1. Specifically, our method samples the parameters of the reward function from a prior distribution to generate different intents for different agents. These pseudo-rewards encourage the agent to go beyond the dataset and exhibit diverse behaviors. It also encourages the behavior to be useful since each behavior is the optimal policy to accomplish some given intent. The acquired behaviors are then applied in conjunction with policy expansion (Zhang et al., 2023) or policy reuse (Zhang et al., 2022) techniques to accelerate online task learning. Surprisingly, we observe that random intent priors is sufficient to produce a wide range of useful behaviors, some of which even approach expert performance levels. We provide theoretical justifications and empirical evidence for using random intent priors in Section 4 and 5, respectively.
Our experiments across multiple benchmarks demonstrate UBER's proficiency in learning behavior libraries that enhance sample efficiency for existing DRL algorithms, outperforming current baselines. By diminishing the reliance on human-designed rewards, UBER expands the potential applicability of reinforcement learning to real-world scenarios.
In summary, this paper makes the following contributions: (1) We propose UBER, a novel unsupervised RL approach that samples diverse reward functions from intent priors to extract diverse behaviors from reward-free datasets; (2) We demonstrate both theoretically and empirically that random rewards are sufficient to generate diverse and useful behaviors within offline datasets; (3) Our experiments on various tasks show that UBER outperforms existing unsupervised methods and enhances sample efficiency for existing RL algorithms.
Unsupervised Offline RL. Yu et al. (2021) considers reusing data from other tasks but assumes an oracle reward function for the new tasks. Yu et al. (2022) and Hu et al. (2023) utilize reward-free data for improving offline learning but assume a labeled offline dataset is available for learning a reward function. Ye et al. (2022); Ghosh et al. (2023) consider the setting where reward and action are absent. We focus on the reward-free setting and leverage offline data to accelerate online learning.
Unsupervised Behavior Extraction. Many recent algorithms have been proposed for intrinsic behavioral learning without a reward. Popular methods include prediction methods (Burda et al., 2018; Pathak et al., 2017, 2019), maximal entropy-based methods (Campos et al., 2021; Liu & Abbeel, 2021b,a; Mutti et al., 2020; Seo et al., 2021; Yarats et al., 2021b), and maximal mutual information-based methods (Eysenbach et al., 2018; Hansen et al., 2019; Liu & Abbeel, 2021a; Sharma et al., 2019). However, they require an online environment to achieve the diversity objective. Ajay et al. (2020); Singh et al. (2020) employ generative models to extract behaviors in the offline dataset but have a limited ability to enrich diversity.
Offline-to-online learning and policy reuse. Offline-to-online RL has been popular recently since it provides a promising paradigm to improve offline further learned policies. Nair et al. (2020) is among the first to propose a direct solution to offline-to-online RL. (Lee et al., 2022) proposes to use a balanced replay buffer and ensembled networks to smooth the offline-to-online transition; Zhang et al. (2023) proposes to use the expanded policy for offline-to-online RL. Ball et al. (2023) proposes to reuse the off-policy algorithm with modifications like ensembles and layer normalization. Zhang et al. (2022) proposes a critic-guided approach to reuse previously learned policies. Zhang et al. (2023) proposes a similar and simpler way to reuse previous policies.
Imitation Learning Our method is also related to imitation learning (Hussein et al., 2017). Popular imitation learning methods include behavior cloning methods Pomerleau (1988) and inverse RL methods Ho & Ermon (2016); Xiao et al. (2019); Dadashi et al. (2020). However, they assume the demonstration in the dataset to be near optimal, while our method works well with datasets of mediocre quality.
2.1 Episodic Reinforcement Learning
We consider finite-horizon episodic Markov Decision Processes (MDPs), defined by the tuple  $(S, \mathcal{A}, H, \mathcal{P}, r)$ , where  $S$  is a state space,  $\mathcal{A}$  is an action space,  $H$  is the horizon and  $\mathcal{P} = \{\mathcal{P}_h\}_{h=1}^H$ ,  $r = \{r_h\}_{h=1}^H$  are the transition function and reward function, respectively.
A policy  $\pi = \{\pi_h\}_{h=1}^H$  specifies a decision-making strategy in which the agent chooses its actions based on the current state, i.e.,  $a_h \sim \pi_h(\cdot | s_h)$ . The value function  $V_h^\pi : \mathcal{S} \to \mathbb{R}$  is defined as the sum of future rewards starting at state  $s$  and step  $h \in [H]$ , and similarly, the Q-value function, i.e.
$$
V _ {h} ^ {\pi} (s) = \mathbb {E} _ {\pi} \left[ \sum_ {t = h} ^ {H} r _ {t} \left(s _ {t}, a _ {t}\right) \mid s _ {h} = s \right], \quad Q _ {h} ^ {\pi} (s, a) = \mathbb {E} _ {\pi} \left[ \sum_ {t = h} ^ {H} r _ {h} \left(s _ {t}, a _ {t}\right) \mid s _ {h} = s, a _ {h} = a \right]. \tag {1}
$$
where the expectation is with respect to the trajectory  $\tau$  induced by policy  $\pi$ .
We define the Bellman operator as
$$
\left(\mathbb {B} _ {h} f\right) (s, a) = \mathbb {E} \left[ r _ {h} (s, a) + f \left(s ^ {\prime}\right) \right], \tag {2}
$$
for any  $f: \mathcal{S} \to \mathbb{R}$  and  $h \in [H]$ . The optimal Q-function  $Q^{*}$ , optimal value function  $V^{*}$  and optimal policy  $\pi^{*}$  are related by the Bellman optimality equation
$$
V _ {h} ^ {*} (s) = \max  _ {a \in \mathcal {A}} Q _ {h} ^ {*} (s, a), \quad Q _ {h} ^ {*} (s, a) = \left(\mathbb {B} _ {h} V _ {h} ^ {*}\right) (s, a), \quad \pi_ {h} ^ {*} (\cdot | s) = \underset {\pi} {\operatorname {a r g m a x}} \mathbb {E} _ {a \sim \pi} Q _ {h} ^ {*} (s, a). \tag {3}
$$
We define the suboptimality, as the performance difference of the optimal policy  $\pi^{*}$  and the current policy  $\pi_{k}$  given the initial state  $s_1 = s$ . That is
$$
\operatorname {S u b O p t} (\pi ; s) = V _ {1} ^ {\pi^ {*}} (s) - V _ {1} ^ {\pi} (s).
$$
2.2 Linear Function Approximation
To derive a concrete bound, we consider the linear MDP (Jin et al., 2020, 2021) as follows, where the transition kernel and expected reward function are linear with respect to a feature map.
Definition 2.1 (Linear MDP).  $\mathrm{MDP}(\mathcal{S},\mathcal{A},\mathrm{H},\mathbb{P},\mathrm{r})$  is a linear MDP with a feature map  $\phi :\mathcal{S}\times \mathcal{A}\to \mathbb{R}^d$ , if for any  $h\in [H]$ , there exist  $d$  unknown (signed) measures  $\mu_h = (\mu_h^{(1)},\ldots ,\mu_h^{(d)})$  over  $\mathcal{S}$  and an unknown vector  $z_{h}\in \mathbb{R}^{d}$ , such that for any  $(s,a)\in \mathcal{S}\times \mathcal{A}$ , we have
$$
\mathbb {P} _ {h} (\cdot \mid s, a) = \left\langle \phi (s, a), \mu_ {h} (\cdot) \right\rangle , \quad r _ {h} (s, a) = \left\langle \phi (s, a), z _ {h} \right\rangle . \tag {4}
$$
Without loss of generality, we assume  $||\phi (s,a)||\leq 1$  for all  $(s,a)\in \mathcal{S}\times \mathcal{A}$ , and  $\max \{\| \mu_h(\mathcal{S})\| ,\| z_h\|\} \leq \sqrt{d}$  for all  $h\in [H]$ .
When emphasizing the dependence over the reward parameter  $z$ , we also use  $r_h(s, a, z)$ ,  $V_h^\pi(s, z)$  and  $V_h^*(s, z)$  to denote the reward function, the policy value and the optimal value under parameter  $z$ , respectively.
How can we effectively leverage abundant offline reward-free data for improved performance in online tasks? Despite the absence of explicit supervision signals in such datasets, they may contain valuable behaviors with multiple modes. These behaviors can originate from human expertise or the successful execution of previous tasks. To extract these behaviors, we assume that each behavior corresponds to specific intentions, which are instantiated by corresponding reward functions. Then, by employing a standard offline RL algorithm, we can extract desired behavior modes from the offline dataset using their associated reward functions. Such behavior can be reused for online fine-tuning with methods such as policy expansion (Zhang et al., 2023) or policy reuse (Zhang et al., 2022).
Then, the question becomes how to specify possible intentions for the dataset. Surprisingly, we found that employing a random prior over the intentions can extract diverse and useful behaviors effectively in practice, which form the offline part of our method. The overall pipeline is illustrated in Figure 1. Initially, we utilize a random intention prior to generating diverse and random rewards, facilitating the learning of various modes of behavior. Subsequently, standard offline algorithms are applied to extract behaviors with generated reward functions. Once a diverse set of behaviors is collected, we integrate them with policy reuse techniques to perform online fine-tuning.
Formally, we define  $\mathcal{Z}$  as the space of intentions, where each intention  $z\in \mathcal{Z}$  induces a reward function  $r_z:\mathcal{S}\times \mathcal{A}\to \mathbb{R}$  that defines the objective for an agent. Let the prior over intentions be  $\beta \in \Delta (\mathcal{Z})$ . In our case, we let  $\mathcal{Z}$  be the weight space  $\mathcal{W}$  of the neural network and  $\beta$  be a distribution over the weight  $w\in \mathcal{W}$  of the neural network  $f_{w}$  and sample  $N$  reward functions from  $\beta$  independently. Specifically, we have
$$
r _ {i} (s, a) = f _ {w _ {i}} (s, a), w _ {i} \sim \beta , \forall i = 1, \dots , N.
$$
We choose  $\beta$  to be a standard initialization distribution for neural networks (Glorot & Bengio, 2010; He et al., 2015). We choose TD3+BC (Fujimoto & Gu, 2021) as our backbone offline algorithm, but our framework is general and compatible with any existing offline RL algorithms. Then, the loss function for the offline phase can be written as
$$
\mathcal {L} _ {\text {c r i t i c}} ^ {\text {o f f l i n e}} \left(\theta_ {i}\right) = \mathbb {E} _ {\left(s, a, r _ {i}, s ^ {\prime}\right) \sim \mathcal {D} _ {i} ^ {\text {o f f}}} \left[ \left(r _ {i} + \gamma Q _ {\theta_ {i} ^ {\prime}} \left(s ^ {\prime}, \widetilde {a}\right) - Q _ {\theta_ {i}} (s, a)\right) ^ {2} \right], \tag {5}
$$
$$
\mathcal {L} _ {\text {a c t o r}} ^ {\text {o f f l i n e}} \left(\phi_ {i}\right) = - \mathbb {E} _ {(s, a) \sim \mathcal {D} _ {i} ^ {\text {o f f}}} \left[ \lambda Q _ {\theta_ {i}} (s, \pi_ {\phi_ {i}} (s)) - \left(\pi_ {\phi_ {i}} (s) - a\right) ^ {2} \right], \tag {6}
$$
where  $\widetilde{a} = \pi_{\phi_i'}(s') + \epsilon$  is the smoothed target policy and  $\lambda$  is the weight for behavior regularization. We adopt a discount factor  $\gamma$  even for the finite horizon problem as commonly used in practice (Fujimoto et al., 2018).
As for the online phase, we adopt the policy expansion framework (Zhang et al., 2023) to reuse policies. We first construct an expanded policy set  $\widetilde{\pi} = [\pi_{\phi_1}, \dots, \pi_{\phi_N}, \pi_w]$ , where  $\{\phi_i\}_{i=1}^N$  are the weights of extracted behaviors from offline phase and  $\pi_w$  is a new randomly initialized policy. We then use the critic as the soft policy selector, which generates a distribution over candidate policies as follows:
$$
P _ {\widetilde {\pi}} [ i ] = \frac {\exp (\alpha \cdot Q (s , \widetilde {\pi} _ {i} (s)))}{\sum_ {j} \exp (\alpha \cdot Q (s , \widetilde {\pi} _ {j} (s)))}, \forall i \in \{1, \dots , N + 1 \}, \tag {7}
$$
where  $\alpha$  is the temperature. We sample a policy  $i\in \{1,\dots ,N + 1\}$  according to  $P_{\bar{\pi}}$  at each time step and follows  $\pi_{i}$  to collect online data. The policy  $\pi_w$  and  $Q_{\theta}$ -value network undergo continuous training using the online RL loss, such as TD3 (Fujimoto et al., 2018):
$$
\mathcal {L} _ {\text {c r i t i c}} ^ {\text {o n l i n e}} (\theta) = \mathbb {E} _ {(s, a, r, s ^ {\prime}) \sim \mathcal {D} ^ {\text {o n}}} \left[ (r + \gamma Q _ {\theta^ {\prime}} (s ^ {\prime}, \widetilde {a}) - Q _ {\theta} (s, a)) \right] ^ {2}, \tag {8}
$$
$$
\mathcal {L} _ {\text {a c t o r}} ^ {\text {o n l i n e}} (w) = - \mathbb {E} _ {(s, a) \sim \mathcal {D} ^ {\text {o n}}} \left[ Q _ {\theta} (s, \pi_ {w} (s)) \right]. \tag {9}
$$
The overall algorithm is summarized in Algorithm 1 and 2. We highlight elements important to our approach in purple.
Algorithm 1 Phase 1: Offline Behavior Extraction
1: Require: Behavior size  $N$ , offline reward-free dataset  $\mathcal{D}^{\mathrm{off}}$ , prior intention distribution  $\beta$   
2: Initialize parameters of  $N$  independent offline agents  $\{Q_{\theta_i}, \pi_{\phi_i}\}_{i=1}^N$   
3: for  $i = 1, \dots, N$  do  
4: Sample a reward function from priors  $z_i \sim \beta$   
5: Reannote  $\mathcal{D}^{\mathrm{off}}$  as  $\mathcal{D}_i^{\mathrm{off}}$  with reward  $r_{z_i}$   
6: for each training iteration do  
7: Sample a random minibatch  $\{\tau_j\}_{j=1}^B \sim \mathcal{D}_i^{\mathrm{off}}$   
8: Calculate  $\mathcal{L}_{\mathrm{critic}}^{\mathrm{offline}}(\theta_i)$  as in Equation (5) and update  $\theta_i$   
9: Calculate  $\mathcal{L}_{\mathrm{actor}}^{\mathrm{offline}}(\phi_i)$  as in Equation (6) and update  $\phi_i$   
10: end for  
11: end for  
12: Return  $\{\pi_{\phi_i}\}_{i=1}^N$
Algorithm 2 Phase 2: Online Policy Reuse
1: Require:  $\{\pi_{\phi_i}\}_{i = 1}^N$  offline dataset  $\mathcal{D}^{\mathrm{off}}$  , update-to-data ratio  $G$    
2: Initialize online agents  $Q_{\theta},\pi_w$  and replay buffer  $\mathcal{D}^{\mathrm{on}}$    
3: Construct expanded policy set  $\widetilde{\pi} = [\pi_{\phi_1},\dots,\pi_{\phi_N},\pi_w]$    
4: for each episode do   
5: Obtain initial state  $s_1$  from the environment   
6: for step  $t = 1,\dots ,T$  do   
7: Construct  $P_{\bar{\pi}}$  According to Equation (7)   
8: Pick an policy to act  $\pi_t\sim P_{\bar{\pi}},a_t\sim \pi_t(\cdot |s_t)$    
9: Store transition  $(s_t,a_t,r_t,s_{t + 1})$  in  $\mathcal{D}^{\mathrm{on}}$    
10: for  $g = 1,\dots ,G$  do   
11: Calculate  $\mathcal{L}_{\mathrm{critic}}^{\mathrm{online}}(\theta)$  as in Equation (8) and update  $\theta$    
12: end for   
13: Calculate  $\mathcal{L}_{\mathrm{actor}}^{\mathrm{online}}(w)$  as in Equation (9) and update  $w$    
14: end for   
15: end for
How sound is the random intent approach for behavior extraction? This section aims to establish a theoretical characterization of our proposed method. We first consider the completeness of our method. We show that any behavior can be formulated as the optimal behavior under some given intent, and any behavior set that is covered by the offline dataset can be learned effectively given the corresponding intent set. Then we consider the coverage of random rewards. We show that with a reasonable number of random reward functions, the true reward function can be approximately represented by the linear combination of the random reward functions with a high probability. Hence our method is robust to the randomness in the intent sampling process.
4.1 Completeness of the Intent Method for Behavior Extraction
We first investigate the completeness of intent-based methods for behavior extraction. Formally, we have the following proposition.
Proposition 4.1. For any behavior  $\pi$ , there exists an intent  $z$  with reward function  $r(\cdot, \cdot, z)$  such that  $\pi$  is the optimal policy under  $r(\cdot, \cdot, z)$ . That is, for any  $\pi = \{\pi_h\}_{h=1}^H$ ,  $\pi_h : \mathcal{S} \to \Delta(\mathcal{A})$ , there exists  $z \in \mathcal{Z}$  such that
$$
V _ {h} ^ {\pi} (s, z) = V _ {h} ^ {*} (s, z), \forall (s, a, h) \in \mathcal {S} \times \mathcal {A} \times [ H ].
$$
Proof. Please refer to Appendix A.1 for detailed proof.
Proposition 4.1 indicates that any behavior can be explained as achieving some intent. This is intuitive since we can relabel the trajectory generated by any policy as successfully reaching the final state in
hindsight (Andrychowicz et al., 2017). With Proposition 4.1, we can assign an intent  $z$  to any policy  $\pi$  as  $\pi_z$ . However, it is unclear whether we can learn effectively from offline datasets given an intent set  $\mathcal{Z}$ . This is guaranteed by the following theorem.
Theorem 4.2. Consider linear MDP as defined in Definition 2.1. With an offline dataset  $\mathcal{D}$  with size  $N$ , and the PEVI algorithm (Jin et al., 2021), the suboptimality of learning from an intent  $z \in \mathcal{Z}$  satisfies
$$
\operatorname {S u b O p t} (\widehat {\pi}; s, z) = \mathcal {O} \left(\sqrt {\frac {C _ {z} ^ {\dagger} d ^ {2} H ^ {3} \iota}{N}}\right), \tag {10}
$$
for sufficiently large  $N$ , where  $\iota = \log (dN|\mathcal{Z}| / \delta)$ ,  $c$  is an absolute constant and
$$
C _ {z} ^ {\dagger} = \max _ {h \in [ H ]} \sup _ {\| x \| = 1} \frac {x ^ {\top} \Sigma_ {\pi_ {z} , h} x}{x ^ {\top} \Sigma_ {\rho_ {h}} x},
$$
with  $\Sigma_{\pi_z,h} = \mathbb{E}_{(s,a)\sim d_{\pi_z,h}(s,a)}[\phi (s,a)\phi (s,a)^\top ],\Sigma_{\rho_h} = \mathbb{E}_{\rho_h}[\phi (s,a)\phi (s,a)^\top ]$
Proof. Please refer to Appendix A.2 for detailed proof.
$C_z^\dagger$  represents the maximum ratio between the density of empirical state-action distribution  $\rho$  and the density  $d_{\pi_z}$  induced from the policy  $\pi_z$ . Theorem 4.2 states that, for any behavior,  $\pi_z$  that is well-covered by the offline dataset  $\mathcal{D}$ , we can use standard offline RL algorithm to extract it effectively, in the sense that learned behavior  $\widehat{\pi}$  has a small suboptimality under reward function  $r(\cdot ,\cdot ,z)$ . Compared to standard offline result (Jin et al., 2021), Theorem 4.2 is only worse off by a factor of  $\log |\mathcal{Z}|$ , which enables us to learn multiple behaviors from a single dataset effectively, as long as the desired behavior mode is well contained in the dataset.
4.2 Coverage of Random Rewards
This section investigates the coverage property of random rewards. Intuitively, using random rewards may suffer from high variance, and it is possible that all sampled intents lead to undesired behaviors. Theorem 4.3 shows that such cases rarely happen since with a reasonable number of random reward functions, the linear combination of them can well cover the true reward function with a high probability.
Theorem 4.3. Assume the reward function  $r(s, a)$  admits a RKHS representation  $\psi(s, a)$  with  $\| \psi(s, a) \|_{\infty} \leq \kappa$  almost surely. Then with  $N = c_0 \sqrt{M} \log(18 \sqrt{M} \kappa^2 / \delta)$  random reward functions  $\{r_i\}_{i=1}^{N}$ , the linear combination of the set of random reward functions  $\widehat{r}(s, a)$  can approximate the true reward function with error
$$
\mathbb {E} _ {(s, a) \sim \rho} [ \widehat {r} (s, a) - r (s, a) ] ^ {2} \leq c _ {1} \log^ {2} (1 8 / \delta) / \sqrt {M},
$$
with probability  $1 - \delta$ , where  $M$  is the size of the offline dataset  $\mathcal{D}$ ,  $c_0$  and  $c_1$  are universal constants and  $\rho$  is the distribution that generates the offline dataset  $\mathcal{D}$ .
Proof. Please refer to Appendix A.3 for detailed proof.
Theorem 4.3 indicates that RL algorithms are insensitive to the randomness in the reward function used for learning, as long as we are using a reasonable number of such functions. This phenomenon is more significant in offline algorithms since they are conservative in nature to avoid over-generalization. This partly explains why using random rewards is sufficient for behavior extraction, and can even generate policies comparable to oracle rewards. Such a phenomenon is also observed in prior works (Shin et al., 2023; Li et al., 2023).
Figure 2: Comparison between UBER and baselines in the online phase in the Mujoco domain. We adopt datasets of various quality for offline training. We adopt a normalized score metric averaged with five random seeds.
Figure 3: Comparison between UBER and baselines including unsupervised behavior extraction (PARROR OPAL) and data-sharing methods (UDS). The result is averaged with five random seeds. Our method outperforms these baselines by leveraging the diverse behavior set.