SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our evaluation studies UBER as a pre-training mechanism on reward-free data, focusing on the following questions: (1) Can we learn useful and diverse behaviors with UBER? (2) Can the learned behaviors from one task using UBER be reused for various downstream tasks? (3) How effective is each component in UBER?
To answer the questions above, we conduct experiments on the standard D4RL benchmark (Fu et al., 2020) and the multi-task benchmark Meta-World (Yu et al., 2020), which encompasses a variety of dataset settings and tasks. We adopt the normalized score metric proposed by the D4RL benchmark, and all experiments are averaged over five random seeds. Please refer to Appendix D for more experimental details.
We extract offline behavior policies from multi-domains in D4RL benchmarks, including locomotion and navigation tasks. Specifically, The locomotion tasks feature online data with varying levels of expertise. The navigation task requires composing parts of sub-optimal trajectories to form more optimal policies for reaching goals on a MuJoco Ant robot.
Baselines. We compare UBER with baselines with various behavior extraction methods. We compare our method with BC-PEX, which uses a behavior-cloning objective and other unsupervised behavior extraction methods (Yang et al., 2022), including OPAL (Ajay et al., 2020) and PARROT (Singh et al., 2020). We also compare our method with a strong offline-to-online method, RLPD (Ball et al., 2023) and an unsupervised data sharing method, UDS (Yu et al., 2022). Specifi
Figure 4: The return distribution of the dataset (DATASET) and different behaviors generated from random intent priors (UBER) and behavior cloning (BC). UBER can generate a diverse behavior set that span a return distribution wider than the original dataset.
500k step scores	RLPD (w\o true reward)	BC-PEX	UBER (Ours)	RLPD (w true reward)
umaze	0.0±0.0	99.8±0.3	97.9±0.3	99.9±0.1
umaze-diverse	0.0±0.0	99.9±0.1	98.8±0.5	99.9±0.1
medium-play	0.0±0.0	0.0±0.0	94.0±3.1	98.7±0.9
medium-diverse	0.0±0.0	0.0±0.0	96.5±1.5	98.5±1.3
large-play	0.0±0.0	0.0±0.0	83.3±7.7	94.8±1.5
large-diverse	0.0±0.0	0.0±0.0	88.7±1.5	93.5±1.6
100k step scores	RLPD (w\o true reward)	BC-PEX	UBER (Ours)	RLPD (w true reward)
umaze	0.0±0.0	95.8±1.4	90.7±1.8	82.5±0.2
umaze-diverse	0.0±0.0	95.5±3.7	94.5±1.1	83.3±0.3
medium-play	0.0±0.0	0.0±0.0	91.9±2.9	79.5±0.6
medium-diverse	0.0±0.0	0.0±0.0	91.3±1.4	75.3±1.3
large-play	0.0±0.0	0.0±0.0	62.8±2.6	62.1±3.7
large-diverse	0.0±0.0	0.0±0.0	82.7±2.8	61.4±4.4
Table 1: Comparison between UBER and baselines in the online phase in the Antmaze domain at 100k and 500k environment steps. Pure online methods without an offline dataset fail completely due to the difficulty of exploration.
cally, BC-PEX first extracts behavior from the offline dataset based on behavior cloning and then conducts online policy reuse with PEX (Zhang et al., 2023) following the schedule of Algorithm 2. As for OPAL-PEX and PARROT-PEX, we extract behavior based on the VAE and FLOW models, and then conduct the same online policy reuse pipeline. We set the reward of the offline dataset to zero to implement UDS-PEX. As for the pure online baseline, we use RLPD without prior offline data, named RLPD (no prior) as our baseline.
Answer of Question 1: To show that UBER can generate diverse behaviors, we plot the return distribution of behaviors generated by UBER, as well as the original return distribution from the dataset. The experimental results are summarized in Figure 4. We also provide the entropy of the return distribution in Table 3 for a numerical comparison. We can see that (1) UBER can extract policies that perform better than the dataset, especially when the dataset does not have high-quality
Figure 5: Experimental results on the meta-world based on two types of offline datasets, expert and replay. All experiment results were averaged over five random seeds. Our method achieves better or comparable results than the baselines consistently.
Figure 6: Ablation study for the online policy reuse module. Using policy expansion (PEX) for downstream tasks is better than critic-guided policy reuse (CUP) in general.
data; (2) UBER can learn a diverse set of behaviors, spanning a wider distribution than the original dataset. We hypothesize that diversity is one of the keys that UBER can outperform behavior cloning-based methods.
Further, we conduct experiments to test if the set of random intents can cover the true intent. We calculate the correlation of  $N = 256$  random rewards with the true reward and measure the linear projection error. The results in Table 2 indicate that random intents can have a high correlation with true intent, and linear combinations of random rewards can approximate the true reward function quite well.
To test whether UBER can learn useful behaviors for downstream tasks, we compare UBER with BC-PEX, AVG-PEX, and TD3 in the online phase in the Mujoco domains. The experimental results in Figure 2 show that our method achieves superior performance than baselines. By leveraging the behavior library generated from UBER, the agent can learn much faster than learning from scratch, as well as simple behavior cloning methods. Further, the results in Figure 3 show that UBER also performs better than behavior extraction and data sharing baselines in most tasks. This is because prior methods extract behaviors in a behavior-cloning manner, which lacks diversity and leads to degraded performance for downstream tasks.
In addition, to test UBER's generality across various offline and online methods on various domains, we also conduct experiments on Antmaze tasks. We use IQL (Kostrikov et al., 2021) and RLPD (Ball et al., 2023) as the backbone of offline and online algorithms. The experimental results in Table 1 show that our method achieves stronger performance than baselines. RLPD relies heavily on offline data and RLPD without true reward has zero return on all tasks. Differently, UBER extracts a useful behavior policy set for the online phase, achieving similar sample efficiency as the oracle method that has reward label for the offline dataset. We also provide the learning curve for the antmaze environment in Figure 11.
Figure 7: Ablation study for the random intent priors module. Using average reward for policies learning can lead to nearly optimal performances on the original task. Nevertheless, our method outperforms such a baseline (AVG-PEX) consistently, showing the importance of behavioral diversity.
Answer of Question 2: To test whether the behavior set learned from UBER can benefit multiple downstream tasks, we conducted the multi-task experiment on Meta-World (Yu et al., 2020), which requires learning various robot skills to perform various manipulation tasks. We first extract behaviors with random intent priors from the selected tasks and then use the learned behaviors for various downstream tasks. The experimental results in Appendix B show that the prior behaviors of UBER can be successfully transferred across multi-tasks than baselines. Please refer to Appendix B for the experimental details and results.
Answer of Question 3: To understand what contributes to the performance of UBER, we perform ablation studies for each component of UBER. We first replace the policy expansion module with critic-guided policy reuse (CUP; Zhang et al., 2022), to investigate the effect of various online policy reuse modules. The experimental results in Figure 6 show that UBER+PEX generally outperforms UBER+CUP. We hypothesize that CUP includes a hard reuse method, which requires the optimized policy to be close to the reused policies during training, while PEX is a soft method that only uses the pre-trained behaviors to collect data. This makes PEX more suitable to our setting since UBER may generate undesired behaviors.
Then, we ablate the random intent priors module. We use the average reward in the offline datasets to learn offline policies and then follow the same online policy reuse process in Algorithm 2. Using average reward serves as a strong baseline since it may generate near-optimal performance as observed in Shin et al. (2023). We name the average reward-based offline behavior extraction method as AVGPEX. The experimental results in Figure 7 show that while average reward-based offline optimization can extract some useful behaviors and accelerate the downstream tasks, using random intent priors performs better due to the diversity of the behavior set.
This paper presents Unsupervised Behavior Extraction via Random Intent Priors (UBER), a novel approach to enhance reinforcement learning using unsupervised data. UBER leverages random intent priors to extract diverse and beneficial behaviors from offline, reward-free datasets. Our theorem justifies the seemingly simple approach, and our experiments validate UBER's effectiveness in generating high-quality behavior libraries, outperforming existing baselines, and improving sample efficiency for deep reinforcement learning algorithms.
UBER unlocks the usage of abundant reward-free datasets, paving the way for more practical applications of RL. UBER focuses on learning behaviors, which is orthogonal to representation learning. It is an interesting future direction to further boost online sample efficiency by combining both approaches. Consuming large unsupervised datasets is one of the keys to developing generalist and powerful agents. We hope the principles and techniques encapsulated in UBER can inspire further research and development in unsupervised reinforcement learning.
This work is supported in part by Science and Technology Innovation 2030 - "New Generation Artificial Intelligence" Major Project (No. 2018AAA0100904) and the National Natural Science Foundation of China (62176135).
Xiong, W., Zhong, H., Shi, C., Shen, C., Wang, L., and Zhang, T. Nearly minimax optimal offline reinforcement learning with linear function approximation: Single-agent mdp and markov game. arXiv preprint arXiv:2205.15512, 2022.
Yang, L. and Wang, M. Reinforcement learning in feature space: Matrix bandit, kernels, and regret bound. In International Conference on Machine Learning, pp. 10746-10756. PMLR, 2020.
Yang, Y., Ma, X., Li, C., Zheng, Z., Zhang, Q., Huang, G., Yang, J., and Zhao, Q. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34:10299-10312, 2021.
Yang, Y., Hu, H., Li, W., Li, S., Yang, J., Zhao, Q., and Zhang, C. Flow to control: Offline reinforcement learning with lossless primitive discovery. arXiv preprint arXiv:2212.01105, 2022.
Yarats, D., Fergus, R., Lazaric, A., and Pinto, L. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645, 2021a.
Yarats, D., Fergus, R., Lazaric, A., and Pinto, L. Reinforcement learning with prototypical representations. In International Conference on Machine Learning, pp. 11920-11931. PMLR, 2021b.
Ye, W., Zhang, Y., Abbeel, P., and Gao, Y. Become a proficient player with limited data through watching pure videos. In The Eleventh International Conference on Learning Representations, 2022.
Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1100. PMLR, 2020.
Yu, T., Kumar, A., Chebotar, Y., Hausman, K., Levine, S., and Finn, C. Conservative data sharing for multi-task offline reinforcement learning. Advances in Neural Information Processing Systems, 34: 11501-11516, 2021.
Yu, T., Kumar, A., Chebotar, Y., Hausman, K., Finn, C., and Levine, S. How to leverage unlabeled data in offline reinforcement learning. In International Conference on Machine Learning, pp. 25611-25635. PMLR, 2022.
Zhang, H., Xu, W., and Yu, H. Policy expansion for bridging offline-to-online reinforcement learning. arXiv preprint arXiv:2302.00935, 2023.
Zhang, J., Li, S., and Zhang, C. Cup: Critic-guided policy reuse. arXiv preprint arXiv:2210.08153, 2022.
Proposition A.1 (Proposition 4.1 restated). For any behavior  $\pi$ , there exists an intent  $z$  with reward function  $r(\cdot, \cdot, z)$  such that  $\pi$  is the optimal policy under  $r(\cdot, \cdot, z)$ . That is, for any  $\pi = \{\pi_h\}_{h=1}^H$ ,  $\pi_h: S \to \Delta(\mathcal{A})$ , there exists  $z \in \mathcal{Z}$  such that
Moreover, if  $\pi$  is deterministic, then there exists  $z\in \mathcal{Z}$  such that  $\pi$  is the unique optimal policy under  $r(\cdot ,\cdot ,z)$ , in the sense that for all optimal policy  $\pi_z^*,\pi_{h,z}^* (\cdot |s) = \pi_h(\cdot |s),\forall d_h^\pi (s) > 0$
Proof. For any policy  $\pi$  and any reward function  $r_h(\cdot, \cdot, z)$ , we have
$$
V _ {h} ^ {\pi} (s _ {h}, z) = \sum_ {t = h} ^ {H} \int r _ {t} (s, a, z) d _ {t} ^ {\pi} (s, a, s _ {h}) \mathrm {d} s \mathrm {d} a, \tag {11}
$$
where  $d_t^\pi (s,a,s_h)$  is the visitation density at step  $t$  for  $\pi$  starting from  $s_h$ .
Let  $r_h(s, a, z) = \mathbb{1}(\exists s_h \in S, d_h^\pi(s, a, s_h) > 0)$ , then we have
$$
V _ {h} ^ {\pi} (\cdot , z) = H - h = V _ {h, \max } = V _ {h} ^ {*} (\cdot , z). \tag {12}
$$
If  $\pi$  is deterministic, then  $d_h^\pi (s,a)$  is a one-hot distribution at any given  $s$ . So any  $\pi^{\prime}$  such that there exists  $s_h$ ,  $\pi_h(\cdot |s_h)\neq \pi_h'(\cdot |s_h)$  and  $d_h^\pi (s_h) > 0$ , we have
$$
\int r _ {h} \left(s _ {h}, a, z\right) d _ {h} ^ {\pi^ {\prime}} \left(s _ {h}, a\right) \mathrm {d} a <   1,
$$
Then we have  $V_h^{\pi'}(s_h, z) < V_{h,\max}$ , which means  $\pi'$  is not optimal. Therefore, the optimal policy  $\pi$  is unique.
Proposition 4.1 shows that any deterministic policy can be uniquely determined by some intent  $z$  so that it is extractable via standard RL.
Theorem A.2 (Theorem 4.2 restated.). Consider linear MDP as defined in Definition 2.1. With an offline dataset  $\mathcal{D}$  with size  $N$ , and the PEVI algorithm (Jin et al., 2021), the suboptimality of learning from an intent  $z \in \mathcal{Z}$  with size  $|\mathcal{Z}|$  satisfies
$$
\operatorname {S u b O p t} \left(\widehat {\pi}; r _ {z}\right) \leq 4 c \sqrt {\frac {C _ {z} ^ {\dagger} d ^ {2} H ^ {3} \iota}{N}}, \tag {13}
$$
with probability  $1 - \delta$  for sufficiently large  $N$ , where  $\iota = \log \frac{4dN|\mathcal{Z}|}{\delta}$  is a logarithmic factor,  $c$  is an absolute constant and
$$
C _ {z} ^ {\dagger} = \max  _ {h \in [ H ]} \sup  _ {\| x \| = 1} \frac {x ^ {\top} \Sigma_ {\pi_ {z} , h} x}{x ^ {\top} \Sigma_ {\rho_ {h}} x},
$$
$$
\Sigma_ {\pi_ {z}, h} = \mathbb {E} _ {(s, a) \sim d _ {\pi_ {z}, h} (s, a)} [ \phi (s, a) \phi (s, a) ^ {\top} ], \quad \Sigma_ {\rho_ {h}} = \mathbb {E} _ {\rho_ {h}} [ \phi (s, a) \phi (s, a) ^ {\top} ].
$$
Proof. Using the fact that any linear reward function is still a linear MDP, we can reuse the proof for standard offline RL.
Let  $\delta' = \frac{\delta}{|\mathcal{Z}|}$  and following Jin et al. (2021), we have the result immediately with a union bound. Note that the original bound in Jin et al. (2021) scales as  $\widetilde{\mathcal{O}}(d^3)$ , but it can be improved to  $\widetilde{\mathcal{O}}(d^2)$  by finer analysis without changing the algorithm (Xiong et al., 2022).
A.3 Proof of Theorem 4.3
Proof. Each random reward function can be seen as one dimension of random feature for linear regression. Following Theorem 1 in Rudi & Rosasco (2017), and note that the true reward function has zero error, we have that the linear estimator  $\widehat{r}_{\mathrm{ridge}}$  generated by ridge regression satisfy
$$
\mathbb {E} _ {(s, a) \sim \rho} (\widehat {r} _ {\mathrm {r i d g e}} (s, a) - r (s, a)) ^ {2} = \mathbb {E} _ {(s, a) \sim \rho} \left(\sum_ {i} \widehat {w} _ {i} ^ {\mathrm {r i d g e}} r _ {i} (s, a) - r (s, a)\right) ^ {2} \leq c _ {1} \log^ {2} (1 8 / \delta) / \sqrt {M}.
$$
$$
\min _ {w} \mathbb {E} _ {(s, a) \sim \rho} \left(\sum_ {i} w _ {i} r _ {i} (s, a) - r (s, a)\right) ^ {2} \leq \mathbb {E} _ {(s, a) \sim \rho} \left(\sum_ {i} \widehat {w} _ {i} ^ {\mathrm {r i d g e}} r _ {i} (s, a) - r (s, a)\right) ^ {2},
$$
we have result immediately.
B Experiments on Multi-task Transfer
Answer of Question 2: We evaluate our method on Meta-World (Yu et al., 2020), a popular reinforcement learning benchmark composed of multiple robot manipulation tasks. These tasks are correlated (performed by the same Sawyer robot arm) while being distinct (interacting with different objectives and having different reward functions). Following Zhang et al. (2022), we use datasets from three representative tasks: Reach, Push, and Pick-Place, as shown in Figure 8 and we choose Push-Wall, Pick-Place-Wall, Push-Back, and Shelf-Place as the target tasks.
Figure 8: Visualization of three representative tasks in the metaworld.
Two types of offline datasets were considered: an expert policy dataset and a replay dataset generated by TD3. The expert dataset comprised 0.1M data points obtained from expert demonstrations, while the replay dataset consisted of 0.3M data points sampled from 3M TD3 experiences.
Our experimental results, depicted in Figure 9, reveal that UBER outperforms the baselines in most tasks. This underscores UBER's ability to effectively repurpose learned behaviors from various tasks for different downstream applications.
Correlation and projection error of random rewards. To investigate whether random rewards has a good coverage and a high correlation with the true reward, we calculate the maximum correlation between random rewards and the true reward, as well as the projection error of the true reward on the linear combination of random rewards on Mujoco tasks, as shown in Table 2.
Tasks	Max Correlation	Min Correlation	Projection Error
hopper-medium-v0	0.569	-0.568	0.016
hopper-medium-expert-v0	0.498	-0.540	0.015
hopper-expert-v0	0.423	-0.415	0.011
halfcheetah-medium-v0	0.569	-0.568	0.016
halfcheetah-medium-expert-v0	0.605	-0.589	0.047
halfcheetah-expert-v0	0.370	-0.461	0.021
walker2d-medium-v0	0.475	-0.582	0.046
walker2d-medium-expert-v0	0.495	-0.472	0.042
walker2d-expert-v0	0.358	-0.503	0.019
Entropy of return distribution. To numerically show that UBER can generate a diverse behavior set beyond the original dataset, we calculate the entropy of the return distribution for each task, as shown in Table 3.
Table 2: Minimum and maximum correlation and linear projection error for the true reward function using random reward functions on various tasks. The projection error  $\epsilon$  is defined as  $\epsilon = \| r - \widehat{r} \| / \| r \|$ , where  $\widehat{r}$  is the best approximation using linear combination of random rewards. Note that we use random reward functions (represented by neural networks) based on  $(s, a)$  inputs rather than entirely random ones, which have a near-zero correlation with the true reward and  $40\%$  projection error.
Methods	halfcheetah-m	halfcheetah-me	halfcheetah-e	hopper-m	hopper-me	hopper-e
BC	0	0	0	0	0	0
DATASET	1.21	1.47	0.86	0.71	1.97	0.82
UBER	2.01	1.92	2.69	2.45	2.32	1.11
Methods	walker2d-m	walker2d-me	walker2d-e	antmaze-md	antmaze-mr
BC	0	0	0	0	0
DATASET	1.44	1.88	2.43	0.63	0.29
UBER	2.67	2.61	0.41	1.77	1.69
Table 3: Entropy of the return distribution for different methods. Our method generate a behavior set that has a higher entropy in the return distribution than the original dataset consistently across various tasks.
Distribution of behaviors. To show that UBER can generate diverse behaviors, we plot the distribution of behaviors generated by UBER, as well as the original distribution from the dataset. The experimental results are summarized in Figure 10. We can see that when there are expert behaviors, most of the behaviors generated by UBER successfully lock to the optimal policy. When there are multiple modes of behavior like medium-expert, UBER can learn both behaviors, matching the distribution of the dataset. When there are no expert behaviors, UBER can learn diverse behaviors, some of which can achieve near-optimal performance. We hypothesize that diversity is one of the keys that UBER can outperform behavior cloning-based methods.
Learning curve for the Antmaze environment. Here we provide the detailed learning curve for the Antmaze experiment to demonstrate the efficiency of UBER. We can see from Figure 11 that UBER learns a good strategy in hard exploration tasks like medium and large, indicating UBER's strong ability to learn useful behaviors.
Ablation study for the online policy reuse module. We first replace the policy expansion module with CUP (Zhang et al., 2022), UBER (CUP) to investigate the effect of various online policy reuse modules. The experimental results in Figure 12 show that UBER+PEX generally outperforms UBER+CUP. We hypothesize that CUP includes a hard reuse method, which requires the optimized policy to be close to the reused policies during training, while PEX is a soft method that only uses the pre-trained behaviors to collect data. This makes PEX more suitable to our setting since UBER may generate undesired behaviors.
Complete comparison with baselines. Here we provide a complete comparison with several baselines, including OPAL, PARROT and UDS, as shown in Figure 13. Our method outperforms these baselines due to the ability to extract diverse behaviors.
Ablation study for the random intent priors module. we conduct an ablation study for the random intent priors module. We use the average reward in the offline datasets to learn offline policies and then follow the same online policy reuse process in Algorithm 2. Using average reward serves as a strong baseline since it may generate near-optimal performance as observed in Shin et al. (2023). We name the average reward-based offline behavior extraction as AVG-PEX. The experimental results in Figure 14 show that while average reward-based offline optimization can extract some useful behaviors and accelerate the downstream tasks, using random intent priors performs better due to the diversity of the behavior set.
Figure 9: Complete experimental results on the meta-world based on two types of offline datasets, expert and replay. All experiment results were conducted over five random seeds.
Figure 10: Distribution of random intent priors, datasets, and behavior cloning policies. We restrict the y-axis to a small range to make the distribution clearer.
Figure 11: Comparison between UBER and baselines in the online phase in the Antmaze domain. Oracle is the performance of RLPD at  $500\mathrm{k}$  step. We adopt a normalized score metric averaged with three random seeds.
Figure 12: Complete result for the ablation study for the online policy reuse module. Using policy expansion (PEX) for downstream tasks is better than critic-guided policy reuse (CUP) in general
Figure 13: Complete result for the comparison between UBER and baselines including unsupervised behavior extraction and data-sharing method. We adopt a normalized score metric averaged with five random seeds.
Figure 14: Complete result for the ablation study for the random intent priors module. Using average reward for policies learning can lead to nearly optimal performances on the original task. Nevertheless, our method outperforms such a baseline (AVG-PEX) consistently, showing the importance of behavioral diversity.
Experimental Setting. For BC-PEX, we first extract behavior from the offline dataset based on the Behavior Cloning, then conduct online policy expansion based on Algorithm 2. For Avg-PEX, we first use the average reward for the offline optimization, then conduct online policy expansion based on Algorithm 2. For RLPD, we load the offline dataset with the true reward value for the prior data. For RLPD (no prior), we do not load offline-dataset into the buffer.
Hyper-parameters. For the Mujoco and metaworld tasks, we adopt the TD3+BC and TD3 as the backbone of offline and online algorithms. For the Antmaze tasks, we adopt the IQL and RLPD as the backbone of offline and online algorithms. We outline the hyper-parameters used by UBER in Table 4, Table 5 and Table 6.