Unsupervised-to-Online Reinforcement Learning

## Abstract

Abstract Offline-to-online reinforcement learning (RL),
a framework that trains a policy with offline RL
and then further fine-tunes it with online RL,
has been considered a promising recipe for data-driven decision-making.
While sensible, this framework has drawbacks: it requires domain-specific offline RL pre-training for each task,
and is often brittle in practice.
In this work, we propose unsupervised-to-online RL ( U2O RL ),
which replaces domain-specific supervised offline RL with unsupervised offline RL,
as a better alternative to offline-to-online RL.
U2O RL not only enables reusing a single pre-trained model for multiple downstream tasks,
but also learns better representations, which often result in even better performance and stability
than supervised offline-to-online RL.
To instantiate U2O RL in practice, we propose a general recipe for U2O RL
to bridge task-agnostic unsupervised offline skill-based policy pre-training and supervised online fine-tuning.
Throughout our experiments in nine state-based and pixel-based environments,
we empirically demonstrate that U2O RL achieves strong performance
that matches or even outperforms previous offline-to-online RL approaches,
while being able to reuse a single pre-trained model for a number of different downstream tasks.

### 1 Introduction

Across natural language processing (NLP), computer vision (CV), and speech processing,
ubiquitous in the recent successes of machine learning is
the idea of adapting an expressive model pre-trained on large-scale data to domain-specific tasks via fine-tuning.
In the domain of reinforcement learning (RL),
offline-to-online RL has been considered an example of such a recipe for leveraging offline data for efficient online fine-tuning.
Offline-to-online RL first trains a task-specific policy on a previously collected dataset with offline RL,
and then continues training the policy with additional environment interactions to further improve performance.

But, is offline-to-online RL really the most effective way to leverage offline data for online RL?
Indeed, offline-to-online RL has several limitations.
First, it pre-trains a policy with a *domain-specific task reward*,
which precludes sharing a single pre-trained model for multiple downstream tasks.
This is in contrast to predominant pre-training recipes in large language models or visual representation learning,
where they pre-train large models with self-supervised or *unsupervised* objectives
to learn useful representations, which can facilitate learning a wide array of downstream tasks.
Second, naïve offline-to-online RL is often brittle in practice [, ].
This is mainly because pre-trained offline RL agents suffer the distributional shift
between the offline and online interaction data [, ]
or experience feature collapse (Section 5.5),
which necessitates specialized, potentially complicated techniques.

In this work, our central hypothesis is that *unsupervised pre-training of diverse policies* from offline data
can serve as an effective data-driven recipe for *online* RL,
and can be more effective than even domain-specific (“supervised”) offline pre-training.
We call this framework unsupervised-to-online RL (U2O RL).
U2O RL has two appealing properties.
First, unlike offline-to-online RL, a single pre-trained model can be fine-tuned for different downstream tasks.
Since offline unsupervised RL does not require task information,
we can pre-train diverse policies on unlabeled data before knowing downstream tasks.
Second, by pre-training multi-task policies with diverse intrinsic rewards,
the agent extracts rich representations from data,
which often helps achieve *even better* final performance and stability than *supervised* offline-to-online RL.
This resembles how general-purpose unsupervised pre-training in other domains,
such as with LLMs or self-supervised representations [, , , , , ],
improves over the performance of domain-specific specialist pre-training.

Our U2O RL framework consists of three stages:
unsupervised offline policy pre-training, bridging, and online fine-tuning (Figure 1).
In the first unsupervised offline pre-training phase,
we employ a skill-based offline unsupervised RL or offline goal-conditioned RL method, which
trains diverse behaviors (or *skills*) with intrinsic rewards
and provides an efficient mechanism to identify the best skill for a given task reward.
In the subsequent bridging and fine-tuning phases,
we adapt the best skill among the learned policies to the given downstream task reward with online RL.
Here, to prevent a potential mismatch between the intrinsic and task rewards,
we propose a simple yet effective reward scale matching technique that bridges the gap between the two training schemes
and thus improves performance and stability.

Our main contributions in this work are twofold.
First, to the best of our knowledge, this is the first work that makes
the (potentially surprising) observation that it is often better to replace supervised offline RL with unsupervised offline RL in the offline-to-online RL setting.
We also identify the reason behind this phenomenon:
this is mainly because offline unsupervised pre-training learns better *representations* than task-specific supervised offline RL.
Second, we propose a general recipe to bridge skill-based unsupervised offline RL pre-training and online RL fine-tuning.
Through our experiments on nine state-based and pixel-based environments,
we demonstrate that U2O RL
often outperforms standard offline-to-online RL
both in terms of sample efficiency and final performance,
while being able to reuse a single pre-trained model for multiple downstream tasks.

Figure: Figure 1: Illustration of U2O RL. In this work, we propose to replace supervised offline RL with *unsupervised offline RL* in the offline-to-online RL framework. We call this scheme unsupervised-to-online RL (U2O RL). U2O RL consists of three stages: (1) unsupervised offline RL pre-training, (2) bridging, and (3) online RL fine-tuning. In unsupervised offline RL pre-training, we train a multi-task skill policy $\pi_{\theta}(a\mid s,z)$ instead of a single-task policy $\pi_{\theta}(a\mid s)$. Then, we convert the multi-task policy into a task-specific policy in the bridging phase. Finally, we fine-tune the skill policy with online environment interactions.
Refer to caption: x1.png

### 2 Related work

Online RL from prior data.
Prior works have proposed several ways to leverage a previously collected offline dataset to accelerate online RL training.
They can be categorized into two main groups: offline-to-online RL and off-policy online RL with offline data.
Offline-to-online RL first pre-trains a policy and a value function with offline RL [, , , , , , , , , , , , , ],
and then continues to fine-tune them with additional online interactions [, , , , , , , ].
Since naïve offline-to-online RL is often unstable in practice
due to the distributional shift between the dataset and online interactions,
prior works have proposed several techniques,
such as balanced sampling [], actor-critic alignment [],
adaptive conservatism [], and return lower-bounding [].
In this work, unlike offline-to-online RL, which trains a policy with the target task reward,
we offline pre-train a multi-task policy with unsupervised (intrinsic) reward functions.
This makes our single pre-trained policy reusable for any downstream task and learn richer representations.
The other line of research, off-policy online RL,
trains an online RL agent from scratch on top of a replay buffer filled with offline data,
without any pre-training [, , , ].
While this simple approach often leads to improved stability and performance [],
it does not leverage the benefits of pre-training;
in contrast, we do leverage pre-training by learning useful features via offline unsupervised RL,
which we show leads to better fine-tuning performance in our experiments.

Unsupervised RL.
The goal of unsupervised RL is to leverage unsupervised pre-training to facilitate downstream reinforcement learning.
Prior works have mainly focused on unsupervised representation learning and unsupervised behavior learning.
Unsupervised representation learning methods [, , , , , , , , , ] aim to extract useful (visual) representations from data.
These representations are then fed into the policy to accelerate task learning.
In this work, we focus on unsupervised behavior learning,
which aims to pre-train policies that can be directly adapted to downstream tasks.
Among unsupervised behavior learning methods,
online unsupervised RL pre-trains useful policies
by either maximizing state coverage [, , , ]
or learning distinct skills  [, , , ]
via reward-free interactions with the environment.
In this work, we consider *offline* unsupervised RL,
which does not allow
any environment interactions during the pre-training stage.

Offline unsupervised RL.
Offline unsupervised RL methods focus on learning diverse policies (*i.e*., skills) from the dataset,
rather than exploration, as online interactions are not permitted in this problem setting.
There exist three main approaches to offline unsupervised RL.
Behavioral cloning-based approaches extract skills from an offline dataset by training a generative model
(*e.g*., variational autoencoders [], Transformers [], etc.) [, , ].
Offline goal-conditioned RL methods learn diverse goal-reaching behaviors with goal-conditioned reward functions [, , , , , , , ].
Offline unsupervised skill learning approaches learn diverse skills based on intrinsically defined reward functions [, , ].
Among these approaches, we use methods in the second and third categories (*i.e*., goal- or skill-based unsupervised offline RL) as part of our framework.

Our goal in this work is to study how unsupervised offline RL,
as opposed to supervised task-specific offline RL, can be employed to facilitate online RL fine-tuning.
While somewhat similar unsupervised pre-training schemes have been explored in prior works,
they either consider hierarchical RL (or zero-shot RL) with frozen learned skills without fine-tuning [, , , , ],
assume online-only RL [],
or are limited to the specific setting of goal-conditioned RL [, ].
To the best of our knowledge, this is the first work that considers the *fine-tuning* of skill policies
pre-trained with unsupervised offline RL *in the context of offline-to-online RL*.
Through our experiments, we show that our fine-tuning framework leads to significantly better performance than
previous approaches based on hierarchical RL, zero-shot RL, and standard offline-to-online RL.

### 3 Preliminaries

We formulate a decision making problem as a Markov decision process (MDP) [], which is defined by a tuple of $(\mathcal{S},\mathcal{A},P,r,\rho,\gamma)$,
where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space,
$P:{\mathcal{S}}\times{\mathcal{A}}\to\Delta({\mathcal{S}})$ is the transition dynamics,
$r:{\mathcal{S}}\times{\mathcal{A}}\times{\mathcal{S}}\to{\mathbb{R}}$ is the task reward function,
$\rho\in\Delta({\mathcal{S}})$ is the initial state distribution,
and $\gamma\in(0,1)$ is the discount factor.
Our aim is to learn a policy $\pi:\mathcal{S}\to\Delta(\mathcal{A})$ that maximizes the expectation of cumulative task rewards, $\mathbb{E}_{\pi}[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t},s_{t+1})]$.

Offline RL and implicit Q-learning (IQL).
The goal of offline RL is to learn a policy solely from an offline dataset $\mathcal{D}_{\mathtt{off}}$,
which consists of transition tuples $(s,a,s^{\prime},r)$.
One straightforward approach to offline RL is to simply employ an off-policy RL algorithm (*e.g*. TD3 []).
For instance, we can minimize the following temporal difference (TD) loss to learn an action-value function (Q-function) from data:

$$ $\displaystyle\mathcal{L}_{\mathtt{TD}}(\phi)=\mathbb{E}_{(s,a,s^{\prime},r) \sim\mathcal{D}_{\mathtt{off}}}\left[(r+\gamma\max_{a^{\prime}}Q_{\bar{\phi}}( s^{\prime},a^{\prime})-Q_{\phi}(s,a))^{2}\right],$ (1) $$

where $Q_{\phi}$ denotes the parameterized action-value function, and $Q_{\bar{\phi}}$ represents the target action-value function [], whose parameter $\bar{\phi}$ is updated via Polyak averaging [] using $\phi$.
We can then train a policy $\pi$ to maximize $\mathbb{E}_{a\sim\pi}\left[Q_{\phi}(s,a)\right]$.

While this simple off-policy TD learning can be enough when the dataset has sufficiently large state-action coverage,
offline datasets in practice often have limited coverage, which makes the agent susceptible to value overestimation and exploitation, as the agent cannot get corrective feedback from the environment [].
To address this issue, Kostrikov et al. [] proposed implicit Q-learning (IQL),
which fits an optimal action-value function without querying out-of-distribution actions: IQL replaces the $\operatorname*{arg\,max}$ operator,
which potentially allows the agent to exploit Q-values from out-of-distribution actions,
with an expectile loss that implicitly approximates the maximum value.
Specifically, IQL minimizes the following losses:

$$ $\displaystyle\mathcal{L}^{Q}_{\mathtt{IQL}}(\phi)$ $\displaystyle=\mathbb{E}_{(s,a,s^{\prime},r)\sim\mathcal{D}_{\mathtt{off}}} \left[(r+\gamma V_{\psi}(s^{\prime})-Q_{\phi}(s,a))^{2}\right],$ (2) $\displaystyle\mathcal{L}^{V}_{\mathtt{IQL}}(\psi)$ $\displaystyle=\mathbb{E}_{(s,a)\sim\mathcal{D}_{\mathtt{off}}}\left[\ell_{\tau }^{2}(Q_{\bar{\phi}}(s,a)-V_{\psi}(s))\right],$ (3) $$

where
$Q_{\phi}$ and $Q_{\bar{\phi}}$ respectively denote the action-value and target action-value functions,
$V_{\psi}$ denotes the value function,
$\ell_{\tau}^{2}(x)=|\tau-\mathds{1}(x<0)|x^{2}$ denotes the expectile loss []
and $\tau$ denotes the expectile parameter.
Intuitively, the asymmetric expectile loss in Equation 3
makes $V_{\psi}$ implicitly approximate $\max_{a}Q_{\bar{\phi}}(s,a)$
by penalizing positive errors more than negative errors.

Hilbert foundation policy (HILP).
Our unsupervised-to-online framework requires an offline unsupervised RL algorithm that
trains a skill policy $\pi_{\theta}(a\mid s,z)$ from an unlabeled dataset,
and we mainly use HILP [] in our experiments.
HILP consists of two phases.
In the first phase, HILP trains a feature network $\xi:{\mathcal{S}}\to{\mathcal{Z}}$
that embeds temporal distances (*i.e*., shortest path lengths) between states
into the latent space by enforcing the following equality:

$$ $\displaystyle d^{*}(s,g)=\|\xi(s)-\xi(g)\|_{2}$ (4) $$

for all $s,g\in{\mathcal{S}}$,
where $d^{*}(s,g)$ denotes the temporal distance (the minimum number of steps required to reach $g$ from $s$)
between $s$ and $g$.
In practice,
given the equivalence between goal-conditioned values and temporal distances,
$\xi$ can be trained with any offline goal-conditioned RL algorithm []
(see Park et al. [] for further details).
After training $\xi$, HILP trains a skill policy $\pi_{\theta}(a\mid s,z)$
with the following intrinsic reward
using an off-the-shelf offline RL algorithm [, ]:

$$ $\displaystyle r^{\mathtt{int}}(s,a,s^{\prime},z)=(\xi(s^{\prime})-\xi(s))^{ \top}z,$ (5) $$

where $z$ is sampled from the unit ball, $\{z\in\mathcal{Z}:\|z\|=1\}$.
Intuitively,
Equation 5 encourages the agent to learn
behaviors that move in every possible latent direction,
resulting in diverse state-spanning skills [].
Note that Equation 5 can be interpreted as
the inner product between the task vector $z$ and the feature vector $f(s,a,s^{\prime}):=\xi(s^{\prime})-\xi(s)$
in the successor feature framework [, ].

### 4 Unsupervised-to-online RL (U2O RL)

Our main hypothesis in this work is that
task-agnostic offline RL pre-training of *unsupervised* skills can be more effective
than task-specific, supervised offline RL for online RL fine-tuning.
We call this framework
unsupervised-to-online RL (U2O RL).
In this section,
we first describe the three stages of our U2O RL framework (Figure 1):
unsupervised offline policy pre-training (Section 4.1),
bridging (Section 4.2),
and online fine-tuning (Section 4.3).
We then explain why *unsupervised*-to-online RL is potentially better than standard *supervised*
offline-to-online RL (Section 4.4).

#### 4.1 Unsupervised offline policy pre-training

In the first unsupervised offline policy pre-training phase (Figure 1 (bottom left)),
we train diverse policies (or *skills*) with intrinsic rewards
to extract a variety of useful behaviors as well as rich features
from the offline dataset ${\mathcal{D}}_{\mathtt{off}}$.
In other words,
instead of training a single-task policy $\pi_{\theta}(a\mid s)$ with task rewards $r(s,a,s^{\prime})$
as in standard offline-to-online RL,
we train a *multi-task* skill policy $\pi_{\theta}(a\mid s,z)$ with
a family of unsupervised, *intrinsic* rewards $r^{\mathtt{int}}(s,a,s^{\prime},z)$,
where $z$ is a skill latent vector sampled from a latent space ${\mathcal{Z}}={\mathbb{R}}^{d}$.
Even if ${\mathcal{D}}_{\mathtt{off}}$ contains reward labels, we do not use any reward information in this phase.

Among existing unsupervised offline policy pre-training methods (Section 2),
we opt to employ successor feature-based methods [, , , , ] or offline goal-conditioned RL methods [, ]
for our unsupervised pre-training,
since they provide a convenient mechanism to identify the best skill latent vector given a downstream task,
which we will utilize in the next phase.
More concretely, we mainly choose to employ HILP [] (Section 3) as an unsupervised offline policy pre-training method in our experiments
for its state-of-the-art performance in previous benchmarks [].
We note, however, that any other unsupervised offline successor feature-based skill learning methods []
or offline goal-conditioned RL methods []
can also be used in place of HILP (see Appendix A.1).

Figure: Algorithm 1 U2O RL: Unsupervised-to-Online Reinforcement Learning

#### 4.2 Bridging offline unsupervised RL and online supervised RL

After finishing unsupervised offline policy pre-training,
our next step is to convert the learned multi-task skill policy into a task-specific policy
that can be fine-tuned to maximize a given downstream reward function $r$ (Figure 1 (bottom middle)).
There exist two challenges in this step:
(1) we need a mechanism to identify the skill vector $z$ that best solves the given task
and (2) we need to reconcile the gap between intrinsic rewards and downstream task rewards
for seamless online fine-tuning.

Skill identification.
Since we chose to use a successor feature- or goal-based unsupervised pre-training method in the previous phase,
the first challenge is relatively straightforward.
For goal-oriented tasks (*e.g*., AntMaze [] and Kitchen []), we assume the task goal $g$ to be available,
and we either directly use $g$ (for goal-conditioned methods) or infer the skill $z^{*}$ that corresponds to $g$
based on a predefined conversion formula (for successor feature-based methods that support such a conversion [, ]).
For general reward-maximization tasks, we employ successor feature-based unsupervised pre-training methods,
and use the following linear regression to find the skill latent vector $z^{*}$ that best approximates
the downstream task reward function $r:{\mathcal{S}}\times{\mathcal{A}}\times{\mathcal{S}}\to{\mathbb{R}}$:

$$ $\displaystyle z^{*}=\operatorname*{arg\,min}_{z\in\mathcal{Z}}\mathbb{E}_{(s,a ,s^{\prime})\sim\mathcal{D}_{\mathtt{reward}}}\left[\left(r(s,a,s^{\prime})-f( s,a,s^{\prime})^{\top}z\right)^{2}\right],$ (6) $$

where $f$ is the feature network in the successor feature framework (Section 3)
and $\mathcal{D}_{\texttt{reward}}$ is a reward-labeled dataset.
This reward-labeled dataset $\mathcal{D}_{\texttt{reward}}$ can be
either the full offline dataset $\mathcal{D}_{\texttt{off}}$ (if it is fully reward-labeled),
a subset of the offline dataset (if it is partially reward-labeled),
or a newly collected dataset with additional environment interactions.
In our experiments, we mainly use a small number (*e.g*., $0.2\%$ for DMC tasks) of reward-labeled samples from the offline dataset for $\mathcal{D}_{\texttt{reward}}$,
following previous works [, ],
but we do not require $\mathcal{D}_{\texttt{reward}}$ to be a subset of $\mathcal{D}_{\texttt{off}}$
(see Appendix A.4).

Reward scale matching.
After identifying the best skill latent vector $z^{*}$,
our next step is to bridge the gap between intrinsic and extrinsic rewards.
Since these two reward functions can have very different scales,
naïve online adaptation can lead to abrupt shifts in target Q-values,
potentially causing significant performance drops in the early stages of online fine-tuning.
While one can employ sophisticated reward-shaping techniques to deal with this issue [, ],
in this work, we propose a simple yet effective reward scale matching technique
that we find to be effective enough in practice.
Specifically, we compute the running mean and standard deviation of intrinsic rewards during the pre-training phase,
and normalize the intrinsic rewards with the calculated statistics.
Similarly, during the fine-tuning phase, we compute the statistics of task rewards and normalize the task rewards
so that they have the same scale and mean as normalized intrinsic rewards.
This way, we can prevent abrupt shifts in reward scales
without altering the optimal policy for the downstream task.
In our experiments, we find that this simple technique is crucial for achieving good performance,
especially in environments with dense rewards (Section 5.7).

#### 4.3 Online fine-tuning

Our final step is to fine-tune the skill policy with online environment interactions (Figure 1 (bottom right)).
This step is straightforward:
since we have found $z^{*}$ in the previous stage,
we can simply *fix* the skill vector $z^{*}$ in the policy $\pi_{\theta}(a\mid s,z^{*})$
and the Q-function $Q_{\phi}(s,a,z^{*})$,
and fine-tune them with the same (offline) RL algorithm used in the first phase
(*e.g*., IQL [], TD3 [])
with additional online interaction data.
While one can employ existing specialized techniques for offline-to-online RL for better online adaptation in this phase,
we find in our experiments that, thanks to rich representations learned by unsupervised pre-training,
simply using the same (offline) RL algorithm is enough to achieve strong performance
that matches or even outperforms state-of-the-art offline-to-online RL techniques.

#### 4.4 Why is U2O RL potentially better than offline-to-online RL?

Our main claim is that *unsupervised* offline RL is better than supervised offline RL for online fine-tuning.
However, this might sound very counterintuitive.
Especially, if we know the downstream task ahead of time,
how can unsupervised offline RL potentially lead to better performance than supervised offline RL,
despite the fact that the former does *not* use any task information during the offline phase?

We hypothesize that this is because unsupervised multi-task offline RL enables better *feature learning*
than supervised single-task offline RL.
By training the agent on a number of diverse intrinsically defined tasks,
it gets to acquire rich knowledge about the environment, dynamics, and potential tasks in the form of representations,
which helps improve and facilitate the ensuing task-specific online fine-tuning.
This resembles the recent observation in machine learning that
large-scale unsupervised pre-training improves downstream task performances over task-specific supervised pre-training [, , , , , ].
In our experiments, we empirically show that U2O RL indeed learns better features than its supervised counterpart (Section 5.5).

Another benefit of U2O RL is that it does not use any task-specific information during pre-training.
This is appealing because we can reuse a single pre-trained policy for a number of different downstream tasks.
Moreover,
it enables leveraging potentially very large, task-agnostic offline data during pre-training,
which is often much cheaper to collect than task-specific, curated datasets [].

### 5 Experiments

In our experiments, we evaluate the performance of U2O RL in the context of offline-to-online RL.
We aim to answer the following research questions:
(1) Is U2O RL better than previous offline-to-online RL strategies?
(2) What makes unsupervised offline RL result in better fine-tuning performance than supervised offline RL?
(3) Which components of U2O RL are important?

Figure: Figure 2: Environments. We evaluate U2O RL on nine state-based or pixel-based environments.
Refer to caption: x2.png

Figure: Figure 3: Online fine-tuning plots of U2O RL and previous offline-to-online RL frameworks (8 seeds). Across the benchmarks, our U2O RL mostly shows consistently better performance than standard offline-to-online RL and off-policy online RL with offline data.
Refer to caption: x3.png

#### 5.1 Experimental setup

Environments and offline datasets.
In our experiments, we consider nine tasks across five benchmarks (Figure 2).
ExORL [] is a benchmark suite that consists of offline datasets collected by exploratory policies (e.g., RND []) on the DeepMind Control Suite []. We consider four embodiments (Walker, Cheetah, Quadruped, and Jaco), each of which has four tasks.
AntMaze [, ] is a navigation task,
whose goal is to control an $8$-DoF quadruped agent to reach a target position. We consider the two most challenging mazes with the largest sizes, large and ultra, and two types of offline datasets, diverse and play.
Kitchen [, ] is a robotic manipulation task, where the goal is to control a $9$-DoF Franka robot arm to achieve four subtasks sequentially. The environment has four subtasks in total. We consider two types of offline datasets from the D4RL suite [], partial and mixed.
Visual Kitchen [, , ] is a pixel-based variant of the Kitchen environment,
where an agent must achieve four subtasks purely from $64\times 64\times 3$ pixel observations instead of low-dimensional state information.
Adroit [] is a dexterous manipulation benchmark,
where the goal is to control a $24$-DoF robot hand to twirl a pen or open a door.
We provide further details in Appendix B.1

Implementation.
In our experiments, we mainly employ HILP []
as the unsupervised offline policy pre-training algorithm in U2O RL.
For the offline RL backbone, we use TD3 [] for ExORL
and IQL [] for others
following previous works [, ].
Since both IQL and TD3+BC [, ]
have been known to achieve strong performance in the offline-to-online RL setting [],
we use them for the online fine-tuning phase in U2O RL as well.
For sparse-reward tasks (AntMaze, Kitchen, and Adroit), we do not apply reward scale matching.
For AntMaze, Kitchen, and Adroit, we report normalized scores, following Fu et al. [].
In our experiments, we use $\mathbf{8}$ random seeds and report standard deviations with shaded areas, unless otherwise stated.
We refer the reader to Appendix B for the full implementation details and hyperparameters.

#### 5.2 Is U2O RL better than previous offline-to-online RL frameworks?

We begin our experiments by comparing our approach, unsupervised-to-online RL, with two previous offline-to-online RL *frameworks* (Section 2):
offline-to-online RL (O2O RL) and off-policy online RL with offline data (Online w/ Off Data).
To recall, offline-to-online RL [, , , , ]
first pre-trains a policy with *supervised* offline RL using the task reward,
and then continues training it with online rollouts.
Off-policy online RL [, , ] trains a policy from scratch on top of a replay buffer filled with offline data.
Here, we use the *same* offline RL backbone (*i.e*., TD3 for ExORL and IQL for AntMaze, Kitchen, and Adroit)
to ensure apples-to-apples comparisons between the three frameworks.
We will compare U2O RL with previous specialized offline-to-online RL techniques in Section 5.3.

Figure 3 shows the online fine-tuning curves on $14$ different tasks.
The results suggest that U2O RL generally leads to better performance than both offline-to-online RL and off-policy online RL across the environments, despite not using any task information during pre-training.
Notably, U2O RL significantly outperforms these two previous frameworks
in the most challenging AntMaze tasks (antmaze-ultra-{diverse, play}).

#### 5.3 How does U2O RL compare to previous specialized offline-to-online RL techniques?

Next, we compare U2O RL with $13$ previous specialized offline-to-online RL methods,
including CQL [], IQL [], AWAC [], O3F [], ODT [], CQL+SAC [, ], Hybrid RL [], SAC+od (offline data) [, ], SAC [], IQL+od (offline data) [, ], RLPD [], FamO2O [] and Cal-QL [].
We show the comparison results in Table 1,
where we take the scores from Nakamoto et al. [], Wang et al. [] for the tasks that are common to ours.
Since Cal-QL achieves the best performance in the table,
we additionally make a comparison with Cal-QL on antmaze-ultra-{diverse, play} as well,
by running their official implementation with tuned hyperparameters.

Table 1 shows that U2O RL achieves strong performance that matches or sometimes even outperforms
previous offline-to-online RL methods,
even though U2O RL does *not* use any task information during offline pre-training
nor any specialized offline-to-online techniques.
In particular, in the most challenging antmaze-ultra tasks,
U2O RL outperforms the previous best method (Cal-QL) by a significant margin.
This is very promising because,
even if U2O RL does not necessarily outperform the state-of-the-art methods on every single task
(though it is at least on par with the previous best methods),
U2O RL enables reusing a single unsupervised pre-trained policy for multiple downstream tasks,
unlike previous offline-to-online RL methods that perform *task-specific* pretraining.

**Table 1: Comparison between U2O RL and previous offline-to-online RL methods. We denote how performances change before and after online fine-tuning with arrows. Baseline scores except RLPD [] are taken from Nakamoto et al. [], Wang et al. []. Scores within the $5\%$ of the best score are highlighted in bold, as in Kostrikov et al. []. We use 8 random seeds for each task for U2O RL.**
| Task | antmaze-ultra-diverse | antmaze-ultra-play | antmaze-large-diverse | antmaze-large-play | kitchen-partial | kitchen-mixed |
| --- | --- | --- | --- | --- | --- | --- |
| CQL | - | - | 25 $\rightarrow$ 87 | 34 $\rightarrow$ 76 | 71 $\rightarrow$ 75 | 56 $\rightarrow$ 50 |
| IQL | 13 $\rightarrow$ 29 | 17 $\rightarrow$ 29 | 40 $\rightarrow$ 59 | 41 $\rightarrow$ 51 | 40 $\rightarrow$ 60 | 48 $\rightarrow$ 48 |
| AWAC | - | - | 00 $\rightarrow$ 00 | 00 $\rightarrow$ 00 | 01 $\rightarrow$ 13 | 02 $\rightarrow$ 12 |
| O3F | - | - | 59 $\rightarrow$ 28 | 68 $\rightarrow$ 01 | 11 $\rightarrow$ 22 | 06 $\rightarrow$ 33 |
| ODT | - | - | 00 $\rightarrow$ 01 | 00 $\rightarrow$ 00 | - | - |
| CQL+SAC | - | - | 36 $\rightarrow$ 00 | 21 $\rightarrow$ 00 | 71 $\rightarrow$ 00 | 59 $\rightarrow$ 01 |
| Hybrid RL | - | - | $\rightarrow$ 00 | $\rightarrow$ 00 | $\rightarrow$ 00 | $\rightarrow$ 01 |
| SAC+od | - | - | $\rightarrow$ 00 | $\rightarrow$ 00 | $\rightarrow$ 07 | $\rightarrow$ 00 |
| SAC | - | - | $\rightarrow$ 00 | $\rightarrow$ 00 | $\rightarrow$ 03 | $\rightarrow$ 02 |
| IQL+od | $\rightarrow$ 04 | $\rightarrow$ 05 | $\rightarrow$ 71 | $\rightarrow$ 56 | $\rightarrow$ 74 | $\rightarrow$ 61 |
| FamO2O | - | - | $\rightarrow$ 64 | $\rightarrow$ 61 | - | - |
| RLPD | 00 $\rightarrow$ 00 | 00 $\rightarrow$ 00 | 00 $\rightarrow$ 94 | 00 $\rightarrow$ 81 | - | - |
| Cal-QL | 05 $\rightarrow$ 05 | 15 $\rightarrow$ 13 | 33 $\rightarrow$ 95 | 26 $\rightarrow$ 90 | 67 $\rightarrow$ 79 | 38 $\rightarrow$ 80 |
| U2O (Ours) | 22 $\rightarrow$ 54 | 17 $\rightarrow$ 58 | 11 $\rightarrow$ 95 | 14 $\rightarrow$ 88 | 48 $\rightarrow$ 75 | 48 $\rightarrow$ 74 |

Figure: Figure 4: Learning curves during online RL fine-tuning (8 seeds). A single pre-trained model from U2O can be fine-tuned to solve multiple downstream tasks. Across the embodiments and tasks, our U2O RL matches or outperforms standard offline-to-online RL and off-policy online RL with offline data even though U2O RL uses a single task-agnostic pre-trained model.
Refer to caption: x18.png

#### 5.4 Can a single pre-trained model from U2O be fine-tuned to solve multiple tasks?

One important advantage of U2O RL is that it can reuse a single task-agnostic dataset for multiple different downstream tasks,
unlike standard offline-to-online RL.
To demonstrate this, we train U2O RL with four different tasks from the same task-agnostic ExORL dataset on each DMC environment,
and report the full training curves in Figure 4.
The results show that, for example, a single pre-trained model on the Walker domain can be fine-tuned for all four tasks (Walker Run, Walker Flip, Walker Stand, and Walker Walk).
Note that even though U2O RL uses a single task-agnostic pre-trained model, the performance of U2O RL matches or even outperforms O2O RL, which pre-trains a model with task-specific rewards.

#### 5.5 Why does U2O RL often outperform supervised offline-to-online RL?

Figure: Figure 5: Feature dot products during offline RL pre-training (lower is better, 8 seeds). The plots show that *unsupervised* offline pre-training effectively prevents feature collapse (co-adaptation), yielding better representations than supervised offline pre-training.
Refer to caption: x35.png

In the above experiments, we showed that our unsupervised-to-online RL framework
often even outperforms previous supervised offline-to-online RL methods.
We hypothesized in Section 4.4 that this is because unsupervised offline pre-training yields
better *representations* that facilitate online task adaptation.
To empirically verify this hypothesis,
we measure the quality of the value function representations using the method proposed by Kumar et al. [].
Specifically, we define the value features $\zeta_{\phi}(s,a)$
as the penultimate layer of the value function $Q_{\phi}$,
*i.e*., $Q_{\phi}(s,a)=w_{\phi}^{\top}\zeta_{\phi}(s,a)$,
and measure the dot product between consecutive state-action pairs, $\zeta_{\phi}(s,a)^{\top}\zeta_{\phi}(s^{\prime},a^{\prime})$ [].
Intuitively, this dot product represents the degree to which these two representations are “collapsed” (or “co-adapted”),
which is known to be correlated with poor performance [] (*i.e*., the lower the better).

Figure 5 compares the dot product metrics of
unsupervised offline RL (in U2O RL) and supervised offline RL (in O2O RL) on four benchmark tasks.
The results suggest that our unsupervised multi-task pre-training effectively prevents feature co-adaptation
and thus indeed yields better representations across the environments.
This highlights the benefits of unsupervised offline pre-training,
and (partially) explains the strong online fine-tuning performance of U2O RL.
We additionally provide further analyses with different offline unsupervised RL algorithms
(*e.g*., graph Laplacian-based successor feature learning [, ]) in Appendix A.1.

#### 5.6 Is fine-tuning better than other alternative strategies ( e.g ., hierarchical RL)?

In this work, we focus on the *fine-tuning* of offline pre-trained skill policies,
but this is not the only way to leverage pre-trained skills for downstream tasks.
To see how our fine-tuning scheme compares to other alternative strategies,
we compare U2O RL with three previously considered approaches:
hierarchical RL (HRL, *e.g*., OPAL [], SPiRL []) [, , , , ],
zero-shot RL [, ],
and PEX [].
HRL additionally trains a high-level policy that combines fixed pre-trained skills in a sequential manner.
Zero-shot RL simply finds the skill policy that best solves the downstream task, with no fine-tuning or hierarchies.
PEX combines fixed pre-trained multi-task policies and a newly initialized policy with a multiplexer that chooses the best policy.

Figure: Figure 6: Fine-tuning is better than previous strategies, such as hierarchical RL, zero-shot RL, and PEX (8 seeds).
Refer to caption: x39.png

Figure 6 shows the comparison results
on top of the same pre-trained unsupervised skill policy.
Since PEX is not directly compatible with IQL, we evaluate PEX only on the tasks with TD3 (*e.g*., ExORL tasks).
The plots suggest that our fine-tuning strategy leads to
significantly better performance than previous approaches.
This is because pre-trained offline skill policies are often not perfect
(due to the limited coverage or suboptimality of the dataset),
and thus using a fixed offline policy
is often not sufficient to achieve strong performance in downstream tasks. We provide additional results in Appendix A.6.

#### 5.7 Ablation study

Figure: Figure 7: Ablation study of reward scale matching (4 seeds).
Refer to caption: x42.png

Reward scale matching.
In Section 4.2, we propose a simple reward scale matching technique that bridges the gap between intrinsic rewards and downstream task rewards.
We ablate this component, and report the results in Figure 8.
The results suggest that our reward scale matching technique
effectively prevents a performance drop at the beginning of the online fine-tuning stage,
leading to substantially better final performance on dense-reward tasks (*e.g*., Walker Run and Cheetah Run).

Value transfer vs. policy transfer.
In our U2O RL framework, we transfer *both* the value function and policy from unsupervised pre-training to supervised fine-tuning.
To dissect the importance of each component, we conduct an ablation study,
in which we compare four settings: (1) without any transfer,
(2) value-only transfer, (3) policy-only transfer,
and (4) full transfer.
Figure 8 demonstrates the ablation results on Walker and AntMaze.
The results suggest that both value transfer and policy transfer matter in general,
but value transfer is more important than policy transfer.
This aligns with our findings in Section 5.5 as well as Kumar et al. [],
which says that the quality of value features often correlates
with the performance of TD-based RL algorithms.

We refer to the reader to Appendix for further analysis including (1) combining U2O RL with other offline unsupervised skill learning methods (Appendix A.1), (2) U2O RL on a different type of datasets (Appendix A.2), (3) comparisons between U2O RL and pure representation learning schemes (Appendix A.3), (4) U2O RL without reward samples in the bridging phase (Appendix A.4), (5) an ablation study with different skill identification strategies (Appendix A.5), and (6) additional results with different online RL strategies (Appendix A.6).

### 6 Conclusion

In this work, we investigated how unsupervised pre-training of diverse policies
enables better online fine-tuning
than standard supervised offline-to-online RL.
We showed that our U2O RL framework often achieves even better performance and stability
than previous offline-to-online RL approaches,
thanks to the rich representations learned by pre-training on diverse tasks.
We also demonstrated that U2O RL enables reusing a single offline pre-trained policy for multiple downstream tasks.

Do we expect U2O RL to be *always* better than O2O RL?
While we showed strong results of U2O RL throughout the paper, our framework does have limitations.
One limitation is that U2O RL may not be as effective
when the offline dataset is monolithic and heavily tailored toward the downstream task (see Appendix A.2),
which leaves little room for improvement for unsupervised offline RL compared to task-specific supervised offline RL.
We believe U2O RL can be most effective (compared to standard offline-to-online RL)
when the dataset is highly diverse so that unsupervised offline RL methods can learn a variety of behaviors and thus learn better features and representations.
Nonetheless, given the recent successes in large-scale self-supervised and unsupervised pre-training from unlabeled data,
we believe our U2O RL framework serves as a step toward a general recipe for scalable data-driven decision-making.

### Acknowledgments and Disclosure of Funding

We thank Qiyang Li, Mitsuhiko Nakamoto, Kevin Frans, Younggyo Seo, Changyeon Kim, and Juyong Lee for insightful discussions and helpful feedback on earlier drafts of this work.
This work was partly supported by the Korea Foundation for Advanced Studies (KFAS), ONR N00014-21-1-2838, and Intel.
This research used the Savio computational cluster resource provided by the Berkeley Research Computing program at UC Berkeley.

## Appendices

### Appendix A Additional Experiments

#### A.1 Can U2O RL be combined with other offline unsupervised RL methods?

Figure: Figure 9: U2O RL with Laplacian-based successor feature learning (8 seeds).
Refer to caption: x48.png

While we employ HILP [] as an offline unsupervised skill learning method in the U2O RL framework in our main experiments,
our framework can be combined with other offline unsupervised skill learning methods as well.
To show this, we replace HILP with a graph Laplacian-based successor feature method [, ] or goal-conditioned IQL (GC-IQL) [, ],
and report the results in Figures 10 and 10, respectively.
The results demonstrate that U2O RL with different unsupervised RL methods also improves performance over standard offline-to-online RL.

Figure: Figure 11: Feature dot product analysis with Laplacian-based successor feature learning (8 seeds).
Refer to caption: x54.png

Additionally, we show that other unsupervised skill learning methods also lead to better value representations.
We measure the same feature dot product metric in Section 5.5 with the graph Laplacian-based successor feature learning method and report the results in Figure 11.
The results suggest that this unsupervised RL method also prevents feature co-adaptation, leading to better features.

#### A.2 When is U2O RL better than O2O RL?

Figure: Figure 12: Online RL learning curves with expert-only datasets (4 seeds).
Refer to caption: x57.png

In Section 6, we discussed that U2O RL is advantageous over O2O RL especially when the offline dataset allows the underlying unsupervised RL method to learn diverse behaviors and thus capture diverse features.
Conversely, we do not particularly expect U2O RL to be better than O2O RL when the dataset is monolithic (*e.g*., consists of only expert trajectories, has less diversity, etc.).
To empirically show this, we conduct an additional experiment with a different dataset in AntMaze-large and AntMaze-ultra that consists of monolithic, expert trajectories (we collect a 1M-sized dataset by rolling out an offline pre-trained policy). Figure 12 shows the results.
As expected, we find that the performance difference between U2O and O2O is marginal in this monolithic setting.

#### A.3 Do we need to use unsupervised RL for pre-training representations?

**Table 2: Comparison between U2O RL and pure representation learning algorithms (4 seeds).**
| Task | antmaze-large-diverse |
| --- | --- |
| U2O (HILP, Q Ours) | 94.50 $\pm$ 3.16 |
| U2O (HILP, $\xi$) | 5.50 $\pm$ 1.91 |
| Temporal contrastive learning | 37.50 $\pm$ 15.00 |

In Sections 4.4 and 5.5, we hypothesized and empirically showed that U2O RL is often better than O2O RL
because it learns better representations.
This leads to the following natural question: do we *need* to use offline unsupervised *reinforcement learning*, as opposed to general representation learning?
To answer this question, we consider two pure representation learning algorithms as alternatives to unsupervised RL: temporal contrastive learning [] and Hilbert (metric) representation learning [], where the latter is equivalent to directly taking $\xi$ in the HILP framework (Equation 4) (note that the original U2O RL takes the Q function of HILP, not the Hilbert representation $\xi$ itself, which is used to train the Q function).
To evaluate their fine-tuning performances, for the temporal contrastive representation, we fine-tune both the Q function and policy with contrastive RL []; for the Hilbert representation, we take the pre-trained representation, add one new layer, and use it as the initialization of the Q function.
Table 2 shows the results on antmaze-large-diverse.
Somewhat intriguingly, the results suggest that it is important to use the full unsupervised RL procedure,
and pure representation learning methods result in much worse performance in this case.
This becomes more evident if we compare U2O RL (HILP Q, ours) and U2O RL (HILP $\xi$),
given that they are rooted in the same Hilbert representation.
We believe this is because, if we simply use an off-the-shelf representation learning,
there exists a discrepancy in training objectives between pre-training (*e.g*., metric learning) and fine-tuning (Q-learning).
On the other hand, in U2O RL, we pre-train a representation with unsupervised Q-learning (though with a different reward function),
and thus the discrepancy between pre-training and fine-tuning becomes less severe.

#### A.4 Can we do “bridging” without any reward-labeled data?

Figure: Figure 13: U2O RL without using reward-labeling in the offline dataset (8 seeds).
Refer to caption: x60.png

In the bridging phase of U2O RL (Section 4.2),
we assume a (small) reward-labeled dataset ${\mathcal{D}}_{\mathtt{reward}}$.
In our experiments, we sample a small number of transitions (*e.g*., $0.2\%$ in the case of DMC) from the offline dataset
and label them with the ground-truth reward function, as in prior works [, ].
However, these samples do not necessarily have to come from the offline dataset.
To show this, we conduct an additional experiment where we do not assume access to any of the existing reward samples or the ground-truth reward function in the bridging phase.
Specifically, we collect $10$K online samples with random skills and perform the linear regression in Equation 6 only using the collected online transitions.
We report the performances of U2O (without offline samples) and O2O in Figure 13.
The results show that U2O still works and outperforms the supervised offline-to-online RL baseline.

#### A.5 How do different strategies of skill identification affect performance?

Figure: Figure 14: Ablation study of skill identification (4 seeds).
Refer to caption: x62.png

To understand how skill identification strategies affect online RL performance, we compare our strategy in Section 4.2 with an alternative strategy that simply selects a random latent vector $z$ from the skill space. Figure 15 shows that the skill identification with a randomly selected latent vector performs worse than our strategy. This is likely because modulating the policy with the best latent vector helps boost task-relevant exploration and information.

#### A.6 Additional experiments on fine-tuning strategies

We additionally provide experimental results of fine-tuning strategies on a different task (*i.e*., Cheetah Run). Figure 15 shows that our fine-tuning strategy outperforms previous strategies, such as zero-shot RL and PEX. This result further supports the effectiveness of fine-tuning.

### Appendix B Experimental Details

For offline RL pre-training, we use $1$M training steps for ExORL, AntMaze, and Adroit and $500$K steps for Kitchen,
following Park et al. [].
For online fine-tuning, we use $1$M additional environment steps for ExORL, AntMaze, and Adroit and $500$K steps for Kitchen
with an update-to-data ratio of $1$.
We implement U2O RL based on the official implementation of HILP [].
We evaluate the normalized return with 50 episodes every 10k online steps for ExORL tasks, and every 100k online steps for AntMaze, Kitchen, and Adroit tasks.
We run our experiments on A5000 or RTX 3090 GPUs.
Each run takes at most $40$ hours (*e.g*. Visual Kitchen).
We provide our implementation in the supplementary material.

#### B.1 Environments and Datasets

ExORL []. In the ExORL benchmark, we consider four embodiments, Walker, Cheetah, Quadruped, and Jaco. Each embodiment has four tasks: Walker has {Run, Flip, Stand, Walk}, Cheetah has {Run, Run Backward, Walk, Walk Backward}, Quadruped has {Run, Jump, Stand, Walk}, and Jaco has {Reach Top Left, Reach Top Right, Reach Bottom Left, Reach Bottom Right}. For all the tasks in Walker, Cheetah, and Quadruped, the maximum return is $1000$, and Jaco has $250$. Each embodiment has an offline dataset, which is collected by running exploratory agents such as RND [], and then annotated with task reward function. We use the first $5$M transitions of the offline dataset following the prior work [, ]. The maximum episode length is $250$ (Jaco) or $1000$ (others).

AntMaze [, ].
In AntMaze, a quadruped agent aims at reaching the (pre-defined) target position in a maze and gets a positive reward when the agent arrives at a pre-defined neighborhood of the target position. We consider two types of Maze: antmaze-large [] and antmaze-ultra [], where the latter has twice the size of the former. Each maze has two types of offline datasets: play and diverse. The dataset consists of $999$ trajectories with an episode length of 1000. In each trajectory, an agent is initialized at a random location in the maze and is directed to an arbitrary location, which may not be the same as the target goal. At the evaluation, antmaze-large has a maximum episode length of $1000$, and antmaze-ultra has $2000$. We report normalized scores by multiplying the returns by 100.

Kitchen [, ]. In the Kitchen environment, a Franka robot should achieve four sub-tasks, microwave, slide cabinet, light switch, and kettle. Each task has a success criterion determined by an object configuration. Whenever the agent achieves a sub-task, a task reward of $1$ is given, where the maximum return is $4$. We consider two types of offline datasets: mixed and partial. We report normalized scores by multiplying the returns by $100$. For Visual-Kitchen, we follow the same camera configuration as Mendonca et al. [], Park et al. [], and Park et al. [], to render $64\times 64$ RGB observations, which are used instead of low-dimensional states. We report normalized scores by multiplying the returns by $25$.

Adroit [].
In Adroit, a $24$-DoF Shadow Hand robot should be controlled to achieve a desired task.
We consider two tasks: pen-binary and door-binary, following prior works [, ].
The maximum episode lengths of pen-binary and door-binary are $100$ and $200$. respectively. We report normalized scores by multiplying the returns by $100$.

#### B.2 Hyperparameters

**Table 3: Hyperparameters of unsupervised RL pre-training in ExORL.**
| Hyperparameter | Value |
| --- | --- |
| Learning rate | $0.0005$ (feature), $0.0001$ (others) |
| Optimizer | Adam [] |
| Minibatch size | $1024$ |
| Feature MLP dimensions | $(512,512)$ |
| Value MLP dimensions | $(1024,1024,1024)$ |
| Policy MLP dimensions | $(1024,1024,1024)$ |
| TD3 target smoothing coefficient | $0.01$ |
| TD3 discount factor $\gamma$ | $0.98$ |
| Latent dimension | $50$ |
| State samples for latent vector inference | $10000$ |
| Successor feature loss | Q loss |
| Hilbert representation discount factor | $0.96$ (Walker), $0.98$ (others) |
| Hilbert representation expectile | $0.5$ |
| Hilbert representation target smoothing coefficient | $0.005$ |

**Table 4: Hyperparameters of unsupervised RL pre-training in AntMaze, Kitchen, and Adroit.**
| Hyperparameter | Value |
| --- | --- |
| Learning rate | $0.0003$ |
| Optimizer | Adam [] |
| Minibatch size | $256$ (Adroit), $512$ (others) |
| Value MLP dimensions | $(256,256,256)$ (Adroit), $(512,512,512)$ (others) |
| Policy MLP dimensions | $(256,256,256)$ (Adroit), $(512,512,512)$ (others) |
| Target smoothing coefficient | $0.005$ |
| Discount factor $\gamma$ | $0.99$ |
| Latent dimension | $32$ |
| Hilbert representation discount factor | $0.99$ |
| Hilbert representation expectile | $0.95$ |
| Hilbert representation target smoothing coefficient | $0.005$ |
| HILP IQL expectile | $0.9$ (AntMaze), $0.7$ (others) |
| HILP AWR temperature | $0.5$ (Kitchen) $3$ (Adroit-door), $10$ (others) |

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: - Why does the feature dot product in the O2O framework for Walker Run and Cheetah Run (Fig. 4) diverge, indicating much poorer representation learning compared to U2O, yet the performance of O2O in these environments is similar or even superior to U2O? Intuitively, poorer representations should lead to poorer performance.
soundness: 2
strengths: - The paper is well-written, and the proposed U2O RL framework is explained clearly.
- Research into specific paradigms within the pretrain-finetune framework for RL is valuable, and this paper contributes to that discussion.
summary: This paper introduces Unsupervised-to-Online Reinforcement Learning (U2O RL) as an alternative to the conventional Offline-to-Online RL (O2O RL) framework. U2O RL replaces domain-specific, supervised offline RL with unsupervised offline RL, allowing a single pre-trained model to be adapted for multiple tasks. Experiments conducted across nine environments demonstrate that U2O RL can match or even surpass previous methods.

I do not currently agree with the direct acceptance of this paper. However, I recognize the importance of research on RL in pretrain-finetune frameworks. If the authors can satisfactorily address my concerns, I would be open to increasing my rating.
weaknesses: - While this paper proposes the U2O RL framework, it does not introduce any novel methods. Both the unsupervised offline RL pre-training and the online fine-tuning stages rely on existing algorithms. Additionally, the reward scaling adjustment in the bridging stage has already been employed in prior reward design approaches. Proposing a new “framework” is reasonable, but I believe the paper needs to provide more substantial evidence on why this U2O RL framework is more effective than the traditional O2O approach. Simply demonstrating feature co-adaptation to support the efficacy of representation learning in the offline phase may be insufficient. Furthermore, the paper should conduct a broader set of experiments: for instance, by testing various offline and online algorithms and using a wider range of environments to validate the framework’s effectiveness. Without these enhancements, the work seems somewhat incremental.
- For general reward maximization tasks, such as Walker, Cheetah, and Jaco, the U2O RL framework does not demonstrate a notable advantage; in some cases (e.g., Cheetah), performance is even lower. This could stem from difficulties in selecting an optimal skill latent vector  $z^*$  for these tasks, highlighting a potential limitation of the U2O framework.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. Can the explanation of O2O vs off-policy online RL can be made clear in the paper? Those are important baselines. It seems to be important to put the U2O algorithm in main paper as it wasnt clear that the old dataset was kept around.  It would help to be very clear the differences between U20, O2O and Off-policy online RL 
2. Why are results missing in Table 1?
3. In line 249, it might be helpful to have a citation for the reward regression technique with successor features reward.
soundness: 3
strengths: 1. The paper is clearly written and well motivated, having a single reusable model to perform finetuning makes sense and can improve RL pretraining.
2. The paper looks at a particular method HILP and provides a reward scale matching scheme to enable finetuning. This turns out to be quite important in performing efficient finetuning.
3. The paper considers a wide variety of tasks to demonstrate potential improvements over offline to online finetuning.
summary: The paper presents an empirical analysis of how unsupervised to online finetuning for RL is better than offline to online finetuning.
weaknesses: 1. Insufficient empirical evidence to claim U20>O2O: In figure 3, it seems the results are not significant in 10/14 environments. How can we claim the U20 is a better strategy? Furthermore, insufficient details are provided about baselines of O2O and  off policy RL - eg. do they use the same network sizes and discount factor? It is clear in Table 1 that the baselines and U2O use different network sizes and discount factor as the prior entries are based on discount of 0.99 and use a network size of (256,256) for most tasks. These raises a number of questions on empirical evaluations - maybe the improvements in some domain is because of discount factor?
2. Claim of better features: As I understand, the main claim of the paper is that U2O learn better features. Feature rank collapse is a known problem with offline RL but there have been fixes provided for it in the past. Ex. DR3. It seems comparisons are not made to those modifications at all in this paper.
3. Generality of approach: Unsupervised RL encompasses a broad range of methods. Methods that are based on maximizing mutual information; methods that discover options; methods that capture a bag of skills and update it over time (eg. Voyager). This method relies on reusing policy and value functions to initialize and reward shaping; how will this method work for all the other unsupervised RL approaches. To be calling this a framework might be an overstatement as they consider very related unsupervised approaches based on a single bucket of successor features. 

4. Prior works have proposed using unsupervised to online objectives and can be attributed correctly: [1,2,3,4]. I believe the claiming of an entirely new framework is somewhat overclaimed.

[1]:Nasiriany, Soroush, et al. "Planning with goal-conditioned policies." *Advances in neural information processing systems* 32 (2019).

[2]:Eysenbach, Ben, Russ R. Salakhutdinov, and Sergey Levine. "Search on the replay buffer: Bridging planning and reinforcement learning." *Advances in neural information processing systems* 32 (2019).

[3]: Ma, Yecheng Jason, et al. "Vip: Towards universal visual reward and representation via value-implicit pre-training." *arXiv preprint arXiv:2210.00030* (2022).

[4]: Ma, Yecheng Jason, et al. "Liv: Language-image representations and rewards for robotic control." *International Conference on Machine Learning*. PMLR, 2023.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 4
questions: - Do you agree that your method does not outperform O2O methods when using expert data sets?
- Why have you only included one experiment using exert data sets, put it in the appendix, and not explained the results in the main text?
soundness: 1
strengths: Overall, I found the paper well-written and easy to follow.
The problem that the authors are working on -- pre-training in RL -- is important and definitely of interest to the community.

For the most part, I found the experiments insightful and thorough.
I particularly like the feature dot product analysis, this is a nice addition to the paper.
summary: This paper presents a new offline-to-online (O2O) reinforcement learning (RL) method, where they replace the offline RL pre-training 
stage, with offline unsupervised RL pre-training.
The offline unsupervised RL pre-training consists of learning an offline policy conditioned on a latent skill vector, resembling a multi-task or goal-conditioned policy.
The paper proposed to name this problem setting as unsupervised-to-online RL (U2ORL).
They evaluate their method in nine tasks from five benchmarks.
They claim that their results suggest that their approach either matches or outperforms O2O RL methods.
weaknesses: The biggest issue with this paper is that the abstract claims "we empirically demonstrate that U2O RL achieves strong performance that matches or 
even outperforms previous offline-to-online RL approaches ...".
However, Figure 12 suggests that O2O outperforms U2O when the offline data only contains expert data.
As such, this claim that U2O either outperforms or matches O2O is false. It is dependent on the type of offline data.
The authors clearly address this in the conclusion but as a reader, I am very disappointed getting to the 
end of the paper to find the claim in the abstract is false.
The authors need to update the abstract and correct this claim.

Further to this, I want to know how U2O compares to O2O in Kitchen when using the "complete" data set.
It seems like the authors have avoided including results when using expert data sets because their method 
performs worse than O2O RL methods in this setting.
These results are important and should be included.
I suggest the authors include results for the Kitchen task with the complete data set and include a discussion of how their method performs against O2O RL when using different types of offline data sets.

How I see it, this method requires a diverse data set that is collected by an unsupervised RL method.
This paper then proposes a new way to pre-train on this diverse data set.
That is a good contribution, but do not overclaim your contribution.
This could, for example, motivate collecting diverse data sets and the investigation of how to best incorporate expert data into this method.

Finally, this paper is fairly incremental as it simply combines existing methods. 
As such, I think it is important that the experiments are thorough so that we gain lots of insights about why we should care about this U2O method.
I think the authors are almost there as the results provide insights that I think are valuable to the community.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: N/A
soundness: 4
strengths: 1. Replacing O2O with U2O is a reasonable and innovative approach.
2. The authors comprehensively discuss the advantages of U2O over O2O, supported by extensive empirical evidence.
3. Experimental results indicate that U2O outperforms O2O in terms of performance.
4. The paper is well-written, clearly highlighting the challenges and contributions.
5. The detailed explanations in the Experiments and Appendix effectively justify why U2O is superior to O2O.
summary: This paper introduces a novel concept termed unsupervised-to-online reinforcement learning (U2O), which replaces the conventional task-specific offline RL pretraining in the offline2online RL paradigm with task-agnostic unsupervised offline RL pretraining. The study reveals that unsupervised pretraining enables agents to adapt more quickly to downstream tasks compared to domain-specific offline RL pretraining. This advantage stems from the richer and more general-purpose representations fostered by unsupervised learning, which encode a broader range of task features and thus prevent feature collapse specific to any single task. The paper presents a practical and straightforward implementation of their U2O method. Empirical experiments conducted on state-based and image-based tasks, along with extensive ablations, demonstrate the effectiveness of leveraging unsupervised pretraining for general-purpose representation over traditional offline RL pretraining.
weaknesses: 1. One potential limitation is the skill identification process. In this paper, the authors determine the optimal skill $z*$ by minimizing the MSE loss w.r.t to the single-step reward using a small reward-labeled dataset. Ideally, a skill should encapsulate more long-horizon information and should not be solely defined by single-step rewards. More advanced skill identification process can be developed in the future.
2. The scope of this paper is relatively narrow. In my view, unsupervised pretraining can facilitate various types of fine-tuning, including offline RL fine-tuning, imitation learning, and others. Limiting the study to only online RL fine-tuning could be expanded for a more comprehensive evaluation.

Meta-review: This paper proposes a recipe for doing unsupervised-to-online RL. The idea is that if you have a big dataset of observation-action pairs from different tasks, you can learn a task-conditioned policy $\pi(a \mid s, z)$ where your $z$ is your task vector. No rewards are used during this part. Then given a task, you can identify the right task vector $z^\star$ using reward values, and do online RL to further improve the performance of the policy $\pi(a \mid s, z^\star)$. 

The main strengths of the paper are:

- RL is sample expensive, so pre-training before doing online RL is an important task. 
- Approach is simple and easy and also composable (e.g., one can use any online RL approach). 


Main weakness:

- The main issue, as raised by reviewers, concerns comparison with O2O. As pointed out by reviewers, U2O does not uniformly outperform offline-to-online (O2O) works across the board and either matches O2O in some domains or even gets outperformed by O2O when the offline data is of expert type. The authors agree with this. The main counterpoint is that expert data is hard to collect and U2O uses a single pre-trained model which is still an advantage. I agree with the latter point, but one would still wish for U2O to match O2O when the latter has expert data.

- Reviewers have mentioned that the paper overclaims its contribution of a general framework when the approach is more specific and also overclaims gains over past O2O work in the abstract. In response, the authors have edited the writing to reduce their claims.

- Reviewers have also mentioned that the approach isn't novel. However, this in itself doesn't concern me. In fact, it is great to reuse existing approaches. But what would have then been good is to see a more real-world application rather than simulator-based evaluations.

The authors have addressed these claims by removing overclaims of a framework, toning down the abstract, and adding additional experiments. I worry that this is a non-trivial amount of change that should be another submission. Further, while the approach is simple, neither it is strictly better than the past approach, nor is the core idea of using pre-training for RL novel, nor it is demonstrated in a real-world application. For this reason, I am currently leaning towards rejection but I won't mind if it gets accepted.

Decision: Reject