Unsupervised Zero-Shot Reinforcement Learning via Dual-Value Forward-Backward Representation
Authors: Jingbo Sun
Songjun Tu
qichao Zhang
Haoran Li
Xin Liu
Yaran Chen
Ke Chen
Dongbin Zhao
Venue: ICLR 2025 Poster (2025)
Keywords: unsupervised reinforcement learning
zero-shot generalization
skill discovery
successor representation
Primary area: reinforcement learning
TL;DR: The proposed dual-value forward-backward representation framework is the first method to simultaneously achieve superior zero-shot generalization and fine-tuning task adaptation capabilities in online URL.

Abstract: Online unsupervised reinforcement learning (URL) can discover diverse skills via reward-free pre-training and exhibits impressive downstream task adaptation abilities through further fine-tuning.
However, online URL methods face challenges in achieving zero-shot generalization, i.e., directly applying pre-trained policies to downstream tasks without additional planning or learning.
In this paper, we propose a novel Dual-Value Forward-Backward representation (DVFB) framework with a contrastive entropy intrinsic reward to achieve both zero-shot generalization and fine-tuning adaptation in online URL.
On the one hand, we demonstrate that poor exploration in forward-backward representations can lead to limited data diversity in online URL, impairing successor measures, and ultimately constraining generalization ability.
To address this issue, the DVFB framework learns successor measures through a skill value function while promoting data diversity through an exploration value function, thus enabling zero-shot generalization.
On the other hand, and somewhat surprisingly, by employing a straightforward dual-value fine-tuning scheme combined with a reward mapping technique, the pre-trained policy further enhances its performance through fine-tuning on downstream tasks, building on its zero-shot performance.
Through extensive multi-task generalization experiments, DVFB demonstrates both superior zero-shot generalization (outperforming on all 12 tasks) and fine-tuning adaptation (leading on 10 out of 12 tasks) abilities, surpassing state-of-the-art URL methods.

Published as a conference paper at ICLR 2025
UNSUPERVISED ZERO-SHOT REINFORCEMENT LEARNING VIA DUAL-VALUE FORWARD-BACKWARD REPRESENTATION
Jingbo Sun Songjun Tu Qichao Zhang".; Xin Liu Haoran Li Yaran Chen Ke Chen', Dongbin L2.
"Inttitute of Automation, Chinese Academy of Sciences, PPengcheng Laboratory Uniiersityy of Chinese Academy of Sciences, *Xi' an inunbonn,,2022,, tusong Jun2223,, zhanggichao22144 (liuxin2021, 1ihaoran2015 dongbin. aia.) on chenk02@pcl ac. en, yaran. cwwn@xjtlu.eeeeee edu. cn
University
ac en
ABSTRACT
Online unsupervised reinforcement learning (URL) can discover diverse skills via reward-free pre-training and exhibits impressive downstream task adaptation abil- ities through further fine-tuning. However, online URL methods face challenges in achieving zero-shot generalization, i.e., directly applying pre- pre-trained policies downstream tasks without additional planning or learning. In this paper, we pro- pose a novel VulleValue Forward-Backward Backward representation (DVFB) framework with a contrastive entropy intrinsic reward to achieve both zero-shot generalization and fine-tuning adaptation in online URL. On the one hand, we demonstrate that representations can lead to limited data di-
poor exploration in Norward-backwerr versity in online URL, impairing successor measures, and ultimately constraining generalization ability. To address this issue, the DVFB framework learns succes- sor measures through a skill value function while promoting data diversity through exploration value function, thus enabling zero-shot generalization. On the other hand, and somewhat surprisingly, by employing a straightforward dual-value fine- tuning scheme combined with a reward mapping technique, the pre-trained pol- icy further enhances its performance through fine-tuning on downstream tasks. Through extensive experiments, DVFB demonstrates both superior zero-shot gen- eralization (outperforming on all 12 tasks) and fine-tuning adaptation (leading on 10 of 12 tasks) abilities, surpassing state of-the-ar (SOTA) URL methods. Our code is available at https:://gtthtii com http://ouuuu///ee FDSF.
INTRODUCTION
In recent years, deep reinforcement learning (DRL) (Sutton & Barto 1999) has achieved remark- able success in various fields, including game AI (Tang et al. 2023), autonomous driving (Wu al.) 2022). robot control (Li et al., 2024). However. most DRL approaches rely on predefined task- specific rewards for training, which limits the generalizability of the learned policies to new tasks. Given the varied demands of real-world applications, it is crucial to develop agents capable of ad- dressing multiple tasks directly, as zero-shot reinforcement learning (zero-shot RL) (Touati al, 2023), or through task adaptation. This need has spurred growing interest in training task-agnostic policies that can generalize across tasks, a paradigm known as unsupervised reinforcement learning (URL). In this paper, we aim to achieve downstream task generalization in online URL with both zero-shot capability and the potential for further improvement through fine-tuning.
Previous research (Eysenbach al. 2019 Laskin et al. 2022 Park et al. 2024) has explored un- supervised skill discovery (USD) for task generalization, focusing on developing diverse skills via reward-free learning to enable task adaptation through fine-tuning. However, USD methods aim to maximize the divergence between the state distribution skills and the average state distribution, posing challenges for reoo generalization. Other studies, such BBrreto et al. (2017); Touati et al. (2023) utilize successor representation (SR) techniques (Dayan 1993) to facilitate zero-shot
"Corresponding author.

---

Published as a conference paper at ICLR 2025
generalization. SR methods can be categorized into successor features (SF) (Barreto et al. 2017 and forward barkwrdd representations (FB) (Touati & Ollivier 2021). They define skill vectors based on successor representations, enabling them to infer near-optimal skills from minimal demonstra- tions of downstream tasks. Unfortunately, SR methods rely on pre-collected diverse datasets (Jeen et al. 2024) and constrained to offiine settings. Our findings further reveal their poor zero-shot generalization performance in online URL. These insights trigger our further thought: Can the SR mechanism, such as FB, be extended to online URL enable zero-shot generalization capability?
Extending the zero-shot generalization of forward- backward representations online URL presents several challenges. First, we find that FB mechanism typically exhibit limited exploration capabilities, re- sulting in reduced data diversity. This limitation hin- ders the effectiveness of successor measures and fur-
DVF
Fine-tune Performance
800
BcCL
CeSD
700
RND
ther restricts zero-shot generalization in downstream tasks. Second, FB mechanism rely on implicit re- wards that have fixed relationship with skill during
600
CIC
DIAYN
500
SMM
pre-trate treiniiinnng, which prevents the direct use of intrin- sic rewards enhance online exploration, methods do. Therefore, the key to extending FB to online URL for zero-shot generalization develop- ing mechanism that enhances policy exploration for diverse buffer data while maintaining efficient successor representation and skill learning.
400
USD
100
200
300
400
500
600
700
Zero-shot Performance
Figure DVFB achieves leading zero-shot and fine-tuning performances in Walker and Quadruped Domain.
In this paper, we explore the underlying reasons for the failure of the FB mechanism in online URL. Our findings reveal that insufficient exploration in FB leads to low data diversity, which impeees the learning of successor measures and results in inaccurate value functions for downstream tasks, ultimately limiting zero-shot capabilities. To address these challenges, we propose Dual- Value Forward- Backurrrd representation (DVFB) framework, which leverages a skill value function to learn forward-backwant representations while incorporating an exploration value function to courage exploration. We introduce a novel intrinsic objective based on contrastive learning, which enhances exploration while learning distinguishable state transitions, thereby improving zero-shot generalization across various downstream tasks. Additionally, we introduce a dual-value fine-tuning scheme with a reward mapping technique that leverages both the pre-trained skill value and the downstream task value to ensure performance improvement beyond zero-shot capabilities.
In summary, contributions
follows:
We investigate the reasons for FB's failure in online URL and find that its inadequate ex- ploration leads low data diversity. This hinders the learning of successor measures and the value network, ultimately limiting zero-shot generalization capabilities.
We propose novel dual-value forward-backwant representation (DVFB) architecture with intrinsic reward based on contrastive learning. The pre-trained DVFB agents can zero- shot generalize to various downstream tasks.
We introduce dual-value fine-tuning scheme with a reward mapping technique, enhancing performance stability during downstream task fine-tuning beyond zero-shot capabilities by leveraging the pre-trained skill value function with forward-backward representations.
Our method comprehensively validated across twelve challenging robot control tasks in the DeepMind Control Suite, demonstrating superior generalization performance. To the best our knowledge, our proposed DVFB is the first method to achieve both zero-shot generalization and fine-tuning capabilities in online URL, shown in Figure
2 RELATED WORK
2.1 ZERO-SHOT TASK GENRALIZATION IN RL
Zero-shot task generalization in RL refers to an agent's s ability to apply its knowledge directly to new tasks after training, without any additional training (Touati et al. 2023 JJen et al. 2024). Methods
2

---

Published as a conference paper at ICLR 2025
for addressing task generalization in RL include model-baseds RL, multi-task RL, unsupervised skill discovery, and successor representations. Model based reinforcement learning (RL) (Chua et al.
2018
Hatner al.) 202I) trains a task-agnostic world model to assist in decision-making; however,
it still requires a reward function to plan for new tasks. Multi-task RI. (He et al. 2024 Lan et al.,
2024 enables generalization across predefined related tasks but cannot be applied ied unfamiliar
tasks. USD [Eysenbach et al., 2019 Laskin et al., 2022) learns diverse skills in reward-free envi- ronments, but its objective focuses on maximizing the distance between the skill's state distribution and the average state marginal distribution. Consequently, USD is unable to achieve zero-shot gen- eralization (Eysenbach et al.
2022). None of these methods can achieve zero-shot generalization.
Successor feature (SF) [Dayan 1993) methods establish successor features using handcrafted or learned reward-related features e which can then be zero-shot generalized to downstream tasks [Barreto et al. [017]] Zhang et al. 2017). To relax the linear assumption of & and reward in SF, suc- cessor measures directly learn future state distributions and avoid the need for & (Touati & Ollivier) 2021 FB methods (Touati et al. 2023) achieve zero-shot generalization by learning successor mea- sures from pre-collected diverse data. FRE (Frans et al. 2024) focuses on learning functional rep- resentations by encoding state-reward samples with variational auto-encoder to achieve zero-shot generalization. However, these methods require access to offline datasets for pre-training, which cannot be expected for most real problems (Jeen et al.) 2024). In addition, they show high data sen- sitivity and limited performance on pre-collected fixed dataset (see detailed analysis in Appendix ( Extending FB the online URL setting without the offline dataset collection process is a promising direction. Unfortunately, we find it fails to achieve zero-shot generalization in online URL. Hence, this paper proposes a DVFB framework that extends FB methods to online URL for both zero-shot generalization and efficient task adaptation to downstream tasks.
2.2 UNSUPERVISED PRE-TRAINING IN RL
Unsupervised pre-training methods in reinforcement learning can be divided into two categories: online and offline. This paper primarily focuses on the former. Online URL typically involves learning useful representations or skills for downstream tasks through intrinsic rewards. Based on the modeling of intrinsic rewards, URL can be further divided into unsupervised skill discovery
(USD) methods(Eysenbach et al., 2019; Sharma et al.) 2020 Park et al. 2022 Laskin et al. 2022 Park et al.] 2024) and data coverage maximization methods (Liu & Abbeel| 2021b; Yarats et al.
2021 Lee et 2019)) Existing USD methods leverage mutual information as an intrinsic reward
to learn diverse skills that can rapidly adapt downstream tasks(Eyyenbach et al.] 2019. Sharma
jet al. 2020 Liu et al., 2025) Laskin et al., 2022). Some (EysensEyyennaac et al., [202]] Yang
al. 2024) indicate that maximizing mutual information equivalent to maximizing the difference
between the average state distribution and the most distinct skill state distribution. As a result, al-
though these methods enhance fine-tuning by providing optimal initialization, they lack the capacity for zero-shot generalization to downstream tasks. Some studies attempt to improve state coverage with Euclidean distance constraints Park et al. 2022), Wasserstein distance constraints(Park et al.) 2024), and guided skills(Kin et al. 2024). However, they lack clear definition of skills and their connection to downstream task rewards, which hinders zero-shot generalization.
PRELIMINARIES
3.1 REWARD-FREE MARKOV DECISION PROCESSES AND GENRALIZATION IN URL
In unsupervised RL, a reward-free Markov Decision Process (MDP) (Sttonn & Barto 9999 Touati et al., (202) is defined as the tuple M = (S, A, P.7). where S is the state space, is the ac- tion space, and P(ds'[s. [s, a) is the probability measure of the next state s' E S, which defines the stochastic transition from state 8 to state s' after taking an action a. 7 is the discount factor. Given state-action pair (so, ao) E S x A and a policy
S - Prob(AA), we use Pr(- So, Go, т) and E(1 so, do, T) to represent the probability and expectation of the state-action sequence (st, at) obtained by starting from the initial state (5o, an) and using the policy n, where states are sampled as St ~ P(ds, $t-1, 14-1) and actions are sampled as a₂ ~ T(da; | st). The state transition probability under the policy is defined as PM (ds' s) = P(ds' s, ad)(aa s). Given a ward function r : S + R. the Q-function of a policy π with respect to the reward r is defined as QF (No, an) := - (B)(r(s))) 80: 0o,
3

---

Published as a conference paper at ICLR 2025
During the pre-training phase in URL, the agent interacts with reward free environment and learns skills using data collected by periodically updating the skills. At the testing time, the policy aims to maximize the expected discounted return for the task, given by Teoall zero-shot generalization, the agent must infer skills from small dataset Dinter r() where k < 10000, obtained through limited interactions with the downstream task environment.
So: ao, TT. For
3.2 THE FORWARD-BACKWAN REPRESENTATIONS
Following (Touati Ollivier 2021). the successor measure M (s0, ao. represents the cumulative discounted time spent in each state s1+1, starting from a state-action pair (so, ao) and following the policy æ, namely:
M (So, X) = - Pr E E 80, ao, T); S
(1)
220
The FB framework provides approximately optimal policy for any reward by learning tractable representation of the successor measure. Let Rd be representation space, and let p be an arbitrary distribution states. The FB framework estimates the successor measure M by learning forward representation : x A Rd + Rd,, a backward representation B + Rd, and a set of skill policies (Nz)zERd, such that:
M (su. ao. ao. X) = / F(su, aosu, 2) B(s)p(ds), Aso E S, E A,X R"
(s))) & arg max F(s. 0, =)
(x.a)ExAA,1CC
If the approximation 2 holds, for any reward function r, the policy тz, is optimal for with the optimal Q-function given by: Q Fss a, 2 2,. The near-optimal skill represented 2, B(s")r. However, FB's limited exploration capability makes it challenging to sample diverse data in online URL setting, hindering the validity of the approximation in (2) for all states. Therefore, struggles to maintain zero-shot generalization ability in the online setting.
WHAT HINDERS THE APPLICATION OF FB TO ONLINE URL?
Does FB's exploration affects zero-shot performance in online URL? Unlike offline URL, where the agent learns skills from pre-collected diverse data, in online URL the agent is trained using self- generated trajectories through online exploration. Consequently, whether FB's exploration ability impacts its zero-shot performance in online URI. remains a question. To explore this impact, we first evaluate FB's online exploration performance. Specifically, we compare the motion trajectories generated by FB with those produced by random network distillation (RND), a pure exploration method, within the Walker domain. The trajectories for FB are generated by randomly sampling 20 skills after million steps of pre-training, with each skill repeated four times, whereas RND generates trajectories 80 times. The walking distances for the different trajectories of the RND and FB agents are presented in Figure ((a) and (b), respectively. In the figure, lines of the same color represent trajectories corresponding to the same skill. Compared to RND, FB explores only
80
60
60
60
Skill Inoex
Skill Index
Index
Skill
20
20
-0
-40
-20
20
40
60
-60
-40
-20
20
40
60
60 -40
-20
40
60
Position
Position
Position
(a) Trajectories of RND agent
(b) Trajectories of FB-online agent
(c) Trajectories of FB-offline agents
Figure 2: The trajectories of the RND, FB-online, and FR-offline agents in the Walker walk task. The FB-online agent learns short-range trajectories, in contrast to the RND agent's diverse exploration and the FB-offline agent's ability to master long-range locomotion skills.
4

---

Published as a conference paper at ICLR 2025
short-distance trajectories, resulting in less diverse exploration outcomes. Thus, infer that insuf- ficient exploration in FB leads to limited data diversity, which impacts its zero-shot generalization. To verify this conjecture,
trained the FB using the diverse dataset generated by well-trained RND agent, as described in (Touati et al. 2023), which we refer to as FB-offline. Figure (igureeec) illus- trates the trajectories of different skills from FB-offline, some of which exhibit significant positional changes. Furthermore, we conduct an additional experiment on specific downstream task: Walker Walk, with the training curves shown in Figure Ba)). The zero-shot generalization performance of FB FB-offline improves with training, while FB-online, lacking any prior data, consistently performs poorly. Therefore, we conclude that the impact of exploration on data diversity limits FB's zero-shot generalization ability.
How do exploration and data diversity affect FB's zero-shot performance? FB achieves zero-shot generalization by ensuring that the successor representation satisfies Equation 2 for all states. However, the online URL, only the states that are explored meet this requirement. If high-reward states related to downstream tasks re- main unexplored, may result inaccurate succes- measures and value estimates, ultimately affecting the capability for zero-shot generalization. To vali- date this hypothesis, value functions composed of successor measures for FB- online and FB-offline in the Walker walk task after pre-
FB-online
Episode Rewards
600
FB-offline
500
compare the accuracy of the
400
300
200
training. As shown at the top of Figure (b)), we ana-
100
lyze the normalized values of trajectories across differ- ent ranges of episode rewards. The value function of FB-online estimates higher values for low-return trajec-
Timestops
x10
(a) G Generalization performance curve
tories, while FB-offline assigns higher value for high- return trajectories. This suggests that FB-offline, trained on diverse data, possesses accurate successor measures and value functions for downstream tasks, whereas FB- online, trained self-generated data, does not. Addi- tionally, we introduce Spearman correlation (Spearman 1987) reflect the correlation between the prior skill value (QM = F(ss, a, =)] 2) and the episode rewards of trajectories. Spearman correlation measures the strength and direction of monotonic relationship between two
Nomalized Value
5peaman Carrelation
0.75
0.50
FD-online
FB-offline
ranked variables. A higher Spearman correlation sug- gests that the inferred value function of the policy aligns more closely with the rewards of the downstream tasks. As shown at the bottom of Figure bb. FB-offline ex- hibits improved Spearman correlation across all trajecto- ries, while FB-online displays poor correlation with the episode rewards of high-return trajectories. This finding indicates that FB- FB-online's successor measures for down- stream tasks are inaccurate. Therefore, we conclude that exploration hinders FB's zero-shot ability by impacting its successor measure of high-reward states.
0.00
Episode Reward Ranges
Value function properties
Generalization curve and value function properties. (a) presents the agents' performance during pre- training. while (b) illustrates the malized skill value and the Spearman correlation between skill value and turn across different return ranges.
Figure 3:
METHODOLOGY
5.1 DUAL-VALUE FORWARD-BACKWER REPRESENTATION PRE-TRAINING FRAMEWORK
To enhance the exploration capability of forward-backe representations, straightforward idea is to incorporate exploration-base intrinsic rewards. The forward-backwall representation achieves zero-shot generalization by learning the successor measure M= that satisfies a Bellman-like equa- tion: A = P P+ To satisfy Bellman's 's equation, the value function for skill 2 is associated with fixed implicit reward, given by: Pimplseit B (B, Touati et al. 2023). There- fore, directly training the value function QM using intrinsic rewards for exploration, which are in conflict with Timplici for FB representations, would impede the skill learning.
5

---

Published as a conference paper at ICLR 2025
Unsupervised Pre-training Phase
(ee)))
Contrastive Loss h
Entropy
Task-agnostic Environment
A
Replay Buffer
(Br(s))))
Bridd
Unsupervised Interaction
Intrinsic Reward
2r Critic
(5..
E.M
Qp
Marget
Qp +Taer
Torger
x
+
Critic Training
Lgg
Qp(Ae .c.))
Qp Critic
Exploration Critic Training
Dual-Value Actor Training
Dual-Value Critic Training
II. Fine-tuning Phase
Mapping
Qui Critic
xy Gprget
Aget
Dowastrom Tank
Lig Blage
Eg Yees
@r(s..).2)
OF Critic
Downstream Task Interaction
Downstream Task Critic Training
Dual-Value Acoor Finetuning
Exploration Value Function
Skill Value Function
Downstream Task Value F Function
Figure 4: The overall framework of DVFB. During the pre- pre-training, the exploration Critic Qp and skill Critic Qu are trained and utilized to train the actor. During the fine-tuning, skills are inferred, and the downstream task Critic Qr is trained with mapped extrinsic reward ry. Finally, the policy is fine-tuned with the downstream task Critic QF and the prior skill Critic QM-
Dual- Value Pre-training Architecture. To address this reward conflict problem, propose dual-value network architecture including the exploration value and skill value. The pseudocode for pre-training is presented in Appendix [ Specifically, we train additional exploration value function Qr to promote policy exploration. The training objective for Qr is defined as:
[(Qr) = (e (e... (t.e....)>(((( ((rrr 0), (Fintr + ( (1)1,
(3)
where Tintr represents the intrinsic reward that encourages exploration, D refers the replay buffer, representa- tions to learn the skill value QM F(st, 0t, T The training objective for keward-backwarr representations is given by:
and Qr denotes the target networks of Qp. In parallel, we train the
L(F, B) B(e.ou ( (s, B(s') ((41) B(s')) (4)
(e [F( at, 2), 2) B(st+1)]
where s'' represents the future state of 8t, and B are the target networks for the forward and backward representations, respectively. We update the policy network by utilizing the exploration value function Qp and skill value function QM- The training objective for the policy network π is:
L (a
F(st, ae, 2) (- aQr(si, a,, 2)],
(5)
where a is coefficient. This dual-value architecture encourages the policy æ to explore diverse data in online URL guided by exploration value Qp without impairing its skill learning capabilities. Con- sequently, the DVFB agent learns successor measures for various states using diverse buffer data, thereby enabling zero-shot generalization for downstream tasks with different reward functions.
Contrastive Entropy Intrinsic Reward. Although pure exploration rewards such as RND improve policy exploration by focusing exclusively unseen states, they overlook the discriminability of learned skills during exploration, which is helpful for FB-based skill learning. We introduce a novel contrastive entropy reward encourage skill discrimination during exploration. First, we utilize the intrinsic reward Fred from RND as a pure exploration reward:
T Trad (|(())))) (aap))))=(((+1(()11) J(s4+1))1

---

Published as a conference paper at ICLR 2025
where f is the representation network of states, and f is the target network of f with frozen parame- ters. Then, we train the encoders op and ø for state transitions and skills using contrastive learning to enhance skill discriminability. This approach improves the similarity between state transitions and their corresponding skills by updating the encoders with NCE loss (Gutmann & Hyvärinen 2010) within contrastive learning framework (Chen al., 2020). Since skills derived from historical samples may not satisfy Equation to disturbances caused by exploration, our method leverages skills inferred from the next state train a skill discriminator that aligns with the forward-backwanl representation. The training objective of the encoders is:
(TT)) T ((B(s(+)))) () [T) ( (B (8+++1):)][T
N Σ j=1
(T(TT) oi ((B(sc-)))) [|T((;))]]l ((s+11)i
log N exp
where and é are the encoders for the transition and skill = B(s+11,, T is the temperature parameter, т denotes a state transition (st, St+1), N is the number of contrastive pairs, and i and are their indices. We leverage the trained encoders to compute similarities in contrastive entropy, enhancing skill discernibility by using particle estimates of dissimilarity entropy for each state tran- sition discriminative intrinsic rewards. The contrastive entropy intrinsic reward rce is:
1
Σ log (2
((TT)) o(²")
Tee x No
(8)
S'EN
where 2" is a KNN skill, N₂ is the number of KNNs. The total intrinsic reward Tintr is:
Tintr = Trad + Brees
where / is coefficient. We combine the pure exploration reward and contrastive entropy reward as intrinsic rewards, enhancing exploration while keeping the skill learning ability of FB. Furthermore, we offer theoretical analysis to support the zero-shot generalization of DVFB in Appendix H
5.2 DUAL- VALUE FORWARD-BACKWARD REPRESENTATION FINE-TUNING SCHEME
Value Fine-tuning Scheme. Furthermore, we propose a dual-value fine-tuning scheme that leverages prior skill value function and zero-shot generalized policy to achieve improved perfor- mance through fine-tuning. During fine-tuning, we use environmental rewards from downstream tasks train task value function QF, guiding the agent learn policies with improved task- specific performances. The training objective for the downstream task value function as follows:
L(QF) =)s.,.r.)>
dt, z) (ry (04411),4441;
(10)
where r is the downstream task reward used to fine-tune. Since the downstream task value function trained from scratch, its inaceuracy during the early stages of fine-tuning can disrupt the zero-shot initial policy, hindering lin-ttunig efficiency. To address this issue, we incorporate prior knowledge from the pre-training phase mitigate this impact. Specifically, We fine-tune the policy network using both the downstream task value function QF and the prior skill value function Q D a, =) F(s, a, 2) The training objective of the policy network during fine-tuning as follows:
((T)) (1)
at, 2) QF(se, de, 2)],
(11)
where z is the skill inferred from minimal demonstrations in the downstream task, is a coefficient. In the early fine-tuning stage, the downstream task value function, trained from scratch, holds min- imal values. In contrast, the prior skill value function maintains stable values, thereby assuming a more sigmificant role and encouraging the policy to preserve effective zero-shot performance. In the later fine-tuning phase, the downstream task value function also exhibits stable values, fostering further improvements in the agent's 's zero-shot performance. Thus, the dual-value fine-tuning scheme ensures stable performance improvements over the zero-shot baseline.
Reward Mapping Technique. After pre-training, the ferent skills for zero-shot generalization across multiple downstream tasks, resulting in task-specific variations in the prior implicit rewards Timpliest = B B" (E,BBT)- 2. The range of prior implicit re- wards relative to downstream task rewards significantly impacts mulli-task fine-tuning performance. Excessively high prior implicit rewards can restrict the policy's 's ability to surpass its zero-shot per- formance, whereas overly low implicit rewards may undermine the zero-shot policy, leading to
representation selects dif-

---

Published as a conference paper at ICLR 2025
relearning process from scratch. To mitigate the impact of the prior implicit reward range fine-tuning performance, we propose a reward mapping technique that balances the importance of rewards by mapping downstream task rewards into the range of prior implicit rewards. During fine- tuning, we dynamically track the means of both prior implicit rewards and downstream task rewards, adjusting the downstream task rewards align with the range of prior implicit rewards follows:
H (Mimppiiit) p((r))
ry
(12)
where ry represents the mapped downstream task reward for fine-tuning, refers to downstream task environmental reward, and μ denotes the dynamically computed mean of the reward. Through the reward mapping, the agent achieves stable performance improvements during fine-tuning.
EXPERIMENTS
In this section, we conduct experiments to address these questions: (1) Does DVFB exhibit zero-shot generalization in online URL.? (2) Can DVFB build upon zero-shot generalization for efficient and stable fine-tuning? (3) What is the impact of various modules generalization performance?
6.1 DMC CONTINUOUS CONTROL TASKS
Environment Setup. Following the latest advancements (Yang et al. 2023
Bai et al. 2024), we
evaluate task generalization performance using 12 downstream tasks across
3 domains URLB
(Laskin et al. 2021) and DeepMind Control Suite (DMC) (Tassa et al. 2018). Details of tasks can be found in Appendix
Baselines. We compare DVFB with several strong baselines for unsupervised task generalization. In zero-shot generaaiztni,,,,nnnn, we evaluate DVFB against (i) successor feature methods, including SF-
CL (Balestriero & LeCun 2022), SF-LRA-SR (Touaii et al. 2023). SF-LRA-P (Ren et al.) (023),
SF-Lap (Wu et al. 2018). and (Touati et al. 2023), and (ii) skill discovery methods, including
CIC [Laskin et al.) 2022). BeCL. (Yang et al.) 2023). ComSI) (Liu et al. (025), and CeSD) (Bai
cct al. 2024). For fine-tuning, we compare DVFB with (i) pure exploration methods, including
RND (Burda et al.] 2019) and Disagreement (Pathak al. 2019). and (ii) skill discovery methods, including SMM (Lee et al. 2019), DIAYN (Eysenbach et al. 2019). APS (Liu & Abbeel 2021a),
CIC, BeCL, ComSD, and CeSD. details and clear classification given in Appendix C.2
Does DVFB exhibit zero-shot generalization in online URL? We evaluate the zero-shot perfor- mance of DVFB across 12 tasks in the DMC. During pre-training, each method trained for million steps without rewards. Subsequently, each agent interactes with environment for 1e steps infer the skills most suitable for downstream tasks. As shown in Table our method exhibits significantly improved zero-shot performance across all tasks and domains. In the challenging Hop- per domain, DVFB exhibits enhanced zero-shot generalization performance, while all other baseline methods struggle to acquire valuable skills. Notably, the zero-shot performance of DVFB remains competitive, even when compared the fine-tuning performance of USD methods in Table The
Table I: DMC Zero-shot Performance
Domain
Task
CL
Lap
LRA-P
LRA-SR
CIC
BcCL
ComSD
CeSD
Ours
Stand
170
337
905
Walk
45 19
118
82 20
53 27
48 +
179 +
29 +
124 + 13
64 64-+34
900 + 53
Walker
Hipp
53
112
56
49
213 +:
27 +
162::
106 - 65
515 ::
Run
34 17
14
11
38
78
28 4:
00
65 65:+:1
423 53
Average
76
204
166
90
114
207
59
161
113
686
Stand
539
301
61
352 1:
335 2016
897 =
109 1:
154 :1:
526 ::
516 2
953 :: 15
153
268
624
Quadruped
Jump
400
218
280
58
251 +: 80
261 + 162
716 58
330 4: 51
115 4: 63
438 +: 56
452 :: 16
816 +: 19
Run
288
154
217
179 d
170 d: 102
450 02
251 1:
77
283 L:
264
467 1: 18
Average
Hop
U-t.D
0=:00
0:::0
2 2:::2
4+::1
01:t:0
1
2:::2
73
Flip
04::0
141
0=
0
1
7
1
1 1::00
1:::1
110 18
Hop backward
125
Hopper
Flip backward
0
1+1
0=00
0 0
8 #
15 3
1
4
4 1::4
#
Average
0
3
101
8

---

Published as a conference paper at ICLR 2025
Table 2: DMC Fine-tune Performance
Disagreement
BeCl.
ConSD
Stand
753 = 87
901 + 19
886 +
789 + 48
702 112.#:00 6-
919 +
962
962 +:22-99
960 +:3
972 =
142
7U2 4-42
547 4-38
21
883
918 52
961
Walker
Flip
335 22
506 1:
500 1
361 1:
488
1.
641
1:
611
630
5d1 1:
927 1:
Run
213 = 32
403 4.
395 a
184 1: 23
170
450 4: 19
387 4.
447 64
337 4: 19
548
Average
454
643
436
364
738
708
729
668
852(+15%)
Stand
512 115
809 +: 25
206 48
718 81
436
700 +: 55
815
824 86
919 +:
963 =
Walk
358 40
154 4:36
385
743
140
Quadruped
Jump
403 86
626 1: 23
167 5:
408 +: 65
360 7:
565 +: 64
727 1:15
080 1: 66
756 +: 14
831 :: 20
Average
4005
800
182
517
353
383
720
6866
787
804()2%)
Hop
4:
Plip
108 26
97 +
29 + 16
7
3 4
96
13 15
61 47
48 +:40
116 18
Hopper
Hop backward
231 60
230 +
29
40
117
Flip buckward
173 1:
203 1
19 1
211
10
154 1
22
59 11 63
74.1.71
189
Average
147
152
21
4
120
20
63
82
163(+7%)
experimental results demonstrate that DVFB exhibits excellent zero-shot generalization capabilities in online URL. Further comparison with offline zero-shot methods is provided in Appendix
Can DVFB build upon zero-shot generalization for efficient and stable fine-tuning? We evaluate the fine-tuning performance of DVFB across 12 tasks with the same pre-training setup as zero-shot generalization. shown in Table 2 DVFB demonstrates superior performance across all three domains compared to current SOTA methods. Specifically, DVFB surpasses baseline methods by 15%, 2%, and 7% the Walker, Quadruped, and Hopper domains, respectively. Unlike previous URL methods, DVFB possesses outstanding zero-shot performance and pre-trained prior knowl- edge. We further explore whether DVFB can utilize these advantages improve performance in downstream tasks fine-tuning. We compare the performance variations of DVFB with 8 SOTA URL methods during fine-tuning on the Walker and Quadruped domains. As shown in Figure 5 DVFB exhibits outstanding initial performance across most environments compared to SOTA meth- ods. Through fine-tuning its performance further improves upon the initial performance with lower variance, demonstrating more stable and efficient fine-tuning compared SOTA methods.
Walker Stand
Walker Walk
Walker Flip
Walker Run
1000
6:00
Epirade Rewards
800
Spicode Rewardss
800
Eploode Rewardss
800
Episede Rewards
800
600
600
3:00
400
400
400
2.0
200
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Timestaps
Timesteps
Timestaps
Timesteps
Quadruped
Quadruped Walk
Quadruped Jump
Quadruped
10000
1000
1000
600
Episade Rowards
803
Epiecde Rowards
800
Eploade Rowards
800
Episode Rewards
5.00
4.00
2.00
400
400
2.00
200
200
200
100
0.00
0.75
1.00
8.00
0.25
1,00
0.25
1,00
0.00
0.25
0,50
1.00
Timestops 14
Timestaps 148
Timestops
1s6
Timestops
46
DVRStours]
Disa groggremrrt
RND
DIAYN
SMM
CIC
BACL
ComSD
CaSD
Figure 5: Finetuning Curve on the Walker and Quadruped Domains. DVFB begins with out- standing zero-shot performance and further enhances performance efficiently through fine-tuning.
6.2 WHLT IS THE IMPACT OF VARIOUS MODULES ON GENERALIZATION PERFORMANCE?
Ablation study on the dual-value pre-training framework. We evaluate the zero-shot generaliza- tion performance across 12 tasks using 5 seeds for pre-training and analyze the Spearman correla- tion and value accuracy of the skill value function to evaluate its capability in learning the successor measure. As shown in Figure (a), DVFB w/o CE represents DVFB with only RND rewards the intrinsic reward. The task names in the figure left are abbreviations of domain and task. Compared to FB's failure in most tasks, both DVFB w/o CE and DVFB achieve zero-shot generalization across

---

Published as a conference paper at ICLR 2025
1000
Value Retrun Accuracy Spearman Cornulation
1.0
DVFB w/o CE
800
0.5
Episods Rewards
600
0.0
Walker walk Walker flip
1.0
400
200
0.5
0.0
W-5
W-W
W-f
Wir
Q-5
Q PW
Q1 Tasks
Qir
Hih
H-f
H-h-b
H-fbb
Wal ker walk Walker flip
Tasks
Figure 6: Results of dual-value pre-training framework ablation study.
all tasks. As illustrated in (b), the absence of the dual-value framework results the skill value function exhibiting poor Spearman correlation and value accuracy with the downstream episode rewards. The experiments show that the proposed dual-value pre-training framework is essential for effectively learning the successor measure and facilitates zero-shot generalization. Compared to DVFB w/o CE, DVFB exhibits improved learning of successor measure and achieves higher episode rewards across all tasks. Thus, the proposed contrastive entropy intrinsic reward encourages explo- ration while ensuring the skill learning capability of FB, further enhancing zero-shot generalization.
Ablation study on the dual-value fine-tuning scheme. To evaluate the contributions of various components within the dual-value fine-tuning scheme, we compare the performance curves of DVFB using various fine-tuning methods across tasks in 3 domains. As shown in Figure 7 DVFB w/o SVF SVF employs a task value function without the prior skill value function for fine-tuning, while DVFB w/o MAP refers DVFB without reward mapping. The figure shows that the performance of DVFB w/o SVF declines during the early stage of fine-tuning. In contrast, DVFB and DVFB w/o MAP maintain their initial performance while achieving further improvements. value fine-tuning scheme ensures stable performance improvements over the zero-shot baseline. DVFB w/o MAP improves slowly because the downstream reward values are lower than those of the implicit reward. In contrast, DVFB rapidly enhances its zero-shot performance through fine- tuning, since the reward mapping technique enables stable and rapid improvements by aligning the range of downstream and implicit rewards. More ablation results are given in Appendix
shows that the dual-
Walker Flip
Walker Run
Quadruped Walk
Hopper Flip Backward
1000
600
1000
Episade Rewards
800
Epioode Rewards
800
Episede Reeards
2.0
800
1.5
600
600
400
Epicode
400
1.00
200
50
0.00
8.00
0.25
0.50
8.00
0.50
Timestops
14
Timesteps
Timesteps
us
Timesteps
DVFB
DVFB w/o SVF
DVFB w/o MAP
Figure 7: Results of dual-value fine-tuning scheme ablation study.
7 CONCLUSION
This paper introduces a general framework for achieving both zero-shot generalization and task adaptation in online URL. We first demonstrate that insufficient exploration in FB restricts the zero- shot generalization capability of the FB mechanism in online URL. To address this issue, we propose a dual-value hewward-backwarr framework (DVFB) that encourages exploration while preserving the learning of successor measures, enabling zero-shot generalization to downstream tasks. Surpris- ingly, within the dual-value scheme, DVFB can further enhance performance with a maaping technique by building upon its zero-shot capabilities. Notably, to the best of our knowledge, the pro- posed DVFB approach the first method to simultaneously achieve zero-shot generalization and fine-tuning capabilities in online URL. The method is evaluated on 12 robot control tasks, demon- strating superior performance in both zero-shot generalization and fine-tuning compared to current SOTA methods. Although DVFB has demonstrated remarkable zero-shot generalization and fine- tuning abilities, this study mainly focuses on the URL problem within state space. Exploring the extension of the nawaad-backward approach to visual URL offers promising direction.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
We extend our gratitude to Fan Yang and Mingshuang Luo for their valuable and insightful discus- sions during the preparation of this paper. This work is supported by the National Key Research and Development Program of China under Grants 2022YFA1004000 the Beijing Natural Science Foundation under No. 4242052, the National Natural Science Foundation of China under Grants 62173325, and the CAS for Grand Challenges under Grants 104GJHZ2022213GG..
REFERENCES
Chenjia Bai, Rushuai Yang, Qiaosheng Zhang, Kang Xu, Yi Chen, Ting Xiao, and Xuelong Li. Constrained ensemble exploration for unsupervised skill discovery. In International Conference on Machine Learning, volume 235, pp. 2418-2442, 2024.
Randall Balestriero and Yann LeCun. Contrastive and non-contrastive self-supervised learning re- cover global and local spectral embedding methods. In Advances Neural Information Process- ing Systems, volume 35, pp. 26671-26600 1-6685,, 2022.
André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. In Advances in Neural Information Processing Systems, volume 30, pp. 4058 4068, 2017.
Yuri Burda, Harrison Edwards. Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pp. 1597-1607, 2020.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing in handful of trials using probabilistic dynamics models. In Advances in Neural Information 2018.
Processing Systems, volume 31, pp.
Peter Dayan. Improving generalization for temporal difference learning: The successor representa- tion. In Neural Computation, volume 5, pp. 613-624, 1993.
Jacob Devlin, Ming-Wei Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pp. 4171-4186, 2019.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without reward function. In International Conference on Learning Representa- tions, 2019.
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. The information geometry of unsupervised reinforcement learning. In International Conference on Learning Representations, 2022.
Kevin Frans, Seohong Park, Pieter Abbeel, and Sergey Levine. Unsupervised zero-shot reinforce- ment learning via functional reward encodings. In International Conference on Machine Learn- ing, volume 235, pp. 13927-13942, 2024.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference artificial intelligence and statistics, volume 9, pp. 297-304, 2010.
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with dis- crete world models. In International Conference Learning Representations, 2021.
Haoran He, Chenjia Bai, Kang Xu, Zhuoran Yang, Weinan Zhang, Dong Wang, Bin Zhao, and Xue- long Li. Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning. volume 36, pp. 64896-64917, 2024.

---

Published as a conference paper at ICLR 2025
Scott Jeen, Tom Bewley, and Jonathan M. Cullen. Zero-shot reinforrmmen learning from low quality data. volume 37, pp.
2024.
Hyunseung Kim, Byung Kun Lee, Hojoon Lee, Dongyoon Hwang, Sejik Park, Kyushik Min, and Jaegul Choo. Learning discover skills through guidance. volume 36, pp. 28226-28254, 2024.
Siming Lan, Rui Zhang, Qi Yi, Jiaming Guo, Shaohui Peng, Yunkai Gao, Fan Wu, Ruizhi Chen, Zidong Du, Xing Hu, et al. Contrastive modules with temporal attention for multi-task reinforce- ment learning. Advances in Neural Information Processing Systems, 36:36507-36523, 2024.
Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. Urlb: Unsupervised reinforcement learning benchmark. Advances in Neural Information Processing Systems, 2021.
Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel. Unsupervised reinforcement learning with contrastive intrinsic control. volume 35, pp. 34478- 34491, 2022.
Lisa Lec, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdi- nov. Efficient exploration via state marginal matching. arXiv preprint 1rXiv:/906.00 2019.
Dong Li, Dongbin Zhao, Qichao Zhang, and Yaran Chen. Reinforcement learning and deep learning based lateral control for autonomous driving. IEEE Computational Intelligence Magazine, 14(2): 83-98, 2019.
Haoran Li, Zhennan Jiang, Yuhui Chen, and Dongbin Zhao. Generalizing consistency policy to visual rl with prioritized proximal experience regularization. Advances in Neural Information Processing Systems (NeurIPS), 2024.
Hao Liu and Pieter Abbeel. Aps: Active pretraining with successor features. In International Conference on Machine Learning, volume 139, pp. 6736-6747, 2021a.
Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training, In Advances in Neural Information Processing Systems, volume 34, pp. 18459-18473, 2021b.
Xin Liu, Yaran Chen, and Dongbin Zhao. Balancing state exploration and skill diversity in unsuper- vised skill discovery. IEEE Transactions Cybernetics, 2025.
Seohong Park, Jongwook Choi, Jaekyeom Kim, Honglak Lee, and Gunhee Kim. Lipschitz- constrained unsupervised skill discovery. In International Conference on Learning Represen- tations, 2022.
Seohong Park, Oleh Rybkin, and Sergey Levine. Metra: Scalable unsupervised with metric-aware abstraction. In International Conference on Learning Representattions, 2024.
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. In International Conference on Machine Learning, volume 97, pp. 5062-5071, 2019.
Tongzheng Ren, Tianjun Zhang, Lisa Lee, Joseph E Gonzalez, Dale Schuurmans, and Bo Dai.
Spectral decomposition representation for reinforcement learning. In International Conference on Learning Representations, 2023.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In International Conference on Machine Learning, volume 37, pp. 1312-1320, 2015.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 58877839):004-609 2020.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. In International Conference on Learning Representations, 2020.
Charles Spearman. The proof and measurement of association between two things. The American
journal of psychology, 100:441-471, 1987.
12

---

Published as a conference paper at ICLR 2025
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In International Conference on Machine Learning, volume 139, pp. 9870-9879, 2021.
Jingbo Sun, Songjun Tu, Qichao Zhang, Ke Chen, and Dongbin Zhao. Salience-invariant consistent policy learning for generalization in visual reinforcement learning. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, 2025.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. Robotica, 17(2): 229-235, 1999.
Zhentao Tang, Yuanheng Zhu, Dongbin Zhao, and Simon M Lucas. Enhanced rolling horizon evolution algorithm with opponent model learning: Results for the fighting game ai competition. IEEE Transactions Games, 15(1):5-15, 2023.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq. et al. Deepmind control suite. arXiv preprint arXiv: 000090, 2018.
Ahmed Touati and Yann Ollivier. Learning one representation to optimize all rewards. In Advances in Neural Information Processing Systems, volume 34, pp. 13-23, 2021.
Ahmed Touati, Jérémy Rapin, and Yann Ollivier. Does zero-shot reinforcement learning exist? In International Conference on Learning Representations, 2023.
Songjun Tu, Jingbo Sun, Qichao Zhang, Yaocheng Zhang, Jia Liu, Ke Chen, and Dongbin Zhao. In-dataset trajectory return regularization for offline preference- based reinforcement learning. In Association for the Advancement of Artificial Intelligence, 2025.
Jingda Wu, Zhiyu Huang, and Chen Lv. Uncertainty-aware model-based reinforcement learning: Methodology and application autonomous driving. IEEE Transactions on Intelligent Vehicles, 8(1):194 203, 2022.
Yifan Wu, George Tucker, and Ofir Nachum. The laplacian in rl: Learning representations with efficient approximations. In International Conference on Learning Representations, 2018.
Rushuai Yang. Chenjia Bai, Hongyi Guo, Siyuan Li, Bin Zhao, Zhen Wang. Peng Liu, and Xuelong Li. Behavior contrastive learning for unsupervised skill discovery. International Conference on Machine Learning. volume 202, pp. 09188339204, 2023.
Yucheng Yang. Tianyi Zhou, Qiang He. Lei Han, Mykola Pechenizkiy, and Meng Fang. Task adap- tation from skills: Information geometry, disentanglement, and new objectives for unsupervised reinforcement learning. In International Conference Learning Representations, 2024.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with pro- totypical representations. In International Conference on Machine Learning, volume 139, pp. 11920-11931. PMLR, 2021.
Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering atari games with limited data. In Advances in Neurul Information Processing Systems, volume 34, pp. 25476 25488, 2021.
Jingwei Zhang, Jost Tobias Springenberg, Joschka Boedecker, and Wolfram Burgard. Deep rein- forcement learning with successor features for navigation across similar environments. In 20/7 JEEF/RSJ International Conference on Intelligent Robots and Systems, pp. 2371-2378. IEEE, 2017.
13

---

Published as a conference paper at ICLR 2025
A
PSEUDO-CODE
We provide the complete pseudocode for DVFB, with the unsupervised pre-training phase described Algorithm D and the downstream task fine-tuning phase described in Algorithm
A.1 PRE-TRAIN PSEUDO-CODE
Algorithm DVFB Algorithm: Unsupervised Pre-training Phase
1: Inputs: replay buffer D, randomly initialized networks Fs. B., RND representation network fR. transition representation network ie., skill representation network (çş, exploration value function Qr... actor network मÇ learning rate 1), mini- size b, number of episodes E, number of gradient updates N, skill update period temperature т, and regularization coeffi- cient X.
2: for m 1 do
3: 1* Collect E episodes
4: eor =
1 to E
5: Sample p(2)
6:
Observe an initial state 80
7:
for t to 7 do
8:
if mod T = then
9:
Sample z p(z)
10: end if
Select action a, = ( ((,t,,,,)) d2,
12: Observe next state 81+1
Store transition (8,, (t, 8(+1) in the replay buffer end for
14:
15:
end for
16:
1* Perform N stochastic gradient descent updates
17:
for = to do
18:
Sample mini-batch of transitions (ss, as 8441 2) her D of size ||| = b.
19:
Sample mini-batch of target state-action pairs {(8, a C D of size |/| =
20:
Sample mini-batch of {zi p(=) of size |7| b.
21:
Compute RND loss function L(k) (1fx(8++1) [(s++1)1B..
22:
Update k + K L(x).
23:
Compute contrastive loss function L(v,)) g) with equation 7
24:
Update v + v. ((,,),
Compute intrinsic reward Tiner with equation
26:
Compute exploration value loss function T.(a) with equation
Update +- Ø 7TVL(())
28:
Compute skill value loss function L(0) with equation
29:
Compute regularization loss Lreg (0) = Enjer B (s,, az).
30:
Update 0 +- 0 a VVV L(0).
31:
Compute actor loss function L(C) with equation 5
32:
Update Ç + - T L().
end for
34:
/* Update target network parameters */
A 70 + (1 7)0-
36:
--
+ Tw + T)W
37:
K x TK
TK + (1 T)K-
38:
end for=0
14

---

Published as a conference paper at ICLR 2025
A.2 FINE-TUNE PSEUDO-CODE
Algorithm DVFB Algorithm: Downstream Fine-tuning Phase
1: Inputs: replay buffer D, initialized networks Fs. Bu, actor network सς. downstream task value function FF learning rate 7,, mini-bate size b, number of episodes E, number of gradient updates N, number of inference step M.
2: 1* Inference skill
3: for t to M do
4: Sample p(z)
5: Select action a₃ (st, 0t 2)
6: Observe next state 81+1
7: Store transition (st, ae S141, r) in the replay buffer D
8: end for
9: Inference skill: B(ss)r.
/* Fine-tune
11: for m = do
12: /* Collect E episodes
13: for = to E do
14:
Observe an initial state 80
15:
for =
1 to 7 do Select action de = (8t, dz2 2)
16:
Observe next state 8
18:
Store transition (s;, az Sc+1) in the replay buffer D end for
20:
end for
1* Perform N stochastic gradient descent updates
22:
for n = 1 to N do
23:
Sample mini-batch of transitions {(s, as, 8141 1141,=))ee =)]er D size / b.
24:
Compute downstream task value loss function L(u) with equation 10
25:
Update μ + م L(p).
26:
Compute actor loss function L({) with equation I1
27:
Update Ç + Ç - V (CC.
28:
end for
29:
end for=0
15

---

Published as a conference paper at ICLR 2025
B EXTENDED BACKGROUND
In the presence of extrinsic rewards, Reinforcement Learning (RL) has demonstrated its efficacy in learning powerful task-specific skills (Li et al. 2019; Wu et al 2022 Tu al. 2025 Sun al. 2025). The success of unsupervised learning in computer vision () (Chen et al. 2020 and natural language processing (NLP) (Devlin et al., 2019) inspires researchers and benefits task task-specific RL with complex visual imput by enhancing representation learning in a target manner. Nevertheless, the task-specific supervision of extrinsic reward makes it difficult for the agents who have been trained with significant amount of effort to generalize their knowledge to novel tasks (Stooke et al. 2021 In order to enhance this generalization, unsupervised RI. is proposed, which the task-agnostic re- ward is specifically designed for unsupervised pre-training. The pre-trained feature encoders (Liu & Abbeel 2021b; Yarats et al. 202T) and exploration policies (Burda et al.) 2019; Yarats et al. 2021) be subsequently implemented to facilitate efficient RL on variety of downstream tasks. In ad- dition to the generalization, intelligent agents should also be capable exploring environments and acquiring variety of useful behaviors without any extrinsic supervision, similar human beings. For the reasons above, unsupervised skill discovery (Eysenbach et al. 2019 Laskin et al. [002] Lee al. 2019) is proposed and becomes novel research hotspot. As branch of unsupervised RL, unsupervised skill discovery approaches also design task-agnostic intrinsic rewards and achieve pre-training with these rewards through RL, which guarantees multi-task downstream generalization
(Laskin et al. 2022 Liu & Abbeel 2021a Liu et al. 2025 Yang et al. [023 Bai et al. 2024). The
primary distinction is that skill discovery necessitates additional input in the form skill vectors, which are conditions. Their objective is to identify task-agnostic policies that are distinguishable by skill vectors (Park et al. 2024, Liu et al. 2025 Lee al., 2019). These methodologies demonstrate promising outcomes in variety of disciplines, including manipulation, video games (Schrittwieser ct al. 2020; Ye et al. 2021). D). robot locomotion (Laskin et al. 202I; Tassa et al., 2018), and so on. However, observe that existed unsupervised skill discovery methods always lack the ability of zero-shot policy learning, i.e., their learned skills can't directly achieve multi-task adaptation with- out further task-specific adjustment.
Zero-shot RL methods are typically based on successor representations (Dayan 1993), universal
value function approximators (Schaul et al. 2015), successor features (Barreto et al., 2017). and successor measures. The state-of-the-art methods achieve these ideas either by using universal suc- cessor features (USFs) (Schaul al.) 2015) or forward- bawkkrd representations (FB) (Touati & Ollivier 2021 Touati et al. 2023). Methods based on USFs require learning representation for suc- cessor features. Previous approaches have utilized techniques such as Laplacian Figenfunctions (Wu et al. 2018). Low-Rank Approximation of Transition Probabilities (Ren et al. 2023). Contrastive Learning (Balestriero & LeCun (022), and Low-Rank Approximation
Successor Representa-
tions (Touati et al.) 2023) to learn representations. FB methods directly learn successor measures using forward and backward representations, avoiding the need to learn representations. However, these successor ceature-based methods rely diverse offline datasets, and in online URL, the lack of exploration capabilities results in poor data diversity, thereby affecting the learning of successor measures and zero-shot generalization. This paper proposes a dual-value forward-backwar sentation framework (DVFB) that enables zero-shot generalization in online URL by encouraging exploration while ensuring the learning of successor features.
repre-
16

---

Published as a conference paper at ICLR 2025
C EXPERIMENTAL SETUP
C.I ENVIRONMENTS
Following recent advancements (Liu et 2025 Bai al. 2024), we evaluate task generalization performance on 12 downstream tasks across three domains: Walker, Quadruped, and Jaco, using the URLB (Laskin et al. 2021) and DMC (Tssaa et al. 2018) benchmarks. Here are the detailed domains and corresponding tasks:
The Walker domain includes walker control tasks with a state space S IR24 and action space A E RR This domain includes four tasks: Walker stand, Walker walk, Walker flip, and Walker run.
The Quadruped domain involves quadruped control tasks with a higher-dimensional state space S E R and action space A R It includes the tasks of Quadruped stand, Quadruped walk, Quadruped jump, and Quadruped run.
The Hopper domain consists of one-legged hopper control tasks, which present more challenging exploration of rewards, with a state space E R and action space E R It includes the tasks of Hopper hop, Hopper slip, Hopper hop buckward, and Hopper flip backward.
In the experiments, each method is pre- mre-trained in the aforementioned environments without rewards for 2e steps and adapts to downstream tasks through zero-shot for generalization and fine-tuning.
Walker
Quadruped
Hopper
stand
walk
stand
walk
hop
flip
flip
run
jump
hop backward flip backward
Figure 8: Illustration of domains and downstream tasks in DMC, Each domain has four downstream tasks.
C.2 DETAILED BASELINES
In zero-shot generalization, compared DVFB with (i) successor feature methods that per- form well offline settings and (ii) skill discovery methods, including CIC(Laskin et al. 2022),
BeCI.(Yang al. 2023), ComSD et al. 2025), and CeSD(Bai et al. 2024). In successor feature methods, we compared approaches based on contrastive learning (ClstBalstrriero & LeCun 2022 Low-Rank Approximation of successor representations (LRA-SR)(Toalle et al.] (023). Low-Rank Approximation of state transition probabilities (LRA-P)(Ren et al. (202)), and Laplacian Eigen- functions (Lap) Wu al., 2018), along with the FB methodyToual et al., 2023). The successor representation methods learn representations in various ways to compute successor features for gen- cralization, while the FB method achieves generalization by learning successor metrics. The detailed baseline methods with successor feature methods are presented below:
Successor Feature with Laplacian Eigenfunctions (Lap): Lap focuses on learning the eigen- functions of the symmetrized MDP graph Laplacian, which is defined through ploratory policy. seeks to minimize the difference between feature representations of consecutive states while ensuring that the features constitute orthonormal basis. Lap
17

---

Published as a conference paper at ICLR 2025
encourages feature clustering among states and their neighbors while effectively separating distant states.
Low-Rank Approximation of Transition Probabilities (LRA-P): LRA-P approximates the transition probability densities using low-rank models. It learns state-action features by minimizing contrastive loss that compares samples of consecutive states with independent samples drawn from a stationary distribution. The method produces optimal policies if the model perfectly matches the transition probabilities.
Contrastive Learning (CL): CL uses SimCLR-like objective to learn representations by contrasting positive pairs with negative pairs.CL necessitates full trajectory data and is closely related to the spectral decomposition of the successor measure associated with the behavior policy.
Low-Rank Approximation of Successor Representations (LRA-SR): LRA-SR builds on CL but reduces variance by learning successor measures via temporal difference learning, rather than through Monte Carlo sampling- It normalizes feature representations to form identity matrix, ensuring low-rank approximation of the successor measure.
Forward backward Representations (FB): FB learns successor measure with forward rep- resentation and backward representation. It avoids reliance on features, demonstrating out- standing performance in successor feature methods.
In the fine-tuning task adaptation, we compared DVFB with (i) pure exploration methods, including RND(Burda et al. 22099) and Disagreemeen(Pathak al. 2019), and (ii) skill discovery methods, including SMM(Lee et al. 2019), DJAYY(EEyeenbace et al. 2019). APS(Liu & Abbeel, 2021a), CiC(Laskin et al., 2022). BeCL Yang et al. 2023), ComSD et al. 2025), and CeSD( Bai et al, The detailed baseline methods in task adaptation are as follows:
2024
Disagreement: Disagreement enhances exploration by training ensemble of dynamics models, encouraging the agent to maximize the disagreement between them.
RND: RND is pure exploration method that encourages the agent to explore unseen states by using randomly generated network.
DIAYN: DIAYN maximizes the mutual information (MI) between skills and states by em- ploying discrete uniform prior to enhance skill entropy. trainable discriminator is used to estimate state-conditioned entropy, allowing for the computation of intrinsic rewards.
SMM: SMM aligns the state marginal distribution with a target distribution by formulating a zero-sum game between state density model and parametric policy, explicitly maxi- mizing state entropy through intrinsic rewards.
APS: APS employs another MI decomposition for improved mutual information estimation and utilizes particle-basee entropy estimation to enhance exploration.
CIC: CIC is the first incorporate contrastive learning into skill discovery, maximizing
implicit skill-conditioned entropy by comparing state transitions with skill vectors.
BeCL: introduces novel MI objective to mitigate exploitation issues in CIC, providing theoretical upper bounds.
ComSD:
ComSD employs a contrastive learning-basee diversity reward to help agents identify existing skills, combined with particle-based exploration reward to facilitate the discovery of new behaviors.
CeSD: CeSD introduces URL framework that uses an ensemble of skills for partitioned exploration based on state prototypes, enabling local exploration within clusters while max- imizing overall state coverage.
While URL methods enable rapid adaptation to downstream tasks through fine-tuning, achieving zero-shot generalization remains difficult due to the absence of direct link between the learned skills or policies and the downstream tasks. We further demonstrate the properties of baseline method in Table
18

---

Published as a conference paper at ICLR 2025
Table Properties of Baseline Methods
Method
Zero-shot
Online
Offline
Exploration
Skill Discovery
Successor Representation Methods
CL
X
Lap
X
LRA-P
X
X
X
LRA-SR
FB
X
X
Pure Exploration Methods
Disagreement
X
RND
X
X
Unsupervised Skill Learning Methods
DIAYN
X
SMM
x
APS
X
CIC
X
X
BeCL
V
X
ComSD
X
V
X
CeSD
X
DVFB(ours)
X
C.3 DETAILED ZERO-SHOT GENRALIZATION SETUP
We evaluated the zero-shot performance of DVFB across 12 continuous control tasks in the Deep- Mind Control Suite. During the pre-training phase, each method trained for 2 million steps using self-generated data without rewards. Subsequently, each agent interacted for 10,000 steps in the downstream tasks to collect demonstration data, allowing them to acquire the skills most suitable for those tasks. FB method infers skills through switching skills in each episode and selects the skill with the highest episode reward as the down- stream task skill. During testing, the chosen skills are fixed evaluate the zero-shot performance of different methods in downstream tasks. Each method was run with 6 random seeds to assess its performance.
methods, skills 2, are inferred using the formula r(s) = (x))Tz, while the B(s')r The USD methods acquire demonstration data by
During the adaptation to downstream tasks, DVFB and USD methods identify the skills that best match these tasks through skill inference and the selection of the highest-performit tively. During the fine-tuning phase, USD methods fix the skill vector and trains a new value function using the rewards from downstream tasks, subsequently fine-tuning the skill policy with this updated value function.
skills, respec-
19

---

Published as a conference paper at ICLR 2025
D HYPER-PARAMETER SETTINGS
In this section, we provide the detailed hyper-parameter settings of our proposed method, as shown Table
Table 4: Hyper-parameter settings.
Hyper-parameter
Setting
Pre-training frames
2e
Finetuning frames
1e5
Zero-shot selection frames
4
RL replay buffer size
le6
Frame stack
1
Action repeat
1
Seed frames
4000
vector dimensions
50
vector space
continuous
update frequency
300
RL backbone algorithm
DDPG
Return discount
0.99
Discounted steps return
Batch size
1024
Optimizer
Adam
Learning
le
Actor network (MLP)
dim(s) + 64 + 1024
+ 1024 dim(a)
Actor activation
layernorm CannnoormTannni (Tanh) + ReLU
+ Tanh
Critic network (MLP)
dim(s) + dim (a)
+ 1024 + 1024 +
Actor activation
layernorm (Tanh) ReLU
Agent update frequency Target critic network EMA Exploration stddev clip Exploration stddev value coefficient coefficient 3
2
0.01
0.3
0.2
5
0.5
coefficient 7)
0.5
20

---

Published as a conference paper at ICLR 2025
E FULL EXPERIMENTS RESULT
As shown in Figure 9 we present the fine-tuning curves of our method compared to eight baselines all twelve tasks in three domains. Our method demonstrates strong initial performance and achieves further improvements through fine-tuning with greater efficiency.
Walker Stand
Walker Walk
Walker Flip
Walker Run
Epizade Rewards
800
Epiedde Rowards
800
Epizado Rowards
800
Eppede Newards
5.00
600
600
600
3.00
400
2.0
0
0.50
0.75
2,00
0.00
0.25
0.50
2,00
0.00
0.25
1,00
0.00
0.25
0.50
1.00
Timesteps
18
Timesteps
148
Timesteps
1nt
Timesteps
Quadruped Stand
Quadruped Walk
Quadruped Jump
Quadruped Run
1000
1000
1000
6.00
Epsode Rewardss
Eptode Rewards
Episodo Rewards
Bptode Rewards
5.00
600
600
600
8.00
400
400
400
100
8.00
0.25
0.50
0.75
1.00
8.00
0.25
0.50
0.75
1.00
8.
0.25
0.50
0.75
1.00
8.00
0.25
0.50
0,75
1.00
Timestops
Timesteps
Timestops
limesteps
Hopper Hop
Hopper Flip
Hopper Hop Backward
Hopper Flip Backward
125
125
400
Epocdd Rewards
100
Epicode Rewards
Epicade Rewards
300
Epirede Rewards
1.50
50
50
200
1000
25
25
100
8.00
0.50 Timesteps
0.75
8.00
Timesteps
Timesteps
Timesteps
DVFB
Disagremment
RND
DWANN
SHN
QC
BeCL
ComSD
CuSD
Figure 9: Finetuning Curve on the Walker and Quadruped Domains. DVFB begins with out- standing zero-shot performance and further enhances performance efficiently through fine-tuning.
21

---

Published as a conference paper at ICLR 2025
F FULL ABLATION STUDY
Figure TO presents the ablation study results of DVFB's fine-tuning technique across 12 tasks in three domains. The results demonstrate that the proposed dual-value fine-tuning technique ensures efficient and stable fine-tuning.
Walker Stand
Walker Walk
Walker Flip
Walker Run
Epizade Rewards
800
Epeedd Rowards
800
Epizado Rowards
800
Eppede Newards
5.00
600
600
600
1.00
400
2.0
0
0.50
0.75
2,00
0.00
0.25
0.50
3,00
0.00
0.25
0.75
1,00
0.00
0.25
0.50
1,00
limesteps
14
Timesteps
148
Timesteps
1nt
Timesteps
Quadruped Stand
Quadruped Walk
Quadruped Jump
Quadruped Run
1000
1000
6.00
Epsode Rewards
Eptode Rewards
Episode Rewards
Eptsode Rewards
5.00
600
600
600
8:0
400
400
400
100
8.0
0.25
0.50
0.75
1.00
8.00
0.25
0.50
0.75
1.00
0.
0.25
0.50
0.75
1.00
8.00
0.25
0,75 limesteps
0.50
1.00
Timestops
Timesteps
Timestops
Hopper Hop
Hopper Flip
Hopper Hop Backward
Hopper Flip Backward
125
125
400
Epocdd Rewards
100
Epicode Rewards
Epicdd Rewards
Epirede Rewards
1.50
50
50
200
1000
25
25
100
8.00
0.50
0.75
Timesteps
Timesteps
Timesteps
Timesteps
DVFB
DVFB w/o SVF
DVFB w/o MAP
Figure 10: Ablation study results for the dual-value fine-tuning scheme. We present the fine- tuning curves for DVFB, DVFB w/o SVF, and DVFB w/o MAP across all 12 downstream tasks. DVFB w/o SVF denotes DVFB fine-tuning with downstream task value function trained from scratch, while DVFB w/o MAP refers to DVFB fine-tuning without the reward mapping technique in the dual-value scheme.
22

---

Published as a conference paper at ICLR 2025
G COMPARISON WITH OFFLINE ZERO-SHOT METHODS
In this section, we compare DVFB against the offlin zero-shot methods LAP, LRA-SR, LRA-P and FB in Walker, Quadruped and Cheetah domains. The performances of offline methods are from (Touati et al., 2023), where offtine datasets are collected by APS, Proto, and RND exploration methods. The results, summarized in Table 2, highlight the following key differences between offline and online methods:
High Data Sensitivity for Offline Methods: Offline methods exhibit significant performance vari- ation depending on the quality of exploration data. On the one hand, the same algorithm requires different exploration datasets different domains. For example, FB trained with Proto data in the Walker domain achieves best performance (666), while in the Quadruped domain using Proto data yields a huge performance drop (222). On the other hand, different algorithms require different exploration datasets. For example, LRA-P performs best with RND data, while FB performs best with APS data. When designing a novel algorithm, how to make sure what kind of exploration dataset is most suitable? The intuitive idea is to train the models different exploration datasets, and compare to find the best performance models. Obviously, it will lead to high computational costs. In contrast, DVFB does not depend offline datasets, and requires only single agent pre- training phase to achieve strong zero-shot capability, significantly reducing time and computational overhead. It is simpler and easier to deploy than offline zero-shot methods.
Offline methods rely on the diversity and quality of fixed pre-collected datasets, which limits their generalization performance. In con- trast, DVFB balances the exploration and exploitation online with an intrinsic reward based on con- trastive learning, leading to enhanced skill learning and better zero-shot performance. Experimental results across twelve tasks in Mujoco domains demonstrate that DVFB consistently outperforms zero-shot offline methods.
Performance Limitation on Pre-collected Fixed Dataset:
In summary. the results demonstrate that DVFB offers superior performance and efficiency com- pared to both offline and online methods, establishing its significance zero-shot online URL.
Table 5: Zero-shot Generalization Performance Comparison
Doman
task
LAP
LRA-PT
LRA-SR
Offline FH
DVPR
[APS]
(Proto)
(RND)
(APS)
(Proto)
(RND)
(APS)
(Proto)
(RND)
(APS)
(Prot0)
(RND)
Stand
895
937
853
643
657
591
874
$28
822
890
905
Walker
Flip
569
340
512
551
454
413
Run
289
280
209
115
183
325
254
301
350
346
136
423
Average
506
662
582
314
363
640
413
671
621
600
666
654
686
552
924
712
Qundruped
Jump
177
490
309
184
147
134
731
649
183
Run
491
125
399
238
166
301
448
113
461
476
137
429
467
Average
674
167
505
318
197
403
604
140
663
690
233
606
715
276
rum buckward
221
133
261
230
83
238
185
Cheetah
walk
900
722
330
75
770
29
918
860
446
844
844
827
walk backward
937
798
499
8
629
14
98.3
979
053
98.
981
793
978
564
452
242
23
430
13
570
235
585
583
513
619
evegge
513
Averago best
487
615
647
Results marked with are sourced from FB(CCR 23).
23

---

Published as a conference paper at ICLR 2025
H THEORETICAL ANALYSIS FOR IMPROVEMENTS OF DVFB
In this section, we offer a thcoretical guarantee for zero-shot generalization and a detailed analysis of how the DVEB framework improves FB's zero-shot generalization online URL.
Theorem 1. (Touati et al. 2023) Let F: S A Z approximation functions, and define policy T. (8) = arg maXa F(s, a, z) for each 2 E Z.
and S A Z be successor state
Given positive probability distribution p on S A, let m² be the density of successor state measure M under policy with respect to p. Define the model estimates:
nh" (8, a, aaa,s',aa) ,,) a() B(d, ,a) MM a, ds', da') = fn inn
(s, a, s', d d')p(d'', da')
For any bounded reward function r function value function of policy T2, and 2g = [(s.a)-pl((s, B(s, a)}. Then:
x A R. let V* be the optimal value function, v be the
1. Under bounded density estimation enror: m= (8, a, a')l S for all (s,a) E A, then
If (o
3//1r|1oo
1-7 - - S
2. For Lipschitz continuous rewards: If |f| (s, :)- Ma (8, |kn S e e all (8,a) E
S x A, then
3s max (|oo ||
IV
Vo S
1-7
3. For general norm pairs: Given norms I1A on functions and [1-IIB on measures satisfying
f fdp S [lf [||||||BBll for any reward function with ||l||a 00,
3||r||A sup (s, -M
-| V I
s.a
Furthermore, the Q-function approximation satisfies:
2||l|A
sup sup|F(s,a,2R) (s, <
sup (s,aRs, M as,
8.0
a.a
The Theorem D establishes that for any reward function the error in value estimation is bounded as follows:
sup = - Q 3/r|A sup JEzN (s, (BB
(13)
SxAA
SxA
where Esp (s, a. = M (s, a, ds', da') - F(s, a, between the true successor measure and the estimated successor measure. This bound ensures that the zero-shot generalization error depends on the accuracy of the successor measure estimation.
B(s', p(ds'' da') represents the difference
Theoretical Analysis for Improvements of DVFB. In online URL, the performance of the -awkardd (FB) method is hindered by insufficient exploration, leading to inaccurate es- timation of successor measures. Our empirical analysis in Section 4 highlights this limitation by showing the poor correlation between the value function and returns on downstream tasks under FB. To address this, we propose the Dual-Value Forward-Backs warkwardd (DVFB) framework, which in- troduces a dual-value structure to enhance exploration and improve successor measure estimation. Specifically. let DVFB FB and FB represent the successor measure errors under DVFB and FB, spectively. DVFB reduces these errors, such that:
1
DV DVFB
B
< 1
FB B.
24

---

Published as a conference paper at ICLR 2025
As result, for any unseen reward function Inews the zero-shot generalization error of DVFB is bounded by:
3/1 3/r A
n/wwew A
TDVIB
Tew Prann
DVFB B
B
1 y
This tighter bound demonstrates that DVFB achieves superior zero-shot generalization by reducing successor measure estimation errors via enhanced exploration.
Our experimental results validate this theoretical analysis, with the DVFB framework consistently outperforming FB across range of unseen tasks. These findings confirm that DVFB offers improved zero-shot generalization in online URL..
25

---

Published as a conference paper at ICLR 2025
I FURTHER EXPERIMENTAL RESULTS ON NAVIGATION AND ROBOTIC MANIPULATION DOMAIN
distinct domains: Point-Mass Maze navigation environment and a Meta- Meta-World robotic manipulation environment. Although the offline methods FB-offline
have conducted additional experiments
al.) 2023) and MCFB-offline [2] rely offline data and offline settings (in contrast to online URL, which requires multi-stage training and more expensive computation), we provide a comparison with these offline reinforcement learning methods for a more comprehensive evaluation of our approach. As they didn't tt do the Meta- World experiment, we summarize the results of FB-offline and MCFB-offline with RND offline data in Point-Mass Maze domain. As shown in Table 6 DVFB demonstrates better performance across both domains. The additional results demonstrate that DVFB is not only effective robotic control tasks but also generalizes well to other domains, such navigation and robotic manipulation.
Table 6: Performance comparison across different domains. For Point-Mass Maze, results show mean + standard deviation across three seeds. For Meta- World, results show success rates.
Domain
Task
FB
CIC
CcSD
FB-offline*
MCFB-oflline*
DVFB
Reach Tef-left
69 +
6
12 1
612
773
932 + 10
Reach Top-right
77 95
5
5
270
203 81
Point-Mass
Reach Bottom-lcft
3 3
7
18 21
268
1
94 94445
Reach Bottom-right
0
2
2
0
0
4
Average
37.3
8.0
9.3
219
261
308.3
Faucet Open
0.18
0.04
0.00
0.60
Meta- World
Faucet Close
0.10
0.52
Results marked with * sourced from MCFB
2024
26

---

Published as a conference paper at ICLR 2025
J IMPORTANCE OF CONTRASTIVE ENTROPY REWARD
In this section, we discuss the importance of the contrastive entropy reward. The contrastive en- tropy reward is designed promote skill discrimination during exploration, thereby preserving the skills learned through the FB mechanism. To assess the role of the contrastive entropy reward, we conducted two of comprehensive experiments.
Ablation Study. We perform an ablation study on the coefficient 3 of the contrastive entropy reward, as shown in Table The results demonstrate that increasing / from 0.1 0.7 consistently improves performance, validating that the contrastive entropy reward enhances generalization by promoting skill separability.
Table Ablation study on contrastive entropy coefficient 8 on Walker tasks. Results show mean standard deviation across three seeds.
Task 3=0.1
3=0.3
$=0.5
3=0.7
3=0.9
Stand
819+32
862+9
905
898#62
91949
Walk
819+38
861+18
900
926*17
873432
Flip 428+10
501+18
515
616+129
453+30
Run
344+28
397+40
423
434+54
342+35
Average
603
655
686
719
647
Comparison Experiment. We compare DVFB with variants that use alternative intrinsic rewards (ICM-APT, Proto, and CIC), shown in Table
Table 8: Comparison with alternative intrinsic rewards on Walker tasks. standard deviation across three seeds.
Results show mean +
Task
DVFB(ICM-PPT)
DVFB(Proto)
DVFB(CIC)
DVFB
Stand
883#106
844#101
846#74
905#27
Walk
840+85
821427
825+24
900+53
Flip
436168
454451
436+140
515:67
Run
354#15
358+17
342+14
423+53
Average
628
619
612
686
The results reveal several key insights: (1) The scalability of DVFB. All variants achieve reasonable zero-shot generalization performance, demonstrating DVFB's compatibility with different intrinsic rewards. (2) The advantage of the CE reward. DVFB with contrastive entropy consistently out- performs the other variants, achieving the highest average performance.
These experiments provide strong evidence for the effectiveness of our contrastive entropy design and demonstrate DVFB's 's flexibility in incorporating different intrinsic rewards.
27

---

Published as a conference paper at ICLR 2025
K IMPORTANCE OF REWARD MAPPING TECHNIQUE
In this section, we provide a detailed justification and explore potential alternatives for reward map- ping technique during our experiments.
Reward Mapping Implementation. Compute the implicit reward using backward network (backward_ned and a latent variable (z). (2) Normalize both implicit and extrinsic rewards using running mean and standard deviation track- crs. (3) Rescale the extrinsic reward to align with the scale of the implicit reward.
The reward mapping process involves three key steps: (1)
The detailed implementation is shown in the following pseudocode:
Algorithm 3 Reward Mapping Mechanism
1: Input: downstream reward To next state Snexts backward representation network Bnet. skill 2, implicit reward normalization function rmb- downstream reward normalization function Tt-ms.
1: procedure REWARD MAPPING
2: 1* Compute features from backward network
2: B < (8nett
3: 1* Compute Covariance matrix
3: Σ + T B
3: Σ- + inverse(E)
4: /* Compute Implicit reward
4: Timplicit + >Ee
5: 1* Update normalization functions
5: (Pib; (
5: (M4, (m),0²) x- t_rmsuuddate(()
6: 1* Rescale reward
6: Trescaled Ti ArtE
6: return Frescaled
6: end procedure=0
Potential Fine-tuning Techniques. The goal fine-tuning to ensure stable policy improvements, guided by both the prior skill value QM and the downstream task value Qp. The reward mapping technique is designed to directly use the downstream task rewards for the task value, adjusting the coefficient 7) in Eq- 11 to balance the importance of Q and Qr- We refer to this as DVFB w/o MAP, which employs dual-value fine-tuning scheme based downstream task rewards rather than ry- Additionally, we explore an adaptive approach where the parameter is dynamically adjusted as r Q which we call DVFB w/o MAP adaptive.
balance the influence of these two values. A straightforward alternative is
Table 9: Performance comparison of different fine-tuning approaches on the quadruped domain. Results show mean over three runs.
DVFB w/o MAP
Task
DVFB
1) 0.02
= 0.1
0.5
adaptive 17
Stand
954 +
951 10
961
960 +
965 7
Walk
753 - 32
765 + 18
752 + 31
820 + 77
908 + 21
Jump
819 = 28
811 + 16
784 59
830 + 12
831 20
Run
496 + 23
491 +
490 + 14
496 + 48
536 + 27
Average
756
755
747
777
804
Experimental Results. We perform comparative experiments in the quadruped domain, with
sults presented in Figure [] and Table 9 Our findings show that DVFB w/o MAP provides stable but limited improvements with higher coefficients, while performance becomes unstable with lower coefficients, ultimately limiting overall performance. Moreover, DVFB w/o MAP adaptive strug- gles to achieve superior improvements due to the nonlinear relationship between skill value and
28

---

Published as a conference paper at ICLR 2025
downstream value. In contrast, DVFB with the reward mapping scheme, which uses all coefficients, consistently delivers stable and superior improvements across all tasks. These results strongly vali- date the effectiveness of our chosen approach.
Quadruped Stand
Quadruped Walk
Quadruped Jump
Quadruped Run
1000
1000
1000
600
Episode Nowards
800
Epicode Rowards
800
Eploode Rowards
800
Epicode Rewards
5.00
400
400
400
400
200
200
200
100
0.00
0.75
0.25
1,00
0
1,00
0.00
0.25
1.00
Timestaps
Timestaps
Timestaps
146
Timestaps
Figure 11: Finetuning Curve on the Quadruped Domains for DVFB with different fine-tuning approaches.
29

---

Published as a conference paper at ICLR 2025
L THE SENSITIVITY OF DVFB TO HYPERPARAMETERS
In this section, we conduct a series of ablation studies to evaluate the impact of the key hyperparam- a, 3 and 7) on DVFB's performance. Below, we present the experimental results.
Ablation Study on a. Table [0] shows the results for varying in Walker domain. The experiments indicate that while changes of DVFB remains stable, with the best results observed at a = 5.
affect performance slightly, the overall generalization performance
Table 10:
Performance of DVFB with different a values on the Walker domain. Results show mean standard deviation across three seeds.
Task
1 911
3
= 5
Q= 7
= 9
Stand
912
905 27
888 5
807 + 33
Walk
835 = 71
895 + 41
900 5:
862 7
707 + 49
Flip
464 = 76
522 + 92
515 5
489 + 18
423 + 19
Run
350 = 444 +
423 + 53
345 + 5
266 + 45
Average
640
693
686
646
551
Ablation Study on 3. Similarly, Table 11 reports the performance for different 3 values in Walker domain. The results reveal that DVFB achieves optimal performance at 3 = 0.7, with only minor deviations observed across other values. This demonstrates the robustness of our framework to changes in $3
Table 11: Performance of DVFB with different 3 values on the Walker domain. Results show mean standard deviation across three seeds.
Task
3= 0.1
3 0.3 0.5
3= 0.7
3= 0.9
Stand
819 + 32
862 + 9
905
898 + 62
919 9
Walk
819 38
+
900
926
873 + 32
Flip
428 + 10
501 + 18
675
616 129
453 + 30
Run
344 + 28
397 + 40
423 + 53
434
342 + 35
Average
603
655
686
719
647
Ablation Study 1). We conduct ablation study in Quadruped domain evaluate the sensitivity of the DVFB framework to the hyperparameter during the fine-tuning phase. As shown in Table 12 the overall average performance across all tasks remains consistent, indicating that DVFB is relatively resilient to changes in y).
Table 12: Performance of DVFB with different 7) values on the Quadruped domain. Results show mean t standard deviation across three seeds.
Task
== 0.02
1 0.1.1
0.1
0.5.5
0.5
1= 1.0
Stand
957-4
96416
96517
954=10
Walk
8911:32
908+30
908121
88618
Jump
830+:18
838+111
831+20
835+8
Run
557+11
530+=15
536+227
543-26
Average 809
810
810
804
The results of our sensitivity analysis demonstrate that DVFB is resilient to variations in key hyper- parameters a, 3 and 17. While some fluctuations performance occur, the overall generalization ability remains largely unaffected, indicating the robustness of our framework across different pa- rameter settings. Furthermore, for all other neural network hyperparameters (e.g., learning rate), we adopt the default settings of URL with DDPG, ensuring consistency with prior work.
30


Reviews:
[{"review_id": "AjZ843XD3P", "reviewer": "Reviewer_U97w", "summary": "This work presented a pre-training framework for zero-shot reinforcement learning by leveraging forward-backward (FB) representation. Unlike some previous study on zero-shot RL, this work analysed the performance gap of FB in the online learning setting compared with a specific exploration strategy. The authors then proposed a new exploration reward based on contrastive learning and incorporated this into FB traning by end-to-end online learning. The proposed method is evaluated in zero-shot online URL and fine tuning settings. Experimental results suggest that it achieved improved performance than some baseline methods given limited interactions.", "questions": "Please see the weaknesses part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This work is well motivated and the promotion of exploratory behaviour during the pre-training phase to increase the data coverage is reasonable.\n\nThe paper is well written and easy to follow.", "weaknesses": "The major contribution in this work is the combination of an exploration reward with FB learning, where the technical novelty is limited.\n\nAlthough the performance gain shown in Table 1 looks strong, I have concern on the baselines used in comparison for this setting. It is unclear why these are suitable baselines here for the problem of zero-shot online URL. Many baselines here are either not designed for online learning with self-generated trajectory (for example, LRA-SR used in (Touati et al., 2023)) or not zero-shot testing (if I’m not mistaken some baseline finetunes longer steps, for example, CeSD with 100k interactions rather than 1e^4 used in this work). So it does not look so exciting when explicit exploration techniques are used in combination with a zero-shot technique. A naive approach for this problem would be using a method of pure exploration (e.g. r_{ce} proposed in this work as the proposed intrinsic reward itself has the capability to collect an offline dataset.) or a method of skill discovery to collect an offline dataset with better data coverage than FB, then training FB on top of this dataset and testing its ability in zero-shot generalisation. This could probably better demonstrate the advantage of combining exploration reward with FB in online URL.\n\nFollowing the previous comment, for Table 1, it would be better to group the baselines into several categories so that it is clear from the table which property (zero-shot, online, offline,  exploration or skill discovery) each method has or does not have.\n\nThere is no theoretical analysis to support the proposed objective function and reward function either in the FB pre-training stage or fine-tuning stage. It is unclear what guarantee of zero-shot generalisation of the proposed method can have.\n\nQuestions and suggestions:\n\nFor Fig. 2 and Fig. 3, Please add more descriptions to the caption so that the reader can understand the main discovery and meaning of the figure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FCCyqpXbi8", "reviewer": "Reviewer_yJG4", "summary": "This work introduces the Dual-Value Forward-Backward (DVFB) representation framework for unsupervised reinforcement learning (URL). It tackles the challenge of enabling agents to generalise to new tasks without further training (zero-shot generalisation) in addition to fine-tuning adaptation in online settings.\n\nIt builds on successor representation (SR)-based approaches which aim to learn a representation of expected future states and have been shown to aid zero-shot generalisation in RL. In particular, the work extends forward-backward (FB) representations by learning both forward and backward dynamics. The authors explore failures in FB-based approaches in online URL settings and find that it is due to inadequate exploration. They address this by introducing an intrinsic reward based on contrastive learning to encourage exploration, combining this “exploration value” function to the usual “skill value” function to arrive at their DVFB. The authors also introduce a fine-tuning scheme using a reward mapping technique to add further online adaptation capabilities to their method. \n\nThe authors validate DVFB across twelve robot control tasks in the DeepMind Control Suite and demonstrate the approach gives both superior zero-shot and fine-tuning performance.", "questions": "Reflecting the weaknesses discussed above, my key questions are:\n - How broadly applicable is the method, particularly beyond robotic control tasks? Are there any preliminary results in other domains that the authors could include?\n - How important is their particular choice of reward to encourage exploration -- the contrastive entropy reward? How well would other rewards stand-in for this, or is it particularly well suited?\n - Similar questions for the reward mapping technique. Could we see more justification for their approach and other alternatives explored?\n - Can the authors provide code so that others can directly reproduce the results?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Zero-shot generalisation in online settings is an important problem in RL, where progress is going to be essential for successfully deploying RL in real-world applications. DVFB advances the field's understanding of how to create agents that can solve and adapt to new tasks immediately, without requiring extensive retraining. \n - The authors build on foundational concepts in the field such as SR and FB representations. The paper does a good job identifying the limitations of FB in online settings, pinpointing insufficient exploration as a core issue, and using their insights to justify the extensions of FB into DVFB. The introduction of a novel exploration value function alongside the skill value function is an original approach that enhances exploration and, as shown in their motivation and results, improves generalisation. Furthermore, the addition of a reward mapping is a valuable addition that enables them to demonstrate both zero-shot generalisation and fine-tuning in an online setting.\n - Impressive results: the paper presents rigorous experiments across 12 diverse control tasks in a widely used benchmark for tasks requiring fine-grained motor control. In terms of zero-shot performance, their method outperforms baseline methods across the 12 tasks, particularly in tasks where others struggle with exploration (further supporting their motivation). It also outperforms on fine-tuning performance, showing faster adaptation and greater stability compared to the state-of-the art in URL.\n - The paper is well written, guiding the reader through the problem being addressed, relevant related work, the motivations for their extensions, implementation, results and conclusions. The methodology section is nicely laid out, with clear explanations and schematics detailing components of the model. Their experimental results are clearly presented and easy to understand.", "weaknesses": "- Potential for broader applicability: the paper focuses on tasks in the DeepMind Control Suite. This demonstrates DVFB’s capability in robotic control tasks, but leaves one wondering about the framework's versatility which otherwise seems very general. Could the authors discuss the potential for adapting DVFB to other domains, such as navigation? If possible, preliminary results or  discussion on expected performance in different contexts would broaden the scope of the work.\n - Other intrinsic rewards: the paper attributes improvements to enhanced exploration, but it doesn’t delve into specific advantages that contrastive entropy is providing over other intrinsic rewards. Going beyond the DVFV w/o CE ablation experiment and trying out other intrinsic rewards (beyond just RND rewards) could add further insight into their particular choice of contrastive entropy.\n - Sparse presentation of reward mapping technique: there’s limited detail and justification for the reward mapping technique. It’s unclear whether this particular mapping method is optimal or if other strategies might perform equally well or even better in different tasks.  Further exploration would clarify its effectiveness and limitations. Could the authors discuss more justification for this approach, as well as analysing some alternatives? \n - Reproducibility: lack of code to reproduce the results: providing code would significantly enhance the paper’s accessibility. While the inclusion of pseudocode and hyperparameters is appreciated and provides important details for the method, the absence of actual code makes it challenging for others to fully replicate the experiments or apply the DVFB framework to other settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WAkl9HJNGg", "reviewer": "Reviewer_pWXy", "summary": "The study presents the Dual-Value Forward-Backward (DVFB) framework for zero-shot generalization in online unsupervised reinforcement learning (URL). DVFB integrates a skill value function with an exploration value function to enhance data diversity and generalization in the absence of task-specific rewards. It utilizes a contrastive entropy intrinsic reward to improve exploration and a dual-value fine-tuning method to optimize downstream task performance, claiming good results in continuous control tasks.", "questions": "1. Could you furnish more detailed explanations regarding the metrics employed in the studies, especially for figures where the axes and comparisons lack clarity? Supplementary labeling and contextual information would assist readers in appropriately interpreting your findings.\n\n2. What is the performance of DVFB in relation to other contemporary zero-shot generalization methods, and what are the reasons for the selection or exclusion of specific baselines? Incorporating a broader array of comparisons or elaborating on these selections would bolster the assertion of enhanced performance.\n\n3. Could you provide a detailed explanation of the practical execution of the reward mapping technique, possibly including pseudocode? Additional detail would elucidate this component's impact during fine-tuning.\n\n4. In what manner does the contrastive entropy reward facilitate skill differentiation, and can you present empirical data that substantiates its efficacy? An elucidation or ablation of the role of this reward would improve comprehension.\n\n5. Have you performed any analysis to assess the sensitivity of DVFB to essential hyperparameters? This would be beneficial to evaluate the resilience of the framework across diverse contexts and circumstances.\n\n6. Could you elaborate on the possible limits of DVFB, including computational complexity and scalability in practical applications? Considering these criteria would yield a more equitable perspective on the approach's practical viability.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The research presents a novel Dual-Value Forward-Backward (DVFB) paradigm that integrates skill and exploratory value functions to improve data variety and zero-shot generalization in online URL, providing an innovative method for reward-free learning.\n\n2. Should the suggested DVFB approach demonstrate efficacy, it may rectify a basic constraint in reinforcement learning by facilitating zero-shot generalization absent task-specific incentives, hence potentially enabling RL agents to adapt more readily to varied real-world contexts.", "weaknesses": "1. The experimental configuration and illustrations are challenging to interpret, with scant explanation offered for particular measures and comparisons. Enhanced labeling, elucidation of axes and benchmarks, and uniform layout throughout figures would facilitate comprehension of the data and augment the paper's readability. Figure 6 has mixed x-axis labels, which needs an improvement. Legends can be bigger w/o affecting the size of total figure for example Figure 7. \n\n2. The method depends on several essential network hyperparameters given in Table-3 yet the research fails to analyze the sensitivity of the results to these selections. An investigation of network hyperparameter sensitivity would enhance confidence in the robustness and generalizability of the findings.\n\n3.The implementation and/or utilization of the reward mapping technique for fine-tuning can be clarified. Integrating pseudocode would improve the accessibility and reproducibility of this component.\n\n4. The report omits a discussion of potential limitations, including computing cost, scalability, and difficulty in real-world implementation. Recognizing these factors might yield a more equitable viewpoint and inform subsequent research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)