Exploratory Diffusion Model for Unsupervised Reinforcement Learning
Authors: Chengyang Ying
Huayu Chen
Xinning Zhou
Zhongkai Hao
Hang Su
Jun Zhu
Venue: ICLR 2026 Oral (2026)
Keywords: reinforcement learning
diffusion policy
unsupervised reinforcement learning
exploration
Primary area: reinforcement learning
TL;DR: We propose Exploratory Diffusion Model (ExDM), boosting unsupervised exploration and few-shot fine-tuning by diffusion models.

Abstract: Unsupervised reinforcement learning (URL) pre-trains agents by exploring diverse states in reward-free environments, aiming to enable efficient adaptation to various downstream tasks. Without extrinsic rewards, prior methods rely on intrinsic objectives, but heterogeneous exploration data demand strong modeling capacity for both intrinsic reward design and policy learning. We introduce the **Ex**ploratory **D**iffusion **M**odel (**ExDM**), which leverages the expressive power of diffusion models to fit diverse replay-buffer distributions, thus providing accurate density estimates and a score-based intrinsic reward that drives exploration into under-visited regions. This mechanism substantially broadens state coverage and yields robust pre-trained policies. Beyond exploration, ExDM offers theoretical guarantees and practical algorithms for fine-tuning diffusion policies under limited interactions, overcoming instability and computational overhead from multi-step sampling. Extensive experiments on Maze2d and URLB show that ExDM achieves superior exploration and faster downstream adaptation, establishing new state-of-the-art results, particularly in environments with complex structure or cross-embodiment settings. The source code is provided at https://github.com/yingchengyang/ExDM.

Published as a conference paper at ICLR 2026
EXPLORATORY DIFFUSION MODEL FOR UNSUPERVISED REINFOR LEARNING
Chengyang Ying', Huayu Chen', Xinning Zhou', Zhongkai Hao", Hang J Zhu
'Department of Computer Science & Technology, Institute for AI, BNRist Center, Tsinghua-Boss Bosch Joint ML Center, THBI Lab, Tsinghua University
ABSTRACT
Unsupervised reinforcement learning (URL) pre-trains agents by exploring di- verse states in reward-free environments, aiming to enable efficient adaptation various downstream tasks. Without extrinsic rewards, prior methods rely on intrinsic objectives, but heterogeneous exploration data demand strong model- ing capacity for both intrinsic reward design and policy learning. We introduce the Exploratory Diffusion Model (ExDM), which leverages the expressive power diffusion models to fit diverse replay-buffer distributions, thus providing ac- curate density estimates and score-based intrinsic reward that drives explo- ration into under-visited regions. This mechanism substantially broadens state coverage and yields robust pre-trained policies. Beyond exploration, ExDM of- fers theoretical guarantees and practical algorithms for fine-tuning diffusion poli- cies under limited interactions, overcoming instability and computational over- head from multi-step sampling. Extensive experiments on Maze2d and URLB show that ExDM achieves superior exploration and faster downstream adapta- tion, establishing new state-of-ther art results, particularly in environments with complex structure cross enossemmbo settings. The source code is provided https: / / thub. com ngchnngyang EXDM.
INTRODUCTION
Developing agents that generalize across diverse tasks remains a central challenge in reinforcement learning (RL). Unsupervised RL (URL) (Eysenbach et al., 2018; Laskin et al., 2021) aims to address this by pre-training in reward-frre environments acquire diverse skills or transferable representa- tions. In the absence of extrinsic rewards, agents often rely on intrinsic objectives that are frequently hand-crafted, myopic, and weakly aligned with downstream tasks. The data collected through ex- ploration are highly heterogeneous, demanding representations that are expressive yet stable against collapse or spurious correlations. In addition, policies trained in fixed reward-free settings often fail to transfer under shifts in dynamics, embodiment, or semanties.
central obstacle in URL is the demand for strong modeling capacity during both pre-training and fine-tuning. Effective exploration in reward-free environments hinges on intrinsic rewards derived from accurate estimates of the underlying state distribution, which is typically heterogeneous and collect diverse trajectories but often rely on simple pre-
difficult to capture. Existing methods trained policies-such as Gaussian (Pathak et al., 2017: Mazzaglia et al., 2022) or discrete skill- based policies (Eysenbach et al., 2018; Laskin et al., 2022)-chosen for their ease of training and sampling. Such policies fail to capture the full diversity of explored data in the replay buffer, limiting both unsupervised exploration and downstream adaptation. This calls for more powerful modeling approaches, where diffusion models stand out for their stability and strong density estimation ability.
To address these challenges, we propose the Exploratory Diffusion Model (ExDM), which leverages diffusion-be based density estimation to address the exploration bottleneck in unsupervised RL while providing reusable prior for downstream adaptation. At its core, ExDM trains a diffusion model on the heterogeneous and nonstationary state distribution in the replay buffer, and uses the resulting
"H. Su and Zhu corresponding authors.

---

Published as a conference paper at ICLR 2026
ctimpy polacy altoys complex
Pre-train
Exploratory Diffusion Model
policy
Fine-tune
Gaussion Palicy "3
Diffosion
Reply Buffer
Pollcy Na
Compriie
Viltig div hekarders
Fast terraction
Dirfusion
Model
Policy
Gzaaviia
Tiak
Dowenaeen
Timls
Figure I: Overview of Exploratory Diffussio Model (ExDM), Different from standard RL, URL aims to explore in reward-free environments, requiring expressive policies and models to fit het- erogeneous data (Theorem 4.1). During pre-training, ExDM employs the diffusion model to model the heterogeneous exploration data and calculate score-based intrinsic rewards to encourage explo- ration. Moreover, we adopt a Gaussian behavior policy to collect data that avoids the inefficiency caused by the multi-step sampling of the diffusion policy.
score function to define an intrinsic reward Rscore that explicitly targets under-visited states. This drives broad state coverage and maximizes entropy during reward-free exploration. Unlike conven- tional generative uses of diffusion, ExDM employs the score function for exploration utility rather than sample fidelity, and must learn from online replay buffer with shifting visitation distribu- tions. To keep train efficiency, ExDM decouples modeling from acting-replacing costly multi-step diffusion sampling with lightweight Gaussian policy trained to maximize Recore- This design preserves the modeling strength of diffusion while enabling scalable training and action selection.
Beyond enhancing unsupervised exploration, ExDM also provides a strong initialization for down- stream tasks. In addition to fine-tuning the Gaussian hehavior policy with standard RI. algorithms, ExDM allows the diffusion model itself to be adapted for downstream control. This adaptation is particularly challenging in URL., where fine-tuning must succeed with limited online interaction. To this end, we analyze the fine-tuning objective and derive an alternating optimization procedure whose convergence and optimality are formally established in Theorem 4.2.
We evaluate ExDM both unsupervised exploration and downstream adaptation across standard benchmarks, including Maze2d (Campos et al., 2020) and continuous control in URLB (Laskin al., 2022). Maze2d, ExDM consistently achieves substantially higher state coverage than all baselines. On the most challenging mazes with many branching paths and decision points, ExDM attains up to 51 % higher coverage and reaches comparable performance using only 37% of timesteps, demonstrating its ability to efficiently explore diverse regions under strict interaction budgets, while baselines often stall near corners and fail to cover the maze. Beyond exploration, URLB experiments, including single-embodiment and cross-embodiment settings, further show that ExDM adapts rapidly to diverse downstream tasks and outperforms SOTA URL and diffusion fine-tuning baselines by large margins, highlighting its effectiveness general framework for exploration and transfer.
In summary, the main contributions are as follows:
To the best of our knowledge, this is the first work to introduce diffusion models into unsu- pervised RL, enabling accurate modeling of heterogeneous state distributions and defining a score-based intrinsic reward that substantially improves exploration.
Beyond exploration, ExDM develops an efficient decoupled training scheme and fine- tuning algorithm for adapting pre-trained diffusion components to downstream tasks under limited interaction, with theoretical guarantees of convergence and optimality.
Extensive experiments Maze2d and URLB benchmarks demonstrate that ExDM achieves broader state coverage and faster adaptation than prior methods, establishing new state-of-the-arn performance in both exploration and transfer.
2
RELATED WORK
Unsupervised Pre-training in RL. For achieving zero-shot generalization in RL, there are lots of attempts like in-context RI. (Rakelly et al., 2019; Zintgraf al., 2021) or forward-backward repre-
2

---

Published as a conference paper at ICLR 2026
sentations (Touati & Ollivier, 2021; Tirinzoni et al., 2025). However, these methods require sam- pling from tasks during pre-training or reward signals from the offline datasets. Differently, URL pre-trains agents in reward-free free environments to acquire knowledge for fast fine-tuning downstream tasks. Existing methods mainly rely on intrinsic rewards to guide agents to explore the environment, falling into two categories: exploration and skill discovery. Exploration methods typically explore diverse states by maximizing intrinsic rewards designed to estimate either uncertainty (Pathak et al., 2017: Burda al., 2018; Pathak al., 2019; Raileanu & Rocktäschel, 2020; Mazzaglia al., 2022; Yuan et al., 2023; Ying et al., 2024) or state entropy (Lee et al., 2019; Liu & Abbeel, 2021; Seo al., 2021; Mutti et al., 2021). Skill-discovery methods hope to collect diverse skills by maxi- mizing the mutual information between skills and states (Eysenbach et al., 2018; Lee et al., 2019; Campos et al., 2020; Kim al., 2021; Park et al., 2022; Laskin et al., 2022; Yuan et al., 2022; Zhao et al., 2022; Yang et al., 2023b; Park et al., 2023; Bai et al., 2024; Wilcoxson et al., 2024). Although exploring diverse states, existing methods always neglect the expression ability of pre-traines poli- cies and choose simple Gaussian policies (Pathak et al., 2017; Mazzaglia et al., 2022) policies (Eysenbach et al., 2018; Yang et al., 2023b), which fail to capture the diversity present in the explored data. Consequently, applying generative models with strong expressive ability for improving the diversity of pre-trained policies is still less studied.
kissl-based
RL with Difffusion Models. Recent advancements have shown that high-fidelity diffusion models can benefit RL from different perspectives (Zhu et al., 2023). In offline RI, diffusion policies (Wang et al., 2023; Chen et al., 2023; Lu al., 2023; Chi al., 2023; Hansen-Estruch et al., 2023; Kang et al., 2024) excel at modeling multimodal behaviors, outperforming previous policies such as Gaus- sians. Besides policies, diffusion planners (Janner et al., 2022; Ajay et al., 2023; He et al., 2023; Liang et al., 2023; Nuti et al., 2023; Chen et al., 2024a) have demonstrated the potential in long-term sequence prediction and test-time planning. Some works have also investigated online training dif- fusion policies to improve performance (Psenka et al., 2023; Li et al., 2024; Ren et al., 2024; Mark et al., 2024; Celik et al., 2025; Ma et al., 2025; Ishfaq et al., 2025). However, the computational cost of multi-step sampling remains the efficiency bottleneck. In addition to behavior modeling, diffusion models have also been employed as world models (Alonso et al., 2024; Ding et al., 2024), augmented replay buffer (Lu et al., 2024; Wang et al., 2024a), hierarchical RL (Li al., 2023; Chen et al., 2024b), and so on. To the best of our knowledge, this work represents the first attempt to leverage the strong modeling capabilities for heterogeneous distribution of diffusion models for unsupervised exploration.
3 BACKGROUND
3.1 UNSUPERVISED REINFORCEMENT LEARNING
Here S and A denote the state and action spaces, respectively. For V(s, a) E S x A. P(.|s, a) is distribution on S, representing the dynamic of M, and R(s, a) is the extrinsic task reward function. po is the initial state distribution and y is the discount factor. For given policy π : S - A(A). define the
RL considers Markov decision processes (MDP) M (S,
R,,,R,,,,77) (...A......
discount state distribution of π at state & as (s) = (1 - 7) Ei PP' = = The objective of RL is maximize the expected cumulative return of क over the task R:
J(x) A R. [R(T)]
E Band, [R(8 a)).
(1)
To boost agents' generalization, unsupervised RL (URL) typically includes two stages: unsuper- vised pre-training and few-shot fime-tuning. During pre-training, agents explore the reward-free environment M i.e., M without the reward function R. Thus, URL requires designing intrinsic rewards Riat to guide policies to maximize the state entropy (-). During fine-tuning, agents adapt pre-trained policies to handle downstream tasks represented by extrinsic task-specific rewards RR through limited interactions (like one-tenth of pre-training steps, the formulation is in Eq. 9).
3.2 DIFFUSION MODELS IN REINFORCEMENT LEARNING
Recent studies have demonstrated that diffusion models excel at accurately representing heterogeneous behaviors in continuous control, particularly through
et al., 2015: Ho et al., 2020)
3

---

Published as a conference paper at ICLR 2026
the use of diffusion policies (Wang al., 2023; Chi et at., 2023). Given state-action pairs (s, a) sam- pled from some unknown policy ({a(s), diffusion policies consider the forward diffusion process that gradually injects standard Gaussian noise e into actions:
ae aga + ore, t [0, 1],
(2)
here dt,, of are pre-defined hyperparameters satisfying that when t = O, we have a, = a, and when t 1, we have a, Fs e. For Vt [0, 1]. we can define the marginal distribution of a, as
/ ((ar ()(aas)dd.
pr(a, (,
Then we train conditional "noise predictor" Es(ar) t) to predict the added noise of each timestep:
min Er. (a: s, E el
The learned Eg can estimate the score function Va, log pr (a (s,,). We can discretize diffusion ODEs of the reverse process (Song et al., 2021b) and sample actions with numerical solvers (Song et al., 2021a; Lu et al., 2022) in around 5 ~ 15 steps, to approximate the original policy μ(als). However, this multi-step sampling affects the training efficiency, especially in online settings.
METHODOLOGY
Below, we introduce the Exploratory Diffusion Model (ExDM) to capture diverse data to boost unsupervised exploration (Sec. 4.1) and obtain powerful initialization for fast fine-tuning (Sec. 4.2).
4.1 EXPLOORATORY DIFFUSION MODEL FOR UNSUPERVISED PRE-TRAINING
The major challenge and objective during unsupervised pre-training is to explore diverse states in reward-free environments. Consequently, a natural pathway to pre-train the policy maximize the entropy of the state (Liu & Abbeel, 2021), i.e., H(d=(-)) = -d- (s) log dx (s)ds. Although the optimal policy fully-observable single-agent RL is simple deterministic policy, we prove that, even if the environment is discrete, policies with the maximum state entropy are still complicated and not deterministic with a high probability, requiring much stronger modeling abilities.
Theorem 4.1 (Policy with maximal state entropy). When S,A are discrete spaces, i.e., [S] S, |AI = A, there are M & AS deterministic policies. Set á arg max, H(dz(-)), under some mild assumptions, have
P(r is not deterministic policy and H(d+) log (S)) 21 M &
(5)
will fast converge to with the increasing of A, and here v(S) is a constant only related to S and satisfies 0 vSS) < 1.
Details and proof are in Appendix B.1 (we also discuss continuous situations there). This theorem demonstrates that maximizing state entropy requires policies with strong expression abilities, rather than simple deterministic policies. Despite previous work mainly considering simple Gaussian poli- cies or skill-based policies, in practice, explored replay buffer is always diverse and heterogeneous, the policy continuously changes to visit states during pre-training. Consequently, URL quires capturing the heterogeneous distribution of collected data and obtaining policies with high diversity. These challenges pose the requirement of strong density estimation and fitting abilities, while maintaining training stability and efficiency. Inspired by the recent great success of diffusion models modeling diverse image distributions (Dhariwal Nichol, 2021) and behaviors (Chi al., 2023; Janner et al., 2022), ExDM proposes to utilize the diffusion models and to model the distribution states and state-action pairs in the replay buffer D collected before:
min E, a [ (st(t) e|la + E.,- ( t) =
(6)
To maximize the entropy of the state distribution, states the replay buffer. Consequently, we design - log por 8) as the intrinsic reward to encourage the agent explore these regions. Although estimating the bog-probability of the diffusion model
use log per (s) to measure the frequency of
4

---

Published as a conference paper at ICLR 2026
Algorithm Pre-training of ExDM
Require: Reward-free environment M replay buffer D, Gaussian behavior policy Tg. diffusion policy иa parameterized with the score model Eæ, state diffusion model Egr.
1: for sample step = 5. do
2: for update step = 1, 1.2..... do
3: Sample pairs
aa"")) from D.
4: Update Eg and Egr via optimizing with Eq. (6) with sampled data.
5: Calculate score-based intrinsic rewards r via Eq. (8) for each sampled pair (s" am)).
6: Train Fg with (s"", a",, r)) by any off-policy RL algorithm.
7: end for
8: Utilize the behavior policy "g to interact with M and store state-action pairs into D.
9: end for
challenging, it is well known that log po (8) can be bounded by the following evidence lower bound (ELBO) (Ho et al., 2020):
log Po (s) S Be,, (ee.:[wellea (s. <l + C,
here C is a constant independent of 0', and we are parameters related to Otz, tt which are typically ignored (Ho et al., 2020). Consequently, we propose our score-based intrinsic rewards as:
Recore (s) = E.,, (
<l|²].
Intuitively, our score-based intrinsic rewards can measure the fitting quality of the diffusion model to the explored data, thereby encouraging the agent explore regions that poorly fitted explored (more analyses between Rycore and - log Per are in Appendix C.1). By maximizing these intrinsic rewards, ExDM trains agents discover unseen regions effectively. However, directly ing diffusion policies to interact with reward-free environments during pre-training is inefficient and unstable due to the requirement of multi-step sampling. To address this limitation, ExDM incor- porates Gaussian behavior policy Tg for efficient action sampling. Gaussian behavior policy Tg can be trained using any off- off-policy RL algorithm, guided by score-based intrinsic rewards Rooore (s). This encourages the exploration of regions where the diffusion model either fits poorly or has not yet been exposed. The pseudo code of the unsupervised exploration stage of ExDM is in Algorithm 1.
4.2 EFFICIENT ONLINE FINE-TUNING TO DOWNSTREAM TASKS
When adapting pre-trained policies downstream tasks with limited timesteps, existing URL meth- ods always directly apply online RI algorithms like DDPG (Lillicrap. 2015) or PPO (Schulman al., 2017) for fine-tuning. The behavior policy Fg in ExDM also be fine-tuned to handle the downstream task with the same online RL algorithms, performing fair comparison of exploration efficiency between ExDM and baselines (detailed experimental results are in Sec. 5.3).
Besides Tg. ExDM has also pre-trained the diffusion policy ma, which can better capture the hetero- geneous explored trajectories for adapting to downstream tasks. Unfortunately, it is challenging to online fine-tune diffusion policies due to the instability caused by the multi-stpp sampling and the lack of closed-form probability calculation (Ren et al., 2024). To address these challenges, we first analyze the online fine-tuning objective for URL. Given the limited fine-tuning timesteps, the objec- tive can be formulated as the combination of maximizing the cumulative return and keeping close to the pre-trained policy over all (Eysenbach et al., 2021) (more analyses are in Appendix C.2);
3
max [+( A(()) 7
Eand, (
(1 (1-7)
7 Eg arut anun
[R(s,a) (
(9)
π <aas)) R( -Blg
E Eg awt
(a 8)
here 3 >> 0 is an unknown trade-off parameter that is related to fine-tuning steps. (T() can be inter-
preted penalizing the probability offset of the policy in (s, a.) over and ma- More specifically,
5

---

Published as a conference paper at ICLR 2026
Algorithm 2 Diffusion Policy Fine-tuning of ExDM
Require:
Environment M with rewards R. replay buffer D, pre-trained diffusion policy param- eterized with the score model EØ, fine-tuned diffusion policy 6x.
1: for update iteration = 2.,,..., N do
2: Sample s-a-r pairs {(s"), a".. ,r'") from D.
3: Update Q function with IQL and update Guidance føn-, with CEP.
4: Optimize ve by score distillation with Eq. (14).
5: for interaction step = 1,2,-. .... S do
6: Interact with M by Eg. and store state-action-reward pairs into D.
7: end for
8: end for
aims to maximize a surrogate reward of the form R(s, a) - B log =a(ala) However, this surrogate
m(ale)
reward depends on the policy и,, and we cannot directly apply the classical RL analyses. Inspired by
soft RL (Haarnoj et al., 2017) and offline RL (Peng et al., 2019), we define our Q functions as:
) (a (8;)
(s( =E R(s a) + Σ
(R(5), as) 3 log
π (ai(8:))
(10)
1=1
Based on this Q function, we can simplify Jr as
(+( -Es~po,a~t 1Q a) 3D... (T([|)) (a((ss))]
(11)
To optimize Jr. ExDM decouples optimizing Q functions and diffusion policies. detail, we initial To = Fd. Qo = Q.. then for n = 1, 2, .... we set
Tn (-1s) ) - arg max Eavr [Qmn- (s, a) - (=Ks((((()) = [a(a)s)c 1(.a)/8
Z(s)
(12)
On Q
here 7(s) ( (n Building on soft RI analysis (Haarnoja al., 2017: 2018), we show the policy improvement of each iteration and the optimality of the alternating optimization:
Theorem 4.2 (Optimality of ExDM, Proof Appendix B.2). ExDM can achieve policy improve- ment, Le., J)( 2 (1( 1) for Vn 2 1. And nr will converge to the optimal policy of Jr-
Compared with offline RL, in which Q functions are related to offline datasets, Q functions here are related to current policies, which introduces extra challenges as Q functions change correspondingly during fine-tuning. Below, we introduce the practical diffusion policy fine-tuning method ExDM for both updating Q functions and diffusion policies, respectively (pseudo-code Algorithm 2).
Q function optimization. Our principle for updating Q functions is to penalize actions with large log probability ratios between and πa. Thus, we apply implicit Q-learning (IQL) (Kostrikov al., 2022), which leverages expectile regression to penalize out-of-distributis actions (Appendix C.3).
Diffusion policy distillation. At each iteration n with Q function Qn- calculating тr by Eq. (12) is difficult as (()) ) is complicated integral. However, sampling from can be regarded as sam- pling from za with energy guidance Qn-1, i.c., guided sampling (Janner et al., 2022). Especially, we employ contrastive energy prediction (CEP) (Lu al., 2023) to sample from x Tn-0011/0 and parameterize fon (8, a,, to represent timestep f's energy guidance, which can be optimizated as:
for (s. ai,t) 2 ( log Σ So. (8, ai
min E,, a On-1
(13)
Then ExDM fine-tunes diffusion policies by distilling the score of ग. parameterized by evi s,)]s,t):
min E... (a,ls, ca(a,|stt) fan-1 a. as, t)
(14)
Finally, directly sample from e, to generate action of n. (details are in Appendix C.4).

---

Published as a conference paper at ICLR 2026
MEPOL
Sqguee- Squirehootizeeek
Normalized Value
Figure Heatmap of explored regions by URL methods in the most complicated mazes.
Square-a
Square-b
Square-c
Square-d
Square-tree
Square-botlleeekk
Square-large
0.58
0.49
0.49
RND
0.50 1: 0.14
0.39 1: 0.08
0.52 1: 0.16
0.32 2 0.05
0.28 0.06
0.33 1 0.06
0.33 :: 0.08
LBS
0.32 2 0.04
0.29 + 0.09
0.27 + 0.05
0.25 = 0.03
0.22 0.03
0.21 0.02
0.19 - 0.06
MEPOL
0.98 + 0.03
0.99 +: 0.02
0,96 = 0.07
0.77 = 0.01
0.89 0.06
0.62 0.01
0,59 = 0.04
0.06
0.00
SMM
0.13
0.45 +: 0.20
0.36 =: 0.08
0.28 = 0.04
0.25 0.02
0.1 = 0.13
0.34 0.10
LSD
0.03
0.38 1: 0.05
0,36 -:
0.35 0.03
0,28 0.03
0.34
0.32 0.03
CIC
0.94 - 0.02
0.98 +: 0.01
0,86 + 0.03
0.74
0.89 = 0.01
0.88 0.05
0.47 = 0.01
BeCI.
0.50 3.00
0.11
0.42
0.37
0.36 0.06
0.29
0,25 0.05
CeSD
0.70 0.04
0.79 1 0.04
0.67 1. 0.06
0.46 0.06
0.37 0.06
0,46 0.03
0.40 0.01
FXDM
0.99 0.02
0.99 3 0.01
0.98 1 0.02
0.78 6.01
0.11 0.01
0.75
0.71 2 0.07
Table 1: State coverage in Maze. We report the mean and std of 10 seeds for each algorithm.
5
EXPERIMENTS
In this section, we present extensive empirical results to mainly address the following questions:
Can ExDM boost the unsupervised exploration efficiency, especially in complicated mazes with numerous branching paths and decision points? (Sec. 5.2)
What about the adaptation efficiency of the pre-trained Gaussian policies of ExDM com- pared to other URL baselines? (Sec. 5.3)
As for fast fine-tuning pre-trained diffusion policies to downstream tasks, how does the performance of ExDM compare to existing baselines? (Sec. 5.4)
EXPERIMENTAL SETUP
Maze2d. We first evaluate the exploration diversity during the unsupervised stage in widely used maze2d environments (Campos et al., 2020; Yang et al., 2023b): Square-a, Square-b, Square-c, Square-d, Square-tree, Square-bottleneck, and Squgre-large. Observations and actions here belong to 2 When interacting with mazes, agents will be blocked when they contact walls.
To evaluate the performance of fine-tuning in downstream tasks, we choose 4 continuous control settings in URLB (Laskin et al., 2021): Walker, Quadruped, Jaco, and Hopper. Each domain contains four downstream tasks. More details are in Appendix D.I.
Continuous Control.
Baselines. Maze2d experiments, we take exploration baselines: ICM (Pathak et al., 2017), RND (Burda et al., 2018), Disagreement (Pathak et al., 2019), RE3 (Seo et al., 2021), MEPOL (Mutti et al., 2021), and LBS (Mazzaglia et al., 2022); as well as 6 skill discovery baselines: DIAYN (Ey- senbach et al., 2018), SMM (Lee et al., 2019), LSD (Park al., 2022), CIC (Laskin et al., 2022), BeCL (Yang et al., 2023b), and CeSD (Bai et al., 2024). which are standard and SOTA. As for fine-tuning in URLB, consider three settings: (a) for fair comparison, directly utilize

---

Published as a conference paper at ICLR 2026
Median (1)
IOM (1)
Mean
Optimality Gap
(4)
LSD
DIAYN
SMM
RE3
MEPOL
Disagresment
RND
EXDM (Durs)
0.2
0.8
0.2
0.8
Expert Nomallzzd
(a) URLB finc-tuned by DDPG
Optimality (4)
ICM Disagresment
DIAYN
SMM
CIC
BdC
PEAC
EXBM (Ours)
0.45
0.80 0.75
0.60
0.75
0.45
0.60
0.75
0.30
0.45
0.60
Expert Nomalized Score
(b) Cross-embodiment URLB fine-tuned by DDPG
Median (?)
1OM (f)
Mean (f)
Optimality Gap (1)
Q5M
DO1
IDOL
DIPO
EXOM (Ours)
0.4 0.6
0.4
0.6
Exppert Normalized Score
(c) URLB for fine-tuning diffusion policies
Figure 3: Aggregate metrics (Agarwal et al., 2021) for three settings. Details are in Appendix D.5.
DDPG (Sohl-Diekstein et al., 2015) to fine-tune the pre-trained behavior Gaussian policy in ExDM, compared to existing URL baselines, including ICM, RND, Disagreement, RE3, MEPOL, LBS, DIAYN, SMM, LSD, CIC, BeCL, and CeSD (all baselines fine-tuned by DDPG, the standard RL backbone in URLB, except CeSD fine-tuned by ensembled DDPG); (b) We consider complicated cross-embodiment URL (Ying et al., 2024), comparing ExDM with ICM, Disagreement, RND, LBS, DIAYN, SMM, CIC, BeCL, and PEAC (Ying et al., 2024); (c) ExDM also fine-tunes pre-trained diffusion policies, compared diffusion policy fine-tuned baselines, like DQL (Wang al., 2023), IDQL (Hansen-Estruch et al., 2023), QSM (Psenka et al., 2023), and DIPO (Yang et al., 2023a).
Metries. Maze2d, we pre-train agents in reward-free environments with 100k steps and visu- alize all collected trajectories. Moreover, to quantitatively compare the exploration efficiency, evaluate the state coverage ratios, which are measured as the proportion of 0.01 x 0.01 square bins pre-train agents in reward-free environments
visited. As for URLB, following standard settings, for 2M steps and fine-tune pre- preitrained policies to adapt each downstream task within extrinsic re- wards for 100K steps. All settings are run for 10 seeds to mitigate the effectiveness of randomness.
5.2 UNSUPERVISED PRE-TRAINING FOR EXPLORATION
In Fig. 2, we visualize the heatmap of collected trajectories during pre-training in complicated Square-botlenecc and Square-large (results all mazes and 13 baselines are in Appendix D.4). To quantitatively evaluate the exploration efficiency of each algorithm, we further report the state coverages in Table (training curves are in Fig. 8 of Appendix D.4). In both qualitative visualization and quantitative metrics, ExDM outperforms baselines by large margins. Especially, in complicated mazes like Square- bottleneck and Square-large with many different branching points (Fig. 2), all baselines will struggle some wall corner and cannot explore the entire maze. In contrast, ExDM
Square-bottleneck
8

---

Published as a conference paper at ICLR 2026
-IM
-ND
18
Qcaloped
Quacegee
Qucraged
0.0
Sxper
3.34 334.556 8
15
Diflaom Surpiing Sep
Tronalong
(a) Pre-training timesteps
(b) Q Learning choices
(c) Diffusion steps
(d) B for Fine-tuning
Figure 4: Ablation studies.
successfully explores almost the whole maze, demonstrating that our Racore, leveraging the accurate data estimation ability of diffusion models, can effectively guide agents explore diverse states.
5.3 FINE-TUNING THE GAUSSIAN POLICY TO DOWNSTREAM TASKS
We verify the ability of ExDM to fine-tune downstream tasks in both single-embodiment and cross- embodiment URLB. As existing URL methods directly fine-tune policies with DDPG, for a fair comparison, we also use DDPG to fine-tune pre-trained Gaussian policies 7g ExDM. Following previous settings, we train DDPG agents for each downstream task with 2M steps to obtain the expert return and calculate the expert-normalized score for each algorithm. Fig. 3(a)-3(b), we compare all methods with four metrics: mean, median, interquartile mean (IQM), and optimality gap (OG), along with stratified bootstrap confidence intervals. ExDM significantly outperforms all existing SOTA methods, demonstrating that introducing diffusion models can lead to more efficient generalization in downstream tasks. Details of each downstream task are in Appendix D.5.
5.4 FINE-TUNING THE DIFFUSION POLICY TO DOWNSTREAM TASKS
Moreover, ExDM has pre-trained diffusion policies that can capture the diversity of explored trajec- tories and adapt to downstream tasks. Consequently, in Fig. 3(c), ExDM substantially outperforms alternating op-
existing diffusion online tinettunig baselines, demonstrating the efficiency timization. However, diffusion policy fine-tuning performance is still lower than Gaussian policy performance, which may be due to limited interaction timesteps during fine-tuning. is an interest- ing future direction design more efficient diffusion online fine-tuning methods.
5.5 ABLATION STUDIES
Pre-training Steps. We first do ablation studies on pre-trained steps (100k, 500k, 1M, and 2M) to evaluate fine-tuned performance (100k fine-tuned steps). As shown in Fig. 4(a), ExDM markedly exceeds all baselines from 500k steps, indicating that the diffusion model enhances fine-tuning. Moreover, ExDM substantially improves with increasing pre-training timesteps, showing that the unsupervised exploration benefits downstream tasks. Additional results are in Appendix D.6.
Q function optimization. We conduct ablation studies to evaluate the impact of Q learning meth- ods during fine-tuning. In detail, we introduce ExDM w/o IQL, which utilizes In-support Softmax Q-Learning (Lu et al., 2023) for optimizing functions. Fig. 4(b) demonstrates that ExDM consis- tently outperforms ExDM w/o IQL, verifying the efficiency of IQL in diffusion policy fine-tuning.
Sampling steps of diffusion policies. During fine-tuning diffusion policies, ExDM requires sam- pling actions from diffusion policies for both trajectory generation and final evaluation. We adopt DPM-Solver (Lu et al., 2022) to accelerate sampling. For trajectory collection, we set the diffusion step 15, following previous works (Lu et al., 2023). Then we conduct ablation studies between the diffusion sampling steps used during inference and the fine-tuned performance. Fig. 4(c) shows that performance improves diffusion steps increase and gradually stabilizes when the step exceeds 5.

---

Published as a conference paper at ICLR 2026
Ablation study 3. We consider the objective Jr with the behavior regularization term because the tine-tuning step is limited (more analyses are in Appendix C.2). The parameter 3 implicitly relies on the fine-tuning steps. If the fine-tuning steps are infinite, the optimal 3 should be 0, and Jr degrades into J. We set /3- = 1 /3.0 3.0 in experiments (following previous work (Lu et al., 2023)) and do ablations with different /3 in Fig. 4(d), showing that ExDM performs relatively stably of [3.
Time cost of ExDM. One of the major concerns for diffusion models is their time cost due to multi-step sampling. This problem may be more severe in online RL, as each collected trajectory re- quires sampling from diffusion policies. To address it, ExDM decouples modeling from acting, i.e., utilizing Gaussian behavior policies 7g for sampling. Thus, ExDM exhibits high training efficiency for both timesteps and training time. For example, in Square-large, RND achieves the state coverage of 0.33 with 100k timesteps and 1000s of time. ExDM can achieve 0.71 state coverage within 100k timesteps, and achieve 0.4 state coverage with the same time cost (1000s) and only 16.6k timesteps.
CONCLUSION
Unsupervised exploration is one of the major problems in RL for improving task generalization, it relies on accurate intrinsic rewards to guide exploration of unseen regions. In this work, we address the challenge of limited policy expressivity in previous exploration methods by leveraging the pow- erful expressive ability of diffusion policies. detail, our Exploratory Diffusion Model (ExDM) improves exploration efficiency during pre-training while generating policies with high behavioral diversity. We also provide theoretical analysis of diffusion policy fine-tuning, along with practi- cal alternating optimization methods. Experiments various settings demonstrate that ExDM can effectively benefit both pre-training exploration and fine-tuning performance. We hope this work can inspire further research in developing fighfiidliity generative models improve unsupervised exploration, particularly in large-scale pre-trained agents or real-world control applications.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
Designing generalizable agents for varying tasks one of the major concerns in reinforcement learning. This work focuses utilizing diffusion policies for exploration and proposes novel algorithm ExDM. One of the potential negative impacts that algorithms mainly use deep neural networks, which lack interoperability and may face robustness issues. There are no serious ethical issues, as this is basic research.
REPRODUCIBILITY STATEMENT
To ensure that our work is reproducible, we have submitted the source code htttps://githi: com/yinie wwiyyyinnngggannnnnmmm anggyynn//ai ExDM. We also provide the pseudo-code of ExDM in Algorithm 1-2 and implementation details of ExDM, including hyper-parameters in Appendix D. Moreover, for all theoretical results, have provided all details and proofs in Appendix B.
ACKNOWLEDGEMENT
This project is supported by Fundamental and Interdisciplinaty Disciplines Breakthrough Plan of the Ministry of Education of China (No. JYB2025XDXMi01, NSF of China Projects (Nos. U25B6003, 62550004, 92370124, 92248303); Beijing Natural Science Foundation L247011; the High Performance Computing Center, Tsinghua University. J.Z was also supported by the XPlorer Prize.
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural informa- fion processing systems, 34:29304-223 2222304-22220, 2021.
Anurag Ajay. Yilun Du, Abhi Gupta, Joshua B. Tenenbaum, Tommi S. Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? In The Eleventh Interna- tional Conference Learning Representations, 2023. URL /tttts:://pprreeeee
n
forum? Pd= Fo2K9DFG.
Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos Storkey, Tim Pearce, and François Fleuret. Diffusion for world modeling: Visual details matter in atari. arXiv preprint arXiv:2405. 12399, 2024.
Adrian Baddeley, Imre Bárány, and Rolf Schneider. Random polytopes, convex bodies, and ap- proximation. Stochastic Geometry: Lectures given the CIME Summer School held in Martina Franca, Italy, September 13-18, 2004, pp. 77-118, 2007.
Chenjia Bai, Rushuai Yang, Qiaosheng Zhang, Kang Xu, Yi Chen, Ting Xiao, and Xuelong Li. Con- strained ensemble exploration for unsupervised skill discovery. arXiv preprint arXiv:2405. 16030, 2024.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference Learning Representations, 2018.
Víctor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giró-i Nicto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. In Interna- tional Conference on Machine Learning, pp. 1317-1327. PMLR, 2020.
Onur Celik, Zechu Li, Denis Blessing, Ge Li. Daniel Palanicek. Jan Peters, Georgia Chalvatzaki, and Gerhard Neumann. Dime: Diffusion-basee maximum entropy reinforcement learning. arXiv preprint ar arxiv:2502.022 02..5002..236..... 2025.
Boyuan Chen, Diego Marti Monso, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitz- mann. Diffussin forcing: Next-token prediction meets full-sequence diffusion. arXiv preprint arXiv:2407.01332, 2024a.
11

---

Published as a conference paper at ICLR 2026
Chang Chen, Fei Deng, Kenji Kawaguchi, Caglar Gulcehre, and Sungjin Ahn. Simple hierarchical planning with diffusion. arXiv preprint 2at.:24400.02244,, 2024b.
Huayu Chen, Cheng Lu, Chengyang Ying. Hang Su, and Jun Zhu. Offline reinforcement learn- ing via high-fidelity generative behavior modeling. In The Eleventh International Confer- ence on Learning Representations, 2023, URL https:/ /wppnreeeeeeww..///// .nt/ forum? = 422s3qa2kpy.
/
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The Inter- national Journal of Robotics Research, pp. 02783649241273666. 2023.
Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, and Pieter Abbeel. Model-based reinforcement learning via meta-policy optimization. Conference on robot learn- ing. pp. 617-629. PMLR, 2018.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Zihan Ding, Amy Zhang, Yuandong Tian, and Qinqing Zheng. Diffusion world model. arXiv preprint arXiv:2402.03570 2024.
Yan Duan, John Schulman, Xi Chen, Peter I. Bartlett Ilya Sutskever, and Pieter Abbeel. RI 2: Fast reinforcement learning via slow reinforcement learning, arXiv preprint arXiv:1611.0277,, 2016.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representa- fions, 2018
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. The information geometry of unsupervised reinforcement learning. In International Conference on Learning Representations, 2021.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1126-112 135. PMLR, 2017.
1135,
Sebastian Flennerhag, Yannick Schroecker, Tom Zahavy, Hado van Hasselt, David Silver, and Satin- der Singh. Bootstrapped meta-learning. arXiv preprint urXiv:2 109.044044 2021.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference on machine learning, pp. 1352-1361. PMLR, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforeement learning with stochastic actor. In International confer- ence on machine learning. pp. 1861-1870. PMLR, 2018
Philippe Hansen-Estruchh Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304. 10573, 2023.
Haoran He, Chenjia Bai, Kang Xu, Zhuoran Yang, Weinan Zhang, Dong Wang, Bin Zhao, and Xue- long Li. Diffusion model an effective planner and data synthesizer for tusk reinforcement learning. Advances in neural information processing systems. 36:64896 (164499666917,,,, 2023.
Takuya Hiraoka, Takahisa Imagawa, Voot Tangkaratt, Takayuki Osa, Takashi Onishi, and Yoshi- masa Tsuruoka. Meta-modell-asse meta-policy optimization. In Asian Conference on Machine Learning, pp. 129-144. PMLR, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6883 6851, 2020.
Haque Ishfaq, Guangyuan Wang, Sami Nur Islam, and Doina Precup. Langevin soft actor-critic:
Efficient exploration through uncertainty-drive driven critic learning, arXiv preprint arXiv:2501.1782 2025.
12

---

Published as a conference paper at ICLR 2026
Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, pp. 9902-9915. PMLR. 2022.
Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Jaekyeom Kim, Seohong Park, and Gunhee Kim. Unsupervised skill discovery with bottleneck option learning. In International Conference on Machine Learning, pp. 5572-5582. PMLR, 2021.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-
learning. In International Conference on Learning Representations, 2022. URL https:// openreview wwtpeeewwwwwwwweeeeeeeeeeeeeeeee 2
d-d 68n2s
Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. Urlb: Unsupervised reinforcement learning benchmark. In Thirty-fftt Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel. Unsupervised reinforcement learning with contrastive intrinsic control. Advances in Neural In- formation Processing Systems, 35:34478 34491, 2022.
Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdi- nov. Efficient exploration via state marginal matching. arXiv preprint arXiv:1906.0522 2019.
Wenhao Li, Xiangfeng Wang, Bo Jin, and Hongyuan Zha. Hierarchical diffusion for offline decision making. In International Conference on Machine Learning, pp. 20035-20064. PMLR, 2023.
Zechu Li, Rickmer Krohn, Tao Chen, Anurag Ajay, Pulkit Agrawal, and Georgia Chalvatzaki. Learning multimodal behaviors from scratch with diffusion policy gradient. arXiv preprint arXiv:2406.0062 2024.
Zhixuan Liang, Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, and Ping Luo. Adaptdiffuser: Diffusion models as adaptive self-evolving planners. In International Conference on Machine Learning, pp. 20725-20745. PMLR, 2023.
TP Lillicrap. Continuous control with deep reinforcement arXiv: 1509.02909 2015.
learning.
arXiv preprint
Bo Liu, Xidong Feng, Jie Ren, Luo Mai, Rui Zhu, Haifeng Zhang, Jun Wang, and Yaodong Yang. A theoretical understanding of gradient bias in mentrreeinfor learning. Advances Neural Information Processing Systems, 10599-30331 2022.
Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. Advances in Neurul Information Processing Systems, 34: 8459-18873, 2021.
Xu-Hui Liu, Tian-Shuo Liu, Shengyi Jiang, Ruifeng Chen, Zhilong Zhang, Xinwei Chen, and Yang Yu. Energy-guided diffusion sampling for offline-to-online reinforeement learning. arXiv preprint arXiv:2407. 12448, 2024.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact energy- energy-guided diffusion sampling in offline reinforcement learning. In Inter- national Conference on Machine Learning, pp. 22825-22855 PMLR, 2023.
Cong Lu, Philip Ball, Yee Whye Tch, and Jack Holder. Synthetic experience replay. Ad- vances Neural Information Processing Systems, 36, 2024.
Haitong Ma, Tianyi Chen, Kai Wang, Na Li, and Bo Dai. Soft diffusion actor-critic; Efficient online reinforcement learning for diffusion policy. arXiv preprint arXiv:2502.003:: 00361, 2025.
13

---

Published as a conference paper at ICLR 2026
Max Sobol Mark, Tian Gao, Georgia Gabriela Sampaio, Mohan Kumar Srirama, Archit Sharma, Chelsea Finn, and Aviral Kumar. Policy agnostic rl: Offline rl and online rl fine-tuning of any class and backbone. arXiv preprint arXin:2412.06685, 2024.
Pietro Mazzaglia, Ozan Catal, Tim Verbelen, and Bart Dhoedt. Curiosity-driven exploration via latent bayesian surprise. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 36, pp. 7752-7760, 2022.
Mirco Mutti, Lorenzo Pratissoli, and Marcello Restelli. Task-agnostic exploration via policy gradi- ent of a non-parametric state entropy estimate. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 9028-9036, 2021.
Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Picter Abbeel, Sergey Levine, and Chelsea Finn. Learning adapt in dynamic, real-world environments through meta- reinforcement learning. arXiv preprint /803. 2018.
Felipe Nuti, Tim Franzmeyer, and João F Henriques. Extracting reward functions from diffusion models. Advances in Neural Information Processing Systems, 36:50196-50220, 2023.
Scohong Park, Jongwook Choi, Jaekyeom Kim, Honglak Lee, and Gunhee Kim. Lipschitz- constrained unsupervised skill discovery. In International Conference on Learning Represen- tations, 2022.
Seohong Park, Oleh Rybkin, and Sergey Levine Metra: Scalable unsupervised with metric-aware abstraction. arXiv preprint arXiv:2310.0888/7 2023.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp. 2778-2787. PMLR, 2017.
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. In International conference on machine learning, pp. 5062- 5071. PMLR, 2019.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00117, 2019.
Michael Psenka, Alejandro Escontrela, Pieter Abbecl, and Yi Ma. Learning a diffusion model policy from rewards via q-score matching. arXiv preprint arXiv:2312.11152 2023.
Roberta Raileanu and Tim Rocktäschel. Ride: Rewarding impact-driven exploration for procedurally-gennratee environments. arXiv preprint arXiv:2002.12222. 2020.
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinffrrcmmet learning via probabilistic context variables. In International conference on machine learning, pp. 5331-5340. PMLR, 2019.
Allen Z Ren, Justin Lidard, Lars L Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majum- dar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimiza- tion. arXiv preprint arXiv:2409.0055 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1707v01707.0633 2017.
Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lec, Pieter Abbeel, and Kimin Lee. State entropy maximization with random encoders for efficient exploration. In International conference on machine learning, pp. 9443-9454, PMLR, 2021.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing, pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2021a. URL https: htttps://ppereeeeeeee http://pppeeeeeeeeeeeeee ferum?id=SSe g arCHIP.
14

---

Published as a conference paper at ICLR 2026
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2021b. URL https:://peereviiiie nnet/ forum? d=p.m 2RRHS.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, al. Deepmind control suite. arXiv preprint arXiv:1801.00650 000900 2018.
Andrea Tirinzoni, Ahmed Touati, Jesse Farebrother, Mateusz Guzek, Anssi Kanervisto, Yingchen Xu, Alessandro Lazaric, and Matteo Pirotta. Zero-shot whole-body humanoid control via behav- ioral foundation models. arXiv preprint 2504: 2025.
Ahmed Touati and Yann Ollivier. Learning one representation to optimize all rewards. Advances in Neural Information Processing Systems, 34:13 13-2,, 2021.
Renhao Wang, Kevin Frans, Pieter Abbeel, Sergey Levine, and Alexei A Efros. Prioritized genera- tive replay. arXiv preprint arXiv:2410.1808; 2024a.
Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023. URL https: http//ppeeeee.....o//ee/. net/ forum? wiw.mmiiiiiiiiii FAA.
Zhendong Wang, Zhaoshuo Li, Ajay Mandlekar, Zhenjia Xu, Jiaojiao Fan, Yashraj Narang, Linxi Fan, Yuke Zhu, Yogesh Balaji, Mingyuan Zhou, et al. One-step diffusion policy: Fast visuomotor policies via diffusion distillation. arXiv preprint arXiv:2410.21257, 2024b.
James G Wendel. A problem in geometric probability. Mathematica Scandinavica, 11(1):109-111, 1962.
Max Wilcoxson, Qiyang Li, Kevin Frans, and Sergey Levine. Leveraging skills from unlabeled prior data for efficient online exploration. arXiv preprint arXiv:2410. 18076, 2024.
Long Yang, Zhixiong Huang, Fenghao Lei, Yucun Zhong, Yiming Yang, Cong Fang, Shiting Wen, Binbin Zhou, and Zhouchen Lin. Policy representation via diffusion probability model for rein- forcement learning. arXiv preprint arXiv:2305. 13 1222, 2023a.
Rushuai Yang, Chenjia Bai, Hongyi Guo, Siyuan Li, Bin Zhao, Zhen Wang, Peng Liu, and Xuelong Li. Behavior contrastive learning for unsupervised skill discovery. In International Conference on Machine Learning. pp. 39183-39204. PMLR, 2023b.
Chengyang Ying. Xinning Zhou, Hang Su, Dong Yan, Ning Chen, and Jun Zhu. Towards safe rein- forcement learning via constraining conditional value-at-risk. arXiv preprint arXiv:2206.0444 2022.
Chengyang Ying, Hao Zhongkai, Xinning Zhou, Xuezhou Xu, Hang Su, Xingxing Zhang, and Jun Zhu. Peac: Unsupervised pre-training for cross-embodiment reinforcement learning. Advances in Neural Information Processing Systems, 37:54632-54669, 2024.
Mingqi Yuan, Bo Li, Xin Jin, and Wenjun Zeng. Automatic intrinsic reward shaping for exploration in deep reinforcement learning. In International Conference on Machine Learning, pp. 40531- 40554. PMLR, 2023.
Yifu Yuan, Jianye Hao, Fei Ni, Yao Mu, Yan Zheng, Yujing Hu, Jinyi Liu, Yingfeng Chen, and Changjie Fan. Euclid: Towards efficient unsupervised reinforcement learning with multi-choice dynamics model. arXiv preprint arXiv:2210.004990 2022.
Andrew Zhao, Matthieu Lin, Yangguang Li, Yong-Jin Liu, and Gao Huang. A mixture of surprises for unsupervised reinforcement learning. Advances Neural Information Processing Systems, 35:26078-2660 2022.
Zhengbang Zhu, Hanye Zhao, Haoran He, Yichao Zhong, Shenyu Zhang, Haoquan Guo, Tingting Chen, and Weinan Zhang. Diffusion models for reinforcement learning: A survey, arXiv preprint 1.X0223111.01223. 2023.
15

---

Published as a conference paper at ICLR 2026
Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl. Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep rl via meta- learning. arXiv preprint arXiv:1910.08348, 2019.
Luisa Zintgraf, Sebastian Schulze, Cong Lu, Leo Feng, Maximilian Igl, Kyriacos Shiarlis, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: Variational bayes-adaptixe deep rl via meta-learning. Journal of Machine Learning Research, 22(289):1-39, 2021.
16

---

Published as a conference paper at ICLR 2026
A EXTENDED RELATED WORK
A.1 META RL ANE UNSUPERVISED RL.
Improving the generalization of RL agents is a long-term challenge. There are several settings aiming at pre-training agents and then fine-tuning the pre-trained agent to fast adapt to various tasks, like unsupervised RL. and meta RL. Unsupervised RL, as we have discussed above, pre-trains agents in reward-free environments, i.e., the environment without task rewards. Thus, we will design intrinsic rewards to pre-train agents to fully explore the environment. And the pre-trained agent needs to fast adapt to downstream tasks that it has not seen during the pre-training stage. Meta RL pre-trains agents several sampled tasks from a task distribution T. And the pre-trained agent needs to fast adapt to test tasks that sampled from T but are not seen during the pre- training stage. There are two major types of methods Meta RL:: gradient- based (Finn et al., 2017; Flennerhag al., 2021; Liu et al., 2022) and context-based methods (Duan et al., 2016; Rakelly et al., 2019; Zintgraf et al., 2019). Besides model-free methods, there are also some methods that apply learned dynamic models boost the sample efficiency of meta RL (Nagabandi et al., 2018; Clavera et al., 2018; Hiraoka et al., 2021). Consequently, the major difference between unsupervised RL and meta RL that, during the pre-training stage, there are no task rewards in the former setting, but the agent can access several task rewards sampled from the same distribution of the test tasks in the latter setting.
A.2 ONLINE FINE-TUNING DIFFUSION POLICIES
Although diffusion policies have been proven to be expressive models for capturing diverse of- fline datasets (Wang et al., 2023; Chen et al., 2023), online fine-tuning diffusion policies (Psenka al., 2023; Li et al., 2024; Ren et al., 2024; Mark al., 2024; Celik al., 2025; Ma et al., 2025; Ishfaq et al., 2025) still face many challenges and are a key issue of concern for the community. For example, the instability caused by multi-step sampling and the lack of closed-form probabil- ity calculation are significant concerns. Many recent works hope to address these challenges and improve the efficiency of online training diffusion policies; they mainly consider two settings: on- line training diffusion policies from scratch with enough timesteps (Psenka et al., 2023; Yang et al., 2023a) or online fine-tuning diffusion policies pre-trained from an offline dataset sampled from same task (Hansen-Fstruud et al., 2023; Ma et al., 2025). Furthermore, there are several poten- tial directions for improving fine-tuning efficiency, including: more energy-guided sampling tech- niques Liu et al. (2024), more efficient diffusion sampling techniques (c.g., distillation, deterministic sampling) (Wang et al., 2024b), and so on.
17

---

Published as a conference paper at ICLR 2026
B THEORETICAL ANALYSES
B. THE DETAILS AND PROOF OF THEEREM 4.1
In this part, we narrate Theorem 4.1 in detail as well as provide its proof.
Assume that S, are discrete spaces and |S| ||| define the discount state distribution of π as d = (1- (11--))> distribution of the state space
A. For any policy क 8 - A(A),
h"P(s' hi = s)}. Naturally, dz (-) S. Specially, when is discrete, following previous work (Ey- senbach et al., 2021; Ying et al., 2024), we can regard dz
point of the probability simplex in R. i.e., d, H & { Ei= 24 1,0 I 2t 1] (for example, the light blue plane in the left part of Fig. 1). Moreover, we consider D A {d E H H representing all feasible state distribution.
is natural that there are M A A different deterministic policies. Previous work (Eysenbach al., 2021) has proven that D is convex polytope, of which the vertices are the state distributions of the deterministic policies (for example, the green points of Fig. 1). Consequently, for any downstream task represented by some extrinsic reward function R, the optimal policy is one the vertices of D, i.e. some deterministic policy.
Differently, the unsupervised exploration in URL aims to maximize the entropy of the policy, i.e., we hope to optimize म = arg max, (-). As (-)) = -ogg can be regard as maximizing a surrogate reward - llo (s). However, the surrogate reward is related to the current policy æ, thus the analyses of standard RL may not hold. Actually, if we consider all distribution over S, i.e., all points in H. it is well known that the distribution with the maximal distribution the center of H, i.e. O = (1/S, 1/5, 1/S) E (for example, the red point of Fig. 1). Although O D may not hold, hope to claim that the optimal policy á with the maximal state distribution entropy may not be deterministic, and its state distribution is O, i.e., O E D, with high probability.
this problem
We consider the distribution in which the state distributions of the M deterministic policies are i.i.d., and all follow the uniform distribution on H. Therefore, our problem can be transformed into: there are M i.idd points uniformly sampled from H, and the convex polytope formed by these M points D C H, then calculating the probability of the event O D.
Based on the results in geometric probability (Eq. (13) of paper (Baddeley et al., 2007), extending the Wendel Theorem in (Wendel, 1962)), we have
S-1 P)OED)2LE
Σ K=U
(u(O)
(1-
u(())
M-k
(15)
2
2
here u(O)= = vo((20--)))H)/volll Below we first estimate u(O), which obviously belong to [0. 1]. We have
vol(H)) =
dæ dxg...ds
1,,220
1-21
=Vs
dr
dzz...
drs-1
1-11
23-3
-VS
dr
dæz...
(1 - -
[/ dr
1-11
1 1
(1 ... )3) da dn S-3
(16)
=V8
dz
-V8 / (1 -x1) --1)-2
dx
(S - 2))
VS
-1!
18

---

Published as a conference paper at ICLR 2026
Then we estimate vol((20 H) H). H is convex polytope surrounded by S points A (1,0, .......,, O).... Ag = surrounded by S points B, = (2/S__..,5, is difficult to directly calculate the volume of (20 - H) n H (although we can calculate it by
As = (1/5,1/5
1/5). 20 - H is a convex polytope
1/s), ..... Bs (1/S. 1/S...... 1/S. 1).
the inclusion-exclusion principle). We can provide lower bound of (20 H)) n H. Consider convex polytope C' surrounded by S points C₁ = (0,1/(s-)),
1055-1,,,1 1.,,,,,,,( - 1)), .. Cs = (1/(S 1/S5 ..,, 1/(S 1). is easy to show that C' C (20 - H)
(0)0-))n n H and C H.
As the side length of H and C' is / and / S-L- Thus we have (ol((20HH)) H) Z vol(C) =
-
(S-1) 15-1)1 and (O)) vol((20 H) n H))vol(H) (5-1)8- Assume that M S 2 + (S 1) x(O)
v/S
3- then for YA S S S 2, we have
u(O))
k
m(O)
M-K
OM M
11
2
2
++1 2 - u(0)
[u(O)) K+1
1 u(O)
M-k-1
Ck+1
M k m(O)
M
(
2
I 2-2+1 2 (0))
Ck+1 (n(O)
k+1
1 1 u(O))
M-
(17)
M - + (())
2
S 1
2 (0)
Ch+1
(u(O)
k+1
(1 n(O)
M-k-1
(8 1) 2-
w0))
2
2
a(O)
xk+1 (u(O)
k+1
(1- (())
M-R-1
2
2
Consequently, we set v(S) =
00)) 2
< 1 and have
8-1
((OO)
&
m(O)
M-k
P(O E D)
Σ
OM
1
2
2
k=0
u((O)
S-1
u(O)
M-8+1
21 S-19 '
1
2
2
(O))
8-1
u(0)²
M
=1 S 1s
1
2-w(O)
2
(18)
21 C. (MM
=1 M x (M- 1) X x (M - S + 2)
Sv(S)
1 2 X ... (8 1)
21 M x (M - 1) x... (M- - -S- S x
S
21 M M(S)
If we fix S, with the increasing of A, AS MS will fast converge to 0 as 0 < v(S) < 1. Moreover, the right part of the first line of Eq. 18 is always larger than 0 when M > S (the sum of these combination numbers is always less than 1 when M > S). Thus, we have proven Theorem 4.1.
Below, we will discuss the situation when the state space and action space are continuous. As the optimal state distribution with the maximal state entropy is somewhat uniform, in continuous environments, there may still not be simple unimodal distributions. For example, assume a one-step MDP, we only take one action from [0, 1]. and the state is the same as the action, thus the optimal policy should be the uniform distribution over [0, 1] rather than simple unimodal distributions.
19

---

Published as a conference paper at ICLR 2026
B.2 THE PROOF OF THEOREM 4.2
Below, we first analyze our fine-tuning objective Eq. and then prove Theorem 4.2. Assuming Po is the original state distribution of the MDP M, have
3 BanBad,
J+(T) [TT)
1-7 Band,
E Barde (aw(( [R(s, a) ( 1-7
Σ
(R(s;, a:) ( 80 = 8, a0 = a
1=0
(19)
Ba-po
R(s,a) [DKx((((())]]
+
Σ i=1
n2 (B)sss.a)l)D
E - (x, a) - (allaa-BBBBBaa (D
here we set
(+(8,)) = R R(s.a) Σn ((s log <aa;ss)) (a, (s:)
i=1
(20)
= E R(s a)
Σ r' (R(s;: - (x...(((-l))))lllllll
As discussed in Sec. 4.2, ExDM applies the following alternative optimization method:
n (-18) = arg max E. ( (8, - (=)))
(21)
On =
Now we prove that T. manaaaa 8) I
Z(5] z(zz (alaa More generally, we define
F(T,"'.0 Eawn((ls) [Q (s, a) (
(22)
Using the calculus of variations, we can calculate the optimal point x- of F satisfying that Q=1(8, a) B log T- (a|s)
+ b3,,
(23)
(als))
here is constant related to n², and have (aass) Ta(als)e nr (als)da 1, we can calculate that
As
log Ta(als)e da, x'(as)) Ta (a(ass) (in))
(24)
ger(n.a)
J Ta(als) da
i.e. have arg max, F(T, na( 9ar and thus mn(als)
Below will prove Theorem 4.2.
Proof. Based on the definition of F, we have Ji(x) = Ea-no F(T,, ग, 8). Thus we require to prove Egmpo F F(TT, Tn, s) 2 Egmpp Caop,,F(Tu F(Tn-1: Tn-1. s). As we have discussed above,
Ta(+ss) = arg F(T,, 8) Z(s) (a(a)s) Q - (s.a)/8
x
(25)
F F( (
20

---

Published as a conference paper at ICLR 2026
In other words, we have proven that Eg F(Tn: Tn-1.8) 2 Ee F (In-1: 8). Moreover, we have
=R(s, (sa a) + E y (R(s;, - B (1
su = =, ao = a
=R(8, a) - -BylEa, (Tn
?Bay [0 (81,
=R(s, a) + +R,,, F
(26)
Thus
(x, Q
=ylBes [F(s 81) - nn -1-1 2 yle, [F(Tns Tni Tn, F(Tn, mn
=y|Ee, BaInx [Q,, (81, a1) 3DkL ( (-15)) [aa]]]aa(((s)))
(27)
QF (81, an + (DR.. (+181) [Ta()
=, Ea
[TT, (81,
(51, a1) - Qx (51. a1]]
Given the property of dx that dxx (8) - (1 - ()) = = drr da 7(al))l a) (Ying al., 2022), have
E gndd avva)()s) [Q+, (8, - Q= (8,
21 E Bandon (-)) R2, Bajns, [QT,, (81, a1) - QT (81,
/ (dan (81) (1 (s))) Barra 10t, (81, a1) Q. (81, dsi
(28)
EBy Md ar aaiaann (Q+,, (81, a1) Q:n (81, a1)l
-(1 )) Bar 1T., (81, 01) Q+.-1 ($1, a1)
Consequently,
E Eanpo F ((TT Tn-1, - Eanpo P kn
(29)
-B Barmplaiiaaaall 10+, (81, a1) Q. (81, a1) 20
Finally, we have
J((Ta) = Es-ro F (Tn, nn, 8) 2 Ea-po F(Tns In 8) 2 Es-po F (In-1,
J (n 1))
(30)
Thus, policy iteration improve the performance. Moreover, under some regularity conditions (following Haarnoja et al. (2017; 2018), for example, considering the policy mapping arg max. F'(T, x', 8) and assume that a compressed mapping with fixed point), म. converges to To.. which is the fixed point with the maximal Jr. Since non-optimal policies can be improved by our iteration, the converged policy Too is optimal for Jj.
-
21

---

Published as a conference paper at ICLR 2026
C ANALYSES AND DETAILS OF EXDM
Below, we discuss more analyses and details about ExDM, including analyses of Recore analyses of the objective Jr. details of the Q function optimization, and details of diffusion policy optimization of ExDM during the fine-tuning stage.
C.1 ANALYSES OF THE SCORE INTRINSIC REWARD Reccore
is an important result stemming from diffusion models that the KL divergence between the original distribution and the diffusion model estimated distribution. In detail,
LYLB BRg log g(81.T|80) 80) = log p( ) log ((11T(80)) ((
p(so:r)/((s))
(31)
-- log p(ogp(0)))) (E) log ( so) = logp(so) + DKL Dkrfplssirloollleeennra ((00:T 80)
i.., logp(sg) + LVLB = DKL (ol.p(ee
Consequently, the gap between LVLB and - log p(50) is the KL divergence between the original distribution and the diffusion model estimated distribution, which will be controlled when training the diffusion model. Then LVLB can be simplified maximize the state entropy during the training of the diffusion model.
score intrinsic reward Rycore. Thus, maximizing the score intrinsic reward can
C.2 ANALYSES OF THE OBJECTIVE Jr WHEN FINE-TUNING DIFFUSION POLICIES
The final objective of the fine-tuning stage is maximize J of the policy æ within limited steps, and may use any online RL methods like DDPG (Lillicrap, 2015), PPO (Schulman et al., 2017), and so on.
Unfortunately, when fine-tuning diffusion policies, directly using existing RL methods may en- counter many challenges. For example, in the diffusion policies, the log p is hard to estimate (while log p is important in policy-gradient-bue based methods), and taking action from the diffusion policies requires the multi-step forward process of the neural network (so is unstable to update the policy via (s), like DDPG). There several diffusion RL methods that hope to address these issues, like DQL (Wang et al., 2023), IDQL (Hansen-Estree DIPO (Yang al., 2023a), and so on. And have included these methods baselines in Fig. 3(c).
et al., 2023), QSM (Psenka et al., 2023),
Given that the fine-tuning stage can only access the limited steps, we directly consider the optimal policy
obtain, i.e., arg max, Jr(m), and hope to directly sample from the arg max, (T) without using methods like policy gradient. (Thus, here the 3 implicitly relies on the fine-tuning steps: if the fine-tuning step infinity, the optimal we can access is arg max,, (TT) i.c., 0: the fine-tuning step is O, the optimal we can access is Ta. i.c., 3 = xc)
Given Jr. our results demonstrate that arg max, (TT) the form of x 0/ This also provides another important insight: the more fine-tuning steps optimal Q function, and at this point, we should choose a smaller beta.
take, the closer learn to the
Thus, the core idea of ExDM is: learning the Q function, then utilizing guided sampling techniques to sample from 2/0 avoiding calculating log (a]ss Vano(s), which are difficult to timate in diffusion policies. In practice, we find that distilling the pre- pra-trained policy as well as the learned Q function into the fine-tuned policy can further improve the performance (this training is similar to the supervised training of the diffusion model, thus it is stable).
22

---

Published as a conference paper at ICLR 2026
C.3 Q FUNCTION OPTIMIZATION
For the Q function optimization, we choose to use implicit Q-learning (IQL) (Kostrikov et al., 2022), which is efficient penalize out-of-distribution actions (Hansen-Estruuch al., 2023). The main training pipeline of IQL is expectile regression, i.e.,
Vo(s))), min Lo( -...,.'' [||((s a) +V(((s)) a))
min Lv (<) = Es (45 -Vo(())) <
(32)
here L3 (u) = |r - 1u < 0)(u² and is a hyper-parameter. In detail, when 7 > 0.5, L3 will downweight actions with low Q-values and give more weight to actions with larger Q-values.
C.4 DIFFUSION POLICY FINE-TUNING
For sampling from n = Z Que we choose contrastive energy prediction (CEP) (Lu et al., 2023), powerful guided sampling method. First, we calculate the score function of TT as
V log n (a Va log (a(a))) + (
(33)
Moreover, calculate the score function म. at each timestep ie., Va, log म. (als), CEP further defines the following Intermediate Energy Guidance:
Qn-1(ss, ao),
t=
(8,
(34)
log E (aoc(a))) [c00-1(..0)11]] t
Then Theorem 3.1 in CEP proves that
T| (ar ()) XcTa aa ()) (s,as))
(35)
V log n. (aals) =Va, log (a s) + Vaen-l at).
For estimating Vact the following objective:
(s, az), CEP considers a parameterized neural network fo.-1 (8, at t) with
K Σ
eln 1/a00)/88
fo.. f 1)
min Ee (-))) 0.
log ΣØ (s, a.,
(36)
cun 1/x
2j=1
Then Theorem 3.2 in CEP (Lu 2023) has proven that its optimal solution for_, satisfying that Vollo (8, az,1 1) V E (s, az).
Consequently, we propose to fine-tune Va. log Tf (a:ls) parameterized as sw (a, t) with the fol- lowing distillation objective:
min E.a, (ar - cs(ar)s, t) fon ar, t)13
(37)
And the optimal solution 3 satisfying that eg (arls, t) the score function of ,. ie.. can sam- ple from cg. (a,(s,t) t) with any unconditional diffusion model sampling methods like DDIM (Song al., 2021a) or DPM-solver (Lu et al., 2022).
23

---

Published as a conference paper at ICLR 2026
D EXPERIMENTAL DETAILS
In this section, we will introduce more information about our experimental details. Sec. D.I, we first introduce all the domains and tasks evaluated in experiments. Then we briefly illustrate all the baselines compared in experiments Sec. D.2. Then Sec. D.3, we introduce the hyperpa- rameters of ExDM. Moreover, we supplement more detailed experimental results about maze2d and URLB in Sec. D.4 and Sec. D.5, respectively. The detailed ablation studies are Sec. D.6. And we finally report the computing resource in Sec. D.7.
D.1 DOMAINS AND TASKS
Maze2d.
This setting includes kinds of mazes: Square-a, Square-b, Square-c, Square-d, Square- tree, Square-bottleneck, and Squgre-large. These mazes are two-dimensional, and agents need to explore as many states as possible during the unsupervised pre-training stage.
Continuous Control. Our domains of continuous control follow URLB (Laskin al., 2021), including domains: Walker, Quadruped, Jaco, and Hopper, each with 4 downstream tasks from Deepmind Control Suite (DMC) (Tassa et al., 2018) (we plot each task of each domain in Fig. 5);
Walker is a two-legged robot, including downstream tasks: stand, walk, run, and flip. The maximum episodic length and reward for each task is 1000.
Quadruped is quadruped robot within a 3D space, including 4 tasks: stand, walk, rum, and jump. The maximum episodic length and reward for each task is 1000.
Jaco is 6-DOF robotic arm with a 3-finger gripper, including 4 tasks: reach-top-left (tl), reach-top-right (tr), reach-bottom-left (bl), and reach-bottom-right (br). The maxi- mum episodic length and reward for each task is 250.
Hopper one-legged hopper robot, including tasks: hop, hop-backward, flip, and flip-backward. The maximum episodic length and reward for each task is 1000.
Walker
Quadruped
Jaco
Hopper
stand
walk
stand
walk
top-left
top-right
hop
hop-back
run
flip
run
jump
bottom-left
bottom-right
flip
flip-buck
Figure 5: Illustration of domains with their downstream tasks in URLB (Laskin et al., 2021). We
consider 4 domains, and each domain has four downstream tasks.
D.2 BASELINES AND IMPLEMENTATIONS
We first introduce all URL baselines in our experiments.
ICM (Pathak et al., 2017). Intrinsic Curiosity Module (ICM) trains forward dynamics model and designs intrinsic rewards as the prediction error of the trained dynamics model.
RND (Burda et al., 2018). Random Network Distillation (RND) utilizes the error between the predicted features trained neural network and a fixed randomly initialized neural network as the intrinsic rewards.
24

---

Published as a conference paper at ICLR 2026
Disagreement (Pathak et al., 2019) The Disagreement algorithm proposes self-supervised al- gorithm that trains an ensemble dynamics models and leverages the prediction variance between multiple models to estimate state uncertainty.
LBS (Mazzaglia et al., 2022). Latent Bayesian Surprise (LBS) designs the intrinsic reward as the Bayesian surprise within latent space, i.e., the difference between prior and posterior beliefs of system dynamics.
DIAYN (Eysenbach et al., 2018). Diversity is All You Need (DIAYN) proposes to learn a di- verse of skills during the unsupervised pre-training stage, by maximizing the mutual information between states and skills.
SMM (Lee et al., 2019). State Marginal Matching (SMM) aims at learning policy, of which the state distribution matches given target state distribution.
LSD (Park et al., 2022). Lipschitz-constrained Skill Discovery (LSD) adopts a Lipschitz- constrained state representation function for maximizing the traveled distances of states and skills.
CIC (Laskin et al., 2022). Contrastive Intrinsic Control (CIC) leverages contrastive learning be- tween state and skill representations, which can both learn the state representation and encourage behavioral diversity.
BeCL (Yang et al., 2023b). Behavior Contrastive Learning (BeCL) defines intrinsic rewards as the mutual information (MI) between states sampled from the same skill, utilizing contrastive learning among behaviors.
CeSD (Bai et al., 2024). Constrained Ensemble exploration for Skill Discovery (CeSD) utilizes an ensemble of value functions for distinguishing different skills and encourages the agent explore the state space with a partition based on the designed prototype.
PEAC (Ying al., 2024). Pre-trained Embodiient-AAare Control (PEAC) analyzes the fine- tuning and pre-training objectives for the cross-embodiment unsupervised RL, resulting the cross- embodiment intrinsic rewards.
In experiments of URLB, most baselines (ICM, RND, Disagreement, DIAYN, SMM) com- bined with RL backbone DDPG are directly following the official implementation in urlb (https://qitie fer to the official implementation Inttps://git httt:p:://gtthbb..oo wttptttt...c..iiii....oo wwweeeemmtttnnggneeeeeee and combine it with the codebase of urlb. For CIC, BeCL, CeSD) and PEAC, also follow their official implementations (https://githe) wttm://--rsseerccchaooooo https: /tgp:///ithhs CeSD, https: : / github. com/tho mhhummmCEURL), respectively.
wwwccooo/riieeee http://1lssseeerrrrieeeeeeeee http://rrr_bbechhrkk..
For LBS,
com/Rooshy yang/ https: com/ htte/BBaccheeee Baichajiaa/
Below, we will list the diffusion fine-tuning baselines in experiments.
DQL (Wang al., 2023). Diffusion Q-Learning (DQL) inherited the idea of policy gradient and proposes to directly backpropagate the gradient of the Q function within the actions (calculated with the diffusion action by multi-step denoising).
QSM (Psenka et al., 2023). Q-Score Matching (QSM) proposes to align the score of the diffusion policy with the gradient of the learned function.
DIPO (Yang et al., 2023a). Diffusion Policy for Model-free Online RL (DIPO) utilizes the Q function to optimize the actions, ie., finding the better action with gradient ascent of the Q function, and then trains the diffusion policy fit the "optimized" actions.
25

---

Published as a conference paper at ICLR 2026
IDQL (Hansen-Estruch et al., 2023). Implicit Diffusion Q-learning (IDQL) considers to sample multiple actions from the diffusion policy and then select the optimal action with the learned Q function.
We implement these methods based on their official codebases: DQL luttps://githi) com/ httpeeoonnn.aaa//lffffoooooooooo github. htt.uu..ccc//llecconnreeeee.ee score http://sooeeettngeeeeeeaaae DIPO whttts:ttss///oooo / com/ Be BeL_mannimmffut/DDPP)) and IDQL (nttgs://github.ccmm http://ttt.bb.com//i.ipppppeeeee Decc/DDD
RL), QSM (https://
D.3 HYPERPARAMETERS
Hyperparameters of baselines are taken from their implementations (see Appendix D.2 above). Here we introduce ExDM's hyperparameters.
First, for the RL backbone DDPG, our code is based URLB (https: //gi: thub. com/ rll-research http//uuul_eeeemrrrreeeee and inherits DDPG's hyperparameters. For the diffusion model hyperparameters, we follow CEP (Lu et al., 2023). For completeness, we list all hyperparam- eters in Table 2.
DDPG Hyperparameter
Value
Replay buffer capacity
10
Action repeat
1
Seed frames
4000
n-step returns
3
Mini-batch size
1024
Seed frames
4000
Discount y
0.99
Optimizer
Adam
Learning rate
le-4
Agent update frequency
2
Critic target EMA rate TQ
0.01
Features dim.
1024
Hidden dim.
1024
Exploration stddev clip
0.3
Exploration stddev value
0.2
Number of pre-training frames
1x10 10 for Mazeld and 2 10 for URLB
Number of fine-turee tne-turning frames
1x for URLB
ExDM Hyperparameter
Value
Diffusion SDE
VP
0g of diffusion model
do t, 9o = 0.1,31 20
dy diffusion model
G:
MLPResnet Blocks, the same as IDQL (Hansen-Estren et al., 2023)
Diffusion model neural network
Optimizer
Adam
Learning rate Energy guidance model Sampling method Sampling step
le-4
4-yyer MLP, aidden.dim=256 the same as CEP (Lu et al., 2023)
DPM-Solver
15
Table 2: Details of hyperparameters used for Maze2d and state-based URLB.
26

---

Published as a conference paper at ICLR 2026
D.4
ADDITIONAL EXPERIMENTS IN MAZE
LSD
DIAYN
SMM
ICM
LBS
Disagreement
RND
Explored
MAnnoo Ceseeeiien volus
RE
MEPOL
CIC
BeCL
CeSD
ExDM
-10
Explored
DIAYN
Disagreement
10-3
Explored
RE3
MEPOL
CIC
BeCL
CcSD
ExDM
10
Explored
LSD
DIAYN
SMM
ICM
LBS
Disagreement
RND
L
UT
Explored
MEPOL
CIC
BcCL
CoSD
ExDM
10
Explored Ratio
Explored Ratio
LSD
DIAYN
SMM
ICM
LBS
Disagreement
RND
GD
Explored Ratio
RE3
MEPOL
CIC
BeCL
CeSD
ExDM
G
Figure 6: Visualization of explored trajectories by URL methods in Square-a, Square-b, Square-c, and Square-d maze.
27

---

Published as a conference paper at ICLR 2026
Explored Ratio
LSD
DIAYN
SMM
ICM
LBS
Disagreement
RND
Explored Ratio
MEPOL
CIC
BeCL
CcSD
ExDM
10-3
LSD
DIAYN
SMM
ICM
LBS
Disagreement
RND
Explored Ratio
RE3
MEPOL
CIC
BeCL
CcSD
ExDM
Explored Rañio
LSD
DIAYN
SMM
ICM
LBS
Disagreement
RND
10
Explored
MEPOL
CIC
BeCL
CeSD
ExDM
Explored Ratio
10
Figure 7: Visualizations of explored trajectories by URL methods in mazes Square-tree, Square- bottleneck, and Square-large, respectively.
Moreover, we include the visualization of all algorithms (ICM, RND, Disagreement, RE3, MEPOL, LBS, DIAYN, SMM. LSD, CIC, BeCL, CeSD, and ExDM) within all mazes: Square-a, Square-b, Square-c, Square-d, Square-tree, aguare-bottleneck. and Square- in Fig. 6 Fig. 7, respectively.
As shown in these figures, although baselines can explore unseen states and try to cover as many states as they can, the behaviors of baselines can not fully cover the explored replay buffer due their limited expressive ability. Using the strong modeling ability of diffusion models, the pre- trained policies of ExDM can perform diverse behaviors, setting a great initialization for handling downstream tasks.
28

---

Published as a conference paper at ICLR 2026
LSD
SMM
BS
RND
MEPOI
BeCL.
FxDM
DIAYN
ICM Square-a
Disagrecment
RE3
CIC
CeSD
Square-b
Square-e
444
State Coverge
03
04
06
04-
02
04
0.
00
108
100h Squaro-tree
801
130
801
304
129:
Square-d
Square-oottlencck
Square-Large
34-
04
12
03
0.
50
08
00
205
624
28
1065
40%
23V
60A
254
.06
BSC
104
1094
Timooleps
Timesleps
Timestops
Timnoolops
Figure 8: State coverage ratios of different algorithms in mazes during pre-training.
D.5 ADDITIONAL. EXPERIMENTS IN URLB
In Table 3. report the detailed results of all methods in the 4 downstream tasks of domains in URLB. both the Quadruped and Jaco domains, ExDM obtains state-of-the- performance in downstream tasks. Overall, there are the most number of downstream tasks that ExDM performs the best, and ExDM significantly outperforms existing exploration algorithms.
Dumenn Tisks
Waller
Quodlsped
Jana
Hupper
stand
walk
JBn
[lp
S1280
wxlt
jurp
1
bop
Iop bock
dip
Fip back
ICM
$28.5
628.8
223.8
400.3
292.9
123.9
92.1
145.5
56.5
51.7
54.3
$3.4
82.1
1625
106.9
107
XND
$78.5
245.4
348.0
454.1
792.0
544.5
442.2
6:2.0
96.2
110.5
101.0
81.3
267.2
132.5
181.0
Dhagpemenn
240.5
$21.9
210.5
340.1
590.8
352.3
361.9
427.9
142.5
135.1
1296
118.1
96-2
255.6
113.0
215.3
LBS
603.2
375.3
4130
255.3
303.8
366.5
166.5
129.6
105
303.0
416.0
MEPOL
298.5
SMM
2239
93.
APS
575.6
472.5
155.4
394.
484.5
315.5
387.8
351.3
34.0
40.0
29.2
LSD
770.2
532.3
167.1
5057
315.4
156.3
1796
253.5
11.6
33.5
22.5
6.7
13.0
6.6
24
12.3
DBC
$41.1
881.1
199.
687.2
788
157.
475
630.6
145.6
167
133.1
1659
81.7
191
96.3
161
B:CL
951.7
912.7
4086
605.2
796.7
694.9
301.7
645.5
114.2
152.2
137.
144.
31.8
58.3
73.6
142.7
CaNB
833
6309
896.
763.4
636.
399,1
115.7
131.1
1184
(OMM))
573.1
913.4
3623
755.1
1335
Table 3: Detailed results in URLB of different pre-trained methods that fine-tune Gaussian
policies with DDPG. Average cumulative reward (mean of 10 seeds) of the best policy.
Metries
Median
IQM
Mean
Optimality Gap
ICM
0.40, [0.34, 0.44)
0.38. [0.33, 0.42)
0.39, [0.35, 0.43)
00,1, 0.65]
0.59. [0.55, 0.63]
[0.56, 0.62]
0.59, [0.56, 0.62]
0.41, [0.38, 0.44]
Disagreement
0.54, [0.50, 0.58]
0.53, [0.49, 0.57]
0.57, [0.51. 0.577
0.46, [0.43, 0.501
LBS
0.48. [0.41, 0.53
0.45, 10.39, 0.51
0.47, [0.43, 0.52)
0.53, [0.49, 0.58)
RE3
0.46, [0.41,
0.46, [0.41, 0.51]
0.46, [0.42, 0.51]
0.54, [0.50, 0.58]
MEPOI,
0.48, [0.43, 0.53]
0.48, [0.42, 0.53]
0.48, [0.44, 0.53]
0.52, [0.48, 0.56]
0.29. [0.24,
[0.18, 0.30
0.29,
0.33]
[0.67, 0.75]
SMM
0.29, [0.23, 0.34]
0.2.1 16, 0.27]
0.29, [0.24, 0.33]
0.71, [0.67, 0.76]
LSD
0.21. 0.26]
0.14. [0. 0.19]
0.21, [0.18, 0.251
0.79, [0.75, 0.82)
CIC
0.66, [0.61, 0.70]
0.68. [0.63, 0.72]
0.66, [0.62, 0.69]
0.35, [0.31, 0.38]
BeCL
0.59, [0.54, 0.66]
0.62, [0.56, 0. 68)
0.60, [0.55, 0.64]
0.40, [0.36, 0.45]
0.67.
0.67,
0.37]
ExDM (Ours)
0.78.
0,80 [0.76, 0.84]
0.77, [0.74,
0.23, [0.20, 0.26]
Table 4: Aggregate metrics (Agarwal et al., 2021) with confidence interval in URLB For every algorithm, there are 4 domains, each trained with seeds and fine-tuned under downstream tasks, thus each statistic for every method has 160 runs.
29

---

Published as a conference paper at ICLR 2026
Moreover, further provide the detailed metrics (including the confidence interval) of all methods within URLB in Table As shown here, ExDM significantly outperforms all baselines, for example, ExDM's IQM is larger than the second-best method, CeSD, by 13%.
In Table 5, we further report the detailed results of all methods in the downstream tasks of domains in cross-embodiment URLB, which is much more challenging as the algorithms require handling various embodiments. In both the Walker-mass and Quadruped-mant obtains state-of-the-art performance in downstream tasks. Overall, there are the number of downstream tasks that ExDM performs the best, and ExDM significantly outperforms existing ex- ploration algorithms.
domains, ExDM
Domains Tasks
Walker-mass
Quadruped-mass
stand
walk
run
flip
stand
walk
run
jump
ICM
665.3
418.0
146.2
246.5
460.2
229.5
215.6
323.5
386.7
253.8
563.7
589.5
Disagreement
549.3
331.6
139.8
250.0
555,5
372.4
329.8
506.
PEAC
823.8
499.9
210.6
320.5
786.0
754.5
388.3
645.6
DIAYN
502.1
245.2
106.8
212.7
682.7
484.3
371.0
469.1
SMM
673.5
509.2
220.7
329.6
357.0
176.4
189.7
277.8
220.7
327.7
762.5
BeCL
838.6
623.6
238.5
348.1
729.8
445.0
349.4
557.1
ExDM (Ours)
872.2
647.6
266.2
396.7
872.7
717.6
465.8
693.7
Table 5: Detailed results in cross-embodiment state-based DMC. Average cumulative reward
(mean of 10 seeds) of the best policy.
Metries
Median
IQM
Optimality Gap
0.42, [0.37, 0.48]
0.41, [0.35, 0.46]
0.42, [0.38, 0.47]
0.58, [0.53, 0.62]
RND
0.57, (0.53,
0.57.(0.53. 0.61]
0.58, [0.54, 0.621
0.42, [0.38, 0.46]
Disagreement
0.47, [0.43, 0.50]
0.45, [0.42, 0.48]
0.47, [0.43, 0.50]
0.53, [0.50, 0.57]
LBS
0.56, 0.60)
0.57]
0.55, [0.50,
0.45.
0.50]
PEAC
0.68, 10.63, 0.74
0.70, [0.64, 0.76]
0.6%, [0.63, 0.721
0.32, 10.28, 0.37|
DIAYN
0,46, [0.40, 0.51]
0.42,[0.35, 0.47]
0.46, [0.42, 0.50)
0.54, [0.30, 0.58]
SMM
0.44, [0.39, 0.52
0.45, [0,37. 0.531
0.45, [0.40, 0.511
0.55. 10.49, 0.601
CIC
0.67, [0.62, 0.73]
0.68, [0.62, 0.74]
0.67, [0.63, 0.71]
0.33, [0.29, 0.37]
0.73]
0.65, 0.701
[0.30, 0.40]
EXDM (Ours)
0.77, (0.73, 0.81)
0.80, [0.75, 0.83]
0.77, [0,74,
0.23, [0.20, 0.26]
Table 6: Aggregate metrics (Agarwal et al., 2021) with confidence interval in cross-embodiment URLB. For every algorithm, there are domains, each trained with seeds and fine-tuned under downstream tasks; thus, each statistic for every method has 80 runs.
Moreover, we further provide the detailed metrics (including the confidence interval) of all methods within URLB in Table As shown here, ExDM significantly outperforms all baselines, for example, ExDM's IQM is larger than the recond-best method, PEAC, by 14%.
D.6 ABLATION OF TIMESTEPS IN URLB
In Figure 9, we show additional results about the performance in four domains of URLB for different algorithms and pre-training timesteps. Overall, ExDM outperforms all methods, while CIC and CeSD are still competitive on some domains.
30

---

Published as a conference paper at ICLR 2026
BROUM SIIV
Elpont hormuta Stote
Expon Morms o2d Score:
Pretaring on 100ks
Prekaring on 500k 2010
BILOO BRUM
BUIUO LINOL WROOM
BIUIO BICU ILIVO (IRNCL WOOM
Water
Contapps
Ovend
Wake
Omei
Pretrahingg saggs
Phetrainnn un staps
Figure 9: Ablation study of pre-training steps in URLB.
D.7 COMPUTING RESOURCE
In experiments, all agents trained by GeForce RTX 2080 Ti with Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz. In maze2d / urlb, pre-training ExDM (each seed, domain) takes around 0.5 days, respectively.
E THE USE OF LARGE LANGUAGE MODELS
In the preparation of this manuscript, LLMs were used solely auxiliary tools for paraphrasing and polishing the writing to improve readability. No LLM was involved in formulating research ideas, designing methods, constructing datasets, implementing experiments, conducting analyses, or drawing conclusions. All scientific contributions and substantive content presented in this paper the original work of the authors.
31


Reviews:
[{"review_id": "53c2ofkTkh", "reviewer": "Reviewer_KGEL", "summary": "-The paper introduces the Exploratory Diffusion Model (ExDM) to address the exploration bottleneck in Unsupervised Reinforcement Learning (URL).\n\n-Unlike prior methods that use simple policies, ExDM leverages the superior expressive power of diffusion models to accurately model the complex and heterogeneous state distributions collected during exploration. A diffusion model ($\\epsilon_{\\theta^{\\prime}}$) is trained on the replay buffer's state distribution. A novel score-based intrinsic reward ($R_{score}$) is calculated from this model's loss (its inability to fit a state), which guides the agent to under-visited regions. To ensure efficiency, a simple Gaussian behavior policy ($\\pi_g$) is trained to maximize this $R_{score}$ and is used for fast data collection, avoiding slow diffusion sampling.\nThe pre-trained Gaussian policy ($\\pi_g$) can be fine-tuned on downstream tasks using standard RL algorithms (like DDPG).", "questions": "Please, refer to the weakness and answer against my concerns.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The author addressed that this is the first work to successfully integrate diffusion models into the unsupervised exploration phase of RL. The concept of using the diffusion model's density estimation loss as the intrinsic reward is a significant contribution over prior reward mechanisms (like RND or ICM).\n- It was impressed that the decoupled training scheme (fast Gaussian actor, slow diffusion reward-calculator) is a clever and practical solution to the primary obstacle of using generative models in online RL: slow sampling speed.\n- I think that the paper provided a novel, non-trivial algorithm for fine-tuning the diffusion policy itself, complete with a formal proof of optimality (Theorem 4.2). This goes beyond just using the model as a static prior.\n- Overall, the method's superior performance is not marginal. Its experiments dramatically outperform all baselines in complex exploration tasks (e.g., Fig. 2, where baselines get stuck and ExDM covers the entire maze) and shows consistent SOTA results across all aggregate metrics in URLB (Fig. 3).", "weaknesses": "- There is a limitation in terms of performance gap: The paper's own experiments (Fig. 3) show that fine-tuning the simple Gaussian policy ($\\pi_g$) actually achieves better final performance than the proposed new, complex diffusion policy fine-tuning algorithm (Algorithm 2). The reason should be explained and analyzed intensively. Compared with Fig. 3(a) and (b), the expert normalized scores of the proposed algorithm in Fig. 3(c) were small. \n- The authors stated that the performance degradation may be due to limited interaction timesteps during fine-tuning. The advanced works to overcome this problem should be discussed further.\n- While their new fine-tuning method (Algorithm 2) is a novel contribution, it is not yet fully optimized and is outperformed by a simpler, standard approach such as DDPG. \nTherefore, it is expected that the paper's primary strength lies in its pre-training exploration (which produces a superior Gaussian policy) rather than its diffusion policy fine-tuning performance. This mechanism could be considered in this discussion of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yOpXf1VShZ", "reviewer": "Reviewer_ueyP", "summary": "This paper introduce the Exploratory Diffusion Model, which leverages the expressive power of diffusion models to fit diverse replay-buffer distributions, thus providing accurate density estimates and a score-based intrinsic reward that drives exploration into under-visited regions. This mechanism substantially broadens state coverage and yields robust pre-trained policies. Beyond exploration, ExDM develops an efficient decoupled training scheme and a fine\u0002tuning algorithm for adapting pre-trained diffusion components to downstream tasks under limited interaction, with theoretical guarantees of convergence and optimality.", "questions": "- How is the intrinsic reward designed, what is the underlying rationale, and were alternative design schemes considered?\n- What is the difference between unsupervised reinforcement learning and Meta-RL, given that both seem to aim at rapidly solving new tasks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Empirical gains across multiple settings: The figure indicates consistent improvements over strong unsupervised exploration baselines in URL, in cross-embodiment transfer, and when initializing diffusion policies.\n- Potentially general mechanism: A diffusion-based exploratory prior could be a broadly applicable way to induce diverse skills or state coverage that helps downstream RL fine-tuning and transfer.\n- Sufficient theoretical proof.", "weaknesses": "None", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aEYY6t4wCF", "reviewer": "Reviewer_dZCp", "summary": "The authors proposed an unsupervised RL algorithm called ExDM with diffusion action head.", "questions": "1. The results on URLB seems to be promising, but why APT and APS by Liu et al. were not included in the baseline?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The performance seems to be very strong compared to baselines\n2. The presentation is clear and easy to follow", "weaknesses": "1. The motivation is somewhat weak. I'll put my questions in the following section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GqbyxJneLf", "reviewer": "Reviewer_UWWM", "summary": "This work proposes a novel approach called ExDM for unsupervised RL using a diffusion model. During pre-training, ExDM trains a diffusion model to model the state and action distributions of interactions with the environment and derives an intrinsic reward to encourage exploration that is inversely proportional with approximate probability of state visitations of the diffusion model. Using these intrinsic rewards, the approach trains a Gaussian policy that explores the environment. During fine-tuning, the Gaussian policy can be trained with task-specific rewards, or the diffusion policy trained during pre-training can be fine-tuned. To enable fine-tuning of the diffusion policy, a novel regularized training objective is being derived, similar to soft RL and using implicit Q-learning from the offline RL literature to avoid out-of-distribution actions being sampled. The new approach is shown to lead to more exhaustive exploration, as measured by state coverage, during pre-training, and the work shows that fine-tuning of the pre-trained Gaussian and diffusion policies lead to higher performance compared to alternative pre-training approaches.", "questions": "1. What are the benefits of the diffusion policy of ExDM, or does it not contribute to the performance of the system (given the fine-tuned Gaussian policy performs better)?\n2. Would the authors be able to elaborate on their statement that the diffusion policy might perform worse than the Gaussian policy \"[...] due to limited interaction timesteps during fine-tuning (Section 5.4). Were the diffusion policies fine-tuned with less than the 2M steps used for Gaussian policies? What other reasons might there be for the more expressive diffusion policies to perform worse after fine-tuning?\n3. How were hyperparameters of all baselines and ExDM selected for the empirical Maze2D experiments?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "## Originality\nThe approach proposed in this work appears original and novel. While diffusion policies are not new, and diffusion models have been used to express various data distributions, their application to URL is novel to the best of my knowledge. Furthermore, the theoretical contributions in Theorem 4.1 justifying the need for more expressive policies for unsupervised pre-training, and in deriving a novel algorithm for online fine-tuning of the diffusion policy are valuable to the community.\n\n## Significance\nAs stated, I consider the theoretical contributions of this work significant and valuable to the community. Similarly, the empirical results indicate a small but consistent improvement of ExDM compared to the strongest URL baselines. Assuming these results were generated under fair hyperparameter tuning (see question 3), they demonstrate that ExDM is a significant contribution to the field.\n\nOne aspect that dampens my otherwise very positive impression of the significance of this work is the unclear benefits of the diffusion policy part of the ExDM algorithm. The diffusion policy is arguably the biggest and most complex novel contribution of this work, but it appears to not contribute meaningfully to the performance of the approach (see Weakness 1.). Without the diffusion policy, the ExDM algorithm could have also \"just\" been a diffusion model of the state distribution to derive a slightly novel intrinsic reward with a Gaussian policy. I would expect further discussion with reviewers and the authors to clarify that part of this work.\n\n## Quality & Clarity\nI find the writing and presentation of this work of a high quality. The empirical evaluation also appears to follow good practice, and provides further ablations and analyzes to shed more light on the learned components. There are few unclear or not well supported statements in this work that are listed below, but none of them are major issues or central to the work.", "weaknesses": "Below, I provide a list of weaknesses that should be addressed. I would consider all weaknesses stated as **major** important to address in order to justify acceptance.\n\n1. **(Major)** The introduction of this work significantly leans into the motivation that typical URL approaches use policies that are not sufficiently expressive (often discrete or Gaussian policies) to properly explore the environment during pre-training. Theorem 4.1 as theoretical contribution of this work further supports this narrative and the pre-training and fine-tuning of the diffusion policy component of ExDM takes over large parts of Section 4. However, despite this motivation and more expressive diffusion policy, the environment interactions during pre-training are still done only using the Gaussian policy (as per line 8 in Algorithm 1). Furthermore, fine-tuning of the Gaussian policy of ExDM still leads to higher performance than fine-tuning the diffusion policy (see Figure 3 (a) vs (c)), a fact that is acknowledged by the authors in Section 5.4. \n   All this makes me question what the diffusion policy of ExDM truly adds to the method. It appears the benefits of ExDM are not from training a more expressive policy in the diffusion policy, but from the diffusion model of state distributions that appears to provide a more informative intrinsic reward to exhaustively explore the environment. Theorem 4.1 still motivates the use of more expressive policies but it appears that the challenges of training diffusion policies remain to outweigh the benefits of a more expressive policy. My questions here are\n\t1. Does the diffusion policy of ExDM provide some benefits, or does it contribute to any component of the approach in a way that I have not seen?\n\t2. Or am I correct in my assessment that it appears to provide no benefits over the Gaussian policy so far?\n\t3. Given Theorem 4.1, one might expect the more expressive diffusion policy of ExDM to provide clear benefits. In Section 5.4, it is stated that the diffusion policy might perform worse than the Gaussian policy \"[...] due to limited interaction timesteps during fine-tuning\". Did you fine-tune the diffusion policies with the same 2M steps as used for Gaussian policy fine-tuning? If not, how many steps were used to fine-tune diffusion policies and why was a different number of fine-tuning steps used for these policies? \n2. **Major:** Appendix C.3 states that \"hyperparameters of baselines are taken from their implementations\". Were these hyperparameters explicitly tuned for the Maze2D tasks evaluated in this work? How about ExDM hyperparameters listed in Table 2? Were these tuned in any form for the Maze2D tasks? I would expect comparable effort to be spent on tuning hyperparameters across all approaches to have confidence in the empirical results presented in this work, and this should be clarified.\n3. The work makes the following two statements that are unclear, imprecise, or incorrect and should likely be expressed more precisely:\n\t1. \"[...] the optimal policy of standard RL is a simple deterministic policy\" (Section 4.1). It is unclear to me what \"standard RL\" means; to the best of my knowledge, the statement is only generally true in fully observable (single-agent) RL; partially observable or multi-agent environments might have stochastic optimal policies.\n\t2. \"[...] the explored replay buffer is always diverse and heterogeneous, as the policy continuously changes to visit new states during pre-training. Consequently, URL requires capturing the heterogeneous distribution of collected data and obtaining policies with high diversity\" -- I would argue that URL does not necessarily require to capture distribution of multiple policies to achieve its objective of learning a policy that maximizes state entropy (as stated in Section 4.1). It is merely the practical approach of using off-policy algorithms and a replay buffer of experiences that requires to capture such distributions.\n\t3. \"The Gaussian behavior policy $\\pi_g$ can then be trained using any RL algorithm\" -- I would argue that this is not true since off-policy RL algorithms would be needed. The experience samples that $\\pi_g$ is trained on, as per Algorithm 1, are from a replay buffer and, thus, off-policy with respect to the trained policy $\\pi_g$.\n4. The baselines visualized in Figure 2 appear to be mostly poor performing or middle of the pack when looking at Table 1. None of the strongest baselines (MEPOL, RE3, CIC) are included in Figure 2, supposedly to make the result of ExDM appear more impressive. I would appreciate Figure 2 would show the strongest 1-2 baselines in each family which appear to be R3 and MEPOL for exploration and CiC for skill discovery baselines. (Visualizations of all baselines are shown in Appendix C.4 but I would prefer for the most relevant ones to be shown in the main corpus of the paper)\n5. The fine-tune box of Figure 1 appears confusing to me and I believe the policy titles should be flipped. The left half appears to show the fine-tuning of the Gaussian policy and the right half the fine-tuning of the diffusion policy (as per plot and legend) but the red titles above them are reversed. \n6. (Minor) I noticed that baseline algorithms do not have identical colors in Figure 3 (a) and (b) which makes it slightly harder to cross-reference these results at a glance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)