Off-policy Reinforcement Learning with Model-based Exploration Augmentation
Authors: Likun Wang
Xiangteng Zhang
Yinuo Wang
Guojian Zhan
Wenxuan Wang
Haoyu Gao
Jingliang Duan
Shengbo Eben Li
Venue: NeurIPS 2025 poster ()
Keywords: Model-based reinforcement learning
Model-based Exploration
Generative model
World model
Primary area: reinforcement_learning
TL;DR: We propose MoGE, which enhances the Off-policy RL exploration by critical experiences generaion, leading to significant improvements in sample efficiency and performance ceilings across various tasks.

Abstract: Exploration is crucial in Reinforcement Learning (RL) as it enables the agent to understand the environment for better decision-making. Existing exploration methods fall into two paradigms: active exploration, which injects stochasticity into the policy but struggles in high-dimensional environments, and passive exploration, which manages the replay buffer to prioritize under-explored regions but lacks sample diversity. To address the limitation in passive exploration, we propose Modelic Generative Exploration (MoGE), which augments exploration through the generation of under-explored critical states and synthesis of dynamics-consistent experiences. MoGE consists of two components: (1) a diffusion generator for critical states under the guidance of entropy and TD error, and (2) a one-step imagination world model for constructing critical transitions for agent learning. Our method is simple to implement and seamlessly integrates with mainstream off-policy RL algorithms without structural modifications. Experiments on OpenAI Gym and DeepMind Control Suite demonstrate that MoGE, as an exploration augmentation, significantly enhances efficiency and performance in complex tasks.

Off-policy Reinforcement Learning with Exploration Augmentation
Likun Wang'
Xiangteng Zhang' Yinuo Wang'
Guojian Zhan' Shengho Eben Li
Wenxuan Wang'
Haoyu Gao?
Jingliang Duan 1.2+
School of Vehicle and Mobility & College of AI, Tsinghua University Sccooo of Mechanical Engineering, University of Science and Technology Beijing
Abstract
Exploration is fundamental to reinforcement learning (RL), as it determines how effectively an agent discovers and exploits the underlying structure of its environ- ment to achieve optimal performance. Existing exploration methods generally fall into two categories: active exploration and passive exploration. The former intro- duces stochasticity into the policy but struggles in high- -iggh-immnsional environments, while the latter adaptively prioritizes transitions in the replay buffer to enhance exploration, yet remains constrained by limited sample diversity. To address the limitation in passive exploration, we propose Modelic Generative Exploration (MoGE), which augments exploration through the generation of under-explored critical states and synthesis of manamics-consiseer experiences through transition models. MoGE is composed of two components: (1) a diffusion-based generator that synthesizes critical states under the guidance utility function evaluating each state's potential influence on policy exploration, and (2) one-step imagina- tion world model for constructing critical transitions based on the critical states for agent learning. Our method adopts modular formulation that aligns with the principles of off- off-policy learning, allowing seamless integration with existing algorithms improve exploration without altering their structures. Empirical results on OpenAl Gym and DeepMind Control Suite reveal that MoGE effectively bridges exploration and policy learning, leading to remarkable gains in both sample efficiency and performance across complex control tasks.
Introduction
Reinforcement learning has demonstrated remarkable potential across various tasks, including autonomous driving, large language models, game playing, and embodied artificial intelligence 667. It optimizes policies through trial and error, with policy performance fundamentally relying on the diversity and coverage of samples collected during interaction with environments.
68
[T
65
49
19
66]
69 21). Similar imitation learning (IL., RL algorithms face diitrr-distribution (OOD) challenges due to limited diversity in training data. However, RI. mitigates this issue by leveraging its exploration capabilities reach unvisited regions of the state space. In this context, enhancing exploration strategies to collect diverse samples and achieve broader state space coverage becomes crucial for improving the effectiveness and generalization of RL algorithms [25] 3
18
Conventional exploration strategies can be broadly classified into two categories: active exploration purely based policy, and passive exploration based on states. Existing approaches achieve active exploration by introducing randomness into the policy [46] [6] or adding exploration bonuses, which help prevent the policy from converging prematurely to narrow subset of actions (2). Specifically,
Corresponding author <duanji @ @uusbb.du.cnt <lishbo @ isinghua.ccu..ns>
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

methods like SAC 22 and DSAC B3 leverage maximum entropy as exploration bonuses to encourage exploration, while algorithms such as MPO [I] employ multimodal policies to maintain action diversity and stochasticity. However, the active exploration mechanisms are inherently limited by the agent's actual interacted trajectories, which are constrained by the environment's 's initial states, finite episode lengths, and the agent's current policy 44 As a result, many critical regions distant from typical rollouts or rarely visited often remain unexplored, reinforcing value estimation errors and narrowing the scope of policy learning [34 [5]. Furthermore, forcing exploration through the policy can divert focus from reward optimization, leading to suboptimal learning [8]
In contrast to active exploration, which relies on policy-driven interactions, passive exploration modi- fies the sample distribution of the agent to incorporate prioritized information. Passive exploration originates from Prioritized Experience Replay (PER) [53], which enhances learning by selectively reusing valuable experiences but remains fundamentally limited to previously collected samples. To address this restriction, certain approaches like SER (38) and PGR [64] leverage generative models to augment the replay buffer, thereby artificially intensifying the state-action distribution. However, this expansion is still confined to the vicinity of observed data, adhering closely to the original data distribution. Other methods integrate world models with current policies to simulate state transitions, potentially generating higher-quality transitions 12 2 24]; nevertheless, these generated samples typically exhibit limited diversity due to their strong dependence on the policy-conditional transitions originating from existing states. Moreover, generating complete transitions in high- -itheeinnniinal state spaces further amplifies the bias introduced by synthetic data, as the complexity grows exponentially with dimensionality [63]
To address the issues above in policy exploration, we propose MoGE, novel exploration paradigm that enhances off- off-po RI. algorithms by generating critical transitions across the entire state space, guided by exploratory priors. It consists of two components: a generator that produces critical states with high exploratory potential for the current policy and value network, and a dynamics model that simulates one-step transitions, enabling the generation of critical transitions. Specifically, our work makes three main contributions: (I) We employ a conditional diffusion generator to sample critical states with high exploratory potential. To guarantee the state-space compliance and feasibility of the generated states, theoretically prove that the state distribution in the replay buffer asymptotically converges to the stationary occupancy measure of the optimal policy. By continuously fine-tuning the generator on the replay buffer, we ensure that its learned distribution shares common support with the optimal policy's 's occupancy measure, generating critical states in a compliant way. (2) To guarantee the dynamical consistency of the generated samples, we design a one-step imagination world model to imitate the dynamics of the environment. This world model allows for efficient pre-training through supervised learning, supporting the construction of training experiences and designing the classifier of the conditional diffusion-based critical state generator. (3) We propose an off-policy RL training framework that integrates MoGE seamlessly into existing algorithms without requiring any modifications to their original structure. By introducing importance sampling that mixes critical transitions generated by MoGE with replay buffer samples, MoGE enhances exploration. leading to improved performance and sample efficiency. Experiments on standard continuous control benchmarks, including OpenAl Gym 5 and DeepMind Control Suite [61] demonstrate that MoGE, as plug-in module, consistently improves both the final performance and the sample efficiency of baseline off-policy RL algorithms.
2 Preliminaries
2.1 Reinforcement Learning and Policy Exploration
In RL, an agent interacts with an environment modeled as a Markov Decision Process (MDP) 660 defined as M = (S, A, P. r, 7), where S and A represent the state and action spaces, P: S x A S denotes the environment's transition dynamics,
S x A + RR is the reward function, and 7 E [0.
is the discount factor 33 To formally describe the target of RL, we introduce the occupancy measure d= (s), which is defined as d= (s) = (1 7) E PIss = s T)). This occupancy measure represents the visitation frequency of a given state under the policy π B2]. Likewise, we can define the dd (s, a) = d= (s)T (als), which represents the visitation frequency of given state-action pair (s, a). The target of policy is to maximize the expected cumulative reward, which can be expressed in terms of the occupancy measure as J(T) = (e (e.a)(r( (r(s, a)). Meanwhile, a value function is trained to estimates the value of the current policy by minimizing the temporal difference (TD) error,
2

---

which is formulated as J(T) ((s.a)aud (s.o)
(QT - .d))) where (" a)
denote the value function that evaluate the quality of given (s, a). Through this formulation, the actor optimizes the policy towards higher cumulative rewards, while the critic evaluates and stabilizes the learning process by minimizing prediction errors.
Effective policy training depends on discovering policy-improving states. Since such critical states often require active exploration, identifying them under the current policy enables targeted updates that accelerate learning. If these states can be estimated, they can be selectively replayed for more efficient updates. In existing RL frameworks, two common metries are used to quantify their criticality:
Policy entropy. The policy entropy reflects the randomness of action selection at given state. High entropy may indicate either insufficient visitation, offering high information gain, or proximity to critical decision points in the MDP where small action changes lead to divergent outcomes [41 14]. Focusing learning on such regions enhances policy robustness and long- nanghhorzon decision-making. For example, in the case of a Gaussian distribution, the utility function is defined as:
f(s) (((=((1s)) log (onr)()))) det E(s))
(1)
where x-1ss) = (1; u()) I(s)) while т and e denote the constant.
TD error. High TD error states critical for exploration, they highlight regions where the value function poorly approximates returns under the current policy [5]. These states often correspond areas of high uncertainty insufficient optimization. Prioritizing them helps reduce value bias, correct suboptimal actions, and improve policy robustness. The TD error for a given state under the current policy can be estimated as:
f() = 5.(s) Rs r(s, (o()) + Qu To(s)), x mo(s)))))) (s,
2.2 Diffusion Models for Generative Tasks
Diffusion models have emerged as effective generative models due to their ability to capture complex data distributions 29 54 [55] Inspired by non-equilibrium thermodynamics, they simulate two
complementary processes: a forward diffusion process incrementally adding noise to the original data, and reverse denoising process reconstructing the data distribution from noise. In the forward diffusion process, an initial data sample ~ ((s) is gradually transformed into Gaussian noise ST by iteratively applying:
(8)(+1-1) = N(s;; v 1,,1
where 3 controls the noise schedule. Letting ar = 1 - Be and āz = II ag the true reverse distribution conditioned on sn is given by:
1 at
ōt-1 3,1
g(st-1 sp) = N 81-1: Var
8t
(4)
VI
Since so is unknown during generation, diffusion models approximate this posterior with parame- terized neural network (8,, t) to predict the noise term G. To align with the real reverse process, the approximate reverse process is expressed
Pe (8 =N 84-1;
1 Vor
8t
Be
())
āt-1 B.1)
(5)
VI - a.
The training objective is to minimize the noise prediction error:
Lyenerator Esp.ct [||
( Voiso + v1 -1-045 1) 1|]]
c NNO, I),
(6)
enabling efficient data generation by progressively denoising from standard Gaussian noise.
3 Methods
In this section, we introduce the MoGE paradigm, illustrated in Figure In Section 3.1 we focus on the method and theoretical foundation of generating critical states through the policy model, as
3

---

Policy Improvement
Policy Evaluation
Entropy regularity
Max return
Actor
Critic
Min
TD-error
Policy-value
Gutdance
MoGE
Critical generation
One-step
Policy
inaging
(+)
(5,a)
(5,4,r,51)
Exploratory Replay
Replay Buffer
Figure I: Overview MoGE. MoGE is composed of two sub-modules: generator and a one-step world model. The generator produces critical states under-explored but potentially valuable for policy exploration under the guidance of policy and value function, while the one-step world model predicts the next state and reward to construct the transitions. The formulated exploratory can be mixed with real samples from the buffer to perform the policy improvement and evaluation.
well as the selection of guidance methods. Section 3.2 we introduce the structure and application of the one-step imagination world model. In Section 3.3.1 we first analyze the quality of samples generated by MoGE both novelty and dynamics consistency, and after that, we propose a training framework that integrates MoGE with existing off-policy actor-critic methods in Section 3.3.2
3.1 Critical State Generation with Steady Occupancy Measurement Alignment
State-based passive exploration leverages prior knowledge to uncover iolicy-improving states without active search, enabling more directed and sample-efficien policy training. However, when such states are not visited by the policy or stored in the replay buffer, they cannot be utilized for learning. Generative models overcome this limitation by enabling the synthesis of specific states once the model is trained. By leveraging advanced generative techniques such as diffusion and flow matching 37 17,, the model is capable of synthesizing high-fidelity samples that accurately approximate even intractable target distributions.
Motivated by the insight above. MoGE adopts classifier-guided diffusion model [10,, Pe (s|c), which serves as a utility-conditione generator to synthesize policy-improv sillyyiippoving states. The unconditional diffusion model learns the manifold of plausible states, while separately trained classifier provides directional guidance, steering generation toward high-utility regions. This guided synthesis bridges generative modeling and policy optimization, allowing the agent to access critical yet previously unvisited states. In this bussifier-guideed diffusion framework, the gradient used during inference combines the unconditional diffusion gradient with classifier-baset gradient. Specifically, imple- menting Bayes' formulation and denoting the classifier e by a continuous utility function f(s), the classifier-guided gradient can be expressed as:
(s: (st) ww (f
Diffusion gradient Clussifier utility gradient
where is the guidance scale, which controls the strength of this guidance, balancing between following the unconditional diffusion prior and aligning with the target utility function. By selecting an appropriate utility function f(s), this approach allows targeted control over state generation probability, enabling effective exploration high-value regions.
However, central challenge in training generative models within RL settings lies in determining an appropriate stationary target distribution. Since the data distribution RL inherently policy- dependent and evolves the policy updates, obtaining stable and realistic state distribution becomes

---

infeasible, thereby complicating the training of the generator. As result, arbitrary or poorly aligned state generation can lead to exploration inefficiencies and instability during policy updates. To address this, the generated states must align closely with an approximately physically grounded and stable occupancy measure, ensuring both state-space compliance and dynamical consistency. practice, we observe that the behavior policy in the replay buffer exhibits statistically diminishing variation over training, inducing an approximately stationary occupancy measure that can serve as a physically grounded target for the generative model. To formally establish this alignment, we first propose the following theorem as a theoretical guarantee.
Theorem 1 (Steady- State Occupancy Measurement Alignment Theorem). Let B(a s) be a behavior policy, It" (a s) be a specific static policy, and let vy(s) and d= (s) represent the state occupancy measures under ((a 8) and (a s), respectively. Assuming that the divergence between the policy and x² (8) gradually decreases over the course of training. and that the replay buffer has finite capacity following First-In- Worst-I--Firs--ur First-Out (FIFO) scheme with states sampled exclusively from the current policy interaction, the following convergence relationship holds:
lim TV(v((s), d" (s)) = =0. 8000
(8)
Proof. See Appendix A.2
By aligning the unconditional diffusion model's training distribution with the stable occupancy measure in the replay buffer, the critical state generator inherently guarantees generated states remain valid within the true state space since the conditional diffusion shares the same support with the unconditional one [TO] 56). This alignment ensures high-quality, stable critical state generation and enhances the reliability and efficacy of subsequent policy improvements. In this work, we introduce two utility functions below that satisfy these requirements and analyze how each facilitates policy exploration and learning. Since the MoGE is equipped with a transition model, which enables the computation and gradient propagation of certain utility functions that would otherwise be intractable. The two criteria are policy entropy and TD error (see preliminaries for a detailed introduction). During training, we use different utility signals for different tasks, which facilitates more effective policy learning and accelerates value function convergence.
3.2 Transition Imaging with One-step World Model
When the environment can not return the reward r and next state s' for arbitrary (s, a), the utility function ff cannot be directly evaluated without environment interaction. Therefore, we employ learned surrogate models-an a reward model r(s, a) and a dynamics model f(s a) approximate these quantities and provide differentiable estimates of and s'. To work in conjunction with the critical state generator, we introduce one-step imagination world model Fg to estimate the environment dynamics. Since we need to estimate the environment's 's transition and reward functions for arbitrary (s, a) pairs, we only require the world model be accurate for one-step dynamics under the current state and policy. Consequently, our world model diverges from conventional designs [2 in both structure and training approach. By focusing on one-step predictions, trades off long-horizon accuracy to ensure reliable transitions within the region characterized by the optimal policy's 's occupancy measure.
Our proposed one-step imagination world model Fs consists of five parameterized components to predict the following variables:
Representation:
2, go(ss)
Reconstruction:
8, ha(Ze)
Latent dynamics:
Ze+1 1
Reward:
F. = Res (2t, ar)
Termination prediction:
G. Colze, az).
The detailed structure of the world model is depicted in Figure of of
Departing from prior work that relies on probabilistic components (23. we find that implementing all modules of
Appendix C.I
the world model as deterministic networks suffices for effective performance. Except for the latent dynamics model, which uses a two-layer Transformer encoder [62] flexibly handle both sequential and single-step inputs, all other components are implemented standard MLPs. At each time step t,
5

---

Algorithm Off-policy RL training framework with MoGE
Input: Policy ne, critical-state generator Pip One-step world model F&z critic Q
Initialize: Te. Pe, Fe. Q+
1: // Pretraining
2: Interact with environment using random actions: (r, .c) env. step(random)
3: Store random transitions (s, Grand: s') into replay buffer
4: Update Føs by minimizing model loss worrkdmodel in (10) using random interaction data
5: Re-warmup the buffer B with the current target TTg
6: for each iteration do
7: // Data Collection
8: Initialize 8
9: Interact with environment: (r,8',c) +- env st).stt(p((ss))
Store transition (s, m,(s),,, into replay buffer B
11:
// Off-policy RL Learning
12:
Sample transitions = {(5r, ae Tt. s=+1)
13:
Update Fo.Pop by minimizing model loss Lworldmodel and Lgenerator in , using I
14:
Generate critical states &a by using classifier guriieeee generatorp, with utility function f(s)
15:
Generate full transitions with one-step imaging: (r, (r. Fa(Se (s.)))
16:
Combine critical transitions Te = {(se. r,, with r r -ure
Update Ta, Q with downstream algorithm using I (for Te, do approximate importance
sampling with X-mixture)
18:
end for
the observation s; is first encoded into latent representation Z, using an encoder network 9á. Given latent state in feature space and the executed action the world model outputs: (1) a prediction of the next latent state Zr+1. (2) the corresponding one-step reward Å and (3) the termination factor that evaluates whether the transition is terminated. During training, we treat the reward signal and termination factor as augmentations of the next-step state, and design separate loss functions through three output heads. This design simplifies the architecture while preserving the predictive capacity of the world model. The total training loss of the world model presented as:
Lwarldmodel
1ise sullz
life 1 rallz + at log +1 c.) log(1 à)
BT
reconstruction
reward
termination
+
B1 [g((((()411)) - (go(( or)ll2
(10)
dynamies
+
32l (00(804)))
sg (do (g)( (z)
representation
where 81 = 0.5, 32 = 0.1,(s;, ae Tt, $t+1)o:T are the transitions sampled from buffer B, and sg(.) ) is the stop-gradient operator. T' is the total length of the transition that depends the sample method, and B is the batch size. The model accommodates both sequence and single-step inputs, with = 1 indicates the single-step setting.
3.3 MoGE with Off-policy RL
3.3.1 Sample Quality of MoGE
As generative exploration framework, MoGE synthesizes novel transitions that extend beyond the replay buffer. In what follows, we examine two key questions that underpin its design and effectiveness:
(I) How do the generated transitions contribute to exploration and policy improvement?
(2) Will the generated transitions influence the policy improvement and evaluation of the algorithm?

---

(a) Novelty and Policy-Dependent Generation. Let D Dreplay denote the replay buffer induced by historical behavior policy B, corresponding to state-action distribution Preplay (s, a). Conventional passive exploration frameworks 48
38
64 intensify transitions from the replay buffer by reorder-
ing or imitating, remaining inherently constrained by 3. Consequently, the generated transitions (s, s') ~ cannot escape the coverage bias of the behavior policy. In contrast, MoGE explicitly constructs a generative distribution
Pgen(s, a) = (s) (a(a)s),
(11)
where (s) denotes a learned state generator guided under time-varying utility landscape f(s (s)).. The critical state distribution is refined as follows:
(s) x pp(()) exp(w (s, mo(s))),
(12)
Because f evolves alongside the policy and critic, the target distribution for generation continuously shifts the policy updates. This property induces continual novelty-Moo new, high-utility regions as the policy changes, rather than remaining tied to the historical behavior distribution. Therefore, MoGE effectively decouples generation from the replay buffer, enabling exploration guided by both the current policy and the estimated value function instead of fully resampling or imitating past experiences.
adaptively generates
(b) Dynamic Consistency and Bellman Validity. While novelty broadens exploration coverage, it must coexist with dynamical validity to ensure that generated transitions remain consistent with the environment's transition kernel Penv (s'[s, a). To this end, MoGE employs learned world model (s (s, a) enforcing the transition-level relation
(s, a') ~ Fo(s'(s. such that fen (s,a),
(13)
where Seny denotes the true environment dynamics. This mechanism ensures that the generated samples adhere to the underlying physical or causal transition relationship.
In contrast, methods that directly synthesize interpolate transitions (e.g.. via diffusion or latent interpolation) without explicit dynamics modeling can easily introduce spurious correlations among (s, s') (loss functions that imitate the replay buffer do not ensure that valid transition dependencies are preserved among samples). Such correlations violate the Markov property and lead to inconsistent TD estimates:
E(s,a,"))~ (r(es"))),,[[ssss a) + (V() * TVV(s),
(14)
where 7 is the Bellman operator. These off-manifold transitions break the Bellman consistency and yield biased policy evaluation. By leveraging the world model Fz. MoGE ensures that generated transitions satisfy the Bellman-connistent transition relation, thus preserving both physical and statistical validity.
(c) Summary. In summary, MoGE achieves continual novelty via evolving utility-guided generation that adapts the current policy, while maintaining dynamic consistency through model-based transition regularization. This synergy enables MoGE to explore beyond the behavior policy's 's support without violating Bellman validity, leading to more reliable and effective policy learning.
3.3.2 Training Off-policy Algorithm with MoGE
To enhance exploration alongside policy improvement and evaluation by introducing the MoGE- generated samples in an off- off-policy manner, we propose a training framework that integrates MoGE into existing off- off-policy policy RL algorithms. Taking Actor-Critic as an example, the training framework is illustrated in Algorithm| ! Notably, due to the existence of the critical generation. a distribution shift arises between the initial state distribution of the buffer and the generator. While this bias is negligible during policy evaluation, it must be addressed during policy improvement through Importance Sampling (IS). Since the distribution of the diffusion model cannot be explicitly represented, the importance sampling ratio for the initial state distribution intractable. Therefore, we employ a sample mixing method to approximate importance sampling under bounded error. The formulas for policy evaluation and improvement are as follows:
( = E (w(ss - 7 Q5s(s' ad]]]
(15)
LFIM(0) = (1 - A) Den (0s.,0) as) + X [
(((,0))
7

---

(a) Humanoid-walk (b) Humanoid-stanm
(c) Humanoid-run
(d) Quadruped-run
(c) Quadruped-walt
( Walker2d-v3
(g)
Humanoid-v3
(h) Ant-v3
ii) Halfcheetah-v3
i Swimmer-v3
DSACAMoGE
DIAC
DSAC-PER
DSAC+POR
TD3+MoGE
TD3-PGR
TD3+PER
TD3
Figure 2: Training curves benchmarks. The solid lines depict the mean performance, while the shaded areas represent the confidence intervals over three seeds. The first row corresponds to the training curves on the DeepMind Control Suite, while the second represents the results OpenAl Gym.
where D2 = (1-k) k)- + kDgen 0 I k < 1, and g(s, a) denotes the return in a single state-action pair (s, It worth noting that the error of this policy improvement method is controllable only when À is sufficiently small. The details of this approximation method, along with the selection of and X, are referred in the Appendix A.3
4 Experiments
4.1 Experimental Setup.
Baselines. We choose two widely used representative off-policy RL algorithms as our MoGE baselines: the stochastic policy algorithm DSAC [3] and the deterministic policy algorithm TD3 [6]. which achieve active exploration through exploration bonus with policy entropy and random noise injection, respectively. To further evaluate the performance of MoGE, compare it with passive exploration methods like PGR (64] and prioritized experience replay (PER) [48), where PGR is state-of-the of-the-art method in data augmentation that directly generates buffer samples using diffusion model that has been shown to significantly enhance downstream learning.
Benchmarks. We evaluate our method on diverse benchmark of 10 challenging locomotion tasks drawn from the DeepMind Control Suite (DMC) [6] and the OpenAl Gym [5]. The Gym Benchmark introduces a wide range of control tasks. For example, the Humanoid environment raises the difficulty with high-dimensional state and action spaces (376/17 state/action dims). The DMC tasks feature complex agents: humanoid tasks (67/24 state/action dims) and quadruped tasks (78/12 state/action dims) that demand sophisticated balance and coordination.
Implementation details. To validate the plug-in capability of the MoGE, we preserve the original off-policy algorithms without further fine-tuning. In this paper, the total training step size for all experiments is set at 1.5 million, with the results of all experiments averaged over 3 random seeds. All hyperparameters are aligned with standard implementations, and the configuration details are documented in the Appendix B
4.2 Experimental Results
All the training curves are shown in Figure[Jand the detailed results are listed in Table Our method, MoGE (with DSAC), consistently achieves superior Total Average Return (TAR) across a wide range of locomotion tasks. Despite the challenges introduced by high-dimensional state and action spaces as well as intricate dynamics, maintains exceptional stability and efficiency, highlighting its robustness and adaptability.

---

In the challenging DMC Suite tasks, MoGE demonstrates substantial performance enhancements over the original TD3 and DSAC algorithms. MoGE achieves an average Total Average Return (TAR) of 817.7, significantly outperforming the original DSAC method (568.5) by a notable +43.8 % In individual tasks, such as Humanoid-walk, MoGE reaches 891.7, remarkable +508.6% % improvement over the original DSAC (146.5). Similarly, TD3 with MoGE significantly surpasses original TD3, delivering a + +73.3% % improvement.
In the OpenAI Gym tasks, MoGE continues to exhibit exceptional performance. MoGE achieves an average TAR of 9135.5, surpassing the standard DSAC (8301.0) by +10.0%. Notably, MoGE sets new benchmark results across all evaluated enviromments. In Humanoid-v3, it attains a score of 12151.1. a substantial +16.8% increase over DSAC (10402.2). MoGE's consistent performance highlights its superior effectiveness, offering clear and significant improvements in both early-stage learning and final asymptotic returns compared to traditional TD3 and DSAC implementations.
Table 1: Total Average Return (TAR) on 5 DMC Suite tasks and 5 OpenAl Gym tasks. Mean + Std over 3 seeds. Bold best; Higher is better.
Environment
TD3
TD3+PER
TD3-PGR
TD3+MoGE
DSAC
DSAC+PER
DSAC+POR
DSAC+MoGE
Humanodd-ealk
120.1 + 106.8
34,4 1.2
2.
222 72.0
146.5 60,9
122.5 48.5
195.0 92.3
891.7 + 19.1
Humanoid stand
1.2
0,9
8.3 0.2
2.7
776.6 15.6
816.5 91.5
751.4116.2 16.2
907.5 6.9
Humanoid
8.7
2..2
271.1 35.5
223.4
Quadruped run
44.1
63.3 2.5
83.2 4 16.6
0.4
793.9 29.9
662.7 à: 162.6
717.9 107.4
824.3 à 19.0
Qundupad-wall
236.5 19.5
200.2 11.7
405.2 54.7
857.9 102.8
649.3 67.7
823.2 17.5
976.2 3.1
AVE. DMC
93.4+226.7
99.62 + 2,9
6.3
161.9 26
568.5-42.7
504.4 + 81.8
542.8 +-47.3
817.7 + 11.4
Walker2d-v3
5031.1 # 84.2
$253.7 2 206.1
6007:5
6082.8 606.7
6258.3 #3.3
63916 246.9
6501.1 87.33
6978.4 68.7
Humomoid-v3
9967.1 I 547..8
5203.9 e 33.9
5531.8 62.5
5885.8 e 38.1
10402.2 2 187.7
109.0 10363.6
11004.0 121.5
12151.1 à 35.4
Ant-13
6037,5 a 119.2
6134,7
212.5
3961.3 170.7
6369.1 265.7
76100 10.0
7637.0 27,1
7837.2
203.0
8176.6 44,9
Halichoetah-r3
7363.0 3666.4
3774.3 7438.1
9687.1
11167.3
17072.0 61.3
16913.0
70.2
17324.7
18064,9 459.6
Swimmzr-y3
5.3
131,8
134.1 2.8
137.3
1323
131.0
136.0 3.0
AVG.Gym
4906
4832.4
5464.4
5928.4 629.7
8 68.5
8535.6 92.7
4.3 Ablation
We perform three ablation studies to evaluate the impact of each core component in our framework:
Utility function for exploration. We compare the choice of utility function for different parts of updating, as illustrated FigureBar Compared to policy entropy. TD error more beneficial during policy evaluation. On the other hand, entropy plays a more significant role in policy improvement since high-entropy regions encourage broader exploration, while high TD-error regions may stem from inaccurate environment estimation, potentially leading to unreliable evaluations.
Guidance scale w. We chose different guidance intensities test the balance between the regions with high potential for exploration and feasibility guarantee, as shown in Figure 3b] When the guidance strength is set I, it effectively balances the generation of states with high exploratory value and feasibility, ensuring that the occupancy measure while maximizing exploration potential.
states maintain alignment with the optimal
Mix ratio X. We vary the value of À by testing from 0.1 to 0.5. Results in Figure 3c show that
= 0.2 stable performance across this range. When is too small, the policy fails to acquire enough critical states for effective exploration. Conversely, if À is too large, the discrepancy in the state distribution becomes neg-negli,,,,, which aligns with the discussion in Appendix A.3
5 Related Works
Active exploration. Active exploration methods explicitly modify policies to enhance exploration, which originates from the epsilon-greedy policy (60 39 46 16). Entropy-based strategies like SAC 21 DSAC 13 DACER 67 66]. and PPO [51] incorporate entropy terms to balance exploration and exploitation, preventing premature convergence and enhancing stability. More advanced explo- ration approaches explicitly encourage policy diversity. Count-based methods, such as pscudo-counts

---

piity
(a) Ablation study on the choice of utility function
(b) Ablation study on the guidance scale
(c) Ablation study on the
mix ratio
Figure 3: Ablation study curves. We select the Humanoid-run task in DMC Suite with high complexity to perform all ablation experiments.
4 and neural density models [43, estimate state visitation frequencies to incentivize exploration in under-sampled regions. Intrinsic motivation strategies, including RND (6) and ICM [45], generate exploration bonuses from novelty signals prediction errors. Bootstrap DQN 42 further introduces uncertainty- exploration by leveraging ensemble networks to identify poorly understood state- action pairs. Recent advances maintain exploration diversity through maximum entropy principles, and exploration-drive policy optimization [1[45 which directly augments policies for broader state coverage. These methods, while easy implement, often require complex parameter tuning and may struggle with scalability in high-dimensional spaces compared with MoGF.
Passive exploration. The policy update process not only depends on its optimization objective, but also on the samples collected for estimating the gradient. By shifting the distribution of replayed samples, the policy is able to cover wider range of states. Prioritized experience replay (PER) [48) dynamically adjusts the replaying frequency collected samples by TD-error. Some further research uses different metrics for experience replay. Rather than only selecting the samples, data generation is more flexible since it not only models the initial state but also captures its transition. Diffuser 30 utilizes a diffusion probabilistic model that plans by iteratively denoising trajectories. Synthetic experience replay (SER) 38 proposes a diffusion
47 57
approach to flexibly upsample an agent's collected experience. Prioritized generative replay (PGR) [64] generates
model--asee RL 200). These methods learns parameterized transition model of state and action and directly optimize the policy through imagined samples, such as Dreamer [22 23 24 and other variants
transitions under a given relevance function. Another paradigm of passive exploration
27 70 26 Our proposed MoGE method can generate critical initial states and their transitions, thus guaranteeing data compliance.
6 Conclusion
We proposed MoGE, a novel exploration paradigm that addresses the limitations of passive exploration in off-policy RL. It enhances exploration by generating critical states guided by exploratory priors and then estimates state transitions through world model, forming valid training samples that guarantee state-space compliance and dynamical feasibility. Experimental results on OpenAl Gym and DeepMind Control Suite benchmarks demonstrate that MoGE significantly improves sample efficiency and overall performance, validating the effecctivnness of this exploration paradigm. We believe MoGE establishes a new perspective for exploration augmentation in reinforcement learning, with significant potential for future improvements, such incorporating adaptive mechanisms for prioritizing critical state transitions or leveraging more expressive generative models beyond conditional diffusion to further boost exploration efficiency and policy robustness.
7 Acknowledgment
This research was generously supported by the Beijing Natural Science Foundation (L257002)). The core of this work was completed in the Intelligence Driving Lab (IDLAB) at Tsinghua University. We extend our sincere gratitude to all the lab members for their constructive feedback during this period.
10

---

References
[1] Abbas Abdolmalcki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum posteriori policy optimisation. arXiv preprint arXiv: htt1::8806.0922 2018.
[2] Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, and Dale Schuurmans. Understanding the impact of entropy on policy optimization. In International conference on machine learning. pages 151-160. PMLR, 2019.
[3] Chayan Banerjee, Zhiyong Chen, and Nasimul Noman. Boosting exploration in actor-critic algorithms by incentivizing plausible novel states. 2023 62nd IEEE Conference on Decision and Control (CDC), pages 7009-7014. IEEE, 2023.
[4] Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
[5] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv: arXiv:1606.00 1600.0166600000 2016.
[6] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: 12894, 2018.
[7] Chang Chen, Yi-Fu Wu, Jaesik Yoon, and Sungjin Ahn. Transdreamer: Reinforcement learning with transformer world models. arXiv preprint arXiv:2202.088 09448, 2022,
[8] Eric Chen, Zhang-War Wei Hong. Joni Pajarinen, and Pulkit Agrawal. Redeeming intrinsic rewards via constrained optimization. Advances in neural information processing systems, 35:4996- 5008, 2022.
[9] Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized ensembled double q- learning: Learning fast without model. arXiv preprint arXiv:2101.05988 2021.
[10] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[11] Oguzhan Dogru, Junyao Xie, Om Prakash, Ranjith Chiplunkar, Jansen Soesanto, Hongtian Chen, Kirubakaran Velswamy, Fadi Ibrahim, and Biao Huang. Reinforcement learning process industries: Review and perspective. IEEE/CAA Journal of Automatica Sinica, 11())283--000 2024.
[12] Jingliang Duan, Yang Guan, Shengbo Eben Li, Yangang Ren, Qi Sun, and Bo Cheng. Distribu- tional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors. IEEE transactions on neural networks and learning systems,
6598, 2021.
[13] Jingliang Duan, Wenxuan Wang, Liming Xiao, Jiaxin Gao, Shengbo Eben Li, Chang Liu, Ya-Qin Zhang, Bo Cheng, and Keqiang Li. Distributional soft actor-critic with three refinements. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
[14] Benjamin Eysenbach and Sergey Levine. If maxent rl is the answer, what is the question? arXiv preprint arXiv: 191.019110001133.. 2019.
[15] Sebastian Flennerhag, Jane X Wang, Pablo Sprechmann, Francesco Visin, Alexandre Galashov, Steven Kapturowski, Diana L Borsa, Nicolas Heess, Andre Barreto, and Razvan Pascanu. Temporal difference uncertainties as a signal for exploration. arXiv preprint arXiv:2010.02255, 2020.
[16] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.
[17] Zhengyang Geng, Mingyang Deng, Xingjian Bai, Zico Kolter, and Kaiming He. Mean flows for one-step generative modeling, arXiv preprint arX1v:2505.13447,200 2025.

---

[18] Ivo Grondman, Lucian Busoniu, Gabriel AD Lopes, and Robert Babuska. A survey of actor- critic reinforcement learning: Standard and natural policy gradients. IEEE Transactions on Systems, Man, and Cybernetics, part C (applications and reviews), 42(6):1211-1307, 2012.
[19] Agrim Gupta, Silvio Savarese, Surya Ganguli, and Li Fei-Fei. Embodied intelligence via learning and evolution. Nature communications, 12(1):5721, 2021.
[20] David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv:1803.101222 2018.
[21] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. Pmlr, 2018.
[22] Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprini arXiv:1912.01603, 2019.
[23] Danijar Hafner, Timothy Lillicrap. Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020.
[24] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.cm 200::0000.00044, 2023.
[25] Seungyul Han and Youngchul Sung. Diversity actor-critic: Sample-aware entropy regularization for sample-efficient exploration. In International Conference Machine Learning, pages 4018-4029. PMLR, 2021.
[26] Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpc2: Scalable, robust world models for continuous control. arXiv preprint arXiv:2310.16822 2023.
[27] Nicklas Hansen, Xiaolong Wang. and Hao Su. Temporal difference learning for model predictive control. arXiv preprint arXiv:2203.0444 04955, 2022.
[28] Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto, Takashi Onishi, and Yoshimasa Tsu- ruoka. Dropout q-functions for doubly efficient reinforcement learning. arXiv preprint arXiv:2110.02034. 2021.
[29] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[30] Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205. 099999, 2022.
[31] David Janz, Jiri Hron, Przemyslaw Mazur, Katja Hofmann, José Miguel Hernández-Loobttt and Sebastian Tschiatschek. Successor uncertainties: exploration and uncertainty in temporal difference learning, Advances in Neural Information Processing Systems, 32, 2019.
[32] Romain Laroche and Remi Tachet Des Combes. On the occupancy measure of non- mon-markovian policies in continuous mdps. In International Conference on Machine Learning, pages 18548- 18562. PMLR, 2023.
[33] Hojoon Lee, Dongyoon Hwang, Donghu Kim, Hyunseung Kim, Jun Jet Tai, Kaushik Subrama- nian, Peter R Wurman, Jaegul Choo, Peter Stone, and Takuma Seno. Simba: Simplicity bias for scaling up parameters in deep reinforcement learning. arXiv preprint arXiv:2410.09754, 2024.
[34] Donghao Li, Ruiquan Huang, Cong Shen, and Jing Yang. Near-optimal conservative exploration in reinforcement learning under episode-wise constraints. In International Conference on Machine Learning, pages 19527-19564, PMLR, 2023.
[35] Shengbo Eben Li. Reinforvement learning for sequential decision and optimal control. 2023.
[36] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint N0Xiv:15090029990 2015.
12

---

[37] Yaron Lipman, Ricky TQ Chen, Heli Ben- Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
[38] Cong Lu, Philip Ball, Yee Whye Teh, and Jack Parker-Holder. Synthetic experience replay. Advances in Neural Information Processing Systems, 36:46323-46344 2023.
[39] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-533, 2015.
[40] Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Mitos, and Marek Cygan. Bigger, regularized, optimistic: scaling for compute and sample efficient continuous control. Advances neural information processing systems, 37:1 13038 13071, 2024.
[41] Gergely Neu, Anders Jonsson, and Vicenç Gómez. A unified view of entropy-regularived markov decision processes. arXiv preprint
2017.
[42] Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn, Advances in neural information processing systems, 29, 2016.
[43] Georg Ostrovski, Marc G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In International conference on machine learning, pages 2721-2730. PMLR, 2017.
[44] Pathmanathan Pankayaraj and Pradeep Varakantham. Constrained reinforcement learning in hard exploration problems. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 15055-15063, 2023.
[45] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity- -rriven exploration by superviue prediction. In International conference on machine learning, pages 2778- 2787. PMLR, 2017.
[46] Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. arXiv preprint arXiv: 170.. 00000, 2017.
[47] Mirza Ramicic and Andrea Bonarini. Entropy-based prioritized sampling in deep q-learning. In 2017 2nd international conference on image, vision and computing (ICIVC), pages 1068-1072. IEEE, 2017.
[48] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.0595: 2015.
[49] Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Si- mon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7839):604-609) 2020.
[50] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.
[51] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint wrXiv:1707.06344 2017.
[52] Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In International conference on machine learning, pages
PMLR, 2020.
[53] Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. Advances in neural information processing systems, 30, 2017.
[54] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamies. In International conference on machine learning, pages 2256-2266 pmlr, 2015.
13

---

[55] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances neural information processing systems, 32, 2019.
[56] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011..33550 2020.
[57] Shivakanth Sujit, Somjit Nath, Pedro Braga, and Samira Ebrahimi Kahou. Prioritizing samples in reinforcement learning with reducible loss. Advances in Neural Information Processing Systems, 36:23237-23258, 2023.
[58] Bhavya Sukhija, Stelian Coros, Andreas Krause, Pieter Abbeel, and Carmelo Sterrazza. Maxin- forl: Boosting exploration in reinforcement learning through information gain maximization. arXiv preprint arXiv:2412. 2024.
[59] Bhavya Sukhija, Lenart Treven, Carmelo Sferrazza, Florian Dorflrr, Pieter Abbeel, and Andreas Krause. Optimism via intrinsic rewards: Scalable and principled exploration for model-based reinforcement learning. In 7th Robot Learning Workshop: Towards Robots with Human-Level Abilities, 2025.
[60] Richard S Sutton, Andrew G Barto, et al. Reinforcement learning An introduction, volume 1. MIT press Cambridge, 1998.
[61] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: 1801.0080000000 2018
[62] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[63] Kevin Wang, Hongqian NIu, Yixin Wang, and Didong Li. Deep generative models: Complexity, dimensionality, and approximation. arXiv preprint wrxiv2500.000820 2025.
[64] Renhao Wang, Kevin Frans, Pieter Abbeel, Sergey Levine, and Alexei A Efros. Prioritized generative replay. arXiv preprint arXiv:2410. 8882, 2024.
[65] Shuhe Wang, Shengyu Zhang, Jie Zhang, Runyi Hu, Xiaoya Li, Tianwei Zhang, Jiwei Li, Fei Wu, Guoyin Wang, and Eduard Hovy. Reinforcement learning enhanced 1lms: A survey, arXiv preprint arXiv: 10400, 2024.
[66] Yinuo Wang, Likun Wang. Yuxuan Jiang, Wenjun Zou, Tong Liu, Xujie Song, Wenxuan Wang. Liming Xiao, Jiang Wu, Jingliang Duan, et al. Diffusion actor-critic with entropy regulator. Advances in Neural Information Processing Systems, 37:54183-544 542244,,,, 2024.
[67] Yinuo Wang, Likun Wang, Mining Tan, Wenjun Zou, Xujie Song. Wenxuan Wang. Tong Liu, Guojian Zhan, Tianze Zhu, Shiqi Liu, et al. Enhanced dacer algorithm with high diffusion efficiency. arXiv preprint arXiv:2505.23420 2025.
[68] Guojian Zhan, Yuxuan Jiang. Shengbo Eben Li, Yao Lyu, Xiangteng Zhang, and Yuming Yin. A cranforration-aggreggttie framework for state representation of autonomous driving systems. IEEE Transactions on Intelligent Trunsportation Systems, 25(7):7311-7322 2024.
[69] Guojian Zhan, Xiangteng Zhang, Feihong Zhang, Letian Tao, and Shengbo Eben Li. Bicriteria policy optimization for high- reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems, 2025.
[70] Weipu Zhang, Gang Wang, Jian Sun, Yetian Yuan, and Gao Huang. Storm: Efficient stochastic transformer based world models for reinforcement learning. Advances in Neural Information Processing Systems, 36:27147-27166 2023.
14

---

A Theoretical Analysis
A.1 Useful lemmas
Lemma 1 (Discounted-Occupare) Lipschitz Lemma). Let 0 < < 1. For any two stochastic policies and x' sharing the same initial state distribution μs their discounted state-occupancy measurements (s) and di (s) meet the following condition:
| -
Y sup (1((1186,)) (-)=((
(16)
1--
sES
where ( denotes the total variation of two distributions:
(a(s)), la(s) b(s)l-
(17)
By the definition, the discounted occupancy measure can be reformulated fixed-point equation:
Proof.
de = +1 7 Prt.
= -1- +7
(18)
where Pa =. Taa Ps s, a) is the one-step transition kernel of It denotes the transition probability from state to s' under the policy x. Defining the subtraction the two transition kernels A, the following equation holds:
- = - + + (x.).
(19)
By rearranging A the hent-hand side and factoring out the common terms, we obtain:
A(T PP=) yd (P= P=.).
(20)
Because the induced norm of the transition kernel |P and y 1, -Px)) exists and (I - -P=)~ (?) converges. Multiplying by this inverse, can derive the Neumann
series as follows:
A
(Tpa)* Co (Pa - Pr)).
(21)
The L1 norm of the above equation is given by:
| 11
Σ E
ll Ih Parll1-
(22)
Due to the definition of the transition kernel, we can derive that:
|r Parll1-1
sup #E$
Σ [m(a| s) n x'(a P) s.a)
AEEA
(23)
(a 8) n' s) |||||
2 (T ())
7 s))
Since ||d] ||1 1, We can obtain that:
27 |||| S 1-7 sup (V ()) r'(: |s)).
(24)
Dividing by converts the L, norm total variation, the proof is completed.
For clarity, we provide the proof in discrete state-action space. Since we assume continuous state-action space is measurable continuous space, the contraction property in the total variation norm and the convergence of the Neumann series of (T - P)) also hold in this setting. Therefore, the conclusion can be directly extended to the continuous state-action space.
15

---

Lemma 2 (FIFO-Bull) Buffer Proximity Lemma), Assume the replay buffer B holds exactly the most recent K transitions (FIFO of fixed size K). We can define the occupancy measurement under the
behavior policy in as: Vs *x Lh= d and let D == supses TV m. s)).
Then the following inequality holds:
- I S
At-j.
(25)
1=
Proof. By the convexity of total variation, we can obtain:
K
|| 1
-d - -TV.
1|d
(26)
K
h=0
By using the triangle inequality in, can obtain that -ld -|d==>- d=II/lv S Fmix Σj=o Ae-3 with Tmix = 7/(1 - 7). Insert this bound and interchange the sums, we can derive that:
K-14-
|| d 1|V
K
h=0
j=0
(27)
K-1
l
K -j K
K -
Tiix
D.-j
S
Tmix
A A--j.
==
Lemma 3 (Behaviour Sechaviuur-Mixing Contraction Lemma). Assuming that each policy TT h (for h E K 1)) writes the same number of transition samples into the buffer during the last time steps, and the difference between consecutive policies remains
negligible scale: sup, TV (V( (s) <
the behuvior policy represented in the buffer can be approximately expressed as Br R E Th-h
Let 6, = supses [V(==(1 s), (. [)), then the following bound holds:
K
sup TV (B.( x (1, (+1 S #ES
d
(28)
h=0
Proof. Fix since the total variation is convex in its first argument, derive that:
K-1
K-1
TV
Σ m h=0
(),
<
Σ
(.1))-
(29)
K
K
h=
Taking the supremum over and moving the sup outside the sum gives the claimed inequality.
Lemma (The X-mixture estimator bias). Let p(s. = (s (s) be the target state-action distribution and p'(s, a) = di a) be the behaviour distribution. For any measurable function g: Sx.AA + R bounded by ||g]|oo, define the X-mixture sampling measure as q2 = (1=A)p++ Xp', s < 1. Denote the exact importance sampling (IS) estimator JIs = ((s,))ya [w(s, a) g(s, a) with weight p/qx, and the X-mixture estimator Jmis (B(s,a)=ga [g(s,a)) a)]. Then the following inequality holds:
E [[mi
[[sss] s SAM (KL(())))) = (AALL(70113))1)
(30)
Proof. Rewrite the bias as follows:
Jys - -1 gl.
(31)
By applying Hölder's inequality, we can obtain that:
|| - p'll S (2
(32)
Because [gl S M, the magnitude of the bias is upper-bounded by X M (T.pppp) Replacing the distribution notation with the corresponding policies gives the stated O KL) dependence.
D
16

---

A.2 Proof of Theorem
Proof. From lemma 3 the upper bound of total variation between the behavior policy Pr and the optimal policy nT" can be derived as:
K
sup (V I
(33)
K
h=0
the target policy converges to the optimal policy, i.e., 8 - 0 holds, then the following relation is satisfied:
lim sup (V;,,, =
(34)
The above expression indicates that the behavior policy B, will gradually converge to the optimal policy π². Then, using the triangle inequality, we can obtain the following:
TV(n,, d S (V + TV(d", d
(35)
A
B:
From lemma 2 the first part A, can be derived as
K
A S Tmin
At-j-
(36)
j=0
Since the policy can converge to the optimal policy within a finite number of steps N, the final policy shift error is considered to vanish after these steps. Therefore, the term A, can be regarded zero From lemma ! the B, can be reformulated as:
B, | d"
Tmix sup TV (Te(-ls), T'' (-1s) Tmixxe.
(37)
TV
Likewise, due to policy convergence in finite steps, 8, holds when & N. Thus, the proof is complete.
Notably, if the policy does not converge to the optimal policy within a finite number of steps, but the policy shift A, decays sufficiently fast after each update, where the following condition holds:
Σ
sup TV s)) (r( s)) oc.
(38)
#ES
Under this condition, it can still be guaranteed that A, = 0, and the original proof remains valid.
A.3 Discussion of importance sampling in MoGE
In MoGF, since the critical experiences are generated by the diffusion-based generator and the one-step imagination world model, both the initial-state distribution and the policy of the experiences may differ from those in replay buffer B under the assumption that the one-step world model can accurately estimate the environmental dynamics, where all transitions share the same dynamics kernel P(s' a,
Problem Setting. We consider training samples composed of two distributions: (1)Denm collected with the behavior policy 3 and the real initial state distribution denv (s) from the buffer 1; (2) Denn generated synthetically with target policy Ty by MoGE, where the initial state distribution denoted (s).
For policy evaluation, let Q" be the action value function the target policy me. obeys the Bellman identity
(= a) Eg--Pa (r(s,a) a) + + (*'''''')),
V(s a).
(39)
Because Eq. holds in a point-wise way, where the (8, a) can be sampled from anonymous distribution, the squared -D-rrror of any parameterised critic Qu:
Lyev( = Bus.ann (Q - (r + Ea'nns (s
(40)
17

---

still attains its global minimum at Qo Q" for any mixture of the training Dj (1 k)D.nv gen S 1. Hence, the critic remains unbiased without importance sampling; oof-py sam- pling only affects optimization variance. An IS correction becomes necessary only when regressing the full Monte-Carlo return Go. which behaviour- hehaviour-dependent. However, for policy improvement, the importance sampling is non-negligible since the existence of distribution mismatch:
CPM (0) = (x,a)) ~ D.n p(s. a) g(s, a) + B(s,a) ~ Dun (s()) g(s, a)
(41)
dem(s) where g(8, a) denote the return differs in different algorithm, and w(s)) )حي(3) p(8, a) ma(als)
p(ale)
However, for algorithms that compute the objective function directly based on the target, i.e., g(s, a) = g(s, To(s)). importance sampling for policy can be omitted. In this case, policy improvement only requires importance sampling for the initial state distribution. Since computing w(s) and intractable in high-dimensional continuous spaces. We therefore approximate the IS expectation by sampling mixture using mixing rate À since the bias can be bound by Lemma The target for policy improvement in MoGE can be finally derived as a sampling mixture:
(0) = (1 () ~ Das [o(s, a)] + >E((,a)) Daca [a(s.a)) a))
(42)
with A [0, 1). For example, the resulting actor loss for SAC-style objectives is like:
LPIM = (1 - X) (eaa))DD.. (ar log No (a 8) - (s,
+ (ss [a log To(a s) (s.
(43)
Note that Eq. 43 is the first-order Taylor expansion of the exact IS estimator; its bias scales with O(A ((ool(a 222). Empirically, small À retains the coverage benefit of the generated data while keeping bias and training instability negligible.
Theoretically, the choice of k is unrestricted; however, to reduce training variance, it is advisable to constrain & within smaller range. this work, we set lt 2A.
B
Environmental configuration
B.1 Environment Introduction
Deep Control Suite. We chose challenging tasks involving the humanoid and quadruped robots. The final reward for each task is the product of the standing reward and the forward velocity reward, expressed as: Reward (Standing Reward) x (Forward Velocity Reward).
(a) Humanoid
(b) Quadruped
Figure 4: DMC environments.
Humanoid tasks: The Humanoid consists of three primary tasks, each designed to challenge an agent's ability control simulated humanoid robot. The three tasks are described as follows:
Stand: The agent's objective is to maintain upright posture. The reward function encourages stability and vertical torso position while minimizing deviations from an ideal standing height.
Walk:
The agent is rewarded for moving forward at target velocity of 1 m/s. This task evaluates the agent's 's capability to coordinate limb movement and maintain balance while walking.
18

---

Run: In this task, the agent is required to achieve a high-speed forward motion of 10 m/s. The challenge includes maintaining dynamic stability and efficient stride patterns.
Quadruped tasks: The Quadruped environment represents four-legged robotic model with two major tasks aimed at testing multi-legged coordination and locomotion:
Walk: The agent must achieve forward movement at a steady pace. This task assesses the stability and synchronization its four legs during controlled walking.
Run: The agent is required accelerate higher velocities, demanding agile gait adjust- ments and robust stability during high-speed movement.
OpenAl Gym. We chose 5 widely used locomotion tasks in various domains:
State-action space: S E 1RT, A R
Objective. Maintain forward velocity fast as possible while avoiding falling over.
Initialization.
The walker is initialized in standing position with slight random noise added to joint positions and velocities.
Termination.
The episode ends when the agent falls, the head touches the ground, or after 1000 steps.
Figure 5: Walker2d-v3
State-action space: S E R A R
Objective. Maintain balance and walk or run forward high velocity while avoiding falls.
Initialization.
The humanoid starts in an upright position with slight random perturbations joint angles and velocities.
Termination. The episode ends when the head height is less than 1.0 meter, the torso tilts excessively, or after 1000 steps.
Figure Humanoid-v3
State-action space: S E [11 A E 12..
Objective. Navigate forward quickly as possible using four legs while maintaining stability.
Initialization. The ant is initialized in stable, upright position with random noise applied to its joints.
Termination. The episode ends if the ant falls, flips over, or reaches the maximum step count of 1000.
Figure 7: Ant-v3
B.2 Reproducibility Statement & Detailed Hyperparameters
In MoGE, we adopt the hyperparameter settings without additional fine-tuning and use the same configuration across all previously demonstrated tasks, which listed in Table
In this paper, experiments are conducted with a total 1.5 million environment interaction steps, and the results are averaged over three random seeds. The experiments are performed AMD Ryzen Threadripper 3960X 24-Core Processor and an NVIDIA GeForce RTX 4090 GPU. Besides, the walltime results(s) of 1.5M steps are included in Table
19

---

State-action space: S E 12 , A R2.
Objective. Achieve maximum forward velocity with smooth. coordinated movements.
Initialization. The agent starts with a slight forward tilt and randomized joint noise.
Termination. The episode ends after 1000 steps or if agent's head touches the ground.
Figure 8: Halfcheetah-v3
State-action space: S E A
Objective. Propel forward through water-like dynamics using sinusoidal wave patters..
Initialization. The swimmer starts a straight posture with minor random perturbations.
Termination. The episode ends after 1000 steps, with explicit termination falling.
Figure 9: Swimmer-v3
The results show that MoGE incurs slightly higher walltime compared to baseline algorithms. No- tably, reduce computational overhead, we perform MoGE training only once every 10 environment interaction steps, which significantly controls the overall compute cost. Moreover, unlike methods requiring retraining the generative model on the buffer until convergence before use, MoGE con- tinuously updates its generator during training. This makes MoGE's s diffusion- based training more efficient and lightweight in practice.
Supplemental clarification
C.1 Design of One-step imagination world model
In MoGE, the structure of the one-step imagination world model is illustrated Figure 10 Since MoGE is developed under the MDP setting. The reason why modeling dynamics in latent space instead of directly learning the mapping (s,a))s is motivated by several practical considerations:
(A)) Learning compact latent representation helps capture abstract, task-relevant features of the environment, improving generalization and training efficiency even under MDPs. Besides, different dimensions of the state may contribute unevenly to dynamic modeling. Using an encoder to transform the raw state allows the model to extract the most relevant components, which facilitates more accurate transition prediction.
(B) Latent dynamics are often smoother and easier to predict compared to raw state transitions, especially in high-dimensional environments. Fitting dynamics in latent space typically improves the accuracy of transition modeling.
(C) Decoupling representation learning from dynamics prediction allows better reuse and transfer of components. fact, MoGE, the policy network is built on top of the same encoder, enabling unified state representation that facilitates more effective policy learning.
To further support our claim, simple experiment is conducted across three environments (each with three random seeds) using the same diffusion and baseline algorithm setup. This experiment compares the TAR between vanilla dynamics (Using the Transformer predictor of next states and rewards) and latent dynamics. The results are demonstrated in the Table
The results show that, compared to directly mapping the current state and action to the next state, intro- ducing latent space leads to better learning of both environment transitions and state representations in the policy network.
20

---

Table 2: Performance comparison of MoGE and related baselines on Humanoid- task.
Env
MoGE
PGR
SER
DSAC
Humamoid-run 151657
20781341464
210321+788
128153+122
Table 3: Hyperparameter settings.
Hyperparameter
Value
Hyperparameter
Value
Training
Learning rate
10-4
Target network soft-update rate
0.05
Batch size
1024
Buffer size
I 0.0 000
Optimizer
Adam
Sampling
Uniform
Discount factor (r))
0.99
Num of vector envs
10 (Only in DMC tasks)
Sample batch size
20
World Model learning
Dynamics loss coefficient (3²)
0.5
Representation loss coefficient (32)
0.1 Adam
Learning reate
x 10
Optimizer
Diffusion model
Diffusion steps
100
Noise Schedule
Cosine
Guidance scale
e-prediction Activation Learning rate
True
Denoising network
[256,256,256] Adam
GELU 1 10
Optimizer
Actor
Minimum policy log std Maximum policy log std Learning rate
-20
Policcy network
[256,256,256]
0.5
Activation in hidden dim Activation in output dim
GELU
x 10-4
Linear
Critic
Value network
[256,256,256] x 10
Activation in hidden dim Activation in output dim
GELU
Learning rate
Lincar
Architecture (8M)
Transformer layers
Latent space dimension
256
Transformer heads
Dropout Normalization
0.1
MLP activation
GELU
LayerNorm
C.2 Broader baseline experiments with model-free algorithms
To further validate the effectiveness MoGE compared with active exploration, we perform extra ex- periments with more mainstream model-free algorithms. All the algorithms are evaluated in standard settings and tested on the OpenAl Gym tasks: Walker2d-v3, Humanoid-v3, and Halfcheetah-v3..
Deep Deterministic Policy Gradient (DDPG) 36 an off-policy actor-critic method that leverages deterministic policies and experience replay for efficient learning in continuous action spaces.
Trust Region Policy Optimization (TRPO) [50]: an on-policy method that optimizes policies by enforcing trust region constraint to ensure stable updates.
Proximal Policy Optimization (PPO) 51 improved upon TRPO by using a clipped surrogate objective for simpler and more efficient training.
Soft Actor-Critic (SAC) 21:: introduces maximum entropy encourage exploration and improve stability, making well-suited for complex, high-dimensional tasks.
All the training curves are illustrated in Figure [I and the detailed results are listed in Table 5
21

---

Table 4: Ablation of MoGE with vanilla vs, latent dynamics on humanoid control tasks.
Env
MoGE w/ latent dynamics MoGE w/ vanilla dynamics
Humanoid-run
489 +
408
35
Humanoid-wai
892 19
684 71
Humanoid -tann
907 = 7
785 88
action
at
reward
Tt
One-step imagination
Z
1++11
Encoder
Decoder
Encoder
Decoder
St
St
S
Figure 10: One-step imagination world model. During training, the current state 8t is encoded by the representation network he into latent representation Given this latent state and the action de. one-step world model predicts the next latent state 1, the immediate reward A, and termination factor é The solid-line process represents inference, while the dashed-line process is used for loss function construction.
C.3 Broader baseline experiments with model-based algorithms
Similarly, to validate the performance of MoGE when integrated with reinforcement learning algo- rithms, additionally select two mainstream model-based RL algorithms for comparison. These algorithms are sourced from the pettps://githan com/ ni ck1 ashansen/tdmppc2 and the experi- ments are performed in DMC Humanoid tasks:
TD-MPC2 [6]
model- -based reinforcement learning algorithm that combines temporal difference learning with model-predictin control to enhance sample efficiency and long-horizon planning.
DreamerV3 24 model-based reinforcement learning framework that leverages world model for imagination-basee training, enabling effective policy learning with high sample efficiency in continuous control tasks.
All the training curves are illustrated in Figure and the detailed results are listed in Table
C.4 Broader baseline experiments with modified Actor-Critic algorithms
To provide more comprehensive evaluation, some folmmlfordlleed methods that were modified based on the traditional Actor-Critic method can be included. We conduct the experiments in DMC Humanoid tasks, which are one of the hardest tasks within 3 seeds, and the introductions of the methods are illustrated as follows:
REDQ 9 model-free RL algorithm that employs large ensemble of fufnctio with randomized subset updates to improve sample efficiency and reduce overestimation bias.
DroQ 28] a lightweight variant ensemble Q-learning that applies dropout regularization to approximate uncertainty and achieve doubly efficient learning.
BRO 40 a scalable actor-critic framework that leverages network scaling, regularization, and optimism to balance compute and sample efficiency in continuous control.
22

---

11400
2000
(a) Walker2d-v3
(b) Humanoid-v3
(c) Halfchectab-v3
Figure 11: Supplemental study curves with mainstream model-free algorithms.
Table 5: Total Average Return (TAR) on 3 OpenAl Gym tasks for supplemental experiments. Mean + Std over 3 seeds. Bold best, Higher better.
Environment
TDS
SAC
DDPG
TRPO
PPO
DSAC
DSAC-MoGE
Walker2d-w3 5031.1 + 84.2
5997.0 291.1
3268.1
240.9
5635.6 = 211.9
3880.9 + 327.7
6501.1 #7.3
6978.4 68.7
Humanoid-v3
5967.1 547.8
8831.7
352.2
4548.7
807.6
947.2
503.9
6011.0 2014.4
11004.0 121.5
12151.1
35.4
Hallcheetah-v3
3666.4
380.8
462.3
4207.4
756.6
17324.7
41.1
18054.9
459.6
AVG.GYM
6120.4 1432.8
10583.3
341.4
7536.6
503.6
3596.7 490.8
5010.4 1578.3
11472.7 93.0
12394.8 187,9
Simba 33
large-scale RL framework that exploits simplicity bias in over-parameterized networks to improve stability, generalization, and training efficiency.
As shown in Table 7 except for the simplest case (llumanoid-Stand) MoGE achieves the best performance across all more challenging tasks.
C.5 Broader baseline experiments with enhanced exploration approaches
We have conducted additional experiments to study the effect of different exploration strategies on the DSAC algorithm via ablation comparisons. Due to the development and implementation effort involved, and time constraints, we focus on three representative environments- the Humanoid tasks in DMC-for this evaluation. The three approaches are as follows:
Plan2Explors [52] model-based exploration method that uses self-supervised world models to plan informative trajectories without external rewards.
exploration framework that maximizes mutual information between states, actions, and returns to encourage diverse and informative behaviors.
MaxInfoRL. 58
OMBRL 59 model-based exploration algorithm that incorporates optimism through intrinsic reward shaping, enabling scalable and principled exploration.
Demonstrated in Table the results show that MoGE still offers significant advantage compared to these methods. Intrinsic exploration methods encourage visiting novel unpredictable states, but their signals (e.g., prediction error, uncertainty) are often task-agnostic and may lead to uninformative or misaligned exploration. In contrast, MoGE leverages task-aware utility functions (c.g., TD- eиror, entropy) to generate states that are directly aligned with policy improvement objectives**. Besides, unlike intrinsic reward methods that require careful reward balancing and affect the policy's optimization objective, MoGE decouples exploration from reward design, enabling more stable training without interfering with the task- specificc learning signal.
D Limitation and Future Work
While MoGE demonstrates strong performance in enhancing exploration and improving sample efficiency in reinforcement learning, several limitations remain. First, the generation of critical states through the diffusion-based generator introduces additional computational overhead compared to
23

---

(a) Humanoid walk
(b) Humanoid-stand
(c) Humanoid-run
Figure 12: Supplemental study curves with mainstream model-based algorithms.
Table 6: Total Average Return (TAR) on 3 DMC Suite tasks for supplemental experiments. Mean
Std over 3 seeds. Bold = best, Higher is better.
Enviromment
DSAC
DreamerV3
TD-MPC2
DSAC+MoGE
Humanoid-wall
146.5 = 60.9
0.9
814.3 + 49.8
891.7 19.1
Humanoid-stane -tand
776.6 15.6
0.3
838.9
907.5 =
Humanoid-run
2.7.4
0.8 0.4
254.6 11.1
488.9 = 8.7
AVG.DMC
396.8 26.8
2.4 + 0.4
635.9 a 33.4
762.7 11.6
standard replay buffer sampling. Second. MoGE assumes that the state distribution learned by the generator aligns well with the replay buffer's occupancy measure. In practice, minor discrepancies may arise. Overall, while MoGE enhances exploration capabilities, its time cost is influenced by the quality of the learned state distribution and the smoothness of the conditional diffusion process.
In the future, we may explore integrating MoGE with on-policy RL frameworks to enable real- time generation of critical states, further enhancing exploration efficiency during live interactions. Additionally, investigating more expressive utility functions for the diffusion-based generator could improve the selection of high-value states, optimizing policy learning. Moreover, dynamically adjusting the generator's 's sampling strategy based on task complexity and training progress may further boost robustness and generalization.
E
Positive and Negative Social Impact
Our method, MoGF, performs exploration augmentation in reinforcement learning by generating critical samples through a diffusion- generator and a world model, significantly improving sample efficiency and policy performance in complex control tasks. This capability has promising implications for real-world applications such as embodied AI, autonomous driving, and large-scale dicision-making systems, where efficient exploration of vast state spaces is crucial. However, the simulation-
ability to synthetically generate exploration samples may lead to overconfidence trained policies, thereby increasing risks if these policies deployed prematurely in real-worl worldw environments. We advocate for careful validation and consideration of ethical implications to ensure the responsible and safe application of MoGE.
24

---

Table 7: TAR for supplemental experiments with modified AC methods.
Table 8: TAR for supplemental experiments with enhanced exploration approaches.
Ens
DSAC-MoGE
Simba
BRO
RedQ
DioQ
Exy
DSACAMoGE Plan2Explore
MaxinfoKL.
OMBRI
Humanoid run
268 1: 40
15
187 4:
161
Homanoid zun
311 12
197 1:
262 13
Humanoid walk
892 19
801 1: 13
881 2:
665 1:5
682 1 14
Humanoid walk
892 19
568 1:
481 1:
678 1:
Humanoid stand
907
1:
920 1:
905 3:3
902 1:4
896 6
Humanoi stand
907
801 14
844 1:
769 1: 11
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
We emphasize our core contributions in the abstract and introduction, high- lighting the critical state generator, the one-step imagination world model, and the proposed RL training framework combined with MoGE.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: In Appendix D we discuss that the limitation of this work is the problem of time cost and the bias in critical state generation.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
25

---

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed
penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We provide complete assumptions in the formulation of the theorem and complete proof of the theorem in Appendix A
Guidelines:
The answer NA means that the paper does include theoretical results. the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the suppiemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We give details the hyperparameters required for the experiments in the Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
26

---

(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We will make the full code publicly available when the paper is accepted, but the core code is public with open access Appendix B
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
Sablic/guides httpi///uuudee//ooSuubsmssssss
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurlPS code and data submission guidelines (https ctttps.cc/pubbll httppppwwppppppppppppppppppaaaep wtw.:iddde///oeedubbeeeeee
ssilioolicyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In the appendix section, we give very detailed selection of all algorithmic hyperparameters, optimizers.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
27

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Whether the data is in table a picture, report the error bars. Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In the Evaluation Setups in Appendix B we give the CPU and GPU servers used for the computation and the time used to train 1.5 million using Humanoid-run as an example.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Yes.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
28

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss both the potential positive societal impacts and negative societal impacts of the work in Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymenn of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
29

---

Justification: All our baseline algorithms are evaluated under standard settings, and all open-source implementations are properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, pappesswi thcode danasatsssss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/mmodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
30

---

Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
31


Reviews:
[{"review_id": "DSTHs0tQZq", "reviewer": "Reviewer_vJeT", "summary": "To speed up RL, the authors propose to generate synthetic transitions with high exploration potential using conditional diffusion. Their method, MPGE, first trains an unconditional diffusion model on the replay buffer. Then imaginary states are generated using classifier guidance, where the utility function is either policy entropy or TD-error. A world model is learned to generate the rest of the synthetic transition. MPGE can be added onto any off-policy RL algorithm, and they show improvements over baselines in 10 continuous control tasks.", "questions": "See weaknesses", "limitations": "yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- an interesting idea of augmenting the replay buffer with imaginary transitions high in learning potential, by using a conditional diffusion model\n- paper is generally well written\n\nWeaknesses\n- Usage of policy entropy is questionable as a utility function, especially in robotics tasks with optimal multi-modal policy distributions (see Diffusion Policy)\n- there seems to be a lot of moving parts and hyperparameters (guidance weight, mixture weights, choice of utility function)\n- gains over baselines is somewhat marginal in 5/10 tasks, with little analysis on why MPGE does better or worse than baselines. the experiments are run with only 3 seeds, which makes me think many of the tasks where MPGE only does marginally better, could be due to statistical error. \n- It would have been great to see some qualitative analysis of what imaginary states MPGE generates, and why this leads to better performance in some tasks. \n- No discussion of the compute costs, such as walltime comparison with other methods. \n\nMinor:\n- I don't think figure 2 is necessary, could be moved to appendix to make space for qualitative analysis", "was_revised": true, "final_justification": "Overall, I am fairly positive on this paper. The authors build on top of prior generative replay buffer work, by proposing to condition the generation on various task-relevant metrics instead of unconditional generation. Most of my concerns were addressed, such as concerns on hyperparameter tuning and seed variance.\n\nWhat would have moved the needle more for me, would have been more qualitative analysis, which the authors provided in text form in the rebuttal. If they had more analysis in the initial submission, I would have increased my score. As such, I will maintain my rating of borderline accept.", "rebuttal": "We sincerely appreciate your detailed and constructive feedback. Below are our responses to your concerns.\n# 1. Discussion of the utility function with Policy entropy \nWe sincerely appreciate your insightful comment. Indeed, policy entropy is only one possible choice of utility function within the MPGE framework. In tasks with inherently multi-modal optimal policies, such as those observed in robotics, we approximate entropy using a Gaussian Mixture Model (GMM). Prior works (e.g., [1]) have demonstrated the feasibility of this approach in the context of diffusion-based policies.\n\nMore generally, since entropy merely quantifies stochasticity, it can be replaced by other metrics to capture utility, such as the distance between a given state and the empirical state distribution in the buffer, or other differentiable surrogate signals. The key requirement is that the utility function must be differentiable with respect to the state, so that it can guide generation. MPGE is fully compatible with a broad family of such utility functions, and we view entropy as one heuristic example rather than a universal choice.\n# 2.Discussion of the hyperparameters \nWe fully understand your concern regarding hyperparameter sensitivity. We will explain the meaning and selection strategy for each of the following components: the utility function, the guidance scale $ω$, and the trade-off parameter \n$λ$. I will also describe how these parameters are chosen in practice.\n\n1. First, regarding the choice of utility functions: as mentioned in the paper, there exists a wide variety of utility functions in reinforcement learning tasks. Our goal is to provide a general architecture that can accommodate and benefit from better-designed or more informative utility signals in the future.\nDepending on the specific task, different utility functions (e.g., Q-values or intrinsic curiosity as used in PGR) can be adopted to guide the generation of task-specific critical states. Our current choiceslike policy entropy and temporal-difference (TD) error are meant to serve as illustrative examples: entropy naturally appears in the policy optimization objective, and TD error is central to value estimation. These choices are heuristic rather than exhaustive, and we encourage practitioners to adapt and extend this framework using other utility formulations suitable for their own downstream tasks.\n\n2. In our ablations, $ω=1$ performs best under standard training steps. However, with extended training (e.g., 3M steps), smaller scales (e.g., \n$ω<1$) also achieve strong performance, differing mainly in convergence speed. Larger scales tend to distort the generated data. For simple tasks, a small \n$ω$ is sufficient, while complex tasks may benefit from adaptive schemes such as annealing: \n\n|  | $\\omega=0.2$ | $\\omega=0.5$ | $\\omega=1$ | $\\omega=2$ |\n|--------|----------------|----------------|------------------|------------------|\n| TAR(3M)   | 598.2±21.3    | 616.2±24.7    | 624±12.5       | 402.3±9.8     |\n\n3. As for the value of $λ$, we have discussed its setting in Appendix A. A large \n$λ$ can lead to a significant distributional shift, which reduces the effectiveness of prioritized imitation mechanisms (PIM). Our ablation studies also confirm this observation. Therefore, \nactually $λ$ is not a freely tunable parameter—it must remain within a small range to be effective, similar in spirit to the clipping threshold in PPO.\n\n# 3.Supplemental experiments\nWe fully understand your concern. To address this, we have re-run the experiments on the Gym tasks using five random seeds based on the original setup. The updated results are presented below:\n\n| Env | MPGE | PGR | DSAC |\n|--------|----------------|----------------|------------------|\n| Walker2d-v3   | **7354±158**    | 6621±237    | 6348±121      |\n| Humanoid-v3    | **12091 ± 146**    | 11395±379   | 10814±219      |\n| Ant-v3    | **8294.6 ± 184**   |  7735.2 ± 454   | 7121±176     | \n| Halfcheetah-v3    | **18124 ± 395**    | 17362± 128   | 17110 ± 14      |\n| Swimmer-v3    | 144 ± 2   | **145±4**   | 131 ± 6      | \n\nThe results show that our method consistently achieves state-of-the-art performance across all five seeds compared to the baseline algorithms. As further discussed in the supplementary experiments (Appendix C), we note that the baseline algorithm DSAC already achieves near-optimal performance on these Gym tasks, leaving limited room for improvement. The fact that both PGR and MPGE still manage to outperform DSAC highlights the importance of passive, utility-guided exploration, which contributes to further policy enhancement even in highly optimized settings.\n\n\n# 4. Quality analysis of transitions generated by MPGE \nMPGE generates imaginary transitions by first sampling critical states under utility guidance, then assembling transitions via a learned world model. These generated transitions exhibit two important properties like novelty and dynamical feasibility, which together contribute to improved policy performance.\n\n## 4.1 Novelty of generated states\nMPGE generates states that are reachable but under-visited, conditioned on utility signals (e.g., TD error or entropy) that reflect the current policy’s learning needs. This design introduces the following advantages:\n\n(i) State-level generation allows MPGE to focus on critical regions beyond the static buffer coverage, rather than simply perturbing stored transitions as done in PGR.\n\n(ii) Dynamic utility evolution enables continual novelty as the utility function shifts with policy updates, steering generation toward states where the policy is uncertain or under-trained.\n\n(iii) Policy-centric guidance means MPGE is not constrained by the historical behavior policy; it rebalances the frequency of critical states under the current policy distribution, leading to better task relevance.\n\nTogether, these results in generated transitions that are more diverse and better aligned with the evolving policy, which supports exploration and evaluation in otherwise poorly covered areas of the environment.\n\n## 4.2 Dynamical feasibility of generated states\nMPGE ensures that all transitions respect environmental dynamics by simulating them through a pretrained world model, following standard practice in model-based RL. The world model is trained using diverse data (e.g., from random policy), covering broad regions of the state-action space and avoiding overfitting to narrow behavioral trajectories.\nIn contrast, methods like PGR directly learn joint distributions over \n$(s,a,r,s \n′)$ from buffer tuples. This paradigm does not enforce the structural constraints of real dynamics (i.e., the mapping from $(s,a)→(r,s′))$. As a result, PGR may capture spurious statistical correlations rather than true functional dependencies, potentially modeling unstructured factorizations.\nSuch dynamics-inconsistent samples may degrade the value estimates and hinder policy improvement.\n\n## 4.3 Summary\nIn summary, MPGE generates imaginary transitions that are:\n\nNovel: they target critical, under-visited states tailored to the current policy;\n\nValid: they are assembled via a learned model trained to respect environment dynamics.\n\nThese properties enable MPGE to produce high-quality synthetic data that complements the replay buffer and improves learning efficiency. Our additional experiments further confirm that transitions generated by MPGE are more dynamically consistent and task-relevant, contributing to its superior performance over PGR in several benchmark tasks. **To find more quantitative analysis, please refer to the rebuttal to reviewer W4cH due to the limitation of space.**\n\n# 5. Compute costs in MPGE \nThank you for the suggestion. We have added the relevant comparison. The walltime results(s) are now included as follows:\n\n| Env | MPGE | PGR | SER | DSAC|\n|--------|----------------|----------------|------------------|------------------|\n| Humanoid-run   | 151657±395    | 207813±1464    | 210321±788       | 128153±122      |\n\n\nThe results show that MPGE incurs slightly higher walltime compared to baseline algorithms. Notably, to reduce computational overhead, we perform MPGE training only once every 10 environment interaction steps(please refer to codebase), which significantly controls the overall compute cost.\n\nMoreover, unlike PGR, which requires retraining the generative model on the buffer until convergence before use, MPGE continuously updates its generator during training. This makes MPGE’s diffusion-based training more efficient and lightweight in practice.\n#  Minor\nWe sincerely appreciate and acknowledge your valuable suggestion. We will carefully consider this point and incorporate the corresponding clarification or discussion in the revised version.\n#  Conclusion\nWe sincerely appreciate your thoughtful and constructive feedback. We hope our responses have effectively addressed your concerns and respectfully ask that you consider a more favorable evaluation.\n\n[1] Wang Y, Wang L, Jiang Y, et al. Diffusion actor-critic with entropy regulator[J]. Advances in Neural Information Processing Systems, 2024, 37: 54183-54204."}, {"review_id": "5EC0oa66y1", "reviewer": "Reviewer_W4cH", "summary": "The authors propose MPGE, a novel generative augmentation framework for off-policy RL algorithms. MPGE consists of three modules: generator and classifier to guide generator to generate under-explored states, and the world model to predict dynamically plausible next states and reward. By using augmented transitions, MPGE seamlessly integrates with off-policy RL algorithms and achieves higher sample-efficiency.", "questions": "Here are some questions I want to ask:\n\n- It seems that the method is susceptible to guidance scale $\\omega$. Is there any rule for deciding such hyperparameter?", "limitations": "Here are some comments I want to suggest:\n\n- As mentioned above, off-policy RL algorithms are very sensitive to hyperparameters. It seems that MPGE achieves better performance in various benchmarks compared to SER and PGR, but the complexity of the proposed method is higher than those baselines. It is hard for me to ensure that MPGE can generalize to different tasks without extensive hyperparameter tuning.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- Idea is straightforward and the paper is easy to follow.\n- Conduct extensive experiments including ablation studies on each design components.\n- Code is publicly available\n\n**Weakness**\n\n- It seems that the main motivation of this work is closely related to PGR, as both utilizes generative models with guidance to generate novel but dynamically plausible transitions. While authors claim that using PGR generates data closely to the original data distribution, it is not true when we check the analysis conducted by PGR. I’m also not sure that MPGE really generate novel but dynamically plausible transitions as there are only plots with the reward metric and lack of analysis. I strongly recommend authors to conduct analysis on novelty and dynamic plausibility of generated transitions by their method.\n- While I acknowledge that authors conduct extensive experiments and get superior results, I’m not completely persuade myself that why it should be better than PGR. It seems that world model is also uncertain in unknown regions during training, so we cannot ensure that the prediction of world model is dynamically plausible. Again, it would be nice to have some additional analysis on why MPGE performs better than other methods.", "was_revised": true, "final_justification": "Authors additionally provide analysis on novelty and dynamic plausibility of generated transitions, which are promising. Therefore, I lean towards the acceptance.", "rebuttal": "We sincerely thank you for the thoughtful and constructive feedback. Please find our detailed responses to the comments below.\n\n# 1. Qualitative Analysis of MPGE\n\nWe will address your concerns through theoretical analysis from two perspectives: novelty and dynamic compliance. Specifically, we aim to clarify the following questions:\n\n1. Why do the actions or transitions generated by MPGE exhibit novelty, and how does this benefit policy optimization and evaluation?\n\n2. How does MPGE ensure compliance with environmental dynamics when generating transitions?\n\n3. Why are the transitions generated by MPGE more effective than those produced by methods like PGR, ultimately leading to improved policy performance?\n## 1.1 The novelty of transitions\nThe joint distribution of transitions generated by MPGE can be derived as follows:\n\n$$p(s,a,s^{\\prime},r)=p(s)p(a|s)p(s^{\\prime},r|s,a)\\quad(1)$$\n\nBased on the structure of MPGE, we can further decompose the source of its novelty into three main components:\n\n### (i) State-Level Generation Instead of Transition Resampling\nMPGE generates high-utility states directly and then uses the world model to simulate plausible transitions. This contrasts with prior methods such as PGR, which often resample or perturb transitions from the buffer. By targeting utility-defined regions, MPGE explores more structurally diverse transitions.\n\n### (ii) Dynamically Shifting Utility Landscape During Training\nThe utility signal in MPGE evolves as training progresses, continuously shifting the generation target. This dynamic conditioning promotes continual novelty, as the generator focuses on states that are currently difficult for the policy, rather than statically defined regions.\n\n### (iii) Policy-Centric Guidance Instead of Buffer Imitation\nCrucially, MPGE does not generate data based on the static behavior policy distribution in the buffer. Instead, the generation is guided by utility functions evaluated under the current policy, such as current-policy TD error or entropy. This ensures that the generated transitions align with the evolving learning needs of the agent, rather than being confined to outdated or suboptimal trajectories. The world model further enables rollouts from generated states, allowing MPGE to synthesize transitions that the current policy is likely to visit but has not yet explored, thus supporting meaningful off-policy generalization. \n\n### Summary:\nMPGE leverages utility functions conditioned on the current policy to generate reachable but under-visited states with long-tail characteristics. Rather than producing out-of-distribution samples, it rebalances in-distribution critical states to align with the training objective.\n\nAs a result, transitions generated by MPGE are more novel and task-relevant compared to methods like PGR, which imitate the buffer and remain constrained by the behavior policy distribution.\n\n## 1.2 The dynamical feasibility of transitions \n### (i) analysis of the dynamical feasibility in MPGE\nIn our setting, the world model is pretrained using data collected from a random policy, ensuring broad coverage of the state-action space and capturing generalizable environment dynamics rather than overfitting to narrow trajectories. This is a standard practice in offline model-based RL to ensure that rollouts remain dynamically plausible.\n\nImportantly, MPGE generates only states, and transitions are simulated through the learned dynamics model. As such, all transitions remain on-manifold with respect to the model, provided it is well-trained. This follows the same principle as in Dreamer [1] and related work, where stochastic data and latent dynamics promote reliable predictions.\n### (ii) Comparison with methods like PGR\n\nIn fact, compared to MPGE—which explicitly assembles transitions using the environment dynamics and ensures that the resulting buffer complies with the true transition function—methods that imitate and resample transitions directly from the replay buffer are more likely to violate dynamics consistency.\n\nTake PGR as an example: PGR models the joint distribution \n$p(s,a,r,s^{{\\prime}})$ by learning directly from tuples in the buffer. However, this paradigm does not enforce any structural constraints that the transitions must satisfy the environment dynamics. This lack of constraint makes the generative model prone to overfitting, potentially capturing spurious correlations between variables rather than learning the true mapping from \n$(s, a)$ to $(s^{{\\prime}}, r)$. For example, although both approaches model joint distributions $p(s, a,s^{\\prime},r)$, PGR may end up modeling a distribution of the form such as $p(s)p(a)p(s^{{\\prime}})p(r)$, which does not reflect the causal or functional dependencies present in the environment and may results in poor policy evaluation. \n\nThis observation is further supported by our experiments below, which demonstrate that MPGE is more capable of generating transitions that are dynamically consistent with the environment, in contrast to methods that rely solely on joint distribution modeling without enforcing transition constraints.\n# 2. Quantitative Analysis of MPGE (Supplementary Experiments)\n\nTo validate our analysis, we conducted an experiment to assess the dynamic compliance of transitions generated by MPGE and PGR. We trained a dynamics discriminator that classifies whether a given tuple \n$(s,a,s \n′)$ aligns with the true environment dynamics.\n\nThe discriminator was trained on positive samples from the replay buffer and a random policy—both guaranteed to be valid. To ensure fairness, MPGE and PGR were trained independently, each generating an equal number of negative samples, while their learning was not influenced by the discriminator’s feedback.\n## Structure of the discriminator:\nTo mitigate potential overfitting as much as possible, we introduced an information bottleneck in the design of the dynamics discriminator. Specifically, we adopt the architecture where the input pair $(s, a)$ is first processed through a multi-layer MLP, which is then directly connected to the encoded representations of \n$s^{{\\prime}}$. This combined representation is subsequently passed through a lower-dimensional MLP before producing a softmax-based binary classification output.\n ```\n    def forward(self, s, a, s_prime):\n        sa_feat = self.mlp1(torch.cat([s, a], dim=-1))\n        sp_feat = self.mlp2(s_prime)\n        fused = torch.cat([sa_feat, sp_feat], dim=-1)\n        out = self.fusion_discriminator(fused)\n        return out \n```\n\n## Results:\nWe conducted a joint training experiment on the  Humanoid-Run with 3 seeds, where the dynamics discriminator, PGR, and MPGE were trained simultaneously. \nThe resulting figure summarizing the method's accuracy (the ratio of the number of samples classified as real to the total number of test samples) over time is shown below:\n| Method | Accuracy (100k) | Accuracy (500k) | Accuracy (1M) | Accuracy (1.5M) |\n|--------|----------------|----------------|------------------|------------------|\n| MPGE   | 10.7%±5.8%    | 66.2%±15.1%    | 75.3%±11.4%       | 92.1%±4.5%      |\n| PGR    | 13.1%±3.2%    | 43.4%±7.9%   | 63.4%±9.3%      | 76.8%±10.6%       |\n\nThe results show that as training progresses, MPGE consistently generates a larger proportion of transitions that do not violate the environment dynamics constraints, thereby demonstrating the higher quality and plausibility of its generated data.\n\n# Q&A1: Discussion on guidance scale\nThe guidance scale plays a critical role in balancing sample diversity and target alignment.\nIn our current implementation, we select $\\omega$ through cross-validation on a held-out set (or via grid search across a small range, e.g., \n$[0.2,0.5,1.0,2.0]$) and apply the same value across all environments for consistency.\n\nIn our ablations, \n$ω=1$ performs best under standard training steps. However, with extended training (e.g., 3M steps), smaller scales (e.g., \n$ω<1$) also achieve strong performance, differing mainly in convergence speed. Larger scales tend to distort the generated data. For simple tasks, a small \n$ω$ is sufficient, while complex tasks may benefit from adaptive schemes such as annealing.\n|  | $\\omega=0.2$ | $\\omega=0.5$ | $\\omega=1$ | $\\omega=2$ |\n|--------|----------------|----------------|------------------|------------------|\n| TAR(3M)   | 608.2±21.3    | 616.2±24.7    | 624±12.5       | 402.3±9.8     |\n\n# Q&A2: Discussion on hyperparameter \nWe fully understand the reviewer’s concern regarding generalization and hyperparameter sensitivity. To address this:\n\n1. We used the same set of hyperparameters across all tasks on both benchmarks (DMC and Gym MuJoCo), without any task-specific tuning for MPGE.\n\n2. The world model parameters are directly adopted from existing implementations of DreamerV3[1] and STORM[2], and the diffusion model settings follow standard DDPM[3] configurations—no further tuning was applied.\n\n3. For our method-specific hyperparameters \n$λ$ and \n$ω$, we conducted ablation studies, and then applied the same values across all tasks.\n\nThese results demonstrate that MPGE maintains strong performance without extensive tuning, and we believe this reflects its robustness across diverse environments.\n\n#  Conclusion\nWe sincerely appreciate your detailed and constructive feedback. We hope our responses address your concerns and kindly request your consideration for a higher score.\n\n[1] D. Hafner et al. *Mastering diverse control tasks through world models*. Nature, 2025.\n\n[2] Zhang W, Wang G, Sun J, et al. Storm: Efficient stochastic transformer-based world models for reinforcement learning[J]. Advances in Neural Information Processing Systems, 2023, 36: 27147-27166.\n\n[3] Ho J, Jain A, Abbeel P. Denoising diffusion probabilistic models[J]. Advances in neural information processing systems, 2020, 33: 6840-6851."}, {"review_id": "jksPyrNqsd", "reviewer": "Reviewer_DNyW", "summary": "The paper proposes an exploration algorithm that uses a diffusion model to generate useful states for exploration (critical states). A world model is learned to augment these states, with transition tuples, and these tuples are integrated into off-policy updates. The algorithm exhibits strong performance on deep RL benchmarks.", "questions": "See Strengths And Weaknesses", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**: \n1. The paper is well-written. \n2. The problem studied in the paper is an important one. \n3. The algorithm exhibits strong performance against the compared baselines.\n\n**Weaknesses**: \n1. The algorithm itself seems much more computationally expensive, since it has to learn a diffusion model for critical state generation and a latent dynamics model. Hence, I believe that compared to standard off-policy RL methods, this would be more computationally expensive to run. This is not at all discussed in the paper. Furthermore, I think baselines such as [1-4] should also be considered in the paper. \n2. Dreamer and TD-MPC learn a world-model for POMDPs, where the dynamics are learned from observations. This requires stacking the observations together and compressing them in a relevant latent space. The encoder and decoder in this paper only compress the current state into the latent space. For POMDPs this would be incorrect. For MDPs, I am not quite sure why a latent model has to be learned instead of just learning a forward dynamics model, which maps the current state and action to the next state. \n3. Intrinsic exploration baselines, such as P2E [5], are not at all discussed in the paper. Moreover, methods such as [6,7,8] which combine intrinsic exploration objectives with the extrinsic rewards, are not discussed or ablated. Given that these methods are much simpler than what is proposed, I think the authors should compare against them to make a convincing case. \n4. Finally, I am not quite sure about the relevance/key novelty or insight of Theorem 1. Especially, given that the authors already assume convergence of the algorithm: \"Assuming that the policy converges to the optimal policy in finite steps\".  \n\n**References**:\n1. Chen, Xinyue, et al. \"Randomized ensembled double q-learning: Learning fast without a model.\" arXiv preprint arXiv:2101.05982 (2021).\n2. Hiraoka, Takuya, et al. \"Dropout q-functions for doubly efficient reinforcement learning.\" arXiv preprint arXiv:2110.02034 (2021).\n3. Nauman, Michal, et al. \"Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control.\" arXiv preprint arXiv:2405.16158 (2024).\n4. Lee, Hojoon, et al. \"Simba: Simplicity bias for scaling up parameters in deep reinforcement learning.\" arXiv preprint arXiv:2410.09754 (2024).\n5. Sekar, Ramanan, et al. \"Planning to explore via self-supervised world models.\" International conference on machine learning. PMLR, 2020.\n6. Chen, Eric, et al. \"Redeeming intrinsic rewards via constrained optimization.\" Advances in neural information processing systems 35 (2022): 4996-5008.\n7. Sukhija, Bhavya, et al. \"MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximization.\" arXiv preprint arXiv:2412.12098 (2024).\n8. Sukhija, Bhavya, et al. \"Optimism via intrinsic rewards: Scalable and principled exploration for model-based reinforcement learning.\" 7th Robot Learning Workshop: Towards Robots with Human-Level Abilities. 2025.", "was_revised": true, "final_justification": "The authors implemented most of the suggestions made during the rebuttal and therefore I have increased my score. However the submission now significantly differs from the initial submission and therefore I have decided to not increase the score beyond 4.", "rebuttal": "We sincerely thank you for the thoughtful and constructive feedback. Please find our detailed responses to the comments below.\n\n# 1. Supplemental experiments of MPGE\nThank you for raising this important point. To address your concern, we have conducted additional walltime experiments to assess the computational cost of MPGE. The updated results are provided below:\n\n| Env | MPGE | PGR | SER | DSAC|\n|--------|----------------|----------------|------------------|------------------|\n| Humanoid-run   | 151657±395    | 207813±1464    | 210321±788       | 128153±122      |\n\nThe results show that MPGE incurs moderate overhead compared to standard off-policy methods due to the added diffusion and world model components. However, this overhead is effectively controlled through two mechanisms:\n\n1. We perform MPGE training only once every 10 environment steps, significantly reducing overall compute.\n\n2. Unlike PGR-style methods that require retraining the generative model to convergence before use, MPGE supports online training with incremental updates, which simplifies training and improves efficiency.\n\nWe will include this discussion and the corresponding results in the final version of the paper.\n\nAdditionally, we appreciate your suggestion regarding baseline selection. We agree that including recent off-policy exploration methods such as [1–4] would provide a more comprehensive evaluation. We conduct the experiments in DMC Humanoid tasks, which are one of  the hardest tasks within 3 seeds, and the results are illustrated as follows:\n\n\n| Env | MPGE | Simba | BRO | RedQ |DroQ |\n|--------|----------------|----------------|------------------|------------------|------------------\n| Humanoid-run   | **489 ± 9**   | 268± 40    | 417± 15       | 187± 12      |164± 21|\n| Humanoid-walk    | **892 ± 19**    | 801 ± 13   | 881 ± 25      | 665± 5        |682± 14\n| Humanoid-stand    | 907 ± 7   |    **920±14**      | 905±3      |902±4| 896±6| \n\nAs shown in this task, except for the simplest case (Humanoid-Stand), MPGE achieves the best performance across all more challenging tasks.\n# 2. Discussion of the Encoder-decoder structure in MPGE\nThank you for this insightful question. We would like to clarify that MPGE is developed under the MDP setting. And why we choose to model dynamics in a latent space instead of directly learning the mapping \n$f(s,a)→s′$ is motivated by several practical considerations:\n\n1. Learning a compact latent representation helps capture abstract, task-relevant features of the environment, improving generalization and training efficiency even under MDPs. Besides, different dimensions of the state may contribute unevenly to dynamic modeling. Using an encoder to transform the raw state allows the model to extract the most relevant components, which facilitates more accurate transition prediction.\n\n2. Latent dynamics are often smoother and easier to predict compared to raw state transitions, especially in high-dimensional environments. Fitting dynamics in a latent space typically improves the accuracy of transition modeling.\n\n3. Decoupling representation learning from dynamics prediction allows better reuse and transfer of components. In fact, in MPGE, our policy network is built on top of the same encoder, enabling a unified state representation that facilitates more effective policy learning.\n\nTo further support our claim, we conducted a simple experiment across three environments (each with 3 random seeds), using the same diffusion and baseline algorithm setup, to compare the TAR between vanilla dynamics (Using Transformer as predictor of next states and rewards) and latent dynamics. The results are as follows:\n\n| Env | MPGE with vanilla dynamics | MPGE with latent dynamics  |\n|--------|----------------|----------------|\n| Humanoid-run   | **489 ± 9**    | 408± 35    | \n| Humanoid-walk    | **892 ± 19**     | 684 ± 71  |\n| Humanoid-stand    |**907 ± 7**   | 785 ± 88       |\n\nThe results show that, compared to directly mapping the current state and action to the next state, introducing a latent space leads to better learning of both environment transitions and state representations in the policy network.\n\n# 3. Comparison with different exploration baselines\nThank you very much for the valuable suggestion. In response, we have conducted additional experiments to study the effect of different exploration strategies on the DSAC algorithm via ablation comparisons. Due to the development and implementation effort involved, and time constraints, we focus on three representative environments—the Humanoid tasks in DMC—for this evaluation.\n\nBefore presenting the results, we provide the following clarifications regarding the methods considered in this comparison:\n\n1. **Plan2Explore**: Due to the limited compatibility of the original codebase, we re-implemented this method within our own framework. Specifically, we replaced the world model with Dreamer, and trained five independent predictors to construct a latent disagreement signal. An exploration policy was then trained using this signal to collect additional data for the main task policy.\n2. **EIPO**:  All experiments of EIPO are conducted on on-policy algorithms, specifically based on PPO, which poses challenges when attempting to transfer the method to off-policy algorithms such as DSAC. Although EIPO is theoretically algorithm-agnostic, **applying it to off-policy methods (e.g., DSAC) poses practical challenges**. Its core mechanism relies on alternating optimization between two policies and dynamically adjusting intrinsic reward via Lagrangian updates. However, off-policy training uses replay buffers, making it difficult to accurately evaluate both policies’ extrinsic returns for updating the multiplier. Additionally, the assumption of policy closeness often fails in off-policy settings, reducing surrogate objective reliability. **Due to these challenges and time constraints, we were unable to include EIPO in our experiments. We appreciate your understanding.**\n3. **MaxInfoRL**: The method is highly transferable, and we directly integrated its core components into our own codebase. We also attempted to build DSAC on top of the original implementation, but the performance was not as good as our reimplementation. Therefore, we report results based on the version implemented in our framework.\n4. **OMBRL**: Since the original implementation is in JAX, we reimplemented the method in our own codebase by replacing the SAC component in MBPO with DSAC. We use Deep Ensembles to compute \n$σ_n​(x,a)$, with each network sharing the same architecture as the predictor used in **Plan2Explore**.\n\nThe final results are as follows:\n| Env | MPGE | Plan2Explore | MaxInfoRL | OMBRL |\n|--------|----------------|----------------|------------------|------------------|\n| Humanoid-run   | **489 ± 9**    | 311 ± 12    | 197± 4      | 262±13      |\n| Humanoid-walk     | **892 ± 19**    | 588 ± 7   | 481 ± 7      | 678± 5      |\n| Humanoid-stand     | **907 ± 7**     | 801± 14   | 844±8      | 769±11       |\n\nThe results show that MPGE still offers a significant advantage compared to these methods.\nIntrinsic exploration methods encourage visiting novel or unpredictable states, but their signals (e.g., prediction error, uncertainty) are often task-agnostic and may lead to uninformative or misaligned exploration. In contrast, **MPGE leverages task-aware utility functions (e.g., TD-error, entropy) to generate states that are directly aligned with policy improvement objectives**. Besides, unlike intrinsic reward methods that require careful reward balancing and affect the policy's optimization objective, MPGE decouples exploration from reward design, enabling more stable training without interfering with the task-specific learning signal.\n\n\n# 4. Clarification of Theorem 1\nWe fully understand your concern regarding this statement. \n\nIn fact, the purpose of this theorem is to ensure that the generated states, which are learned based on the state distribution in the replay buffer, can gradually follow a steady-state distribution.\nThis is motivated by existing studies[4-5] showing that unstable or non-convergent training distributions in generative models often lead to degraded or uncontrolled generation quality. \n**Our theoretical analysis aims to provide a principled guarantee that the generated states remain within the environment’s reachable state space, thereby ensuring that they can effectively support policy exploration and improvement.**\n\n**As stated in line 532 of the paper, we also provide conditions under which the conclusion still holds even if the policy does not satisfy finite-step convergence.**\nAnd the actual assumption made here is that the **policy changes diminish over time as optimization proceeds**. Importantly, **the final stationary policy distribution does not need to correspond to an optimal policy** (our derivation does not rely on any properties of the optimal policy, such as its return being the highest).\nIn our paper,\nLastly, we would like to reiterate that this is merely an idealized assumption. Our theoretical result is not tied to the convergence of the optimal policy, but only to its convergence rate, at which policy updates diminish over time. \n\nWe acknowledge that our previous wording may have caused confusion, and we will clarify it accordingly in the final version. We are sincerely grateful for your thoughtful question and suggestion.\n# Conclusion\nWe sincerely thank you for your thoughtful and constructive feedback. In response, we have conducted substantial additional experiments and analyses to directly address your concerns. We hope these efforts help clarify the points in question, and we would be truly grateful if you would consider them in your consideration for a higher score."}, {"review_id": "h5gOVtCTNQ", "reviewer": "Reviewer_ZXF4", "summary": "This paper introduces Model-based Prioritized Generative Exploration (MPGE), a novel approach to enhance exploration in reinforcement learning, particularly within off-policy settings. MPGE addresses the limitations of traditional passive exploration by generating under-explored, critical states and synthesizing dynamics-consistent experiences. \nThe method comprises two main components: a diffusion generator guided by policy entropy and temporal difference error to identify critical states, and a one-step imagination world model to construct valuable transitions for learning. \nExperimental results on OpenAI Gym and DeepMind Control Suite benchmarks show that MPGE improves exploration efficiency and overall performance in continuous control tasks.", "questions": "1. In Theorem 1, why is it assumed that the policy converges to the optimal policy in a finite number of steps? The convergence of the training policy depends on your algorithm; why make this assumption at the outset?\n2. How does the proposed method compare to on-policy methods in terms of performance?\n3. Can the approach be generalized to discrete action spaces, such as those in Atari environments?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The exploration of off-policy scenarios is both important and interesting. The proposed method appears to be novel.\n2. The introduction is well-written, and the literature review is comprehensive.\n3. Experimental results on OpenAI Gym and DeepMind Control Suite benchmarks show that MPGE improves exploration efficiency and overall performance in continuous control tasks.\n\nWeaknesses:\n\nThe methodology section is somewhat confusing. I have several questions regarding the methods:\n1. The explanation of Equation 5 is unclear; it is not obvious how it is derived or how the variable $w$ is introduced.\n2. The two paragraphs on \"Policy Entropy\" and \"Temporal Difference Error\" appear abruptly in Section 3.1. Are these intended to be part of the method? Concepts like TD error might be better placed in the Preliminaries section.\n3. In Equation 10, the labels for reward and reconstruction seem to be swapped.\n4. In Equation 10, the use of sg() appears to bring $g(s_{t+1}) $and $d(g(s_t, a_t))$ closer together. Does learning both the representation and dynamics simultaneously lead to unstable training?\n5. The experiments are conducted on continuous control tasks. Can the method generalize to discrete-action space cases?", "was_revised": true, "final_justification": "The authors addressed my concerns in the rebuttal. I raise my score to 4.", "rebuttal": "We sincerely appreciate your detailed and constructive feedback. Below are our responses to your concerns.\n\n# 1. Explanation of Equation 5\nEquation (5) follows the standard gradient decomposition for classifier-guidance diffusion models [1]. It is used to guide the generation of states toward those favored by the target function. The guidance scale coefficient $\\omega$ controls the strength of this guidance, balancing between following the unconditional diffusion prior and aligning with the target utility function.\n\n# 2. Description of the two utility functions\n\nWe appreciate the reviewer’s suggestion and agree that these concepts are better introduced earlier. In the revised version, we will move the descriptions of policy entropy and TD error to the Preliminaries section for clarity. \nWe also clarify that these are example utility functions within our framework, and other choices are equally applicable to support the generation of diverse and informative critical states. \n\n# 3. Swapped label\nThank you very much for pointing this out! We confirm the labels are indeed swapped and will correct this in the final version.\n\n# 4. About Equation 10\nThank you for the insightful question. We follow the design principle introduced in Dreamer series[2-3], where stop-gradient (sg) is applied to avoid representational collapse and to stabilize joint training of the encoder $g_\\phi$ and the dynamics model $d_\\phi$. Specifically, the sg operator ensures that the prediction target $g_{s_{t+1}}$ is treated as fixed when training the dynamics model. At the same time, the target is treated as fixed when training the encoder, so that the encoder is not optimized to merely match the dynamics model output, but instead learns representations that are useful for downstream tasks such as value prediction or policy learning.\n\nIn summary, this design is implemented to prevent trivial solutions where both modules co-adapt to minimize the loss without learning meaningful structure. Moreover, Dreamer demonstrates that using such asymmetric losses with sg leads to more stable and scalable representation learning in model-based RL. Empirically, we observe that this design promotes stable convergence across tasks, consistent with prior works.\n\n# 5. Extension\nYes, our method can be extended to discrete action spaces. As long as a differentiable \nmapping from given utility functions to states can be established, it remains feasible to compute utility functions under a state, resulting in critical generation. While our current experiments focus on continuous control tasks, we plan to explore more discrete-action environments in future work.\n\n\n# Q&A1: Clarification of Theorem 1\nWe fully understand your concern regarding this statement. \n\nIn fact, the purpose of this theorem is to ensure that the generated states, which are learned based on the state distribution in the replay buffer, can gradually follow a steady-state distribution.\nThis is motivated by existing studies[4-5] showing that unstable or non-convergent training distributions in generative models often lead to degraded or uncontrolled generation quality. \n**Our theoretical analysis aims to provide a principled guarantee that the generated states remain within the environment’s reachable state space, thereby ensuring that they can effectively support policy exploration and improvement.**\n\n**As stated in line 532 of the paper, we also provide conditions under which the conclusion still holds even if the policy does not satisfy finite-step convergence.**\nAnd the actual assumption made here is that the **policy changes diminish over time as optimization proceeds**. Importantly, **the final stationary policy distribution does not need to correspond to an optimal policy** (our derivation does not rely on any properties of the optimal policy, such as its return being the highest).\nIn our paper,\nLastly, we would like to reiterate that this is merely an idealized assumption. Our theoretical result is not tied to the convergence of the optimal policy, but only to its convergence rate, at which policy updates diminish over time. \n\nWe acknowledge that our previous wording may have caused confusion, and we will clarify it accordingly in the final version. We are sincerely grateful for your thoughtful question and suggestion.\n\n# Q&A2: Comparison of the on-policy methods\n We have reported comparisons with two representative on-policy methods, PPO and TRPO, in Appendix C.2. Please refer to that section for detailed results.\n# Q&A3: Discrete action spaces generalization\nYes, the approach can be generalized to discrete action spaces.  Please refer to our earlier response above for a detailed explanation.\n\n#  Conclusion\nWe truly appreciate your thoughtful and constructive feedback. We hope that our detailed responses and additional efforts have addressed your concerns, and we kindly ask that you take them into consideration during your re-evaluation.\n\n\n[1] Dhariwal P, Nichol A. Diffusion models beat GANs on image synthesis[J]. Advances in neural information processing systems, 2021, 34: 8780-8794.\n\n[2] D. Hafner et al. *Learning latent dynamics for planning from pixels*. ICML, 2019.\n\n[3] D. Hafner et al. *Mastering diverse control tasks through world models*. Nature, 2025.\n\n[4]Argenson A, Dulac-Arnold G. Model-based offline planning[J]. arXiv preprint arXiv:2008.05556, 2020.\n\n[5]Luo C. Understanding diffusion models: A unified perspective[J]. arXiv preprint arXiv:2208.11970, 2022."}]

Author rebuttal:

Meta-review: This paper tries to improve exploration in off-policy RL. They proposed Model-based Prioritized Generative Exploration (MPGE).  The method consists of two parts: a diffusion model that generate critical and underexplored states guided by metrics such as policy entropy, and a world model that learns the transition. While the idea is related to existing methods such as Prioritized Generative Replay (PGR), subtle and critical differences make the proposed method outperforming them in certain environments.  

In the reviewer-author discussions phase, the authors provided additional evidences on the novelty and qualitative analysis that makes the proposed method more convincing. All reviewers' concerns are well-addressed.  Overall, the problem studied is important, the proposed idea is interesting and general, and could definitely benefit the community.  

The authors have to incorporate the new experiments in the discussion phase into the final version of the paper.
Decision: Accept (poster)