GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learning
Authors: Shutong Ding
Ke Hu
Shan Zhong
Haoyang Luo
Weinan Zhang
Jingya Wang
Jun Wang
Ye Shi
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning
On Policy Learning
Generative Diffusion Models
Primary area: reinforcement_learning
TL;DR: We propose GenPO, which effectively incorporates invertible diffusion model into on-policy RL, and deals with the challenge of log-likehood computation in diffusion policies.

Abstract: Recent advances in reinforcement learning (RL) have demonstrated the powerful exploration capabilities and multimodality of generative diffusion-based policies. While substantial progress has been made in offline RL and off-policy RL settings, integrating diffusion policies into on-policy frameworks like PPO remains underexplored. This gap is particularly significant given the widespread use of large-scale parallel GPU-accelerated simulators, such as IsaacLab, which are optimized for on-policy RL algorithms and enable rapid training of complex robotic tasks. A key challenge lies in computing state-action log-likelihoods under diffusion policies, which is straightforward for Gaussian policies but intractable for flow-based models due to irreversible forward-reverse processes and discretization errors (e.g., Euler-Maruyama approximations). To bridge this gap, we propose GenPO, a generative policy optimization framework that leverages exact diffusion inversion to construct invertible action mappings. GenPO introduces a novel doubled dummy action mechanism that enables invertibility via alternating updates, resolving log-likelihood computation barriers. Furthermore, we also use the action log-likelihood for unbiased entropy and KL divergence estimation, enabling KL-adaptive learning rates and entropy regularization in on-policy updates. Extensive experiments on eight IsaacLab benchmarks, including legged locomotion (Ant, Humanoid, Anymal-D, Unitree H1, Go2), dexterous manipulation (Shadow Hand), aerial control (Quadcopter), and robotic arm tasks (Franka), demonstrate GenPO’s superiority over existing RL baselines. Notably, GenPO is the first method to successfully integrate diffusion policies into on-policy RL, unlocking their potential for large-scale parallelized training and real-world robotic deployment.

GenPO: Generative Diffusion Models Meet
On-Policy Reinforcement Learning
Shutong Ding 1,,+
Ke Hu
Shan Zhong
Haoyang Luo
Weinan Zhang?
Jingya Wang
Jun Wang"
Ye Shi
ShangghiiTech University University Electronic Science and Technology of China Shanghai Jiao Tong University *University College London Moo Key Laboratory of Intelligent Perception and Human Machine Collaboration
fdingsht, huke2024,
edu, cn
20221 202211409276std.. uestc.edu.co cn
vnzhang@sjtu.edu...
wangocs.ucl.ac..u ac. (wangjingya, shiyee/eshanghaaitcch...u edu. cn
Abstract
Recent advances in reinforcement learning (RL) have demonstrated the powerful exploration capabilities and multimodality of generative diffusion-based policies. While substantial progress has been made in offline RL and off-policy RL. settings, integrating diffusion policies into on-policy frameworks like PPO remains underex- plored. This gap is particularly significant given the widespread use of large-scale parallel GPU-accelerated simulators, such IsaacLab, which are optimized for on-policy RL algorithms and enable rapid training of complex robotic tasks. A key challenge lies in computing mate-action logeliselssoods under diffusion policies, which is straightforward for Gaussian policies but intractable for flow-based models due irreversible oowarddreverse processes and discretization errors (e.g.. Euler- Maruyama approximations). To bridge this gap, we propose GenPO, a generative policy optimization framework that leverages exact diffusion inversion construct invertible action mappings. GenPO introduces novel doubled dummy action mech- anism that enables invertibility via alternating updates, resolving log- lieelikehhood computation barriers. Furthermore, we also use the action likeliool for unbi- ased entropy and KL divergence estimation, enabling KL-adaptive learning rates and entropy regularization in on-policy updates. Extensive experiments on eight IsaacLab benchmarks, including legged locomotion (Ant, Humanoid, Anymal-D, Unitree H1, Go2), dexterous manipulation (Shadow Hand), aerial control (Quad- copter), and robotic arm tasks (Franka), demonstrate GenPO's 's superiority over existing RL baselines. Notably, GenPO is the first method successfully integrate diffusion policies into on-policy RL, unlocking their potential for large-scale paral- lelized training and real-world robotic deployment. The official implementation of GenPO is provided in https: //githh// cww http:///xxx1//////// cow///axx201///gnnooooooooooo///
Introduction
In recent years, generative diffusion models, such DDPM 25 DDIM [52) and flow match- ing [36] (35 have attracted considerable attention from reinforcement learning researchers due
"Equal contribution. iCorresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

On-policy RL
Off-policy RL
Offline RL
Diffusion policy
Diffusion policy
Diffusion policy
ac+1
dc
S1+1
Log-Ukkethhood diffusion?
Q-grad
data
frain
train
Train
Interaction
Interaction
Innerraa Secalte
Returns
Critic
Offline Dataset
transition
transition
Simulator or Real-wortd
Simulator or Real-world
Simulator or Real-world
Figure 1: Existing Liffusion-basse reinforcement learning algorithms mainly focus on the off off-policy (middle) and offline (right) RL. This is because to update the diffusion policy in off-policy RL and utilize the offline data to train the agent in offline RL. However, as for diffusion-based RL in the on-policy (left) algorithm, there still exists challenge that we cannot obtain the log-likelihooe of diffusion.
generally obtain the gradient of the Q function
their powerful exploration capabilities and multimodality compared with traditional parame- terized Gaussian policies. However, existing works in this field almost merely focus on offline RL 660
59 methods, and the inte- gration of diffusion or flow-based policies into on-policy reinforcement learning like PPO [51] remains largely unexplored. However, existing massively parallel GPU-accelerated simulators, such as Isaac Gym (38) and its successor, IsaacLab, mainly [32] benefit the training of on-policy reinforcement learning algorithms, especially PPO, which are capable of achieving substantial performance in relatively short wall-clock time. This has resulted in fundamental dilemma, hindering the practical deployment of existing diffusion//low-bart RL algorithms real-world
24
40 and off-policy RL [62
applications [10 31 45 [1].
Different from diffusion models 13 applied off-policy RL, the primary challenge in integrating flow policies with on-policy reinforcement learning is how to compute the log- ligeliidd of state- action pairs, while straightforward for Gaussian policies. The underlying reason here lies in the inconsistency and non-reversibility between the forward and reverse processes of the generative diffusion policy, caused by the Euler- Maruuaaa (EM) discretization 48 5) Nevertheless,
optimization objective in on-policy RL necessitates the probability density of the given state-action pair. In this context, estimating the probability density of a given state-action pair in flow or diffusion models is nontrivial, and further utilizing this estimate for diffusion flow policy improvement in the on- on-policy RL. paradigm presents additional challenges.
To address these challenges, we propose GenPO, motivated by the exact diffusion inversion [5]] which constructs an invertible flow mapping and thus avoids the mismatch of forward and reverse processes of the diffusion policy. Specifically, we first construct new Markov process with the action space twice the size of the original in GenPO due to the limitation of exact diffusion inversion, and then enable reversible flow processes by alternately updating the two parts of the dummy action. In that case, we can calculate the exact probability density of the given action in this new Markov decision problem via change of variables theory like normalizing flow [46] [5]. Moreover, the two parts of the dummy action will be averaged as one and then mapped to the original action space during inference. Hence, reformulated Markov decision problem, but finally obtain the optimal solution to the original task.
perform GenPO to update the generative diffusion policy in the
In addition, with the exact probability density, we estimate [ the entropy of the diffusion policy and the Kullback-Leibler (KL.) divergence with respect to the behavior diffusion policy. This allows for the combination of entropy regularization and KL-adaptive learning rate adjustment into GenPO, which has been confirmed effective PPO. To demonstrate the superiority of GenPO, we conduct experiments 8 IsaacLab benchmarks [38), which cover robot ant, humanoid, quadcopter, Franka

---

robot arm [23]. Shadow dexterous hand [56]. ANYbotics anymal-D, and Unitree legged robots [68] The final results show that the proposed GenPO outperforms previous RL baselines in cumulative returns, while maintaining comparable sample efficiency and faster convergence. It is worth noting that GenPO is the first trial to incorporate the generative diffusion policy into on-policy RI. algorithms and opens brand-new avenues for diffusion-type policies applied in large-scale parallel simulators and real-world robot control. Our contributions are summarized as follows:
Bridging generative diffusion models and on-policy RL. GenPO achieves the first stable integra- tion of diffusion based policies with on-policy reinforcement learning, successfully deploying the inherent exploration capabilities and multimodality of diffusion models in large-scale GPU parallel simulators (IsaacLab).
Closing the likelihood gap in diffusion policies. Unlike Gaussian policies with closed-fon densities, diffusion policies lack tractable likelihoods. GenPO overcomes this by using invertible diffusion dynamics, enabling 1) exact log-likelihood computation, 2) unbiased entropy estimation, and 3) analytical KI. preeggnne-bbriggng Gaussian-style advantages to expressive diffusion models.
Comprehensive experiments IsaacLab benchmarks. We evaluated GenPO IsaacLab benchmarks. The experimental results show that in a massively parallel environment, previous diffusion-basee reinforcement learning algorithms are almost ineffective, while GenPO achieves the best performance in terms of sample utilization efficiency and episodic rewards, far surpassing other algorithms.
2 Related Works
In this section, we review the literature of generative models such VAE (28). GAN 112. normalizing flow [46], diffusion [65]. and flow-based model [36] (34] for policy learning, and generally divide them into two classes according their learning paradigm.
Generative Policy for Online Reinforcement Learning. The goal of online RL 54 optimal policy with the interactions of the given environment. It is challenging to train generative model policy within an online RL paradigm due to the absence of the action label. Generative policy learning methods in online RL can be broadly grouped into two paradigms. The first paradigm views the generative policies as a black-box function approximation like neural networks, and optimizes them via the policy gradient. In this paradigm, representative methods include DACER (59), CPQL 19, FlowPG 44. SAC-NF (39). and TRPO-NF [53), which directly apply deterministic or policy gradient on the generative model policies. By contrast, the second category leverages the internal structure and mechanisms of the generative model itself to perform the policy improvement. For instance, DIPO (62) and QVPO [13 respectively utilize the gradient and the value magnitude of the Q-function to identify optimal actions, which are then applied to the variational loss of the diffusion model. Additionally, QSM 43 and MaxEntDP (16) regard the noise network in the diffusion model as the score function of the target policy distribution and then employ the gradient of the Q-function and its variants train the noise network. As normalizing flow-based approaches, MEow (6) exploits the layer-wise nonlinear property of the flow network to train the Q and V functions, which yields the corresponding maximum-entropy normalizing flow policy based on SAC [2 (2). Overall, current approaches that employ generative model policies in online RL have predominantly focused on integrating these models with off off-policy RL algorithms like SAC, with few efforts to combine them with on-policy RL methods.
is to learn an
Generative Policy for Other Learning Paradigms. Compared with generative policy for online RI., more existing research works on generative policy have been devoted to offline RI. (30) and imitation learning paradigms, wherein policies are learned from offline datasets without environment interaction. For offline RL, BCQ [18), PLAS [66], SPOT [61] adopt VAE (28) policy regularizer to
prevent the policy from optimizing outside the support of the dataset. Besides, CPED (64] utilizes
flow GAN 20 to achieve more accurate policy regularization. Furthermore, diffusion- QL [60].
EDP 27.. SRDP [.. CEP [7], and FQL [1 leverage diffusion (25) and flow [36] 34] policies to
accurately model the policy distribution in offline datasets, thereby obtaining policies with superior generalization capability. Regarding imitation learning, diffusion policy [10] and diffuser (26 serve as canonical examples of using diffusion models to fit the state-action mapping and state-trajectory mapping, respectively. The latter one is also referred to as the diffusion planner. Subsequently, FMIL 47). AVDC [29]. and decision diffuser [2] follow their steps and demonstrate enhanced
3

---

performance in different robots. Furthermore, DPPO 44 applies policy gradient to fine-tune the diffusion policy trained with the offline dataset. Nonetheless, DPPO merely explores the use of the RI technique in finetuning an offinne-peetrinee diffusion policy, rather than representing diffusion-based online RI. algorithm.
However, all of the above generative policy learning approaches have not been effectively integrated with on-policy reinforcement learning algorithms, making them unable to sufficiently leverage modern massively parallel GPU-accelerated simulators. Specifically, off-policy algorithms often struggle with convergence in such simulators, while offline RL and imitation learning methods are fundamentally incompatible with them due to the lack of online interaction. In contrast, our proposed Generative diffusion Policy Optimization (GenPO) seamlessly integrates with on polooiicy algorithms. enabling efficient training of high-pes tigh-performing generative policies within short wall-clock time using large-scale GPU-parallelized simulation platforms.
Preliminaries
3.1 On-policy Reinforcement Learning
Reinforcement learning problems [54] are formalized as Markov decision processes (MDPs), defined
by the tuple (S,A, p.r:po 7), where 8 is the state space, A is the action space, p(s' 8, a) function, Po(s) :
x S x A + 0. x)) denotes the transition dynamics, r(s, a) S x A - IR is the reward
[0, oo) is the distribution of the initial state, and y 10, is the discount factor for the value estimation. The goal of RL is to maximize the expected discounted return as (0) Erw E 7'Te]. where To (a 8) is the behavior of the agent parametrized by and induces a corresponding distribution over behavior trajectories T. For convenience, RL also defines two different value functions: the state value function (s) E < Ei= 7're = and the state-action value function Q= a) = Ex iEr= Y 7'relo0 = So s). Besides, the advantage function, which represents the benefit of taking one specific action compared with the expected return at the given state, is defined as A₂ a) = (= a) (*).
On-policy RI. algorithms train the policy with the transitions generated by the current policy itself, while off-policy RL algorithms use the transitions from a different policy as training samples. While on-policy RL algorithms preserve the consistency between the behavior and target policies, but often suffer from high variance and sample inefficiency. Proximal Policy Optimization (PPO) [5]) addresses these challenges by introducing the clipped surrogate objective:
CLII (0) = E((,,,,)
min (r)(0)A,, clip(r.(0), 1001--e,,1
-,, e, e,1++)
where re( To (a:s) and A, is the estimation of the advantage function via GAE 50 By
Tea [as]s.]
leveraging the importance sampling technique and trust-region insights [49] without second-order
complexity for approximation of the KI. divergence constraint, PPO can achieve stable and efficient policy updates. The simplicity, robustness, and strong empirical performance on comtinuous-control
benchmarks [38 688 of PPO have made the first algorithm be adopted in many domains.
3.2 Generative Diffusion Models and Flow Matching
are a kind of powerful latent variable generative model that transform samples from the standard Gaussian distribution Ø7 -N(0,) 7) to the data distribution To ~ Paata via the denoising procedure with the noise network. The relationship between xn and Fr is defined by the forward SDE of diffusion
Generative diffusion models 25
53 52
dz.- [(x,, () di + 9())
where W2 the standard Wiener process (a.k.a., Brownian motion), fIz, t) is vector-valued function called the drift coefficient of It. and g(t)s scalar function known as the diffusion coefficient of x,, According to [53], the reverse-time SDE of diffusion (i.c., denoising procedure) is
dr: = [f(*.. () 2V (og))( dt g(t) dw..
(3)
where W. is a standard Wiener process when time flows backwards from T to 0. In prac- tice, DDPM (25) discretizes the SDE and approximates the forward step as q(I, 2x-1) :=

---

p(zo)
p(a1)
Invertible Diffusion Pollcy
Ni I)
Dummy
action
211
State
avg
Action
yo
Vt
Mr+
y/1
p(yo)
p(y)
Eppos EpposRant Rent p(ur, U1) = plan, vo) det
0(a..)) Mao.00)
Ceompp
Training loss Eypo lent Loomp
Figure 2: Forward and reverse process of GenPO. The forward process is to sample actions with the
given state; the reverse process is compute the probability density of the given state-action pair. Notably, the forward and reverse processes are invertible.
(zys TT Be1). where 3g is the variance schedule. The noise network e ( t) in DDPM can be trained by the variational lower bound loss:
E (I [[|& colVoroo + v1 Cee, 8)]?
where 0t 1 3r a. II. at., and E; 1),
The flow matching model 36 34 is developed for faster generation compared with classical diffusion models like DDPM, and be viewed as special type of generative diffusion model [19]. Similar to DDPM, flow matching model learns a time-dependent vector field vy (z. t), i.e., noise network in DDPM, that transports samples along predefined family of conditional distributions {perrejo.u) satisfying T1 Pdata and zo (N(0, Hence, the sample from the target distribution Pdato can be achieved via solving the ode r 2n Jo v2z,, t)dt.
Besides, The loss of flow matching is given as L()) = fo' Eppp, [2v0 u" (z, t)ll dt, where v" (2,, t) is the true velocity field under the chosen path. When the path is Gaussian diffusion, o" (z, t) coincides with the score-drift term of the reverse SDE in diffusion. However, the log ligleiilidood of the given sample cannot be calculated directly in the generative diffusion framework.
4 GenPO: An On-policy RL Method based on Diffusion Models
As mentioned in Section 3.2 the log-likelihood of the given samples cannot be computed directly in generative models. However, to integrate diffusion policy into on-policy RL, we must present the explicit formula of log-like likeeioddd under the given state-action pair for the policy update. Hence, we first elucidate the intrinsic reasons that make the computation of the log- likelihood diffusion models challenging. Then, we address these challenges by (1) applying the exact diffusion inversion technique (57] and utilizing the change of variables technique like normalizing flow [46) (2) con- structing reformulated Markov decision process with modified action space, along with a trick that maps the generated actions back to the original action space.
Besides, based on this framework, we also introduce an unbiased estimation method for both the entropy of the diffusion policy and the Kullback-Leible (KL) divergence between different generative diffusion policies. Notably, this has not yet been accomplished by previous diffusion-basee RL methods. addition, since policy update actually performed on reformulated MDP, extraneous exploration is induced, thereby degrading RL learning efficiency. To address this issue, we also introduce an auxiliary compression loss term. Leveraging these techniques, we finally present practical implementation of the proposed GenPO algorithm, which successfully incorporates the generative policy to PPO [5I an effective manner. Figure shows the training and inference process of the proposed GenPO.
5

---

4.1 Challenges in Calculating Diffusion Probability
Recalling existing generative models, such as VAE [28]. GAN 12,
normalizing flow [46], and
diffusion model [53]. it can be found that only normalizing flow and its variants like flow GAN 20
allow the exact likelihood computation via the change of variables theorem [46]. In contrast, for
other generative models, probability densities or related statistics can only be obtained approximately
via special designs [13 5 [9] Applying such approximate probability density estimates for policy update is unacceptable on-policy RL algorithms.
According to Lemma 1, if we want to compute the probability density of a generative model exactly, the generative model must be invertible between the sampling distribution, like the standard Gaussian distribution, and the target distribution. Thus, we consider designing a reversible generative diffusion model and employing the change of variables theorem (5) to compute its probability density.
Lemma 1. (Change of Variables (3) Let J :R - TK is an invertible and smooth mapping. If we have the random variable X ~ q(x)) and the random variable Y = S(X) transformed by function f. the distribution ply) of is
p(y) = q(x) det 0s dr
(5)
58 63 it is nontrivial to realize an exact invertible diffusion model. This is because of the inconsistency between the forward and reverse processes of the generative diffusion model. (6) shows the forward and reverse process of DDIM [52]. It can be observed that this inconsistency caused by the approximation of o4 with Ft-1 in the forward process, since the Ie is unavailable in the forward step
However, as mentioned in 57
Jt VI (1eg(Tent) + ar-aa-i
Reverse: It-1 = vot-1
Vai
(6)
Forward: 24 Tt-1 (2co(
81-1 - b,co(oe-1.t))
where 0,
V0t-1/01 by ---vaa Vat-1 an)/ar + v1 0tt.
4.2 Exact Diffusion Inversion in Reformulated MDP
Consequently, realize invertible diffusion model, this issue must be addressed. Motivated by EDICT [57], we realize the invertible diffusion model via maintaining two coupled noise vectors and updating them alternately in the forward and reverse processes of the generative diffusion model, and then extend it into flow matching [35] 36] for fast generation as shown in 77. . However, the two coupled noise vectors also lead doubled sample space. This implies we cannot directly apply this technique to diffusion policy when the dimension of the action space is odd.
To resolve this problem, we reformulate the original MDP problem with doubled dummy action space A in GenPO. Each dummy action ä (Ir, y) consists of two components, which subse- quently averaged to produce single action aty in the original space. It is obvious that, when the policy in the reformulated MDP is optimal, it is also optimal with the average mapping in the original MDP problem.
Reverse:
T+tar = Ft + Uo(ye, t)At,
Ur+a: y: + Ug (ā (c, t)
Mixing:
T++A1 = =p-I+4A + (1 (1-1) pp :) De Dr+ p- Dr+ + (1 -ppp) p) Er+
Notably, different from EDICT, which just applies exact diffusion inversion for the inference of diffusion, GenPO independently samples the standard Gaussian noise for 2o, Vo rather than samples one noise and sets Zg = yo to allow for the probability calculation using changes of variables (5). Moreover, due to the doubled action space the modified MDP, there exists meaningless exploration problem in the diffusion policy. For instance, if the optimal action a* 0, we have (-1, (-2, 2) are both the optimal dummy action. Hence, to prevent unnecessary exploration in the modified MDP, a mixing scheme (7 8 is adopted so that the two parts x. of dummy action components remain closely aligned. In 7 the mixing scheme employed to facilitate information exchange between the and y components of a, thereby ensuring the diffusion policy converges to

---

dummy actions with minimal discrepancy between x and y parts. Here the coefficient is used to control the intensity of the interchanged information.
Unmixing:
M++1 Pt- (1-
L++ Pe-Ar-(1-pia)
Forward:
y/2 = VI+ Uo 10( t)tt
21 = L(+S( - uo vo (vl.htt) t)
4.3 Practical Implementation
To enhance GenPO's exploration and stabilize training, we estimate policy entropy and introduce an adaptive KL-divergence-basee learning rate schedule for flow policies. Unlike Gaussian policies, flow policies lack closed-form expressions for entropy and KL divergence, preventing direct computation. have relied on heuristic approximations
As a result, prior diffusion RL. algorithms 133 59 S
to encourage exploration. In contrast, GenPO exactly invertible (Section 4.2), allowing precise computation of the log-likelihood any state-action pair via the change-of-variabe (5) in Section This enables unbiased estimation of both entropy and KL divergence. Accordingly, define the
entropy loss in Eq. 9
ENNT (TB) == Es,a-a [log (πo(ā s))]
Besides, to enable adaptive scheduling of the learning rate, also present an unbiased estimation of the KL divergence as shown Algorithm| divergence between diffusion policies is
[ and the estimation formula for computing the KL
KL.
T
E, B [log (Noww | [s)
log (#o [))]]
(10)
We also present an instantiation of our algorithm within the PPO framework. The surrogate loss of PPO is modified (1..
min T 8x) Ac, clip( To(ā; 88) 1,
CPP (0) : (s
To.ia (āe |8))
To.ia (āg (88)
(11)
Notably, to mitigate this issue and prevent ineffective exploration of the action space, we introduce a compression loss as Ex, ~ (x1 )1 which diminishes the mean square error between a and y components, in the policy loss. The final diffusion policy loss is shown in 12
(()) = PPO + XCENT VE mate (x] - v1)²}
(12)
where À and v are the coefficients of the entropy and compression loss, respectively.
5 Experiments
In this section, we empirically evaluate the proposed method on several control tasks within IsaacLab. We also conduct ablation studies investigate the impact of key hyperparameters. These experiments aim to compare our approach with popular online RI. algorithms and analyze how hyperparameter choices affect performance.
5.1 Comparative Evaluation
Setup. evaluate our method, we conducted experiments on a suite of benchmark tasks provided by IsaacLab. The benchmark environments used in this study include Isaac-Ant-vO, Isaac-Humanoid-viee Isaac Velie-VVlocttyyPeee Flat-Anytaaaal -D Rough-H1-vO, and Isaac-Que Quadcopte-- Direct-vO. For brevity, we refer to these environments respectively as Ant, Humanoid, Franka Arm, Shadow Hand, Anymal-D, Unitree-Go2. Unitree-H1. and Quadcopter throughout the remainder of the paper. Our algorithm GenPO is compared and evaluated against several well-known model-free algorithms, including DDPG, TD3, SAC, PPO, well two diffusion-ber DACER and QVPO. To assess statistical robustness, all experiments are repeated across five random seeds. More details related to the experiments be found in the supplementary materials.
lsaac-Lift-Cube- Mane-LiftCCube-FFraak--000
Setac-Repose-Cule See--Shado--Direee
v0, Isaac-Velocity Seth--nnnttte----n Go2-vO, Isaac-
off- off-policy algorithms:
Figure presents the mean episodic return (solid line) and one standard deviation (shaded area). GenPO and PPO achieve near-optimal performance with fewer environment interactions compared to

---

Algorithm Generative Diffusion Policy Optimization
Input: generative diffusion policy Te(a s), value network (s).
1: 0' 6-0
2: for in ... do
3: for each actor do
4: Sample dummy actions ā (z, v) from (a s)
5: Use 2g to interact with the environment for N timesteps
6: Calculate the probability Te(ā 8) the executed dummy actions
7: Use GAE to estimate advantages ,,,..., AA and returns R..... Rx
8:
Store transitions, dummy actions, probability, and advantage value in rollout
9:
end for for in 1,2, K do Estimate KL divergence according to (TO) and adjust the learning rate a
11:
DEL 26
a +
2a,
2
DKL SC.
12:
Update Tā s) with the surrogate loss
12) with and mini-batches from rollout
[| A(s, a)
27
13:
Update the value network (s) with loss IE Eg
14:
end for
15:
end for
(4)
Gueelonote
Eproudt Rarvird
Epecec
Edceh
Eaoch
Epech
Epech
12200
Ediosok Rovur
Ediosle Roware
(202h
Epoch
(pacn
CVPO
PPP
Figure 3: Learning curves across 8 IsaacLab benchmarks. Results are averaged over 5 runs. The x-axis denotes training epochs, and the y-axis shows average episodic return with one standard deviation shaded.
off-policy baselines, benefiting from synchronized data collection that enhances training stability and convergence speed. In contrast, off- off-policy methods struggle in large-scale parallel settings. This issue particularly pronounced in diffusion- based algorithms, whose policies have difficulty tracking the rapidly shifting data distribution in the replay buffer. Additionally, GenPO's generative diffusion policy enhances exploration over standard Gaussian policies, contributing to more efficient and robust policy optimization. These factors together explain GenPO's superior performance across diverse tasks.
Table I reports the average episodic return (with standard deviation) across eight benchmark tasks. highest mean return across all environments, significantly outper- forming all algorithms. Notably, on the most challenging tasks, such as Unitree robots locomotion
GenPO consistently achieves the

---

Table 1: Comparison of the average final rewards of GenPO with some prevalent RL methods in LsbacLab Benchmarks. The maximum value for each task is shown in bold. The standard deviation of the five runs is enclosed in parentheses.
Algorithm
And
Humanoid
Franka-Arm
Quadkopter
Anymal-D
Unirre-HI Unitre--Go2
Shadow-Huun Hond
DDPG
9665, 18)
63.34(7.39)
-1.61(0.76)
0.03(0.13)
-0.86(0
9.09(0.81)
-6.63(1.00 .04)
6209 (209559)599
TD3
17
67,80/7.05)
82.36(4.70)
0.03(0.12)
0.17(0.15)
-0.89(0.77)
8.44(1.05)
-4,97(0.87)
9386.644314.177
SAC
22
38.68/21.84) 1,84)
49.52(16.89)
-1.10(0.95)
38.46(22.2) 20)
-5.12:0.14)
6.67(2.02)
-5.05(0.18)
8459.74(335.3)
DACER
59
0.29/0.96) 98)
0,58(0.01)
-3,43(2.82)
25/32222,99 99)
-1.6200.09)
6.37(2.07)
-6.2100.77)
-2077.43888 82)
QVPO
7.19(2.32)
10.59(6.30) 30)
-0.12(0.14)
63.99(45.44 42)
-1.81(0336)
7.34(0.62)
...2(0.16)
134. 144.36(39) (9.065
(6694(10.61)
197.25(1826)
78.14:30,43)
99.08(13.49)
9,80(4.78)
15.67(1.92)
18.9712.02)
8402.21(435.66
GenPO(
177.90(13.87)
273.94((6..6)
144.78(3.10)
137.95(0.84)
19.80(0.16)
28.01(0.76)
26.09(0.68)
11282.35(332.99)
lation for compression loss
(b) Ablation for entropy and learning ratn adaptation.
[c] Ablation for mixin coefficiett
200
200
200
160
150
Epiaode Rowoen
128
12
Epocod Reward
25
22
85
Edtapy/Reed
Ertosy/Fiad V)
1000
1290
11:06
29101
2900
30000
1966
2913
3930
0. 6966
18966
1/091
3000
2000
20930
Egoch
Ezoch
Epoch
Figure 4: Ablation study results. (a) Effect of varying the compression loss coefficient v on training stability and final performance. (b) Impact of entropy and learning rate adaptation exploration and convergence. (c) Performance under different mixing coefficients p in flow policies.
and Shadow Hand manipulation, GenPO demonstrates not only higher returns but also lower variance, indicating greater training stability.
5.2 Ablation Study
To better understand the significance of each component in GenPO and explain its superior perfor- mance the IsaacLab benchmark, conducted the ablation study focusing three key design choices: 1) the effect of the compression loss in the training objective; 2) the impact of entropy and KL divergence estimation; and 3) the choice of the mixing coefficient p. We present results on the Istac-Ant-vO task as a representative example, as similar trends are observed across other environments. Additional ablation results provided in the supplementary materials.
Effect of Compression Loss. To validate the impact of the compression loss on final performance. we vary its coefficient in (12) and analyze the resulting training curves in Figure 4 a).. We evaluate v E {0, 0.01, 0.1, 0.5, 1.0) and find that y = 0.01 provides the best balance between stability and regularization. Larger values overly constrain the policy and lead to poor performance, while
excluding the loss (v 0) slows convergence. Based on this tradeoff, we set v = 0.01 in all experiments.
Entropy and Learning rate adaptation. Figure illustrates the effect of entropy regularization and learning rate adaptation. Adaptive adjustment of the learning rate leads to faster and more stable convergence, while entropy regularization enhances exploration and yields higher returns. Empirical results confirm that both components, implemented described in Section 4.3 substantially improve overall performance. In all tasks, we employ an adaptive learning rate scheme and include an entropy regularization term in the loss defined in Eq. (12)
Mixing coefficient. As shown in Figure (c.. GenPO performs best when the mixing coefficient is set to = 0.9. This is because setting too low will cause the forward mixing process (8) to be numerically unstable, making the value too large and affecting the stability of training. On the contrary, too high p will introduce redundant exploration in the doubled dummy action space, thus slowing down the convergence. Therefore, choosing p = 0.9 can achieve a practical trade-off between stability and exploration efficiency, so we set p to 0.9 for all experiments.

---

6 Conclusion, Limitation and Future Works
This paper proposes Generative Diffusion Policy Optimization (GenPO), a novel approach that integrates generative diffusion policies into the on-policy RI framework. GenPO enables tractable oogliikeihood computation of actions in the diffusion model, which unavailable in prior works, and further unbiasedly estimates the entropy and KL divergence of the diffusion policy, thereby achieving entropy regularization and adaptive learning rate adjustment during policy update. Finally, GenPO achieves superior performance compared with existing RL baselines, including diffusion-based RL methods, on eight IsaacLab benchmarks, with comparable sample efficiency and faster convergence, highlighting the potential of generative diffusion policies in on on-policy RL for large-scale simulation and real-world robotics control.
Despite its excellent performance, GenPO faces the problem of relatively high computational and memory overhead to be resolved the future. While the GPU's parallelism helps to mitigate this issue, GenPO still incurs considerable complexity in computing the Jacobian determinant. Hence, our future work will focus on exploring how to optimize the computational and memory efficiency of GenPO to facilitate its deployment in more real- world applications.
Acknowledgement
This work was supported by National Natural Science Foundation of China (62303319, 62406195), Shanghai Local College Capacity Building Program (23010503 100). ShanghaiTech A14S Initiative SHTAI4S202404, HPC Platform of ShanghaiTech University, and MoE Key Laboratory of Intelligent Perception and Human-Machin Collaboration ShanghaiiTec University), Shanghai Engineering Research Center of Intelligent Vision and Imaging. This work was also supported in part by computational resources provided by Feloud CO., LTD.
References
[1] Suzan Ece Ada, Erhan Oztop, and Emre Ugur. Diffusion policies for outtof--iitribuutin generalization in offline reinforcement learning. IEEE Robotics and Automation Letters, 9(4)33112A43123, April 2024.
[2] Anurag Ajay, Yilun Du Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657 2022.
[3] Joseph K Blitzstein and Jessica Hwang. Introduction to probability. Chapman and Hall/CRC, 2019.
[4] Janaka Brahmanage, Jiajing Ling, and Akshat Kumar. Flowpg: action-constrained policy gradient with normalizing flows. Advances in Neural Information Processing Systems, 36:201 18- 20132, 2023.
[5] Onur Celik, Zechu Li, Denis Blessing, Ge Li, Daniel Palanicek, Jan Peters, Georgia Chalvatzaki, and Gerhard Neumann. Dime: ceffusiio-bbscd maximum entropy reinforcement learning. arXiv preprint arXiv:2502.02336,, 2025.
[6] Chen-Hao Chao, Chien Feng, Wei-Fang Sun, Cheng- Lee, Simon See, and Chun-V Yi Lee. Maximum entropy reinforcement learning via energy- based normalizing flow. arXiv preprint arXiv:2405.13669, 2024.
[7] Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling, arXiv preprint arXiv:2209.144 14548, 2022.
[8] Yuhui Chen, Haoran Li, and Dongbin Zhao. Boosting continuous control with consistency policy, arXiv preprint arXiv:2310.06343, 2023.
[9] Yuhui Chen, Haoran Li, and Dongbin Zhao. Boosting continuous control with consistency policy. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, pages 335-344, 2024.
10

---

[10] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023.
[11] Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
[12]
Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil Bharath. Generative adversarial networks: An overview. IEEE signal processing magazine, 35(1):53-665 2018.
[13] Shutong Ding, Ke Hu, Zhenhao Zhang, Kan Ren, Weinan Zhang, Jingyi Yu. Jingya Wang, and Ye Shi. Diffusion based reinforcement learning via q-weighted variational policy optimization. arXiv preprint arXiv: /6/73, 2024.
[14] Shutong Ding, Yimiao Zhou, Ke Hu, Xi Yao, Junchi Yan, Xiaoying Tang, and Ye Shi. Diopt: Self-supervised diffusion for constrained optimization. arXiv preprint urXiv:2502. 10330, 2025.
[15] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv: 08803, 2016.
[16) Xiaoyi Dong, Jian Cheng, and Xi Sheryl Zhang. Maximum entropy reinforcement learning with diffusion policy. arXiv preprint arXiv:2502.11612 2025.
[17] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMIR, 2018.
[18] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pages 2062, 2019.
[19] Ruiqi Gao, Emiel Hoogeboom, Jonathan Heek, Valentin De Bortoli, Kevin Murphy, and Tim Salimans. Diffusion meets flow matching: Two sides of the same coin. 2024. URL https:/ddiffusionffow github. io, 2024.
[20] Grover, Dhar, and S Ermon. Flow-gan: combining maximum likelihood and adversarial learning in generative models. arxiv e-prints. arXiv preprint arXiv: /705. 08868, 2017.
[21] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[22] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprini arXiv: /e12.5992.0555 2018.
[23] Sami Haddadin. The franka emika robot: A standard platform in robotics research. IEEE Robotics & Automation Magazine, 2024.
[24] Philippe Hansen-Estruch. Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304. 10573, 2023.
[25] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in nearal information processing systems, 33:684006851, 2020.
[26] Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.0999) 2022.
[27] Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
(28) Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 11Xiv:1312.6114, 2013.

---

[29] Po-Chen Ko, Jiayuan Mao, Yilun Du, Shao-Hua Sun, and Joshua B. Tenenbaum. Learning to act from actionless videos through dense correspondences, 2023.
[30] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint urXiv:2005.01643 2020.
[31] Xiang Li, Varun Belagali, Jinghuan Shang, and Michael S Ryoo. Crossway Diffusion: Improving tiffusion-basets visuomotor policy via self-supervised learning. arXiv preprint vrXiv:2307.01844 2023.
[32] Zechu Li, Tao Chen, Zhang-War Wei Hong, Anurag Ajay, and Pulkit Agrawal. Parallel q-learning: Scaling off-policy reinforcement learning under massively parallel simulation. International Conference Machine Learning, pages 19440-19459, PMLR, 2023.
[33] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning, arXiv preprint arX.v::509.002.. 02971, 2015.
[34] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
[35] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference Learning Representations, 2023.
[36] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209 03003, 2022.
[37] Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. In International Conference on Machine Learning, pages 22825-22855. PMLR, 2023.
[38] Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, and Gavriel State. Isaac gym: High performance gpu-based physics simulation for robot learning, 2021.
[39] Bogdan Mazoure, Thang Doan, Audrey Durand, Joelle Pineau, and R Devon Hjelm. Leveraging exploration in off-policy algorithms via normalizing flows. In Conference on Robot Learning, pages 430-444, PMLR, 2020.
[40] Fei Ni, Jianye Hao, Yao Mu, Yifu Yuan, Yan Zheng, Bin Wang, and Zhixuan Liang, Metadiffuser: Diffusion model as conditional planner for offline meta-rl. In International Conference on Machine Learning, pages 26087-26100 PMLR, 2023.
[41] Seohong Park, Qiyang Li, and Sergey Levine. arXiv:2502.025386 2025.
Flow q-learning.
arXiv preprint
[42] Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, and Sam Devlin. Imitating human behaviour with diffusion models, 2023.
[43] Michael Psenka, Alejandro Escontrela, Pieter Abbeel, and Yi Ma. Learning diffusion model policy from rewards via q-score matching. arXiv preprint 12. 2023.
[44] Allen Z Ren, Justin Lidard, Lars Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffussion policy policy optimization. arXiv preprint arXiv:2409.000
2024.
[45] Moritz Reuss, Maximilian Li, Xiaogang Jia, and Rudolf Lioutikov. Goal-conditioned imitation learning using score-based diffusion policies. arXiv preprint arXiv:2304.02553, 2023.
[46] Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International conference machine learning, pages 1530-1538. PMLR, 2015,
12

---

[47] Quentin Rouxel, Andrea Ferrari, Serena Ivaldi, and Jean- Baptiste Mouret. Flow matching imitation learning for multi-support manipulation. In 2024 IEEE-RAS 23rd International Conference Humanoid Robots (Humanoids), pages 528-535, IEEE, 2024.
[48] Simo Särkkä and Amo Solin. Applied stochastic differential equations, volume 10. Cambridge University Press, 2019.
[49] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.
[50] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint 1xXxv:1506.02438. 2015.
[51] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal 2017.
policy optimization algorithms. arXiv preprint
[52] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500. 2020.
[53] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13355, 2020.
[54] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[55] Yunhao Tang and Shipra Agrawal. Boosting trust region policy optimization by normalizing flows policy. arXiv preprint arXiv: 10326, 2018.
[56] Paul Tuffield and Hugo Elias. The shadow robot mimics human actions. Industrial Robot: An International Journal, 30(1):56-60 60, 2003.
[57] Bram Wallace, Akash Gokul, and Nikhil Naik. Edict: Exact diffusion inversion via coupled transformations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22532 22541, 2023.
[58] Fangyikang Wang, Hubery Yin, Yue-Jiang Dong, Huminhao Zhu, Hanbin Zhao, Hui Qian, Chen Li, et al. Belm: Bidirectional explicit linear multi-step sampler for exact inversion in diffusion models, Advances in Neural Information Processing Systems, 11:46118-46155 2024.
[59] Yinuo Wang, Likun Wang, Yuxuan Jiang, Wenjun Zou, Tong Liu, Xujie Song. Wenxuan Wang, Liming Xiao, Jiang Wu, Jingliang Duan, et al. Diffusion actor-critic with entropy regulator. Advances in Neural Information Processing Systems, 37:54183-54204, 2024.
[60] Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2022.
[61] Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy optimization for offline reinforcement learning. Advances Neural Information Processing Systems, 35:31278-312991 2022.
[62] Long Yang, Zhixiong Huang, Fenghao Lei, Yucun Zhong, Yiming Yang, Cong Fang, Shiting Wen, Binbin Zhou, and Zhouchen Lin. Policy representation via diffusion probability model for reinforcement learning. arXiv preprint arXiv:2305.13322. 2023.
[63] Guoqiang Zhang, Jonathan P Lewis, and W Bastiaan Kleijn. Exact diffusion inversion via bidirectional integration approximation. In European Conference on Computer Vision, pages 19-36. Springer, 2024.
[64] Jing Zhang, Chi Zhang, Wenjia Wang, and Bingyi Jing. Constrained policy optimization with explicit behavior density for offline reinforcement learning. Advances in Neural Information Processing Systems, 36:5616-5630, 2023.
13

---

[65] Qinsheng Zhang and Yongxin Chen. Diffusion normalizing flow. Advances in neural information processing systems, 34: 14:16280-16291, 2021.
[66] Wenxuan Zhou, Sujay Bajracharya, and David Held. Plas: Latent action space for offline reinforcement learning. In Conference Robot Learning, pages 1719-1735, PMLR, 2021.
[67] Zhengbang Zhu, Minghuan Liu, Liyuan Mao, Bingyi Kang, Minkai Xu, Yong Yu, Stefano Ermon, and Weinan Zhang. Madiff: Offline multi-agent learning with diffusion models, 2023.
[68] Ziwen Zhuang, Shenzhe Yao, and Hang Zhao. Humanoid parkour learning. arXiv preprint arXiv:2406. 10759, 2024.
14

---

A Environmental Details
(a) Ant
(b) Humanold
(6) Franka Arm
(d) Quadiopter
(o) Anymal-D
(1) Untroe-Go2
Unitrop-H1
(h) Shadow Hand
Figure 5: Eight Isaaclab benchmark visualizations, eight images from Attps://isaac-sin
github. io/ www.a//aaa/aaaeeooorrererrereaaa
html
From (a) to (h) are
Isaac-Ant-v0, Isaac- Homlao--uummiid-vvd, Isaac-Lit-CCub--Fraake-d Isaac- Quadcopter-Drecct-00 Isaac- Velocity-Flt-AAnymal and Isaac- Reanc-RRppse-Cubbe Cube-Shadoww-aaaee Shadow-Direct-vO,
Isaac- Serriity-Rooh-Unne Unitree-Go2-vv. Isaac- Mabocity-Roug---i
IsaacLab is a unified and modular framework for robot learning, designed streamline common workflows in robotics research, including reinforcement learning, imitation learning, and motion planning. The framework leverages NVIDIA Isaac Sim for high-fidelity simulation and benefits from PhysX's GPU-accelerated physics engine well as a tile-based rendering API for vectorized rendering.
we selected representative and challenging environments, which can be roughly divided into the following categories according to the official manual of Isaaclab https 7/is 7/isaac- sim. gi githhb..... 1 wwwaaaaaaaaaainn/aaaaaaaaa http://maiiisuuuceommmm sour cw/ overview/ environments html
As displayed in Figure 5
1. Classie: These classic tasks derived from the IsaacGymEnv implementation of MuloCo- style environments, providing standardized benchmarks for locomotion and control in continuous action spaces.
Isaac-Ant-vO. The state and action spaces are (S. E R The task involves controlling a MuJoCo Ant robot to move in a specified direction.
Asaac-Humanod--vb. The state and action spaces are (S, 1287211 The objective is to control a MuJoCo Humanoid robot to move in a specified direction.
2. Manipulation: These environments involve manipulation tasks performed by fixed-base robotic arm or a dexterous hand, such as object reaching, grasping, or in-hand rotation.
The state and action spaces are (S,A) E 3636x8 The task involves controlling Franka Emika robot arm to pick up cube and transport to randomly sampled target position.
The state and action spaces are (S, 157xx2 The task requires using Shadow Dexterous Hand to perform in-hand reorientation of a cube to match a target orientation.
Ceee---eooosCubbe Seabb-Shado--Diretee
3. Locomotion: This category includes legged locomotion environments that challenge agents to coordinate multiple degrees of freedom to achieve stable and directed movement. Tasks include forward walking, turning, and terrain adaptation, typically implemented with hu- manoid robots.
Isaac- Isaac-Velocity- Flanty--l---A ---O. The state and action spaces are (S, RR 48x12 The task requires controlling an ANYmal D quadruped robot to follow a commanded velocity trajectory on flat terrain
15

---

The state and action spaces are (S, E 2235x 12 The task involves controlling a Unitree Go2 quadruped robot follow commanded velocity trajectory over rough terrain.
Isaac- WelooityyRoughh-nnne Unitrer--o---v.
The state and action spaces (S, 296 19, The task involves controlling a Unitree H1 humanoid robot to follow a commanded velocity trajectory rough terrain.
VVelocity-Rough---e
4. Others:
Isaac-Quass Saaeee-uuaoopte-Drreeeee The state and action spaces are (S, E 112x4, The task control quadrotors fly and hover at designated goal position by applying thrust commands.
Moreover, based the IsaacLab Engine, the anonymous link for the visualization of our experi- htt.ettt..........ii.... github.io
mental results is shown
B
Training setups
B.1 Hardware Configurations
All experiments were carried out on server equipped with two Intel Xeon Gold 6430 CPUs (32 cores per socket, 64 threads total per CPU, 128 threads total), with a base frequency of 2.1 GHz and maximum turbo frequency 3.4 GHz. The system supports 52-bit physical and 57-bit virtual addressing. For GPU acceleration, we used 8 NVIDIA GeForce RTX 4090 D GPUs, each with 24 GB of GDDR6X memory, connected via PCIe. The GPUs support CUDA 12.8 and were operating under the NVIDIA driver version 570.124.04. The machine is configured with NUMA topology across nodes, each handling 64 logical CPUs. No GPU MIG (Multi-Instance GPU) or ECC was enabled during the experiments.
B.2 Reinforcement Learning Framework in IsaacLab
IsaacLab provides native integration with several reinforcement learning libraries, in- cluding RSL-RL https://github. comm//eggeerrbboiicc//sss__ RL-Games https:// githubb.com/DenysBBe http://aaaaaaaaoooooo/oa/aaaaaaa http://hwwwooo.eee/n/eeeeeeeee wt/////aatss//////////a//sss//s/ and
http:///....oee/////////eeeeeeee html each which exposes a distinct API for agent-environment interaction. In this work, adopt SKRL as primary framework for implementing baseline algo- rithms. SKRL. an open-source, modular, and extensible library that provides stan- dardized implementations of widely used reinforcement learning methods. Specifi- cally, we utilize its implementations of PPO api/agentsppol html SAC
Bac. html TD3 https: //skrl http://hh...aaaaaaooo//////////a DDPG https: //pps://skrrl /krll readthedocs. http:///a/aaaeee/////a/aaaaaaaa/ evaluation. For algorithms such as QVPO and DACER, we design custom environment wrap- pers to adapt the existing interface without modifying the underlying simulation environments. Our proposed method, GenPO, is implemented using RSL-RL, a lightweight and high-performance framework tailored for robotics and continuous control, which emphasizes computational efficiency and ease of deployment.
html, and html for baseline
B.3 Hyperparameters
Tables and Table 10 summarize the hyperparameter configurations used across our experiments. For the baseline algorithms: PPO, SAC, TD3, and DDPG, we adopt the default hyperparameter settings provided by SKRL library. For our proposed method, GenPO, we align its hyperparameter configuration with that of PPO to ensure a fair and controlled comparison.
16

---

Table 2:
used in the Isaaclab-Ant-vO.
Heperparandeess
GenPO
PPO
QУPO
DACER
SAC
TD3
DDRG
hidden layens in network
[400, (00,2000100) 100)
(400.200,100)
[256,256, 256]
[256,256255
[400 100,
[400.200,,00)
[400.200, 200.00000
layens
[400,200,100]
[256.256,256]
[256.256,26)]
1400,200,100]
[400,200,100]
[400,200,100
Activation
mish
elu
mish
relu
mish
mish
mish
Batch size
4096
4096
4096
4096
4096
4096
4096
GAR
N/A
NIA
GAB Smoothing Parameter
0.95
0.955
NIA
N/A
NIA
N/A
N/A
Learning rate
10-3
10-3
10-4
5 10-4
5 10
10
Leaming
10"3
Actor Critic grad norm
1.0
1.01
1.
1.0
1O
1.0
1.0
Momory size
NIA
NIA
10F
XX 10
1 10F
X 10P
10R
Encropy coefficiens
Value loss confficient
1.0
1.0
LO
LO
1.0
Nosse clip
NIA
NIA
NLA
0.5
NEN
NIA
NIA
Surrogate clip
0.2
0.2
NIA
NIA
N/A
NIA
N/A
steps
Desired KL.
0.01
0.01
N/A
N/A
N/A
N/A
N/A
Table Hyper-parameters used the Isnaclab-Humanoid--d.
Hyperparameters
QVPO
hidden layes in octor network
[400,200, 100)
(400,200, 100)
[256, 256,6)
(256,266,26)
2000,200,100 100,
[400,200,,0)
2000,200,,10 100]
hisdenn layers critic oetwork
[400,200,
[400.200,
[256,256,256]
[256,256.25)
[-00,200) 100]
[400,200, 100]
1-00,2000100]
Batch size
1096
1096
1096
1096
1096
4096
Use GAB
Tiue
Truc
N/A
N/A
N/A
N/A
N/A
Discount for roward
0.99
0.99
0.99
0,99
0.99
0.99
Smoothing Parameter
Learning rate for actor
5 10-
x 10
3 10
X-X 10
5 X 10
x 10
10
Leaminng eritic
5 100
x 10
3 10
3 10
5 10
5 X 10
10
Actor grad
Memory size
NIA
N/A
X 10F
X T0F
1 X 1(F
X TOP
X 10"
Entropy coefficicnt
0.01
NAA
N/A
N/A
NIA
N/A
Value cocllicient
1.0
Noise clip
NIA
NIA
N/A
NIA
NIA
N/A
Surrogati clip
0.2
0.2
NAA
N/A
N/A
N/A
N/A
Diffusion steps
5
NIA
20
20
N/A
NIA
N/A
Table 4: Hyper-parametenn used in the Lsaac-Lif--Cube-Fraan
Hyperparameters
GenPO
PPO
QVPO
DACER
SAC
TD3
DDPG
layers actor network
(2866128644
1256,128
(256,256,250
2266,266,26)
[256, 128, 64]
[256, 128, 64]
[256, 128, 64]
hidden layas in critic network
[256.128.64]
[256,128.64]
[256,266,256
2256,256,256
[256, 128, 61]
[256, 128, 64]
[256. 128, 64]
Activation
mish
mish
relu
mish
miss
mish
4096
Use GAE
True
True
N/A
NIA
NIA
NIA
NIA
Discount for reward
0.98
0.98
(1.9
0.99
0.99
0.99
0.99
Smoothing Parameter
Learning rate for actor
x 10
1 10
x 10-1
5 10
10-1
1 10
x 10
Leanning for criric
10
10
10-1
10
10
10
10
grad norm
Memory size
N/A
N/A
X 1O
1x 10"
X 10"
X 10"
X 10P
Entropy crefficient
0.006
0.005
N/A
N/A
NIA
N/A
Valuee cocfficient
1.0
1.0
1.0
Noise clip
N/A
N/A
N/A
0.5
NIA
NIA
NIA
Surrogale clip
0.2
0.2
N/A
N/A
NIA
N/A
N/A
Diffusion steps
NIA
Desired Kl.
0,11
0.01
N/A
N/A
NIA
N/A
N/A
17

---

Table 5: Hyper-parameters used in the Isaac- Salia---uuddopee--iieee---
Hyperparameters
GenPO
PPO
QVPO
DACER
SAC
TD3
DDPG
hidden layers in actor network
[64.64]
[64.64]
2556
(256,256.256]
[64, 64]
[64, 64]
[64, 64]
hidklen layers critic network
[64,64]
[64,64]
[256,256,256]
2256,256,256]
[64, 64]
[64, 64]
164, 64)
Activation
mish
mish
mish
Batch size
4096
4096
4096
4096
4096
4096
4096
Use GAE
True
True
N/A
N/A
N/A
N/A
N/A
reward y
0,99
0,99
0,99
GAE Smoothing Parameter A
0.95
0.95
N/A
N/A
N/A
N/A
N/A
Learning rate for actor
5 x 10
10
3 X 10
X 10
5 10
5 10
5 X 10
Learning rate for critic
5 10-
5 10
3 10
X 10
5 10
X 10
5 10
Actor Critic grad
Memory size
N/A
N/A
X 10
X 10
X 10
10
10
Entropy coefficient
0.01
0.01
NIA
N/A
N/A
NA
NIA
Value loss coefficient
Noise clip
N/A
N/A
N/A
D.5
N/A
N/A
N/A
Surrogate clip
0.2
0.2
N/A
N/A
N/A
N/
NIA
Diffusion steps
N/A
20
N/A
N/A
Desired KI.
0.01
0.01
N/A
N/A
N/A
N/
N/A
Table 6: Hyper-parameters used in the Isaac- Velocctt-FFat--Anmmaaaaa-
Hyperparamettes
GenPO
PPO
QVPO
DACER
SAC
TD3
DDPG
hidden layers network
[228
(128.128.128]
066,256,25
(256,256.26)
[128, 128. 128)
[128. 128, 128)
[128. 128)
hidden layers critic oetwork
[128,128,128]
[128.128.15
[256,256.266
[256,256,266]
[128, 128, 128]
11288 128, 128)
[128, 128. 128]
Activarion
mish
ehi
mish
reln
mish
mish
mish
Barch size
4096
4096
4096
4096
4096 N/A
4095
4096
Use GAE
True
True
N/A
NIA
NIA
NIA
reward y
GAE Smoothing Parameter A
N/A
NIA
N/A
NIA
N/A
Learning rate for actoe
1x 10
x 10
3x 10
5 X 10
L 10
1 10
10
carning for critic
10
10
3 10
10
10
10
Critic grad
Memory size
NIA
NIA
X
1 10
1 10P
1OP
Entropy cnefftcient
0,005
0,005
N/A
NIA
N/A
NIA
N/A
Value confficient
1.0
1.0
1.0
LO
Noise
NIA
NIA
Sumugatc clip
0.2
0.2
NIA
NIA
NIA
NIA
N/A
DilTusion steps
N/A
20
20
N/A
NIA
N/A
Desited KL.
0.01
0.01
N/A
N/A
N/A
N/A
N/A
Table 7: parameesrs used the Isaac- Sowpeitty-RRuu--niiitee
Elpeeparameters
hidden layers actor network
1562,256, (28)
[5122 256, 128]
(256.256.25) (256,256,26)
(25.226.226)
[512, 256, 128]
[512, 256, 128)
[512, 256, 128]
hidden layers in critie metwork.
[512, 256, 128]
[512, 256, 128]
(256.256.25)
[512, 256, 128]
[512, 256, 128]
[512, 256, 128] mish
Activation
mish
clu
mish
relu
mish
mish
Batch
4096
4095
1096
4096
4095
4096
4096
Discount reward y
0.99
0.99
0.99
0.99
GAF Smooting Parameter
0.95
0.95
N/A
N/A
N/A
N/A
N/A
Learning rate for octor
1 10
10
3 x 10
5 10
10
x 10
X 10
Learning for
10-3
10-3
10-4
10
10-3
Actor Critic grad noem
Memory size
NIA
N/A
x 10
1 10"
1 10P
X 10F
1 10
Entropy coefficient
0.01
0.11
N/A
NIA
NAA
NIA
NIA
Value cocfficient
LO
LO
L.0
Noise clip
N/A
N/A
N/A
Surrogate clip
Diffusion steps
5
N/A
N/A
NIA
NIA
Desined KL
0.01
0.1
N/A
N/A
NA
N/A
N/A
18

---

Table 8:
used in the Isaac- Velocity-Rough-e
Hyperparmmeters
GenPO
PPO
QVPO
DACER
SAC
TD3
DDPG
hidden layers actor network
[512, 256, 128]
[5122 256, 128)
(256,256,25) (256,256,26)
(256,260256)
[512, 256, 128
[512, 256, 128]
[512. 256, 128]
hidden lyyrs critic network
[512, 256, 128]
[512, 256, 128)
(256.256256)
[512, 256, 128]
[512. 256, 128]
[512, 256, 128)
Activationn
mith
mish
Ratch
<099
4096
4196
4096
Use GAE
True
Trme
N/A
NIA
NIA
NIA
N/A
Discount for reward y
0.99
0.99
0.99
0.99
0.99
0.99
0.99
GAF Smoothing Parameter
0.95
0.95
NIA
N/A
N/A
NIA
N/A
Leurning for
10-3
10
10
10
10
10"
10
Learning rate for crttcc
10-4
3
X
Actor Ciitic grad noem
1.0
1.0
1.0
1.0
1.0
1.
L.D
Memory
NIA
N/A
10
1x 10
1 10P
X 10"
1 10
Entropy coefficient
0.01
0.01
N/A
N/A
NIA
NIA
N/A
Value coefficient
Noise clip
NIA
NIA
N/A
0.5
NIA
NIA
NIA
Surrogate clip
0.2
0.2
N/A
N/A
N/A
NIA
N/A
Diffusion steps
5
N/A
20
20
N/A
NIA
NIA
Desinel KL.
0,01
0.01
N/A
N/A
N/A
NIA
N/A
Table 9:
used in the Isaac- Suw.rse-ouub-SSSh----D---
Ryperpunaneerrs
GeaPO
QNPO
DACER
SAC
TD3
DDPG
biddes layes
[512. 512, 1281
1256,256.250
(256,256,26)
512, 125)
512.
bilser leyers oritic network
[512) 512, 355, 172)
$12, 256, 176]
[256 (16,,256,55
[156,756,25 156)
[512. 517, 256, 175)
512, 256, 126)
[512, 517, 356 1788
Activation
clu
mi5h
ro3a
22143
mihh
88153
Ratch vine
4036
4096
4836
4096
4006
40066
4006
Use GAE
True
True
NIN
NIA
NBA
NIA
N/A
Discount for reward
0.55
0.59
0.99
1199
0,5N
11.99
0.59
Smoothing Panancter
0.95
0.55
NIA
NIA
NIA
Leaming actoe
N/A
NIA
100
10
10
10
Extroyy coefficiezt
NIA
NIA
NGA
NAA
Value lows sncllicient
1.0
1.0
1.0
1.0
3.0
1.0
1.0
Noise clip
MAA
NIA
NIA
0.5
SIA
NIA
MAA
Surmaate clip
0.2
0.3
N/A
NIA
NAA
NIA
NIA
Extlasion slope
NIA
20
20
NEA
NIA
NIA
Destral Ki.
0,015
0.066
N/A
NIA
S/A
NIA
NAA
Table 10: Hyper-parameters used in GenPO.
Hyperpurameter
Ant
Humanoid
Franka Arm
Quadcopperr
Anymal-I>
Unitee-GinZ
Unitrec-HI
Shadow Hand
Compress coefficient
Time embbokding dimcuusio
16
32
32
32
32
Hidden layers in time embedding Mixing coefficient
(256,256]
[256,256]
[256,256]
[128,128]
[256256
[256,256]
[256.256]
[256,256]
0.9
0.9
0.9
C
Details of Model Architecture
C.1 Sinusoidal Positional Embedding
During training, instead of directly concatenating the time step I,, state 8t, and action ar as raw input features, we incorporate sinusoidal positional embeddings to encode the temporal component. This approach, inspired by the positional encoding technique commonly used in denoising diffusion probabilistic models (DDPMs), maps each timestep t E (0, 10.., T to a fixed-dimensional vector e, e IRd as follows:
e(22)) sin 1000022/4
(2++1) cos
t
for i 0, 1,.......
d - 1. (13)
100002:74)
The resulting embedding e, is then passed through a multi-layer perceptron (MLP) to allow for non- linear transformation and projection into the model's feature space. The MLP output is subsequently concatenated with the state 8t and action o to form the final input representation:
2 = concat(s;, st, ae, MLP(s.)),
(14)
which enriches the model input with smooth, continuous representation of temporal progression. Unlike raw scalar timestep concatenation, sinusoidal embeddings provide a structured encoding that helps the model infer temporal relationships and ordering, even the absence of explicit recurrence or attention mechanisms.
19

---

C.2 Actor and Critic Model Architecture
the Isaac-Ant-vi environment, the actor network, corresponding to the flow policy, is implemented multi-layer perceptron (MLP) with hidden layer dimensions [400,200, 100]. The activation function used Mish. The imput to the actor consists of the concatenation the current state, action, and sinusoidal time embedding vectors, resulting in imput dimension of [A|+|ST (T), The output dimension is equal to the action dimension IAI, representing the flow-based action refinement. The critic network is also implemented Mish activation. It receives only the current state as input and outputs a scalar value corresponding to the estimated state value.
MLP with the same hidden layer structure 440,2000,000 and
D Additional Experiments
D.1 Effect of Different Parallel Environments Size
To assess the effect of parallelization on the learning efficiency of GenPO, we conducted an experiment varying the number of parallel environments. We trained GenPO agents using 512, 1024, 2048, 4096, and 8192 parallel environments, respectively. All other hyperparameters were kept consistent across these runs.
As illustrated in Figure 6 increasing the number of parallel environments generally leads improved learning performance. Agents trained with a larger number of environments exhibit faster convergence and achieve higher final rewards compared to those trained with fewer environments. However, the performance gain from using 8 8192 environments over 4096 is marginal. To balance performance and computational efficiency, we fix the number of parallel environments to 4096 in all experiments.
200
512
1024
2048
4096
150
Episodic Reward
125
25
5000
10000
15000
20000
26000
30000
Epoch
Figure 6: Learning curves for the Isaaclab-Ant-vi benchmark with different numbers of parallel environments. Results are averaged over runs. The x-axis denotes training epochs, and the y-axis shows average episodic return with one standard deviation shaded,
D.2 Effect of Different Flow Policy Steps
We investigate the impact of the number of flow steps, denoted by T, on learning performance. To this end, we train agents using different values of T E {1, 2, 5, 10, 20), while keeping all other peerparrameeess fixed. As shown in Figure 7 the number of flow steps influences convergence
20

---

speed and performance stability. Notably, as long as T is not too small (e.g., T 2 2), the overall performance remains robust.
200
175
20
160
Episodic Reward
100
5000
25000
Epoch
Figure 7: Learning curves for different flow policy time steps on the Bhaclab-Ant-vl benchmark. Results are averaged over runs. The x-axis denotes training epochs, and the y-axis shows average episodic return with one standard deviation shaded.
D.3 Effect of Different Operations on Dummy Action Space
We study the impact of recovering single action from the dummy action à = (ax, y) by interpolating between the two partial components z, y. Specifically, we construct the real action as az - (1- a) and {0.0, 0.3, 0.5, 0.7, 1.0} respectively. As illustrated in Figure g the best performance is achieved when a 0.5. This observation highlights the benefit of equally leveraging both components of the dummy action. When a 0.0 or a 1.0, only one side of the dummy action space is utilized, leading to limited exploration and suboptimal performance. In contrast, interpolating between both components enhances exploration diversity and improves sample efficiency, which crucial for effective policy optimization in high-dimensional multimodal action spaces.
D.4 Effect of Time Steps Embedding
We investigate the effect of sinusoidal time embeddings on the performance of the flow policy. As shown in Figure incorporating sinusoidal embeddings significantly improves the policy's 's ability to utilize temporal information across different stages of the diffusion process. The structured, high-dimensional representation provided by the sinusoidal encoding facilitates better discrimination of time steps, which turn enhances the model's capacity learn effective denoising functions. In contrast, using simple concatenation of the scalar timestep with the state and action fails to provide sufficient temporal structure, resulting degraded performance. Based on these observations, we adopt sinusoidal positional embeddings to encode time steps in all subsequent experiments.
D.5 Training and Inference Time
To validate computational efficiency, we also conduct comparison experiments on the proposed GenPO with 6 other baselines, which respectively evaluate the inference, training and parallel training time on single/multiple GPUs, as shown in Figure TO, Figure TJ and Figure [2] can be observed that, while the GenPO's inference time is slightly longer than that of Gaussian policies due to multiple
21

---

200
0.0
0
175
0y
0.y
1,0"x+0.0'y
Episodie Reward
125
100
25
5000
10000
15000
20000
25000
30000
Epoch
Figure 8: Learning curves for different operations on dummy action space on the Ssaacla-Ann-w benchmark. Results are averaged over 5 runs. The x-axis denotes training epochs, and the y-axis shows average episodic return with one standard deviation shaded.
denoising iterations of diffusion, it remains significantly more efficient than previous diffusion- based RL algorithms such as QVPO and DACER. Besides, to our knowledge, inference time of 2.577ms is more than sufficient for the real-time decision in robot control. Furthermore, in terms of training time, although the performance of GenPO on a single GPU is less than ideal (yet still superior to previous diffusion-brit basedd RL methods), can effectively leverage multi-GR parallelism to significantly accelcrate training. In contrast, the traditional RL algorithm PPO cannot obviously benefit from that.
22

---

200
time _ep_emmbediing
wo. time no__time_stp__eeee _ap__mmbedinng
175
Episodie Reward
125
100
5000
10000
15000
20000
25000
30000
Epoch
Figure 9: Learning curves on the Isaaclab-Ante benchmark with or without time step sinusoidal positional embedding. Results are averaged over 5 runs. The x-axis denotes training epochs, and the y-axis shows average episodic return with one standard deviation shaded.
E Proof for Lemma 1
Lemma 1. (Change of Variables (3) Let : - TR is an invertible and smooth mapping. If we have the random variable X ~ q(x) and the random variable Y = S(X) transformed by function f. the distribution ply) of Y
05 dr
p(u) = ((x)) det
(15)
Proof. Here, we take the deeddmmeniional random variables X, Y as an example. For a general n-dimensional case,
obtain the same result by applying multivariable calculus.
Firstly, we can divide f into two categories: (1) f is strictly increasing, (2) f is strictly decreasing. since f is an invertible and smooth mapping.
Strictly increasing. For strictly increasing the CDF of Y is
P(y) P(Y < v) P(((X) 5) P P = Q(x).
Then, according to p(x) = lim P(xte)-PLC P(z) and the chain rule, the PDF of the random variable Y is -0
dx q(x) det
0ʃ dr
ply) q(z) q(z) dy
Strietly decreasing, The proof for f strictly decreasing is analogous. In that case, the PDF of Y is p(y) -q(x)) de dy
since de < O. Hence, we need to add an absolute sign for the final formula, i.c., dy
p(y) q(x)) det
01 2=
23

---

Gaussian Policy Diffusion Policy Flow Policy
4.516
4.029
2.577
10
Inference Time (ms)
0.104
10-4
0.032
0.032
0.003
SAC
TD3
DDPG
PPO Algorithm
QVPO
DACER
GenPO
Figure 10: Inference time comparison on Issac-Ant-v0 environment.
Gaussian Policy
5.091
Diffusion Policy
Flaw Palicy
4,399
Training Time (h)
0.143
0.133
SAC
TD3
DDRG
PPO18 GPU)
QVPa
DACER
CenPOja GPU)
Algorithm
Figure 11: Training time comparison on lsaac-Ant-vi environment.
24

---

2.5
2.429
Training Time (h)
1.5
1.300
1.0
0.5
0.122
0.113
0.106
0.0
GenPO44
Genioo
Algorithm
Figure 12: Comparison of training time between PPO and GenPO on the Ssaac-And-y environment for scaling training on multiple GPUs.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: As stated the abstract and introduction, this paper bridges the gap between on-policy reinforcement learning and generative diffusion models, makes it possible to calculate the log-likes likelihoohood diffusion policies, and finally enables large-scale parallel simulation enviromments to leverage the powerful multimodal capabilities of diffusion models. They can accurately reflect the contributions and scope of this paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The analysis of limitations provided in Section 6
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
26

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
The proofs are provided in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: These details are provided in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
27

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
We will release the code once the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, understand that this might not be possible, so âÄIJNoâĀī is acceptable answer. Papers cannot be rejected simply for not including code, unless this central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.co ccc wtpppp/uguuiiapeeeeepppppppppbpd
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: These details are provided in Section 5 and supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The standard deviation in our experiment results (Figure Figure and Table M shows the statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
28

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The information on the computer resources is provided in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttps://neurpp::: wwwpuripssccppuuliii.e htt.:i///ehhcisuiiellleeeeee
Answer: [Yes]
Justification:
Our research conformed with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our paper primarily focuses on theoretical research in how to employ the diffusion model in on-policy reinforcement learning, with no consideration of societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
29

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our experiment of this paper was conducted on 8 robot control tasks in IsaacLab, which poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The implementation of SAC, TD3, DDPG, PPO, DACER, and QVPO, and IsaacLab simulator are cited properly in the supplementary materials.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
30

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
31

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We merely use the LLM for editing the paper.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy fattps://neune http://errriiccooooooeeeeee///// cc//oonferences/2225//LL. for what should or should be described.
32


Reviews:
[{"review_id": "FYuBAmbBvK", "reviewer": "Reviewer_rVSg", "summary": "The paper introduces GenPO, an on-policy reinforcement-learning (RL) algorithm that replaces the policy network with a conditional diffusion generator. GenPO tackles the non-invertibility of diffusion policies by doubling the action space with a “dummy” copy and applying alternating updates that make the forward and reverse processes analytically tractable. Two additional regularisers: (i) a compression loss that discourages redundant dummy actions and (ii) an entropy bonus that preserves exploration, stabilise training. Experiments on eight IsaacLab robotic benchmarks show that GenPO consistently outperforms standard model-free on-policy baselines and two diffusion-based off-policy methods.", "questions": "1. Which specific Franka-arm tasks were used? If any involve multimodal action distributions (e.g., pick-and-place with multiple grasp configurations), please highlight the corresponding performance curves. Demonstrating gains in explicitly multimodal settings would strengthen the argument that diffusion models excel where unimodal Gaussian policies struggle.", "limitations": "The authors adequately addressed the limitations of their work.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "### Strengths:\n- The paper is well written and clear, with a lot of details on the method, and ablations on its components.\n- The resulting method is able to outperform baselines on average on the 8 IsaacLab environments.\n\n\n### Weaknesses:\n- GenPO is a computationally intensive method.", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the Reviewer rVSg for the detailed and constructive comments. In the following, we have provided an item-by-item response to the comments.\n\n>**Q1**: GenPO is a computationally intensive method.\n\n**A1**：Thank you for raising this concern. We need to clarify that training time is acceptable under multi-GPU settings (see **Figure 11–12 in Appendix**), and inference time is about 2 ms (see **Figure 10 in Appendix**), which is fast enough to meet the real-time need of real-world robotics tasks [R1, R2].\n\n> **Q2(1)**: Which specific Franka-arm tasks were used? If any involve multimodal action distributions (e.g., pick-and-place with multiple grasp configurations), please highlight the corresponding performance curves. \n\n**A2(1)**：Thanks for your question. In **Line 262**, we have specified the exact environment as Isaac-Lift-Cube-Franka-v0, which is one of the most challenging and representative Franka arm tasks in IsaacLab involving picking up a cube and placing it at a randomly sampled target position.\n\n> **Q2(2)**: Demonstrating gains in explicitly multimodal settings would strengthen the argument that diffusion models excel where unimodal Gaussian policies struggle.\n\n\n**A2(2)**: Thanks for your suggestion. We need to clarify that the multimodality of the **final converged policy** is not the focus of our method. In online RL, the multimodality of the diffusion model is leveraged to achieve **better exploration capability** rather than for the final multimodality of the converged policy in [R14]. \n\nWith better exploration capability, GenPO can prevent the policy from converging to a local optimum. This remains meaningful even when the oracle optimal policy is actually unimodal. This advantage can be confirmed by the superior performance of GenPO compared with that of PPO (with Gaussian policy) across IsaacLab Benchmarks in **Figure 3**.\n\n\n\n[R1] He, T., Luo, Z., Xiao, W., Zhang, C., Kitani, K., Liu, C., & Shi, G. (2024, October). Learning human-to-humanoid real-time whole-body teleoperation. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 8944-8951). IEEE.\n\n[R2] Xue, Y., Dong, W., Liuˆ, M., Zhang, W., & Pang, J. (2025). A Unified and General Humanoid Whole-Body Controller for Versatile Locomotion. arXiv preprint arXiv:2502.03206.\n\n[R14] Chi, C., Xu, Z., Feng, S., Cousineau, E., Du, Y., Burchfiel, B., ... & Song, S. (2023). Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, 02783649241273668."}, {"review_id": "la2ly1Iwt4", "reviewer": "Reviewer_kGqW", "summary": "This paper presents GenPO, a framework integrating generative diffusion models into on-policy reinforcement learning (RL). Existing diffusion-based policies have focused on offline and off-policy RL, leaving a gap in on-policy integration. GenPO addresses the challenge of computing state-action log-likelihoods in diffusion policies by introducing a doubled dummy action mechanism for invertible mappings, enabling exact likelihood calculation, unbiased entropy, and KL divergence estimation. Experiments on 8 IsaacLab benchmarks show GenPO outperforms baselines, demonstrating an effective integration of diffusion policies into on-policy RL.", "questions": "1. Clarify the questions mentioned in the above weakness.\n\n2. Evaluation of the effectiveness of adaptive learning rate $\\alpha$ mechanism is missing. How is it compared with constant learning rate?\n\nMinor problem:\n(1) Notation ambiguity in Eq. (7) and Eq. (8) : The coefficient $p$ used in the mixing scheme may be misinterpreted as a probability distribution which is also denoted as $p$ . It is recommended to replace $p$ with an alternative notation to avoid confusion.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength: \n\n1. GenPO is the first framework to combine generative diffusion models with on-policy RL (e.g., PPO), addressing the gap in leveraging diffusion’s exploration and multimodality for large-scale parallel training in simulators like IsaacLab. By introducing a doubled dummy action mechanism and exact diffusion inversion proposed by EDICT, GenPO overcomes the intractability of computing state-action likelihoods in diffusion policies.\n\n2. Superior empirical performance across diverse tasks: Experiments on 8 IsaacLab benchmarks (e.g., legged locomotion, dexterous manipulation) show GenPO outperforms traditional RL baselines (PPO, SAC) and diffusion-based methods. The evaluation is comprehensive.\n\nWeakness:\n\n1. The presentation of this paper needs improvement:\n\n   (1) The Preliminaries section lacks a brief introduction to EDICT, which hinders the reader’s ability to contextualize the proposed exact diffusion inversion mechanism.\n\n   (2) The final loss function in Eq. (12) omits the KL divergence regularizer from Eq. (10).\n\n   (3) The meaning \"this issue\" and \"ineffective exploration\" in Line 250 is ambiguous. The motivation for the compression loss in Line 250 is unclear. The loss aims to reduce discrepancies between two sampled action sequences $x_1$ and $y_1$, which contradicts to the entropy maximization regularizer (e.g., compression loss encourages low-entropy actions while entropy regularization encourages exploration).", "was_revised": true, "final_justification": "My concerns are well clarified and addressed. I think the paper is technically solid with comprehensive evaluation.", "rebuttal": "We gratefully appreciate the reviewer kGqW for recommending acceptance of our submission and thank his/her valuable suggestions to help improve it. We will answer all the questions that the reviewers are concerned about.\n\n> **Q1**: The Preliminaries section lacks a brief introduction to EDICT, which hinders the reader’s ability to contextualize the proposed exact diffusion inversion mechanism.\n\n**A1**：Thank you for your suggestion. We will integrate a description of EDICT into the Preliminaries section in the final version to improve clarity and completeness.\n\n> **Q2**: The final loss function in Eq. (12) omits the KL divergence regularizer from Eq. (10).\n\n**A2**：Thanks for your question. On one hand, GenPO actually adopts the clipped surrogate objective $L^{CLIP}$ as defined in **Eq. (7)** of PPO [R3], without explicitly incorporating a KL divergence penalty term in the loss function.\n\nOn the other hand, the estimation of the KL divergence during training is to **adaptively schedule the learning rate** as shown in **Algorithm 1, line 11**. To avoid ambiguity, we will explain the purpose of estimating the KL divergence in **Eq.(10)** in Section 4.3 of the final version. \n\n> **Q3(1)**: The meaning \"this issue\" and \"ineffective exploration\" in Line 250 is ambiguous. \n\n**A3(1)**：Thanks for your careful review. We occasionally deleted a sentence before \"this issue ...\". The original text should be \"Since the MDP problem to optimize is reformulated from the original one, featuring an enlarged but practically unnecessary exploration space, we introduce a compression loss to mitigate this issue and prevent ineffective exploration of the action space.\" We appreciate your comment and will fix this problem. \n\n> **Q3(2)**: The motivation for the compression loss in Line 250 is unclear. The loss aims to reduce discrepancies between two sampled action sequences $x_1$ and $y_1$, which contradicts the entropy maximization regularizer (e.g., compression loss encourages low-entropy actions while entropy regularization encourages exploration).\n\n**A3(2)**: Thank you for raising the critical question. The compression loss **does not contradict** to the entropy regularization term. Specifically, as stated in **line 250**, the entropy regularizer encourages sufficient exploration, and the compression loss reduces ineffective exploration. **Ineffective exploration** refers to high action diversity in the double action space but low action diversity in interaction with the environment. \n\nFor example, without the compression loss, the policy may sample dummy actions $\\tilde{a} = (x, y)$ such as $[-1, 1], [10, -9], [-20, 19]$(see **Algorithm 1 line4**), which all result in the similar actions $a = \\frac{x_1 + y_1}{2} = 0$, $0.5$, and $-0.5$, respectively (see **Algorithm 1 line 5**). The entropy regularizer doesn't contribute to effective exploration. In that case, the compression loss encourages $x_1$ and $y_1$ to be close to ensure that exploration occurs in the actual action space, which does not contradict the entropy regularizer. \n\n\n> **Q4**: Evaluation of the effectiveness of adaptive learning rate $\\alpha$ mechanism is missing. How is it compared with constant learning rate?\n\n**A4**： Thank you for raising the concern. The effectiveness of the adaptive learning rate $\\alpha$ mechanism is shown in **Figure 4(b)**. The inaccurate subtitle in line 297 may mislead you. This paragraph is actually for the ablation of the adaptive learning rate. The reason we used the subtitle “KL divergence” is that GenPO uses KL divergence to schedule the learning rate adaptively, as mentioned in **A2**. We will replace \"KL divergence\" with \"adaptive learning rate\" to avoid this misleading in the final version. \n\nSpecifically, in **Figure 4(b)**, \"wo_KL\" denotes training without KL estimation (constant learning rate), and \"w_KL\" denotes training with KL-divergence-based adaptive learning rate. The results show that the adaptive learning rate in GenPO leads to faster and more stable convergence.\n\n>**Q5**: Notation ambiguity in Eq. (7) and Eq. (8): The coefficient $p$ used in the mixing scheme may be misinterpreted as a probability distribution, which is also denoted as $p$. It is recommended to replace $p$ with an alternative notation to avoid confusion.\n\n**A5**: Thanks for your suggestion. We will replace the symbol $p$ with $m$ in the final version.\n\n[R3] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347."}, {"review_id": "cBiYOPUiw6", "reviewer": "Reviewer_tqrN", "summary": "The paper introduces the GenPO framework, which combines the diffusion policy with the online on-policy method PPO. The core contribution is to adapt EDICT's exact diffusion inversion to RL using \"double dummy action\", enabling the exact likelihood calculation. To be more specific, the approach reformulates the MDP with a double action space where the average of two actions is the action to take. The authors add a compression loss to prevent unnecessary exploration. \nThe method is evaluated on 8 tasks in IssacLab compared with model-free methods, DDPG, TD3, SAC, PPO, and two diffusion-based off-policy algorithms, DACER and QVPO, and achieves performance gains.", "questions": "1. Can you provide the comparison with Song Yang's exact likelihood method, including the computation cost and performance?  This is critical to to justify the necessity of your doubled action space approach.\n\n\n2. In the appendix, the 5-step diffusion model seems enough for your tasks. What do you think of the reason behind it?\n\n\n3. If standard RL benchmarks (not just IsaacLab) are evaluated, how do you think the results will be? This gives a fairer comparison environment for all methods.", "limitations": "**Novelty and Motivation**:\nYang Song et al.'s probability flow ODE already provides exact likelihood computation for diffusion models. Why is your doubled action space approach necessary when this established method exists?  Especially when the diffusion steps are only 5\n\n\n**Method comparison**: \nThe experimental comparison lacks the most relevant baselines: DPPO used in online settings", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- achieve performance gains compared to their baselines and a comprehesive experimental evaluation \n- well-written with a well-structured presentation \n- nice cross-domain adaptation from image to RL\n\nWeakness:\n1. **Computational inefficiency**: double action space doubles the computation and introduces one more loss and one more hyperparameter to tune.\n2. **likelihood computation**: In SDE from Song Yang, the probability flow ODES already solves the exact likelihood calculation using deterministic ODEs. In the experiments, the diffusion step is only 5, so why not use their method? Why make it more complex to double the action space? Also, though the main idea of DPPO is used in offline-to-online settings, it is possible to apply it in online settings. In DPPO github projects, there are two ways of calculating the likelihood. From my view, it is necessary to add their likelihood calculations as baselines. \n3. **Biased experimental setup**:  IssacLab's parallel environments is not that fit off-pooicy diffusion methods like DACER, QVPO", "was_revised": true, "final_justification": "The authors responses and other reviewers' comments convinced my concerns.", "rebuttal": "Thanks for your valuable comments and suggestions. Here we address your detailed comments as below:\n\n>**Q1**: Computational inefficiency: double action space doubles the computation and introduces one more loss and one more hyperparameter to tune.\n\n**A1**: Thank you for raising this concern. We need to clarify that doubling the action space and the compression loss does not introduce significant computation cost. Training time remains acceptable under multi-GPU settings (see **Appendix Figures 11–12**), and inference takes only about 2 ms (see **Appendix Figure 10**), meeting real-time demands of most robotics tasks [R4， R5]. Compared to Jacobian computation, the added loss is negligible. The hyperparameter $\\nu$ is **easy to tune** and **fixed to 0.01** in all experiments.\n\n\n>**Q2(1)**: likelihood computation: In SDE from Song Yang, the probability flow ODES already solves the exact likelihood calculation using deterministic ODEs. In the experiments, the diffusion step is only 5, so why not use their method? Why make it more complex to double the action space?\n\n**A2(1)**: That is a good question. Although Song Yang's SDE framework [R6] guarantees exact likelihood in theory, **it cannot obtain the exact probability density in practice**. This is because the diffusion process is implemented by discretizing the underlying probability flow ODE in practice, while the computation method proposed by Song does not guarantee exact probability density estimation under such discretization. The concrete explanation can be referred to **line 206-210 and Eq.(6)**. In that case, we must use the exact diffusion inversion technique with doubled action space to compute the exact diffusion probability density.\n\n\n>**Q2(2)**:  Also, though the main idea of DPPO is used in offline-to-online settings, it is possible to apply it in online settings. In DPPO GitHub projects, there are two ways of calculating the likelihood. From my view, it is necessary to add their likelihood calculations as baselines.\n\n**A2(2)**: Yes, certainly we can do that, but not necessary. We view DPPO more as a fine-tuning method than an on-policy method because DPPO is much worse than PPO in online reinforcement learning tasks learned from scratch(see **Table 1 in** [R9]). \n\nUsing two implementations provided by the DPPO project [R12] actually performs worse than the original DPPO(see **Appendix C.6 Figure A10 of DPPO** [R10]). These two ways estimate the Jacobian using Gaussian or Rademacher noise via the Skilling-Hutchinson estimator. Although theoretically unbiased, both suffer from high variance, which undermines the accuracy of the estimated likelihood. If you insist on adding it, we will add it to the final version.\n\n>**Q3**: Biased experimental setup: IssacLab's parallel environments are not that fit off-pooicy diffusion methods like DACER, QVPO\n\n**A3**：Thank you for raising the concern. Due to the fact that GenPO is the first method designed for on-policy diffusion-based RL (existing diffusion-based online RL algorithms are all off-policy), we have no choice but to use DACER and QVPO as diffusion-based baselines to demonstrate both the superiority of our approach and for completeness of experiments.\n\n>**Q4**: Can you provide the comparison with Song Yang's exact likelihood method, including the computation cost and performance? This is critical to to justify the necessity of your doubled action space approach.\n\n**A4**: Thanks for your question. As illustrated in **A2**, Song Yang's method cannot provide an exact likelihood for diffusion models with a discrete step in practice. Therefore, it is necessary for GenPO to double the action space to make the diffusion model reversible and then calculate the exact policy likelihood.\n\n>**Q5**: In the appendix, the 5-step diffusion model seems enough for your tasks. What do you think of the reason behind it?\n\n**A5**：Thanks for your question. Our method actually uses a special case of diffusion models known as Flow Matching, where 5 denoising steps are sufficient for the Flow Matching model to learn a powerful diffusion policy from state to action (see **Figure 10\\(c\\) in [R7]**). \n\n>**Q6**: If standard RL benchmarks (not just IsaacLab) are evaluated, how do you think the results will be? This gives a fairer comparison environment for all methods.\n\n**A6**：Thanks for your question. We believe GenPO can still perform well in other standard RL benchmarks. However, current simulators capable of massive environment-level parallelism (e.g., IsaacGym/IsaacLab [R8]) are much more suitable for evaluating the potential of **on-policy** methods, while off-policy methods often suffer from learning efficiency of value networks under massive parallelism [R13, R15]. This makes IsaacLab the most appropriate platform to benchmark and validate GenPO. \n\n>**Q7**: Novelty and Motivation: Yang Song et al.'s probability flow ODE already provides exact likelihood computation for diffusion models. Why is your doubled action space approach necessary when this established method exists? Especially when the diffusion steps are only 5\n\n**A7**：Thank you for raising the concern. As mentioned in **A2**, since the diffusion model actually discretizes the reverse ODE, it is infeasible to compute the exact policy likelihood $\\pi(a|s)$, a key component for importance sampling in GenPO. Therefore, our motivation and novelty lie in making the diffusion policy **reversible** through a double action space, which allows for the calculation of the exact policy likelihood.\n\n>**Q8**: Method comparison: The experimental comparison lacks the most relevant baselines: DPPO used in online settings\n\n**A8**: Thanks for your suggestions. DPPO cannot be viewed as an online RL method and performs poorly when learned from scratch, as shown in **Figure A7** of [R10], and performs worse than PPO on many tasks in **Table 1** of [R9]. It is also mentioned in the GitHub repo[R11] of DPPO that **DPPO's advantage lies in the fine-tuning of robot manipulation tasks, because the policy does not need to be extensively explored during training**. Therefore, it is appropriate to regard DPPO as a fine-tuning method rather than an online reinforcement learning algorithm.\n\n[R4] He, T., Luo, Z., Xiao, W., Zhang, C., Kitani, K., Liu, C., & Shi, G. (2024, October). Learning human-to-humanoid real-time whole-body teleoperation. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 8944-8951). IEEE.\n\n[R5] Xue, Y., Dong, W., Liuˆ, M., Zhang, W., & Pang, J. (2025). A Unified and General Humanoid Whole-Body Controller for Versatile Locomotion. arXiv preprint arXiv:2502.03206.\n\n[R6] Song, Y., Kingma, D. P., Kumar, A., Ermon, S., & Poole, B. (2020). Score-Based Generative Modeling through Stochastic Differential Equations. arxiv preprint arxiv:2011.13456.\n\n[R7] Park, S., Li, Q., & Levine, S. (2025). Flow q-learning. arXiv preprint arXiv:2502.02538.\n\n[R8] Makoviychuk, V., Wawrzyniak, L., Guo, Y., Lu, M., Storey, K., Macklin, M., ... & State, G. (2021). Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108.10470.\n\n[R9] Ma, H., Chen, T., Wang, K., Li, N., & Dai, B. (2025). Efficient Online Reinforcement Learning for Diffusion Policy. arXiv preprint arXiv:2502.00361.\n\n[R10] Ren, A. Z., Lidard, J., Ankile, L. L., Simeonov, A., Agrawal, P., Majumdar, A., ... & Simchowitz, M. (2024). Diffusion policy policy optimization. arXiv preprint arXiv:2409.00588.\n\n[R11] https://github.com/irom-princeton/dppo/issues/46#issuecomment-2817918130\n\n[R12] https://github.com/irom-princeton/dppo/blob/main/model/diffusion/exact_likelihood.py\n\n[R13] Liang, E., Liaw, R., Nishihara, R., Moritz, P., Fox, R., Goldberg, K., ... & Stoica, I. (2018, July). RLlib: Abstractions for distributed reinforcement learning. In International conference on machine learning (pp. 3053-3062). PMLR.\n\n[R15] Li, Z., Chen, T., Hong, Z. W., Ajay, A., & Agrawal, P. (2023, July). Parallel $ Q $-Learning: Scaling Off-policy Reinforcement Learning under Massively Parallel Simulation. In International Conference on Machine Learning (pp. 19440-19459). PMLR."}]

Author rebuttal:

Meta-review: This paper presents a novel adaptation of generative diffusion from image generation to PPO. This includes some non-trivial novel double-action likelihood computation. The authors test many ablations and outperform strong RL baselines. A downside is the authors preference for massively parallel setups, which bias the evaluated environments somewhat against the tested baselines.

However, all reviewers like the paper and recommend acceptance. It seems to me that there is really nothing substantial wrong with this paper, and I do not see why it should be rejected.
Decision: Accept (poster)