Adversarial Diffusion for Robust Reinforcement Learning
Authors: Daniele Foffano
Alessio Russo
Alexandre Proutiere
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning
Diffusion
Robust
Model-Based Reinforcement Learning
Primary area: reinforcement_learning

Abstract: Robustness to modeling errors and uncertainties remains a central challenge in reinforcement learning (RL). In this work, we address this challenge by leveraging diffusion models to train robust RL policies. Diffusion models have recently gained popularity in model-based RL due to their ability to generate full trajectories "all at once", mitigating the compounding errors typical of step-by-step transition models. Moreover, they can be conditioned to sample from specific distributions, making them highly flexible. We leverage conditional sampling to learn policies that are robust  to uncertainty in environment dynamics. Building on the established connection between Conditional Value at Risk (CVaR) optimization and robust RL, we introduce Adversarial Diffusion for Robust Reinforcement Learning (AD-RRL). AD-RRL guides the diffusion process to generate worst-case trajectories during training, effectively optimizing the CVaR of the cumulative return. Empirical results across standard benchmarks show that AD-RRL achieves superior robustness and performance compared to existing robust RL methods.

to modeling errors is at least as important as Adversarial Diffusion for Robust Reinforcement Learning
Daniele Foffano"
Alessio Russo
Division of Decision and Control Systems KTH, Royal Institute of Technology foffamo@kth.se
Faculty of Computing and Data Sciences Boston University
arusso2@bu.cd edu
Alexandre Proutiere
Division Decision and Control Systems KTH, Royal Institute of Technology aleprockth.se
Abstract
Robustness to modeling errors and uncertainties remains a central challenge in reinforcement learning (RL). In this work, we address this challenge by leveraging diffusion models to train robust RL policies. Diffusion models have recently gained popularity in model- -aaed RL due their ability to generate full trajectories "all at once", mitigating the compounding errors typical of step-by-step transition models. Moreover, they can be conditioned to sample from specific distributions, making them highly flexible. We leverage conditional sampling to learn policies that are robust uncertainty in environment dynamics. Building the established nection between Conditional Value at Risk (CVaR) optimization and robust RI, we introduce Adversarial Diffusion for Robust Reinforcement Learning (AD-RRL). AD-RRL guides the diffusion process to generate worst-case trajectories during training, effectively optimizing the CVaR of the cumulative return. Empirical sults across standard benchmarks show that AD-RRL achieves superior robustness and performance compared existing robust RL methods.
Introduction
Reinforcement Learning (RL) has produced agents that surpass human-level performance in various domains 22 555 46 45 (30 However, the same policies are notoriously sensitive small dynamics changes, sensor noise, or hardware mismatch, all of which can cause dramatic performance collapse. In safety-critical domains such robotics, finance, healtheare-wwerr collecting new data is expensive, risky, or legally maximizing nominal reward.
Model-based RL improves sample efficiency by learning a world-model and planning within it, but faces key robustness obstacles: (i) compounding errors, which accumulate over long horizons and (ii) the Sim2Real gap, where controllers that succeed in simulation fail after minor real-world deviations [1 6 Compounding eтror occurs in autoregressive models, where the model predicts one step ahead and then is fed its own prediction back: the state predicted at time t used to predict the state at 1+1. Small one-step and performance degrades. The Sim2Real gap arises because, even with highly accurate simulator that minimizes unrealistic artifacts, simulated physics can never perfectly replicate reality. This
56 g
accumulate, the trajectory moves away from reality,
Conresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

discrepancy leads to reduced real real-world performance due to inherent modeling inaccuracies. To overcome these challenges, RL algorithms should be made more robust by optimizing not only the expected return but also the performance under adverse or uncertain dynamics.
Recently, diffusion models have been proposed to mitigate compounding errors by generating entire trajectories rather than predicting one step at a time [38] 17 While this reduces error accumulation, diffusion models remain imperfect: the trajectories they generate may deviate from real-world dynamics. As a result, transferring policies learned in simulation to the real world remains challenging. Despite recent progress, diffusion-basee RL methods often struggle to maintain robustness when deployed in environments with unseen or perturbed dynamics.
Adversarial and risk- sensitive approaches have been explored to enhance robustness against model errors. These methods introduce worst-case perturbations during planning 3 37 or optimize Conditional Value at Risk (CVaR) objectives, which have been shown improve resilience both reward variability and model inaccuracies [34 5] In this work, we show that diffusion models and CVaR-based approaches can be seamlessly integrated to complement each other. We propose Adver- sarial Diffusion for Robust Reinforcement Learning (AD-RRL), novel algorithm that combines the strengths of diffusion models and CVaR-based robustness. By leveraging trajectory-level generation to mitigate compounding errors and incorporating risk-aware objectives, AD-RRL enhances the adaptability and robustness of RL agents to modeling mismatches and envirommental uncertainty. More precisely, we make the following contributions.
(a) We present Adversarial Diffusion (AD), guided diffusion model that for given policy, generates trajectories that are challenging for the agent and result in relatively low rewards. These trajectories either rare in the current environment
originate from unexplored regions of the domain. We show that by learning from such adversarial scenarios, the agent can improve its robustness to modeling errors. To generate these trajectories, we leverage the CVaR framework, applied to trajectory rewards, and demonstrate how guided diffusion can be used to efficiently implement this objective. This mechanism forms the foundation of AD.
(b) Building on this, we introduce AD-RRL, our RL algorithm that integrates AD within the Advan- tage Actor-Critic (A2C) framework. AD-RRL significantly enhances the agent's 's adaptability and robustness. We empirically evaluate AD-RRL across multiple environments from the Gym/MuloCo suite, showing that it achieves superior robustness to modeling errors. In transfer scenarios involving variations in physics parameters, AD-RRL consistently outperforms state-of-the-an baselines.
Related Work
Model-Based Reinforcement Learning. In Model-Based Reinforcement Learning, the agent uses model generate new data, through which it is possible plan further without interacting with the environment. This is essential for settings where collecting new data is impossible, illegal or dangerous. The parametric approach has received a lot of attention thanks the constant
improvements of function approximators, such as Deep Neural Networks [32 7 18 16 53 Recently,
Variational Auto Encoders 19 and Transformers [54] have seen many successful applications as
powerful models for environment dynamics (28, 33 42 12 III leading to state-of-the-and arr methods terms of sample efficiency and performance 13. These methods rely on bootstrapping to generate trajectory samples. The state prediction generated by the model is fed again as input to the model to predict the next state. As result, these methods introduce two sources of error: one coming from an imperfect model and one from the input of the model always being wrong, except for the first timestep. The sum of these errors is commonly known as the Compounding Error problem of Model-Based methods. Multi-step prediction solutions have been proven effective even before the introduction of Diffusion models, for example by learning H models to look Hl steps into the future 3. It goes without saying that this approach results in much higher learning complexity with respect to hingl--moel approaches. Only recently, with Diffusion models becoming more popular, we have seen the rise of more efficient multi-step Model-Basee RL methods 38 I7
Diffusion Models in Reinforcement Learning. Diffusion models are inspired by non-equilibrium
thermodynamics [4 15), defining data generation as an iterative denoising process. Beyond being powerful function approximators, they also offer a natural way to condition the data generation process on labels [10]. Recently, diffusion models have gained significant attention in the RL

---

AD-RRL Training
Collect real data D Env
Train RL. agent Improve a on
D' = (re Ji
Diffusion training
Po. Ze
Generate data D' ~ polrolzo & Co)
Improve ps, Zo D
Adversarial Diffusion
9. V., (x)
T Po 16 E C.) = (N) a24..
1)
9-1
potrolto C.)
g(T./T.-))
Figure 1: A high-level overview of AD-RRL. Following a Dyna-like structure 49 iteratively: samples trajectories from the real environment with given policy æ, improves the Diffusion models on the collected data, uses an Adversarially Guided Diffusion model (Section 4 to generate challenging synthetic trajectories which in turn are used improve æ. The loop repeated until convergence to the optimal policy n²
the algorithm
community. They have been used to model system dynamics, generating trajectory segments by
predicting either states 58), actions 4 22), or both [17 24]. Guidance techniques further
refine trajectory generation by conditioning the process on value estimates, promoting high-expected-
reward sequences. Additionally, diffusion models have been employed for policy modeling [56] [14] and value function approximation (26). Most research on diffusion models in RL has focused on the offline setting. In this paper, shift the focus to the online case, building on PolyGRAD
online,
Dyna-style Model-Based RL. method that uses diffusion for modeling dynamics (38). While prior work primarily uses conditioning to generate high-reward trajectories, we take the opposite approach. Our goal is to generate challenging trajectories-thor those that are underexplored or unlikely- the agent can learn a more robust policy, better suited to handling changes in dynamics and modeling errors.
Robust Reinforcement Learning. Classical RL can struggle to generalize when test environments deviate from training due model errors or shifts. Robust RL addresses this by accounting for uncertainty in actions, states, and dynamics. One line work regularizes transition probabilities within a defined uncertainty set [9 20 23) This kind of methods, despite being theoretically sound and robust, do not scale well to more complex environment.
A well-known approach to tackling complex robust RL problems while maintaining theoretical guarantees is to frame the optimization problem as two-player game [1] In this framework, two players are trained iteratively solve maximin optimization problem: the primary agent aims to maximize the expected cumulative reward, while an adversarial agent attempts to minimize by introducing disturbances. For instance, in Robust Adversarial Reinforeement Learning (RARL) [5), the adversary applies external forces to disturb the environment's 's dynamics. Max-min TD3 (M2TD3) [50] follows similar strategy, solving maximin problem to maximize the expected reward under worst-case scenarios within an uncertainty set. In Noisy Action Robust MDPs [51], the adversary perturbs the agent's actions, while in State Adversarial MDPs 48). the adversary introduces perturbations to the state, resulting in a Partially Observable MDP formulation.
Several Robust RL algorithms use CVaR to constrain their optimization problems [57] For instance, CVaR-PPO an extension of PPO 44 solving risk-sensitive constrained optimization problem that constrains the CVaR to given threshold.

---

Finally, we have algorithms using Domain Randomization (DR) [52], where the agent maximizes the expected return on average, over a predefined uncertainty for some given environment parameters. These classes of methods have been proven very effective in domains such as robotics 23] However, they do not aim to be robust to the worst-case scenarios, and might fall short when tested on environments outside of their training distribution.
3 Background and Problem Statement
3.1 Markov Decision Processes and Reinforvement Learning
Consider a Markov Decision Process (MDP) M = A, 7)r,)), where S and A are the state and action spaces, respectively, P()s,,)) a) the transition probability function, r(-]s, a) is the reward function and y is the discount factor. By interacting with the MDP. a Reinforcement Learning agent is able to collect sequences of states, actions and rewards, forming trajectories TH). The objective of the Reinforcement Learning agent is to learn
= (So: Op, , SH an optimal policy œ maximizing the policy value v=(s) E, DE ?'r++i+1|s: s). In this paper, we consider model-basee RL setting, where we use diffusion model to approximate the distribution of trajectories under a given policy. Specifically, if p" denotes the true distribution the trajectories under policy æ, the diffusion model samples trajectories with distribution pe close to y². We adopt Dyna-style approach [4),, where the diffusion model and the policy are iteratively updated: the policy is improved using data collected from the model, while the model is improved using samples gathered from the target environment using the learned policy.
3.2 Robust RL through the Conditional Value Risk.
We now discuss Conditional Value at Risk (CVaR) and its connection to Robust RL.
Conditional Value at Risk. When learning policies robust to modeling errors, framework commonly used is the one of Conditional Value at Risk. We define the return of a trajectory T by
((T) = =>in 7'r where T2 is the reward obtained at time / in this trajectory Under policy æ ((T) a random variable with edf F. The Value-at-Risk (VaR) of Z at confidence level (0,1) 1) corresponds to its a quantile:
VaR = max(22 S a).
(1)
The Conditional Value-at-Risk (CVaR) of Z is then defined as the expected value of on the lower a-portion of its distribution
CVaR (2) Ez| < (ZR
CVaR dual formulation and its connection to robustness to modeling errors. An alternative way of defining CVaR stems from its dual formulation 2 40
CVaR (Z)
min &EMo
Er~p" [(T))
where & acts as perturbation of the return 2. This perturbation belongs to the set U called the risk envelope and defined as
U
{ VTV,TE(T)
0, [ á
EE--p- [E(T]]
33 states that the CVaR of Z can be defined its expected value under a worst-case perturbed distribution.
In RL, optimizing a CVaR objective introduces robustness to model misspecification. This is exactly because of the dual form CVaR, where the trajectory distribution distorted by an adversarial density {(T). CVaR optimization in this case equals maximizing the worst-case discounted reward when adversarial perturbations are budgeted over the whole trajectory rather than at each time step [5]. The connection between CVaR and robustness modeling errors is well established in the RL field [34] 37 35), and the dual formulation is at the core of our method, as explained in Section 4 and Appendix B
"To avoid cluttering, write Z instead of Z(T) unless it is required avoid misunderstandings.

---

3.3
Diffusion Models
In this work, we adopt model-based approach learn robust policies. To achieve this, we harness the efficiency of diffusion processes to learn parameterized model of the trajectory distribution. This model allows us to sample trajectories T as if they were generated by the true MDP, enabling policy training on these synthetic trajectories.
Diffusion models generate data by progressively refining noisy inputs through iterative denoising process, Pe Pe(T:-1|T.) This process reverses the forward diffusion, q( ((T.(T:-1), which gradually cor- rupts real data by adding random noise. Each step of the denoising process is typically parameterized as Gaussian distribution
Pe p [T)) (To((T) E.),
(5)
with learned mean and fixed covariance matrices, both depending on the diffusion step
The denoising process is formulated as
po(To.N) = p(TN) II (T
(6)
where p(TN) =e ( I) and To is the real (i.e., noiseless) trajectory. The parameters 0 are learned by optimizing the variational lower bound on the negative log likelihood:
0 arg min ET.l log (To)),,
where (T() (o(To:N)dTI.N)
Guided diffusion. A classifier p(ylTa (T)) adding information about the sample to be reconstructed (e.g., the optimality of the trajectory) enhance the generative performance of the diffusion model 10
Po (Toly) o po(To))((lTTo.
(8)
By leveraging the classifier's gradient, can guide the denoising process toward generating samples that align more closely with the classifier's 's predictions. This method, called Classifier-Guidee Diffusion, generates trajectory samples according
Po(T:-IlTs,9) N (Ho(TT))) + E.g. E,)
where g, = V
3.4 Problem statement
We consider model-based Reinforcement Learning setting. For given policy we learn model Po of the distribution of the corresponding trajectories. Our goal is use po to improve the policy and its robustmess to modeling errors.
We formulate the problem of learning robust policy using the following optimization problem:
Ta arg max CV
(10)
arg max min Ey-p- ((T))((T)] EElf
(11)
(1)) describes the objective to obtain the policy that maximizes the return on the worst a-percentile of the trajectories, in terms of cumulative return. However, directly sampling trajectories from this worst a-percentile is challenging. In our approach, we leverage the dual definition of CVaR, solving instead the double optimization problem described in (TT). The problem can be seen game where an adversarial agent & is perturbing the trajectories distribution under given policy We model this distribution via diffusion model pe. which allows to leverage guiding techniques. We introduce adversarial guiding, method that steers the diffusion process toward sampling trajectories that minimize the expected return for the agent. Because the adversarial guide actively seeks to reduce return, the generated trajectories naturally fall within the worst a-percentile. We formally demonstrate that the resulting adversarially guided diffusion models can be adapted to actually sample from the worst a-percentile. We also empirically validate approach.
5

---

Adversarially Guided Diffusion Models
In this section, consider fixed policy n, and for notational convenience, we drop the correspond- ing superscripts. We explain below how to efficiently generate adversarial trajectories, sampled from the set of trajectories C. = {T ((r) I (aR,(2);
Sampling suboptimal trajectories. To steer the diffusion process towards the set Ca we need to define the proper guidance classifier, as in (8). We start from the definition of CVaR (Z) given in (2) The index Po indicates that the trajectory T from which the return computed is generated using the diffusion model po- We have:
CVaRa.po (7) ET To Cal, =
(12)
/
Z (TTO)PO(TOlTO E Ca)dTo.
(13)
min &El
(14)
where the last equality follows from the dual definition of CVaR presented in (3 To steer the generating process towards trajectories from Ca. can the classifier (o E CalTo) since Po(TulTo E Ca) pe (Too)e E Ca To).
Notice also that if we define (To) the solution to the minimization problem in 14 i.e., the bounded trajectory perturbation minimizing the cumulative reward under the dynamics Pa. we have
{"(To)Po (To) po oo(TolTo E C.) x po (ooo) E CalTT)-
In other words, weighting the distribution with the classifier po (To E Ca To) is equivalent, up to a proportionality constant, to weighting po (T() according an adversarial perturbation 5*. We can hence think of applying a guided diffusion to implement this perturbation. However, the set C. is not known. To address this limitation, and following the approach of [2I], we introduce smooth approximation of (T E C. To), namely exp(-co Σ 7'r.) for some constant co 0. This approximation is intuitively reasonable, as it biases the generation process toward trajectories with lower cumulative rewards.
In the following two subsections, we describe how this guided diffusion can be implemented and how it influences the diffusion process. We also discuss how tune the guided diffusion to ensure that the resulting adversarial perturbation & remains within the risk envelope ue defined in (4)
4.1 Perturbed diffusion model
We use the classifier Pe(T; E CalT:)) so that the trajectories т; generated at every step i of the diffusion process belong the set C.- We assume that PO(T, CalT,) exp (-c; Σt= (')
for some value > 0 as we did for To. In the last approximation, the reward (i) represents the reward collected at time trajectory T; for the i-th step of the diffusion process.
As a slight extension of the guided diffusion principle presented in Section 3.3 we establish the following result (essentially obtained by applying (9) with y = {T; E Ca)).
Lemma 4.1. Assume that the denoising process is Gaussian, that is (5) holds. Assume that for all
i E [N], the approximation Po(T; Ca TT)) exp (-c, EH Et=1'rr )-r(() holds. Then, we can sample
(2)
trajectories from po TolTT E Ca) using diffusson steps of the form:
Po(Ti-1 Ti-1 E C.) =N()o(T -GEgg, where 9; = V ZIooTT, ()) for [N].
(15)
The lemma is proved in Appendix A for completeness. The conditional sampling procedure induces the following perturbed model:
/
N PT Ti-1 E Ca)p(T
Po(To) Po(TolTo E C.) x
(16)
We refer to this sampling procedure an Adversarially Guided Diffusion Model.

---

4.2 Selecting C1 C1......
Note that the Adversarially Guided Diffusion Model depends on the constants C1 CN, and recall that the resulting perturbation must lie within the risk envelope defined in (4). In the following, establish conditions on these constants to ensure this requirement is satisfied. To that end, we first show in Appendix Baaaiiixttaaatt our model pe admits product-form representation:
Lemma 4.2. The Adversarially Guided Diffusion Model can be expressed as po(To) = {(To)Po(TO).
where {(To) [( (To. TToN 1 and where E(To:N) I ((Ti;TT-)) with
(T()
exp (-2 Dr [g.)
(17)
and D; = (T((T))
Note that by definition (since Po is a distribution), we have that Er-pe [5(To)] 1. Hence, we just need to verify that {(To) S 1/a for all To to ensure that & belongs to the risk envelope. We define p such that the trajectories т; lie in bounded space C = [T: Tillo S p} and such that (
(T,, lloo < p. the following proposition, proved in Appendix we provide conditions on
Ci,..., CN that this holds.
Proposition 4.3. For all i E [N], let ni N) 2 0 such that IT ne(a, N) a. (a) When for all E [N]
C S min (/² 2 n (a,N)
P (T,
(18)
9 E.g,
|Eggilo
then we have: for all To, E(TO) S 1/a.
(b) Let E [N]. Assume that D, is a diagonal matrix with (E.) E [0, 1). Assume mi N) (1) Then, for N large enough, 18) holds as soon as S (o..N)
9 E.9,
Since our diffusion model uses cosine noise schedule as in [33. have that for all [N] to ensure that & belongs to the risk
2 log No. Σ; 31I with B, 1), we can set c; 9 E.9, envelope.
Remark 4.4. Note that in analysis, we have assumed for simplicity that the states and the actions were a one-dimensional vector, so that trajectories become Gaussian vectors. We can extend the analysis to the case where states and actions multidimensional at the expense of considering trajectories as Gaussian matrices. Refer to Appendix D for details.
5 Algorithms
We now introduce Adversarial Diffusion for Robust Reinforcement Learning (AD-RRL)) which alternates between model improvement and policy improvement steps (see Algorithm Tp. AD-RRL leverages the adversarial conditional sampling discussed in the previous section to sample trajectories in the worst o-percentile terms of return. Our approach is primarily inspired by PolyGRAD (3 and Diffuser [7].
We adopt the common assumption that the policy follows Gaussian distribution over the action space, parameterized by (s) and (s). The policy is deployed in the real environment to collect new data, which is then used train both the dynamics model pe and the cumulative reward function Zg. Following the standard approach in Dyna-like algorithms [49], we generate synthetic trajectories using our learned models (Algorithm 2 These trajectories are then used to train the policy via an on-policy Reinforcement Learning algorithm.
'The official implementation for AD-RRL can be found on GitHub: damielefoffano/AN RRL
https://githur com/
7

---

Algorithm Adversarial Diffusion for Robust Reinforcement Learning (AD-RRI.)
Algorithm 2 Adversarial Diffusion Trajectory Sampling
1: Input: environment, E;
1: Input: adversarial denoising model po:
2: Initialize: policy, Twi adversarial denoising model, Pe: cumulative reward function Zœi data buffer, D; training iterations M
2: reward model Zo: buffer D; level
3: WN (o(
4: 80 ~ D
3: for 1, ... .M do
5: for i N, ..... do
4: Sample T ~ E using Tux add T to D
6:
set $u t S0 in f;
5: Improve Pa. ZΦ on D & Algorithmm 3
7:
G == v 2 mi (a, N)/g; E:g;
6: Sample (+} ~ Po b Algorithm
F1-1 ~ Nouo -2:0))--3320) G. E.)
7: Improve Th (+} using RL.
9: end for
8: end for
10: return 4o
Algorithm illustrates the training procedure for both our diffusion models and is presented in Appendix alongside additional implementation details. The pseudocode provided is simplified. In reality, the diffusion model consists of a noise prediction function co i) from which the mean is computed in closed form [15]. This model is trained using the following objective function (derived from
(()) E, co(;,,)))
where ~ (d((,,( N)) is the diffusion process step, N(O, 1) is the target noise and T; is the trajectory To ~ after steps of the forward diffusion process adding noise e. We update K times, each time by randomly sampling the step i. The model Z is trained to predict the cumulative reward of the trajectory samples т;.
Both the adversarial diffusion model pe and the cumulative reward function Zg are used to sample adversarially generated trajectories in the worst a- percentile. At every step the diffusion process, we perform inpainting by substituting a real starting state 8o into the generated noisy trajectory 4;- We then proceed to compute C; according gradient is taken with respect to so we only adversarially corrupt the states of the trajectory. To ensure that the generated actions are consistent with the generated states, we use the PolyGRAD diffusion guidance method (38), which generates sequence of actions guided by the gradient of the policy Twe
(T8) and the gradient 9; V. Za Notice that the
Experiments
In this section, empirically evaluate how robust method is. During training, the agent interacts with a fixed instance of the environment, At test time, we alter key physics-related parameters and assess the agent's performance against both robust and non-robust basclines. Our experiments conducted on several optimal control tasks from the MuloCo suite: InvertedPendullmm Reacher, Hopper, HalfCheetah, and Walker. All agents trained in the default MuJoCo/OpenAl Gym environment (fixed physics), for 1.5M steps. Additional results are provided Appendix F
Baseline methods. We evaluate AD-RRL against several state-of-the baselines for robust reinforcement learning:
(a) Domain Randomization (DR) (22_ widely used in robotics 23 27 improves policy generalization by maximizing expected return over distribution of dynamics. However, it does not explicitly account for worst-case lower-percentile outcomes. We implement DR using PPO and refer to the resulting method as DR-PPO.
(b) Max-Min TD3 (M2TD3) 50 frames robustness as a minimax optimization problem, training an actor-critic model to maximize performance under the worst-case dynamics sampled from predefined uncertainty set.
(c) CVaR-PPO (CPPO) [57] augments Proximal Policy Optimization with CVaR constraint, leading to policy-gradient algorithm that explicitly controls the policy's 's risk.
Additionally, compare AD-RRL other baselines in RL.

---

(d) PolyGRAD 38 diffusion-basse model that our work builds upon, generates synthetic trajec- tories via policy-guided diffusion and trains policies in an online model-based setting. improves sample efficiency but lacks explicit robustness to adverse dynamics.
two strong model-free baselines, are also included for comparison. TRPO constrains policy updates using a KL-divergence trust region, while PPO employs a clipped surrogate objective for improved computational efficiency.
(e) TRPO 4 and PPO 4.
Robustness under varying physical parameters. To verify the robustness of AD-RRL, we vary several physical parameters of the environment at test time. For Hopper and Cheetah, we vary body mass, ground friction and environment gravity. For Walker, we modify friction and mass. For Reacher, vary all the actuators' gears (i.e., the torque produced by the actions). For InvertedPendulum, change the cart mass, the pole mass and environment gravity.
In Figure 2 we plot the return under the different algorithms and for selected environments and varying parameters. Additional plots are provided in Appendix [.2 In most environments, AD-RRL consistently outperforms both robust and non-robust baselines. PPO and TRPO appear surprisingly stable, which is likely a consequence of the well-tuned Stable-Baselines3 implementations still matched surpassed by AD-RRL. At the same time, AD-RRL consistently outperforms both DR-PPO and M2TD3, demonstrating greater stability and achieving higher cumulative rewards across all environments.
Furthermore, a direct comparison with PolyGRAD (the foundation of our algorithm) highlights that our modifications significantly improve performance under diverse test-time conditions, enhancing robustness to large parameter shifts and model misspecificatiios. This can clearly seen in Figure 2d] or Figure 2g In the Reacher environment (Figure 2 the difference in performance is less evident, but our model still performs consistently better or on par with the baselines. is also clear from Figures 2a and 2b that while PolyGRAD achieves slightly better performance on the nominal environment (as observed in Table it sacrifices robustness under perturbed conditions.
For some environments see for example Figure 2a and Figure 4b (presented in Appendix F AD- RRL performance degrades for extreme changes the modified parameter (but remains better than other algorithms). We hypothesize that this is because our model is generating challenging trajectories which are nonetheless plausible under the agent policy and environment dynamies. Extreme changes in the environment physics do not reflect these constraints, and relevant trajectories might not be generated often.
Performance on the nominal (training) environments. Table I reports the final episode returns (mean + one standard error) for five MuJoCo continuous-control tasks. Best results are highlighted in bold. The results are obtained on the training environment, with the nominal physics parameters. AD-RRL attains the best mean return on four of the five domains, substantially outperforming the other baselines, showing that our risk-ssensitvy training does not trade nominal optimality for robustness. Only on Hopper, PolyGRAD performs better than AD-RRL, but the margin falls within overlapping confidence intervals. On the easy Inverted Pendulum task, multiple methods (AD-RRI, DR-PPO, PPO) reach the maximum score of 1000, as expected.
Hopper
Cheetah
Walker
Reacher
InvertedPendulum
AD-RRL
3280.23 i 13.83
4126.11 = 246.96
4357.80
187.37
-3.97 1 0.13
1000.00 : 0.00
PolyGRAD
3346.99 1 52.39
3879.16 : 626.40
3489.48
456.70
-4.48
0.13
1000.00 :: 0.00
M2TD3
361.73 13.71
3117.16 55.34
2048.03
508.77
-21.28
5.75
634.76 + 192.46
CPPO
2595.64 = 298.35
2173.30
422.97
2164.30
510.60
-6.06 0.32
979.85 20.15
DR-PPO
2315.90 = 482.17
2429.46 1 558.93
2385.19
1 589.49
-15.80 1.44
1000.00 + 0.00
PPO
2996.90 = 432.28
2408.20
546.33
1894.03
349.06
5.17::057
1000.00 1 0.00
TRPO
3270.27 273.04
2014.91 539.64
3090.80 267.79
-.22 + 0.85
960.60 39.40
Table 1: Return on the training environment (nominal physics parameters) for MuJoCo continuous-
control tasks.
Sample efficiency, The learning curves in Figure 3 (see Appendix F.I)) show that AD-RRL reaches higher or matching final performance with the same number of samples as the baselines, and in several cases converges faster. Hence, our adversarially guided diffusion not only preserves (or improves) performance the training environment (with nominal physics parameters), but also matches the sample efficiency of state-of-the-an model-based and model-free alternatives. In Appendix F.3 we perform ablation study comparing AD-RRL trained 1.5M samples against the Model-Free

---

2550
013
2030
2030
630
500
05
27
06
03
3-6
314
32
605
130
335
130
1.55
110
0.5
3.3
3,5
25
Gravity change
(a) Hopper- Mass
(b) Hopper Gravity
(c) InvertedPendulum Mass Pole
00.00
0--11
2939
1990
1530
1030
37
08
20
13
change
change
Gravity change
(d) Walker Mass
(c) Walker Friction
(f) Cheetah
Gravity
4030
00.00
2030
1030
Mass relatve change
Friction reattre Change
Actaator ge3r
relative change
-AD-AAL
M2T03
PolyGRAA CPPO
AD ARL
M2TD3
-PUYGRAAD CCPP
AD-ARL
M2TD3
PsSyGRRA
CPPP
D8-1PD
7P0
TRPO
08-1P0
-PP
TRPP
08-1PD
-PR
TRPP
(g) Cheetah Mass
(h) Cheetah Friction
Reacher Actuator Gear
Figure 2: Average return across variations in selected physics parameters. Shaded regions indicate + one standard error.
Basclines trained 3M samples. This way allow the model-free methods compensate for their lower sample efficiency by leveraging their computational efficiency. Overall, Figures and show show that even when model-free baselines given twice many samples, AD-RRL remains average better or on par across the proposed tasks.
Conclusion and Future Work
In this work we introduced AD-RRL, novel approach to robust RL. AD-RRL is based on Ad- versarial Diffusion (AD), diffusion model that can sample adversarial trajectories by leveraging the Conditional Value at Risk (CVaR) framework. AD enables agents learn from adversarial scenarios that are either rare unexplored in the environment. We demonstrated that AD-RRL, based on this diffusion model, significantly enhances the robustness of RL agents in the presence of modeling errors. Through empirical evaluation multiple Gym/MuloCo environments, we showed that AD-RRL outperforms current state-of-the-an robust RL. methods.
AD relies on a specific strategy for guiding the diffusion process, and exploring alternative guidance methods presents a promising avenue for future work. Potential directions include (i) modifying the overall diffusion objective beyond the current CVaR framework, and (ii) enhancing the diffusion model architecture algorithms reduce computational overhead.
10

---

8 Acknowledgments
The computations were enabled by resources provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS), partially funded by the Swedish Research Council through grant agreement no. 2022-06725. We also thank Adam Miksits for the invaluable support provided in the development of the numerical results.
References
[1] A. Ajay, Y. Du, A. Gupta, J. Tenenbaum, T. Jaakkola, and P. Agrawal. Is conditional generative modeling all you need for decision- making? arXiv preprint arXiv:2211.15657, 2022.
[2] P. Artzner, F. Delbaen, J.-M. Eber, and D. Heath. Coherent measures of risk. Mathematical finance, 93)):033-228, 1999.
[3] K. Asadi, D. Misra, S. Kim, and M. L. Littman. Combating the compounding-error problem with multi-step model. arXiv preprint arXiv: 13320, 2019.
[4] C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 02783649241227366, 2023.
[5] Y. Chow, A. Tamar, S. Mannor, and M. Pavone. Risk-sensitive and robust decision-making: ovar optimization approach. Advances in neural information processing systems, 28, 2015.
[6] P. Christiano, Z. Shah, Mordatch, J. Schneider, T. Blackwell, J. Tobin, P. Abbeel, and W. Zaremba. Transfer from simulation to real world through learning deep inverse dynamics model. arXiv preprint arXiv:1610.035118 2016.
[7] K. Chua, R. Calandra, R. McAllister, and S. Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advances in neural information processing systems, 31, 2018.
[8] I. Clavera, J. Rothfuss, J. Schulman, Y. Fujita, T. Asfour, and P. Abbeel. Model-based reinforce- ment learning via meta-policy optimization. In Conference on Robot Learning, pages 617-629. PMLR, 2018.
[9] E. Derman, M. Geist, and Mannor. Twice regularized mdps and the equivalence between robustness and regularization. Advances in Neural Information Processing Systems, 34:22274- 22287, 2021.
[10] Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8799 2021.
[11] D. Ha and Schmidhuber. Recurrent world models facilitate policy evolution. Advances in neural information processing systems, 31, 2018.
[12] D. Hafner, Lillicrap, M. Norouzi, and Ba. Mastering Atari with discrete world models. arXiv preprint arXiv:20/0.02/93, 2020.
[13] D. Hafner, Pasukonis, Ba, and T. Lillicrap. Mastering diverse domains through world models, arXiv preprint arXiv:2301. 201422301.04044, 2023.
[14] Hansen-Fstruch, I. Kostrikov, M. Janner, J. G. Kuba, and S. Levine. IDQL: Implicit g-larnning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304. 10573, 2023.
[15] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-688 6851, 2020.
[16] T. Jafferjee, E. Imani, E. Talvitie, M. White, and M. Bowling. Hallucinating value: A pitfall of dyna-style planning with imperfect environment models. arXiv preprint arXiv:2006. 04363, 2020.

---

[17] M. Janner, Y. Du, J. B. Tenenbaum, and S. Levine. Planning with diffusion for flexible behavior synthesis, arXiv preprint WxXiv:2205.0999,, 2022.
[18] L. Kaiser, M. Babaeizadeh, P. Milos, B. Osinski, R. H. Campbell, K. Czechowski, D. Erhan, C. Finn, P. Kozakowski, S. Levine, et al. Model-based reinforcement learning for Atari. arXiv 2019.
preprint
[19] D. P. Kingma. Auto-encoding variational bayes. arXiv preprint 2013.
[20] N. Kumar, K. Levy, K. Wang, and S. Mannor. Efficient policy iteration for robust markov decision processes via regularization. arXiv preprint arXiv:2205.14337, 2022.
[21] Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint 1805.00989099 2018.
[22] X. Li, V. Belagali, J. Shang, and M. Ryoo. Crossway diffusion: Improving diffusion-based visuomotor policy via self-supervised learning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 16841-168444 IEEE, 2024.
[23] Z. Li, X. B. Peng, P. Abbeel, S. Levine, G. Berseth, and K. Sreenath, Reinforcement learning for versatile, dynamic, and robust bipedal locomotion control. The International Journal of Robotics Research, page 02783649241285 161, 2024.
[24] Z. Liang, Y. Mu, M. Ding, Ni, M. Tomizuka, and P. Luo. Adaptdiffuser: Diffusion models as adaptive sell-evolving planners. arXiv preprint arXiv:2302.01877. 2023.
[25] Z. Liu, Q. Bai, J. Blanchet, P. Dong, W. Xu, Z. Zhou, and Z. Zhou. Distributionally robust g-learning. In International Conference on Machine Learning, pages 13623-13643. PMLR, 2022.
[26] B. Mazoure, W. Talbott, M. A. Bautista, D. Hjelm, A. Toshev, and J. Susskind. Value function estimation using conditional diffusion models for control. arXiv preprint arXiv:2306.07290 2023.
[27] B. Mehta, M. Diaz, F. Golemo, C. J. Pal, and L. Paull. Active domain randomization. In Conference on Robot Learning, pages 1162-1177 1176. PMLR, 2020.
[28] V. Micheli, E. Alonso, and F. Fleuret. Transformers are sample-efficient world models. arXiv preprint arXiv:2209. 00588, 2022.
[29] A. Mirhoseini, A. Goldie, M. Yazgan, J. W. Jiang, E. Songhori, S. Wang, Y.-. Lee, E. Johnson, O. Pathak, A. Nazi, al. graph placement methodology for fast chip design. Nature, 594(7862):207-212, 2021.
[30] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep rein- forcement learning. nature, 518(7540):529-533 2015.
[31] Morimoto and K. Doya. Robust reinforcement learning. Neural computation, 17(2):335-359, 2005.
[32] A. Nagabandi, G. Kahn, R. Fearing, and Levine. Neural network dynamics for model- based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 7559-7566. IEEE, 2018
[33] A. Q. Nichol and Dhariwal. Improved denoising diffusion probabilistic models. In Interna- tional conference on machine learning, pages 8162-8171. PMLR, 2021.
[34] Osogami. Robustness and sinsiiinsy in markov decision processes. Advances in neural
information processing systems, 25, 2012.
[35] I.. Pinto, J. Davidson, R. Sukthankar, and Gupta. Robust adversarial reinforcement learning.
In International conference on machine learning, pages 2817-2826. PMLR, 2017.
12

---

[36] A. Rafin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann. Saable-asselinss:: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1-8, 2021.
[37] A. Rajeswaran, S. Ghotra, B. Ravindran, and S. Levine. Epopt: Learning robust neural network policies using model ensembles. arXiv preprint arXiv:1610.01283, 2016.
[38] M. Rigter, J. Yamada, and I. Posner. World models via policy-guided trajectory diffusion. arXiv preprint arXiv:2312.08833: 2023.
[39] Robine, M. Höftmann, T. Uelwer, and S. Harmeling. Transformer-based world models are happy with 100k interactions. arXiv preprint arXiv:2303.07109, 2023.
[40] R. T.. Rockafellar. Coherent approaches risk in optimization under uncertainty. In OR Tools and Applications: Glimpses of Future Technologies, pages 38-61. Informs, 2007.
[41] A. A. Rusu, M. Večerík. T. Rothörl, N. Heess, R. Pascanu, and R. Hadsell. Sim-to-real robot learning from pixels with progressive nets. In Conference on robot learning, pages 262-270. PMLR, 2017.
[42] I. Schubert, J. Zhang. J. Bruce, S. Bechtle, E. Parisotto, M. Riedmiller, J. T. Springenberg, A. Byravan, L. Hasenclever, and N. Heess. generalist dynamics model for control. arXiv preprint 2.05. 2023.
[43] Schulman. Trust region policy optimization. arXiv preprint arXiv: 1ati::::0200777777 2015.
[44] Schulman, Wolski, Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.000 1007:06477777 2017.
[45] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
[46] D. Silver, T. Hubert, Schrittwieser, Antonogiou, M. Lai, A. Guez, M. Lanctot, L. Sifrre, D. Kumaran, T. Graepel, et al. Mastering chess and shogi by self-play with a general reinforce- ment learning algorithm. arXiv preprint 1rXiv: 2017.
[47] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-2265. PMLR, 2015.
[48] S. Stanton, R. Fakoor, J. Mueller, A. G. Wilson, and A. Smola. Robust reinforcement learning for shifting dynamics during deployment. 2021.
[49] R. S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, ((4):160-663 1991.
[50] T. Tanabe, R. Sato, K. Fukuchi, J. Sakuma, and Y. Akimoto. Max-min off-policy actor-critic method focusing on worst-case robustness to model misspecification. Advances in Neural Information Processing Systems, 35:6967-6981, 2022.
[51] C. Tessler, Efroni, and S. Mannor. Action robust reinforcement learning and applications in continuous control. In International Conference on Machine Learning, pages 6215-6224 PMLR, 2019.
[52] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RS/ international conference on intelligent robots and systems (IROS), pages 23-30. IEEE, 2017.
[53] E. van der Pol, T. Kipf, F. A. Olichock, and M. Welling, Plannable approximations to mdp homomorphisms: Equivariance under actions. arXiv preprint arXiv:2002.11963 2020.
[54] A. Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
13

---

[55] O. Vinyals. I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, et al. Grandmaster level starcraft ii using multi-agent reinforcement learning. nature, 575((782)3350-3333 354, 2019.
[56] T. Wang, X. Bao, I. Clavera, J. Hoang, Y. Wen, E. Langlois. S. Zhang, G. Zhang. P. Abbeel, and J. Ba. Benchmarking model- based reinforcement learning. arXiv preprint wrXiv:1977.0203: 2019.
[57] C. Ying, X. Zhou, H. Su, D. Yan, N. Chen, and J. Zhu. Towards safe reinforcement learning via constraining conditional value-at-risk. arXiv preprint arXiv:2206.044: 2022.
[58] Z. Zhu, M. Liu, L. Mao, B. Kang, M. Xu, Y. Yu, S. Ermon, and W. Zhang, Madiff: Offline multi-agent learning with diffusion models. arXiv preprint arXiv:2305. 17330, 2023.
14

---

Contents
Introduction
2
Related Work
Background and Problem Statement
3.1
Markov Decision Processes and Reinforcement Learning
3.2
Robust RL through the Conditional Value at Risk.
Dillusion Models
Problem statement
4
Adversarially Guided Diffusion Models
Perturbed diffusion model
4.2
Selecting Cl,..., CN
5
Algorithms
Experiments
Conclusion and Future Work
10
8
Acknowledgments
11
Adversarially Guided Diffusion Models: Proofs of results from Section 4.1
16
Adversarial guide as a multiplicative perturbation: Proof of Lemma 4.2
17
Attaining the duality constraint on {(To): Proof of Proposition 4.3
17
Adaptation to Matrix Normal Distribution
18
Implementation details
20
Additional Results
22
F1 Training curves
22
2 Varying parameters
23
Ablation study
23
I Limitations and future work
24
Societal impact statement
24
15

---

A Adversarially Guided Diffusion Models: Proofs of results from Section 4.1
Proof of Lemma 4.7 For sufficiently smooth function the conditional distribution (T x (++1)) can be approximated using a Gaussian. Following 47 Appendix CJ (see also (10) we know that p Ti-1 E C.) (Nuo((T + E,h, E.) where h; V log PT T E C. (r
Since we assume that the approximation po (T: CalT.) exp (-c Σ p(r()) holds, we get that
h; = V logp(T Cal
T
V aasa (st, ai)
-V po
where ( is the t-th state-action pair of no (T i). Substituting, we get Po E Ca) = (((H((TT))) - ...g, E.), where g: V+
(19)
Proof of Equation (16) As mentioned earlier, sample from po To E Ca), we multiply each intermediate distribution in the diffusion process by ra(TT), with rT(T.) = exp(-c: E (0(())
where the notation v refers to the t-th reward in for the i-th diffusion step. This means that the corresponding modified distribution po satisfies in the intermediate diffusion step i:
1 PO(T:) 7 (pT)))(((T)))
(20)
Zi
where Z is the normalizing constant. Next, we the same strategy that used in the diffusion process po (T:1) Note first that:
47 determine
po (T:) = f po (oo
Plugging 20). the previous condition can be rewritten as
Zi ri+1(T.+1) Zit1 ri(T.)
Po(T.) = po((TI/T+1))
po Po Po(T:+))dT:H)
(21)
However, know that pę also satisfies:
po(T:) = [o(T7/TH)))(())))))
This implies that 21 holds if:
Z(417i(Te)
(.r:4((Ti+1)
Now defining the normalization constant Zi(Ti+1) = Zirts1(Fi-1)
2.4
we get
1
To [o(T,IT(+1)= Zi(Ti1)
Po Po(T.lTH4)mmim
We conclude that po Ti Ca) po(T:(TT-1). Therefore, we have shown that:
Pe(TolTo C.) Te(T)).
/ po(Tol7)))e(TT)dd)
/ po((olTi)... pof TN,
x To
E C.) 1.. E (T))o(TT)dd).... TNN
16

---

B Adversarial guide as a multiplicative perturbation: Proof of Lemma 4.2
Proof of Lemma 4.2 Let's define denoising diffusion model po(Ti-ilTi). and perturbed de- c; E E.). Since the two
noising step of the form pe (T;-11/ 7i-1 E C.) = distributions are Gaussians with known mean and covariance matrices, we have
Po Ti-1 C.) = -Noo 1) c, 2. 2,)
exp
(D, (D++cZgg)) E. (D; +
2
exp
2
1 D. D.',,D exp
(-) D. g: +ggg E,gi)
exx
-N(T:-1lHOTT i), Ex) exp
(2, gi + + Egg,)
where K (2)))7112,
D. (Ti-1 - uo( i)), and E(T,, Ti-1)
exp (-) 9: + eg Eg.)). N(T:-IlHO(T:.))) E.) is the density of the Gaussian distribution T1-1, with mean (io i) and covariance matrix E.
we define {(To.N) II E(T T(-1), we have
Po(To.N To E Ca) = =((O:N)((TO.))
So we can define
po = po(TolTo E Ca)
polTo:NITO E Ca) = 1)dT
/ ((ToN))(TO::)
J [(( P(To) P(To)
E(TO)P(TO)
with {(ro) f (((T:x))( P(ro)
C Attaining the duality constraint on E(To): Proof of Proposition 4.3
Proof of Proposition
Proof of (a): From Lemma 4.2 we know that E(To)
[ [[(T W We want to have P(ro)
{(TO) S 1. this is equivalent to
f (( P(To)
1
/ ((T S p(To:N)dTI.N.
One way to achieve this is to impose {(TO.N)= = II T(T,, S w. constraining the single terms of the product using 7) (a,aNN) such that E(T,, Ti-1) I mi N) and I. (a, NN 1
1. This is satisfied also by
However, T;-1 a random quantity to which we do not have access at step of the diffusion process. Therefore, to satisfy the constraints the single terms impose
max ((T Ti) S ne(a, N). Ti-1
17

---

5(T:-1, T:) is maximized for (T:-1 Mo(T;, i)) < 0. so we want (Ti-1 (o(TT, i))² to be vector opposite to g. We can take Ti-1 = To. e.g.
We assume that the trajectories т; lie in a bounded space C = (Ti ||Tx|/oc $ p), where ||Ti|ls0 maxser; ||||| oc. From this assumption it follows that
| 1100 - p
[Iua(T;;)) 1) c; g figgllo- ~0 S ( i) + 1 P
(p
(,, |Eig, 1o:0
Substituting 7i-1 (,)(T - Dig, into {(T:-1,7:) S (, N) and developing we get
2 (-2c E,g, + og Egg.) S log (a,
log (a, N) S
g E,g
So combining the two inequalities we can take
C S min (1 log no (a,N)
N) P (T.,
(22)
g Eig,
EEgillo 0o
Proof (b): To find the minimum between the terms in Equation and numerators. For the denominators, v / E,g, and || |E max, ( (Egg.); equivalent compare 9 E,g, and 11.
analyze the denominators
Since our diffusion model adopts a cosine scheduler for the covariance matrix, M, is diagonal with elements (Ex),s [0, 1). We can write the j-th element of E,g, as (E,g)) ej E.g,, where e; a basis vector. Then using Cauchy- Schwarz inequality we get
((Eg.),)² = 1 1
< Eee E..
E.g.),
with e E,e, < 1 by definition of Σ, It follows that (E.g.); S 9 E.g., and since this is true for all we can conclude that
max (Eig.,y) < g E.g,
E.g,lle $ V g E.g,
When comparing the numerators of both terms in Equation since log ma(a, = log (t), for N large enough, 2 log (a, N) < [T loo
So C S
log [: satisfies the dual CVaR constraints.
g E.g.
D Adaptation to Matrix Normal Distribution
Here we extend the analysis the case where states and actions are multidimensional, and consider trajectories as Gaussian matrices.
We define p())
P(T.-I/T.) |M
18

---

where (-N(T:1I|MM U, ,, V v a Matrix Normal Distribution with mean Mo(T., E R=*x, row and column covariances U E R and V, E RPxp.
The probability density function of this Matrix Normal Distribution is defined as
(-2 1 Tr -
M U, Vi) = K2 exp
Mo(T., ()
- - i)]]
where Tr[-] is the trace operator, K; (2n)mm/3[V 1,/ and is the determinant of a matrix.
Define G, Rexp as the gradient V with respect to the second order tensor representing the trajectory 7 E R=xp evaluated at Mo (T i). Also define I; = U,G.V, for notation convenience.
Consider the perturbed distribution with mean Mo(T;, GT;; we get
pe [T Th T1-1 (-)
- + Mo(T(,)) cT.ill ( (ra Mo(Tn.))] E(TA.T-))
=exp
TriV
K expp
with
((Ta)T-1)) = = ( ( U ",
As we did Appendix C we take {(T(, S na,, N). We can take T:-1 Mo(T,, c. and get
exp
--2 - -,
N),,N)
1
T.) (a,
TV; < (o, N),
giving
2oo 17: (1, N)
a S V
THV; JU, (U (U.G.V.)
Under the same assumptions of Appendix
C
we get that
||o Sp p
|| - V. Sp
| + e. U., x Sp
G S (p 1|0 (T.
|U, V 1/o0
So we can pick
C min (V
2 2 (a,
(p- Mo (T.,
U...
Following the same reasoning as in Appendix C if the covariance matrices U, and Vx are diagonal with elements in [0, 1) we can pick
2 (a,
4:
Tr
19

---

E Implementation details
Our method makes use of three MLPs: the policy Tws the adversarial denoising diffusion model Po and the learned cumulative reward function Z
Policy network and training. The policy Tw is parameterized in the same way as PolyGRAD [38] We consider a Gaussian policy of the form tw = =N( (+), ow), where w are the parameters of the MLP. The standard deviation of the policy is a single learnable parameter dw. independent of the state.
The policy is trained using Advantage Actor Critic (A2C) with Generalised Advantage Estimation (GAE). The optimizer used ADAM. The hyperparameters can found Table
Parameter
Value
Batch size
512
Synthetic trajectory length
10
GAE A
0.9
Critic learning rate
3e-4
Actor learning rate
3e-5
Discount factor, 7
0.99
Entropy bonus weight
le-5
Table 2: Hyperparameters for A2C training.
Adversarial Diffusion and Cumulative Reward models. Our implementation builds directly top of PolyGRAD. We follow the same training procedure, summarized in Algorithm 3 For the Diffusion Model, we use the same MLP architecture as PolyGRAD, trained by minimizing the L2 loss with ADAM optimizer. The MDP has skip connections every layer, and features learneable embedding of the diffusion step i, which is common for Diffusion Architectures [17. The hyperparameters are summarized in Table
Parameter
Value
Hidden size
1024
Length of generated trajectory
10
Batch size
256
Diffusion step embedding size
128
Number layers
Learning rate
3e-4
Table 3: Hyperparameters for adversarial diffusion training.
When computing according to 18) we chose 30,, where σ; is the standard deviation of the diffusion process4 at step i. In our implementation we choose (7,a, = (2) (
The cumulative reward model Z follows the structure and hyperparameters of the Diffusion Model (also the step emhedding), with a final linear layer producing a scalar output. It optimized using the I.2 loss and the ADAM optimizer.
'In Denoising Diffusion Probabilistic Models, the standard deviation is fixed at every step according to known scheduling [15].
20

---

Algorithm 3 Diffusion model training
1: Input: adversarial denoising model po: cumulative reward function ZΦ: data buffer, D; diffusion
steps N; training iterations K
2: for k = 1..... K do
3: Improve Po (7) using {To} D.
4: Train Zạ to predict the reward (TT)).
5:
end for
Baselines implementation The implementation of PolyGRAD was taken from the respective github repository [38]. The same was done CPPO and M2TD3: for M2TD3, we created new config file for Reacher, where we set the range of the actuator gear to [50.0, 500.0). For TRPO and PPO we used the implementation from Stable- - [36]. We used Domain Randomization on top of the PPO bascline, training with values of the mass and parameters uniformly sampled according to the uncertainty intervals specified Table
Environment
Mass
Friction
Mass pole
Mass cart
Act. gear
Hopper
[0.5, 6.5]
[0.1,3.0]
HalfCheetah
[3.5, 9.5]
[0.2, 0.8]
Walker
6.55,
[0.5,2.0]
Cartpole
[2.5, 10.0]
[5.0, 20.0]
Reacher
[50.0, 500.0]
Table Uncertainty sets used for domain randomization.
Computational resources The training of AD-RRL and Polygrad was performed on three different machines. On cluster node with one A100 GPU, Icelake CPU and 256 GB of RAM.
The remaining freee-frree baselines were trained on laptop with an Intel 17-1185G7 CPU, Mesa Intel Xe Graphics GPU and 32 GB of RAM.
In 5 we report the wall-clock training time for each algorithm. As expected, the model-based algorithms (AD-RRL and PolyGRAD) are slower than the model-free ones. This a well-known shortcoming of Model-Based RI. methods, even more when using Diffusion Models, known for their longer training times when compared to standard MLPs. AD-RRL is slower than PolyGRAD since employes additional Diffusion Model approximate the cumulative reward of trajectory.
Algorithm
Hopper
Halfcheetah
Walker
InvertedPendulum
Reacher
AD-RRL (ours)
3-20-00
3-20-00
3-20-00
3-20-00
3-20-00
Polygrad
2-14-00
2-14-00
2-14-00
2-14-00
2-14-00
M2TD3
0-02-00
0-03-30
0-02-45
0-02-45
0-02-45
CPPO#
0-00-30
0-00-30
0-00-30
0-00-30
0-00-30
PPO#
0-00-30
0-00-30
0-00-30
0-00-30
0-00-30
TRPO1
0-00-30
0-00-30
0-00-30
0-00-30
0-00-30
DR-PPOR
0-00-30
0-00-30
0-00-30
0-00-30
0-00-30
Table 5: Wall-clock training time (days- hoosshhour-mmiutes needed reach the reported performance on the MuJoCo tasks. Times are rounded up to the nearest quarter hour. TTriied on cluster node. TTraindd on laptop.
21

---

Additional Results
In this section, we provide additional plots support our conclusions.
F.1 Training curves
shows the learning curves for AD-RRL and all the baselines for the considered MuloCo tasks. Across seeds, AD-RRL reaches its final performance at least quickly the other methods. AD-RRI. also achieves a final score matching or surpassing that of the baselines. This is particularly clear for the Cheetah and Walker environments, presented in Figures 3d and Bel These results confirm that our method is more robust to modeling errors but does not sacrifice optimality in the training environment learning speed.
Figure shows
600-
100
30.
Stops
Stops
AD-831
CPPO
M2-DD
PPO
RolyGRAD
AD-88L
-3-4P0
100
PPO -FolGRRAD
TRPO
(a) InvertedPendulum
(b) Reacher
4090
2000
2000
(c) Hopper
Cheetah
5000
1000
0.5
3.0
1.4
AD--3L
-CPP
-D0-PP
-N2TD3
-PolyGRAD
TRPO
(c) Walker
Figure 3: Training-return curves on the nominal environment for five MuJoCo tasks. Shaded areas represent standard error over runs.
22

---

F.2 Varying parameters
In Figure we present additional parameters variations for the InvertedPendulum and Cartpole environment. The pattern is consistent with the plots presented in Figure AD-RRL achieves on par or higher returns than both robust and non-robust baselines as the dynamics deviate from nominal values. The only exception appears to be for higher variations of the cart mass, in the Inverted Pendulum environment (Figure(4b) ( where the additional inertia pushes most methods toward failure and AD-RRL similarly shows performance decline.
3500
2500
2000
500
Friction relative change
(a) Hopper - Friction
1000
1000
600
600
400
1.2
Mass Cart relative change
Gravity relative change
AD-RRL -M2DD3
PolyGRAD CPPO
AD-RRL
M2TD3
PolyGRAD - CPPP
DR-PPO
-PPO TTPO
DR-PPO
-PPO
TRPO
(b) InvertedPendullmm Mass Cart
(c)
InvertedPendullmm Gravity
Figure 4: Average Return for varying physical parameters, Shaded areas represent one standard eπгоr
over runs.
F.3 Ablation study
In this section, compare AD-RRL trained on 1.5M environment samples with model-free baselines trained on 3M samples. This setup allows model-free methods to compensate for their lower sample efficiency by leveraging their greater computational efficiency. In Figure we present the learning curves on the nominal environment for the Model-Free algorithms. We can observe that AD-RRL still outperforms (in the worst case) matches the performance of the Model-Free baselines in all of the environments. This is also reflected when testing the robustness of the algorithms, as shown in Figure 6 AD-RRL still achieves on par or higher returns than robust and non-robust Model-Free baselines. When comparing with Figure 2 we can see that some baselines perform worse than their 1.5M samples versions (c.g.. DR-PPO on the Walker and Hopper environments: TRPO on Hopper). This is expected, since an increased amount of samples might improve the performance on the nominal training environment, but will also lead overfitting, making the agents more vulnerable to parameter changes in the test environment. The only exception appears to be M2TD3, which is able outperform AD-RRL on the Walker environment at test time, when varying the physics parameters, despite achieving the same performance in the nominal environment. However, M2TD3 seems to be more prone to overfitting easier environments, as seen in Figure where the baseline cannot reach the optimal solution but instead suffers substantial performance decline in the latest training stages. Finally, M2TD3 remains stuck suboptimal policy the Hopper environment, similarly to what we observed when training 1.5M samples. The doubled amount of samples does not seem
23

---

to benefit the learning process of this baseline. Overall, even when model free baselines are given twice many samples, AD-RRL remains on average better on par across the proposed tasks.
Dec 000 9oom
200
camplest
CHO
(a) InvertedPendulum
(b) Reacher
2010
2000
000
Staps
146
29P0 DR 3P0 M2703
7P0
TAPO
-D-ARL C sampless
-290
0R -PO
M22D3
-P0
TSP0 -DD-RR (1.H camples]
(c) Hopper
(d) Cheetah
4000
2000
Staps
CHPO -DR PPO
-M2103
-#0
TTPP -- ADDAAL sam.loamples samples!
(e) Walker
Figure 5: Training-return curves on the nominal environment for the Model-Free baslines trained on 3M samples. Shaded areas represent one standard error over 5 runs. The dashed line represents the reference final cumulative reward achieved by AD-RRL trained 1.5M samples.
G Limitations and future work
Computation time. Guided diffusion requires dozens of reverse diffussio steps for every synthetic trajectory and an extra gradient evaluation at each step. Consequently AD-RRL slower than its precursor PolyGRAD, and both model-based methods need higher training time (wall-clock) than the model-free baselines, even though they are more sample efficient. Improving the training time for diffusion models (e.g., fine-tuning the network size the number of denoising steps) sounds like natural next step.
Smooth-dynamic, assumption. Our derivation employs Gaussian approximation and the computa- tion of gradients V, (T). Both of these presuppose reasonably smooth rewards and state transitions. Some tasks might break this assumption, causing inaccurate guidance. Extending adversarial diffusion to domains with non-smooth dynamics left for future work.
Scope of the evaluation. Our evaluation focused on simulated control tasks. natural next step Sim2Real study. That is, AD-RRI is trained entirely in simulation and then deployed real hardware, measuring how much the adversarial-diffuioo training reduces the Sim2Real performance drop.
H Societal impact statement
Our contribution methodological: we propose technique for making model-based RI. more robust to dynamics misspecification. Robustness is typically beneficial-e.g. e.g., safer robot control or fewer

---

3030
20:0
2000
203
19:00
1530
420
0090
1030
200
530
500
06
37
06
33
3-
314
32
605
130
335
130
1.35
140
0.5
3.3
2.5
2.3
25
Gravity change
(a) Hopper- Mass
(b) Hopper Gravity
(c) InvertedPendulum Mass Pole
10.00
0.0
30.00
630
400
230
15:00
09
23
Cartrelativ change
change
Gravity change
(d) InvertedPendullmm Mass Cart
(c) Hopper Friction
(f) Cheetah Gravity
-30
-15
3000
-303
2000
Masa reletive change
Pricion welaiie change
Actuelor gaat reletive change
(g) Cheetah Mass
(h) Cheetah Friction
(i) Reacher Actuator Gear
4000
2040
120
1000
1700
25
24
375
change
reladve change
neavve change
-AD-RAL 12 13.501
TRPD
-A3-88 12.50
-TPD OM
ADD-RL 13.5MI
18P0 CIPO (M
DI1-FO 138)
-PEySRAD 12.5HI
CIYO
DI1-PF (3M)
-PEySRA 11-3M) CIPO (M)
DI1-PO 128)
-MAySRAD 12.5H1
M2703
M22D3 OH)
M2703
Walker Friction
Walker Mass
(I) InvertedPendulum Gravity
Figure 6: Average return across variations in selected physics parameters. Shaded regions indicate + one standard error over runs. AD-RRL and PolyGRAD are trained on 1.5M samples, while the remaining Model-Free Baselines are trained on 3M samples.
failures in medical-decisteee support-yet any improvement in sample efficiency or policy quality also lower the barrier to deploying RL in high-stakes settings. In domains such healthcare, finance, or autonomous driving, deployment must therefore be accompanied by domain-speciff safety checks, bias audits, and human oversight. Our work does not introduce practices, nor does it touch sensitive attributes, but it could be combined with decision pipelines that do. We encourage future users of this method evaluate downstream ethical, legal, and safety implications before real-world deployment.
data-collection
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We specified all the claims and assumptions in the abstract and introduction. The claims are backed up by lemmata, propositions and empirical results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: We discuss the limitations of our algorithm in the Experiments and Conclusions and Future work sections.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
26

---

Answer: [Yes]
Justification: We provide the full set of assumptions in the main body of lemmata and propositions, while the complete proofs provided in the appendix (and referenced in the main body).
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: In the appendix we provide the details regarding hyperparameters for our algorithm and baselines. We also provide pseudocode for our algorithm.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
27

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [NA]
Justification:
The code cannot be submitted in the workshop OpenReview form.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
public/guides
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
descoeuuuus//oo 7/nips cc/ wtc/publicc/ subbiic/guides gww.///ieee//eee// Codeeoobiiieseee SanSSuubissss issionn ssionnoiccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide hyperparameters in the appendix section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Result tables and plots always include the standard error across all the simula- tions.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
28

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In the appendix we provide detailed information about the computing resources used to perform the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We reviewed the NeurIPS Code of Ethies and we do not sensitive data.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss potential negative societal impacts in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
29

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We credit all the authors of software used in the paper (like the experiments baselines).
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
30

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide details about our model (hyperparameterss and pseudocode) and the code itself.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
31

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttpps////eurpss.... /ccc////eeeeeeecs///22/////1 for what should or should be described.
32


Reviews:
[{"review_id": "Kx2nEde1zO", "reviewer": "Reviewer_iLrN", "summary": "This paper proposes a model-based reinforcement learning approach that integrates diffusion models with adversarial robustness. Specifically, the diffusion-based world model is guided to generate trajectories on which the learned policy would achieve low returns, and the policy is subsequently trained to maximize returns on these challenging trajectories. Empirical evaluations demonstrate that the proposed method outperforms baseline approaches, particularly in noisy test environments.", "questions": "- L206: The choice of the approximation formula seems a bit arbitrary. Can the authors clarify why that specific formula was used? For example, what happens if you remove the exponential term?\n- To determine if guiding the diffusion model to generate adversarial trajectories is necessary, it would be helpful to compare the proposed method with simply introducing random noise to the diffusion model (similar to DR-PPO).", "limitations": ".", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Diffusion models have recently gained widespread adoption in reinforcement learning, and this paper presents a compelling approach to enhancing robustness through diffusion-based trajectory modeling.\n- The paper is well-written and easy to follow.\n- Experimental results demonstrate that the proposed method matches or exceeds the performance of baseline methods, particularly when the test environment contains additional noise not encountered during training.\n\nWeaknesses\n- As noted in L51, the proposed method may tend to sample trajectories that are rare or insufficiently explored during training. This raises concerns about increased modeling errors, as the diffusion model may not have been adequately trained on such trajectories.\n- The experiments are limited to relatively small-scale locomotion tasks. Evaluating the method’s scalability to more complex, high-dimensional environments would significantly strengthen the paper.\n- Training guided diffusion models and computing return gradients at every diffusion step can be computationally expensive.", "was_revised": true, "final_justification": "Most of my concerns are addressed. This paper presents a compelling approach to enhancing robustness through diffusion-based trajectory modeling. I keep my current positive rating.", "rebuttal": "We thank the reviewer for their feedback and for the time spent evaluating our paper. We are glad to see that the reviewer found the paper well-written and easy to follow, and appreciated the integration of diffusion-based trajectory modeling with adversarial robustness as a compelling and timely contribution to the field. We are also pleased that the reviewer highlighted the strength of our empirical results, noting that our method matches or outperforms strong baselines, particularly in noisy or perturbed test environments, thereby demonstrating the robustness benefits of our approach.  \n\nBelow, we provide detailed responses to their comments and questions.  \n\n---\n\n**Q1. The proposed method may tend to sample trajectories that are rare or insufficiently explored during training. This raises concerns about increased modeling errors, as the diffusion model may not have been adequately trained on such trajectories.**\n\nWe appreciate the concern raised by the reviewer, and are happy to clarify this point.\n\nNote that our CVaR-based sampling process simply biases the diffusion process toward **trajectories with low return**, which do not need to be *rare* in general.\n\n- While diffusion models are typically trained to replicate the distribution of historical trajectories, our approach modifies the sampling process through CVaR-based adversarial guidance, biasing the model toward low-return trajectories, i.e., regions where the agent tends to fail. Among these low-return trajectories, we expect some to originate from under-explored areas of the environment, as poor cumulative return may suggest the agent has not yet learned optimal behavior in those regions. However, low returns can also arise for other reasons, such as stochastic dynamics or a sub-optimal current policy.  \n- Regarding model accuracy, the training curves in the appendix show that the agent’s performance on the nominal environment continuously improves throughout training, suggesting small modeling errors.  \n  - Importantly, our architecture builds on **PolyGRAD**, conditioning the diffusion process on the policy gradient. Thus, the sampled trajectories are not arbitrary outliers but are likely under the current policy, ensuring they remain realistic.  \n  - The combination of adversarial guidance (AD-RRL) and policy-informed conditioning (PolyGRAD) steers sampling toward challenging yet plausible trajectories, balancing robustness and model fidelity.\n\nWe will clarify these points in the final version of the manuscript.  \n\n---\n\n**Q2. The experiments are limited to relatively small-scale locomotion tasks. Evaluating the method’s scalability to more complex, high-dimensional environments would significantly strengthen the paper.**\n\nWe understand the reviewer’s concern. Our current experiments focus on **well-established continuous-control benchmarks** that allow clear comparison to prior work (see for example [1,2,3,4], a small selection of papers using these same benchmarks), highlighting the advantage of our adversarial guidance mechanism. We agree that evaluating scalability to more complex, high-dimensional environments (e.g., vision-based settings) is a valuable future direction.\n\nFurthermore, our approach is modular: the adversarial guidance does not depend on domain-specific structure, suggesting that it should generalize well to more complex tasks. We aim to explore such extensions in future work.  \n\n---\n\n**Q3. Training guided diffusion models and computing return gradients at every diffusion step can be computationally expensive.**\n\nThis is an important practical point, but note that our methods is approximately $1.5\\times$ more computationally expensive than PolyGRAD, which does not include adversarial guidance (note that compared to PolyGrad our method requires an additional model).\n\n- Being a model-based approach, it is natural to expect it to be more expensive than model-free approaches. However, we believe this overhead is justified by the significant gains in robustness and nominal performance, as demonstrated in Figures 1–3 and Table 1.  \n- AD-RRL consistently outperforms all baselines, including model-free and model-based alternatives, on both nominal environments and unobserved test-time conditions. This highlights a positive tradeoff between computational resources and robustness.\n\nWe agree that reducing this computational overhead is an important direction for future work.\n\n---\n\n**Q4. The choice of the approximation formula seems a bit arbitrary. Can the authors clarify why that specific formula was used? For example, what happens if you remove the exponential term?**\n\nWe would like to highlight that we are not the first to employ such ideas. In fact, such approximation is also used in other relevant prior works (e.g., [5,6]) focusing on RL for continuous tasks. \n\n**The main idea** borrows from [5], which casts the control problem as a probabilistic inference problem. Practically speaking, they introduce a binary random variable $O_t$ indicating whether timestep $t$ is optimal, and parameterize the probability of $O_t$ as $P(O_t=1|s_t,a_t)=\\exp(r(s_t,a_t))$. This is a **smooth** approximation of the indicator function $P(O_t=1|s_t,a_t)={\\bf 1}_{(s_t,a_t)\\in C}$, where $C$ is the set of optimal state-action pairs.\n\n- While such approximation may seem arbitrary, in fact it leads to a very natural posterior distribution $P(\\tau|O_1,\\dots,O_T)\\propto P(\\tau)\\exp(\\sum_t r(s_t,a_t))$. That is, the probability of observing a  trajectory (given its optimality) is given by the product between its probability to occur and the exponential of the total reward along that trajectory.\n- In AD-RRL we follow this approach, but need to adapt it to our setting: a diffusion process does not generate one state-action, but an entire trajectory at the time. For this reason, we do not model the \\emph{optimality of a time-step}, but rather the optimality of an entire trajectory $\\tau$: \n  - We introduce a binary random variable $O(\\tau)$ indicating wether $\\tau$ is optimal, and parametrize such random variable by $P(O(\\tau)=1|\\tau)=\\exp(-c_0\\sum_t \\gamma^t r_t)$ (we consider the discounted return since the optimality of a trajectory is based on its total discounted return). \n  - Since this is a smooth parametrization, it enables gradient-based guidance in the diffusion process that steers sampling toward low-return (adversarial) trajectories. \n  - Without such an approximation, the adversarial objective would be **intractable** for diffusion-based planning.\n\n---\n\n**Q5. To determine if guiding the diffusion model to generate adversarial trajectories is necessary, it would be helpful to compare the proposed method with simply introducing random noise to the diffusion model (similar to DR-PPO).**\n\nThis is a great point, we thank the reviewer for the suggestion. We believe that just adding random noise to the diffusion model would not yield improvements in performance or robustness, since diffusion models inherently inject Gaussian noise at every denoising step. \n\n- As an alternative, for this rebuttal, we implemented a Domain Randomization variant of PolyGRAD (which we refer to as DR-PolyGRAD), where the agent is trained to maximize expected return over a uniform distribution of randomized environment dynamics, in line with the DR-PPO formulation.\n- Note that Domain Randomization methods assume access to the environment simulator to randomize physical parameters. This assumption is often impractical in many real-world settings where the agent does not have access to a simulator. In contrast, AD-RRL learns a dynamics model and uses it to generate entire trajectories into the future, enabling adversarial sampling without requiring additional environment access during training.\n- Due to time and resource constraints, we were able to compare the algorithms only on one environment (Hopper). Since we are not allowed to share plots over anonymized links, we provide point-wise evaluations of the two methods in the following table. We can see that AD-RRL still outperforms DR-PolyGRAD in robustness to unobserved test conditions. AD-RRL also achieves better performance than DR-PolyGRAD when the mass is close to its nominal value.\n\n| **Mass relative change** | **AD-RRL (Ours)** | **DR-PolyGRAD** |\n| --- | --- | --- |\n| 0.506 | **2431.643 ± 543.152** | **2678.814 ± 191.608** |\n| 0.749 | **3147.467 ± 12.367** | 2842.534 ± 234.759 |\n| 0.992 | **3279.055 ± 14.408** | 2952.995 ± 240.776 |\n| 1.331 | **3280.748 ± 81.662** | 2865.380 ± 17.254 |\n\n*Table 1 — Performance across mass-relative changes for Hopper (best results in bold).*\n\n---\n\nWe hope that these answers address the reviewer’s concerns and that, given the additional comparison with DR-PolyGRAD, they will consider raising their final score for our paper.\n\n---\n\n[1] Pinto, Lerrel, et al. \"Robust adversarial reinforcement learning.\" International conference on machine learning. PMLR, 2017.\n\n[2] Ying, Chengyang, et al. \"Towards safe reinforcement learning via constraining conditional value-at-risk.\" arXiv preprint arXiv:2206.04436 (2022).\n\n[3] Rigter, Marc, Jun Yamada, and Ingmar Posner. \"World models via policy-guided trajectory diffusion.\" arXiv preprint arXiv:2312.08533 (2023).\n\n[4] Zeng, Siliang, et al. \"Maximum-likelihood inverse reinforcement learning with finite-time guarantees.\" Advances in Neural Information Processing Systems 35 (2022): 10122-10135.\n\n[5] Levine, Sergey. \"Reinforcement learning and control as probabilistic inference: Tutorial and review.\" arXiv preprint arXiv:1805.00909 (2018).\n\n[6] Janner, Michael, et al. \"Planning with diffusion for flexible behavior synthesis.\" arXiv preprint arXiv:2205.09991 (2022)."}, {"review_id": "I5BtlsRUhg", "reviewer": "Reviewer_pD4N", "summary": "This paper presents an adversarially robust reinforcement learning algorithm, termed AD-RRL, which leverages conditional value at risk (CVaR) optimization and diffusion models to generate worst-case trajectories. The paper is well-structured, and the problem being addressed is both interesting and significant. The proposed approach is theoretically sound, and the experimental results effectively demonstrate its efficacy. Overall, I believe this paper exceeds the acceptance threshold for NeurIPS.", "questions": "1. In the proposed method, classifier-guided diffusion is employed, built upon a classifier-free diffusion model $p_\\theta(\\tau_0)$. By design, diffusion models aim to replicate the historical data distribution within the dataset. Given this, how is it possible for the model to generate \"trajectories that are either rare in the current environment or originate from unexplored regions of the domain,\" as described in line 51 of the paper? Could the authors clarify this discrepancy?\n2. I can understand the approximation that $p_\\theta(\\tau_0\\in C_\\alpha|\\tau_0)\\approx exp(-c_0\\sum\\gamma^t r_t)$. Could authors provide more intuitions that why $p_\\theta(\\tau_i\\in C_\\alpha|\\tau_0)\\approx exp(-c_i\\sum\\gamma^t r_t^{(i)})$, and how to get $r_t^{(i)}$ for each $i$?\n3. In Algorithm 2, $c_i$ is compute with Eq.(18), but Eq.(18) is an inequality. As stated by the authors in line 243, it should instead be calculated using Eq. (19) and set as an equality. Is this correct?", "limitations": "The limitations of the paper are discussed by the authors.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n1. The motivation of the paper is clearly articulated, and the content is well-written. I found the paper enjoyable to read.\n2. Most of the mathematical notations are well-defined. Despite the large number of notations, they are presented in a clear and non-confusing manner.\n3. This paper is theoretically sound. All theoretical components are rigorously proven or appropriately cited when sourced from other works. Additionally, all assumptions are thoroughly explained.\n4. The experimental results convincingly demonstrate the effectiveness of the proposed method.\n\n\n**Weaknesses:**\n\nI did not observe any significant weaknesses in this paper.\n\n**Minor issues:**\n1. The abbreviation \"RL\" is introduced in line 16 but is repeated unnecessarily in line 83. Please consider removing the repetition.\n2. Please ensure proper capitalization of article titles in the references. For example, \"atari\" should be \"Atari,\" and \"Idql\" should be \"IDQL.\"\n3. In Eq. (18), the notation $r$ does not appear to represent reward information, as its definition is introduced in Appendix C. The authors should consider using a different notation to avoid potential misunderstanding and restate its definition explicitly in the main text for clarity.", "was_revised": true, "final_justification": "The authors’ rebuttal satisfactorily resolves the majority of my concerns. Consequently, I will maintain my positive assessment of this paper.", "rebuttal": "We thank the reviewer for the interesting questions and for the time dedicated to reading and evaluating our paper.  \n\n- We are pleased to see that the reviewer found the paper well-structured, clearly motivated, and enjoyable to read, and appreciated the significance and theoretical soundness of our contribution.  \n- We are especially grateful for the positive assessment of both the clarity of our mathematical formulation and the strength of our experimental results, as well as the overall recommendation that the paper exceeds the acceptance threshold for NeurIPS.  \n\nBelow, we address their questions in detail.\n\n---\n\n**Q1. By design, diffusion models aim to replicate the historical data distribution within the dataset. Given this, how is it possible for the model to generate “trajectories that are either rare in the current environment or originate from unexplored regions of the domain,” as described in line 51 of the paper?**\n\nWe thank the reviewer for highlighting this point. While diffusion models are indeed trained to replicate the distribution of historical trajectories, our approach modifies the sampling process via CVaR-based adversarial guidance, which biases the model toward trajectories associated with low return (i.e., areas where the agent tends to fail). Among low-return trajectories, we expect some to come from under-explored regions of the environment, since a low cumulative return may indicate that the agent has not learned how to behave optimally in these regions.\n\nWe agree that our original phrasing may have been ambiguous and will revise the sentence in line 51 to make this distinction clearer in the final version.\n\n---\n\n**Q2. I can understand the approximation that $p_\\theta(\\tau_0 \\in C_\\alpha \\mid \\tau_0) \\approx \\exp(-c_0 \\sum \\gamma^t r_t)$. Could the authors provide more intuition for why $p_\\theta(\\tau_i \\in C_\\alpha \\mid \\tau_i) \\approx \\exp(-c_i \\sum \\gamma^t r_t^{(i)})$, and how to obtain $r_t^{(i)}$ for each $i$?**\n\nThank you for the question; we are happy to clarify.\n\nThe reason we need this approximation at each step is as follows:\n\n- The goal is to be able to sample from $\\bar p_{ \\theta}(\\tau_0)$. From eq. (16), and equation below line 201, we have that $\\bar p_{ \\theta_0}(\\tau_0)\\propto p_{\\theta}(\\tau_0)p_\\theta(\\tau_0\\in C_\\alpha|\\tau_0)$.\n  - However, multiplying the distribution by the second term (a conditional probability) is usually costly and difficult. For diffusion models, the second term can be either treated as a small perturbation in each step in the diffusion process, or (often) obtained adding a multiplicative factor to the distribution in each diffusion step. \n  - Our approach consists in adding a multiplicative factor in each diffusion step (see also proof of eq. 16). Therefore, we use the same kind of approximation $p_\\theta(\\tau_i\\in C_\\alpha|\\tau_i)\\approx exp(-c_i\\sum\\gamma^t r_t^{(i)})$, so that we can exploit its smoothness for guiding the model toward adversarial rollouts. \n  - The intuition is that this form enables us to reweight trajectories at every diffusion step based on their cumulative return.\n\n- How we get $r_t^{(i)}$: note that for every generated trajectory $\\tau_i$ we predict the cumulative reward $\\sum_t \\gamma^tr_t^{(i)}$ using a separate network, that takes as input $\\tau_i$ and the diffusion step $i$. The implementation of the guidance procedure is described in section 5 of our paper (see, e.g., lines 270/271 onwards). More details on the implementation are presented in Appendix E.\n\n---\n\n**Q3. In Algorithm 2, $c_i$ is computed with Eq. (18), but Eq. (18) is an inequality. As stated in line 243, should $c_i$ instead be calculated using Eq. (19) and set as an equality?**\n\nThank you for pointing this out. You are correct — in Algorithm 2, $c_i$ should be computed using Eq. (19), where the value is set using an equality. However, we would like to note that using Eq. (18) would not change the computed value of $c_i$ in this setting, as the first term in the $\\min$ is always greater than the second term, as shown in Proposition 4.3.\n\nWe will fix this in the final version by using Eq. (19) with equality to compute $c_i$ in Algorithm 2.\n\n---\n\n**Q4. Minor issues and notation fixes**\n\nWe thank the reviewer for carefully reading the paper and noting these points. We will correct the notation and writing issues in the final version.\n\n---\n\nWe hope that, given our detailed answers to the reviewer's questions, they might consider raising the final score for our paper."}, {"review_id": "bJu9wudK3d", "reviewer": "Reviewer_2eCV", "summary": "The paper proposes a diffusion model to generates trajectories which are biased towards low returns. These low returns trajectories are then used to train the RL models yielding robust policies.", "questions": "***Q1) Could  the work be compared and contrasted with  [1] that generates trajectories with worst kernel with theoretical guarantees?***\n\n***Q2) The paper is well motivated and the approach looks good. Can we say anything about theoretical performance of the approach?***\n\n***Q3) Could you please summarize the key challenges of the paper?***\n\n***Q4) What are the limitations of the work?***\n\n***Q5) There are many works on uncertainty sets bounded by $L_p$ balls, revealing many structural properties such as 'adversary is rank one perturbation of the nominal kernel' [2], etc..  Can these properties be used to design better diffusion based algorithms for robust RL?***\n\n\n[2] @inproceedings{10.5555/3666122.3668721,\nauthor = {Kumar, Navdeep and Derman, Esther and Geist, Matthieu and Levy, Kfir and Mannor, Shie},\ntitle = {Policy gradient for rectangular robust Markov decision processes},\nyear = {2023},\npublisher = {Curran Associates Inc.},\naddress = {Red Hook, NY, USA},\nbooktitle = {Proceedings of the 37th International Conference on Neural Information Processing Systems},\narticleno = {2599},\nnumpages = {25},\nlocation = {New Orleans, LA, USA},\nseries = {NIPS '23}\n}\n\n\n\n\n\n\n[1]@inproceedings{\ngadot2024bring,\ntitle={Bring Your Own (Non-Robust) Algorithm to Solve Robust {MDP}s by Estimating The Worst Kernel},\nauthor={Uri Gadot and Kaixin Wang and Navdeep Kumar and Kfir Yehuda Levy and Shie Mannor},\nbooktitle={Forty-first International Conference on Machine Learning},\nyear={2024},\nurl={https://openreview.net/forum?id=UqoG0YRfQx}\n}", "limitations": "NA", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths: 1) The paper is well motivated. 2) The claims and approach seems intuitive and reasonable.\n\nWeakness: 1) The presentation of the paper can be improved, maybe using block diagrams, illustration of the approach etc... 2) See the questions.", "was_revised": true, "final_justification": "Authors response is convincing, hence I am increasing the score 4 but I don't find myself very familiar with the topic, hence I reduce my confidence to 2.", "rebuttal": "We thank the reviewer for the helpful comments and valuable questions. We’re pleased to hear that the paper was found to be well-motivated, with an approach that is both intuitive and reasonable. Below, we address each comment in turn.  \n\n---\n\n**Q1. Could the work be compared and contrasted with [1] that generates trajectories with worst kernel with theoretical guarantees?**\n\nThank you for pointing to this relevant work. The key idea of EWoK presented in [1] is to bias transitions toward next states with lower estimated value, thereby encouraging the policy to learn from pessimistic outcomes.  \n\nHowever, there are a number of distinctions to highlight:  \n\n1. EWoK assumes access to a generative model, allowing multiple samples to be drawn from any $(s,a)$ pair. AD-RRL does not.  \n   - This assumption is often impractical in real-world environments where the agent cannot reset and resample transitions on demand.  \n   - In contrast, AD-RRL learns a dynamics model and uses it to generate entire trajectories into the future, enabling adversarial sampling without requiring additional environment access during training.  \n\n2. While EWoK can offer strong theoretical guarantees due to its assumption of a known transition function, AD-RRL does not make such assumptions. We believe this leads to a more realistic framework for robust RL in environments with unknown or partially observable dynamics.  \n\nWe agree that comparing AD-RRL with EWoK would be valuable for future work, and we will strive to include such a comparison in the final version of the manuscript.  \n\n---\n\n**Q2. The paper is well motivated and the approach looks good. Can we say anything about theoretical performance of the approach?**\n\nThank you for the positive feedback and for this question.  \n\n- In the paper, we explore how the dual CVaR formulation connects to robustness. In particular, Proposition&nbsp;4.3 formally describes how to select the scalar $c_i$ such that the adversarial guidance at each diffusion step produces trajectories within the CVaR risk envelope.  \n- Choosing $c_i$ in this way ensures that the perturbations introduced by the adversarial objective remain valid from a risk-sensitive optimization perspective, and provides a principled foundation for the guided sampling process. Note that $c_i$ controls the sharpness of the approximation to the indicator function $p_\\theta(\\bar O(\\tau_i) = 1 \\mid \\tau_i) = 1_{\\tau_i \\in C_\\alpha}$ used in the CVaR formulation (see, e.g., Eq.&nbsp;12 and&nbsp;13).  \n  - Larger values of $c_i$ make the function sharper (closer to a hard threshold), while smaller values smooth the weighting more broadly.  \n- In practice, the values of $c_i$ increase as $i$ decreases. This means that as the model approaches the final denoised trajectory $\\tau_0$, the guidance becomes more selective, making the approximation sharper and closer to a hard threshold.  \n  - This trend results from the definition of $c_i$ (see Eq. 19) and the diffusion model’s cosine schedule.  \n  - We use the cosine schedule proposed by [3], which decreases the variance of the diffusion model as $i$ decreases. This reduces the denominator in Eq.&nbsp;19 and thus increases $c_i$ as $i$ decreases.  \n\nWhile we acknowledge that exploring other theoretical properties—such as regret bounds for AD-RRL—is an exciting direction, we consider this beyond the scope of the current paper.  \n\n---\n\n**Q3. Could you please summarize the key challenges of the paper?**\n\nThank you for the question. The key challenges we address in this work are:\n\n- **Learning robust policies in uncertain environments:** Robust RL aims to prepare agents for rare or worst-case scenarios, but identifying and training on such situations is non-trivial—especially when they are not encountered during standard exploration.  \n- **Generating informative, low-return trajectories:** Our challenge is to guide a diffusion model to generate *low-return (adversarial) trajectories* from the tail of the return distribution (i.e., within a CVaR risk envelope), while ensuring they remain realistic and actionable.  \n- **Providing theoretical guarantees on adversarial guidance:** We derive and prove conditions (Proposition&nbsp;4.3) under which the gradient guidance coefficients $c_i$ ensure that the perturbed samples remain within the CVaR risk envelope. This offers a principled foundation for adversarial sampling throughout the diffusion process.  \n- **Maintaining model fidelity while increasing robustness:** Training on adversarial samples can cause a mismatch with the data distribution. A key challenge is to ensure that the generated trajectories remain likely under the current policy, which we address by leveraging PolyGRAD’s policy-conditioned diffusion model.  \n\n---\n\n**Q4. What are the limitations of the work?**\n\nAs also discussed in Appendix&nbsp;G of our paper, we acknowledge the following key limitations:\n\n- **Computational overhead:** While AD-RRL significantly improves robustness and performance, it does so at the cost of increased computational overhead, due to both the learned diffusion model and the adversarial guidance mechanism. As discussed in our response to Reviewer TZ4u, we believe this cost is justified by the observed robustness and nominal performance gains, and could be mitigated in future work by incorporating more efficient generative architectures.\n- **Smoothness assumptions:** Our derivation employs a Gaussian approximation and requires computing gradients $\\nabla_{\\tau}Z(\\tau)$, which presupposes reasonably smooth rewards and state transitions. Tasks with non-smooth dynamics may violate this assumption, leading to less accurate guidance. Extending adversarial diffusion to such domains is an important direction for future work.\n- **Benchmark:** Finally, our experiments are currently limited to continuous control tasks in MuJoCo. While these are standard benchmarks, evaluating AD-RRL in higher-dimensional settings—such as vision-based environments (e.g., Atari)—would further clarify the method’s scalability. \n\n---\n\n**Q5. There are many works on uncertainty sets bounded by $L_p$ balls, revealing many structural properties such as “adversary is rank-one perturbation of the nominal kernel” [2], etc. Can these properties be used to design better diffusion-based algorithms for robust RL?**\n\nThank you for the insightful question. Indeed, many works in robust MDPs (e.g., [2]) define uncertainty sets using $L_p$-norm balls around nominal transition kernels, which leads to useful structural properties such as rectangularity, convexity, and tractable dual forms.  \n\nOur method generates samples at the trajectory level using a learned diffusion model, which makes it difficult to directly apply kernel-level uncertainty set formulations.  \n\nThat said, these structural insights could certainly inspire future improvements. For instance, incorporating constraints or priors on the learned dynamics model (e.g., Lipschitz continuity or rectangularity) may lead to better-calibrated adversarial sampling and stronger theoretical guarantees.  \n\n---\n\n**Q6. Clarity of presentation**\n\nThank you for suggesting a concrete way to improve the clarity of our paper. In the final version, we will include an additional illustration to provide a simplified overview of the AD-RRL pipeline.\n\n--\n\nWe hope that our detailed comparison with EWoK, along with the comprehensive responses to the reviewer’s questions, helps clarify the theoretical contributions, the challenges (both present and future), and the strengths of our work. We the reviewer might consider raising the final score when assessing their final evaluation.\n\n--\n\n[1] Gadot, Uri, et al. \"Bring your own (non-robust) algorithm to solve robust MDPs by estimating the worst kernel.\" Forty-first International Conference on Machine Learning. 2024.\n\n[2] Kumar, Navdeep, et al. \"Policy gradient for rectangular robust markov decision processes.\" Advances in Neural Information Processing Systems 36 (2023): 59477-59501.\n\n[3] Nichol, Alexander Quinn, and Prafulla Dhariwal. \"Improved denoising diffusion probabilistic models.\" International conference on machine learning. PMLR, 2021."}, {"review_id": "85EgFkVr4i", "reviewer": "Reviewer_TZ4u", "summary": "This paper proposes a new solution to robust reinforcement learning under environmental uncertainty. The objective is to learn a policy that maximizes the Conditional Value-at-Risk (CVaR) of trajectory returns—a problem previously explored in the literature. The authors build on PolyGRAD, a model-based reinforcement learning (RL) method that leverages a diffusion world model and policy-guided trajectory diffusion. The main contribution of the paper is the integration of adversarial guidance into PolyGRAD to generate the worst α-percentile trajectories under a given policy, which are then used to iteratively improve policy robustness. Experimental results show that the proposed method, AD-RRL, improves both the robustness and performance of policies across several MuJoCo tasks, outperforming PolyGRAD and other baseline approaches.", "questions": "1. The proposed AD-RRL method is much slower than the PPO and TRPO baselines. How can it be improved using a more efficient diffusion model? \n2. How does AD-RRL compare with CVaR-PPO in terms of robustness and computational efficiency?\n3. The proposed adversarial diffusion guidance is based on a smooth approximation of the classifier. How tight is this approximation? What are the limitations of this approach?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The proposed method demonstrates that a diffusion world model can be adversarially guided to incorporate robustness with theoretical guarantees, highlighting the potential of diffusion models in robust sequential decision-making.\n\nHowever, a major limitation of AD-RRL lies in its substantial computational demands. Trajectory diffusion is inherently slow, and the proposed adversarial guidance further increases computational complexity by requiring gradient evaluations with respect to entire trajectories. This makes the approach unlikely to scale effectively to environments with high-dimensional state spaces, such as Atari games with image-based inputs.\n\nAs shown in Table 5, training AD-RRL policies is over 100 times slower than training PPO or TRPO policies in the same environments, with even simple MuJoCo tasks taking more than three days to complete. This raises serious concerns about the practicality of the approach for real-world applications. While AD-RRL employs the same diffusion architecture as PolyGRAD, more efficient alternatives—such as latent diffusion models and EDM—have been recently proposed. To achieve a more practical solution, it is crucial to explore these more scalable methods and adapt them to trajectory diffusion.\n\nIn addition, the paper primarily compares AD-RRL to non-robust reinforcement learning baselines. However, robust RL has received increasing attention, including methods that optimize for CVaR objectives, such as CVaR-Proximal Policy Optimization (CPPO) [1]. It is expected that CPPO and similar methods incur significantly lower computational overhead compared with AD-RRL. A more meaningful comparison would include such robust RL baselines.\n\nTable 1 shows that AD-RRL outperforms PolyGRAD, PPO, and other baselines across several MuJoCo tasks, even in the training (non-adversarial) environments. This is somewhat surprising, as robust methods like AD-RRL are typically expected to trade off performance in nominal settings for improved robustness. Further explanation is needed here.\n\nFinally, the proposed adversarial diffusion guidance is based on a smooth approximation of the classifier. Although Proposition 4.3 shows that constants c_i can be chosen to ensure that the perturbed trajectories lie within the risk envelope, the tightness of this approximation is unclear. Including ablation studies or empirical analyses of the approximation's impact would strengthen the paper.\n\n[1] Chengyang Ying, Towards Safe Reinforcement Learning via Constraining Conditional Value at Risk, 2021.", "was_revised": true, "final_justification": "I've reviewed the rebuttal and the comments from the other reviewers. I acknowledge that the paper has merit and appreciate the authors' efforts in addressing my concerns, so I've increased my rating from 3 to 4. However, I still think it is crucial to provide a more systematic and fair evaluation of all the baselines in the revised paper. Also, while developing a new diffusion model may be beyond the scope of this paper, it is both reasonable and feasible to adapt the approach to other existing diffusion models, such as EDM and latent diffusion models, that are known to be more efficient.", "rebuttal": "We thank the reviewer for their detailed and thoughtful evaluation of our work. We are pleased that the reviewer recognized the novel integration of adversarial guidance into diffusion-based planning for robust reinforcement learning and appreciated our method’s ability to improve both robustness and performance across multiple tasks. We are also grateful for the reviewer’s recognition of the theoretical grounding of our approach and the broader potential of diffusion models in robust sequential decision-making. At the same time, we acknowledge and value the reviewer’s detailed comments, and we now provide responses to each point below.  \n\n---\n\n**Q1. The proposed AD-RRL method is much slower than the PPO and TRPO baselines. How can it be improved using a more efficient diffusion model?**\n\nThank you for raising this important practical point. Our method is approximately $1.5\\times$ more computationally expensive than PolyGRAD, which does not include adversarial guidance (note that AD-RRL requires an additional model).\n\n- As a model-based method, higher computational cost compared to model-free baselines is expected. However, we believe this overhead is justified by the significant gains in robustness and nominal performance, as demonstrated in Figures 1–3 and Table 1. AD-RRL consistently outperforms all baselines — including model-free and model-based approaches — in both nominal and perturbed environments, illustrating a favorable trade-off between computational resources and robustness.  \n- Importantly, the proposed adversarial conditioning method is architecture-agnostic and can be applied on top of more efficient diffusion models. This means that AD-RRL can directly benefit from advances in efficient generative modeling, such as latent diffusion or EDM (as mentioned by the reviewer).\n\nWe agree that reducing computational overhead is an important avenue for future research and consider integrating such efficient architectures a promising direction.  \n\n---\n\n**Q2. How does AD-RRL compare with CVaR-PPO in terms of robustness and computational efficiency?**\n\nWe thank the reviewer for suggesting this additional baseline. For a fair comparison, we trained CVaR-PPO (CPPO) for 1.5 M steps — consistent with AD-RRL and the other baselines — and present point-wise evaluations below.\n\n|  | HalfCheetah |  |\n| --- | --- | --- |\n| **Mass relative change** | **AD-RRL (Ours)** | **CPPO** |\n| 0.500 | **3061.153 ± 548.862** | −73.265 ± 120.968 |\n| 0.806 | **3898.686 ± 414.957** | 1555.600 ± 351.794 |\n| 1.112 | **4198.342 ± 388.928** | 2488.860 ± 453.486 |\n| 1.265 | **3716.105 ± 282.915** | 1823.855 ± 370.183 |\n| 1.725 | **2691.277 ± 279.085** | 1297.462 ± 344.145 |\n\n*Table 1 — Performance across mass-relative changes for HalfCheetah (best results in bold).*\n\n\n|  | Hopper |  |\n| --- | --- | --- |\n| **Mass relative change** | **AD-RRL (Ours)** | **CPPO** |\n| 0.506 | **2431.643 ± 543.152** | **2035.358 ± 350.707** |\n| 0.627 | **3062.959 ± 10.649** | 2617.749 ± 204.166 |\n| 0.749 | **3147.467 ± 12.367** | 2902.990 ± 77.162 |\n| 0.870 | **3215.967 ± 15.982** | 2603.035 ± 215.011 |\n| 1.161 | **3280.748 ± 81.662** | 2189.888 ± 379.293 |\n\n*Table 2 — Performance across mass-relative changes for Hopper (best results in bold).*\n\n|  | Walker |  |\n| --- | --- | --- |\n| **Mass relative change** | **AD-RRL (Ours)** | **CPPO** |\n| 0.506 | **3095.562 ± 467.163** | 1045.981 ± 171.053 |\n| 0.627 | **3293.219 ± 368.472** | 1366.939 ± 131.585 |\n| 0.749 | **3364.389 ± 363.136** | 1514.123 ± 124.977 |\n| 0.870 | **3556.677 ± 384.626** | 2033.778 ± 86.922 |\n| 1.161 | **3361.651 ± 644.848** | 831.669 ± 55.307 |\n\n*Table 3 — Performance across mass-relative changes for Walker (best results in bold).*\n\n\nIn terms of efficiency, one CPPO run takes roughly 30 minutes, compared to slightly under 4 days for AD-RRL. This is expected, since CPPO is a model-free method that does not rely on learned dynamics or trajectory guidance. However, as the tables show, AD-RRL vastly outperforms CPPO and is significantly more robust to variations in environment parameters. Its performance slowly degrades as the mass parameter shifts, whereas CPPO’s performance declines sharply. We believe this greatly increased robustness justifies the additional overhead, and again highlight the future potential of incorporating more efficient architectures into AD-RRL.  \n\n---\n\n**Q3. The proposed adversarial diffusion guidance is based on a smooth approximation of the classifier. How tight is this approximation? What are the limitations of this approach?**\n\nThank you for this interesting question.  \n\nOur approach builds on prior work (e.g., [1]), which casts control as probabilistic inference. There, the probability of optimality at each time step is approximated as $P(O_t = 1 \\mid s_t, a_t) = \\exp(r(s_t, a_t)),$ a smooth relaxation of the hard indicator $P(O_t = 1 \\mid s_t, a_t) = \\mathbf{1}_{(s_t, a_t) \\in C}$.\n\n- In our setting, a diffusion model generates entire trajectories rather than individual actions. We therefore model the optimality of an entire trajectory $\\tau$:  \n  - Let $O(\\tau)$ be a binary variable indicating whether a trajectory is optimal. We approximate $P \\bigl(O(\\tau)=1 \\mid \\tau\\bigr) = \\exp\\bigl(-c_0 \\sum_t \\gamma^t r_t\\bigr),$ where $c_0$ controls the sharpness of the approximation.  \n  - This smooth form allows gradient-based guidance to steer the diffusion model toward low-return (i.e., adversarial) trajectories.  \n  - Without such an approximation, adversarial guidance would be intractable in diffusion-based planning.  \n\n- The tightness of this approximation is governed by the scalar $c_0$: \n  - Larger values of $c_0$ make the approximation closer to a hard threshold; smaller values produce smoother weighting.  \n  - Similarly, at each diffusion step $i$, a corresponding $c_i$ governs the sharpness of the step-specific approximation.  \n\n- In practice, we observe that $c_i$ increases as $i$ decreases (i.e., as the model approaches the final, denoised trajectory $\\tau_0$), making the approximation sharper at later steps. This effect arises from Eq. (19) and from the cosine noise schedule of the diffusion model [2], which reduces variance as $i$ decreases — thus increasing $c_i$. \n\n- A limitation is that this smooth approximation may not sharply distinguish trajectories that lie near the CVaR threshold. Nevertheless, it provides a practical and differentiable surrogate for otherwise intractable objectives.\n\nWe acknowledge that exploring tighter or alternative approximations, and empirically validating their effects, would be a valuable direction for future work and will mention this in the final version.\n\n---\n\n**Q4. AD-RRL outperforms PolyGRAD, PPO, and other baselines across several MuJoCo tasks, even in the training (non-adversarial) environments.**\n\nIndeed, while AD-RRL is designed for robustness, it also performs strongly in the nominal setting. We believe this stems from two main factors:\n\n- First, the CVaR objective lower-bounds the expected return. Maximizing CVaR thus implicitly promotes strong average performance.\n\n- Second, adversarial guidance encourages the policy to learn from failure modes that might be underexplored in other approaches. This leads to more informative training signals and improved generalization, even under nominal dynamics.\n\n--\n\nWe hope that, given the detailed comparison with CVaR-PPO and our in-depth answers to the reviewer's questions, they might consider raising the final score for our paper.\n\n--\n\n[1] Levine, Sergey. \"Reinforcement learning and control as probabilistic inference: Tutorial and review.\" arXiv preprint arXiv:1805.00909 (2018).\n\n[2] Nichol, Alexander Quinn, and Prafulla Dhariwal. \"Improved denoising diffusion probabilistic models.\" International conference on machine learning. PMLR, 2021."}]

Author rebuttal:

Meta-review: This paper develops a new method for robust RL with environmental uncertainty.  The objective is to learn a policy that maximizes the Conditional Value-at-Risk (CVaR) of trajectory returns—a problem previously explored in the literature. The authors build on PolyGRAD, a model-based RL method that leverages a diffusion world model and policy-guided trajectory diffusion. The main contribution of the paper is the integration of adversarial guidance into PolyGRAD to generate the worst α-percentile low return trajectories under a given policy, which are then used to iteratively improve policy robustness. Experimental results show that the proposed method, called AD-RRL, improves both the robustness and performance of policies across several MuJoCo tasks, outperforming PolyGRAD and other baseline approaches.

The reviewers noted that the paper was novel and well structured, clear to read, and addresses a significant problem. Assumptions are clearly stated and theoretical aspects are rigorous. The approach is theoretically sound, and the experiments effectively demonstrate the value of the proposed method compared to baselines, and demonstrate robustness when the test environment has additional noise not encountered during training. Overall, the paper demonstrates that a diffusion world model can be adversarially guided to incorporate robustness with theoretical guarantees, highlighting the potential of diffusion models in robust sequential decision-making.

A key limitation is computational complexity and scaling.  The experiments were for relatively small-scale locomotion tasks, and exploring scalability to more complex, high-dimensional environments would strengthen the research.  The method trains guided diffusion models and computing return gradients at every diffusion step (over entire trajectories) can be computationally expensive.  So the method as-is may be unlikely to scale to problems with high-dimensional state spaces.  

The authors do compare to PPO and TRPO, and show that the proposed AD-RRL policy is more than 100x slower, so the proposed method as-is seems to be bordering on impractical.  Reviewers suggest considering recently proposed diffusion architecture alternatives such as latent diffusion models and EDM.  One reviewer suggests that “To achieve a more practical solution, it is crucial to explore these more scalable methods and adapt them to trajectory diffusion.”  This is considered reasonable and feasible.

It is also suggested by the reviewers that more comparisons are needed to existing robust RL methods as improved baselines, such as CVaR-Proximal Policy Optimization (CPPO) that has much lower complexity.  

Summary: The reviewers noted the significant strengths of the paper in clarity, novelty, theoretical underpinnings, and empirical performance.  The consensus was positive, while suggesting aspects that need further work, especially exploring alternative diffusion models with lower complexity, and comparisons with state of the art robust RL methods.
Decision: Accept (poster)