DR-SAC: Distributionally Robust Soft Actor-Critic for Reinforcement Learning under Uncertainty
Authors: Mingxuan Cui
Duo Zhou
Yuxuan Han
Grani A. Hanasusanto
Qiong Wang
Huan Zhang
Zhengyuan Zhou
Venue: ICLR 2026 Poster (2026)
Keywords: Distributionally Robust Optimization
Robust Reinforcement Learning
Primary area: reinforcement learning

Abstract: Deep reinforcement learning (RL) has achieved remarkable success, yet its deployment in real-world scenarios is often limited by vulnerability to environmental uncertainties. Distributionally robust RL (DR-RL) algorithms have been proposed to resolve this challenge, but existing approaches are largely restricted to value-based methods in tabular settings. In this work, we introduce Distributionally Robust Soft Actor-Critic (DR-SAC), the first actor–critic based DR-RL algorithm for offline learning in continuous action spaces. DR-SAC maximizes the entropy-regularized rewards against the worst possible transition models within an KL-divergence constrained uncertainty set. We derive the distributionally robust version of the soft policy iteration with a convergence guarantee and incorporate a generative modeling approach to estimate the unknown nominal transition models. Experiment results on five continuous RL tasks demonstrate our algorithm achieves up to $9.8\times$ higher average reward than the SAC baseline under common perturbations. Additionally, DR-SAC significantly improves computing efficiency and applicability to large-scale problems compared with existing DR-RL algorithms.

Published as a conference paper at ICLR 2026
DR-SAC: DISTRIBUTIONALLY ROBUST SOFT ACTOR- CRITIC FOR REINFORCEMENT LEARNING UNDER UN- CERTAINTY
Mingxuan Cui", Duo Zhou
Yuxuan Han', Grani A. Hanasusanto, Qiong Wang, Huan Zhang, Zhengyuan Zhou' University of Illinois Sebana-Champaigee
NNew York University.
ABSTRACT
Deep reinforcement learning (RL. has achieved remarkable success, yet its deploy- ment in real-world scenarios is often limited by vulnerability environmental uncertainties. Distributionally robust RL (DR-RL) algorithms have been pro- posed to resolve this challenge, but existing approaches are largely restricted value-based methods in tabular settings. In this work, we introduce Distribu- tionally Robust Soft Actor-Critic (DR-SAC), the first actor-critic based DR-RL algorithm for offline learning in continuous action spaces. DR-SAC maximizes the entropy-regularized rewards against the worst possible transition models within an KL-divergence constrained uncertainty set. We derive the distributionally robust version the soft policy iteration with a convergence guarantee and incorporate a generative modeling approach estimate the unknown nominal transition models. Experiment results on five continuous RL tasks demonstrate our algorithm achieves up 9.8x higher average reward than the SAC baseline under common pertur- bations. Additionally, DR-SAC significantly improves computing efficiency and applicability large-scale problems compared with existing DR-RL algorithms. Code is publicly available at http://gituub.com//ccuussss
INTRODUCTION
The field of deep reinforcement learning (DRL) has witnessed remarkable progress, enabling agents to learn complex behaviors across a wide range of domains, from game playing to robotic con- trol (Arulkumaran et al., 2017; Francois-Lavet et al., 2018; Chen et al., 2024). In particular, offline deep reinforcement learning, which learns policies from fixed datasets without additional environment interaction, has gained increasing attention due to its practical relevance. By eliminating online exploration, this paradigm improves safety guarantees and data efficiency. Among these methods, Soft Actor-Critic (SAC; Haarooja al. (2018a;b) is principled algorithm based entropy regular- ized reinforcement learning, commonly known as the soft value framework. This maximum entropy formulation grounded in theoretical foundations (Ziebart, 2010) and has been applied to various contexts, including stochastic control (Todorov, 2008; Rawlik et al., 2012; Messaoud et al., 2024) and inverse reinforcement learning (Ziebart al., 2008: Zhou et al., 2018).
However, a persistent challenge limiting the deployment of offline deep RL in real-world systems is the sensitivity of learned policies environmental uncertainties (Whittle, 1981; Enders et al., 2024). Policies trained in nominal environment often exhibit significant performance degradation when deployed under slightly different one. This model mismatch may arise from uncertain transition and reward functions, observation and actuator noise, parameter variations, or adversarial perturbations.
Distributionally robust reinforcement learning (DR-RL) addresses this challenge by optimizing policies against the worst-case scenario. Instead of assuming a single Markov Decision Process (MDP), DR-RL adopts a Robust Markov Decision Process (RMDP) framework, which includes a set of MDPs defined by uncertainty set of distributions around the nominal onc.
'Equal Contributaion. Corresponding to duozhou20i i edu

---

Published as a conference paper at ICLR 2026
Although both value-based (Liu et al., 2022: Lu et al., 2024) and policy-gradient gradient (Wang & Zou, 2022; Kumar et al., 2023) DR-RL algorithms have been proposed, most existing work focuses on performance guarantees and sample complexity in the tabular setting and cannot be deployed in continuous environments. The only notable exception is Robust Fitted Q-Iteration (RFQI; Panaganti al. (2022)). However, several fundamental research gaps remain. First, RFQI considers uncertainty sets defined by the Total Variation (TV) distance, which is analytically convenient due to its piecewise linear dual formulation, but does not extend to other divergence measures. Second, its non-robust baseline, Fitted Q-Iteration (FQI; Ernst et al. (2005)), is value-based and suffers from critical limitations, including deterministic learned policies, low applicability to nigh-dimensional action spaces and high sensitivity to errors in the state-action function (Degris et al., 2012). In contrast, actor- critic algorithms combine low-variance value estimation with scalable policy optimization, making them preferred in continuous control benchmarks and practical applications (Konda & Tsitsiklis, 1999; Grondman et al., 2012). Despite their empirical success, no distributionally robust counterpart has been developed. This gap motivates our Distributionally Robust Soft Actor-Critt (DR-SAC) the first actor-critic-bant -assed DR-R RL-RL algorithm for offline learning in continuous action spaces.
In this work, we assume access only dataset collected from the training environment, and the transition distribution in the deployment environment lies within an uncertainty set, defined as a Kullback-Leible (KL) divergence ball centered at the nominal distribution. The goal is learn a policy that maximizes the soft value function under the worst possible distributions. The main contributions of this work are:
Distributionally Robust Maximum Entropy Framework. We formulate the maximum entropy learning framework under transition uncertainties modeled by dedivvrgence-conntraiinn am- biguity sets. Within this framework, we derive distributionally robust soft policy iteration with convergence guarantees and develop distributionally robust counterpart of SAC.
Scalable Reformulation via Functional Inner Optimization. We exploit the interchange property to reformulate the per-(8, a) scalar inner optimization problems into shared functional optimization. This reformulation eliminates the dependence state-actin action space dimensionality, enables application to continuous action space and saves training time by over 80.0% compared to the existing DR-RL algorithm RFQI.
Generative Modeling for KL-Based Robustness. We introduce generative models (VAEs) to estimate unknown nominal transition distributions and construct empirical measures while preserving robustness. This design resolves the double-sampling issue inherent in the non-linear KL-divergence dual formulation and enables distributionally robust soft policy learning in offline and continuous control tasks. Extensive experiments across five environments demonstrate that DR-SAC achieves up to 9.8x higher average reward than the SAC bascline under perturbations.
1.1 RELATED WORKS
Robust RL. The RMDP and Robust Dynamic Programming method were first introduced in (lyen- gar, 2005; Nilim & El Ghaoui, 2005) and have been widely studied in planning settings (Xu & Mannor, 2010; Wiesemann al., 2013; Yu & Xu, 2015). Beyond classical robust formulations, various approaches have been developed to address uncertainty in reinforcement learning, including soft-robustness (Derman et al., 2018; Lobo et al., 2020; Park et al., 2024), risk sensitivity (TTamr et al., 2015; Pan et al., 2019; Singh et al., 2020: Queeney & Benosman, 2023), and adversarial training (Pinto al., 2017: Zhang et al., 2020; Cheng al., 2022). In recent years, many distributionally robust RL algorithms have been proposed with provable guarantees in the tabular setting. These include model-free algorithms based on Q-learning (Wang et al., 2023; 2024; Liang et al.. 2024) and model-based algorithms extending value iteration (Zhou et al., 2021; Panaganti & Kalathil, 2022; Xu et al., 2023; Ma et al., 2023; Liu & Xu, 2024). However, these algorithms are not applicable to continuous action space environments.
Model-Free Algorithms for Distributionally Robust RL. In DR-RL, the nominal transition distributions typically appear in the optimization objective but are unknown in practice. To address this challenge, some model-free algorithms assume access to a simulator generating i.i.d. samples from the nominal environment (Liu et al., 2022: Zhou et al., 2023; Ramesh al., 2024), which violates the offline setting. Other algorithms compute empirical transition frequencies from offline datasets (Derman & Mannor, 2020; Clavier et al., 2023; Shi & Chi, 2024), which applicable
2

---

Published as a conference paper at ICLR 2026
in continuous spaces, Lastly, Empirical Risk Minimization (ERM) has also been used to estimate the robust objectives under special structures (Mankowitz et al., 2019; Kordabad et al., 2022).
VAE in Offline RI.
Variational Autoencoders (VAEs) have been widely used in non-robust offline RL. A common application is to estimate the behavior policy from offline data, and impose policy constraints pessimistic value regularization to mitigate distributional shift (Fujimoto et al., 2019; Wei et al., 2021; Xu et al., 2022: Lyu et al., 2022). VAEs have also been used for state reconstruction and representation learning in RL (Van Hoof al., 2016). To the best of our knowledge, this work is the first to incorporate VAE models in DR-RL algorithm to estimate nominal transition distributions and generate synthetic samples in the absence of a simulator.
is worth noting that Smirnova et al. (2019) proposed algorithm with a similar name. However, the problem formulation is completely different from ours and most DR-RL literature. Their framework accounts for estimation error in the evaluation step and employs KL divergence to constrain deviations from the behavior policy within single MDP rather than RMDP.
2
FORMULATION
2.1 NOTATION AND BASICS OF SOFT ACTOR-CRITIC
standard framework for reinforcement learning is the discounted Markov Decision Process, for- mally defined as tuple M = (S, A, R, P),), where S and A denote the state and action spaces, respectively, both assumed to be continuous this work. The random reward function modeled as mapping R S x A P((0, Rmax)), where P((0, Rmax)) is the set of random variables supported on [0, R.ax] The transition distribution is denoted by P:
A + ASS), where A(S) is the set of probability measures over S, and y [0, 1) is the discount factor. We denote random reward and s' as the next state drawn from the transition distribution Ps.a = P(- s, s,a). A policy S ((AA) represents the conditional probability over actions given state. We consider the stochastic stationary policy class, denoted by II. The entropy of a stochastic policy π at state defined as Hs) E |- llgg (a[s)], measuring the randomness of action. The set of integers from 1 to n is denoted as [m]]
R(s, a) the
In maximum entropy RL, the objective comprises the cumulative discounted reward and entropy regularization term to encourage exploration. Specifically, given an MDP M, the soft value function under policy is defines as
(
Σ xt 1
(r. (1 ((((T(()))))))))))))))))())))))) T, =
The temperature 2 0 controls the trade-off between reward maximization and policy stochasticity. The optimal soft value and optimal policy are defined as
max V TM
argmax VM-
Hell
=Ell
Similarly, the soft state-action value function (soft Q-function) under policy æ is defined
a)( E r
(r+ f (o(o))(((()))))
π,, 81 = 5
For any mapping Q: x A + R. Haarnoja et al. (2018a) defined soft Bellman operator as {s) = B[r] + Ep...: [Q (8', a') log (a' |s)]]
(4)
The Soft Actor-Critic (SAC) algorithm updates the policy through soft policy iteration, which is guaranteed to converge in the tabular case. In each iteration, the soft Bellman operator 7 is applied update the estimation of the soft Q-function under the current policy æ. The policy is then updated by minimizing the KL divergence between the candidate policy and the exponential of the soft --ffucctio:
exp (a (' / Z(s)) k- 0,1,
Tk+1 = argmin DKL ((. (((1s))
s)
(5)
Tell
where DRL || Q) = Ep log
Q(z) denotes the KL divergence and the function 2(-) is the normalizing function ensuring that the exponential term defines valid probability distribution.
3

---

Published as a conference paper at ICLR 2026
2.2 ROBUST MARKOV DECISION PROCESS
In real-world RL tasks, the transition distribution P and reward function R in the deployment environment may differ from the environment which the model is trained in the offline dataset is collected from. Such environmental shifts motivate the study of the Robust Markov Decision Process framework and the goal of learning policies robust to distributional perturbations. Since the analysis and algorithm design for reward perturbations are similar, we assumee the reward function R unchanged and focus only on uncertainty the transition distributions.
The RMDP framework is denoted as MM = (S, A, R, P(5), 7). We consider transition distributions perturbed within KL-divergence ball. Specifically, let po {p2.. (s.a)Exx be the nominal transition distributions. For each state-action pair (s, a) centered at P2..
and 6 O, we define the KL ball
(o)a {Ps,a A(S) (ps 113..) S 0}
(6)
The ambiguity set P6) is defined as the Cartesian product P..a (8) for pairs (s,a) A. which belongs to the (s, a) rectanguul set (Wiesemann et al., 2013).
Under the RMDP framework, the goal is to optimize performance under the worst-case transition model within the ambiguity set. Given Mas the distributionally robust soft value function under policy æ is defined
Σ Ext En-i(r:+ (o++a-((T(s))))) H(((s)))) 8, t=1
(+) = inf Ep DEP
Similarly, the distributionally robust soft Q-function given by
(s, inf Ep Σ Σ + a ((((())))) T, =,0
(8)
PEP(
The distributionally robust optimal value and optimal policy are defined as:
Vine (8)
max VM, (s) and TiM argmax VM, (8). well
#Ell
3 ALGORITHM: DISTRIBUTIONALUS ROBUST SOFT ACTOR-CRITT
In this section, we develop the Distributionally Robust Soft Actor-Critic algorithm. We first derive the distributionally robust soft policy iteration and establish its convergence the optimal policy. To improve computational efficiency, we develop a scalable implementation by replacing the per-(s, a) scalar inner optimization with shared parametric optimization. Lastly, to handle the unknown nominal distribution in offline settings, we incorporate generative modeling to construct the empirical transition measures.
Assumption 3.1. To ensure that the policy entropy H(((s)) E log T (a]s)) [)] is bounded, we assume |A| < co.
Remark 3.2. Assumption 3.1 inherited from the non-robust baseline SAC (Haarnoja et al., 2018a), which establishes theoretical guarantees in the tabular setting while being empirically used in continuous control benchmarks. Our work extends the performance properties of SAC to the DR-RL. framework. In Section 3.3, design practical algorithm in continuous action spaces.
3.1 DISTRIBUTIONALLY ROBUST SOFT POLICY ITERATION
We begin with the distributionally robust soft policy iteration, which iterates between DR soft policy evaluation and DR soft policy improvement. We further show that this iteration guaranteed to converge to the DR optimal policy.
DR soft policy evaluation. For fixed policy ,, the DR soft Q-function is estimated by iteratively applying the distributionally robust soft Bellman operator, which considers the worst possible transition distribution within the uncertainty set. For any bounded mapping Q S A + R, the distributionally robust soft Bellman operator is defined as:
4

---

Published as a conference paper at ICLR 2026
(s) EEE[ +
P (6) { [Q(s', .')
inf
log r(a' |s)]}
(10)
Following Iyengar (2005); Xu & Mannor (2010), the DR soft Q-function can be computed via distributionally robust dynamic programming. and Qú, is fixed point of TT". However, operator TT'' generally intractable because it requires solving an infnite-dimensione the transition distributions. To address this, we apply the strong duality for worst-case expectations over KL-divergence ball and derive a equivalent dual formulation.
optimization problem over
Proposition 3.3 (Dual Formulation of the Distributionally Robust Soft Bellman Operator). Suppose a(s,a) bounded, the distributionally robust soft Bellman operalor in Equation (10) can be reformulated into:
exp (- [(5) 3
()) -86
QTs - E(r] 7 -Blog
Epp...
(11)
020
where
V(s) Eaw 1Q(s, a)
log |s]]
(12)
Derivation is provided in Appendix B.1. Importantly, the dual form depends only the nominal transition distribution Po. instead an infinite number of distributions in the uncertainty set P(0). Also, the inner optimization problem is reduced one-dimensional problem the scalar B, rather than infinite- dimensiooamn distributions. Using this tractable dual form operator, the DR soft be computed by iteratively applying T/"
Q-value for a fixed policy
Proposition 3.4 (Distributionally Robust Soft Policy Evaluation). For any fixed policy E II, starting from any bounded mapping Qº S x A - R. define a sequence {Q* by iteratively applying distributionally robust soft Bellman operator: Q++1 Tg Q*. This sequence converges to the DR soft Q-value Q"M,, as + oc.
The proof shows that the operator T is 7-contraction mapping, with details in Appendix B.2.
DR soft policy improvement. The distributionally robust soft policy improvement step similar to that in SAC, with DR soft Q-value QM, replacing its non-robust counterpart. The new policy in each update defined as
((( (((15)) exp (-
(z) k 0,
Tk+1 = argmin DKL xEIl
(13)
With the above policy updating rule, mi has a non-decreasing value with respect to the DR soft Q- function. This extends the non-robust soft policy improvement to the uncertain transition distribution case.
Proposition 3.5 (Distributionally Robust Soft Policy Improvement). Suppose |A| < 0o, let Tht be the solution the optimization problem above. Then
Q Th+t (s, a) Qhi, (s, V(s,a) AxA.
(14)
The proof is provided in Appendix B.3. The DR soft policy iteration algorithm alternates DR soft policy evaluation and DR soft policy improvement. The following theorem shows convergence to the DR optimal policy, with proof in Appendix B.4.
Theorem 3.6 (Distributionally Robust Soft Policy Iteration). Suppose |A| < co, starting from any policy T E II, the policy sequence { k converges to the optimal policy n as l - Cc.
Key Challenges. Although DR soft policy iteration converges to the optimal policy the tabular setting, several challenges arise in extending it to continuous action space and offline setting: 1) the DR soft policy evaluation step is computationally expensive at scale due to the per-(s, a) inner optimization over 8; 2) the nominal transition distribution p2 is typically unknown in offline RL tasks, and 3) exact DR soft policy iteration is not directly applicable in continuous action space. We will resolve these issues step by step in the rest of this section.
5

---

Published as a conference paper at ICLR 2026
3.2 SOLVING DUAL OPTIMIZATION USING GENERATIVE MODEL
In offline reinforcement learning, the goal is to learn the optimal policy from pre-collected dataset D [((,, di, ri, &{) where (82, 0z) ft, with μ denoting the data generation distribution determined by the behavior policy, r; = R8 (4) and 8 ~ ( 86, 0.). In this section, we address the key computational and modeling challenges of DR soft policy iteration. Specifically, we (i) develop a scalable reformulation of the dual Bellman operator by approximating the per-(s, a) scalar optimization into shared optimization problem over a function space, and (ii) introduce a generative modeling scheme to estimate unknown nominal transition distributions.
Dual Reformulation via Functional Optimization. In DR soft policy evaluation, the Bellman operator 7y" is iteratively applied to the Q-function. From the dual form operator in (11), each application requires solving an optimization problem over a scalar 3 > O. While this optimization is tractable, it must be solved separately for every (s, a) pair, which becomes computationally expensive for large-scale problems. To improve training efficiency, we convert a group of scalar optimization problems into single functional optimization problem. Intuitively, instead of solving for the optimal 3* separately at each state-action pair, we learn function that approximates these optimal values jointly across the dataset. This can be achieved by the interchange of minimization and integration in decomposable spaces (Rockafellar & Wets, 2009).
Formally, consider the probability space (S A, E(S x A), and let L' A, A), p)) be the set of absolutely integrable functions on that space, abbreviated as Lt For any 5 > 0 and value function V > [0, (Rmax + a log]/A))/(1 - r)]. define
exp ( (- 33 )) ,38.
=- -3 log
E.g..
(15)
Assume |A| < co, define the admissible function set
g== ge L g(s,a) E
(o. Rauax alllg
V(s,a) E e E
(16)
(1 7)8
Proposition 3.7 (Interchange of Minimization and Expectation). For any 8>0 0.
E (s,a)~D
sup f ((s,)((s))))) a), $20
= sup B( [ ((s.0))..(..)))
(17)
gEg
The proof is provided Appendix B.5. The results allow us to solve single optimization problem over the function g instead of the [D] scalar optimization problems. In practice, . is learned jointly across the dataset, substantially reducing training time while preserving robustness.
Based on Proposition 3.7, we define a finctional DR soft Bellman operator by replacing the scalar / with a function g(s. a) and removing the inner optimization. For any function E 9 and mapping Q S x A + [0, Rmmx + a (ogg 7)1, let
Th, ((, [r] + 7 ((),a))((ss)))
g a) log E,g.. exp (-
(((,)) (-
)1) g(s. s(s.a)o).
(18)
-E +
where (s) Egsw [Q(s, a) - a- log T(a |s]]
Generative Modeling for Nominal Distributions. In offline RL, we assume that the nominal distributions 7d are unknown, and no simulator available to generate additional samples. Under the KL-constrained uncertainty set, the dual optimization problem in the DR soft Bellman operator (both original and functional) is non-linear, Directly estimating the required expectations from the offline dataset D suffers from the double-sampling issue, making empirical risk minimization inapplicable. detailed discussion is provided in Appendix A.
To enable practical implementation of operator T5 in continuous space, incorporate gener- ative model to estimate the nominal transition distributions. To be specific, we train a variational autoencoder (VAE) model collected data (s, a, s') D to learn transition po The trained VAE

---

Published as a conference paper at ICLR 2026
generates next-state samples ( and construct empirical measure P2.a: For any function h S R, the empirical expectation defined as Exsppp [h(s')] = Ei= h(3)). We define the empirical DR soft Bellman operator as
The ((s,) = 7 a((s,
(19)
where
(-V(6))
V(s') B )) 36.
8)) - log
Epp..
exp
(20)
3.3 DISTRIBUTIONALLY ROBUST SOFT ACTOR-CRITIE
We now extend the action space to the continuous setting and use neural networks to approximate the DR soft value function and policy. We consider RMDP MM and omit subscripts in V and Q. Our algorithm includes value network (s). Q-networks Qa a) and stochastic policy To(a s), parametrized by vb, 0 and Φ. t and are the target network parameters to stabilize training (Mnih et al., 2015). Let &o be the parameters of VAE model. We use parametrized neural network Gy to approximate the function set G.
The core idea of our DR-SAC algorithm is to alternate between empirical DR soft policy evaluation and DR soft policy improvement. The loss Q-network
JDR (0) (e,,,)
2 (Qo(s, a) Tog Qo(s,
(21)
where
9" argsup (G, (((ss))).0s..))) gEg,
(22)
The loss functions of vh, ó and a are the same as SAC, and the loss function of is the standard VAE evidence lower bound (ELBO) loss. We adopt the SAC-vl algorithm (Haarnoja et al., 2018a) with explicit V-function, as we observe empirically that including V-network reduces sensitivity to the behavior policy underlying the offline dataset. Ablation studies are presented in Appendix C.3.3. To mitigate overestimation bias, we employ multiple Q-functions Qa, (i E [m]), train independently, and use the minimum in updating the value and policy networks. This has been shown to outperform the clipped double Q-learning (n = 2) in offline RL (An et al., 2021). We formally present the Distributionally Robust Soft Actor-Critic in Algorithm 1. Detailed loss functions are provided in Appendix A.2. We also derived regret bound in Appendix D.
Algorithm Distributionally Robust Soft Critic (DR-SAC)
Require: Offline dataset D {s a, Fs. sf) }. V-function network weights v). Q-function
network weights 0,, E [n]. policy network weights Φ, transition VAE network weights y, weight for moving average, function class Gn
1: J 0 + 02 for i [ri]
Initialize target network weights for soft update
2: for each gradient step do
3: + X V (v)
b Update transition VAE weights
4: Generate samples m from VAE, form empirical measures Ph.
5: Compute optimal function g" according to (22)
6:
vb - Φb - A V (b))
Update V-function weights
7:
0, + 0, - \Q Ve, Ve (D.)
(0.) for i E [n]
Update Q-function weights
8:
ò + Φ A, V (d)
D Update policy weights
9:
a a - Xa (a)
> Adjust temperature
+ T + T) 0. =0, (1 7)0, for [n]]
Update target network weights
11: end for
Ensure: à
7

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
The goal of our experiments is to demonstrate the robustness of DR-SAC under environmental uncertainties in offline RL tasks. We measure performance by the average episode rewards under different perturbations, and compare DR-SAC with non-robust baselines and RFQI. To the best of our knowledge, RFQI the only offline DR-RL algorithm applicable to continuous action spaces. Moreover, extensive ablation studies demonstrate that VAE-based DR-SAC with functional approximation achieves the best trade-off between robustness and computational efficiency.
4.1 SETTINGS
We implement SAC and DR-SAC based on the multiple critic version SAC-N (An et al., 2021). Besides RFQI, we also compare DR-SAC with Fitted Q-Iteration (FQI), Deep Deterministic Policy Gradient (DDPG; Lillicrap et al. (2015)). and Conservative Q-Learing (CQL; Kumar et al. (2020)).
We consider Pendulum, Cartpole, Lunarlander, Reacher and HalfCheetah environments from Gymnasium (Towers et al., 2024). For Cartpole, we use the continuous action space version in Mehta et al. (2021). For LunarLander, we also adopt continuous action space setting. All algorithms are trained the nominal environment and evaluated under various perturbations. In our experiments, perturbations include environment parameter changes, random noise added to observed states and random actuator noise applied to actions. Detailed experimental settings are provided in Appendix C.1
4.2
PERFORMANCE ANALYSIS
-10
SAC
DisAACTDurs)
DR SACIOUrS)
RFOI
-00
DORG
FGN
282
50
-309
-8
SAC
DRSSACOOuss
-50
FQI
100
30
DDPG
26 Percentage Change Value
1o
-0
-3
Prob. Picking Random Action
Percontage Change from Noninal
(a) Length Perturbation Pendulum
(b) Action Perturbation Cartpole
(c) Engine Perturbation Lunarl Lnder
SAC
SAC
3AC
D8-GACIOun))
09 G0u
POI
6430
-16
4220
152
3000
-222
-13
3600
Standard Deviation of Obse vetion Noise
Damping
Percentage Change from Nominnal veluv
(d) Observation Perturbation
(c) Damping Perturhation Reacher
(1) Back Damping Perturbation HalfCheetah
Reacher
Figure 1: Robustness performance comparison. The curves show the average reward over 50 episodes, with shaded regions indicating +0.55 standard deviation (see Table 4 for standard deviations in sub-figure (e)). Environmental perturbations include parameter shifts, state and actuator noise.
This section reports selected experiment results. Additional experiments are provided in Appendix C.2. In the Pendulum environment, we evaluate robustness against parameter length perturbations. RFQI is omitted due to poor performance in the nominal environment. In Figure I (a)), DR-SAC performance outperforms SAC by 35% when the length changes by 20%. In the Cartpole environment, the actuator is perturbed by taking random actions with different probabilities. DR-SAC consistently outperforms RFQI, especially when the probability of random action is below 50%. In the LunarLander envi- ronment, we jointly perturb main engine disturbance. DR-SAC shows consistently robust performance compared to other algorithms. In
engine_power ppower and dide_eeniie__ee power to model engine power
8

---

Published as a conference paper at ICLR 2026
Figure I(c), under 20% perturbation, DR-SAC achieves average reward of 240 while the rewards of all other algorithms drop under 180. Moreover, DR-SAC achieves 9.8x higher reward than SAC when parameters reduce by 30%.
To further evaluate robustness in more complex environments, we conduct experiments in HalfCheetah and Reacher from MuloCo (Todorov et al., 2012). In the Reacher environment, consider Gaussian observation noise and parameter joint damping perturbation. In Figure (d), DR-SAC shows the best performance across all observation noise levels. In Figure (()), DR-SAC clearly outperforms SAC. In the HalfCheetah environment, we present the experiments of SAC and DR-SAC due to the poor performance of FQI and RFQI. When back damping varies within 50%, DR-SAC maintains stable average reward of over 6300, while the average reward of SAC keeps decreasing below 5950.
Discussion on FQI Failure. It is worth noting that FQI and RFQI perform poorly even in unper- turbed Pendulum and HallCheetah environments. One possible reason is that offline RL algorithms exhibit different sensitivities to dataset distributions. SAC works well when the dataset provides broad coverage over the action space (Kumar et al., 2019). In contrast, FQI is implemented Cateh-Constrained Deep Q-learning (BCQ; Fujimoto et al. (2019)), which restricts the agent to selecting actions close to the behavior policy. This conflicts with the gpsilon-greedy-methie generation process in our experiments. One primary goal of our experiments to demonstrate that DR-SAC improves robustness over SAC under common environmental perturbations. Investigating the sensitivity of offline RL algorithms to dataset distribution is out of the scope of this work.
data
4.3 ABLATION STUDIES
To better understand the design choices DR-SAC, we conduct a series of ablation studies focusing computational efficiency and generative model selection. Specifically, examine (i) the impact of functional approximation on robustness and training time, (ii) the optimization efficiency compared to RFQI, and (iii) the sensitivity of DR-SAC to different generative modeling choices. These studies aim to validate that the proposed design achieves a favorable trade-off between robustness and efficiency.
4.3.1 TRAINING EFFICIENCY OF DR-SAC.
Comparison with Accurate Bellman Operator Section 3.2, we approximate the Bellman operator 75" with T3", to avoid solving optimization problems for each (s, a) pair. To evaluate the impact this approximation, we additionally implement an algorithm using the accurate operator. We refer to this variant as DR-SAC-Accuree section. As shown in Figure 2, performance while requiring less than 2% training time. These results validate that functional approximation significantly improves computational efficiency without sacrificing robustness. Variant algorithm details and training time are provided in Appendix C.3.1.
and denote Algorithm as achieves comparable and even better robustness
in this
-100
200
-300
500
-500
SAC
-000
SAC
-700
DR-SAC-Functionel DR-SAC-Accurate
-800
DR-SAC-Accurate
-700
Percentage Change from Nominal Value
Percentage Change from Nominal Value
(a) Mass Perturbation. Pendulum
(b) Length Perturbation. Pendulum
Figure 2: Ffficiency-Rohustnees Robustness Trade-Off. These figures show that DR-SAC with functional
approximation maintains robustness.

---

Published as a conference paper at ICLR 2026
Comparison with RFQI In Section 4.2, RFQI achieves comparable robustness to DR-SAC in certain environments. However, DR-SAC demonstrates substantial improvement in training efficiency. Table 1 shows that RFQI requires up 23.2x the training time of DR-SAC. Compared with their non-robust counterparts, RFQI requires at least 11.3x the training time of FQI, while DR-SAC training time is at most 2.6x that of SAC.
Further analysis suggests that this efficiency gap primarily stems from optimization complexity. Although RIQI involves a similar functional approximation step as Equation (22), it requires 1000 gradient descent (GD) steps in each update to find the optimal function. In contrast, DR-SAC requires only 5 GD steps to achieve comparable performance. Empirically, reducing the number of GD steps in RFQI leads to severe performance drop. even in unperturbed environments (results see Appendix C.3.1), )). This indicates that the structure of RFQI's loss function inherently results in slower convergence and more optimization steps.
Table Training time in different environments (minute)
Env
SAC
DR- PR-SAC
FQI
RFQI
Carppole LunarLander 16
4
7
93
36
17
238
Reacher
13
32
14
159
4.3.2 SELECTION OF GENERATIVE MODEL
Although the VAE models inevitably introduce estimation error when constructing empirical transition measures, we find that DR-SAC is largely insensitive to such modeling choices. As shown in Figure 3(a), varying the VAE latent dimension between and 20 Pendulum does not degrade robustness, and DR-SAC consistently outperforms the SAC baseline.
To demonstrate the choice VAE over other generative models, we implemented diffusion models and normalizing flows as alternative generative models in DR-SAC. Ablation studies in Figure 3(b) and (c) show that flow-based models exhibit unstable performance, even in the unperturbed Pendulum environment. The diffusion-biis based model achieves comparable robustness but requires at least 4.5x the training time the VAE-based model. We emphasize that the VAE not necessarily the optimal choice in all settings. Our DR-SAC algorithm can incorporate alternative transition models depending on the task and computational constraints.
DR DRISKCANA, Dasi
-109
300
489
Sic
DRSACI445, casl
-60
D8 ANC PR1
FAC DA-SEEIW,, Dett
Parcentage Change from Nominal Value
Parcentage Change from Nominal Value (8)
Standard Deviation of dbservation Noise
35
(a) Mass Perturbation. Pendulum
(b) Length Perturbation. Pendulum
(c) Observation Noise. Cartpole
Figure 3: Generative Model Selection. Subfigure (a) shows that VAF-based DR-SAC is insensitive to the latent dimension. Subfigures (b) and (c) compare robustness performance with diffusion and flow- models.
CONCLUSIONS
We propose DR-SAC, the first actor-critic-based DR-RL algorithm for offline learning in contin- uous action spaces. Our framework establishes distributionally robust soft policy iteration with convergence guarantees, reduces training time by over 80.0% compared to RFQI through functional optimization, and resolves the houble-sampling issue in estimating nominal distributions via genera- tive modeling. Experimental results show that DR-SAC attains up to 9.8x higher reward than SAC under perturbations, highlighting both robustness and efficiency in practical offline RL tasks.
10

---

Published as a conference paper at ICLR 2026
Acknowledgement. Grani A. Hanasusanto is supported in part by NSF (CCF2343869 and ECCS- 2404413). Huan Zhang is supported in part by the AI2050 program at Schmidt Sciences (A12050 Early Career Fellowship) and NSF (II-22331677.. Zhengyuan Zhou is supported in part by NSF ONR-13983263. and the 2027 New York University Center for
CCCF-2312205, (CCSS22419564 Global Economy and Business grant. We thank Yunfan Zhang for valuable discussions and the anonymous reviewers for constructive feedback.
Ethics Statement. Ethics.
All authors of this submission have read and adhered to the ICLR Code of
Reproducibility Statement. We provide our code with detailed comments in the supplementary materials. The detailed experiment settings, dataset processing steps and the devices used in our experiments are provided in Appendix C to ensure reproducibility.
The Use of Large Language Models. The authors use Large Language Models (LLMs) assist with grammar checking and language polishing in this submission. LLMs do not play significant role research ideation writing to the extent that they could be regarded contributor.

---

Published as a conference paper at ICLR 2026
REFERENCES
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-but based offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems, 0147436-7447, 2021.
Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine, 34(6))26-38, 2017.
Leemon Baird et al. Residual algorithms: Reinforcement learning with function approximation. In Proceedings of the twelfth international conference on machine learning, pp. 30-37, 1995.
Yanjun Chen, Xinming Zhang, Xianghui Wang. Zhiqiang Xu, Xiaoyu Shen, and Wei Zhang. Corrected soft actor critic for continuous control. arXiv preprint arXiv:2410.1673 2024.
Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning, In International Conference on Machine Learning, pp. 3852-3878. PMLR, 2022.
Pierre Clavier, Erwan Le Pennec, and Matthieu Geist. Towards minimax optimality of model-based robust reinforcement learning. arXiv preprint arXiv:2302.05372, 2023.
Thomas Degris, Martha White, and Richard Sutton. Off-policy actor-critic. arXiv preprint arXiv: 4839, 2012.
Esther Derman and Shie Mannor. Distributional robustness and regularization in reinforcement learning. arXiv preprint arXiv:2003.02894 2020.
Esther Derman, Daniel J Mankowitz, Timothy A Mann, and Shie Mannor. Soft-robust actor-critic policy-gradient. arXiv preprint arXiv: I:0::8033300 04848, 2018.
Tobias Enders, James Harrison, and Maximilian Schiffer. Risk-sensitive soft actor-critic for robust deep reinforcement learning under distribution shifts. arXiv preprint arXiv:2402.09992, 2024.
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6, 2005.
Vincent Francois-Lavet, Peter Henderson, Riashat Islam, Mare G Bellemare, Joelle Pineau, al. An introduction to deep reinforcement learning. Foundations and Trends@ in Machine Learning, (3-4):219-354, 2018
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference Machine Learning, pp. 1582-1591, 2018.
Scott Fujimoto. David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 2052-2062. PMLR, 2019.
Ivo Grondman, Lucian Busoniu, Gabriel AD Lopes, and Robert Babuska. A survey of actor-critic reinforcement learning: Standard and natural policy gradients. IEEE Transactions on Systems, Man, and Cybernetics, part C (applications and reviews), 42(6): 12t1119991110 2012.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp. 1861-1870. Pmlr, 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, al. Soft actor-critic algorithms and applications. arXiv preprint 1812 2018b.
Zhaolin Hu and Jeff Hong. Kullback-leible divergence constrained distributionally robust opti- mization. Available at Optimization Online, 1(2):9 2013.
Garud N. Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2): 257-280, 2005.
12

---

Published as a conference paper at ICLR 2026
Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems, 12, 1999.
Arash Bahari Kordabad, Rafael Wisniewski, and Sebastien Gros. Safe reinforcement learning using wasserstein distributionally robust mpe and chance constraint. IEEE Access, 2022.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in neural information processing systems, 33:1179-1191, 2020.
Navdeep Kumar, Esther Derman, Matthieu Geist, Kfir Y Levy, and Shie Mannor. Policy gradient for rectangular robust markov decision processes. Advances in Neural Information Processing Systems, 36:59477-59511, 2023.
Zhipeng Liang, Xiaoteng Ma, Jose Blanchet, Jiheng Zhang, and Zhengyuan Zhou. Single-trajectory distributionally robust reinforcement learning, 2024.
11721.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint 1.X.00950990997... 2015.
Zhishuai Liu and Pan Xu. Minimax optimal and computationally efficient algorithms for distribution- ally robust offline reinforcement learning. arXiv preprint arXiv:2403.09621, 2024.
Zijian Liu. Qinxun Bai, Jose Blanchet, Perry Dong, Wei Xu, Zhengqing Zhou, and Zhengyuan Zhou. Distributionally robust q-learning. In International Conference on Machine Learning, pp. 13623-13643. PMLR, 2022.
Elita A Lobo, Mohammad Ghavamzadeh, and Marek Petrik. Soft-robust algorithms for batch reinforcement learning. arXiv preprint arXiv:201 1. 14495, 2020.
Miao Lu, Han Zhong, Tong Zhang, and Jose Blanchet. Distributionally robust reinforcement learning with interactive data collection: Fundamental hardness and near-optimal algorithm. arXiv preprint arXiv:2404.03578, 2024.
Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative q-learning for offline reinforcement learning. Advances Neural Information Processing Systems, 35:1711-1724, 2022.
Xiaoteng Ma, Zhipeng Liang, Jose Blanchet, Mingwen Liu, Li Xia, Jiheng Zhang, Qianchuan Zhao, and Zhengyuan Zhou. Distributionally robust offline reinforcement learning with linear function approximation, 2023. URL https: arxiv. org/ abs 2209. 06620.
Daniel J Mankowitz, Nir Levine, Rae Jeong, Yuanyuan Shi, Jackie Kay, Abbas Abdolmaleki, Jost To- bias Springenberg, Timothy Mann, Todd Hester, and Martin Riedmiller. Robust reinforcement learning for continuous control with model misspecification. arXiv preprint arXiv:1906.07511 2019.
Viraj Mchta, Biswajit Paria, Jeff Schneider, Stefano Ermon, and Willie Neiswanger. An experimental design perspective on model-based reinforcement learning. arXiv preprint 1 2021.
Safa Messaoud, Billel Mokeddem, Zhenghai Xue, Linsey Pang, Bo An, Haipeng Chen, and Sanjay Chawla. S ac: Energy-based reinforcement learning with stein soft actor critic. In The Twelfth International Conference on Learning Representations, 2024. URL htttpp:://ppereevie. net/ fatt/Eorum?id-ie wwt.eiidrr.ctaa.a........
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 518(754):559-5333 533, 2015.
13

---

Published as a conference paper at ICLR 2026
Arnab Nilim and Laurent El Ghaoui. Robust control of markov decision processes with uncertain transition matrices. Operations Research, 53(5):780-773) 2005.
Xinlei Pan, Daniel Seita, Yang Gao, and John Canny. Risk averse robust adversarial reinforcement learning. 2019 International Conference on Robotics and Automation (ICRA), pp. 8522-8528. IEEE, 2019.
Kishan Panaganti and Dileep Kalathil, Sample complexity of robust reinforcement learning with generative model. In International Conference Artificial Intelligence and Statistics, pp- 9582-9602. PMLR, 2022.
Kishan Panaganti, Zaiyan Xu, Dileep Kalathil, and Mohammad Ghavamzadeh. Robust reinforcement learning using offline data. In Advances in Neural Information Processing Systems, volume 35, pp. 32211-32222 Curran Associates, Inc., 2022.
Hyuk Park, Duo Zhou, Grani A Hanasusanto, and Takashi Tanaka. Distributionally robust path integral control. 2024 American Control Conference (ACC), pp. 1164-1171. IEEE, 2024.
Lerrll Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial rein- forcement learning. International conference on machine learning, pp. 2817-2826. PMLR, 2017.
James Queeney and Mouhacine Benosman, Risk-averse model uncertainty for distributionally robust safe reinforcement learning, Advances in Neural Information Processing Systems, 16:1659-1680, 2023.
Shyam Sundhar Ramesh, Pier Giuseppe Sessa, Yifan Hu, Andreas Krause, and Ilija Bogunovic. Distributionally robust moodel-based reinforcement learning with large state spaces. In International Conference Artificial Intelligence and Statistics, pp. 100-108. PMLR, 2024.
Konrad Rawlik, Mare Toussaint, and Sethu Vijayakumar. On stochastic optimal control and rein- forcement learning by approximate inference. Proceedings of Robotics: Science and Systems VIII, 2012.
Tyrrell Rockafellar and Roger J-B Wets. Variational analysis, volume 317. Springer Science & Business Media, 2009,
Takuma Seno and Michita Imai. d3rlpy: An offline deep reinforcement learning library. Journal of Machine Learning Research, 23(315): 1-200, 2022. URL http: jmlr. org papers/v23/ 22-0017. html.
Alexander Shapiro. Distributionally robust stochastic programming, SIAM Journal on Optimization, 27(4):2258 2275, 2017. doi: 10.1137/16M105829
Laixi Shi and Yuejie Chi. Distributionally robust model-based offline reinforcement learning with near-optimal sample complexity. Journal of Machine Learning Research, 25(200):1-91 2024.
Rahul Singh, Qinsheng Zhang, and Yongxin Chen. Improving robustness via risk averse distributional reinforcement learning. Learning for Dynamics and Control, pp. 958-968. PMLR, 2020.
Elena Smirnova, Elvis Dohmatob, and Jérémie Mary. Distributionally robust reinforcement learning. arXiv preprint arXiv: (tt::::/02.0000000 2019.
Aviv Tamar, Yinlam Chow, Mohammad Ghavamzadeh, and Shie Mannor. Policy gradient for coherent risk measures. Advances in neural information processing systems, 28, 2015.
Emanuel Todorov. General duality between optimal control and estimation. In 2008 47th IEEE conference on decision and control, pp. 4286-42999 4292. IEEE, 2008.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 0026-5033. IEEE, 2012. doi: 10.1 209//RRSS.212..336111
14

---

Published as a conference paper at ICLR 2026
Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17022, 2024.
Herke Van Hoof, Nutan Chen, Maximilian Karl, Patrick Van Der Smagt, and Jan Peters. Stable rein- forcement learning with autoencoders for tactile and visual data. In 2016 JEEE/RS/ international IEEE, 2016.
conference on intelligent robots and systems (IROS), Pp.
Shengbo Wang, Nian Si. Jose Blanchet, and Zhengyuan Zhou. A fimite sample complexity bound for distributionally robust q-learning. In Proceedings of The 26th International Conference Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pp. 3370-3398. PMLR, 2023.
Shengbo Wang, Nian Si, Jose Blanchet, and Zhengyuan Zhou. Sample complexity of variance- reduced distributionally robust q-learning. Journal of Machine Learning Research. 25(341):1-77, 2024.
Yue Wang and Shaofeng Zou. Policy gradient method for robust reinforcement learning. In Interna- tional conference machine learning, pp. 23484 23526.. PMLR, 2022.
Hua Wei, Deheng Ye, Zhao Liu, Hao Wu, Bo Yuan, Qiang Fu, Wei Yang. and Zhenhui Li. Boosting offline reinforcement learning with residual generative modeling. arXiv preprint arXiv:2106.10411 2021.
Peter Whittle. Risk-sensitive linear/quaaraatccgaasssaa control. Advances in Applied Probability, 13 (4):764-777, 1981.
Wolfram Wiesemann, Daniel Kuhn, and Bere Rustem. Robust markov decision processes. Mathe- matics of Operations Research, 3853)1153-183 2013.
Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu. Constraints penalized q-learning for safe offline rein- forcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8753-87600 2022.
Huan Xu and Shie Mannor. Distributionally robust markov decision processes. Advances in Neural Information Processing Systems, 23, 2010.
Zaiyan Xu, Kishan Panaganti, and Dileep Kalathil. Improved sample complexity bounds for distribu- tionally robust reinforcement learning. Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Leurning Research, pp. 9728-9754. PMLR, 2023.
Pengqian Yu and Huan Xu. Distributionally robust counterpart in markov decision processes. IEEE Transactions on Automatic Control, 61(9):2538-2543 2015,
Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations on state observations. Advances in Neural Information Processing Systems, 33:21024-21037, 2020.
Ruida Zhou, Tao Liu, Min Cheng, Dileep Kalathil, PR Kumar, and Chao Tian. Natural actor-critic for robust reinforcement learning with function approximation. Advances in neural information processing systems, 36:97-133, 2023.
Zhengqing Zhou, Zhengyuan Zhou, Qinxun Bai, Linhai Qiu, Jose Blanchet, and Peter Glynn. Finite-sample regret bound for distributionally robust offline tabular reinforcement learning. In International Conference Artificial Intelligence and Statistics, pp. 3331-3339. PMLR, 2021.
Zhengyuan Zhou, Michael Bloem, and Nicholas Bambos. Infinite time horizon maximum causal entropy inverse reinforcement learning. IEEE Transactions on Automatic Control, 63(9):2787- 2802, 2018. doi: 10. 10.11099TAC.2201:27755666
Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Cameeii Mellon University, 2010.
Brian Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 1433-1438. Chicago, II., USA, 2008
15

---

Published as a conference paper at ICLR 2026
Appendix
CONTENTS
A Discussion
17
A.1 Necessity of Generative Model
17
A.2 Algorithm Details
17
Proofs
19
B.1 Proof of Proposition 3.3
19
B.2 Proof of Proposition 3.4.
19
B.3 Proof of Proposition 3.5
20
B.4 Proof of Theorem 3.6
B.5 Proof of Proposition 3.7
21
C Experiment Details
23
C.1 More Setting details
23
C.2 Extra Experiment Results
23
C.3 Ablation Study Details
26
D Regret Bound
29
16

---

Published as a conference paper at ICLR 2026
A DISCUSSION
A.1 NECESSITY OF GENERATIVE MODEL
In this section, we explain why model-free empirical risk minimization (ERM) is not applicable under a ww..dieergnce--costraanee uncertainty set and why a generative model (VAE) is necessary.
In offline RL, the nominal transition distributions po are unknown, and no simulator is available to generate additional samples. Under KL-based uncertainty sets, the dual formulation of the DR soft Bellman operator is nonlinear in the transition distribution. As result, naive empirical estimation from the dataset 'D leads to the well-known double- souble-sampling issue (Baird et al., 1995), caused by the nested structure of inner and outer expectations. In our algorithm, we want to apply operator Th."
where g* = argsupneg (e.,a)DD ( ((8,,)),((..)))
Define the objective function as
[(] = (a.)~D
{(s).a),
(>r()))) g(8, 0(8,))0)
(23)
=B (a.)~D
g(s. a) log BBg. exp ((
To construct a consistent empirical estimator of [gg,, we would need independent samples to approximate both the outer expectation over (8, a) and the inner expectation over ~ ph.a- This requires splitting the dataset into disjoint subsets D Douter and Dinner- For each (s, a) E Dinners we aggregate the corresponding samples starting from (s, a) contained in Douters denoted by (v,,a) and the empirical risk of objective (g) becomes
(-V(8))
))
F(g) = Dour
Σ (E
a( log
[D(o.a)) (2..,"))
Σ
exp g(s,a)
(00)-(0ss0) a(s,a)o
(24)
However, in continuous state and action spaces, it is nearly impossible to revisit the exact same Htte-action pair, leading empty conditional sample sets D(..a) and making ERM infeasible.
Importantly, this difficulty arises from the nonlinear structure of the KL-based dual formulation rather than from our functional optimization approach. Even we abandon the functional approximation and use the exact dual Bellman operator, the same double-sampling issue persists due to the nested expectation:
LQ = (s,jeD [Q(s,a) ((s,)a))))
exp (
(-V(=))) 38 ]. (25)
((s,a) E[r]
sup log $20
E,g.
In contrast, under TV-based uncertainty sets, the dual formulation is linear in the transition distribu- tion (Panaganti et al., 2022), which avoids this nested expectation and therefore does not suffer from the same issue.
Existing KL-based distributionally robust RL algorithms overcome this difficulty by using a Monte- Carlo rollouts (Liu et al., 2022; Wang et al., 2023). estimation from transition frequency (Wang et al., 2024), or direct estimation of nominal expectations (Liang al., 2024). However, these approaches rely on either simulators or repeated visitation of state-action pairs and are therefore not applicable in continuous offline RL. settings.
A.2 ALGORITHM DETAILS
In this section, we present a detailed description of the DR-SAC algorithm. In our algorithm, use neural networks (s), Qo(s,a) a) and na 8) to approximate the value function, the Q-function and the stochastic policy, respectively, with th, 0, ò the network parameters. We also utilize target network V (s) and Qs(s, a). where parameters & and 0 are the exponential moving average of respective network weights. Similar to SAC-v1 algorithm (Haarnoja et al., 201 8a), the loss function of V-nntworr is
Jv(w) = EAND
/ (Vo(s) - Earxo [Qa(s. a) a log (a {)]]
(26)
17

---

Published as a conference paper at ICLR 2026
As introduced in Section 3.3, in our algorithm, we modify the loss function of Q-network to
12 (0) (0,0))~
2 (Qa(s,a) T5g- 2o(s,a1)2
where
9 argsup (DD sE9,
F((s,a),
(27)
(o g(s, a) (2))(())))
argsup ((,,)E gEs,
-g(s,a) log
Rpe...
exp
9(8, a))
Optimal dual function I" can found with backpropagation through 7, We also keep the assumption of policy network in the standard SAC algorithm by reparameterizing the policy using neural network transformation a = fo(e; s), where e an imput noise vector sampled from a spherical
Gaussian. The loss of policy is
Ja (Ø) = EaNDDeNN a log To (folc;s) s) Qo(s, fo(c:s))-
(28)
In the SAC-v2 algorithm (Haarnoja et al., 2018b), the authors propose an automated entropy tempera- ture adjustment method by using an approximate solution to a constrained optimization problem. The loss of temperature is
(a) = Ea x, [-a log aa(a s) - -aH].
(29)
where H is the desired minimum expected entropy and is usually implemented as the dimensionality of the action space.
In addition, incorporate generative model into our algorithm. VAE is one of the most popular methods learn complex distributions and has shown superior performance in generating different types of data. In the DR-SAC algorithm, modeling the conditional distribution of next states. It assumes a standard normal prior over the latent variable, p(=) (0, T). The encoder maps (s, a, ))) to approximate posterior g(= 8, a, 8'), and the decoder reconstructs s' from the latent sample z and input (8, a). The training loss is the evidence lower bound (ELBO):
VAE to learn the transition function ( s, a) by
JvAr( By(=(s.a,e") [||s" 3'1[3] + DRI. (a(= ",,,,'')lW((,
(30)
where 8 are the reconstructed states from the decoder.
18

---

Published as a conference paper at ICLR 2026
B PROOFS
B.1 PROOF OF PROPOSITION 3.3
We first provide established result in DRO to compute the worst-case expectation under perturbation in KI-ddiergence constrained uncertainty set.
Lemma B.1 (Hu & Hong (2013), Theorem 1). Suppose G(X) has a finite moment generating
function in the neighborhood of zero. Then for any >>0 0,
exp (G(X)) + ,88
sup (P (P))56
Ep(G(X)] inf $20
3 log
Ero
(31)
Proof of Proposition 3.3.
Q²,(s, = E[rr] +
inf pEP...()
{E (Bgip(())))) (1)) - a log t (a'[s'))]]
= E(r] sup
(6)
Blog (PInp)((aa))) exp (-V(o)) + ,36
11
E[r] Y inf
(Lemma B.1)
020
B
(-v(d)))
Eir] + Y sup 820
- log Egg (EInnp((-aa)) exp
38
B
To apply Lemma B.1, let P = p(-11,, a), Po ('),,, and G(X) G(8') -V('). We assume rewards = R(s, a) are bounded, and the discount factor 7 E [0. 1). Since Q(s, a) is bounded, we know (ss) is bounded well. This implies that G(s') Vs'') has a finite moment generating function (MGF) under the nominal distribution (-l+,, a), i.e., Eg' a.a) (e)(e)G(e oc, for x in neighborhood of zero. This ensures that G(s") has finite MGF under Po required by Lemma B.I.
B.2 PROOF OF PROPOSITION 3.4
Before providing the proof of Proposition 3.4. present the optimality conditions of Lemma B.I.
Lemma B.2 (Hu & Hong (2013), Proposition 2). Let 3* be an optimal solution of the optimization problem in Equation (31). Let H esssup PP G(X) and K. = =xxmpy (xxPP(((() = -f). Suppose the
assumption in Lemma B. 1 still holds, then 3
G(X) has finite moment generating function and log + 0.
at 1/3*. Moreover, 3 0 if and only if H co,
This lemma tells us the optimal solution unique when 8* This happens if and only if there is a large enough probability mass on the finite essential supremum of X, under the distribution center Po. We use this lemma to discuss either 3 0 or 3 the following proof.
Proof of Proposition 3.4. Similar to the standard convergence proof of policy evaluation, we want to prove that the operator T is 7-contraction mapping. Suppose there are two mappings Q1, Easx (2 - (H)((ss)), i 1,2. For any state S, we have
S x A + R and define Vi(s)
(1(()) V (=) = Ea [Q - Q2(8, a)ll < 1|Q1 Q21loc.
Thus, 1|V Vo 1 ||Q: Q21loc
Next, for any 3 > 0 and (s, a) fixed, define function
cxp (-V(s) )
(V) = log Egg
35. 8
(32)
Let ||[ V22lo0 d. Then for any s' S. V3(8') d (s') S V2(s') + After exponential, expectation, and logarithm operations, monotonicity is preserved. We have
-Blog Epy. exp (-Va(o)) V2(s")
}
vi(s))
- d < - Bog Epp.
exp
B
B
Ve(s') B
){
S - 3 log Eyp
exp
+ d.
19

---

Published as a conference paper at ICLR 2026
This gives us [V1) Fo((2)) - [ V2lloo-
Lastly, we reformulate DR soft Bellman operator as
(T)((,,a)= Eir] 7 sup (V..V. 880
(33)
Let 3 be an optimal solution of sup (V),, i 1. From Lemma B.2, we know 5 is unique when 1 0 is optimal. And the optimal value is the essential infimum H, when S 0. We want to show FR; (V1) - Faz (V2) is bounded in all cases of Br.
Case 1: B1 52
In this case, the optimal value is the essential infimum value for both V. We have
Fay (Vi) Pag (V2) = essinf Vi(s") essinf Vi((')
S [V - Vo
s'~P2
s'-Po
The last inequality holds because monotonicity is preserved after taking the essential infimum. Case 2: B = 0, 32 > 0, WLOG.
In this case, know from optimality that
H1 essinf Vi((') 2 Fay (V1), H2 = essinf 1/((') S Fo; (V2). e'~Po
s'~PP.
Then we have
H1 - Fay (V2) S H1 - H2 S 1| - -V21
Fsz (V2) - H1 S (V2) - (V1) S SIV - V2
Thus, |Fgy (V1) - Fø3 (V2)1 = HH Faz (V2) - ||V 1/2lo0.
Case 3: B > 0, [3 > 0.
Suppose F3; (Vi) S FRz (V2), WI.OG. Then
[Fs (Vi) - Fag (V2) = F3y (V22 Fsy (Vi) < (V2) Pa5 (V)) S V - V22lo0
where the first inequality comes from the optimality of Bi-
Thus for any (s,a) pair, we have we
- =y
sup Fs, (V1)
sup Es, (V2)
8:20
8o20
Ey 11 - Volloc
$9 Q1 Q21loo-
Since T is 7-contraction mapping, the Banach Fixed-Point Theorem implies that the sequence {Q*) convergences to the unique fixed-point of TT. From lyengar (2005); Xu & Mannor (201 thi fixed point corresponds to the distributionally robust soft Q-value.
B.3 PROOF OF PROPOSITION 3.5
Proof. Given mi II, let QM, and VM, be the corresponding DR soft Q-function and value function. Denote the function for determining the new policy as
(=(((: = DKL ( (7'(11s)) s))
exp (x))) log 1 2*)
(34)
According to Equation (13), Th+1 = argmin Jap (TT' and Jmp. (TR+1) S Jay (Tx). Hence
bologTa+ile ( 8) Qúa (s. + a log 2T (s)) < <Bans (a log Th + (a s) Qui, (s. -) + a log TM (s)],
E la
20

---

Published as a conference paper at ICLR 2026
and after deleting = (s) both sides, the inequality is reformulated to
BawTT [QTh (s, - a log tT+1 s)] 2 Vid, (s).
Next, consider the DR soft Bellman equation:
Q (s.a) -E(r] +
inf Pa Paal6))
{ (Bar~pan VMs (s')]
inf S1(r] Pr.aPpaa())
(Eornpen Ba'ntee Q (s', ,a) log m4ti (1)]
=Tg (9M)) (s,a)
S 5M (s,a), V(s,a)
(35)
where operator T is repeatedly applied to Qū,, and its convergence is guaranteed by Proposi- tion 3.4.
B.4 PROOF OF THEOREM 3.6
Proof. By Proposition 3.5, QM, is non-decreasing with k. Since function QM, is bounded by (Rmax + loooo/)))/(( - 7), sequence {QM,, converges. Thus policy sequence { conver- gences to some n². It remains to show that x' is indeed optimal. According to Equation (13), Jz (x*) S Jr (T), Vi II. Using the same argument in proof of Proposition 3.5, we can show that QM, (8, a) S QM, (s, a) for any æ E II and (s, a) A. Hence " an optimal policy.
B.5 PROOF OF PROPOSITION 3.7
Before providing the proof, first introduce two technical lemmas. Specifically, Lemma B.4 establishes the interchange of minimization and integration property in decomposable spaces. This property has wide applications in replacing point-wise optimality conditions by optimization in functional space (Shapiro, 2017; Panaganti et al., 2022).
Lemma B.3 (Rockafellar Wets (2009), Exercise 14.29), Function f: x R2" + R (finite- valued) is normal integrand if fw, z) is measurable in for each and continuous in 2 for each
Lemma B.4 (Rockafellar & Wets (2009), Theorem 14.60, Exercise 14.61). Let J Ω x R + I (fnite-valued) be a normal integrand. Let M(2, A; R) be the spuce of all measurable functions Ω + R, My be the collection of all MMΩ, A; R) with Swen f(w, (o,x)(w))) for any space with M, C C A; we have
Then,
inf / f(w, (w)))))))( / JugR
inf f(w)x) uldw). (CER
xEX wee
Proof of Proposition 3.7. First we want to prove 3 = argsup f((s, a), 3) is bounded in interval Ig 0, Rmax t log Al for any (s, E S x A. Rewriting the optimization problem to its primal
(1-7)0
form, it is clear that
f((s,a),8*) (e,,())))
inf E [(s 2
When @ is greater than Rmasto loglA it can never be optimal since (1-7)8
(-v( 11 338
v(s')
[((s,a), = log
Egg.a
exp
3
Rmax + a logl logl_A)
)) 38
< - B log
exp
7)8
Rmax + a loglAA 1-y
38 < 0.
21

---

Published as a conference paper at ICLR 2026
Now know that f((s,a), () is finite-valued function for each (s, a) S x A and B E Ig.
Also, is E(S x meessuralll in (s, a) E S x .A for each Ig and is continuous in 3 for each (s, a) E S x A. From Lemma B.3, we know that f((s,a), B) is normal integrand.
Moreover, all functions in G is upper bounded and measurable so M, C 9 C CM((S x A) x A); R). Proposition 3.7 is direct conclusion of Lemma B.4.
22

---

Published as a conference paper at ICLR 2026
C EXPERIMENT DETAILS
C.I MORE SETTING DETAILS
To allow for comparability of results, all tools were evaluated on equal-cost hardware, Ubuntu 24.04 LTS system with one Intel(R) Core(TM) 17-6850K CPU, one NVIDIA GTX 1080 Ti GPU with 11 GB memory, and 64 GB RAM. All experiments use 12 CPU cores and I GPU.
We implement FQI and REQI algorithms from https: //ggtgittuuub com/ za -aaaannn/ggF DDPG and CQL are implemented from the offline RL library d3rlpy (Seno & Imai, 2022).
Hyperparameter Selection Across all environments, we use y 0.99 for discount rate, T = 0.005 for both V and Q critic soft-update, = 0.12 as initial temperature, |B| 256 for mini-batch size, [D] = 100h for data buffer size. Actor, Q and critic and VAE networks are multilayer perceptrons (MLPs) with (256, 256] hidden dimension. In the HalfCheetah and Reacher environments, we use two hidden layers in the actor and critic networks. All other networks have one hidden layer.
There are multiple learning rates in our algorithm. Learning rate for VAE network X is 5 10-5 in the Pendulum environment and 5 x 10 in others. In Step 5 of Algorithm 1, optimal function 9" is found via backpropagation with learning rate Ay. All other learning rates XØ. Xg. Ao and Xa are the same in each environment and represented by X
Value of learning rates Ay and X, number of Q-crities and latent dimensions in VAE are separately tuned in each environment and presented in Table 2.
Table 2: Hyper-parameters selection in SAC and DR-SAC algorithm training.
Environment
X
Ag
Q-Critic Number
latent dimensions
Pendulum
5 x 10
5 x 10
2
5
Cartpole
3 10-1
5 x 10-4
2
LunarLander
5 10-4
5x 10-4
2
10
HalfCheetah
3 x 10-4
5 x 10-5
5
32
Reacher
x 10-4
x 10-5
5
Offline Dataset To ensure a fair performance comparison, all models within each environment are trained on the same offline dataset. Each datasets contains 10th samples, generated by first training behavior policy and applying the gpsilon-greedy method. For most environments, the behavior policy trained by the Twin Delayed DDPG (TD3; Fujimoto al. (2018)) implemented from the d3rlpy library (Seno & Imai, 2022). In the Cartpole environment we use SAC to train the behavior policy. To ensure fair robustness evaluation, all models are trained to achieve the same performance (500, the maximum reward) in the unperturbed Caripole environment. Datasets generated by TD3-trained (or SAC-trained) behavior policies are denoted as TD3-datasets (or SAC-datasets). Additional details, including the algorithm to train behavior policy, training steps and the random-action probability are presented Table 3.
Table 3: Experiment details in dataset generation
Environment Pendulum Cartpole
Behavior Policy Algorithm
Training Steps
Random-Action Probability c
TD3
5 x 10a
0.5
SAC
5 x 105
0.5
LunarLander
TD3
3 x 105
0.5
HalfCheetah
TD3
106
0.3
Reacher
TD3
106
0.3
C.2 EXTRA EXPERIMENT RESULTS
Pendulum In the Pendulum environment, we compare DR-SAC with SAC, FQI, and DDPG. All models are trained on TD3-dataset. The robust algorithm RFQI does perform well even in the
23

---

Published as a conference paper at ICLR 2026
nominal environment. To evaluate the robustness of trained models, change the environment parameters length, mass, and gravity, with nominal values as 1.0, 1.0 and 10.0 respectively. To further show model performance under heavy-tailed perturbation, state observations. The distribution noise is defined as standard Cauchy distribution multiplied by parameter noise scale. We grind search å {0.1,0.2, 1.0} and find model under 0.5 have the best overall robustness.
also add Cauchy-distributed noise to
DR-SAC shows consistent robustness improvement compared to all other algorithms. The perfor- mance under length perturbation is presented in Figure ((a). In the mass perturbation test, DR-SAC has the best performance in all cases. For example, the average reward over 40% higher than SAC when mass changes 120%. In Figure 4 (b), there is a notable gap between DR-SAC and SAC performance when gravity acceleration changes 40%. In Figure (c), DR-SAC achieves consistent the best performance when noise scale increases.
DNSSAC/Duni
-108
T3
IQI
1000
Nominal Vakue
Percentage Change Nominal
Nolse
(a) Mass Perturbation.
(b) Gravity Perturbation.
(c) Observation Noise.
Figure 4: Additional Pendulum experiment results on TD3-dataset.
Cartpole the Cartpole environment, we compare the DR-SAC algorithm with non-robust algo- rithms SAC, DDPG, FQI, and robust algorithm RFQL. All algorithms are trained on the SAC-dataset. In the Cartpole environment, the force applied to the cart continuous and determined by the actuator's action and parameter force_man The highest possible reward is 500 in each episode. To ensure fair comparison, models are trained to have average rewards of 500 in the nominal environment.
We test the robustness by introducing two changes to the environment: applying action perturbation and adding observation noise. In the action perturbation test, the actuator takes random actions with different probabilities. In the observation perturbation test, noise with zero mean and different standard deviations is added to the nominal states in each step. We grind search 8 E {0.25, 0.5, 0.75. 1.0) and find DR-SAC has the best performance when 8 = 0.75. We also use = 0.75 to train the RFQI model.
In the Cartpole environment, DR-SAC has the best overall performance under both type of perturba- tion. Figure 5(a) extends Figure 1(b). DR-SAC has performance improvement over 75% compared to non-robust algorithms SAC and DDPG when the standard deviation of noise is 0.2 and 0.3.
LunarLander In the LunarLander environment, we compare DR-SAC with non-robust algorithms SAC, CQL, FQI, and robust algorithm RFQI. All algorithms are trained on TD3-dataset. In the LunarLander environment, the lander has main and side engines, and the actuator can control the throttle of the main engine. We change environment parameters engine pegineepowwrrree (main and side engine power) and wind_ _ower (magnitude of linear wind) validate algorithm robustness. We grind search 8 E {0.25, 0.5, 0.75, 1.0) and find DR-SAC has the best performance when 8 = 0.25. We also use p = 0.25 to train the RFQI model.
Under all types of perturbations, DR-SAC shows superior robustness compared to other algorithms. The performance under engine nogine_powerr power perturbation is presented in Figure I(c). In Figure 5(b), DR- SAC shows the highest average reward in most levels of wind perturbation. It is worth noting that the robust algorithm RFQI does not have an acceptable performance this test, even compared to its non-robust counterpart FQI.
24

---

Published as a conference paper at ICLR 2026
500
SAC
250
DR-SAC(Ours)
FQI
200
RFQI
DDPG
100
DR-SAC(Ours)
0
FQI
RFQI
COL
Stndard Deviation of Observation Noise
Wind Power
(a) Observation Perturbation Cartpole
(b) Wind Power Perturbation LunarLander
Figure 5; Additional Cartpole results on SAC-datasct and LunarLander results on TD3-dataset.
Reacher In the Reacher environment, compare DR-SAC with non-robust algorithms SAC, FQI, CQL,, and robust algorithm RFQI. All algorithms are trained on TD3-dataset. In Reacher environment, the actuator controls two-jointed robot reach target. We joint moint_damping to denote the damping factor of both joint0 and jointl, with default value as 1.0. We grind search ò E {0.1, 0.2, 0.3) and find DR-SAC has the best performance when 0.2. We also use p 0.2 to train the RFQI model.
To test the robustness of all algorithms,, we compare their performance after adding observation noise and changing parameters joint damping. In the observation perturbation test, we add zero- mean Gaussian noise the nominal state in dimensions trigonometric function values and are kept unperturbed. Performance under both perturbations is presented in Figure (d) and (e). Moreover, Figure (e), the standard deviation regions were computed but omitted from the final plot because the overlapping shaded areas of multiple algorithms made the figure unreadable. provide them in Table
The first dimensions state are
Table 4: Standard Deviation Model Performance under Damping Perturbation in Reacher
Joint Damping Value
SAC
DR-SAC
FQI
RFQI
CQL
2.0
1,452
1.459
1.638
1.509
1.396
3.0
1.534
1,540
1.754
1.601
1,456
4.0
1,631
1.628
1.865
1.699
1.529
5.0
1.735
1.721
1.969
1.802
1.614
6.0
1.841
1.819
2.057
1.910
1.711
7.0
1.946
1.925
2.135
2.018
1.816
HalfCheetah In the HalfCheetah environment, we compare DR-SAC with SAC baseline only due to the unsatisfactory performance of FQI and RFQI. All algorithms are trained on TD3-dataset. In the HalfCheetah environment, the actuator controls a cat-like robot consisting of body parts and 8 joints to run. We use fronissttif and front_ damping to denote the stiffness and damping factor of joint fthigh, fshin, and ffoot. Also, back back_stiff and back baccaaaaaaaaaaaa damping can be denoted similar way. The default value of these parameters can be found through the environmental assets of Gym- nasium MuJoCo https: / giggiituuu
com/F a Farama comm/Frrama-Foundaa Foundation honn/Gymaaiiii uummmm/// blob/ hhaaffcheetahh. cheetah.ee.... xml. We grind search 8 (0.1, 0.2, 0.3} and find DR-SAC has the best performance when 8 0.2.
main/gynne am/envs / /u joco/ assets s ff
Performance of back damping test is presented in Figure 1(f). Combining it with Figure 6, DR-SAC has notable robustness improvement across all perturbation tests. For example, in front_stiff perturbation test, DR-SAC achieves improvement much 10% when the change is 80%.
25

---

Published as a conference paper at ICLR 2026
6800
SAC
SAC
DR-SACIOUrS)
DR-SACIOurs)
6400
6200
6000
6200
3500
5700
5600
20
40
60
90
100
20
40
60
80
100
Percentage Change from Nominal value
Percentage Change from Nominal value
(a) Front Stiffness Perturbation
(b) Front Damping Perturbation
Figure 6: Additional HalfCheetah results on TD3-dataset.
C.3 ABLATION STUDY DETAILS
C.3.1 TRAINING EFFICIENCY OF DR-SAC.
In this section, we want to show that DR-SAC with functional optimization finds good balance between efficiency and robustness. We compare training time and robustness of Algorithm 1, DR- SAC without functional optimization, and robust algorithm RFQI, to show our DR-SAC algorithm has the best overall performance.
Comparison with Accuaaee Bellman Operator We first introduce DR-SAC algorithm without functional optimization. Most steps are the same as Algorithm 1, instead following modifications. Step 5 in Algorithm I is removed. Q-network loss is replaced by
J0 DR_ (s,a)>> [o a) T,5 (s,
(36)
where T is the empirical version of T by replacing ph.. with Po... We call this modified algorithm DR-SAC-Accurate and call Algorithm DOR-SAC-Functiona
this section.
We train SAC, DR-SAC-Funncti The optimization problem in Equation (11) is a problem over scalar 3 > and solved via Scipy for each (s. a) pair. Table 5 presents the training steps and time for three algorithms. Training time of DR-SACCAccurate is over 150x longer than standard SAC and over 50x longer than DR-SAC- Functional. Considering Pendulum environment is relatively simple, DR-SAC-Accurute algorithm is hard to utilize large-scale problems.
and DR-SAC-Accuree algorithms in Pendulum environment.
Table 5: Training steps and time for three algorithms in Pendulum
Algorithm SAC
Training Steps
Training Time (Minute)
10k
1.7
DR-SAC-Function
10k
4.7
DR-SAC-Accurate
8k
260
Moreover, we test the robustness of three algorithms by comparing their average reward under different perturbations. To be specific, we change Pendulum environment parameters: length and mass.
and DR-SAC-Accurate DR-SAC-Functione achieves comparable and even better performance under small-scale perturbation. For example, PPR-SAC-Functinn and DR-SACCAccurate have almost the same performance under mass perturbation test when change is less than 120%. In length perturbation test, MRR-SAC-Function has better performance when the change is less than 30%.
trained with 8 0.5. Figure 2 shows that
Efficiency Comparison with RFQL In Section 4.2, existing DR-RL algorithm RFQI also shows comparable performance under some perturbations. In this paragraph, want to show that DR-SAC
26

---

Published as a conference paper at ICLR 2026
requires much less training time than RFQI, improving its applicability to large scale problems. Table 1 lists the training time of SAC, DR-SAC, FQI, and RFQI algorithms in three testing environments. DR-SAC is demonstrated to be well-trained in at most 20% time required by RFQL Compared with each non-robust counterpart, the training time of DR-SAC is at most 3.6x of SAC, while RFQI requires 10 - 13x more training time than FQI. In Figure 7, we provide plot of performance changes against the training time in the Reacher environment, where RFQI is shown to be under-trained when the curve of DR-SAC converges.
MM
-8-
SAC
DR-SAC(OUrS)
FQI
-12
RFQI
10
Training Time (1e3 second)
Figure 7: Average Reward of 20 Episodes over Training Time in Reacher Environment.
Moreover, this efficiency improvement does not solely arise from the functional approximation step. but also from the inherent optimization efficiency in the loss function structure. The RFQI algorithm considers the RMDP framework with uncertainty sets defined by the TV distance and is empirically built on the BCQ algorithm. In RFQI, there exists a step similar to Equation (22) to find the optimal functional under empirical measurement. Experimental results show that the efficiency gap arises from the number of GD steps in solving this optimization problem. RFQI sets the default GD steps as 1000 while DR-SAC achieves comparable robustness performance with only 5 steps. To further investigate, we vary the GD steps in RFQI to 5, 10 and 100 in the LunarLander environment and report the model performance in the unperturbed environment. As shown in Table 6, performance drops sharply when RFQI uses fewer GD steps, indicating that the loss function structure in RFQI inherently leads to slower convergence and requires more optimization steps. In framework, the choice of actor-critic based non-robust baseline, KI. divergence induced uncertainty set and generative modeling nominal distribution estimation together yields a more optimization-frient formulation, contributing to our method's practical efficiency.
Table 6: GD steps, training time and performance LunarLander
Algorithm
DR-SAC
RFQI
RFQI
RFQI
RFQI (Used)
GD Steps
5
10
100
1000
Training Time (min) 240.0
36
12
21
139
238
Nominal Env Performance
175.9
181.9
192.9
201.2
C.3.2 SELECTION OF GENERATIVE MODEL.
Robustness of VAE. A consistent challenge in DR-RL algorithm design that unknown nominal distributions Pa.. often appear in the loss function. In Section 3.2 and Appendix A.1, we review methods used in other model-free DR-RL algorithms and motivate the necessity of generative models in our setting. Although generative models inevitable introduce additional estimation eтror when constructing empirical measures P.... our ablation studies demonstrate that DR-SAC is largely insensitive to the VAE modeling, therefore improving its applicability. In the Pendulum environment, where the state and action space dimensions are 3 and 1 respectively, we train DR-SAC with VAEs of latent dimensions 5, 10, 20, 50 and evaluate performance under perturbed pendulum mass. As shown in Figure 3(a), DR-SAC maintains superior robustness over the SAC baseline as long as the latent dimension lies within reasonable range (between 5 and 20 in experiments).
27

---

Published as a conference paper at ICLR 2026
Comparison with Other Generative Models. To demonstrate the choice of VAE other generative models, we implemented Diffusion Probabilistic Models and Normalizing Flows as alternatives to the VAE in DR-SAC and conducted ablation studies Pendulum and Cartpole. Model performance is provided in Figure 3(b), (c) and Figure 8. Flow-based models showed unstable performance even in unperturbed Pendulum environment. DR-SAC with Diffusion models achieved comparable robustness to the VAE in Pendulum. Crucially, the efficiency of sampling process with Diffusion models is major bottleneck. Diffusion-basee training is at least 4.5x slower than VAE-based training.
SAC DRIGACIAL Our!
-250
-200
300
170
-156
BR-ACCING, Dan
-30
06 30.SAClamruien
Percentage Change from Nominal Value (k)
Percentage Change
Prob. of Picking Rendem Action (k) (c) Action Perturbation Cartpole
(a) Gravity Perturbation Pendulum
(b) Mass Perturbation Pendulum
Figure 8: Additional results with different generative models in Pendulum and Cartpole.
C.3.3 USAGE OF V-NETWORK
In this section, demonstrate that keeping the V-network in the SAC algorithm reduces the sensitivity on dataset distribution. As introduced in Appendix C.1, offline datasets in this work are generated by first training behavior policy and applying the epsilon-greedy method collect data. Experimental results shows that SAC without the V-network exhibits unstable performance when the behavior policy differs across datasets.
Our experiments are conducted in the Pendulum environment. We generate two datasets with behavior policy trained by online version of SAC and TD3, denoted SAC-dataset and TD3- dataset, respectively. Figure 9 presents the average reward of 20 episodes against training steps in four scenarios: SAC-dataset vs. TD3-dataset, SAC algorithm with vs. without V-network. Removing the V-network shows minor influence on offline SAC learning using SAC-dataset. However, for TD3-dataset, SAC with V-network achieves stable average reward around -150 quickly, average reward of SAC without --newwrk fluctuates intensely and never exceeds 200. This validates that SAC with a netwoork is less sensitive to behavior policy and dataset distribution.
200
DM MM MM
600
1000
-100
SAC-w/
SAC-WAl V
TD3-w/
-1600
TD3-w/o
10
40
70
Training Step (x100)
Figure 9: Average Reward of 20 Episodes over Training Step in Pendulum Environment.
28

---

Published as a conference paper at ICLR 2026
D REGRET BOUND
Definition D.1. The distributionally robust regret RMs (T) of policy a E II is defined as:
RM,
VM, Vallloo
For any policy n, the soft value and soft Q-functions satisfy:
Vū, (s) (a)1()))) [QMs (2, a log :ta {)]
The following inequality holds:
VV - Vu = 1Q2M, - 1bx
Based on the estimator pa we define the corresponding estimate DR soft value function
oo
VM, (3) inf DEP(8)
Ep
(rr a. HT (s:))) و, =
where (d) = {Pa,a E A((S)) DKL (pe.a (lP2..)) =0}. Similarly, the estimate DR soft Q-
function is given by
(8, inf Ep PEP(8)
ri + + Σ 1-1 (re+ + a H (st))) 8,81 = -,
Define VM, = max Vh, and ai, E arn Vises
The estimate 13 is defined
- log Egg. exp (-V (6) )1)
()) = E[r] + sup 920
- 38
B
Assumption D.2. Assume KL 11 S S and supp(p?.a) = supp(p (a..)
By Pinsker's inequality, TV : P2.a) (LLpg,allp S fE.
Bound of [ - Qss a)ll- Lemma D.3. Under Assumption D.2,
Rmax log |A| tog [AD/(1--))
- 150--T70(s0 (TF(ss)))T7 s 2781
(1 7)-
Proof. As we defined in Section 3.2,
f((s, a). = - [Epg (c- (c-V66)/3]) (((s,,), - (Epp.. ( - 33..
From (Xu 2023, Proposition 5), the maximums of f((s,a), and f((s, 3) are achieved at 3*, 3,,3* 3 E [0, Vmax/6], that is,
((, Eirl sup ((sss)), Eirl
sup BE V
f((s,a), 3),
#20
Q(, E s (((,a)) = Eirl
f((s,a), B)-
sup AE Vmaa
820
Hence,
( Q(s - - Q(s, a) 5y
sup Vesoe
Note that supp(p?.a)
supp(ES.a). which implies that F2(0) essinf, V(s)
essinf V(s')= (0) Now, can assume that the optimal 8 ..- is achieved in [3, V Vmax/ where 3 = min (00mm(3/2241) 3/////// 1/2).
29

---

Published as a conference paper at ICLR 2026
Then, aim to bound the supremum of - over the interval
[$. Vmax /ö].
Since log 2 S x- - 1 when x 2 1, we have
1 log Epp, e
lf((s,a), B) [(]s,a),8)1 3 log [Bp
Egg0
-V00//08 -Bpg. (-Vidd)/8
V Ymax
8 min
Epg
[e-V(1)/8) BBBo
[e- (e-V(o1/88)
Since TV(p", a,a Pha)) I E, and Vin 2 0, we have
Epp e -V6///8 - Eyg (
6-V663/88
Σ (8))a(8)) e
E'ES
S Pa (s') pbo (s') P
-Vaiu/8 S 2TV Pb.a) S E1.
S'ES
In addition,
min EBpp.. [e (e-V(os/8)
Eyg... (e v(63/883 } e Vinaaa/
Thus, we obtain
Vimax V/nnre where V Viaxx
Rimax log |A|
()(s 3) f((s, a), B)1 S E1 Combining this with the earlier inequality gives
1-7 7
Rmax + a log |A| ( log [A))/(1-)) = E2-
a T5 (5, a))l S yE1
(1 - 7)6
Bound of 1i - -r,ll- For any π E II, let QK+L = Q² By Proposition 3.4, we know that Q will converge the DR soft Q-value QT which is the fixed point of TT. That is, T3 QM, = =Qh, and Q* + QMss Similarly, there exists a fixed point of 27 such that 13. QM, Qu, and Q+ +Mn, Then
= = Q² and @o = Q
1Q4% -Mu, = =112 QM, - T3 Q',,
=117 Qu, T3 Qu, + +TT Qu, 73 Qua
S + 711 Qim - -mall
1QQM, - Qu, S 1 7
E2
Regret bound.
We define the updating policy as
( (.
exp
#K+1 = argmin DKL
T[-
k = 1,
z**
nEll
By Proposition 3.6, the policy sequence (**) converges to the optimal policy ATMe under the estimate
DR soft policy iteration as k + cc.
For each state s E S, we have (s) - Ms (s) 2 0. By definition, (s) = A we have
(s). Then, MM
Vid (8) VM
#ims (s) <
Vin (8) Vig (6)
W (8) vitii
sup VM, (8) -sup Vz (s)
Mrs (s)
(s) V (8)
+ Mi (M) yime (s)
sup VM, (s) ViLa (8)
2sup VM, (s) V (s)
30

---

Published as a conference paper at ICLR 2026
Thus,
RMs (FJs))
Vis
AMgs Ma
suup x
VMs
- -ine 11=
2
Qum - Qiu S 1-7
252
x
31


Reviews:
[{"review_id": "AMWWY2reEs", "reviewer": "Reviewer_ftXw", "summary": "The paper introduce a new  Distributionally Robust SAC algorithm. Theoretical analysis and numerical examples are given to show the advantages of the proposed methods.", "questions": "Is there any limitation for the general distributional robust RL framework? \nIs it possible to learn the uncertainty of the environment and onine adapt?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduce a new  Distributionally Robust SAC algorithm. Theoretical analysis and numerical examples are given to show the advantages of the proposed methods.", "weaknesses": "The benchmarks used are low dimension examples. It would be good to add some higher dimension examples with continuous action spaces such as Ant in Mujoco environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "es2rSwGE7V", "reviewer": "Reviewer_B6wb", "summary": "This paper tackles the problem of developing a distributionally robust version of the Soft Actor-Critic (SAC) for offline reinforcement learning (RL) from a dataset with a continuous action space. Specifically, the authors develop a maximum entropy framework for policy iteration to accomplish the RL objective across an uncertainty set of transitions defined by KL-divergence, showing that their algorithm converges. They introduce a technique that reformulates the harder state-action optimizations into a single optimization problem over a function space. Through the use of a Variational Autoencoder (VAE), they are able to estimate the nominal transitions and thus generate samples without requiring access to a simulator.", "questions": "1. On line 122, adding \"discounted\" in front of \"Markov Decision Process\" adds clarity to exactly what problem setting you are considering.\n2. Extra closing parenthesis on line 125.\n3. Inconsistent notation of $\\Delta(\\mathcal{S}), \\Delta(S),$ and $\\Delta(|\\mathcal{S}|)$ on line 126 and equation 5.\n4. Addition of $\\forall s\\in\\mathcal{S}$ to make equations 6-8 more precise.\n5. Use of $|A|<\\infty$ in Assumption 3.1 versus $|\\mathcal{A}|<\\infty$ elsewhere.\n6. Please clarify what values $\\tau$ takes, and it's impact in the algorithm.\n7. It would help to make your work more appealing to a wider audience unfamiliar with deep learning by precisely clarifying that $\\hat{\\nabla}$ denotes the empirical stochastic gradient estimate in Algorithm 1.\n8. Is the standard deviation plotted in figure 1.e?\n9. Typo of \"grind\" instead of \"grid\" on lines 1190 and 1240.\n10. Typo of \"larges\" on line 1379.\n11. Addition of grid lines in figures 7 and 9 for consistency.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written and easy to follow. Namely, the paper systematically addresses the challenges discussed in section 3.1 in the subsequent sections.\n2. The use of interchanging minimization and integration in a decomposable space under the stated assumption is clever.\n3. The introduction of a VAE to estimate the nominal transition kernel for a given state-action pair, $p_{s,a}^0$ alleviates the double-sampling issue encountered in the calculation of the empirical risk by producing synthetic samples. This combined with the above point and the use of a neural network allows for the extension to a continuous action space.\n4. Extensive experimentation indicates the validity of the author's theoretical claims.", "weaknesses": "1. There appears to be a fundamental disconnect between the convergence guarantees in Theorem 3.6 (which rely on the assumption that $|\\mathcal{A|<\\infty}$ and hence tabular) and the practical extension to continuous actions in the algorithm's construction. Namely, the convergence guarantees are not formally proven under continuous actions, but rather implied through the use of a neural network and empirical validation.\n2. The use of a VAE to approximate the nominal transition kernel introduces additional risk due to the optimization occurring on this estimation and not the true environment. Should a given dataset not have sufficient coverage, the VAE would learn a model that is not necessarily robust to the true environmental shifts, but rather be robust to some other environmental shifts. The authors should consider including further discussion on this topic in the ablation studies in section 4.3 or additional theory-based discussion on this in the appendix. Can you provide a reference that provides some additional clarity on the above point?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ALO1dJt1so", "reviewer": "Reviewer_ku4q", "summary": "This work proposes a DR-SAC algorithm under a KL-divergence-based uncertainty set. It employs a Variational Autoencoder (VAE) to estimate the empirical distribution from the offline dataset. The proposed algorithm is model-free and can be applied to environments with continuous action spaces.", "questions": "1. Could the authors provide a more detailed discussion on why the VAE can effectively avoid the double-sampling process, and whether the estimated distribution satisfies the requirement of being an unbiased estimator? In previous offline DR-RL works, the use of double-sampling and treating the empirical distribution derived from offline data has been theoretically justified — how does this approach maintain similar rigor?\n\n2. Are there any additional requirements or assumptions regarding the offline dataset (e.g., data coverage, distributional support, or sample quality) for the proposed method to perform effectively?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. This algorithm can be applied in continuous action/state space setting. \n\n2. Using VAE to avoid double sampling in offline DR-RL problem. \n\n3. This work provide convergence guarantee.", "weaknesses": "1. The presentation of this work is not very clear or professional. The gradients used in the algorithm section are not defined where they first appear—I had to consult the appendix to understand their meaning. Moreover, the assumptions on which the convergence analysis relies are not stated explicitly, making it difficult to follow the theoretical results.\n\n2. The proof section lacks sufficient rigor. The main theorem is presented without clearly stating the required assumptions, and the accompanying proof is incomplete and lacks detail. A more thorough and transparent exposition of the assumptions and proof steps is necessary to make the theoretical claims convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xRK8LgDNrP", "reviewer": "Reviewer_aZrL", "summary": "This paper proposes Distributionally Robust Soft Actor-Critic (DR-SAC), a model-free reinforcement learning algorithm that integrates distributionally robust optimization into the Soft Actor-Critic framework. The key idea is to handle environment uncertainty via a KL-divergence–bounded uncertainty set over transition distributions. The authors derive a distributionally robust soft Bellman operator using KL duality, which introduces a scalar dual variable $\\beta$ for each $(s,a)$ pair. To avoid solving numerous local optimizations, they replace these with a single functional optimization $g(s,a)$ over the entire dataset, justified by interchange theorems from variational analysis. To handle continuous state–action spaces and offline settings, the paper incorporates a VAE-based generative model to approximate the nominal transition distribution $p^0(s'|s,a)$, thereby mitigating the double-sampling issue that arises from nonlinear expectations in the KL dual form.", "questions": "1. With errors from function approximation, VAE modeling, and functional optimization present, can the authors provide partial error–convergence guarantees (e.g., one-step contraction with bias)?\n2. What is the quantitative rationale for choosing a VAE over flow-, diffusion-, or score-based transition models?\n3. Do the authors believe the KL-ball assumption adequately captures realistic structural or regime shifts? Would a Wasserstein-ball or f-divergence family extension offer advantages?\n4. (minor) Please fix typographical errors (\"In iour algorithm\") and ensure consistent terminology (e.g., \"KL ball\").", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Methodological innovation**: Introducing KL-based DR soft Bellman operator and replacing numerous scalar $\\beta$-optimizations with a single functional optimization are novel and computationally effective.\n- **Efficiency-performance trade-off**: Demonstrates superior robustness compared to SAC/RFQI while reducing training time.", "weaknesses": "1. **Theory–implementation gap**: The convergence and monotonicity proofs apply only to tabular finite-action cases, whereas the implemented DR-SAC relies on continuous Gaussian policies and neural approximators.\n2. **VAE dependency**: While the need for a generative model (to avoid double-sampling) is well justified, empirical validation of the VAE’s adequacy is limited (mostly latent-dimension sweeps) and lacks comparisons with alternative generative models such as flows or diffusion models.\n3. **Limited robustness evaluation**: Perturbations focus on Gaussian noise and parameter scaling. Tests under more challenging conditions (e.g., heavy-tailed, regime-switching, or reset/restart scenarios) are absent. The adequacy of the KL-ball assumption versus, e.g., Wasserstein metrics remains unexamined.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)