Robust Reinforcement Learning in a Sample-Efficient Setting
Authors: Siemen Herremans
Ali Anwar
Siegfried Mercelis
Venue: Accepted by TMLR ()

Abstract: The performance of reinforcement learning (RL) in real-world applications can be hindered by the absence of robustness and safety in the learned policies. More specifically, an RL agent that trains in a certain Markov decision process (MDP) often struggles to perform well in MDPs that slightly deviate. To address this issue, we employ the framework of Robust MDPs (RMDPs) in a model-based setting and introduce a second learned transition model. Our method specifically incorporates an auxiliary pessimistic model, updated adversarially, to estimate the worst-case MDP within a Kullback-Leibler uncertainty set. In comparison to several existing works, our method does not impose any additional conditions on the training environment, such as the need for a parametric simulator. To test the effectiveness of the proposed pessimistic model in enhancing policy robustness, we integrate it into a practical RL algorithm, called Robust Model-Based Policy Optimization (RMBPO). Our experimental results indicate a notable improvement in policy robustness on high-dimensional control tasks, with the auxiliary model enhancing the performance of the learned policy in distorted MDPs, while maintaining the data-efficiency of the base algorithm. Our methodology is also compared against various other robust RL approaches. We further examine how pessimism is achieved by exploring the learned deviation between the proposed auxiliary world model and the nominal model. By introducing a pessimistic world model and demonstrating its role in improving policy robustness, our research presents a general methodology for robust RL in a model-based setting.

Published in Transactions on Machine Learning Research (12/2025)
Robust Setting
Reinforcement
Learning in a Sample-
Siemen Herremans
sienennkereennnsSuutuuurrn...
IDLab Department of Electronics and ICT, Faculty of Applied Engineering University of Antwerp imec
Ali Anwart
ali www.aaaarraaaam Crwwarruannuuurper.....
IDLab Deparbment of Electronics and ICT, Facully of Applied Engineering
University of Antwerp
imec
Siegfried Mercelis'
siegfried,
IDLab Department of Electronics and ICT, Faculty of Applied Engineering University of Antwerp- imec
Reviewed on OpenReview: https: oppnreveee.. net/ forum? id= EaJ6nLYL5F
Abstract
The performance of reinforcement learning (RL) in real-world applications be hindered by the absence of robustness and safety in the learned policies. More specifically, an RL agent that trains in certain Markov decision process (MDP) often struggles to perform well in MDPs that slightly deviate. To address this issue, we employ the framework of Robust MDPs (RMDPs) in a model-based sctting and introduce a second learned transition model. Our method specifically incorporates an auxiliary pessimistic model, updated adversarially, to estimate the worst-case MDP within a Kullback-Leibbe
uncertainty set. In comparison to several existing works, our method does not impose any additional conditions on the training environment, such the need for parametric simulator. To test the effectiveness of the proposed pessimistic model in enhancing policy robust.ness, we integrate it into practical RL algorithm, called Robust Model-Based Policy Optimization (RMBPO). Our experimen- tal results indicate a notable improvement in policy robustness on high-dimensional control tasks, with the auxiliary model enhancing the performance the learned policy in distorted MDPs, while maintaining the data-efficiency the base algorithm. Our methodology is also compared against various other robust RL approaches. We further examine how pessimism achieved by exploring the learned deviation between the proposed auxiliary world model and the nominal model. By introducing pessimistic world model and demonstrating its role in improving policy robustness, our research presents general methodology for robust RL in a model-based setting.
1 Introduction
Reinforcement learning (RL) has been shown to perform well in many environments. However, the per- formance of a trained RL agent can rapidly decrease when the agent evaluated a slightly altered enviromment (Christiano
al. 2016 Rusu al., 2017). This one of the issues that has limited the adoption of RI. in real-world scenarios, more specifically due to the simulation-to-reality (sim2real) gap and inherent variability in real control systems. For example, these control systems could be robots, where the friction changes over time due to the oil in the joints. Therefore, there is a need for policies that are robust enough to perform well in environments that differ from the training environment. Due to this necessity,
iThese authors provided equal supervision the work.

---

Published in Transactions on Machine Learning Research (12/2025)
various approaches tackle the sim2real issue, often using different problem formulations Zhao al. 2020
Some of these approaches include domain randomization or transfer learning. In our work, however, we aim to maximize the worst-case performance of the RL agent under bounds on the uncertainty, commonly for- malized as a robust Markov decision process (RMDP) (Wiesemann et al. 2013). This formalism defines an uncertainly set of multiple Markov decision processes (MDPs), where the agent oblivious to which MDP of the set it is acting in. The objective in an RMDP then becomes to maximize the return in the worst (i.e., lowest cumulative reward) MDP of the uncertainty set. In previous research, methods that work within the RMDP formalism have demonstrated enhanced robustness against perturbations between the train and test environment (Gadot et al.) 2024 Pinto et al. 2017). However, these works often impose extra requirements on the training environment, such as the ability to re-sample a transition multiple times or to have access to a parametric environment during training, meaning that these algorithms can modily simulator parameters (such as friction or gravity) during training. A second challenge for RL in some real-world applications is the sample efficiency, since it is often slow to perform exploration (e.g., on physical robot). poodel-basod an approach that has remenstrated significant progress in sample effi- ciency, such as the work of Janner et al. (2019) for simulated robotics the more general work by Hafner et al. 2023) that allows visual state representations.
reinforcement learning (MBRL)
This paper adopts the RMDP setting and proposes an algorithm that improves the robustness of a learned policy, without placing any additional requirements on the training environment. Importantly, we work within the MBRL framework and aim to maintain the sample efficiency of these methods. Inspired by the ideas of Rigter et al. (2022) Pinto al. (2017), our approach introduces an auxiliary model that acts as an adversary to minimize the cumulative reward under the current policy. This auxiliary objective then defines two-player Markov game with the policy optimization objectívc. By scquentially optimizing these two competing objectives, our algorithm can optimize towards more robust policy. Our main contributions are firstly (i), proposing a robust MBRL algorithm to improve robustness in an online setting, while remaining sample efficient. This is achieved by adding an auxiliary model to model-based policy optimization (MBPO) which learns a pessimistic world model via adversarial updates. Secondly (ii) we evaluate the empirical performance of our algorithm on high-dimensional Multiple Joint Control (MuJoCo) and Deepmind Coltrol Suite (DMC) benchmarks under simultaneous parameter distortions + Thirdly (šii), we interpret and quantify how the predictions of the learned robust model differ from the nominal model, demonstrating how the agent achieves robustness. The remainder of this work will first highlight relevant background to our approach. Then, our methodology is described in detail. Subsequently, the results demonstrate the improvement in robustness that our method provides to MBPO (Janner et al. 2019 in multiple MuJoCo Todorov et al. 2012) and DMC (Tassa et al. 2018 control environments. Finally, we draw conclusions and outline future research directions.
2 Background
In this section, we first introduce MBRL within the broader context of MDPs. Secondly, RMDPs are described and an adversarial framework to tackle them is highlighted. Finally, the Kullback-Leibler (KL) uncertainty set is defined.
2.1 Model-Based Reinforcement Learning
MBRL (Moerland et al. 2023 operates within the framework of an MDP, defined by the tuple (8, P, P, P,),) Po), where S and A denote the state and action space. The transition distribution P(s', rls',rls,a a) defines the transition and reward probabilities given state s and an action o, with the set of possible rewards denoted by a real interval R. is the discount factor, and Po(s) is the initial state distribution. The objective in RL is to identify an optimal policy x¹ that maximizes the expected sum of discounted rewards:
I Evaluation code and weights available at https///s/tttss/ttttubbb.oc http:///gthtuuccccm//m/rreeeella whtp:///mbbooeeellaaaooooo--oe 2 Recorded examples availahle at /attps//siites, www.aaiitte...ooooeeeee wwwg//e.coo//i//////ppp.
2

---

Published in Transactions on Machine Learning Research (12/2025)
arg max ET,,
y'r
(1)
In addition, we denote the state visitation distribution of the MDP as d", which defines the likelihood of being in a certain state when following policy иτ In MBRL, the agent learns a model of the environment's rss, dynamics, represented by po(s',r|s, a); from the data collected through its interactions with the MDP. This model is then used to simulate future states and rewards, reducing the number of interactions with the real environment. In most MBRL algorithms, the agent's policy is updated based on both real experiences and simulated experiences from the learned model, balancing between exploration for model learning and exploitation of the learned model for policy improvement. For notational simplicity, we use 8, a and s to denote St, Ge, St+1 respectively, when it is clear from context.
2.2 Robust Markov Decision Processes
In traditional MDP, the agent optimizes its policy in static transition model P. However, in some real-world problems, the dynamics can change over time. Hence, (Wiesemann et al. 2013) where the agent acts on an uncertain transition distribution P(..-ss.)) that is assumed to lie within an uncertainty set pas., Following other works, we only consider so-rectangular uncer- tainty sets, as finding the optimal robust policy for general uncertainty sets is NP-hard (Gadot et al. 2024 Zhou et al. 2024: Wiesemann et al. (013)). This assumption ensures that the uncertainty set in (8, a a-pair is independent from the uncertainty
define Robust MDP
in other (s, paairs Furthermore, we define the global uncertainty set P the Cartesian product of each independent marginal uncertainty set pa,a. The robust objective Jrobust(T)) can now be defined as the objective function in the worst-case MDP of given uncertainty set. This objective is staled in Eq. 2
Jrobuel(T)) = min E.,, PEP
r'r
(2)
The goal of robust reinforcement learning is then to find the optimal robust policy T that maximizes this worst-case performance:
7P arg max Jrobust (T)
#ETl
Additionally, the algorithm is dependent on knowing the worst-case MDP at every time step, we call this the inner-loop problem. Under the roctangularity assumption, for a small uncertainty set, the inner-loop problem can be solved iteratively by evaluating transitions in each MDP P P.P. However, when the uncertainty set becomes very large or continuous, the inner-loop problem can be challenging. We will follow related works by considering this combined optimization objective as a two-player zero-sum Markov game (Rigter et al., 2022 Pinto al., 2017). In this game, one player optimizes the policy, to maximize the expected return, whilst the other player tries to find p E P, which minimizes the expected return.
2.3 KL Uncertainty set
Since the 'true" uncertainty set is often not known or ill-defined, a common choice is the KL uncertainty set, denoted as Pkt (Hu & Hong] 2013 Gadot et al. 2024 Shi & Chi 2024). The KL uncertainty for a transition from a state-action pair 8,a is delined as:
P (p, ASxR (rr,((ss a)) S )x.a)
where is the nominal kernel, i.c., the environment with which the agent interacts during training. AsxR denotes the probability simplex over S x R. Furthermore, DRI(P(, 1s, a) ((, is the KT. divergence
3

---

Published in Transactions on Machine Learning Research (12/2025)
between the transition distributions of model P and the nominal model P, given current state and action. Again, we can define the global KT. uncertainty set Pkt. in an analogous manner to P, denoting the Cartesian product of independent marginal sets Pki.- In our work, we consider a constant threshold, i... Es.a for all s, a. Notably, a limitation of the KL uncertainty set is the dependonce on a stochastic transition model, since it would require the (ill-defined) KT.-divergence between two Dirac functions the deterministic setting. However, this limitation can be circumvented without loss of generality by adding action noise between the agent and the MDP (Gadot et al. 2024 Zhou et al. 2024).
3 Auxiliary Model Learning
The goal of this section is to tackle the inner-loop problem of the robust objective, as defined by the minimization problem in Eq. 2 i.c. approximating the worst-case MDP, denoted as P E EP, where we choose P to be the global KL uncertainty set centered around the nominal model P. This choice of uncertainly set follows a common choice literature (Gadot et al., 2024; Hu & Hong. 2013). To describe our methodology. Section 3.1 introduces the auxiliary adversarial model as an addition to traditional world model learning (c.g. via maximum likelihood estimation (Janner et al. 2019)). The auxiliary model has a well-defined KL divergence with the approximated nominal model. Secondly (Section 3.2), we introduce the loss function to train the auxiliary model to maintain a low KL divergence with the normal learned transition model, whilst also learning to be pessimistic (i.e., minimizing the return of the transition).
3.1 Auxiliary Model
Since we work within the context of MBRL, we have direct access to a parameterized approximation, pols', rls, a), of the nominal transition model P(s', rls,,), a). However, this does not directly provide with method to approximate DKz(Po (s, a))l a)), since we do not have access to the transition probabilities of the training environment P, needed to construct the KL uncertainty set. Hence, we propose to not directly try to approximate the pessimistic transition model, thus leaving Pe untouched. As an alternative, we pro- pose an auxiliary paramcterized model, Iv, which takes as input the outputs of the Icarned transition model po; in addition to 8 and a. Both gv and pe are parametrized as neural networks (multilayer perceptrons). Next states and rewards can now be sampled according to Eq.
8 8 ~ as a, po pl(s,,))).))
(5)
As it not practical to compute D P a)) we compute Dxi a)llpo(s, a)) and use this divergence for the uncertainty set. This practical step introduces an error pe does not perfectly capture the training environment. In fact, during numorical measurements, we noted that ( a) [P(s, a)) is mostly not well defined Je and pe predict Gaussian with infinite support, where P(s, a) always has a finite support (see Appendix H However, useful metric to limit the potential decline of the performance in the nominal environment. We rely on the empirical results to demonstrate that our algorithm succeeds in improving the robustness of a learned policy, In our work, both po and Je define the mean and covariance matrix of a diagonal multivariate Gaussian distribution, so the KL divergence can be computed closed-form. In practice, we provide the predicted mean μ4 and covariance matrix Eg as inputs to the auxiliary model 9v, since a Gaussian is fully defined by these two components. Strietly speaking, the addition of Pe as input to the auxiliary model is not necessary, however, this greatly cases the optimization of go, which will be explained in Scction 3.2
describe in Section 4.2 Drz(ge a) [po a)) is still a
3.2 Training the Auxiliary Model
The goal of the auxiliary model is to minimize the value of each transition under the current pol- icy while remaining within the desired uncertainty set PKL- We employ an expected uncertainty set Ee [Dxz (ox(())))llp))))) instead of bounding the clement-wise divergence. This allows us to use common deep learning techniques for optimization. Note that it follows from the Markov inequality that a limited expected KL. divergence also limits the probability of high individual KI. divergences (see Appendix R.1
4

---

Published in Transactions on Machine Learning Research (12/2025)
Using this inoquality, one could set the bound expected KL divergence (denoted by ce) in function of acceptable probability that the element-wise KT. divergence (i.e., <) is violated. By applying Lagrangian relaxation to the constraint problem, we can formulate this objective as a dual problem in Eqn. G The first term is proposed by Rigter et al. (2022) and forces the auxiliary model to minimize the value of transitions. v denotes the learned value function, parametrized by o, which are the parameters of the agent used to solve the outer-loop problem. The second term limits the expected KL divergence between the auxiliary model and the approximate model.
max min [E ((())))a vanr [logls (s,, +XX a)]] (6) X20 Be
Equ. 6 can directly be approached by Lagrangian dual descent. However, this method is known to be
unstable and oscillate around the constraint boundary (Stooke et al 2020 Platt & Barr 1987). Following Rigter et al. (2022) and a practice used in other works that theoretically rely on a constrained objective (such as Higgins et al. (2017)) we choose fix À as a static hyperparameter and optimize the linear combination of the primal objective and the constraint. Therefore, we optimize the auxiliary model using gradient descent, following the gradient provided in Eqn. (note that the constant À in the second term can be replaced by y in the first term, which is equivalent up to scaling factor).
V Vv)((V) = (o (e do)g.. anv (n. (r (r+ (V) ( rls, po(-)) + V V KL (lloo)))))))
The hyperparameter 7) controls the influence of the value function: for small 7, the auxiliary model will be almost identical to the approximale model and therefore our method will not induce any significant pessimism. For larger values of 17, 96 will grow more pessimistic and therefore the auxiliary model can learn to be significantly more pessimistic than the approximate model. Values of that were used in this work can be found in Appendix C. Formal guarantees that the auxiliary model remains in the uncertainly set are left for future work.
3.3 A Supervised Toy Experiment
Before moving to a RL algorithm in the next section, we set up a supervised toy problem, this will allow us to choose some hand-crafted value functions and interpret their effect on the pessimistic model learning visually. We create dataset with samples of standard normal distribution. This dataset represents samples of the transition model, given single state and action. As next step learn nominal parameters 0 (Pnominals nominal that define the approximated nominal distribution. In the final step, follow the methodology of Sec. 3.2 to learn the parameters yb (Ppessimistic: @ ppsssimistie)) which define the pessimistic auxiliary distribution. Note that it is not strictly necessary to approximate 0, and we could just provide the ground truth nominal model to compute the KL-divergence. However, we wanted to remain as close as possible the setting of Section
The following three value functions are used: (x) = x, v2(x) = --r and vz(x) = =² For U1, we expect the pessimistic model to biased towards lower values of x, since there is linear correlation between x and the value of æ. For U2 make analogous reasoning for bias towards higher values of æ. Lastly, we expect unbiased distribution for V32 however the standard deviation is expected to be smaller. This follows from the fact that the normal distribution is already centered around the point where the value function is minimized, Lee x = O. The results these experiments are shown in Fig. [ which confirm that the auxiliary model is biased towards low-value points, and that this bias scales with n. We also performed experiments when learning a categorical distribution instead of Gaussian, these can be found in Appendix B together with summary of the supervised isee algorithm that was used.
5

---

Published in Transactions on Machine Learning Research (12/2025)
V1(x) x
V2(x) -x
V3(x) x2
0.4
0.4
0.3
0.3
0.4
Density
0.2
0.2
0.2-
0.1
0.1
0.0
0.0
0.0
-2.5
0.0
5.0
-5.0
-2.5
2.5
-5.0
-2.5
0.0
5.0
Nominal Kernel
n= 0.25
n=
1.0
Figure 1: A toy experiment where we learn a pessimistic auxiliary model. The nominal model is a standard Gaussian, the associated value function highlighted on each plot.
4 Robust Policy Learning
We propose robust model-based policy optimization (RMBPO), a RL algorithm that incorporates the aux- iliary model to improve the robustness of the learned policy. Furthermore, we discuss the implications of RMBPO on the performance bound of MBPO, where we motivate the choice for the KT. uncertainty set.
4.1 Proposed Reinforcement Learning Algorithm
To improve policy robustness, combine the auxiliary model with MBPO (Janner al. 2019) to create RMBPO. MBPO approximates the training enviromment by maximizing the log likelihood of experienced transitions under its learned model Pa- This model is a neural network that predicts a mean and covariance matrix over the next states and rewards, conditioned on the current state and action. On-policy rollouts are then performed on the learned model. Finally, the unrolled data is used to update policy via Soft Actor- Critic (SAC) (Haarnoja al. 20188.. We modify MBPO by training an auxiliary model in addition to the existing model, via Eq. Since these two models not hinder the accuracy or precision of pa. During the model unroll, pass the current state through the learned model po, after which we use the output of that model (μo, Σa) as inpul to the auxiliary model. The auxiliary model will then predict a modified (My, Ex), predicting a pessimistic transition. This procedure is fully doscribed in Algorithm [] where our additions are highlighted in blue. Following other works (Gadot et al. 2024; Zhou et al.) 2024), we add small amount of action noise to the environment, otherwise, the uncertainty set would not be well-defined. More details on the action noise are provided in Appendix D
trained separately, the auxiliary model learning does
4.2 Implications on the performance bound
Letting the auxiliary model minimize the value of transitions, is direct application of the definition of an RMDP. However, we also want to limit the difference in returns in the nominal MDP. This section will motivate that bounding the KT, divergence is a way to limit this loss in episode returns. By starting from the theoretical insights, provided by Janner et al. (2019), we can see that certain policy has a bounded difference between the returns under the real model (environment) and the learned model (po). The data collecting policy defined as wD- If the expected total variation (TV) distance between two transition distributions is bounded at each time step by 6m max, EyLD,a [Dry (Psssa)lloo(ssa) and the policy divergence be bounded by éx 2 (rx)))(((())) the approximate model returns is bounded, this bound is restated by Eqn. S The true return G[z] denotes the expected return of a policy in the nominal environment. The model return G(r) denotes the expected return of a policy in the approximate model Pa-
then the difference between the true returns and
12y (Em + 2Ex) 4r 4mmarca
G | 2 G(T)
(8)
7
(1

---

Published in Transactions on Machine Learning Research (12/2025)
Algorithm RMBPO (Additions in blue)
Initialize policy wh: predietive model po auxiliary model Dør environment dataset Denv, model dataset model
for N epochs do
4:
while improving on holdout set do
5: Update model parameters on environment data Dene via maximum likelihood
6: end while
while improving on holdout set do Update auxiliary model parameters &b according Eq. V v do V N Denvi pe. Pa. TT) end while
10:
for E steps do
11:
Take action in environment according to n&i add to Denu for M model rollouts do
12:
13:
Sample & uniformly from Denv
On-policy rollout according to Eq. starting from 8, using policy Tæz add model 15: end for
14:
16: Perform (soft) actor-critic updates & using samples from modell-
17:
end for
18:
end for
We can employ this insight to bound the difference in returns of the optimal policy on the learned model and the auxiliary model, as the learned model serves as the data- mata-generating 'environment for the auxiliary model (i.c., we are interested in lower bounding the return under Pe. given the return under 9p). This would mean that improving the policy under the auxiliary model also improves the policy under the nominal learned model, which provides lower bound on the performance the real training environment. Therefore, we define Emouz max, E (DV (s', r|s, a) ())llp(ss', rls, a))] as the expected maximum TV distance between the auxiliary model and the learned model. Let Gauz[z] be defined as the expected return of a policy under the auxiliary model. Furthermore, because the two models are unrolled under the same policy, we know that the policy divergence Ezauz is O. Employing Eqn. in this setting provides Eqn. g
27H ( (Emaux)
[(x]] Gann(TT)
(9)
7)2
We can combine Eqn.
and Equ. to become:
2 (6m + 2ex + maus)
4rmart (1-7)
G(m)
Gauz [T]
(10)
(1 y)2
This makes intuitive sense, if Go is very different from Pa. our agent will perform poorly in the training environment. de (almost) identical to po, the RL agent learns from model that identical to the nominal MBPO model, and the performance bound becomes identical. Since emaus denotes the TV distance, and we know from Pinkster's inequality that the KL-divergence bounds the TV distance, therefore we know that minimizing the KL divergence will hower-bound the performance in the nominal environment (Pinsker) 1964). With a very small the auxiliary model will focus on minimizing the KL divergence, and hence the TV distance. The more is increased, the less the loss function will focus on the KL divergence compared lo value minimization, hence becoming more pessimistic, but losing performance in the nominal environment (and probably everywhere). This trade-off between adversarial robustness and optimality is well studied in literature (Xu& & Mannor 2006 Moos et al. 2022). Empirical results on the relation between n and the KL divergence can be found in Appendix A.2

---

Published in Transactions on Machine Learning Research (12/2025)
MBPO
RMBPO in = 1)
RMBPO In 2)
RMBPO (n = 4.0)
RMBPO in 6)
RMBPO (n = 8)
RMBPO (n 10)
4000
4.3
3000
3.5
2000
2.5
44
-0.2
0.2
04
-24
-02
0.2
0.4
004
-22
0
0.2
0.4
-24
-02
0,2 0.4
-0.4
-02
0.2 0.4
4.4
-02
0
52
0.4
-04
422
0.2
0.4
Friction Coefficient
Figure 2: Comparing the robustness of RMBPO (ours) for multiple 7 values in Hopper-v4. The colorbar denotes mean episodic return.
5
Main Results
The following section aims to answer three main research questions empirically; (i) 'Can the auziliary model make a learned policy more robust?", (û) 'How does RMBPO compare against other robust RL approaches in simvllated locomotion lasks?" and (iii) 'Can we guán some insight in how the auziliary model modifies state tranistions to be pessimistic?". The first two questions are investigated in Section [5.1 and Section 5.2 where invostigate the effect of the auxiliary model, after which RMBPO is compared against SAC and robust natural actor-critic (RNAC) (Zhou et al.
2024 robust adversarial reinforcement learning (RARL) PPinto
et al. 2017), Quantal Adversarial RI. (QARL)
(Reddi et al. 2024), mixed Nash Equilibrium via Langevin 2020) and Estimate the Worst Kernel (EWoK) Gadot et al. (2024). The final question is investigated in Section 5.3 where we perform a limited case study on the
dynamics (MixedNE-LD) (Kamalaruban et al.
Hopper-v4 environment to examine which changes are made by the auxiliary model. For all our results, each algorithm trained five times using different initial sceds. We tune the rj hyperparameter by increasing its value until we note an unacceptable performance drop in the nominal environment. To illustrate this, we include a sensitivity analysis over range of 1) values for Hopper-v4 in Fig. 2 where it can be seen that starting from Notably, this trade-off is mostly a design choice rather than a hyperparameter with an optimal value, and depends on the desired result. However, for an excessive value of 7, the algorithm becomes unstable, as a too large uncertainty set makes the auxiliary model deviate far from the approximate transition model, causing highly unrcalistic transitions that make the loss of the SAC critic model explode, which in turn significantly affects auxiliary model learning.
8, the nominal performance does not surpass 3000 anymore, which we deemed too low.
5.1 Effect of the auxiliary model
We evaluate the hepothesis that our proposed auxiliary model aids MBRL algorithms in being more robust. Therefore, we evaluate RMBPO under two simultaneous distortions. For the MuJoCo environments, we follow Pinto et al. (2017) and perform a sensitivity analysis on the combination of torso mass and friction distortions. The main robustness results are displayed in Fig. 3 We include the results for Walker2d-v4 in Appendix A.T In all three tested environments, it is clear that RMBPO is more robust than MBPO, confirming that the auxiliary model aids the robustness in this setting. Additionally, we investigate the performance of our method in DMC Walker Run and Walker Walk. These results are shown in Fig. 4 and c). In both of there DMC tasks, RMBPO improves the robustness, compared to MBPO. Using the data displayed in Fig- and Fig 4 (a and c), we make an empirical cumulative density plot in Fig. 5 This figure demonstrates significant reduction in the number of distortion combinations that deliver a (very) low return. The improvement in robustness can be related to decrease in optimality in the nominal environment, as can be seen in our experiments in HalfCheetahh-v4 and DMC Walker Run. The trade-off between nominal optimality and robustness is controlled by the hyperparameter 7). This relates to the theory in Section 4.2 and is a well-known Irade-off that is affirmed by previous work (Lee et al. 2024 Gadot el al. 2024). It can be seen in the results that we choose a significantly higher value of v for Hopper-v4, compared to the other environments. This related to much lower variance on state transitions in the Hopper environment, we provide more details on this in Appendix F Additional results on the magnitude of n can be found in Appendix A.4
8

---

Published in Transactions on Machine Learning Research (12/2025)
RMBPO
4000
4.5
3000
4.0
3.5
2000
1000
2.5
-0.4
-0.2
0.0
0.4
Relative Normalized Friction
EWOK
QARL
RARL
4000
4.5
Torso Mass
3000
1000
2.5
-0.4
-0.2
0.2
0.4
-0.4
-0.2
0.2
0.4
-0.4
-0.2
0
0.2
0.4
Relative Normalized Friction
(a) Hopper-vi
MBPO
SAC
RMBPO (n = 0.5)
12000
10000
Torso Mass
8
8000
Mean Return
6
6000
4000
2000
2
1
2
Relative Normalized Friction
EWOK
QARL
RARL
12000
Torso Mass
8-
8000
6000
4000
2000
0
2
Relative Normalized Friction
(b) HalfChectah-vá
Figure 3: Comparing MBPO, SAC, RNAC, RARL, QARL, EWoK and RMBPO (ours) under two distortions MuJoCo Hopper-v4 and HaliChoctah-v4. Results for Walker2d-v can be found in Appendix A.1

---

Published in Transactions on Machine Learning Research (12/2025)
MBPO
SAC
EWOK
RMBPO (n 0.25)
800
Joint Damping
Mean Return
400
300
200
150
0.1
0.3
0.5
0.7
0.1
0.3
0.5
0.7
0.1
0.3
0.5
0.7
0.3
0.7
Torso Length
Walker Run, horizon= 1000
QARL
RARL
SAC MixedNE-LD
RMBPO (n 0.25)
250
Joint Damping
200
Mean Return
150
-50
0.3
0.7
0.1
0.5
0.1
0.3
0.5
0.7
0.3
0.7
Length
(b) Walker Run, horizon=500
MBPO
SAC
EWOK
RMBPO (n 0.25)
1000
Joint Damping
800
Mean Return
500
0.1
0.3
0.5
0.7
0.1
0.3
0.5
0.7
0.1
0.3
0.5
0.7
0.1
0.3
0.5
0.7
Torso Length
(c) Walker Walk, horizon=
QARL
RARL
SAC MiXedNE-LD
RMBPO (n 0.25)
Joint Damping
Mean Return
300
200
100
0.3
0.7
0.1
0.5
0.1
0.3
0.5
0.7
0.3
0.7
Length
(d) Walker Walk, horizon=500
Comparing MBPO, SAC, MixedNE-LD, RARL, QARL, EWoK and RMBPO (ours) under two distortions on DMC Walker- Walk and Walker-Run.
Figure
10

---

Published in Transactions on Machine Learning Research (12/2025)
Proportion Below Mean Return
Hopper-v4
Walker2d-v4
HalfCheetah-v4
RMBPO
RMBPO MBPO
RMBPO
MBPO
MBPO
0.8
0.6
0.0
500
1000
1500
2000
2500
3000
3500 0
1000
2000
3000
4000
2000
4000
6000
8000
10000
12000
Return
Return
Mean Return
Walker Walk
Walker Run
Proportion Below Mean Return
1.0
Proportion Below Mean Return
RMBPO MBPO
RMBPO
MBPO
200
400
600
800
1000
0
100
200
300
400
500
600
700
Mean Retum
Moan Retur
Figure 5: Empirical cumulative density plot demonstrating the robustness of our method. RMBPO success-
fully reduces the number of low-relurn episodes. Samples are combined distortions, displayed in Fig. 3 for MuJoCo and in Fig. (a. and c) for DMC.
5.2 Comparing with robust RL approaches
In this section, the robustness of RMBPO will be compared with six other algorithms. Crucially, we provide each baseline with all of the privileges needed by the respective algorithm during training. such as simulator access for the adversary (QARL, RARL) or the ability to resample multiple next states (EWoK). During evaluation, all algorithms are compared in an identical setting, using only the trained policy. As commonly done, use SAC baseline, this algorithm is trained using the same action noise RMBPO for 1M steps. Secondly, we compare with EWoK, algorithm that approximates KL uncertainty set by sampling multiple next states in the environment, after which importance resampling is performed, based on the value of these sampled states (Gadot et al.) 2024). Notably, this requires the environments to support sampling multiple next states for a cortain (s, ap-ir.. EWoK is trained for 1M stops in cach environment, except HalsCheetah--v, where performs 3M steps. Furthermore, RNAC is included as baseline as well using 3M samples all experiments, as described in the original paper (Zhou et al., (202)). For the most direct comparison, we compare against the integral probability metric (IPM) version of RNAC, since this version produces the strongest results. Additionally, and RART (Pinto et al. 2017). These algorithms are both based on the inclusion of an ad adversary that has direct access to the training simulator, allowing it to apply perturbation forces on the simulated locomotion problems. Notably, this means that these approaches assume a simulator that allows such functionality. We follow Reddi et al. (2024) and train these algorithms for 1M environment steps in DMC Walker, with an update-to-data (UTD) ratio of 1/3. However, the MuJoCo tasks, we increase the UTD ratio to the more common value of as 1/3 did not converge in these environments within LM steps. Also, the total step count was increased to 3M steps for HalfCheetah-v4, and was necessary for convergence to good policy. Finally, we included the MixedNE-LD algorithm Kamalaruban et al. 2020). For MixodNE-L.D, we also followed Reddi et al. (024) for the hyperparameters. RMBPO (ours) uses 125k samples for Hopper-v4, 300k for Walker2d-v4, 400% for HallCheetah-v4, and 200k for DMC Walker, leaving the data-efficient setting of MBPO unaltered. To compare the algorithms, our first experiment (MuJoCo environments) evaluates the mcan performance in two environments, under two simultaneous distortions. In the experiments shown in Fig. 3 RMBPO performs competitively, showing high degree of robustness compared to the baselines. Interestingly, F.WK outperforms method and
compare
approach with QARL (Reddi et al.) 2024
is common practice for SAC-type algorithms,
11

---

Published in Transactions on Machine Learning Research (12/2025)
Table 1:
The difference in transition predictions between go and pg. Values indicate angular velocity (w) or
lateral velocity (v). Expressed in rad/s and m/s respectively.
Torso (w)
Thigh Hinge (w)
Leg Hinge (w)
X- x-Coordinate Torso (v)
= 0.25
4.29e-4
3.02e-4
1.20e-4
2.89e-5
= 0.5
1.01e-3
5.10e-4
2.83e-4
2.11e-4
= 1
1.39e-3
1.14e-3
4.16c-4
2.01e-4
=2 2
3.16e-3
1.39e-3
1.22e-3
3.13e-4
= 4
4.08e-3
2.17c-3
1.79c-3
8.30c-4
all other algorithms by large margin in our tests of HalfChoctah-v4, We hypothesize that this difference can be attributed to the grounding of EWoK in realistic transitions, due to its reliance on re-sampling the simulator. In contrast, our method can potentially predict unrealistic transitions, especially at higher yf values. We noted during experimentation that in HalfCheetah-v4 specifically, larger v values led to instability in the value function, which led the SAC critic value to explode. Which prevented us from tuning y to a desired trade-ofT, we did in other enviromments. Furthermore, in the experiments shown in figure Fig. compare RMBPO with the mentioned baselines DMC Walker Walk and Walker Run. We follow Gadot et al. 2024 by distorting the Joint Damping and Torso Length parameters in these enviromments. To allow for a fair comparison with QARL, RARL and MixedNE-LD using the hyperparameters provided by Reddi et al. (2024). we follow their experimental setting and compare with these algorithms using a horizon of 500, instead 1000. These comparisons with a horizon of 500 can be found in Fig. | (b and d), the comparisons with the other basclines in the traditional sctting of 1000 steps can be found in (a and c). Additionally, we also compare with QARL, RARL and MixedNE-LD when distorting the same parameters as Reddi et al. (2024) in Appendix A.3 In all DMC experiments, RMBPO performs competitively compared to the baselines, often outperforming other algorithms. Notably, EWoK is often the strongest baseline.
5.3 What is the model learning (in H4pp
In addition to the quantitative results in this section, we perform a limited case study on how P& modifies the state transitions compared to the approximated nominal model pg. In Hopper-v4, the observa- tion space consists of 11 values describing the angles and angular velocities of the joints in the robot and the position and (angular) velocity of the top of the robot. For an exhaustive list, the reader is deferred to Todorov et al. 2012p. The goal of the enviromment is to use three rotors (in the foot, leg, and thigh) to make the robot move forward fast as possible, without falling. Therefore, we would expect the auxiliary model to modify the transitions in such way that the robot, moves forward more slowly and becomees more prone to falling. To examine the learned model, we display the four largest modifications that are made by the auxiliary model in Table [ It can be seen that increasing consistently increases the distance of the robust predictions from the predictions of the nominal model. The four state variables that are the most influenced by the adver- sarial updates are the angular velocity of the torso, the thigh hinge, the leg hinge and the x-velocity, More importantly, it is shown that the robust model incrcases the angular velocity of the torso, whilst it decreases the other two angular velocities. This aligns with the intuition of the system, since higher mobility of the torso makes the Hopper harder to control and therefore increases the probability of it falling. The results also demonstrate lower angular velocity on the actuated parts (such as the leg and thigh). Since these limbs used to control the robot, this makes
Figure 6: A render of Hopper-v4, an- notated with the four largest modili- cations of the auxiliary model. An in- crease in (angular) velocity is denoted with '++', a decrease with 1.
12

---

Published in Transactions on Machine Learning Research (12/2025)
the system harder to control. Finally, the lateral velocity of the robot is lowered, which directly reduces the step-wise reward of the environment. All these transition modifications are visually illustrated in Fig. D
6 Related Works
Many works focus on robust reinforcement learning in a tabular setting. These works include robust policy gradient (Wang & Zou 2022 Kumar ct al. 2024) and tractable approach to tackle non-roctangular
RMDPs (Goyal& & Grand-Clemeni, 2023). In a step lowards generality, Wang & Zou 2021] and Morimoto &
Doya 2005) consider robust reinforcement learning with function approximation on the inverted pendulum problem. Recently, Wang et al.] (2024) provide a robust RL algorithm with sample complexity analysis. As
many works exist that consider tabular robust RL, the reader is referred to Moos et al. (2022) for more
information on the topic.
In the context of high-dimensional state and action spaces, Pinto et al. 2017) propose adversarial RL for robustness. They show that an adversarial approach can make RL robust towards differences between the training and evaluation environment. In contrast to our work, the adversary in their methodology has access to paramcters of the simulator during training. Gadot et al. (2024) propose a methodology where multiple next states are sampled at each time step from a stochastic transition model. Subsequently, a single next state is resampled with an importance weight, based on the value of that state. Similar to this work, the KL uncertainty set is considered, however, their methodology requires simulator where multiple next states can be sampled at any time step. Rajeswaran et al. (2017) investigate an approach that, similar to ours, makes use of MBRL with ensemble world models. However their methodology explicitly requires training randomization over the distortion parameter that is evaluated. [Rigter et al. (2022) propose an approach similar to ours, with the goal of being robust to out-of-distribution data in offline RL. More recently, Zhou eet al. (202)) provide a model-free alternative to our work. Improved robust against transition dynamics is demonstrated in the MuJoCo environment, in addition to exhaustive thcoretical motivation. Recently, Liu et al. (2024) introduce robust RL algorithm, with theoretical guarantees on the robustness and sample complexity of their approach. However, their work is restricted to action robust setting. Additionally, the work of |Rigter et al. (0024) demonstrates the benefit of adversarial robustness in a reward-free RL setting. Queeney et al. (2024) introduce a novel uncertainty set, called Optimal Transport Perturbations, and demonstrate its effectiveness in improving robust and safety in a simulated robolics selting. Finally, Queeney & Benosman (024) also consider model- free robust RL to improve the safety of a learned policy.
7 Conclusion and Future Works
This work proposed a novel approach for robust adversarial RL in an online, high-dimensional sctting, We have motivated the use of an auxiliary model to tackle the inner-loop optimization problem of the RMDP formulation and provided a version of this auxiliary model, based on the KT. uncertainty set.. This pessimistic auxiliary model was then implemented in a practical MBRL algorithmm,, called RMBPO. Our experiments demonstrate that the auxiliary model improves the robustness of MBRL, while remaining in the same data-efficient setting. Secondly, our hot was compared to other recent model-free robust RI. approaches. RMBPO matched or outperformed the robustness of these algorithms in most experiments using significantly less data. Finally, we performed a limited case study which interprets the way in which the auxiliary model helps policy robustness. A limitation of our work is that we compute the KL divergence w.r.t. the approximate model instead of the real nominal model, future research could try to quantify the error that arises from this practical approach. Another limitation is the fixed Lagrangian hyperparameter, which does not tackle the constrained problem methods such as the modified method of differential multipliers (MDMM) might be an interesting research direction (Platt & Barr 1987p. Furthermore, it would be interesting to further investigate the performance discrepancy between our method and EWoK in HallCheetah-v4, and improve the stability of our method for high values. A potential direction to improve the stability might be to make the pessimistic model not fully rational by adding noise to the state modifications, as proposed by Reddi et al. (2024) A final limitation that we would like highlight is the fact that we evaluate in simulation toaea-to-siii to-simulation setting, leaving real-world experiments for future work. This potentially leaves some challenges untackled, such as the question if our
hard constraint. We believe that improved Lagrangian
13

---

Published in Transactions on Machine Learning Research (12/2025)
method generalizes to real-world model inaceuracies potential variability of system troughout single episode. As additional future work, we want to tackle the setting of very noisy nominal MDPs, such as explored in Gadot et al. (2024). Other interesting areas for future work could include policy mixing between a traditional and a robust policy, to limit the potential downside of not exploiting the environment optimally. Furthermore, might be interesting to look at way to formally ensure that the auxiliary model remains within the desired uncertainty set, combined with theoretical guarantoes on the robustness of the policy, as we believe that this vital step towards RL in industrial applications.
Acknowledgments
This work was supported by the Research Foundation Flanders (FWO) under Grant Number 1SHAI24N.
14

---

Published in Transactions on Machine Learning Research (12/2025)
References
Paul Christiano, Zain Shah, Igor Mordatch, Jonas Schneider, Trevor Blackwell, Joshua Tobin, Pieter Abbeel, and Wojeiech Zaremba. Transfer from simulation to real world through learning deep inverse dynamics model. arXiv preprint arXiv:1610.0855 03518, 2016.
Uri Gadot, Kaixin Wang, Navdeep Kumar, Kfir Ychuda Levy, and Shic Mannor. Bring your own (Non-robust) algorithm to solve robust. MDPs by estimating the worst kernel. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 14408 14432. PMLR, 21 27 Jul 2024. URL https://prooceeings... mmlr. mlr. press/v235/
htt.:/gs.........e22/////2. https///dot22.aaaa .tmll.
Vineet Goyal and Julien Grand-Clement. Robust markov decision processes: Beyond rectangularity. Math- ematics of Operations Research, 48(1):203 226, 2023.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104 2023.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae learning basic visual concepts with a constrained variational framework. In International conference on learning representations, 2017.
Zhaolin Hu and L Jeff Hong. Kullback-leibler divergence constrained distributionally robust optimization.
Available at Optimization Online, 1(2):9, 2013.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32, 2019.
Parameswaran Kamalaruban, Yu-Ting Huang, Ya-Ping Hsieh, Paul Rolland, Cheng Shi, and Volkan Cevher. Robust reinforcement learning via adversarial training with langevin dynamics. In H. Larochelle, M. Ran- zato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neurul Information Processing Systems, volume 33, pp. 8127 8138. Curran Associates, Inc., 2020. URL htttps://proceeeee/ee htwpooceediinnn.nniisss... cc/ pdf
paper
Navdeep Kumar, Esther Derman, Matthieu Geist, Kfrr Y Levy, and Shie Mannor. Policy gradient for rectangular robust markov decision processcs. Advances in Neural Information Processing Systems, 36, 2024.
Bruce D Loe, Thomas TCK Zhang, Hamed Hassani, and Nikolai Matni. Porformance-obuustness tradcoffs in adversarially robust control and estimation. IEEE Transactions on Automatic Control, 2024.
Guanlin Liu, Zhihan Zhou, Han Liu, and Lifeng Lai. Efficient action robust reinforcement learning with probabilistic policy execution uncertainly. Transactions on Machine Learning Reseurch, 2024. ISSN 2835- 8856. URL httpps://oeenrevier net wttee/oouummmdd-22eeeez..
Thomas M. Moerland, Joost. Broekens, Aske Plaat, Catholijn M Jonker, et al. Model-based reinforcement learning: A survey. Foundations and Trendso in Machine Learning, 16(1):1-118, 2023.
Janosch Moos, Kay Hansel, Hany Abdulsamad, Svenja Stark, Debora Clever, and Jan Peters. Robust reinforcement learning: A review of foundations and recent advances. Machine Learning and Knowledge Extraction, 4(1):276-311 22576631,,20 2022.
Jun Morimoto and Kenji Doya. Robust reinforcement learning. Neural computation, 17(2):335 359, 2005.
Mark S Pinsker. Information and information stability of random variables and processes. Holden Day, 1964.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. In International Conference on Machine Learning, pp. 2817-2826. PMLRR, 2017.
15

---

Published in Transactions on Machine Learning Research (12/2025)
John Platt and Alan Barr. Constrained differential optimization. In D. Anderson (ed.), Neural Information Processing Systems, volume 0. American Institute of 1987. URL /ttppppp/////p/reedddd.. neurips.
cc/paper_ http:////psssss.aaaaaataaaaaaaaa
James Queeney and Mouhacine Benosman. Risk-averse model uncertainly for distributionally robust safe reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
James Queeney, Erhan Can Ozcan, Toannis Paschalidis, and Christos Cassandras. Optimal transport per- turbations for safe reinforcement learning with robustness guarantees. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL
Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, and Sergey Levine. EPOpt: Learning robust ncural network policies using model ensembles. In International Conference on Learning Representations, 2017. URL
Aryaman Reddi, Maximilian Tölle, Jan Peters, Georgia Chalvatzaki, and Carlo D'Eramo. Robust adversarial reinforcement learning via bounded rationality curricula. In The Twelfth International Conference on Learning Representations, 2024. URL
Mare Rigter, Bruno Lacerda, and Nick Hawes. Rambo-rl: Robust adversarial model-based offline reinforce- ment learning. Advances in neurol information processing systems, 10116022-16999 2022.
Mare Rigler, Minqi Jiang, and Ingmaar Posner. Reward-Iree curricula for training robust world models.
The Twelfth International Conference on Learning Representations, 2024. URL https://openreries:
Andrei A Rusu, Matej Večerík, Thomas Rothörl, Nicolas Heess, Razvan Pascanu, and Raia Hadsell. Sim- to-real robot learning from pixels with progressive nets. In Conference robot learning, pp. 262-270. PMLR, 2017.
Laixi Shi and Yuejie Chi. Distributionally robust model-based offline reinforcement learning with near- optimal sample complexity. Journal of Machine Learning Research, 25(200):1 91, 2024.
Adam Stooke, Joshua Achiam, and Picter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning, pp. 9133 9143. PMLR, 2020.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy P. Lillicrap, and Martin A. Riedmiller. Deepmind control suite. CoRR, abs/ 10s/1800000600, 2018. URL battp://arxir ortp:///orgabbb/1/1011 00690
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RS) International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012. doi: 10.1 10.1109/TROS.... IRR0..ROS.2000 TTt.T/........880000000 6012.6386109.
Yudan Wang, Shaofeng Zou, and Yue Wang, Model-free robust reinforcement learning with sample com- plexity analysis. In Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence, UAI 24, 2024.
Yue Wang and Shaofeng Zou. Online robust reinforcement learning with model uncertainty. Advances in Neural Information Processing Systems, 34:7193 7206, 2021.
Yue Wang and Shaofeng Zou. Policy gradient method for robust reinforcement learning. In International conference on machine learning, pp. 23484 23526. PMLR, 2022.
Wolfram Wiesemann, Daniel Kuhn, and Berç Rustem. Robust markov decision processes. Mathematics of Operations Research, 38(1):153 183, 2013.
Huan Xu and Shie Mannor. The caantuees-performaae tradeoff in markov decision processes. In B. Schölkopf, J. Platt, and T. Hoffman (ods.), Advances in Neural Information Processing Systems, vol- ume 19. MIT Press, 2006. URL
16

---

Published in Transactions on Machine Learning Research (12/2025)
Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund. Sim-to-real transfer in deep reinforcement learning for robotics: survey. In 2020 TEEE symposium series on compulational intelligence (SSCT), pp. 737 744. JEEE, 2020.
Ruida Zhou, Tao Liu, Min Cheng, Dileep Kalathil, PR Kumar, and Chao Tian. Natural actor-critic for robust reinforcement learning with function approximation. Advonces in newral information processing systems, 36, 2024.
17

---

Published in Transactions on Machine Learning Research (12/2025)
A
Additional Results
A.1
Evaluation on Walker2d-v4
MBPO
RNAC
SAC
RMBPO 0.5)
4000
3000
2000
1000
0.5
0.0
0.5
0.5
0.0
0.5
Relative Normaizzed Friction
1.0
0.5
0.0
0.5
0.5
0.0
0.5
EWOK
QARL
RARL
5000
4000
Mean Return
3000
4
2-
1000
Relative Normalized Friction
Figure 7: Walker2d-v4
A.2 Empirical effect of on KL divergence
Hopper-v4
Walker2d-44
1.4
Data points
Data points
1.2
Linear fit
Linear fit
0.04
1.0-
KL Divergence
KL Divergence
0.8-
0.03
0.6
0.02
0.4
0.01
0.2
1
0.0
0.1
0.3
0.5
n Value
n Value
(a) Hopper-vd
(b) Walker2d-v4
Figure 8: The KL. divergence between the approximated nominal model and the auxiliary model, in function of Linear fit included for visual reference.
18

---

Published in Transactions on Machine Learning Research (12/2025)
A.3
Extra results in Deepmind Control Suite
QARL
RARL
SAC MixedNE-LI
RMBPO (n 0.25)
300
14
Mass of Torso
250
Mean Return
150
100
0.5
1
0.5
0.5
0.5
1
Friction Coefficient
Walker Run, horizon=500
QARL
RARL
SAC MixedNE-LD
RMBPO (n 0.25)
500
Mass of Torso
450
12
Mean Return
10
400
350
0.5
1
0.5
0.5
D.5
Friction Coefficient
(b) Walker Walk, horizon=500
Figure 9: Evaluation of DMC Walker Walk and Walker Run the disturbance setting of Reddi et al. (0024)
19

---

Published in Transactions on Machine Learning Research (12/2025)
A.4
Effect of 17 on robustness
MВPО
RMBPO (n 0.04)
RMBPO (n = 4)
4.5
Mass
4.0
3000
Mean Retum
83.5
2000
3.0
1000
2.5
-0.2
Relative Normalized Friction
(a) Hopper-va v4
MBPO
RMBPO (n 0.25)
RMBPO (n 0.5)
8-
4000
Torso Mass
Mean Return
3000
4
2000
2-
1000
0.5
0.0
0.5
1.0
0.5
0.0
0.5
1.0
-0.5
0.0
0.5
1.0
Relative Normalized Friction
(b) Walker2d-vl
MBPO
RMBPO (n 0.25)
RMBPO (n 0.5)
12000
10000
8
Torso Mass
8000
6000
6
4000
4-
2000
0
1
0
1
2
Relative Normalized Friction
(c) HalfCheetah-vd
Figure 10: Influence auxiliary model on policy robustness under two distortions. All experiments demon- strate that larger 77 increascs robustnoss, possibly the cost of optimality.
20

---

Published in Transactions on Machine Learning Research (12/2025)
Proportion Below Mean Return
Hopper-v4
Walker2d-v4
HalfCheetah-v4
= 0.5
7= 0.5
m 0.25
A 0.25
- 0.25
0.8
0.6
0.0
500
1000
15:00
2000
25.00
3000
35000
1000
2000
3000
4000
2000
4000
6000
8000
10000
12000
Figure 11: Cumulative proportion of samples below a certain mean return. A higher y value successfully reduces more low-return episodes. Samples are combined distortions, identical Fig.
Table 2: Hyperparameters
Hyperparameter
Hopper-v4
Walker2d-v4
HalfCheetah-v4
DMC Walker
4
0.5
0.25 / 0.5
0.25
X
le-4
le-4
le-4
le-4
Total environment steps
125k
300k
400k
200k
B Toy Experiment Details
A summary of the algorithm that was used in the toy experiments provided Algorithm
Algorithm Supervised Pessimistic Distribution Learning with an Auxiliary Model
Choose an arbitrary value function R" R Initialize dataset D with samples from the training distribution Initialize nominal parameters
4: Initialize pessimistic paramcters 2/
5: while improving do
6: Update model parameters on environment data: 0 Ap Vo,,(00,
7: end while
while improving do
Update model parameters v according to Eq. <- Aa J D. p.po,r)
10: end while
Furthermore, to demonstrate that the methodology is not exclusive to Gaussians, we also perform exper- iments on categorical distributions. dataset was generated by sampling from a categorical distribution with 10 categories, with the following (randomly generated) probabilities: [0.0364, 0. 0.1044, 0.1335, 0.1107, 0.0668, 0.1367, 0.0558, 0.1067, 0.0981, 0.1529). Identical to the Gaussian experiments, we follow Algorithm 2 learn nominal and an auxiliary model from this data. Instead of parametrizing mean and variance, we parametrize the 10 logits. We perform experiments with two value functions. The first value function provides value of for even categories (0, 2, ...) and -1 for odd categories, the results are shown in Fig, 12a The second value function just returns the number of the category (e.g. category 4 has a value of 4),
these results are shown in Fig. 12b
C Hyperparamters
We "tune" n by performing a sweep and taking the largest value for which we still find an adequate nominal performance. Note that this hyperparameter is actually more of design choice, since it trades optimality
21

---

Published in Transactions on Machine Learning Research (12/2025)
7 = 0.0
= 0.2
0.7
0.20
Probability
0.15
0.10
0.00
Category
Category Auxiliary
Category
Nominal
(a) Value function rewards even categories (0, 2, 4, 8).
=
0.0
= 0.05
0.2
0.20
0.15
Probability
0.10
0.05
0.00
Category
Category
Category
Nominal
Auxiliary
(b) Value function equal to the label the category (1, 2, 3, ...).
Figure 12: Difference between nominal categorical model and pessimistic categorical model.
for robustness. The desired magnitude of 7) influenced by the variance on transitions of the nominal environment. Namely, large transition variance already allows for meaningful pessimism introduced by de at lower 1 values. Environments with less variance need higher 77 (and therefore higher KL) to introduce pessimism. For more details on this, see Appendix F The pessimistic model lcarning rate (Aa) is set to of the normal MBPO model learning rate, this signilicantly reduces variance on the return during training. Note that we use the same amount environment steps as MBPO in all environments.
All other hyperparamoters remain identical to MBPO oJanner et al. 2019), the auxiliary model go also has the same architecture single model of the the MBPO ensemble world model.
D Implementation details and reproducibility
Following related work (Zhou et al. 2024), we add uniform noise to the action: <- at +1-5- 5e 3). Since this action noise is invisible to the agent, it introduces stochasticity in the MDP. Inspired by the existing MBPO world model, we standardize the outputs of ps before providing them as inputs to Jø. this showed incremental stability improvements
some training runs. As proposed in appendix A.1 of Rigter et al.) (2022), we subtract V.8((s) (s) as a bascline from the return in Eq. 7 this does not influence the expectation of the gradient but significantly reduces its variance. Note that MBPOO/RMBR
does not employ value approximate this with on-policy samples from the Q-value network. We want to highlight that Po is an ensemble of seven neural networks in MBPO and RMBPO, in contrast, Pe is single neural network. This provides two advantages. First, the computational effort is only increased by a small fraction (not doubled). Secondly, the ability Po to capture the epistemic uncertainty during training is maintained by including samples the ensemble as input to gy-
network directly, however,
Experiments were run on Ubuntu20.04 (Docker) machine with a single NVIDIA Quadro RTX4000 GPU, two CPU cores, and 38GB of memory. We provide the trained weights of the learned policies as supplementary
22

---

Resampling Published in Transactions on Machine Learning Research (12/2025)
Table 3: The environment settings for training our method and the baselines.
Algorithm
Simulator Access
RARL
Yes
No
QARL
Yes
No
MixedNE-Ie
No
No
EWoK
No
Yes
RNAC
No
No
SAC
No
No
RMBPO (ours)
No
No
materials, together with the modified enviromments and an evaluation script This allows for a clear comparison with our research. We choose to distort the same model parameters as Pinto et al. (2017) and Zhou ct al. (024)) to add perspective to the rcsults and case future benchmarking in the community, also, this avoids cherry-iccking the best conditions for RMBPO. To ease implementation, we also release the source code of the toy experiment. Furthermore, we include a relevant part of the source code in Appendix G this code should be regarded as pseudo code and serves to provide necessary implementation details. The authors are not able to release the full source code of RMBPO at the time of submission of this paper, however, the reader is encouraged to contact the first author of this work with any related questions.
To implement the robust baselines, we used the code provided by the auth for RNAC (Zouu et al. 2024
Simílarly, we directly use the author code for EWoK. Following Gadot et al. (2024) we always use a normal distribution with a standard deviation 0.2 for EWoK and we fix the temperature parameter k to 2. For all other baselines (QARL, RARL, MixcdNE-LD)), we use the implementation provided by Reddi et al. (2024) This means that both RARL and MixedNE-LD instead of TRPO and TD3 respectively. As all these algorithms were already tuned for DMC, we did not have to do any tuning there. However, for the MuJoCo experiments, we changed the UTD ratio from 1/3 to and the actor learning rate from 1e 4 to 3e 4, as the lower UTD and learning rate failed to converge after 1 1M steps in MuJoCo. Additionally, we allowed a maximum adversary force of 5 in MuJoCo for RARL, compared to for QARL, following the original RARL paper Pinto et al. 2017).
'modernized' versions, using SAC the base algorithu
It important to note that we (a) always run the basclines in their respective training settings from the original publication, this means for example that RARL-based adversary policies have the privileged ability to apply forces in the simulator during training; and (b) evaluate all methods (ours and baselines) in a separate setting, where no adversaries are used and no additional noise is added anywhere. The evaluation setting consists of the default MuJoCo and DMC enviromment, where the parameters under evaluation (e.g. friction and mass) are set to a fixed perturbation value throughout a full episode. For a summary on these settings, please see Table 3
E Probability bound on KL divergence
In this section, we explore the relationship between the expected approximate uncertainty set and the approximate uncertainty set. We begin by employing the Markov incquality as worst-case bound, which provides probabilistic limit on the size of the approximate uncertainty set. This analysis is followed by empirical measurements that demonstrate how minimizing the expected divergence can effectively bound DKl(ge(s, ())lpo(ss ())l a)) with a certain confidence.
23

---

Published in Transactions on Machine Learning Research (12/2025)
E.1 Applying the Markov Inequality worst- bound
The well-known Markov inequality stales that for nonnegative random variable X and real number / >>> 0:
E X
P(X 2 1) S
To apply this to the auxiliary model loss function, we note that the KL term of Eq. (and Eq. 7 does not depend on s' or r, therefore we have:
(s (t,))) aa DKLlas(e (po a))] E B aaaan
(s,0))saaa)))) a)llpo(s,a)))
As the KL divergence is calculated between two continuous distributions, it is nonnegative everywhere. This means that can apply the Markov inequality, for any >> O:
(a a)) 21) Eondy a. aove (DLlow(s,a))lpdc
(11)
t
which provides us with a bound that limits the probability (defined by P())) of encountering states outside of the desired (approximate) uncertainty set. Another known form of the Markov inequality can be stated by setting = t/EEndd paann [DKL a))lpo(s,))) this allows us to rewrite Equ. I1 as follows:
P ( a)) B aant [DKL(ov(s, (1)e0000
(12)
The form in Eqn. 12 is useful, since it can compare how well algorithms fit the expected uncertainty set, regardless of the size of that expected uncertainty set.
E.2 Empirical Measurements
We now measure the empirical performance of RMBPO. Figure 13 shows the relationship between the expected KL divergence and the probability of encountering larger KI. divergences for specific (s, a--ammlles. Note that the probabilities remain siguificantly below the Markov inequality bound in all environments. The probability of encountering a value that is larger than double the measured expectation is already lower than 10%. Additionally, for Hopper-v4, we measure this quantity for large array of n values in Figure 14 Again, for all mcasured 1 values, the probabilitics remain significantly below the Markov incquality bound and the probability of encountering double the expectation remains below 10%. For visual clarity, all plots are shown in function of instead of t, however, we also include Table 4 that contains all unscaled KL values.
24

---

Published in Transactions on Machine Learning Research (12/2025)
Hopper-v4
Walker2D-v4
HalfCheetah-v4
0.20
0.2523.6%
6.7%
15.4%
(KK [ E[KL))
0.15
0.15
0.20
11.3%
11.5%
0.15
0.10
0.10
10.3%
0.10
6.2%
5.9%
0.05
3.6%
0.05
3.1%
0.05
.3.3%
1.8%
1.2%
0.9%
2.4%
1.7%
1.9%
1.4%
0.00
0.00
0.00
2
3
5
9 10
1
2
3
8
9
10
2
6
7
8
9
10
Measured
.....*
Markov Inequality Bound (2)
Figure 13: Probability of encountering values larger than factor of the expected divergence. The n values of 4, 0.5 and 0.5 were used for Hopper-v4, Walker2d-v4 and HalfCheetah-vá respectively. We used P(KLe 2 * E(KL) as shorthand notation for P(Dxz (ow ())lpo(sss)))))))o))))))))) a)) 2 E. B,ddar= (DxL ()) a)lloo (s)))
0.6
Markov Inequality Bound ()
Markov Inequality Bound Hp)
0.25
9.0
n-
0.0.0
7= 4.0
plKLs [*5(KL])
0.25
0.4
0.20
0.15
0.2
0.1
0.05
3
E
Figure 14: Hopper-v4, multiple n values. The probability of encountering values larger than factor of the expected divergence. We used P(KL, 2 E(]] as a shorthand notation for ()) 2 7. Eandaaaa (Du (8,a)))llle (s))
25

---

Published in Transactions on Machine Learning Research (12/2025)
Table 4: The expected KL divergence and the maximum KL divergence between Py and Ge. Note that higher 7 values monotonically increase the KL divergence all our measurements. Mcasurements are computed a holdout set during training, values are reported the average over the final 10% of training.
Min KL
Expected KL
Max KL
Hopper-v4 1 0.25
7.53e-04
1.77e-02
1.06e-01
Hopper-v4 = 0.5
3.13e-03
3.92c-02
1.33e-+00 00
Hopper-v4 r 1.0
6.81e-03
9.07e-02
6.52e+000
Hopper-v4 n 2.0
1.73e-02
2.69e-01
7.80e+00
Hopper-v4 7 4.0
5.55e-02
7.84e-01
3.29e+01
Walker2d-v4 1) 0.25
4.87c-04
1.32e-02
1.09e++00
Walker2d-v4 1] 0.5
1.25e-03
4.91e-02
3.35e++00
HalfCheetah-v4 0.25
3.95e-04 1.49e-03
1.08e-02 3.73e-02
1.30e+000
HalfCheetah-v4 f) = 0.5
4.58e++00
Table 5: The measured variance of the training environment P and the predicted variances of the approximate model po and the auxiliary model Dv- The variance is defined over next states and rewards, given the same current state and action (for on-policy rollouts). Variance is calculated independently per state dimension, average over dimensions is reported.
P
pe
9y
Hopper-v4
4.67c-06
2.57c-06
2.58c-06
Walker2d-v4
2.39e-04
2.68e-04
2.69e-04
HalfChectah-v4
2.51e-04
8.19e-04
8.22c-04
F More details on the hyperparameter 7)
A natural question that arises from the results of this work is why the hyperparameter 7) is set at significantly higher value for Hopper-v4 than Walker2d-vd or HallCheetah-vd. This arises from the siguificantly lower variance state transitions in Hopper-v4 compared to the other two environments, as shown in Table 5 The KL divergence between two distributions increases much faster for low-variance distributions than high-varinn variance distributions, given a certain difference in their means. This means that, to allow significant average changes between the approximate model and the auxiliary model, 1) should be set higher in Hopper- v4. We quantify this reasoning Table o which confirms that Hopper-vá needs relatively larger 1) values to allow certain average difference on state transitions, compared to Walker2d-vá or HallCheetah-vd. The assumption that larger setting allows for larger KL divergences was already confirmed in Table 4
26

---

Published in Transactions on Machine Learning Research (12/2025)
Table 6: The average absolute differences of predictions between Iv and pe (measured). The five state dimensions with the largest average absolute differences are shown per environment. Note the significantly lower values of Hopper-v4 than the other environments, the same r) levels. Differences are measured over holdout set during training, reported metrics averaged over the final 10% of training.
Avg. diff.
Avg. diff. 2 Avg. diff. 3 Avg. diff. 4 Avg. diff.
Hopper-v4 7 0.25
4.29e-04
3.02e-04
1.20e-04
2.89e-05
2.53e-05
Hopper-v4 7) = 0.5
1.01c-03
5.10e-04
2.83e-04
2.11e-04
1.41e-04
Hopper-v4 " 1.0
1.39e-03
1.14e-03
4.16e-04
2.02e-04
1.48e-04
Hopper-v4 n 2.0
3.16e-03
1.39e-03
1.22e-03
3.13e-04
3.11e-04
Hopper-v4 7 4.0
4.08e-03
2.17e-03
1.79e-03
8.31e-04
3.19e-04
Walker2d-v4 1) 0.25
9.92e-03
5.16e-03
2.29e-03
1.63e-03
1.12e-03
Walker2d-v4 !7 0.5
2.39c-02
1.13e-02
5.08c-03
4.06c-03
2.71c-03
HalfCheetah-v4 17 0.25
4.65e-04
4.60e-04
4.50e-04
2.91e-04
1.51e-04
HalfCheetah-v4 17 0.5
1.26c-02
8.69c-03
7.34c-03
4.39c-03
2.14c-03
G Code Snippet
Listing provides implementation of the function to compute the gradient of the auxiliary model (as defined by Eqn. 7 and upgrade the model parameters according to this gradient.
update_rmbpo
auxiliary_moda model: TrainState, mle :le_model: TrainState, batch: DatasetDict, obs_meant imp.ndarray obs_std: jup. ndarray,
critic: TrainState, actor: TrainState, rng: tax.Array
eta: float,
elites: jax.Array
termination_fas fn: Callable [[jap. [allable[[jpp.naarrr]] ndarrayl, jap. ndarrayl,
12
-> Tuple[TrainStan Dict[str, float], jap ndarray, jnp ndarrayl:
14
state batch observvaiiios
16
def mux_nodel_params: Params,
aux
-> Tuple ndarray, [str, aap.rdarray]] policy jax lax maxxsttop_ae stop_ groppggradeet (actor apply_ (a"aaams" actor params), state))
local_rng, seed randoa split (rng)
on.policy_actions = pollicy..... mode ()
pred_mle_dict jax stop_ (
mle apply_fn(
24
("params": mle all_ params),
atate,
26
on.policy_actions,
obs_mean, oba_std,
)
Sample mixture
27

---

Published in Transactions on Machine Learning Research (12/2025)
mie_means pred_ _erd_mle_dis_______ _distt.........................d mean
mle_ var = pred_mle dist variance
batch, size
mle means shape [0],
shape [1]
local_rng seed randon haaarrando.sspit (local_rng)
Only select from elites
random_ elite, indices max.random w.w.domm....aii
seed, (batc_bbach__si200
ninval=0, maxval=elites shape [0]
)
40
randon randon_ indices elites trandon_elite iiattee_ndiess]
mle_meand mle_means [jup. arange [batch_sizee
:]
42
mle_var mle_var [jmp- (ratchebaacc_ randos_indices, :] 1e-8
pred_mle_dist = distrax. KultivarratteNormallaag pocmmle_mean. scale_diage sqrt (mle_var) var)
)
# Forward auxiliary model tred_stateddiie dist auxiliary wwo.eel.pppl............ apply_ppp......... fn
("params": aux_model_parans
state, on.policy_action,
mle_means,
jup log (ale_var),
53
abs obe_mean [0], obs_std[0],
)
KL loss
kl _1_loss = jap mean (preed_ttate_ddee kl_divergence (pred fered_mme__ee m_t_mle__de_
Robustness loss
local_rng seed
spocal
prediction pred_state_diett sample (seed=seed)
log_probe poo_ (japaaaaa......t..graaiaaaaaa (poopggaddeet(peeitttti))))))))) (prediction)) pred_state, pred_revard prediction :-1], prediction(.... -1]
local_rng, seed jax rww.oa. spw.tnmnllptttllalpppa.oo
64
next_action jax lax stop_gradient(
actor.apply. (("params: actor parans), pred_state) ) mode((
66
Baseline
q. g_baseline jax. lax. stop_gradient
critic pritic.pply... fn ((" pa"aaaa"s" critic, params), state, on-policy_actionn
) (in
Next 9
next_q = jax.lax.
wax.xxxxxooo. stop_gradient antii..ppll__n(
critic. params), prad_ next_action)
m.ii((xxs=))
terminals termination_fi (pred__stppddd_ttattet
critic value = prwa_reward 0.99 next_q * jnp logical not (terminals)
advantage critic_value b_baseline
adv_mean, adv_std jnp mean (advantage) jnp. std (advantage)
79
aoraaai (advantage adv_mean) (adv_ std 1e-8) robustness_ jup mean
81
jax lax. stop stop-gradient Lnormalized_ddvantaae
log_probs
combined_lore = xl_1oss + eta * robustness_loss return combined_loss
grads jax. grad (rmbpo Lospoo__oss_____ fn) (auxiliaryymodel params)
new new_mowwee _v_moddel auxiliary model. apply_graddeeee -grddiegrddntsss (grads-grads
return new_moded
Listing 1: RMBPO Auxiliary Model Update Function.
28

---

Published in Transactions on Machine Learning Research (12/2025)
Where the forward function of the auxiliary model (as called by auziliary_model.opo wwwpe//aaoallalao......o. is defined by Listing 2
class Simpleeaassiimkkkllleeryoooa (nn Module)
madden_dimst int
num num_layers: int
obs_dim int
Onn. compact def ___all__
self,
Real observation vector observations: jnp. .ddrrry,,,,
action: jup. ndarray, mle nle_means: jup. .darray,
Real action vector Mean of pred. nominal distr.
12
mle_logvars: jap. ndarray, mean: jnp.
Log variances of pred. nominal distr. Current observation/action batch means Current observation/actior batch stds
14
std: adarray
distrax distrax.Distre Distribution
16
Dutput of network
and variances of state vector and reward
18
layers [self.hiddes hidden whidhenndddsss
(self. num_layers) + [2*(self. obs_dim+1)] Standardize the real state and action vector
state = jmp concatenateeeeeeee poncaatentte(ooberraaaaa (lobservations, action], axis=-1) state_inp (state mean) (std 1e-8)
# Standardize the next state predictions of the nominal (MLE) model mean_ (mle [ma.mmenns...... :-1] mean self obs_ dim])
stdl: self [is_dim]] + 1e-8
Concatenate real state + action and nominal (MLE) predictions input jup. concatenate(
27
state_inp,
mean mean_pred,
jap eppand_ dims (ale_means[_.. -1], axis=-1), mle_logvars,
1, arig=-1,
outputs MLP (layers, activations-m.. silu, aetivate_final-Palt (input) means aed_rdeardd, logvar = jnp.jjpiiie split 2, -1)
38
Only learn to predict deltas means_ aams_and_rewaree logvar jax. www..ttooogradieettteee_eeee # Return pessimistic (s', r) distribution
jax lax. stop (mee_gradienttmle_meaaa + means_and_rewane
40
logvar
return distrax Matt www.tiaarrrttttttrraa..aag
loc=means and_ rewards, scale_dias dip.-jnpppp. exp (0.5 logvar)
Listing The forward function of the auxiliary model.
29

---

Published in Transactions on Machine Learning Research (12/2025)
H Qualitative examples on Po
Hopper-v4
400000
Empirical P (hist)
Density
Density
200000
Empirice rical mean
10000
1.4185
1.4190
1.4195
1.4200
1.4205
-0.0845
-0.0840
00.8335
-0.0830 -0.0825 -0.0820
Value
Value
Dim
Dim
20000
Density
15000
Density
10000
10000
5000
-0.4755-04750--
0.3015-0.301000.30000
Value
Value
Dim
Dim
20000
750
Density
Density
500
250
-0.680
-0.678
-0.676
-0.674
-0.672 -06670 -0.668
2.126
2.128
2.130
2.132
2.134
Value
Value
Dim
Dim
1500
Density
Density
1000
0.966
0.968
0.970
0.972
0.974
-0.37
-0.36
-0.35
-0.34
-0.33
-0.32
Value
Value
Density
Density
25
-0.36
-0.35
-0.34
-0.33
-0.32
-0.31
1.71
1.72
1.73
1.74
1.75
1.76
1.77
Value
Value
Dim
10
75
Density
50
6.12
Value
Figure 15: Qualitative evaluation fit of the approximate model pe to the nominal simulator P. Note that the log standard deviations of po are clipped above -5. These plots were created on random transition during an evaluation run.
30

---

Published in Transactions on Machine Learning Research (12/2025)
HalfCheetah-v4
Empirical (hist) m
Dim
Density
200
Pa
Density
Empirical mean
50
0
-0.060
-0.055
-0.050
-0.045
-0.040
-0.035
Value
Value
Dim 2
Dim
Density
50
Density
0.20
0.22
0.24
0.26
-0.08
-0.06
-0.04
-0.02
0.00
Value
Value
Dim 4
Dim
Density
Density
25
25
-0.26
-0.24
-0.22
-0.20
-0.38
-0.36
-0.32
-0.30
-0.28
Value
Value
Dim 6
Dim
Density
Density
25
25
0.04
0.06
0.08
0.10
0.12
0.14
0.08
0.10
0.12
0.14
0.16
Value
Value
Dim 8
Dim
Density
Density
5
1.7
1.8
1.9
2.0
1.1
1.3
1.5
1.7
Value
Value
Dim 10
Dim 11
Density
2.5
Density
1
0.0
0
1.4
1.6
1.8
2.0
2.2
7.5
8.5
9.0
Value
Value
12
Dim
Density
Density
0.5
0.5
-12.0
-115 -1.0
10.5
10000
-9.5
-9.0
-8.5
7
10
11
Value
Value
Dim 14
Dim 15
Density
Density
2.5
2.5
0.0
0.0
-10.5
Value
Value
Dim 16
Density
-1.5
-1.0
-055
0.0
Value
Figure 16: Qualitative evaluation fit of the approximate model po to the nominal simulator P. These plots were created on random transition during an evaluation run.
31


Reviews:
[{"review_id": "lpfUglSlAP", "reviewer": "Reviewer_Le1w", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Robust Model-Based Policy Optimization (RMBPO), a model-based reinforcement learning (MBRL) algorithm designed to improve robustness to environment perturbations while retaining sample efficiency.\n\n- The key contribution is an auxiliary adversarial transition model, which is updated by minimizing returns under the current policy while remaining close (in KL divergence) to the nominal learned world model.\n- This is integrated into MBPO as a two-player zero-sum game, where one player optimizes the policy to maximize the return, while the other tries to optimize the transition to minimizes the expected return.\n- RMBPO is competitive with several model-free and model-based baselines on MuJoCo and DeepMind Control Suite benchmarks under bivariate parameter distortions\n- The authors include analyses on how the auxiliary model modifies transitions (e.g., the auxiliary model increasing torso angular velocity and damping limb velocities in Hopper-v4).", "claims_and_evidence_yn": "Yes", "claims_explanation": "- The authors test on a reasonable breadth of tasks: 5 continuous-control benchmarks and 2 distortion axes per task\n- The experiments include strong model-free and model-based baselines. In particular, RMBPO is competitive wrt the strongest baseline. EWoK is still best on HalfCheetah.\n- Table 2 and implementation details support the sample efficiency claim of RMBPO.\n- The paper ablates on $\\eta$ and reports its impact on KL divergence and robustness in Appendix A, showing that robustness and optimality trade off as intended.", "audience_yn": "Yes", "audience_explanation": "This method represents a fairly simple drop-in extension to MBPO. It would be of interest to RL researchers working on robustness and safety under distribution shift, particularly in settings with sim2real gap where sample efficiency matters.", "requested_changes": "- As the authors acknowledge, the KL constraint is computed wrt the learned model rather than the true environment. It would be helpful to see quantitatively how much error is introduced, e.g., by estimating $KL(p_\\theta, P)$ using the final policy in the simulators and comparing to $KL(g_\\psi, p_\\theta)$. \n- Please comment on the diagonal Gaussian specification for $p_\\theta, g_\\psi$. If the application involves correlated transition noise, it would help to specify full covariances for $p_\\theta$ and $g_\\psi$, and the KL would still be closed-form. Perhaps this extension would also reduce the size of the ensemble required for $p_\\theta$?\n- Please review the manuscript for typos (e.g., \"our method will not any significant pessimism\" in Section 3.2 and \"emperical\" throughout)", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SeTyZYNfT8", "reviewer": "Reviewer_LAwW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose the following main contributions: \n1. Sample-efficient, robust MBRL algorithm to improve robustness in an online setting; this is achieved by incorporating an auxiliary model in the MBPD that learns a pessimistic world model via adversarial updates. \n2. Demonstrate their work in complex MuJoCo & DMC virtual simulation experiments\n3. Interpret how the agent is robust", "claims_and_evidence_yn": "Yes", "claims_explanation": "In relation to each main contribution:\n1. Yes, the experiments are very convincing, and the performance difference is quite noticeable, especially in the videos linked by the paper with the HalfCheetah and Hopper. They also compare their work to the latest state-of-the-art, such as Gadot et al. EWoK (2024), Zhou et al. RNAC (2024), Reddi et al. QARL (2024), among others. The empirical experiments are comprehensive and impressive.\n2. See #1.\n3. Yes, I was actually pleasantly surprised by the interpretation that was possible with this technique. It is fascinating to identify the exact features or attributes that can be problematic in sim2real settings and to understand how to solve these discrepancies. I think that this would be very interesting for the eXplainable AI (XAI) community, in applications such as self-driving vehicles.", "audience_yn": "Yes", "audience_explanation": "This paper is extremely well-written and accessible to readers without a background in the subject. I am especially grateful that the authors took the space on the page to explain concepts such as KL uncertainty sets, why they approached the problem in the manner they did (e.g., a 2-player zero-sum Markov game), as well as the overall field of RMDPs. The topic and solution are compelling and seem like an exciting direction of research. I can at least speak from my own experience after reading this submission that I am quite interested in this work, and I have not explored it before.", "requested_changes": "1. (end of abstract) Change \"reinforcement learning\" to \"RL\" (use your previous definition of RL).\n2. (First paragraph of introduction) A bit awkward use of \"for example\" in the middle of the sentence. Consider changing it to: \"For example, these control systems could be robots...\"\n3. Cite RMDP when you mention it is \"commonly formalized as...\"\n4. Can you explain a parametric environment in the context of this work? Perhaps in the appendix? Or in the text, whichever you feel is appropriate.\n5. [Clarify] The slippery environment of MBPO should show the agent fall, but it is cut off before this occurs. Same issue with the heavy torso setting. Can you rerun those settings to show it more clearly that the MBPO agent fails? The 2-second clip is just a bit too abrupt.\n6. [Clarify] How is it possible for the HalfCheetah to continue to be able to run once it leaves the floor? Is it falling but still running, movement against a floor not needed, or is the floor simply not rendered past a certain distance?\n7. Section 2.1 \"R is a real interval\" feels out of place. Perhaps consider reorganizing this.\n8. The MDP tuple definition appears more like an RMDP version since the initial state is represented as an \"initial state distribution\". Is it more of an RMDP version?\n9. [Clarify] What are sa-rectangular uncertainty sets? Perhaps explain that in a quick sentence in the text.\n10. [Clarify] The KL uncertainty sets, can you further explain the following: (1) why do we need, or what is, this constant threshold for all s, a? (2) Can you explain the part about the Dirac functions a bit more?\n11. (end of 1st paragraph, Section 3.1) Change \"Next states and rewards\" so that there is a comma after \"Next\"\n12. (end of page 4) \"therefore our method will not any significant pessimism\" seems like it should be \"therefore our method will not have any significant pessimism\"\n13. Consider changing at the end of page 6: \"model. I.e., we are interested in...\" to model (i.e., we are interested in ...)\"\n14. Page 7: \"... we know from Pinkster’s inequality that the KL-divergence bounds the TV distance, we know that minimizing the KL divergence will lower-bound the performance in the nominal environment\", the second \"we know\" should probably have a preceding \"then\" or similar\n15. Cite the \"well-studied in literature\" work that supports your claim: \"This trade-off between adversarial robustness and optimality is well studied in literature\" (pg 7) or give examples\n16. Figures 2 and 3 should have the y-axis consistently labeled with \"mean return\" for all rows, not just the first row of Figure 2", "broader_impact_concerns": "I do not think there are any negative ethical implications of this work; this methodology could actually improve policy deployments and avoid disasters that may harm human lives, as it can accommodate worst-case scenarios.", "additional_comments": "Clear, well-motivated paper with an excellent methodology and convincing empirical results to back their claims. I strongly advise acceptance if the authors can address my few requested changes.", "extra_scores": {}, "extra_text": {}}, {"review_id": "huLuKLq5g9", "reviewer": "Reviewer_2ekk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "here are the core contributions of the paper:\n\n1. Novel Robust MBRL Framework (RMBPO): Introduces a model-based RL method that integrates robustness directly into the dynamics model learning process.\n2. Adversarially Learned Pessimistic Model: Proposes a key innovation - an auxiliary model (gψ) trained adversarially against the policy to generate worst-case transitions within a physically realistic KL uncertainty set. This is the core mechanism for robustness.\n3. Assumption-Free Robustness: Eliminates restrictive requirements common in prior robust RL work:\nNo parametric simulator access needed (unlike Pinto et al., RARL).\nNo multi-state resampling required (unlike Gadot et al.).\nNo predefined training randomization (unlike Rajeswaran et al.).\n4. Maintained Data Efficiency: Leverages the sample efficiency advantages of model-based RL (MBPO foundation) while adding robustness, avoiding the high sample complexity of model-free robust methods (e.g., EWoK, RNAC).\n5. Demonstrated Effectiveness: Shows significantly improved robustness against simultaneous physical distortions (e.g., mass, friction) across diverse continuous control tasks (MuJoCo, DMC Walker) compared to the base MBPO method.\n6. Competitive Performance: Achieves state-of-the-art or competitive robust performance compared to specialized baselines (EWoK, RNAC, QARL, RARL, MixedNE-LD), particularly excelling in DMC Walker tasks and demonstrating superior data efficiency (3-24x).\n7. Interpretable Adversarial Distortions: Provides analysis showing the auxiliary model learns physically meaningful and targeted distortions (e.g., increasing torso wobble, reducing joint power, lowering speed) that exploit task vulnerabilities, explaining the robustness gains.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Based on the provided summary, the claims made in the paper appear to be strongly supported by accurate, convincing, and clear evidence. Here's a breakdown:\n\n1. Claim: Improved Robustness via Pessimistic Model (Core Contribution)\nEvidence: Section 5.1 provides direct, comparative results (Fig 2, 3a/c, 4).\nRMBPO consistently outperforms the base MBPO method under simultaneous physical distortions (mass + friction) across 5 environments (MuJoCo HalfCheetah, Hopper, Walker2d; DMC Walker Walk/Run).\nSignificantly lower cumulative failure rates (Fig 4) demonstrate better generalization under extreme conditions.\nConvincing & Clear: Direct comparison against the non-robust foundation (MBPO) isolates the effect of the auxiliary model. Multiple environments and metrics (performance, failure rate) strengthen the claim.\n\n\n2. Claim: Competitive Performance vs. SOTA Robust RL Baselines\nEvidence: Section 5.2 compares RMBPO against diverse baselines (QARL, RARL, EWoK, RNAC, SAC, MixedNE-LD).\nAchieves SOTA performance on DMC Walker tasks (Fig 3a-d).\nCompetitive performance on MuJoCo tasks (2nd to EWoK in HalfCheetah).\nConvincing & Clear: Benchmarks against relevant, recent SOTA methods covering different robust RL approaches (simulator-dependent, sampling-intensive, general). Clear presentation of results across tasks.\n\n\n3. Claim: Superior Data Efficiency\nEvidence: Explicit sample count comparison in Section 5.2.\nRMBPO uses 125K-400K samples vs. 1M-3M+ for baselines (EWoK, RNAC).\nQuantified as 3-24x more efficient.\nAccurate & Convincing: Concrete numbers and multipliers provide unambiguous evidence for this key advantage.\n\n\n4. Claim: No Restrictive Assumptions (Simulator Access, Multi-State Resampling, Predefined Randomization)\nEvidence: Implicitly validated by the experimental setup and results in Sections 5.1 & 5.2.\nSuccessfully trains and achieves robust performance without the specific requirements of Pinto (simulator), Gadot (resampling), or Rajeswaran (randomization).\nOutperforms simulator-dependent methods (QARL, RARL) in \"fair tests\" (presumably where those methods don't have an unfair advantage from simulator access).\nConvincing: The paper's ability to achieve its results using only standard interaction data, contrasted with the requirements of the cited methods, strongly supports this claim.\n\n\n5. Claim: Interpretable Adversarial Distortions\nEvidence: Section 5.3 (Hopper-v4 analysis).\nIdentifies specific state variables modified by the auxiliary model (Torso Ang. Vel ↑, Joint Vel ↓, X-Vel ↓).\nProvides plausible physical interpretations of these modifications (increased instability, reduced mobility/progress).\nShows distortion scales with pessimism hyperparameter η.\nClear & Convincing: Directly links the model's internal mechanism (modifying specific states) to understandable physical effects and adversarial strategies, explaining how robustness is achieved.\n\n\n6. Claim: Balancing Robustness & Optimality (Theoretical Trade-off)\nEvidence: Section 5.1 results combined with theoretical grounding (Sec 4.2).\nObserved moderate drop in nominal performance (HalfCheetah, DMC Walker Run) aligns with the theoretical expectation of a trade-off.\nAccurate & Convincing: Empirical observation matches the theoretical prediction, demonstrating a controlled trade-off managed by the KL constraint and η.", "audience_yn": "Yes", "audience_explanation": "1. Core Relevance to RL Research:\nRobustness is a Fundamental Challenge: Ensuring RL policies perform reliably under distribution shift or uncertainty is a critical, unsolved problem central to real-world deployment. This paper directly tackles this.\nModel-Based RL (MBRL) is a Major Paradigm: MBRL is a highly active research area due to its sample efficiency. Improving its robustness is a key goal for many researchers.\nAdversarial Training is Trending: Techniques involving adversarial perturbations or training are widely studied for robustness in ML, including RL. This paper's specific adversarial model approach is novel within MBRL.\n\n\n2. Addressing Key Pain Points in Robust RL:\nRemoving Restrictive Assumptions: The paper explicitly solves a limitation of prior SOTA methods (requiring parametric simulators, multi-state resampling, or predefined randomization). This makes robust RL more accessible and practical, which is a major interest area.\nMaintaining Efficiency: Demonstrating robust performance without sacrificing the core sample efficiency advantage of MBRL (unlike model-free robust methods) is a significant practical contribution valued by the community.\nInterpretable Mechanisms: The analysis showing how the adversarial model distorts states to induce robustness (Sec 5.3) provides valuable insights beyond just performance numbers, appealing to researchers interested in understanding why methods work.\n\n\n3. Strong Empirical Validation on Standard Benchmarks:\nResults on widely used MuJoCo and DeepMind Control Suite benchmarks allow direct comparison to numerous existing works.\nComprehensive comparisons against relevant SOTA baselines (QARL, RARL, EWoK, RNAC, SAC, MixedNE-LD) provide a clear picture of where the method stands.\nQuantifying the significant data efficiency gains (3-24x) is highly compelling for an audience concerned with training costs.\n\n\n4. Appeal Across Sub-Communities:\nTheorists: The use of KL uncertainty sets and the theoretical grounding of the robustness/optimality trade-off (Sec 4.2) will interest those focused on formal guarantees.\nPractitioners/Algorithm Developers: The novel adversarial model architecture, the practical removal of restrictive assumptions, and the strong empirical results make it immediately relevant for those building and applying RL algorithms.\nResearchers in Robustness & Generalization: Anyone working on making ML systems reliable under distribution shift will find the approach and findings pertinent.\nResearchers in Adversarial ML: The specific adversarial training mechanism within the dynamics model is a novel application.\n\n\n5. Timeliness:\nRobust RL, efficient RL, and adversarial robustness are all highly active and timely research areas. This work sits at their intersection.", "requested_changes": "Critical Adjustments (Necessary for Recommendation of Acceptance): Failure to address these could undermine core claims or reproducibility.\n\n\n1. Clarify & Analyze EWoK Performance Gap in MuJoCo HalfCheetah:\n1_1. Issue: The summary notes RMBPO trailed EWoK under distortion in MuJoCo HalfCheetah despite superior efficiency. The paper must provide a deeper analysis or plausible explanation for this specific result.\n1_2. Action Needed: Add discussion exploring potential reasons (e.g., task-specific physics favoring EWoK's resampling, architecture mismatch, hyperparameter sensitivity in this domain). Acknowledge the limitation clearly.\n1_3. Why Critical: Leaving this unexplained weakens the claim of being \"competitive\" or \"SOTA\" in MuJoCo and could suggest a fundamental limitation of the approach in certain environments. It's a key counterpoint to the otherwise positive results.\n\n2. Explicitly Detail Hyperparameter Selection & Sensitivity (Especially η):\n2_1. Issue: The pessimism level η is crucial, and results (Sec 5.3) show performance is sensitive to its scaling. The summary lacks details on how η was chosen for the main experiments and the extent of its sensitivity across tasks.\n2_2. Action Needed:\nClearly state the η values used for each environment/baseline comparison in the main results (5.1, 5.2).\nInclude a brief sensitivity analysis (e.g., a table or small figure) showing performance variation over a reasonable range of η for at least one key environment (like Hopper or DMC Walker). Discuss the implications for tuning.\n2_3. Why Critical: Reproducibility is paramount. Without knowing the exact η used and understanding its impact, others cannot reliably replicate the results. Sensitivity analysis demonstrates the method's robustness (or lack thereof) to this key hyperparameter.\n\n3. Strengthen the \"No Restrictive Assumptions\" Claim for Baselines:\n3_1. Issue: While the paper positions itself as removing requirements (simulator, resampling, randomization), the summary doesn't explicitly state how the baselines (QARL, RARL, Gadot/EWoK, Rajeswaran) were run without their typical advantages to ensure a fair comparison.\n3_2. Action Needed: Clearly describe in the methodology (or results section) the setup for each baseline. Emphasize that simulator-dependent methods (QARL, RARL) were run without privileged simulator access for perturbation, and that EWoK was run without its multi-state resampling advantage if applicable. State explicitly if Rajeswaran-style pre-randomization was not used for RMBPO or the baselines in the comparison.\n3_3. Why Critical: The core novelty claim hinges on removing these assumptions. If baselines were handicapped by not using their defining features, the comparison becomes unfair and the novelty claim is weakened. Proving a fair test is essential.\n\nStrengthening Adjustments (Highly Recommended to Improve Impact & Clarity):\n\n4. Expand Discussion on Robustness-Optimality Trade-off & KL Constraint:\n4_1. Issue: The nominal performance drop is noted and linked to theory, but a deeper discussion on managing this trade-off practically is missing.\n4_2. Action Needed: Discuss how the KL constraint size and η interact to control this trade-off. Could the KL budget be adapted? Suggest strategies for practitioners to choose these parameters based on desired robustness level. Briefly connect the observed drops (e.g., HalfCheetah nominal) more explicitly to the theoretical result (Sec 4.2).\n4_3. Why Strengthening: Provides greater practical guidance and deeper insight into the method's behavior, enhancing its value to practitioners.\n\n\n5. Discuss Sim2Real Implications & Limitations More Directly:\n5_1. Issue: The summary flags the sim2real gap as unresolved. This limitation should be explicitly stated and discussed in the paper's conclusion or limitations section.\n5_2. Action Needed: Add a paragraph in the Discussion/Limitations/Conclusion explicitly stating that validation is currently only in simulation, that the KL constraint is a proxy for real-world uncertainty, and that sim2real transfer remains future work. Briefly discuss potential challenges (e.g., whether the learned distortions generalize to real-world inaccuracies).\n5_3. Why Strengthening: Provides crucial context for the work's applicability and sets realistic expectations, strengthening the paper's credibility.", "broader_impact_concerns": "Safety-Critical Sim2Real Gap: Robustness demonstrated only in simulation. Premature deployment in real-world safety-critical systems (e.g., autonomous vehicles) based on these results carries significant risk. The KL uncertainty set is a proxy, not a real-world guarantee.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)