Time-Constrained Robust MDPs
Authors: Adil Zouitine
David Bertoin
Pierre Clavier
Matthieu Geist
Emmanuel Rachelson
Venue: NeurIPS 2024 poster (2024)
Keywords: Robust reinforcement learning
Primary area: reinforcement_learning

Abstract: Robust reinforcement learning is essential for deploying reinforcement learning algorithms in real-world scenarios where environmental uncertainty predominates.
Traditional robust reinforcement learning often depends on rectangularity assumptions, where adverse probability measures of outcome states are assumed to be independent across different states and actions. 
This assumption, rarely fulfilled in practice, leads to overly conservative policies. 
To address this problem, we introduce a new time-constrained robust MDP (TC-RMDP) formulation that considers multifactorial, correlated, and time-dependent disturbances, thus more accurately reflecting real-world dynamics. This formulation goes beyond the conventional rectangularity paradigm, offering new perspectives and expanding the analytical framework for robust RL.
We propose three distinct algorithms, each using varying levels of environmental information, and evaluate them extensively on continuous control benchmarks. 
Our results demonstrate that these algorithms yield an efficient tradeoff between performance and robustness, outperforming traditional deep robust RL methods in time-constrained environments while preserving robustness in classical benchmarks.
This study revisits the prevailing assumptions in robust RL and opens new avenues for developing more practical and realistic RL applications.

Time-Constrained Robust MDPs
Adil Zouitine DDavid Bertoin 1,3,, Pierre Clavier*4.5 Matthieu Geist", Emmanuel Rachelson
IRT Saint-Exupéry, IISAESUUPARO, Université de Toulouse, IMT, INSA Toulouse *Ccole Polytechnique, CMAP, IInia Paris, HeKA
6 ANITI, 'Cohere
david dww.d.berrochhhrrrttaaaneeppppp com,
pierre. pierre.clavier saareeecllieer@lllyecmiiiuuu
Abstract
Robust reinforcement learning is essential for deploying reinforcement learning algorithms in real-world scenarios where environmental uncertainty predominates. Traditional robust reinforcement learning often depends on rectangularity assump- tions, where adverse probability measures of outcome states are assumed to be independent across different states and actions. This assumption, rarely fulfilled in practice, leads to overly conservative policies. To address this problem, we intro- duce new constraiid robust MDP (TC-RMDP) formulation that considers multifactorial, correlated, and time-dependent disturbances, thus more accurately reflecting real-world dynamics. This formulation goes beyond the conventional rectangularity paradigm, offering new perspectives and expanding the analytical framework for robust RL. We propose three distinct algorithms, each using varying levels of environmental information, and evaluate them extensively continuous control benchmarks. Our results demonstrate that these algorithms yield an effi- cient tradeoff between performance and robustness, outperforming traditional deep robust RL methods time-constrained environments while preserving robustness in classical benchmarks. This study revisits the prevailing assumptions in robust RL and opens new avenues for developing more practical and realistic RL applications.
Introduction
Robust MDPs capture the problem of finding control policy for a dynamical system whose transition kernel is only known to belong defined uncertainty set. The most common framework for analyz- ing and deriving algorithms for robust MDPs is that of sa-rectangularity [ 21, where probability measures on outcome states are picked independently in different source states and actions (in formal notation, P(s'|s, [s, a) and IP((''|8 a) are independent of each other). This provides
appreciable decoupling of worst transition kernel search across time steps and enables sound algorithms like robust value iteration (RVI). But policies obtained for such sa-rectangular MDPs are by nature very conservative 3 4] as they enable drastic changes in environment properties from one time step to the next, and the algorithms derived from RVI tend to yield very conservative policies even when applied to con-sa-rect robust MDP problems.
In this paper, we depart from the rectangularity assumption and turn towards a family of robust MDPs whose transition kernels are parameterized by vector This parameter vector couples together the outcome probabilities in different (s, a) pairs, hence breaking the independence assumption that problematic, especially in large dimension 3 This enables accounting for the notion of transition
These authors contributed equally this work
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

model consistency across states and actions: outcome probabilities are not picked independently anymore but are rather set across the state and action spaces by drawing a parameter vector. In turn, we examine algorithms for solving such parameter-based robust MDPs when the parameter is constrained to follow a bounded evolution throughout time steps. Our contributions are the following.
1. We introduce formal definition for parametric robust MDPs and time-constrained robust MDPs, discuss their properties and derive a generic algorithmic framework (Sec. (
2. We propose three algorithmic variants for solving time-constrained MDPs, named vanilla TC, Stacked-TC and Oracle-TC (Sec. 4. which use different levels of information in the state space, and come with theoretical guaranties (Sec. (
3. These algorithms are extensively evaluated in MuJoCo [5] benchmarks, demonstrating they lead to non-conservative and robust policies (Sec.
2 Problem statement
(Robust) MDPs. A Markov Decision Process (MDP) is a model of a discrete-time, sequential decision making task. At each time step, from a state 8, E S of the MDP, an action E A is taken and the state changes to St+1 according to a stationary Markov transition kernel p(st+1 1/s,, az). while concurrently receiving reward r(st, az). S and A are measurable sets and we write As and A A the set of corresponding probability distributions. stationary policy π r-|ss) is mapping from states to distributions over actions, prescribing which action should be taken in 8. The value function 05 of policy π maps state s to the expected discounted sum of rewards pp 1E, 7ree when applying from s for infinite number of steps. An optimal policy for MDP function is maximal in any state. In a Robust MDP (RMDP) [ 2]. the transition kernel p is not set exactly and can be picked in an adversarial manner at each time step. from uncertainty set P. Then, the pessimistic value function of a policy is (s(ss) minpep 2nl (s). An optimal robust policy is one that has the largest possible pessimistic value function thp in any state, hence yielding an adversarial max, min, optimization problem. Robust Value Iteration (RVT) 7 solves this problem by iteratively computing the one-step lookahead best pessimistic value:
whose value
Un+ (8) = Tn (+) == max min (s (r a) + E,p (s((v())]]]
=(s)EA PEP
The Ti operator called the robust Bellman operator and the sequence of Un functions converges to the robust value function E.p as long as the adversarial transition kernel belongs to the simplex of As.
Zero-sum Markov Games. Robust MDPs can be cast zero-sum two-players Markov games R 9 where B is the action set of the adversarial player. Writing न : S x A - An the policy of this adversary, the robust MDP problem turns max, ming ~. where ". (s) is the expected
sum of discounted rewards obtained when playing (agent actions) against स (transition models) each time step from s. This enables introducing the robust value iteration sequence of functions
1 (s) == (s) := max min ((""",,)) =(*)EA R(c.a)EAs
where T=. == Ee + "B.rra (s')) is a zero-sum Markov game operator. These operators are also 7- contracction and converge to their respective fixed point " and 8"* = = [9] This formulation will be useful to derive practical algorithm in Section 4
Often, this convergence uncertainty set P is a set product of independent subsets of As in each 8, pair. Quoting M. rectangularity is a sort of independence assumption and a minimal requirement for most theoretical results to hold. Within robust value iteration, rectangularity enables picking ( az) completely independently of (s) 1))).. To set ideas, let us consider the robust MDP of a pendulum, described by its mass and rod length. Varying this mass and rod length spans the uncertainty set of transition models. The rectangularity assumption induces that π (s,, az) can pick a measure in Ag corresponding to mass and a length that completely independent from the ones picked in the previous time step. While this might be a good representation in some cases, in general it yields policies that are very conservative as they optimize for adversarial configurations which might not occur in practice.
analyzed under the assumption of sa-rectangularity, stating that the
We first step away from the rectangularity assumption and define a parametric robust MDP as an RMDP whose transition kernels are spanned by varying parameter vector & (typically the mass

---

and rod length in the previous example). Choosing such a vector couples together the probability measures on successor states from two distinct (s, a) and (s', pairs. The main current robust deep RI algorithms actually optimize policies for such parametric robust MDPs but still allow the parameter value at each time step to be pieked independently of the previous time step.
Parametric MDPs. A parametric RMDP is given by the tuple (S, A, M Pp, r) where the transition kernel Pa (s, a) E Ag is parameterized by v. and V is the set of values & can take, equipped with an appropriate metric. This yields the robust value iteration update:
Un+1 (+) = max min (T un)) == max min (a a) + + (e (s')) x(s)EA NEW
A(s)EAA NEW
A parametric RMDP remains a Markov game and the Bellman operator remains contraction mapping as long as Pv can reach only elements in the simplex of As, where the adversary's 's action set is the set of parameters instead of (possibly so-rectangular) set of transition kernels.
Time-constrained RMDPs (TC-RMDPs), We introduce TC-RMDPs as the family of parametric RMDPs whose parameter's s evolution is constrained to be Lipschitz with respect to time. More formally a TC-RMDP is given by the tuple (S, A, Vr pv. r, L), where || - whel s L, that is the parameter change is bounded through time. In the previous pendulum example, this might represent the wear of the rod which might lose mass or stretch length. Similarly, and for larger scale illustration, TC-RMDPs enable representing the possible evolutions of traffic conditions in path planning problem through a busy town. Starting from an initial parameter value 1 the pessimistic value function of policy is non-stationary, th is constrained to lay most L-far away from w--1, 1) from vho and so on. Generally, this yields non-stationary value functions as the uncertainty set at each time step depends the previous uncertainty parameter. To regain stationarity without changing the TC-RMDP definition, we first change the definition of the adversary's 's action set. The adversary picks its actions in the constant set B B(00g, T.), which is the ball of radius I. centered in the null element in V. In turn, the state of the Markov game becomes the pair 2/ and the Markov game itself given by the tuple ((S) x ) A, B, pv. where the Lipschitz constant is included in B. Thus, given an action be E B and a previous parameter value 2 the parameter value at time t the = i + by Then, we define the pessimistic value function of a policy as function of both the state and parameter ub:
()) = min E
Σ 7rrel/b-1 w.
8, be E B, = fh-1 + be, a nI, 8t ~ Pv.
(hr)cct
E
"ig (s. b)=
max
(3 v.).
In turn, an optimal robust policy is a function of s and vo and the TC robust Bellman operators are:
max 11, 1)) (s, =(s.))EAA
1b),
max min E (r))ss) a) + Mn (s',, b)]. =(x.))EAA BEB
This iteration scheme converges to a fixed point according to Th. 2.1
Theorem 2.1. The time-constrained (TC) Bellman operators Th and Ti are contruction mappings. Thus the sequences un+1 TBUn and Un+1 -T converge to their respective fixed points v and vzg.
Proof of Th. 2.1 can be found in Appendix B We refer to this formulation as algorithm Oracle-TC (see Section for implementation details) since an oracle makes the current parameter & visible to the agent. Therefore, it possible to derive optimal policies for TC-RMDPs by iterated application of this TC Bellman operator. These policies have the form (s,, vb) In the remainder of this paper, we extend state-of-the-art robust deep RL algorithms to the TC-RMDP framework. In particular, compare their performance and robustness properties with respect to classical robust MDP formulations, we also discuss their relation with the (s)) robust policies of classical robust MDPs.
If the agent is unable to observe the state variable v/. is not possible to guarantee the existence of a stationary optimal policy of the form (s). Similarly, there is no guarantee of convergence of value functions to fixed point. Nonetheless, this scenario, which access to the vi parameter not available, is more realistic in practice. It turns the two-player Markov game into partially observable
3

---

Markov game, where one can still apply the TC Bellman operator but without these guarantees of convergence. We call vanilla TC the repeated application of the TC Bellman operator this partially observable case. Vanilla TC will be tested in practice, and some theoretical properties of the objective function will be derived using the Lipschitz properties (Sec 6).
3 Related works
Since our method is a non-rectangular, Deep Robust RL algorithm, (possibly non-stationary for Stacked-TC and TC) ), we discuss the following related work.
Non-stationary MDPs. First, non-stationarity has been studied in the Bandits setting in [1] Then, for episodie, non-stationary MDPs [T 12 [3] have explored and provided regret bounds for algorithms that use oracle access to the current reward and transition functions. More recently 14 15] have facilitated oracle access by performing a count-based estimation of the reward and transition functions based on the recent history of interactions. Finally, for tabular MDPs, past data from a non-stationary MDP can be used to construct a full Bayesian model [16] or a maximum likelihood model [17] representations
of the transition dynamics. We focus on the setting not restricted to tabular
Non-rectangular RMDPs. While rectangularity in practice is very conservative, it can be demon- strated that, in an asymptotic sense, non-rectangular ellipsoidal uncertainty sets around the maximum likelihood estimator of the transition kernel constitute the smallest possible confidence sets for the ground truth transition kernel, as implied by classical Cramér-Rac bounds. This is in accordance with the findings presented in $ 5 and Appendix A of (7]. More recently, [3] extends the rectangular assumptions using a factored uncertainty model, where all transition probabilities depend on a small number of underlying factors denoted w1,.... w, R$ such that each transition probability P.. for every (s, a) is a linear (convex) combination of these r factors. Finally, 4 use policy gradient algorithms for non-rectangular robust MDPs. While this work presents nice theoretical guarantees of convergence, there is no practical Deep RL. algorithms for learning optimal robust policies.
Deep Robust RL Methods. Many Deep Robust algorithms exist such M2TD3 18] M3DDPG 19], or RARL (0), which are all based on the two player zero-sum game presented in 2 We will compare our method against these algorithms, except [19] which is outperformed by [8 in general. We also compare our algorithm to Domain randomization (DR) [21] that learns a value function (ss) max,, Bp-2((p) p (s) which maximizes the expected return average across fixed (generally uniform) distribution on P. As such, DR approaches do not optimize the worst-case performance but still have good performance average. Nonetheless, DR has been used convincingly in applications [22] 23]. Finally, the zero-sum game formulation has lead to the introduction of action robustness 9 which is specific case of rectangular MDPs, in scenarios where the adversary shares the same action space as the agent and interferes with the agent's s actions. Several strategies based this idea have been proposed. One approach, the Game-theoretic Response Approach for Adversarial Defense (GRAD) (24) builds on the Probabilistic Action Robust MDP (PR-MDP) [9] This method introduces cone-contrriined perturbations in both the action and state spaces and employs a theorettc approach with a population of adversaries. In contrast to GRAD, where temporal disturbances affect the transition kernel around a nominal kernel, our method part of broader setting in which the transition kernel is included in larger uncertainty set. Robustness via Adversary Populations (RAP) 25 introduces population of adversaries. This approach ensures that the agent develops robustness against a wide range of potential perturbations, rather than just single one, which helps prevent convergence to suboptimal stationary points. Similarly, State Adversarial MDPs 26 27 28 24 address adversarial attacks on state observations, effectively creating partially observable MDP. Finally, using rectangularity assumptions, [29] 30 use Wasserstein and x² balls respectively for the uncertainty set in Robust RI.
4 Time-constrained robust MDP algorithms
The TC-RMDP framework addresses the limitations of traditional robust reinforcement learning by considering multifactorial, correlated, and time-dependent disturbances. Traditional robust reinforcement learning often relies rectangularity assumptions, which are rarely met in real-

---

Environment
ag
a;
4/+
π
Adversary
Agent
St 4/ St-1 a+-
5 4/+
TC-RMDP training involves a temporally-constrainne constrained adversary aiming maximize the effect of temporally perturbations. Conversely, the agent aims to optimize its performance against this come-constrained adversary. In orange, the oracle observation, and in blue the stacked observation.
Figure 1:
world scenarios, leading to overly conservative policies. The TC-RMDP framework provides more accurate reflection of real-world dynamics, moving beyond the conventional rectangularity paradigm.
We cast the TC-RMDP problem as a two-player zero-sum game. where the agent interacts with the environment, and the adversary (nature) changes the MDP parameters vb. Our ap- proach is generic and can be derived within any robust value iteration scheme, perform- ing maXx())çAA mingev Eawn)ss) (r(s,)) a) (s (s')) updates, by modifying the ad- versary's action space and potentially the agent's state space to obtain updates of the form maxx(..))) m Ea- a) 7E,'--e-asss.. Un (n In Section 5 we will introduce time constraints within two specific robust value iteration algorithms, namely RARL (20) and M2TD3 [8] by simply limiting the search space for worst-case & at each step. This specific im- plementation extends the original actor-critic algorithms. For the sake of conciseness, we refer the reader Appendix E. for details regarding the loss functions and algorithmic details.
Three variations of the algorithm are provided (illustrated in Figure I but all fall within the training loop of Algorithm
Algorithm constcontt robust training
Input: Lime-constraines MDP: (S, A, V. P,Per r, L), Agent Adversary म
for each interaction time step f do
G: me (8(8,, v/2))
// Sample action with Oracle-TC // Sample an action with Stacked-TC // Sample action with TO // Sample the worst TC parameter // Sample a transition B + BU ((st, ar, (st, ar). Hh, (441, // Add transition to replay buffer {5i, as r(s,, ai), th, thtt 3141 ~ // Sample a mini-batch of transitions // Update Agent
or de ~ (8t; at-1::111 8t-1)
ag (st)
1'++1 ~ (5, a1, v/2)
S1+1 Ports (st, a1)
T++1 + UpdatePolicy(nn (olicy(TT)
me+1 CanaatePolcygl
// Update Adversary
Oracle-TC: As discussed in Scction the Oracle-TC version includes the MDP state and parameter value input, true parameters of the environment, allowing it to make the most informed decisions and possibly reach the true robust value function. However, these parameters & are sometimes non-observable in practical scenarios, making this method not always feasible.
V A. This method assumes that the agent has access to the
Stacked-TC. Since v might not be observable but may be approximately identified by the last transitions, the Stacked -TC policy uses the previous state and action as additional inputs in an attempt to replace v/, S Ax S + A. This approach leverages the information in the transitions, even though might be insufficient for a perfect estimate of sp. It aims to retain (approximately) the convergence propertics of the Oracle-Pn algorithm.
5

---

Vanilla TC. Finally, the vanilla TC version takes only the state, π. 8 A, as input, similar to standard robust MDP policies. This method does not attempt to infer the environmental parameters or the transition dynamies explicitly. Instead, it relies on the current state information to guide the agent's actions. While this version is the most straightforward and computationally efficient, it may not perform as robustly as the Oracle-TC or Stacked-TC TC versions in environments with significant temporal disturbances, since it attempts to solve a partially observable Markov game, for which there may not exist stationary optimal policy based only the observation. Despite this, it remains viable option in scenarios where computational simplicity and quick decision-making are prioritized.
Results
Experimental settings. This section evaluates the robust time-constrained algorithm's performance under severe time constraints and in the static settings. Experimental validation was conducted in continuous control scenarios using the MuJoCo simulation environments [5]. The approach was categorized into three variants. The Oracle-TC, where the agent accessed environmental parameters (s,, 4) the Stacked-TC where the agent took in input π (St, St-1, a;-1); and the vanilla TC, which did not receive any additional inputs (s). For each variant of the tine-cnnstrained algorithms,, we applied them to RARL (0).. and M2TD3 18] renaming them TC-RARL. and TC-M2TD3, respectively. The algorithms were tested against two state-of-the-an art robust reinforcement learning algorithms, M2TD3 and RARL. Additionally, the Oracle versions of M2TD3 and RARL, where the agent's policy included va in the input S x V - A, were evaluated for more comprehensive assessment. Comparisons were also made with Domain Randomization (DR) 21 and vanilla TD3. 3 to ensure thorough analysis. A 3D uncertainty set is defined in each environment P normalized between [o, 1]3. Appendix provides detailed descriptions of uncertainty parameters. Performance metrics were gathered after five million steps to ensure fair comparison. All baselines were constructed using TD3, and a consistent architecture was maintained across all TD3 variants. The results presented below were obtained by averaging over ten distinct random seeds. Appendices E.2. and H.2 discuss further details on hyperparameters, network architectures, and implementation choices, including training curves for methods and baseline comparisons. In the following tables ! 2 the best performances are shown in bold. Oracle methods, with access optimal information, are shown in black. Items in bold and green represent the best performances with limited information on v making them more easily usable in many scenarios. When there is only one element in bold and green, this implies that the best overall method is non-oracle method.
E.4
E.3
Ant
HalfCheetah
Hopper
Humanoid
Walker
Agg.
Orucle M2TD3
1.11 0.07
0.95 = 0.1
1.11 0.84
2.07 + 0.19
1.31 + 0.36
1.39 + 0.31
Oracle RARI.
0.72
-0.71 0.05
-1.3+
1.62
-0.19 + 0.2
-0.86 0.47
Oracle-TC -M2TD3
1.61 0.33
2.76 0.16
7.79 = 1.0
1.69 + 2.14
1.49 + 0.41
3.07 + 0.81
Oracle TC- RARL.
1.66 = 0.32
2.63 + 0.12
6.86 1.46
0.19 + 1.68
1.34 + 0.11
2.54 4: 0.74
Stacked TC -MTD3
0.22
0.19
6.51 0.59
-1.42 1.44
1.69 + 0.33
2.1
Stacked TC RARI.
1.48 0.22
1.76 + 0.08
3.28
0.27
1.39 + 0.57
1.01 + 0.21
1,78 0.27
TC -2TDD
1.2
2.42 0.1
5.16 0.2
4.02 + 1.23
1.38 + 0.25
2.9 0.4
TC -RARL
0.26
0.15
0.01
1.25 1.91
0.89 - 0.2
1.6
TD3
0.0 0.19
0.0
0.0 1.27
0.0 1.18
0.0 4- 0.23
DR
1.58 + 0.2
1,59 0.12
2.28 128800.00 0.42
0.87 +: 1.79
1.03 0.19
1.47 1.770000
M2TD3
1.0 0.19
1.0
1.0+0..14
1.31
1.0 +
1.0 0.58
RARL
0.63 0 0.2
-0.61 + 0.18
-1.5 0.33
0.88
0.27 + 0.25
-0.08 = 0.37
Table 1: Avg. of normalized time-coupled worst-case performance over 10 seeds for each method
Performance of TCRMDPs in worst-case time-constrained. Table T reports the worst-case time-constrained perturbation. To address the worst-case time-constrained perturbations for each trained agent τ², we utilized a time-constrained adversary using TD3 algorithm #* [r(s, a) + >EEs' Po+ a)v, (s')] within perturbation radius of L = 0.001 for a total of 5 million steps. The sum of episode rewards was averaged over 10 episodes. A heuristic choose the perturbation radius is explained in Appendix[K] To compare met- rics across different environments, each method's 's score was standardized relative to the reference score of TD3. TD3 was trained on the environment using default transition function parameters, with its score denoted as e7 The M2TD3 score, UM27D3, was used as the comparison target. The formula applied was (w WTDs)/ ((ars)// - VTDa)). This positioned e' as the minimal baseline and UM2T D3 the target score. This standardisation provides metric that quantifies the
min

---

improvement of each method over TD3 in relation to the improvement of M2TD3 over TD3. In each evaluation environment, agents trained with the time-constrained framework (indicated by TC in the method name) demonstrated significantly superior performance compared to those trained using alternative robust reinforcement learning approaches, including M2TD3 and RARL.. Furthermore, they outperformed those trained through domain randomisation (DR). Notably, even without directly conditioning the policy with 1h, the time-constrained trained policies excelled against all baselines, achieving up to 2.9-fold improvement. The non-normalized scores reported in Appendix H Ad- ditionally, when policies were directly conditioned by & and trained within the robust reinforcement learning framework, they tended be overly conservative in the time-constrained framework. This is depicted in Table comparing the performances of Oracle RARL, Oracle M2TD3, Oracle TC-RARL, and Oracle TC-M2TD3. Both policies also observe v). The only difference is that Oracle RARL and Oracle M2TD3 were trained in the robust reinforcement learning framework, while Oracle TC-RARL. and Oracle TC-M2TD3 were trained in the time-constrained framework. The performance differences under worst-case time-coupled perturbation are as follows: for Oracle RARL (resp. M2TD3) and Oracle TC-RARL (resp. M2TD3), the values -0.86 (1.39) 2.54 (3.07). This observation highlights the need for a balance between robust training and flexibility in dynamic conditions. A natural question arises regarding the worst-case time-constrained perturbation. Was the adversary in the loop adequately trained, or might its suboptimal performance lead to overestimating the trained agent's 's reward against the worst-case perturbation? The adversary's performance was monitored during its training against all eeed-trained agents. The results in Appendix | show that our adversary converged.
Training under various time fixed adversaries. The method was evaluated against various fixed adversaries, focusing the random fixed adversary shown in Figure 2 This evaluation shows that robustly trained agents can handle dynamic and unpredictable conditions. The random fixed adversary simulates stochastic changes by selecting parameter t/, each timestep within a radius of L = 0.1. This radius is 100 times larger than in our training methods. At the start of each episode, tho is uniformly sampled from the uncertainty th ((P). This tests the agents' udaptability to unexpected changes. Figures 2a Pajthrough Selshow our agents' performance. Agents trained with our robust framework consistently outperformed those trained with standard methods. The policy was also assessed against five other fixed adversaries: cosine, exponential, linear, and logarithmic. Detailed results are provided in the Appendix. H.I
Robust
Nein Neen-pooo spoode rowed
Monea spponie reward
Gredo womss
dralk K.oiet
scadded TC
TDV
Goeda sorss
sams
Onerk TCMOYT
Drade Eciie
Marded TC 1C-27713
mh
drede axTo!
gett
aee--ainn addd
stacke
sta
Statted
stacke
Algunktin
Alpentiin
thm
(a) Ant
(b) HalfCheetah
(c) Hopper
Moan apsode
42700
Greas TCades
Redded 42/773
Ald
plcchel yadd
areds sael
torta
Gunde (C-2/11
dracee IC stee
Steceed 7013
you
dteciel
Aportthm
Algonttim
(d) Humanoid
(e) Walker
Figure 2: Evaluation against a random fixed adversary, with radius 0.1
Performance of Robust Time- Constraind MDPs in the static setting. In static environments, the Robust Time-Constrained al algorithms were evaluated for worst-case and average performance
metrics, shown in Tables 2 and 3 A fixed uncertainty set P was used, dividing each dimension of V into ten segments, creating grid of 1000 points (103). Each agent ran five episodes at each grid point, and the rewards were averaged. The scores were normalized as described for the
7

---

time-constrained adversary analysis in Table I The raw data is provided in Appendix 9 and T0 Performance scores were adjusted relative the baseline DT and UM2TD3- As a result, normalized results reveal distinct trends among agent configurations within the TC-RMDP framework. The Oracle TC-M2TD3 variant achieved an average score of 3.12 while the Stacked TC-M2TD3 scored 2.23, indicating its resilience. Furthermore, in the worst-case scenario, the TC-RARL and Stacked TC-RARL variants demonstrated adaptability, with TC-RARL scoring 0.92 and TC-M2TD3 scoring This performance highlights its reliability in challenging static environments.
1.02
Ani
HalfCheetah
Hopper
Humanoid
Walker
Agg
Oracle M2TD3
1.02 = 0.19
0.34
0.97 + 0.55
3.9 1 3.65
0.3 + 0.45
1.31 +: 1.01
Oracle RARL
0.62 + 0.32
0.02
0.48 + 0.19
-2.99 + 2.18
0.16 + 0.21
-0.25 0.58
Dracle-TC-oe
TT-2Ie-T--M2ID3 -M2TD3
0.25
1.87 0.1
0.49 1.07
-0.8 1.8:300 3.05
0.28 1: 0.38
0.39 1: 0.97
Dracle-TC RARL
0.59 +: 0.36
1.55 0.35
0.16
1.1
0.56 + 0.39
0.86 + 0.5
Stacked-IC -M2TD3
-0.05
1.56 + 0.16
1.08 +
-0.83 2.62
1.12 0.5
0.58 = 0.85
Baacked-TC-RRR
0.07 0.13
1.35 +: 0.93
1.75 + 2.48
0.67 +: 0.32
0.92 0.84
TC -M2TD3
1.49 0.006
1.29
1.21
1.19 0.34
1.02 0.68
RARL
0.24
0.3
0.76
1.5
0.67 + 0.59
1.29
TD3
0.34
0
0.2
0,0
0.0
0.52 0.0
DR
0.06 4: 0.16
1.07 -
0,86 0.82
0,04
0.57 + 0.37
M2TD3
1.0 + 0.27
L0
1.0 0.05
1.0 3.32
1.0
1.0 1.01
RARL
0.44
0.13 0.08
0.5 0.22
0.41
+
0.33 - 0.73
Table 2: Avg. of normalized static worst-case performance over 10 seeds for each method
Ant
HalíCheetah
Hopper
Humanoid
Walker
Agg
Oracle M2TD3
1.13 0.08
1
1.12 0.46
1.96 1,53
1,23 : 0.3
1.4 0.52
Oracle RARI.
0.7 0.22
-1.4
-0.77 + 0.24
-2.6 +: 2.88
-1.13 + 0.84
-1.04 + 0.86
Dracle-TC- TC-MMDD -M21D3
1.73 0.00
4.35 0.26
5.54 0.13
2.12 : 1.4
1.84 1 0.37
3.12 0.445
Dracle-TC- RARL
1.78
5.08 1 0.48
0.4
1.68 1 0.24
2.66 - 0.77
Stacked-TC -M2TD3
0.38
0.299
5.2 + 0.29
-138 1.07
0.52
2.23 0.63
0.11
2.29
0.67
2.19
1.21 0.46
1.81 - 0.73
TC -M2TD3
1.6
3.21
4.4
3.28
3.28 2.52
1.56 0.23
2.91 + 0.73
TC -RAR
1.67 0.07 0.07
0.22
0.79
0.89 0.8022.11
1.01 1.01::221
1.53 1 0.64
TD3
0.0
0.2
0.8
0.0
0.0
0.0 0.68
DR
1.65
2.31 0.27
2.08 0.449
1.15
1.22
1.68 - 0.72
M2TD3
1.01
0.19
0.55
1.0 1.43
1.0
1.5
RARL
0.69 0.13
1.3 0.54
0.911
0.47 = 1.92
0.35 1 0.83
0.3 0.71
Table Avg. of normalized static average case performance over seeds for each method
6
Some Theoretical properties of TC-MDPS.
6.1 On the optimal policy of TC
Following Lemma 3.3 of (T) it known that in the rectangular case, there exists an optimal policy of the adversary that is stationary, provided that the actor policy is stationary. The TC-RMDP definition enforces a limitation on the temporal variation of the transition kernel. Consequently, all stationary adversarial policies are constrained by this stipulation. In turn, this guarantees that (under the hypothesis of so-rectangularity) there always exists a solution to the TC-RMDP that is also solution to the original RMDP. In other words: optimizing policies for TC-RMDss do not exclude optimal solutions the underlying RMDP. This sheds an interesting light on the search for robust optimal policies, since TC-RMDPs shrink the search space of optimal adversarial policies. In practice, this is confirmed by the previous experimental results (Figure 2} where the optimal agent policy found by either Oracle-To Stacked TC, vanilla TC actually outperforms the one found by M2TD3 or RARL in the non time-constrained setting.
6.2 Some prophitz-properties for non-stationary TC-RMPDS.
In this subsection we slightly depart from the framework defined in Section and study the smoothness of the robust objective for vanilla TC or Stacked-TC. Th. 2.1 is no longer applicable as s is not
observed. However, can still give smoothness of the objective starting from Lipchichz conditions
8

---

on the evolution of the parameter that leads to smoothness on reward and transition kernel in the following definition 6.1
Definition 6.1 (Reward/Kemell Lipchitz TC-RMDPs [3).. We say that parametric RDMPs is time constrained if the parameter change is bounded through time ie. [ - 1/ S L. Moreover, we assume that this variation in parameter implies a variation in the reward and transition kernel of
vs E S, Va E P. 8, - Pa+1( I aall S Lp :rr(s, (ale,a)--Errttte [rt+1(8,,0) S Lt
From theoretical point of view, TC-RMDP can be seen sequence of stationary MDPs with time indexed reward and transition kernel Tr, P. that have continuity. More formally for M, = (S, A, V, PPt, Tt, (Lep Ir)), we can then define the sequence of stationary MDPs with Lipchitz variation
M (M 31, RYs E Va A, PIPC (s Patil |8,a)ll S Lp ;
(ess,)--++1(s.000 s L.}
Defining r the random variable corresponding the reward function at time step & for stationary MDPs, but iterating with index k, the stationary rollout return at time is G(T,, M,) Σx>0 r/r. Assuming that at a fixed & the reward and transition kernel rı, P. are fixed, the robust objective function is:
min (m, = m- (m EMF
E [ (T, m)]
This leads to the following guarantee for vanilla TC and Stacked-TO TC algorithms.
Theorem 6.2. Assume TC-RMPDS with (Lr, Lp) smoothness. Then Vt NNr E [0,1].
Vt Nt, Vto N+, |J to) RR to + +)) L't.
with L' := (1- Lp
1 Ltp
This theorem states that small variation of the Kernel and reward function will not affect too much the robust objective. In other terms, despite the fact that the TC Bellman operator may not admit a fixed point and yield non-stationary sequence of value functions, variations of the expected return remain bounded. Proof of the Th. 6.2 can be found in Appendix
Conclusion
This paper presents novel framework for robust reinforeement learning, which addresses the limitations traditional methods that rely on rectangularity assumptions. These assumptions often result in overly conservative policies, which not suitable for real-world applications where environmental disturbances are multifactorial, correlated, and time-constrained. In order to overcome these challenges, proposed Process (TC-RMDP). The TC-RMDP framework is capable of accurately capturing the dynamics of real-world environments, due to its consideration of the temporal continuity and correlation of disturbances. This approach resulted in the development of three algorithms: The three algorithms, Oracle-TC, Stacked-TC, vanilla TC which differ in the extent which environmental information incorporated into the decision-making process. A comprehensive evaluation continuous control benchmarks using MuJoCo environments has demonstrated that the proposed TC-RMDP algorithms outperform traditional robust RL methods and domain randomization techniques. These algorithms achieved a superior balance between performance and robustness in both time-constrained and static settings. The results confirmed the effectiveness of the TC-RMDP framework in reducing the conservatism of policies while maintaining robustness. Moreover, we provided theoretical guaranties for Dracle-Tee -C in Th. 2. and for Stacked-TT TC and vanilla TC in Th. 6.2 This study contributes to the field of robust reinforcement learning by introducing a time-constrained framework that more accurately reflects the dynamics observed in real-world settings. The proposed algorithms and theoretical contributions offer new avenues for the development of more effective and practical RI. applications in environments with complex, time-constrained uncertainties.
formulation, the time-Conttaaieed Robust Markov Decision

---

Acknowledgments
The authors acknowledge the support of the DEEL and ENVIA projects, the funding of the Al Interdisciplinary Institute ANITI funding, through the French "Investing for the Future PIA3" program under grant agreement NNR-119P13A-0004. This work benefited from computing resources from CALMIP under grant P21001. Pierre Clavier has been supported by a grant from Region Ile-de-France; DIM Math Innov and also by Fondation Mathématique Jacques Hadamard.
References
[1] Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2):257-285 2005.
[2] Arnab Nilim and Laurent EI Ghaoui. Robust control markov decision processes with uncertain transition matrices. Operations Reseurch, 53(5):780-798, 2005.
[3] Vineet Goyal and Julien -lemen.. Robust markov decision process: Beyond rectangular- ity. arXiv preprint 1eXiv:/811.002/15 2018.
[4] Mengmeng Li, Tobias Sutter, and Daniel Kuhn. Policy gradient algorithms for robust mdps with non-rectangular uncertainty sets. arXiv preprint arXiv:2305. 19004, 2023.
[5] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012.
[6] Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
[7] Wolfram Wiesemann, Daniel Kuhn, and Berç Rustem. Robust markov decision processes. Mathematics of Operations Research, 18(1):153-183 2013.
[8] Michael L Littman. Markov games as a framework for multi-agent reinforcement learning, In Machine learning proceedings 1994, pages 157-163. Elsevier, 1994.
[9] Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applications in continuous control. In International Conference on Machine Learning, pages 6215-6224. PMLR, 2019.
[10] Aurelien Garivier and Eric Moulines. On upper-confidence bound policies for non non-stationary bandit problems. arXiv preprint arXiv:0805.3415, 2008
[11] Fyal Even-Dar, Sham M Kakade, and Yishay Mansour. Experts in a markov decision process. Advances in neural information processing systems, 17, 2004.
[12] Yasin Abbasi Yadkori. Peter Bartlett, Varun Kanade, Yevgeny Seldin, and Csaba Szepesvári. Online learning in markov decision processes with adversarially chosen transition probability distributions. Advances in neural information processing systems, 26, 2013.
[13] Erwan Lecarpentier and Emmanuel Rachelson. Non-stationary markov decision processes, worst-case approach using model-based reinforcement learning. Advances in neural information processing systems, 32, 2019.
[14] Pratik Gajane, Ronald Ortner, and Peter Auer. A sliding-window algorithm for Markov decision processes with arbitrarily changing rewards and transitions. arXiv preprint TrXiv:I805. 10066, 2018.
[15] Wang Chi Cheung, David Simchi-Levi and Ruihao Zhu. Reinforcement learning under drift. arXiv preprint arXiv: / /9066990000 02922,, 2019.
[16] Nicholas Jong and Peter Stone. Bayesian models of nonstationary Markov decision processes.
Planning and Learning in A Priori Unknown Dynamic Domains, page 132, 2005.
10

---

[17] Melkior Ornik and Ufuk Topcu. Learning and planning for time-varying mdps using maximum likelihood estimation. arXiv preprint arXiv:1911.12976 2019.
[18] Takumi Tanabe, Rei Sato, Kazuto Fukuchi, Jun Sakuma, and Youhei Akimoto. Max-min off- off-policy actor-critic method focusing on worst-case robustness to model misspecification. In Advances in Neural Information Processing Systems, 2022.
[19] Shihui Li, Yi Wu, Xinyue Cui, Honghua Dong, Fei Fang, and Stuart Russell. Robust -aget reinforcement learning via minimax deep deterministic policy gradient. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 4213-4220, 2019.
[20] Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. In International Conference on Machine Learning, pages 2817-2888
PMLR, 2017.
[21] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 JEEE/RSJ international conference on intelligent robots and systems (IROS), pages 23-30. IEEE, 2017.
[22] Bhairav Mehta, Manfred Diaz. Florian Golemo, Christopher J Pal, and Liam Paull. Active domain randomization. In Conference on Robot Learning, pages 1162-1176. PMLR, 2020.
[23] Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's s cube with a robot hand. XrXiv preprint arXiv:1910.07111 2019.
[24] Yongyuan Liang, Yanchao Sun, Ruijie Zheng. Xiangyu Liu, Tuomas Sandholm, Furong Huang, and Stephen Marcus McAleer. Game-theoretic robust reinforcement learning handles temporally- coupled perturbations. In The Second Workshop on New Frontiers in Adversarial Machine Learning, 2023.
(25) Eugene Vinitsky, Yuqing Du, Kanaad Parvate, Kathy Jang, Pieter Abbeel, and Alexandre Bayen. Robust reinforcement learning using adversarial populations. arXiv preprint arXiv:2008.0182 2020.
(26) Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho- Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations on state observations. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 21024-21037, 2020.
[27] Huan Zhang, Hongge Chen, Duane S Boning, and Cho-Jui Hsieh. Robust reinforcement learning on state observations with learned optimal adversary. International Conference on Learning Representations, 2021.
[28] Samuel Stanton, Rasool Fakoor, Jonas Mueller, Andrew Gordon Wilson, and Alex Smola. Robust reinforcement learning for shifting dynamics during deployment. In Workshop on Safe and Robust Control of Uncertain Systems at NeurIPS, 2021.
[29] Mohammed Amin Abdullah, Hang Ren, Haitham Bou Ammar, Vladimir Milenkovic, Rui Luo, Mingtian Zhang, and Jun Wang. Wasserstein robust reinforcement learning. arXiv preprint 197.1:190. 2019.
[30] Pierre Clavier, Stéphanie Allassonière, and Erwan Le Pennec. Robust reinforcement learning with distributional risk-averse averse formulation. arXiv preprint arXiv:2206. 220.::2220000.8111, 2022.
[31] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018
[32] Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1-18. 2022.
11

---

A Appendix
The Appendix is structured follow:
In Appendix B proof for fix point of Oracle-TC algorithm for can be found.
In Appendix proof for algorithm Vanilla TC and Stacked-TC can found about robust
objective.
In Appendix the adversary training was sanity-checked within the time-constrained evaluation.
In Appendix
all implementation details are provided.
In Appendix
H
all raw results are presented.
In Appendix
the computer resources and training wall clock time are detailed.
In Appendix the broader impact and limitations are discussed.
In Appendix
K heuristic for choosing the radius L
B Proof of Theorem 2.1
Proof. The Proof similar to [I. using the fact that Pg-+6 belongs to the simplex, we get contraction of the operator and convergence fix point vz-- Not that to converge to the fix point, there is no need of rectangularity.
Recall the recursion
(n41(8, (8)
max min T5 (,( max min Ba a) 71E S un (5', (')) T(a,))EA, bEB
AEa, BEB
First prove that the TC Robust Operator TH is contraction. Let V1, V/2 E R". Fix S, and assume that 7g (8, w) 2 TEV2(s v)) Then fix >> 0 pick π s.t given 8 E E.
inf Epots Ir (s, + 7V (5', (')] (g bEB
(3)
First we pick a probability measure p' such that Puto: b such that Eg (s, VV2 (s', (')) inf Ey [r (s)) (s', tc.
(4)
Then lead
0 < (8, - -TVV2(8, <
inf Ep (r (s)) + > (5', PEB
(s) ([ (2),
(5)
inf Ep [r (s, V22 (6'',''))
(6)
pEB
$ (Ep (s, + +
(s', Φ3) +
(7)
(L, [r (s, (T))) + + ( -),,
(8)
>lEp 112 V/2 +2,
(9)
S yEpp 1/1 2g-V,
Vvg + 2c
(10)
S V V2lloo + 2e.
(11)
where last inequality is Holder's inequality between Ln and Lox norms, use probability measure in the simplex such as Ip' || 1. Doing the same thing but the case where (1)(()) S <TVV2())
holds
12

---

E E -BV V V2 + 2e,
(12)
i.e. |!7gV, - &2 < > V2ll>o + 2c. As we can choose arbitrary small, this establishes that the TC Bellman operator is a 7-contraction. Since TB is contraction operator a Banach space, the Banach fixed point theorem implies that the operator equation TgV = V has a unique solution V = v A similar proof can be done for optimal operator T- The only difference is the maximum operator which is --Lipschitz. So T is also a contraction. Then, once proved that operators are - contraction, following [1] (Th. 5), we have that the fixed point of this recursion is exactly:
0B (s, 1b) min E
Σ 7'relub-1 v, 80 8, b, B, 1/. 4/-1 bb, nT 8t Pvr
(br)en.
bEB
(13)
8 (s. (6) max E+
(s, zi).
(14)
for (optimal) TC Bellman Operator.
C Guaranties for non-stationary Robust MDPS
Recall that we represent non-stationary robust MDPs (NS-RMDP) as a stochastic sequence, {M = {Mi). to of stationary MDPs Me M, where M is the set of all stationary MDPs. Each M, is tuple, (S, A. T,, p²), where S The set of possible states is denoted by S, the set of actions by A, the discounting factor by y, the start-state distribution by ph, and the reward distribution by rr. The reward distribution, denoted by r. S A A(R), the probability distribution of rewards. The transition function, represented by P: : x A - ASS), is the probability distribution of transitions between states. The symbol A denotes the simplex. For all M, E M. we assume that the state space, action space, discount factor, and initial distribution remain fixed. A policy is represented as a function : S + (A)) In general, we will use subscripts f to denote the time evolution during an episode and superscripts & to denote the time step assuming reward or kernel t which is stationary, assuming that the reward function not changing as it time step stationary. That r is the random variables corresponding to the state, action, and reward at time step t for stationary, but iterating with index k.
Definition C.1 ( (Lipschitz of sequence of MDPs). We denote the sequence of kernel and reward function P ( and R (re)etto We define sequence of MDP is L. (Trr Ly) L-ip-hithtte im = ) E M- with
M₁ { [Mil 3b,, Lp) E E A. 8,a) - (.41(-18,0)) S Lp
(s) (r,) < L.}
Assuming that for a time steps the reward function is stationary, we can compute the average return as:
Definition C.2. Non-robust objective function, assuming that G(T, Mi) = Enzo o the return is we assume stationary with reward function Tt
J ( M.) = (1-2)-- Σ (s, Mi) Σ ((r(s))((ss)))
(15)
#ES
aEA
with d the state occupancy measure defined in (66)
Definition C.3 (Robust (optimal) Return of NS-RMDPs). Let a return of for any m E M, be
G(T, M)) == Exe xh with kernel transition Pt following n, with Vk, t, E [0. 1), and the Robust non-stationary expected return with variation of kernel
Let the robust performance of h for episode I be
J (TT, 1) :=
min (mm EMP
E [ (T, m]]
13

---

D Proof Theom 6.2
WENT VO
JR (6 +1) s L't.
with := == (1-ypapp 1 Lr)
Proof of Theorem 6.2 First, this difference can be upper bounded in the non robust case as: By definition, we can rewrite non-robust objective function and occupancy measure as.
d (s, M.) = -1 (1-7) Σ
Pr( = =,
(16)
k=0
(T. (1- Σ d" (s,
Σ л(a s)ri(s,a).
(17)
First, we can decompose the problem into sub-problems such that
to+t-1
V Vto E N
- -(r, tb) s
Σ
[e(, - J(T Mc41))
(18)
t==oo
using triangular inequality. Looking at differences between two time steps:
(1 - JmT, Mc) - M+1))
= Σ dl M,) Σ T(a a)) - - dr (s, Mc+1) Σ
x(a s))c+1(s,a)
SES
aEA
SES
aEA
T(a (d (s, r() - dr (s, M441) (ez SES
ΣΣ
T(a 8) (d" (s, Mi) (ri41(s, (r)(s,a) Ret1(s, d= d', (s))))))4))ssa))
SES a
Σ Σ
T(a (dF (s, M,) d= (s, (ee))
SES A
Σ
Σ
(a s)d (s, ()) -x+
#ES aEA
Taa (d (s) d (s. Mc+1) (m42(),)))
ΣΣ
x(a s) (s. M.) (n((s,)))))))))))))))){)))) as)
#E8 aEA
(
Taa (d (8, M.) d (s, Mc+1) LR ΣΣ
<(a s)d (s, M.)
BES
RES
dd M,) - d= (8, (..M411 Top
where (a) is triangular inequality, (b) is definition of of supremum of reward in the assumptions and reward bounded by 1. Then, let PM E IR S|x S be the transition matrix ( (' in rows and in
columns) resulting due to क and Pr, i.e. Vt, PT (s',s) == Pr (S.+1 = S. = 8, म, M,), and let d (t, M,) R denote the vector of probabilities for each state, then Finally we can easily bound the difference of occupation measure as
14

---

Σ 1dt M.) d= (s, (e.1))))
(19)
FES
S 57(1 - -) Σ
Σ
-6)
Pit1 (5', d" M.)
(20)
E'ES SES
51(1 - -1)- Σ
Σ
Pe 18) - -P (2116,,) (8,8)) dt Mc)
(21)
s' SEES #ES
--(1-7)~ ΣΣ
Σ
() s, Pr (s' 8 a, Mc+1) d= (s, M.)
(22)
E'ES #ES a
ST(1 - 7)
Σ
<(a [ Pr (,,,,
s,,, M.) - PP a, M(+1] df M))
(23)
E'ES sES AEA
=7(1 (11-))-1 ΣV
Σ
T(a s)d M,) Σ
Pr (s' s, Pr s, (,,0,
(24)
aEAaEA
s'ES
S1(1 - -7)
Σ
Σ
Taa s)d
M.) Lp
(25)
#ES aEA
rhp (1-7)*
(26)
which gives regrouping all terms:
+ 7hp [J(T, - M+11) 1- (1 - 7)2
Ly
(27)
where the stationary MDP Mc+1 can be chosen the minimum over the previous MDPs at time step t such as |Pr (s' 8, a.a,M., M,) Pr (s') 88, a, Mc+1)l T.p Rewriting previous equation (27). it holds that
[Br, (T, min
E.P (T, m]]]
L,
7hp
L'.
(28)
m= (1 1e44
1-7
(1-7)2
Now considering non robust objective
J (T, J (IT, 1 +++1) 1)
(29)
min
E G (x, m)]
min
E (G m)]
(30)
m= [m/] EMC
(m EMC
min
E GG (T,, m]]
min
min
E E (G mT]
(31)
m= (m/)) Vo EMÉ
m= EMP m= =m
E B (T.
min m= (m#
E
[G (IT, m)]]
(32)
m= (mm EM!
where first equality is the definition of the robust objective, second equality is decomposition of minimum across time steps and final inequality is simply a property of the min such as min min bj & sup |a b).
Finally plugging 28 in 32
it holds that
15

---

JR (T, HH (TT, 1)
(33)
Lr 7Lp
min m=/m/by- EMC
Er, [G (TT
min m= (m EMC
Er, [G (T, (TT,
- ? (1 (1-7)²
(34)
:=L'.
(35)
Combining times the previous equation gives the result:
Vt WENT Vto ENt, 1"" +1)l s L't.
with L' (T-ي + ).
E
Implementation details
E.1
Algorithm
Algorithm 2 Time-constrained robust training
Input: Time-constrained MDP: (S, A, P,,,, L), Agent т, Adversary क for each interaction time step t do
ag ~ Te (8t, thx)
// Sample an action with Oracle-TO
dg (8t, a1-10 St-1)
// Sample action with Stacked-TC
ag ~ ,, (5t)
// Sample an action with TO // Sample the worst TC parameter // Sample a transition + BU [ Or, (st, at) ,r, (411) // Add transition to replay buffer
d TT(st; Ot; 1/22)
8141 ~ Poup (8t, (8t,at) az)
{st, ai r(5;; ai), 4/s, U441 8111henN ~ B // Sample mini-batch of transitions
0. < - -Vo. (o(e))
// Critic update phase // Actor update
0g - 0. aVo, (fa)
Φe + Φe + aV L (c)
// Adversary Critic update phase
Φ. 6a aV L (da)
// Adversary update
Note that in Time-constrained robust training Algorithm in section E.1 LQ and I, as defined by 31 double critics and target network updates are omitted here for clarity
In Table 4 for the stack algorithm, 8; is defined as 8; +- U 85-1 Ua for Stacked-TC and for Oracle-TC version, 8; + 8, U
Loss Function
Equation
Lo=. (TC-RARL)
E Qui ax) r(s as) + minj=112 Qo. (s1+1:
(f.)) (TC-RARL)
(s,, no, (s:))]
(O.) (TC-RARL)
E Qu, (86, as. # # ass..:),
La (0.) (TC-RARL)
E Qo, (8i, as) r(s,,( a.) + minn=11.2 Qo, (81+1, no, (8c+1), no. (8 Cst ()++1))
LQo. Shared (TC-M2TD3)
E Qs, 0z) r(s;, a;) + minj=1.2 Qo, (8++1, me, (8++1), Re, (8141, O141 w1+1))]
I. (O.) (TC-M2TD3)
E QQ, (sL, 04, no, (s1, a.). v.)
(O.)) (TC-M2TD3)
[Q, (si, 0s Ne, (8s, ag, vx))]
Table 4: Summary of Loss Functions for TD3 in TC-RARL and TC-M2TD3
16

---

E.2 Neural network architecture
We employ a consistent neural network architecture for both the baseline and our proposed methods for the actor and the critic components. The architecture's ' design ensures uniformity and comparability across different models.
The critic network is structured with three layers, as depicted in Figure 3a the critic begins with an input layer that takes the state and action inputs, which then passes through two fully connected linear layers of 256 units each. The final layer is a single linear unit that outputs real-valued function, representing the estimated value of the state-action pair.
The actor neural network, shown in Figure 3b also utilizes a three-layer design. begins with an input layer that accepts the state input. This is followed by two linear layers, each consisting of 256 units. The output layer of the actor neural network has a dimensionality equal to the number of dimensions of the action space.
S
Linear 256
Linear 256
Linear 1
RR
S
Linear 256
Linear 256
Linear
RR RIAI
(a) Critic neural network architecture
(b) Actor neural network architecture
Figure 3: Actor critic neural network architecture
E.3
M2TD3
We utilized the official M2TD3 18 implementation provided by the original authors, accessible via the GitHub repository for M2TD3 and Oracle M2TD3.
For the TC-M2TD3 or variants, we implemented the M2TD3 algorithm as specified. simplify our approach, we omitted the implementation the multiple & network and the system for resetting øi We replace with adversary which S A x V > T which minimize Q(s, 16).
E.4 TD3
We adopted the TD3 implementation from the CleanRL library, as detailed [3
Sanity check on the adversary training in the time-constrained evaluation
natural question arises regarding the worst time-constrained perturbation. Whether we adequately trained the adversary in the loop, or its suboptimal performance might lead to overestimating the trained agent reward against the worst-case time-constrained perturbation. We monitored the adversary's 's performance during its training against a fixed agent to address this. The attached figure shows the episodic reward (from agent's perspective) during the adversary's training over million timesteps, with perturbation radius of I, = 0.001. Each curve is an average of over 10 seeds. The plots show rapid decline in reward during the initial stages of training, followed by quick stabilization. The episodic reward stabilizes early in the Ant (Figure 4ah environment, indicating
17

---

Hyperparameter
Default Value
Policy Std Rate
0.1
Policy Noise Rate
0.2
Noise Clip Policy Rate
0.5
Noise Clip Omega Rate
0.5
Omega Std Rate
1.0
Min Omega Std Rate
0.1
Maximum Steps
5e6
Batch Size
100
Hatomega Number
5
Replay Size
le6
Policy Hidden Size
256
Critic Hidden Size
256
Policy Learning Rate
3e-4
Critic Learning Rate
3e-4
Policy Frequency
2
Gamma
0.99
Polyak
5e-3
Hatomega Parameter Distance
0.1
Minimum Probability
5e-2
Hatomega Learning Rate (ho [r)
3e-4
Optimizer
Adam
Table 5: Hyperparameters for the M2TD3 Agent
Hyperparameter
Default Value
Maximum Steps
5e6
Buffer Size
1 1 10t
Learning Rate
3 x 10-1
Gamma
0.99
Tau
0.005
Policy Noise
0.2
Exploration Noise
0.1
Learning Starts
2.5 x 10*
Policy Frequency
Batch Size
256
Noise Clip
0.5
Action Min
-1
Action Max
Optimizer
Adam
Table 6: Hyperparameters for the TD3 Agent
quick convergence. Similarly, in the HalfCheetah (Figure 4b) environment, the reward shows a sharp initial decline and stabilizes, suggesting effective training. For Hopper (Figure/4c), the reward decreases and then levels off, reflecting adversary convergence. Although the reward is more variable in the HumanoidStandup (Figure 4d) environment, it ultimately reaches steady state, confirming adequate training. Finally, in the Walker environment, the reward pattern demonstrates a quick drop followed by stabilization, indicating convergence. These observations confirm that the adversaries were not undertrained. The rapid convergence to stable performance across all environments ensures the accuracy of the worst time-constrained perturbations estimated during training.
G Uncertainty set in MuJoCo environments
The experiments of Section follow the evaluation protocol proposed by [8] and based on MuloCo
environments (5). These environments are designed with a 3D uncertainty sets. Table 7 lists all environments evaluated and their uncertainty sets. The uncertainty sets column defines the ranges of
18

---

12000
5000
6200
Hewn apweteplett-o mesndt
#200
2000
(a) Ant: Episodic reward of the trained agent dur- ing adversary training
(b) HalfCheetah: Episodic reward of the trained
agent during adversary training
Hopper
HumonekeStandup
3200
163000
Vear 401004 064800
1500
1000
80000
300
62000
(c) Hopper: Episodic reward of the trained agent during adversary training
HumanoidStandup: Episadic reward the
trained agent during adversary training
Welker
4000
4000
UK
NAHL
NOTh
Starkodd TC-M3T11
Oracle RAAL
TDS
2300
TC-M2TD3
TC-W2TD3
IC-NARL
1300
200
(c) Walker: Episodic reward of the trained agent during adversary training
(f) Legend for algorithm
Figure 4: Episodic reward of the trained agent during the training of the adversary across different environments. Each plot represents the performance over 5 million timesteps, with rewards averaged across 10 seeds. The perturbation radius is set I, 0.001 for adversaries.
variation for the parameters within each environment. The reference parameters column indicates the nominal default values. The uncertainty parameters column describes the physical meaning of each parameter.
H Raw results
Table 8 reports the non-normalized time-constrained (with radius of L, 0.001) worst-case scores, the static worst case score
averaged across 10 independent runs for each benchmark. obtained by each agent across grid of environments, also averaged across 10 independent runs for
19

---

Table 7: List environment and parameters for the experiments
Environment
Uncertainty set P
Reference values
Uncertainty parame-
ters
Ant
0.1,3.0
(0.33, 0.04, 0.06)
torso mass; front left leg mass; front right leg mass
00.01, 3.0]
x
00.01,3.0 3.0)
HalfCheetah
00.1, x (0.1, x
(0.4, 6.36, 1.53)
world friction; torso mass; back thigh mass world friction; torso mass; thigh mass
0.1,3.0
Hopper
0.1,3.0 x (.001...000
(1.00, 3.53, 3.93)
0.1, 4.0)
HumanoidStanduu
0.1, 16.0)
(8.32, 1.77, 4.53)
torso mass; right foot mass; left thigh mass world friction; torso mass; thigh mass
0.0, x 8.0]
Walker
0. 0.1, x
(0.7, 3.93,
0.0,
Table 8: Avg. of time-constrained worst-case performance over 10 seeds for each method
Environment
Ant
HalfCheetah
Hopper
HumanoidStandup
Walker
Method
Oracle M2TD3
5768 + 395
3521 + 187
1241 125
116232 + 1454
4559 757
Oracle RARL
4387 + 667
-50 + 99
344 + 113
68979 + 10641
1811 + 342
Oracle TC-M2TD3
7268 + 704
7507
= 284
3386 = 323
114411 + 16973
5344 536
Oracle TC-RARL
7534 781
7526 311
3169 = 311
101182 + 12083
4783 + 382
Stacked TC-M2TD3
6502 + 450
6377 517
3047 + 394
85524 = 11448
5724 828
Stacked TC-RARL
6955 + 690
5319 + 223
1747 153
107913 + 5514
4152 + 483
TC-M2TD3
7181 591
6516 2
25
129183 9120
4964 531
TC-RARL
7473
4089
1475 158
108669 + 17764
3971 + 351
DR
7247 925
4986
1642 104
+
4380 + 488
M2TD3
5622 + 435
405
1120 220
102839 + 12987
4078 + 644
RARL
4348 + 574
366
240 104
106768 4051
2388 + 559
TD3
2259 +: 424
1808 + 503
777 407
104877 + 12063
1893 + 361
each benchmark. Table T0 reports the static average case score obtained by each agent across a grid of environments, also averaged across 10 independent runs for each benchmark.
H.1 Fixed adversary evaluation
At the beginning of each episode, t/y ~ ((V ) is selected for every fixed adversary. The episode length is 1000 steps. To begin with, the random fixed adversary simulates stochastic changes. It selects a parameter th, at each timestep within a radius of = 0.1, which is 100 times larger than in training methods. This tests the agents' adaptability unexpected changes. In contrast, the cosine fixed adversary introduces deterninistic changes using a cosine function. The radius
Table Avg. of raw static worst-case performance over seeds for each method
Ant
HalfCheetah
Hopper
Humanoid
Walker
dr
19.78 + 394.84
2211.48 88 + 915.64
245.01 01 + 167.2
G4856.87 : 30048.87
1318.36 + 777.51
m2dd3
2322.73 649.3
2031.9 409.7
273.6 + 131.9
71900.97 : 24317.35
2214.16
1330.4
oracle m2td3
2370.93 473.56
319.67
599.26
267 A1 111.47
93123.84 + 26696.17
736.59
1: 911.76
oracle rarl
1396.88 777.46
-78..84 54.36
167.5
45635.24 + 15974.444
459.74 4:
oracle m2d33
1201.4 + 618.23
4273.31 + 246.91
168.7 + 217.94
58687.26 + 22321.77
710.99
799.08
oracle to razl
1328.27 89
893.22 3458.52
150.54 33.12
73276.78 78 91
1299.88
812.63
960.11 744.01
-2118 218.3
170.46 45.73
67821.86 + 21555.2
360.31
186.06
stacked m2dd3
-242.98 212.98
3467 418.64
289.37
182.18
58515.04
19186 25
2475.58
1057.03
stacked to varl
37.77 2 320.71
1414.37 + 876.91
344.37 + 190.1
77357.17
+ 18186.34
1518.86 668.13
-1233.64
-54621 158.81
69.3 42.77
64577.24
16606.51
m2td3 te rarl
-271.34 191.15
9286.67 + 603.
333.36 36 = 60.04
73428.2
17879.28
2603.59 + 706.63
209.04 575.89
1738.59 + 782.71
376.01 - 155.4
74840.68 1 33-96.115
1513.65 + 1239.3
20

---

Table 10: Avg. of raw static average case performance over seeds for each method
name algo-name
HallChhetahh
Hopper
HumanoidStandup
Walker
dr
7500.58 + 143.38
6170.33 442.57
1688.36 :: 225.59
110939.89 99 =: 22292.
4611.24
463.42
m2td3
017-11-2 + 316.95
1000.98
314.76
1193.32 + 254.9
109598. 43 12992.33
4311.2 577
oracle m2dd3
5968.22 068.2112 + 237.32
1830.18:2
390.96
1249.62 - 212.74
118273.54
13691.06
4616.05 = 407.94
oruele rarl
1684.83-4 + 648.
36.19 11992216.00 216.52
380.39 + 110.14
76920.58 = 26135.3
1451.39 + 1132.87
oraclo-te m2td3
7739. 65 25-165
9536,92 129.14
3281.92 61.79
119737
12697.2 2
5112.85 199.78
oracle-te-rarl
7889. 1 56.0
9474.0
3071.17 220.39
104348 01 2-22-99 98
5220.2 + 318.07
rarl
1650.55 - 395.03
206.71
276.37 52.42
10176187
17400.855
2493.26
1113.74 1113.77
stacked m2td3
6912.76 1116.81
8583.55 479.97
3124.06 + 133.27
88039.74
15138.11
5809.54 + 703.92
stacked-te-rull
7123.07 07 + 03:333233 33
6130.71 384.05
2072.75 306.48
11081332 = 19887 .2
1596.79 = 619.2
vanilla
2350.58 337.12
T33.18 + 362.06
100533.0 12298.37
685.39
vanilla-tem2dd3
7366.9 9 = 169.58
8167.64 397.42
2756.5 + 273.91
1 130305 38 22565.1
5070.71 = 315.7
vanilla-to-rarl
7558.58 044.588-2 + 198.37
6092.61 365.68
1358.26 + 242.17
108635.71 71
19848.
4325. 42 = 283.04
Environment Method
Ant
HalfCheetah
Hopper
HumanoidStandup
Walker
Oracle TC-M2TD3
7782 + 915
8805 + 165
2365 + 199
116791 12572
5148 = 558
Oracle TC-RARL
8041 + 470
8727 + 227
2120 + 96
107733 = 11975
4896 = 326
Oracle M2TD3
5830 + 542
4445 + 186
1222 + 111
118861 = 1365
4584 787
Oracle RARL
4628 + 514
-51 = 60
370 = 141
81583 + 16526
1829 = 356
Stacked TC-M2TD3
6888 + 738
7400
2114 + 138
88436 10750
5278 = 845
Stacked TC-RARL
7
5992 427
1940 + 93
106213 + 6770
4430 = 389
TC-M2TD3
7156 + 692
7530
2157 + 112
129599 + 13556
4931 = 568
TC-RARL
5751 #
1445 + 203
105144 16813
4112 = 329
DR
7572 629
6048 349
1416 168
105677 = 16333
4371 = 431
M2TD3
5588 + 516
40
1018 + 271
107692 + 10414
4176 + 783
RARL
4347 + 567
250
390 130
103583 9217
1925 + 501
TD3
4017 + 518
2028 + 1250
1944 + 246
91205 + 11350
2860 + 419
Table I1: Avg. performance against time-constrained fixed random adversary with a radius L L= 0.1 over 10 seeds for each method
of L 0.1 scales the frequency of the cosine function, ensuring smooth and periodic variations. Additionally, a phase shift at the start of each episode ensures different starting points. Meanwhile, the linear fixed adversary employs a linear function. The parameters change linearly from the initial value to either one of vertex of the uncertainty set T over 1000 steps. Furthermore, the exponential fixed adversary uses an exponential function. Parameters change exponentially from e
the initial value to either of vertex of the uncertainty set V over 1000 steps. This ensures smooth and predictable variations. Similarly, the logarithmic fixed adversary uses logarithmic function. Parameters change logarithmically from the initial value to either of a vertex of the uncertainty of the uncertainty set over 1000 steps, ensuring smooth and predictable variations. Agents trained under the tine-constrained framework outperform all baselines in all environments for each fixed adversary, except when compared to the oracle TC method, which has access to v. In this case, the stacked-TC TC methods outperform all baselines in all environments for the cosine, logarithmic, and exponential adversaries and outperform the fixed adversary baseline 4 out of 5 instances for the random and linear fixed adversaries.
H.2 Agents training curve
We conducted training for each agent over a duration of million steps, closely monitoring the cumulative rewards obtained over trajectory spanning 1,000 steps. To enhance the reliability of our results, we averaged the performance curves across 10 different seeds. The graphs in Figures to 15 illustrate how different training methods, including Domain Randomization, M2TD3, RARL. Oracle RARL. .raale M2TD3, TC RARL., TC M2TD3, Stacked TC RARL and Stacked TC M2TD3, impact agent performance across various environments.
21

---

Table 12: Avg. performance against constrrnst fixed cosine adversary with radius L = 0.1
over 10 seeds for each method
Environment
Ant
HalfCheetah
Hopper
HumanoidStandup
Walker
Method
Oracle M2TD3
5528 + 637
3453 + 266
1016 + 48
119813 = 3281
3589 = 863
Oracle RARL
4550 + 626
-79 =
371 = 140
74116 7890
1593 = 326
Oracle TC-M2TD3
7586 + 1345
8174 : 383
1946 + 104
115506 = 12470
4464 = 781
Oracle TC-RARL
7522 1435
7838 810
1735 +
110535 12702
4442 = 591
Stacked TC-M2TD3
6269 + 849
7173 + 509
1734 157
88157 + 10654
4888 = 567
Stacked TC-RARL
6510 + 1395
5385 445
1519 118
105696 = 5243
3848 = 404
TC-M2TD3
6350 = 769
6797 609
1413 167
130892 = 11544
4611 = 632
TC-RARL
712
348
1172 129
102864 = 13308
3548 = 545
DR
6975 992
+
1091 + 169
109227 + 17068
3851 = 612
M2TD3
5330 + 684
321
938 +
108136 + 9755
4126 = 644
RARL
4153 + 602
+
363+55
103366 7604
1689 + 465
TD3
4025 + 557
270
1
94352 10101
2020 = 355
Table 13: Avg. performance against fixed linear adversary over 10 seeds for each method HalfCheetah
Environment
Ant
Hopper
HumanoidStandup
Walker
Method
Oracle M2TD3
5811 +: 121
3560 + 167
1216 + 326
118829 + 846
4431 615
Oracle RARL
4447 + 600
-122 + 64
308 62
81498 12860
1503 450
Oracle TC-M2TD3
7919 + 595
7495 + 268
2983 + 252
117610 + 11682
4952 + 415
Oracle TC-RARL
8069 151
7443 236
2805 + 352
110314 9354
4613
Stacked TC-M2TD3
7003 + 812
6365 + 335
2714 + 198
89556 11115
5256 = 675
Stacked TC-RARL
7328 251
5301 + 86
1616 + 137
105137 7903
4234 = 385
TC-M2TD3
7622 413
6451 + 246
2228 + 131
129501 + 10326
4844 417
TC-RARL
7675 + 143
4881 251
1277 + 288
105566 + 15551
3906 + 381
DR
7713 412
$290 103
1419 122
108711 + 16696
4307 + 309
M2TD3
5444 225
3810
970 = 323
106311 = 9771
4128 = 727
RARL
4651 + 446
138
346 = 22
101477 = 8947
1894 = 515
TD3
3493 + 475
1462 + 1246
1722 + 366
89934 + 10644
2396 = 416
Table 14: Avg. performance against a fixed logarithmic adversary over 10 seeds for each method Hopper
Environment Method
Ant
HalfCheetah
HumanoidStandup
Walker
Oracle M2TD3
5561 +: 580
3086 +: 163
957 + 165
119214 2525
4148 + 630
Oracle RARL
4911 + 177
-145 + 67
293 + 49
79522 + 13470
1618 + 142
Oracle TC-M2TD3
7963 + 796
6625 + 204
2577 + 171
116664 11798
4818 + 451
Oracle TC-RARL
8061 + 821
6532 + 304
2572 + 177
108213 + 10684
4375 + 382
Stacked TC-M2TD3
7315 + 478
5863 + 290
2283 + 122
87691 + 11133
4931 + 735
Stacked TC-RARL
7514
4770 + 145
1426 197
104193 8030
3939 369
TC-M2TD3
7910 90
5657 + 280
1702 + 226
128467 + 10762
4664 1 412
TC-RARL
7686 + 208
4475 + 238
1082 + 298
104835 16040
3636 428
DR
7883 67
4721 146
1166 + 332
106171 16867
3995#31
M2TD3
5371 + 279
3565 + 105
802 - 271
104002 11606
4206 712
RARL
4620 + 763
231 110
340 + 44
102004 + 9925
1919 + 499
TD3
3678 + 623
576 983
1389 + 327
88952 = 11367
1956 360
22

---

Table 15: Avg. performance against fixed exponential adversary over 10 seeds for each method
Environment Method
Ant
HalfCheetah
Hopper
HumanoidStandup
Walker
Oracle M2TD3
5860 + 93
3780 + 137
1271 + 224
119205 + 1217
4767 + 815
Oracle RARL
4585 + 674
-88 79
302 41
82063 = 13274
1611 342
Oracle TC-M2TD3
7491 + 624
8256 + 269
2894 = 244
118476 + 11683
5161 + 289
Oracle TC-RARI.
7724 + 368
8000 250
3036 293
110092 10754
4650 503
Stacked TC-M2TD3
6903 + 365
7041 = 302
2721 + 214
91077 = 11945
5310 882
Stacked TC-RARL
7061 + 222
5741 + 249
1825 + 145
104793 + 6758
4376 342
TC-M2TD3
7318 + 299
7139 + 387
2408 113
129966 10823
4910 663
TC-RARL
7441 + 133
5326 220
1457 + 163
106491 + 14605
4017 439
DR
7389 + 206
5691 + 121
1564 + 99
106290 + 17502
4224 + 660
M2TD3
5466 318
3909 332
1062 + 272
107097 9551
4274 582
RARL
4556 + 729
228 181
351 + 24
102096 8291
2053 + 493
TD3
3771 + 228
2302 343
2201 + 219
90496 = 9487
2768 + 538
Table 16: Average wall-clock time for each algorithm
Wall-clock time
TD3
14h
M2TD3
16h
RARL
18h
TC
16h
Stacked TC
16h
Oracle TC
16h
I Computer ressources
All experiments with single NVIDIA RTX 4090 GPU. Averages and standard deviations were computed from 10 independent repetitions of each experiment.
desktop machine (Intel 19, 10th generation processor, 64GB RAM)
J Broader impact
This paper aims to advance robust reinforcement learning. It addresses general mathematical and computational challenges. These challenges may have societal and technological impacts, but do not find it necessary to highlight them here.
J.1 Limitations
While our proposed Constr-Cntt Robust Markov Decision Process (TC-RMDP) framework significantly advances robust reinforcement learning by addressing multifactorial, correlated, and time-dependent disturbances, several limitations must be acknowledged. The TC-RMDP framework assumes that the parameter vector v that governs environmental disturbances known during training. In real-world applications, obtaining such detailed information may not always be feasible. This reliance on precise parameter knowledge limits the practical deployment of our algorithms in environments where zú cannot be accurately measured or inferred. Our approach assumes that the environment's 's dynamics can be accurately parameterized and that these parameters remain within predefined uncertainty set V. This assumption might not hold in more complex or highly dynamic environments where disturbances are not easily parameterized or when the uncertainty set cannot comprehensively capture all possible variations. Consequently, the robustness of the learned policies might degrade when facing disturbances outside the considered parameter space. Addressing these limitations in future work.
23

---

6200
0300
4020
2000
2000
(a) Training Ant with Domain Random-
(b) Training curve HalfCheetah with Domain Randomization
ization
Hopper
HumenoksStondupp
2200
120000
100000
382
60000
40000
0
0
(c) Training curve Hopper with Domain Ran- domization
Training curve on HumanoidStandup with Do- main Randomization
Valker
2003
3003
2000
12f
(c) Training curve on Walker with Domain Ran-
domization
Figure Averaged training curves for the Domain Randomization method over seeds
K A heuristic for choosing the radius L
We used simple heuristic to select the radius in our experiments. First, we normalized the parameters of the uncertainty set, scaling them lie within the range [0, for each dimension. We then divided this range by the number of steps in an episode, denoted as ep, yielding L = 1. Our experiments, with ep = 1000, resulted in a radius of = 0.001. This choice ensures that any initial parameter th move to any point within the uncertainty set's parameters for episode.
ep
24

---

3032
4000
2000
(a) Training Ant with M2TD3
(b) Training HalfCheetah with M2TD3
Hopper
120000
2350
100000
2000
60000
40000
(d) Training curve HumanoidStandup with M2TD3
(c) Training curve Hopper with M2TD3
Valker
2003
2000
3000
(c) Training curve on Walker with M2TD3
Figure 6: Averaged training curves for the M2TD3 method over 10 seeds
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We propose three distinct constrrnnd robust mdp algorithm algorithms, each using varying levels of environmental information, and evaluate them extensively on continuous control benchmarks. Our results demonstrate that these algorithms yield efficient tradeoff between performance and robustness, outperforming traditional deep robust RL methods in time-constrained environments while preserving robustness in classical benchmarks.
Guidelines:
25

---

3012
-2000
2000
2000
-1000
4000
230
-5000
(a) Training curve Ant with RARL
(b) Training curve HalfCheetah with RARL
HumcnakkStond.up
Hopper
105000
50000
80000
60000
200
(d) Training curve HumanoidStandup with RARL
(c) Training curve on Hopper with RARL
Valker
2520
2000
1500
3000
(c) Training curve Walker with RARL
Figure 7: Averaged training curves for the RARI. method over 10 seeds
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
26

---

14000
6000
17000
6200
1000
2000
(a) Training curve Ant with TD3
(b) Training HalfCheetah with TD3
Hopper
HumcnakkStond.p
1912
110000
3032
2300
50000
392
(c) Training curve Hopper with TD3
(d) Training curve HumanoidStandup with TD3
3003
2003
2000
(c) Training curve Walker with TD3
Figure 8: Averaged training curves for the TD3 method over 10 seeds
Justification:
We have section in the appendix discussing the limitations of our work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
27

---

-2000
2060
-1000
1300
-4000
-3000
(b) Training HalfCheetah with Oracle RART.
(a) Training curve Ant with Oracle RARI.
HumcnoksStond.pp
Hopper
120000
110000
1040
300000
50000
80000
000
70000
200
stop
(d) Training curve HumanoidStandup with Ora- elc RARL
(c) Training curve on Hopper with Oracle RARL
Walker
1020
3500
1500
3000
500
12f
(c) Training curve on Walker with Oracle RARL
Figure 9: Averaged training curves for the Oracle RARL method over 10 seeds
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
28

---

6003
(b) Training HalfCheetah with Oracle M2TD3
(a) Training curve Ant with Oracle M2TD3
HumenoksStondupp
Hopper
210000
120000
3032
2500
80000
60000
1000
535
(d) Training curve HumanoidStandup with Ora- M2TD3
(c) Training curve on Hopper with Oracle M2TD3
Malker
2000
12f
(c) Training curve on Walker with Oracle M2TD3
Figure 10: Averaged training curves for the Oracle M2TD3 method over 10 seeds
judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Hypotheis of proof are well stated in Th 6.2 and 2.1
Guidelines:
The answer NA means that the paper does not include thcoretical results.
29

---

6003
(b) Training HalfCheetah with Oracle M2TD3
(a) Training curve Ant with Oracle M2TD3
HumenoksStondupp
Hopper
210000
120000
3032
2500
80000
60000
1000
535
(d) Training curve HumanoidStandup with Ora- M2TD3
(c) Training curve on Hopper with Oracle M2TD3
Malker
2000
12f
(c) Training curve on Walker with Oracle M2TD3
Figure 11: Averaged training curves for the Oracle M2TD3 method over 10 seeds
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
30

---

10000
8000
6000
6000
8000
2000
(b) Training curve HalfCheetah with Oracle TC- RARL.
Training curve on Ant with Oracle TC-RARI.
Hopper
HumenokkSSond.pp
220000
2300
300000
2200
60000
300
40000
0
(c) Training curve Hopper with Oracle TC-
(d) Training curve HumanoidStandup with Ora-
RARL
clc TC-RARL
Malker
0000
3003
2000
2000
stag
12f
(c) Training curve on Walker with Oracle TC-
RARL
Figure 12: Averaged training curves for the Oracle TC-RARL method over 10 seeds
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide the algorithm, all detail implementations, hyperparameters.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
31

---

10000
1000
$000
5000
8000
2000
2200
0
(b) Training curve HalfCheetah with Oracle TC- M2TD3
(a) Training curve Ant with Oracle TC-M2TD3
Hopger
Humcc Humcnok-Stonduup
130000
2200
Ke/ogk (PAN
100000
1500
60000
40000
0
0
(c) Training curve Hopper with Oracle TC- (d) Training curve HumanoidStandup with Ora- M2TD3
clc TC-M2TD3
Malker
4000
0003
3000
12f
(c) Training curve on Walker with Oracle TC-
M2TD3
Figure 13: Averaged training curves for the Oracle TC-M2TD3 method over 10 seeds
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
detailed
32

---

5000
7032
5000
4000
2000
(a) Training curve Ant with Stacked TC- M2TD3
(b) Training curve on HalfCheetah with Stacked TC-M2TD3
Hopper
HumenokkStonddup
3032
100000
50000
2000
80000
70000
1000
60000
332
50000
40000
3
0
(c) Training curve on Hopper with Stacked TC- M2TD3
(d) Training curve HumanoidStandup with
Stacked TC-M2TD3
Walker
0000
3003
8000
3003
12f
(c) Training curve on Walker with Stacked TC-
M2TD3
Figure 14: Averaged training curves for the Stacked TC-M2TD3 method over 10 seeds
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
33

---

5032
1000
2000
1000
0
(b) Training curve on HalfCheetah with Stacked TC-RARI.
(a) Training curve Ant with Stacked TC-RARL
Hopper
Hume Sumnnoud
2200
100000
1500
80000
40000
0
(c) Training curve on Hopper with Stacked TC-
(d) Training curve on HumanoidStandupp with
RARL
Stacked TC-RARL
Walker
3003
9700
3003
12f
(c) Training curve on Walker with Stacked TC-
RARL
Figure 15: Averaged training curves for the Stacked TC-RARL method over 10 seeds
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We provide the code with clear instruction, to reproduce the experiments.
34

---

Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc)
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
7 //nips.c
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if' applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provides all hyperparameters, optimizer and reinforcement learning algo- rithm. All those details is provided in the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We profide our computers resources to runs all experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
35

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have section in the appendix discussing of the computer resources for our work.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question:
Does the research conducted in the paper conform, in every respect, with the httpp//ccc/bubitttctcccctuuueeee
NeurlPS Code of Ethics
Answer: [Yes]
Justification: Our contribution aims safer RL algorithms, and conducted our experi- ment of open source benchmarks.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., . if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss on the broader impact in the conclusion and the appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
36

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The authors cite the original paper that produced the code package dataset.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
37

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
38


Reviews:
[{"review_id": "nnzswOU43M", "reviewer": "Reviewer_stMZ", "summary": "The paper defines a robust training method for MDPs under uncertainty in the environment dynamics. Current methods assume sa-rectangularity, where the transition dynamics from consecutive states are independent of one another. The authors argue that this assumption is unrealistic and leads to overly conservative policies. Alternatively, this paper breaks the independence assumption in the environment dynamics to use a time-step evolution approach. This paper presents a Time-Constrained TC-MDP where the Time-Constrained Parameterized MDP kernels evolve each time-step constrained to be Lipschitz continuous. The model is trained in an adversarial manner, where the adversarial policy is learned to evolve the model dynamics but is limited to be L-close to the previous dynamics by definition. This learning method is tested under different variant conditions (cosine, exponential, linear, and logarithmic) with positive results. Robustly trained agents that learn under the worst TC adversities outperform vanilla robust algorithms that assume time independence in the model dynamics.", "questions": "- In Algorithm 1, should is be: $b_{t+1} = \\bar{\\pi}(s_t, a_t, \\psi_t)$ ?\n- Typo in Page 7 Line 276: \"Appendix G and G\"", "limitations": "As the authors mention, this method assumes that the environment dynamics can be correctly parametrized in a confident set $Psi$. \n\nAdditionally, selecting the radius L of the proximal adversarial set of actions is not trivial. Fortunately, the authors tested the environment with a large enough L (100 times larger than the one used during training) with positive results, which validates their claim.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The evaluation is fair. The model trained under worst-case time-constrained adversarial dynamics is then evaluated under different fixed evolving dynamics (cosine, exponential, linear, and logarithmic) that range in stochastic changes much larger than the used in learning (L=0.1 in evaluation vs L=0.001 in training). \n- The definition of TC-RMDP is smartly selected to preserve stationarity, defining the state as the tuple ($S \\times \\Psi$) \n- The TC strategy can be straightforwardly implemented in previous algorithms by limiting the search space of the adversarial $psi$ at every time-step.", "weaknesses": "- Reporting that the worst-case TC variants outperform vanilla adversarial models as M2TD3 or RARL is expected. As the decisions of the adversary are rather limited when compared to the vanilla algorithms. Perhaps highlighting the evaluation results (fixed or static setting) in the conclusions would help in emphasizing this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback, their careful analysis of our contributions, and positive assessment of them.\n\n> Reporting that the worst-case TC variants outperform vanilla adversarial models as M2TD3 or RARL is expected. As the decisions of the adversary are rather limited when compared to the vanilla algorithms. Perhaps highlighting the evaluation results (fixed or static setting) in the conclusions would help in emphasizing this.\n\nActually, even though it seems counter-intuitive, there is not immediate theoretical reason for TC algorithms to outperform vanilla ones, despite the fact that their action space is smaller. The reason appears quite clearly when one considers that among the optimal adversaries, at least one is stationary. This adversary is reachable by both vanilla algorithms and TC ones. Consequently the key lesson here is that vanilla methods don't find the optimal adversary in practice, and that using the TC formulation preserves optimality while helping the optimization process (as demonstrated by the empirical results). We propose to better emphasize this in the conclusions as you suggest (but with the slight nuance that dominance of TC over vanilla should not be expected in the first place).\n\n> In Algorithm 1, should is be: $b_{t+1}= \\bar{\\pi}(s_t, a_t, \\psi_t)$\n\nBoth notations seem acceptable, since $\\psi_t$ is within the arguments of $\\bar{\\pi}$. In other words, since $\\psi_{t+1}=\\psi_t+b_t$, the two notations are equivalent. We chose to drop the $b_t$ notation for readability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yUWZLXiCqo", "reviewer": "Reviewer_1Xhb", "summary": "The paper introduces Time-Constrained Robust MDPs (TC-RMDPs) as a novel formulation to address the overly conservative nature of traditional robust RL under sa-rectangularity assumptions. The authors propose three algorithms to handle time-dependent and correlated disturbances in different situations. Extensive evaluations on continuous control benchmarks show that these algorithms outperform traditional robust RL methods in terms of balancing performance and robustness.", "questions": "please see above", "limitations": "The framework assumes that the transition parameter vector $\\psi$ is known during training. However, in real-world applications, it could be challenging. Also, the assumption on the uncertainty set $\\Psi$ is also not practical in practice.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The introduction of TC-RMDPs addresses a significant limitation of overly pessimistic in traditional robust RL, making the approach more applicable to real-world scenarios with time-dependent disturbances, which is interesting and novel. In addition, the paper presents three distinct algorithms with varying levels of information usage, expanding its usage. \n\n2. The authors provide formal theoretical guarantees for the proposed algorithms, which is appreciated.\n\n3. Extensive experiments on continuous control benchmarks demonstrate the efficacy of the proposed methods.", "weaknesses": "# Major\n1. The complexity of the proposed algorithms, especially Oracle-TC, may limit their practical applicability in real-world scenarios where complete environmental information is unavailable.\n\n2. How to select the Lipschitz constant $L$ in practical scenarios? Some ablations studies and analysis regarding $L$ would be beneficial.\n\n3. It would be better if the authors could provide some visualizations of the difference between the traditional independent transitions with the time-constrained parametric transitions.\n\n4. What is the number of worst-case episodes in the experiments? If it is 1, maybe it would be better to display the results as the average of the worst 10% of episodes for better robustness. This is because having only 1 episode might introduce stochasticity.\n\n# Minor\n1. It is suggested to show the equation number for better illustration and reference.\n2. There are some typos in the paper. For instance, in line 102, $B=\\mathcal{B}(0_{\\Psi},L)$ should be corrected to $B=\\mathcal{B}({\\Psi}_0,L)$,  the right-most column in Table 1 should be Avg. The authors should carefully review the paper for such errors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for these insightful comments.  \nWe believe the major concerns raised are actually minor, in the sense that they can all be answered by elements already in the paper, and better explanations and phrasing. We explain why below and welcome further discussion with the reviewer.\n\n> 1. The complexity of the proposed algorithms, especially Oracle-TC, may limit their practical applicability in real-world scenarios where complete environmental information is unavailable.\n\nThe rationale of proposing 3 variations on the same theme (TC, stacked-TC and Oracle-TC) is precisely designed to address your concern. As discussed in the paper, optimality can only be sought for Oracle-TC, since TC only solves a partially observable problem. Conversely, TC is designed to account for the real-world scenarios you mention. Stacked-TC is intended as an in-between solution that retains partial observability, yet recovers optimality when the underlying state can be inferred from the latest observations and actions.  \nWe hope this clarifies things. In our opinion, and unless we misunderstood your concern, we believe this identified weakness is already addressed in the paper.\n\n> 2. How to select the Lipschitz constant $L$ in practical scenarios? Some ablations studies and analysis regarding $L$ would be beneficial.\n\nIndeed, in practical scenarios, it is unrealistic to assume $L$ will be known. We claim TC approaches are actually quite insensitive to $L$'s value. To demonstrate this empirically, we trained our algorithms on the fixed $L=0.001$ setting and then evaluated them on environments with varying $L$ values, with order of magnitude $L=0.1$. This is already reported at line 265. We can stress this out more in the final version, with a particular emphasis on the fact that one needs not know $L$ beforehand and can make a conservative assumption on it, eventually making it a non-critical hyper-parameter.\n\n> 3. It would be better if the authors could provide some visualizations of the difference between the traditional independent transitions with the time-constrained parametric transitions.\n\nThis is a very interesting question. Indeed, in the rectangular case, it is known that the minimum over the simplex of $sa$-local transition probabilities at each time step, is found on the border of this simplex. What the TC hypothesis does, is actually constrain this simplex' radius. So, in environments that respect the rectangularity assumption, we could expect that the difference you are referring to is actually the difference in diameters. But there are two key limitations to this reasoning.  \nFirst, real-world scenarios are precisely not rectangular, and the worst cases need not be found on the border of the uncertainty set. \nBut (secondly) more importantly, as stated at the beginning of Section 6, there always exists a worst-case adversarial policy (ie. a worst-case adversarial transition model) that is stationary (Iyengar, 2005). Therefore, such a transition kernel respects the TC property and can be found by our algorithms. If the worst-case dynamic transition model is unique, then it is necessarily stationary, and hence both classic RVI algorithms and TC ones should converge to it. In this case, the difference between models should be zero. But this might be unverifiable in practice because a span of dynamic models can actually lead to the optimal robust value function. So the difference between the time-constrained dynamic models and the non-time-constrained ones is eventually not very informative.  \nFor this reason, although your remark is very relevant, we believe visualizing such a difference would not be very informative and might be misleading. We propose to add this short discussion to the paper.\n\n\n> 4. What is the number of worst-case episodes in the experiments? If it is 1, maybe it would be better to display the results as the average of the worst 10% of episodes for better robustness. This is because having only 1 episode might introduce stochasticity.   \n\nWe suppose you are referring to the static evaluation case (for the dynamic evaluation case, there is a single adverasary's policy found by the algorithm).\nWe checked the average score across the 10% worst transition models in that case. Although we agree with the statistical robustness argument, we would like to point out that this is eventually a different criterion overall, closer to a CVaR in spirit (or to a DR evaluation on a limited set of transition kernels). The results did not change significantly. We will add them to the appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yEN7pKBpvJ", "reviewer": "Reviewer_ZKxz", "summary": "This paper aims to develop a new framework of robust RL addressing the over-conservability issue of rectangularity and dynamic uncertainty set assumption. This framework allows for time-dependent, correlated and multifactorial disturbance to the dynamics. Three distinct algorithms are developed depending on the available information to the policy. Extensive numerical experiments are further provided to demonstrate the performance of the proposed algorithms.", "questions": "See weaknesses above.", "limitations": "na", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed framework addresses the major drawbacks of existing robust RL approach: overly conservative due to the use of dynamic and rectangular uncerainty sets.\n- Extensive experiments are provided to demonstrate the performance of the three algorithms\n- The approaches can be used with existing robust value iteration approaches.", "weaknesses": "- Thm 2.1 only applies to the case where the policy has the exact information of \\psi, which in practice is usually unavailable. \n- The usefulness of this framework remains questionable, as it poses a significant challenge of constructing such complex uncertainty set with time and state-action correlation.\n- The training algorithm has the same flavor of the adversarial training as those in the literature, e.g., RARL. The novelty seems rather limited as this is mostly an  experimental paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for this feedback.\nWe are quite surprised by the mismatch between the expressed comments and the overall grade attributed to the paper. To us, the three comments somehow discard important parts of the paper and we warmly welcome further discussion to address them if necessary.\n\n> Thm 2.1 only applies to the case where the policy has the exact information of \\psi, which in practice is usually unavailable.\n\nIndeed. We don't think this is really a major limitation: there is no guarantee that an observation-based optimal policy exists for POMDPs and yet people train regular RL algorithms on real-life environments with partial observability all the time. It is quite the same thing here. What theorem 2.1 states is a general property of the Bellman operator, which induces the existence of an optimal value function when (as you accurately point out) $\\psi$ is observable. This does not mean that our algorithms are only applicable in this setting (see answer below).\n\n> The usefulness of this framework remains questionable, as it poses a significant challenge of constructing such complex uncertainty set with time and state-action correlation.\n\nWe beg to differ. When one drives on the highway, the transition models in different geographical conditions are strongly correlated together (by the global weather for instance) and often follow a time-constrained evolution. Similarly, a bicycle's dynamics in different states and actions are coupled by general parameters (friction coefficients for instance) that couple them together. Such uncertainty sets are very easy to construct when designing simulators (one only needs to let the global parameters vary).  \nWe believe there may be a misunderstanding here and warmly welcome further discussion with the reviewer to better address this concern, and understand why it seems so undue to us.\n\n> The training algorithm has the same flavor of the adversarial training as those in the literature, e.g., RARL. The novelty seems rather limited as this is mostly an experimental paper.\n\nIndeed, most algorithms in the literature (most algorithms that scale to large domains at least) are RVI-inspired (like RARL or M2TD3).  We believe this is actually a strength of the current proposal: it can be used in any previous RVI algorithm to make it better. Rejecting the paper on this basis is not a good signal for RL research: if only new algorithms were to be published, a large number of generic, important works would never receive attention.  \nAdditionally, we prove several properties of the TC framework, including that the optimal value function is not excluded by the TC assumption, hence we feel calling this work mostly experimental somehow discards parts of the contribution we deem important. Here again, we welcome further discussion to clarify things.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v3RN3dT2Sx", "reviewer": "Reviewer_aiQz", "summary": "This paper proposed a novel concept, time-constrained robust MDP, to address the conservativeness issue of the rectangular assumption. They assume the transition depends on an underlying parameter, and the parameter can be adversarially chosen from an uncertainty set. Several algorithms are proposed to solve the time-constrained robust MDP. Extensive numerical experiments show the effectiveness of the proposed framework and algorithms.", "questions": "1. In the experiments, the oracle-TC is not always the best method. It seems that the performances are not that stable. Also, the vanilla TC outperforms in almost half cases. It is kind of anti-intuitive.\n\n2. Compared to baseline methods, the TC-based methods show robustness to some extent, but how robust are they? What are the expected best (robust) performances?\n\n3. In algorithm 1, what is \"sample a mini-batch of transitions\" for? How are the agent and adversary updated?\n\n4. On lines 174-175, the authors claim that the rectangularity assumptions are rarely met in real-world scenarios. I think more evidence or references are required to support this claim.", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Given that the field of robust MDP has limited large-scale experiments due to the intractability of the rectangular assumption, the idea in this paper is very nice. I agree that the rectangular assumption can be too general and lead to conservative policies, due to its inefficiency in leveraging the inherent structure of the uncertainty set. The proposed time constraint robust MDP is interesting, especially since the experiment results seem to be promising.", "weaknesses": "My concern is that the methods proposed in this paper might be too heuristic, given the non-stationarity and possibly non-existence of the optimal policy. As I am not very familiar with the experiment side, I am not sure if the evaluation in this paper is sufficient. See my questions for details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and the questions you raised regarding our paper. We appreciate your feedback and would like to address your concerns as follows:\n\n> My concern is that the methods proposed in this paper might be too heuristic, given the non-stationarity and possibly non-existence of the optimal policy. As I am not very familiar with the experiment side, I am not sure if the evaluation in this paper is sufficient. See my questions for details.\n\nActually, because there is always at least one stationary optimal adversary (Iyengar, 2005), the TC hypothesis preserves optimality. Hence, under classical assumptions, the proposed method is not heuristic: the optimal policy is preserved, as stated at the beginning of Section 6.\n\n> Questions:\n> In the experiments, the oracle-TC is not always the best method. It seems that the performances are not that stable. Also, the vanilla TC outperforms in almost half cases. It is kind of anti-intuitive.\n\nWe took great care to avoid over-interpretation of empirical results. Quite often in robust RL, overall scores suffer from high variance. Hence, identifying clear domination between two algorithms can be tricky.  \nOne could conjecture that vanilla TC policies benefits from having a reduced input space (observation only, while the others use a sequence of observations or the information of $\\psi$) which might prevent overfitting, while the observation itself holds enough information for a robust policy. This seems hard to verify though.\n\n> Compared to baseline methods, the TC-based methods show robustness to some extent, but how robust are they? What are the expected best (robust) performances?\n\nThe optimal best robust performance is unknown in most realistic, large robust RL benchmarks. Additionally, non convexity of optimization landscapes prevents from providing optimality certificates for solutions. So the best one can do is rank algorithms based on their average score, and recall the variance.\n\n> In algorithm 1, what is \"sample a mini-batch of transitions\" for? How are the agent and adversary updated?\n\nThis mini-batch is used in the subsequent UpdatePolicy lines (those updates are classical SGD steps, using this minibatch). Specifically, the update in our experiments is that of TD3, but the SAC update could be dropped-in without any modification to the overall training loop.\n\n> On lines 174-175, the authors claim that the rectangularity assumptions are rarely met in real-world scenarios. I think more evidence or references are required to support this claim.\n\nIt is quite hard to prove a negative, but the essence of the rectanglarity assumption is very un-natural in the first place: it implies a bicycle's parts can change mass from one time step to the other, or that the weather is drastically different a few meters apart along the same road. So although the rectangularity assumption makes theoretical analysis possible, it is commonly admitted in robust MDPs that it makes little sense in practice.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Most existing works on robust MDP make the (s,a) rectangularity assumption, which inherently assumes that the transition dynamics from a given current state to the next states are independent even for current states closer to each other. This assumption is strong, and may not be applicable in real-world problems. This paper proposes a new time-constrained robust MDP (TC-RMDP) formulation that considers correlated and time-dependent disturbances. The paper proposes three algorithms to solve this problem, each using varying levels of environmental information. The paper provides an extensive simulation-based evaluation of the proposed algorithms using standard benchmarks. 


We received four expert reviews, with the scores, 7, 6, 5, 5, and the average score is 5.75. Reviewers gave positive comments about the novelty of presenting the TC-RMDP, the novelty of three algorithms, and the extensive simulation-based evaluations of the algorithms. The reviewers are also happy about the quality of the presentation. 

The reviewers have also provided some very useful comments to improve the paper. Reviewer 1Xhb has asked about the complexity of proposed algorithms, selecting the parameter L, and providing a visualization/figure of the traditional (s,a) rectangular uncertainty set and uncertainty considered in this paper. Reviewer ZKxz has asked to include some specific examples of the practical applications where such uncertainty sets are relevant. I think all these are important points, and I recommend authors to include these details in their final manuscript. .
Decision: Accept (poster)