Time-Constrained Robust MDPs

Authors: adil  zouitine
david  bertoin
pierre  clavier
matthieu  geist
emmanuel  rachelson
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 14:40:21

Abstract: Robust reinforcement learning is essential for deploying reinforcement learning algorithms in real-world scenarios where environmental uncertainty predominates.
Traditional robust reinforcement learning often depends on rectangularity assumptions, where adverse probability measures of outcome states are assumed to be independent across different states and actions. 
This assumption, rarely fulfilled in practice, leads to overly conservative policies. 
To address this problem, we introduce a new time-constrained robust MDP (TC-RMDP) formulation that considers multifactorial, correlated, and time-dependent disturbances, thus more accurately reflecting real-world dynamics. This formulation goes beyond the conventional rectangularity paradigm, offering new perspectives and expanding the analytical framework for robust RL.
We propose three distinct algorithms, each using varying levels of environmental information, and evaluate them extensively on continuous control benchmarks. 
Our results demonstrate that these algorithms yield an efficient tradeoff between performance and robustness, outperforming traditional deep robust RL methods in time-constrained environments while preserving robustness in classical benchmarks.
This study revisits the prevailing assumptions in robust RL and opens new avenues for developing more practical and realistic RL applications.

Main paper: 
 introduction: 
 Robust MDPs capture the problem of finding a control policy for a dynamical system whose transition kernel is only known to belong to a defined uncertainty set. The most common framework for analyzing and deriving algorithms for robust MDPs is that of sa-rectangularity [1, 2] , where probability measures on outcome states are picked independently in different source states and actions (in formal notation, P(s ′ |s, a) and P(s ′ |s, ā) are independent of each other). This provides an appreciable decoupling of worst transition kernel search across time steps and enables sound algorithms like robust value iteration (RVI). But policies obtained for such sa-rectangular MDPs are by nature very conservative [3, 4] , as they enable drastic changes in environment properties from one time step to the next, and the algorithms derived from RVI tend to yield very conservative policies even when applied to non-sa-rectangular robust MDP problems.In this paper, we depart from the rectangularity assumption and turn towards a family of robust MDPs whose transition kernels are parameterized by a vector ψ. This parameter vector couples together the outcome probabilities in different (s, a) pairs, hence breaking the independence assumption that is problematic, especially in large dimension [3] . This enables accounting for the notion of transition model consistency across states and actions: outcome probabilities are not picked independently anymore but are rather set across the state and action spaces by drawing a parameter vector. In turn, we examine algorithms for solving such parameter-based robust MDPs when the parameter is constrained to follow a bounded evolution throughout time steps. Our contributions are the following.1. We introduce a formal definition for parametric robust MDPs and time-constrained robust MDPs, discuss their properties and derive a generic algorithmic framework (Sec. 2). 2. We propose three algorithmic variants for solving time-constrained MDPs, named vanilla TC , Stacked-TC and Oracle-TC (Sec. 4), which use different levels of information in the state space, and come with theoretical guaranties (Sec. 6). 3. These algorithms are extensively evaluated in MuJoCo [5] benchmarks, demonstrating they lead to non-conservative and robust policies (Sec. 5).2 Problem statement (Robust) MDPs. A Markov Decision Process (MDP) [6] is a model of a discrete-time, sequential decision making task. At each time step, from a state s t ∈ S of the MDP, an action a t ∈ A is taken and the state changes to s t+1 according to a stationary Markov transition kernel p(s t+1 |s t , a t ), while concurrently receiving a reward r(s t , a t ). S and A are measurable sets and we write ∆ S and ∆ A the set of corresponding probability distributions. A stationary policy π(•|s) is a mapping from states to distributions over actions, prescribing which action should be taken in s. The value function v π p of policy π maps state s to the expected discounted sum of rewards E p,π [ t γ t r t ] when applying π from s for an infinite number of steps. An optimal policy for an MDP is one whose value function is maximal in any state. In a Robust MDP (RMDP) [1, 2] , the transition kernel p is not set exactly and can be picked in an adversarial manner at each time step, from an uncertainty set P. Then, the pessimistic value function of a policy is v π P (s) = min p∈P v π p (s). An optimal robust policy is one that has the largest possible pessimistic value function v * P in any state, hence yielding an adversarial max π min p optimization problem. Robust Value Iteration (RVI) [1, 7] solves this problem by iteratively computing the one-step lookahead best pessimistic value: The T * P operator is called the robust Bellman operator and the sequence of v n functions converges to the robust value function v * P as long as the adversarial transition kernel belongs to the simplex of ∆ S . Zero-sum Markov Games. Robust MDPs can be cast as zero-sum two-players Markov games [8, 9] where B is the action set of the adversarial player. Writing π : S × A → ∆ B the policy of this adversary, the robust MDP problem turns to max π min π v π,π , where v π,π (s) is the expected sum of discounted rewards obtained when playing π (agent actions) against π (transition models) at each time step from s. This enables introducing the robust value iteration sequence of functions v n+1 (s) := T * * v n (s) := maxπ(s)∈∆ A min π(s,a)∈∆ S (T π,π v n )(s)where T π,π := E a∼π(s) [r(s, a) + γE s ′ ∼π(s,a) v n (s ′ )] is a zero-sum Markov game operator. These operators are also γ-contractions and converge to their respective fixed point v π,π and v * * = v * P [9] . This formulation will be useful to derive a practical algorithm in Section 4.Often, this convergence is analyzed under the assumption of sa-rectangularity, stating that the uncertainty set P is a set product of independent subsets of ∆ S in each s, a pair. Quoting [1] , rectangularity is a sort of independence assumption and is a minimal requirement for most theoretical results to hold. Within robust value iteration, rectangularity enables picking π(s t , a t ) completely independently of π(s t-1 , a t-1 ). To set ideas, let us consider the robust MDP of a pendulum, described by its mass and rod length. Varying this mass and rod length spans the uncertainty set of transition models. The rectangularity assumption induces that π(s t , a t ) can pick a measure in ∆ S corresponding to a mass and a length that are completely independent from the ones picked in the previous time step. While this might be a good representation in some cases, in general it yields policies that are very conservative as they optimize for adversarial configurations which might not occur in practice.We first step away from the rectangularity assumption and define a parametric robust MDP as an RMDP whose transition kernels are spanned by varying a parameter vector ψ (typically the mass and rod length in the previous example). Choosing such a vector couples together the probability measures on successor states from two distinct (s, a) and (s ′ , a ′ ) pairs. The main current robust deep RL algorithms actually optimize policies for such parametric robust MDPs but still allow the parameter value at each time step to be picked independently of the previous time step. Parametric MDPs. A parametric RMDP is given by the tuple (S, A, Ψ, p ψ , r) where the transition kernel p ψ (s, a) ∈ ∆ S is parameterized by ψ, and Ψ is the set of values ψ can take, equipped with an appropriate metric. This yields the robust value iteration update : A parametric RMDP remains a Markov game and the Bellman operator remains a contraction mapping as long as p ψ can reach only elements in the simplex of ∆ S , where the adversary's action set is the set of parameters instead of a (possibly sa-rectangular) set of transition kernels.v n+1 (s) = maxTime-constrained RMDPs (TC-RMDPs). We introduce TC-RMDPs as the family of parametric RMDPs whose parameter's evolution is constrained to be Lipschitz with respect to time. More formally a TC-RMDP is given by the tuple (S, A, Ψ, p ψ , r, L), where ∥ψ t+1 -ψ t ∥ ≤ L, that is the parameter change is bounded through time. In the previous pendulum example, this might represent the wear of the rod which might lose mass or stretch length. Similarly, and for a larger scale illustration, TC-RMDPs enable representing the possible evolutions of traffic conditions in a path planning problem through a busy town. Starting from an initial parameter value ψ -1 , the pessimistic value function of a policy π is non-stationary, as ψ 0 is constrained to lay at most L-far away from ψ -1 , ψ 1 from ψ 0 , and so on. Generally, this yields non-stationary value functions as the uncertainty set at each time step depends on the previous uncertainty parameter. To regain stationarity without changing the TC-RMDP definition, we first change the definition of the adversary's action set. The adversary picks its actions in the constant set B = B(0 Ψ , L), which is the ball of radius L centered in the null element in Ψ. In turn, the state of the Markov game becomes the pair s, ψ and the Markov game itself is given by the tuple ((S × Ψ), A, B, p ψ , r), where the Lipschitz constant L is included in B. Thus, given an action b t ∈ B and a previous parameter value ψ t-1 , the parameter value at time t is ψ t = ψ t-1 + b t . Then, we define the pessimistic value function of a policy as a function of both the state s and parameter ψ:v π B (s, ψ) := min (bt) t∈N , bt∈B E γ t r t |ψ -1 = ψ, s 0 = s, b t ∈ B, ψ t = ψ t-1 + b t , a ∼ π, s t ∼ p ψt , v * B (s, ψ) = max π(s,ψ)∈∆ A v π B (s, ψ).In turn, an optimal robust policy is a function of s and ψ and the TC robust Bellman operators are:v n+1 (s, ψ) := T * B v n (s, ψ) := max π(s,ψ)∈∆ A T π B v n (s, ψ), := max π(s,ψ)∈∆ A min b∈B E a∼π(s) [r(s, a) + γE s ′ ∼p ψ+b (s,a) v n (s ′ , ψ + b)].This iteration scheme converges to a fixed point according to Th. Proof of Th. 2.1 can be found in Appendix B. We refer to this formulation as algorithm Oracle-TC (see Section 4 for implementation details) since an oracle makes the current parameter ψ visible to the agent. Therefore, it is possible to derive optimal policies for TC-RMDPs by iterated application of this TC Bellman operator. These policies have the form π(s, ψ). In the remainder of this paper, we extend state-of-the-art robust deep RL algorithms to the TC-RMDP framework. In particular, we compare their performance and robustness properties with respect to classical robust MDP formulations, we also discuss their relation with the π(s) robust policies of classical robust MDPs.If the agent is unable to observe the state variable ψ, it is not possible to guarantee the existence of a stationary optimal policy of the form π(s). Similarly, there is no guarantee of convergence of value functions to a fixed point. Nonetheless, this scenario, in which access to the ψ parameter is not available, is more realistic in practice. It turns the two-player Markov game into a partially observableMarkov game, where one can still apply the TC Bellman operator but without these guarantees of convergence. We call vanilla TC the repeated application of the TC Bellman operator in this partially observable case. Vanilla TC will be tested in practice, and some theoretical properties of the objective function will be derived using the Lipschitz properties (Sec 6). 
related works: 
 Since our method is a non-rectangular, Deep Robust RL algorithm, (possibly non-stationary for Stacked-TC and TC ), we discuss the following related work.Non-stationary MDPs. First, non-stationarity has been studied in the Bandits setting in [10] .Then, for episodic, non-stationary MDPs [11, 12, 13] have explored and provided regret bounds for algorithms that use oracle access to the current reward and transition functions. More recently [14, 15] have facilitated oracle access by performing a count-based estimation of the reward and transition functions based on the recent history of interactions. Finally, for tabular MDPs, past data from a non-stationary MDP can be used to construct a full Bayesian model [16] or a maximum likelihood model [17] of the transition dynamics. We focus on the setting not restricted to tabular representations Non-rectangular RMDPs. While rectangularity in practice is very conservative, it can be demonstrated that, in an asymptotic sense, non-rectangular ellipsoidal uncertainty sets around the maximum likelihood estimator of the transition kernel constitute the smallest possible confidence sets for the ground truth transition kernel, as implied by classical Cramér-Rao bounds. This is in accordance with the findings presented in § 5 and Appendix A of [7] . More recently, [3] extends the rectangular assumptions using a factored uncertainty model, where all transition probabilities depend on a small number of underlying factors denoted w 1 , . . . , w r ∈ R S , such that each transition probability P sa for every (s, a) is a linear (convex) combination of these r factors. Finally, [4] use policy gradient algorithms for non-rectangular robust MDPs. While this work presents nice theoretical guarantees of convergence, there is no practical Deep RL algorithms for learning optimal robust policies.Deep Robust RL Methods. Many Deep Robust algorithms exist such as M2TD3 [18] , M3DDPG [19] , or RARL [20] , which are all based on the two player zero-sum game presented in 2. We will compare our method against these algorithms, except [19] which is outperformed by [18] in general. We also compare our algorithm to Domain randomization (DR) [21] that learns a value function V (s) = max π E p∼U (P) V π p (s) which maximizes the expected return on average across a fixed (generally uniform) distribution on P. As such, DR approaches do not optimize the worst-case performance but still have good performance on average. Nonetheless, DR has been used convincingly in applications [22, 23] . Finally, the zero-sum game formulation has lead to the introduction of action robustness [9] which is a specific case of rectangular MDPs, in scenarios where the adversary shares the same action space as the agent and interferes with the agent's actions. Several strategies based on this idea have been proposed. One approach, the Game-theoretic Response Approach for Adversarial Defense (GRAD) [24] builds on the Probabilistic Action Robust MDP (PR-MDP) [9] . This method introduces time-constrained perturbations in both the action and state spaces and employs a game-theoretic approach with a population of adversaries. In contrast to GRAD, where temporal disturbances affect the transition kernel around a nominal kernel, our method is part of a broader setting in which the transition kernel is included in a larger uncertainty set. Robustness via Adversary Populations (RAP) [25] introduces a population of adversaries. This approach ensures that the agent develops robustness against a wide range of potential perturbations, rather than just a single one, which helps prevent convergence to suboptimal stationary points. Similarly, State Adversarial MDPs [26, 27, 28, 24] address adversarial attacks on state observations, effectively creating a partially observable MDP. Finally, using rectangularity assumptions, [29, 30] use Wasserstein and χ 2 balls respectively for the uncertainty set in Robust RL.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] Justification: Hypotheis of proof are well stated in Th.6.2 and 2.1Guidelines:• The answer NA means that the paper does not include theoretical results. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.• Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.• Theorems and Lemmas that the proof relies upon should be properly referenced. Justification: We provide the algorithm, all detail implementations, hyperparameters. 
time-constrained robust mdp algorithms: 
 The TC-RMDP framework addresses the limitations of traditional robust reinforcement learning by considering multifactorial, correlated, and time-dependent disturbances. Traditional robust reinforcement learning often relies on rectangularity assumptions, which are rarely met in real-Figure 1 : TC-RMDP training involves a temporally-constrained adversary aiming to maximize the effect of temporally-coupled perturbations. Conversely, the agent aims to optimize its performance against this time-constrained adversary. In orange, the oracle observation, and in blue the stacked observation.world scenarios, leading to overly conservative policies. The TC-RMDP framework provides a more accurate reflection of real-world dynamics, moving beyond the conventional rectangularity paradigm.We cast the TC-RMDP problem as a two-player zero-sum game, where the agent interacts with the environment, and the adversary (nature) changes the MDP parameters ψ. Our approach is generic and can be derived within any robust value iteration scheme, performing max π(s)∈∆ A min ψ∈Ψ E a∼π(s) [r(s, a) + γE s ′ ∼p ψ (s,a) v n (s ′ )] updates, by modifying the adversary's action space and potentially the agent's state space to obtain updates of the form max π(s,ψ)∈∆ A min b∈B E a∼π(s) [r(s, a) + γE s ′ ∼p ψ+b (s,a) v n (s ′ )]. In Section 5, we will introduce time constraints within two specific robust value iteration algorithms, namely RARL [20] and M2TD3 [18] by simply limiting the search space for worst-case ψ at each step. This specific implementation extends the original actor-critic algorithms. For the sake of conciseness, we refer the reader to Appendix E.1 for details regarding the loss functions and algorithmic details. Three variations of the algorithm are provided (illustrated in Figure 1 ) but all fall within the training loop of Algorithm 1. i , a i , r(s i , a i ), ψ i , ψ i+1 , s i+1 } i∈[1,N ] ∼ B // Sample a mini-batch of transitions π t+1 ← U pdateP olicy(π t ) // Update Agent πt+1 ← U pdateP olicy( πt )// Update Adversary Oracle-TC . As discussed in Section 2, the Oracle-TC version includes the MDP state and parameter value as input, π : S × Ψ → A. This method assumes that the agent has access to the true parameters of the environment, allowing it to make the most informed decisions and possibly reach the true robust value function. However, these parameters ψ are sometimes non-observable in practical scenarios, making this method not always feasible.Stacked-TC . Since ψ might not be observable but may be approximately identified by the last transitions, the Stacked-TC policy uses the previous state and action as additional inputs in an attempt to replace ψ, π : S × A × S → A. This approach leverages the information in the transitions, even though it might be insufficient for a perfect estimate of ψ. It aims to retain (approximately) the convergence properties of the Oracle-TC algorithm.Vanilla TC . Finally, the vanilla TC version takes only the state, π : S → A, as input, similar to standard robust MDP policies. This method does not attempt to infer the environmental parameters or the transition dynamics explicitly. Instead, it relies on the current state information to guide the agent's actions. While this version is the most straightforward and computationally efficient, it may not perform as robustly as the Oracle-TC or Stacked-TC versions in environments with significant temporal disturbances, since it attempts to solve a partially observable Markov game, for which there may not exist a stationary optimal policy based only on the observation. Despite this, it remains a viable option in scenarios where computational simplicity and quick decision-making are prioritized.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
results: 
 Experimental settings. This section evaluates the robust time-constrained algorithm's performance under severe time constraints and in the static settings. Experimental validation was conducted in continuous control scenarios using the MuJoCo simulation environments [5] . The approach was categorized into three variants. The Oracle-TC , where the agent accessed environmental parameters π(s t , ψ); the Stacked-TC , where the agent took in input π(s t , s t-1 , a t-1 ); and the vanilla TC , which did not receive any additional inputs π(s). For each variant of the time-constrained algorithms, we applied them to RARL [20] , and M2TD3 [18] , renaming them TC-RARL and TC-M2TD3, respectively. The algorithms were tested against two state-of-the-art robust reinforcement learning algorithms, M2TD3 and RARL. Additionally, the Oracle versions of M2TD3 and RARL, where the agent's policy included ψ in the input π : S × Ψ → A, were evaluated for a more comprehensive assessment. Comparisons were also made with Domain Randomization (DR) [21] and vanilla TD3. [31] to ensure a thorough analysis. A 3D uncertainty set is defined in each environment P normalized between [0, 1] 3 . Appendix G provides detailed descriptions of uncertainty parameters. Performance metrics were gathered after five million steps to ensure a fair comparison. All baselines were constructed using TD3, and a consistent architecture was maintained across all TD3 variants. The results presented below were obtained by averaging over ten distinct random seeds. Appendices E.4, E. Performance of TCRMDPs in worst-case time-constrained. Table 1 reports the worst-case time-constrained perturbation. To address the worst-case time-constrained perturbations for each trained agent π * , we utilized a time-constrained adversary using TD3 algorithm π * = min b∈B E a∼π * (s),b∼π(s,a,ψ) [r(s, a) + γEs ′ ∼ p ψ+b (s, a)v n (s ′ )] within a perturbation radius of L = 0.001 for a total of 5 million steps. The sum of episode rewards was averaged over 10 episodes. A heuristic to choose the perturbation radius is explained in Appendix K. To compare metrics across different environments, each method's score v was standardized relative to the reference score of TD3. TD3 was trained on the environment using default transition function parameters, with its score denoted as v T D3 . The M2TD3 score, v M 2T D3 , was used as the comparison target. The formula applied was(v -v T D3 )/(|v M 2T D3 -v T D3 |).This positioned v T D3 as the minimal baseline and v M 2T D3 as the target score. This standardisation provides a metric that quantifies the improvement of each method over TD3 in relation to the improvement of M2TD3 over TD3. In each evaluation environment, agents trained with the time-constrained framework (indicated by TC in the method name) demonstrated significantly superior performance compared to those trained using alternative robust reinforcement learning approaches, including M2TD3 and RARL. Furthermore, they outperformed those trained through domain randomisation (DR). Notably, even without directly conditioning the policy with ψ, the time-constrained trained policies excelled against all baselines, achieving up to a 2.9-fold improvement. The non-normalized scores are reported in Appendix H. Additionally, when policies were directly conditioned by ψ and trained within the robust reinforcement learning framework, they tended to be overly conservative in the time-constrained framework. This is depicted in Table 1 , comparing the performances of Oracle RARL, Oracle M2TD3, Oracle TC-RARL, and Oracle TC-M2TD3. Both policies also observe ψ. The only difference is that Oracle RARL and Oracle M2TD3 were trained in the robust reinforcement learning framework, while Oracle TC-RARL and Oracle TC-M2TD3 were trained in the time-constrained framework. The performance differences under worst-case time-coupled perturbation are as follows: for Oracle RARL (resp. M2TD3) and Oracle TC-RARL (resp. M2TD3), the values are -0.86 (1.39) vs. 2.54 (3.07). This observation highlights the need for a balance between robust training and flexibility in dynamic conditions. A natural question arises regarding the worst-case time-constrained perturbation. Was the adversary in the loop adequately trained, or might its suboptimal performance lead to overestimating the trained agent's reward against the worst-case perturbation? The adversary's performance was monitored during its training against all fixed-trained agents. The results in Appendix F show that our adversary converged.Robust Time-Constrained Training under various time fixed adversaries. The method was evaluated against various fixed adversaries, focusing on the random fixed adversary shown in Figure 2 . This evaluation shows that robustly trained agents can handle dynamic and unpredictable conditions. The random fixed adversary simulates stochastic changes by selecting a parameter ψ t at each timestep within a radius of L = 0.1. This radius is 100 times larger than in our training methods. At the start of each episode, ψ 0 is uniformly sampled from the uncertainty set ψ 0 ∼ U(P). This tests the agents' adaptability to unexpected changes. Performance of Robust Time-Constrained MDPs in the static setting. In static environments, the Robust Time-Constrained algorithms were evaluated for worst-case and average performance metrics, shown in Tables 2 and 3 . A fixed uncertainty set P was used, dividing each dimension of Ψ into ten segments, creating a grid of 1000 points (10 3 ). Each agent ran five episodes at each grid point, and the rewards were averaged. The scores were normalized as described for the time-constrained adversary analysis in Table 1 . The raw data is provided in Appendix 9 and 10. Performance scores were adjusted relative to the baseline v T D3 and v M 2T D3 . As a result, normalized results reveal distinct trends among agent configurations within the TC-RMDP framework. The Oracle TC-M2TD3 variant achieved an average score of 3.12 3, while the Stacked TC-M2TD3 scored 2.23, indicating its resilience. Furthermore, in the worst-case scenario, the TC-RARL and Stacked TC-RARL variants demonstrated adaptability, with TC-RARL scoring 0.92 and TC-M2TD3 scoring 1.02 2. This performance highlights its reliability in challenging static environments. 6 Some Theoretical properties of TC-MDPS.6.1 On the optimal policy of TC Following Lemma 3.3 of [1] , it is known that in the rectangular case, there exists an optimal policy of the adversary that is stationary, provided that the actor policy is stationary. The TC-RMDP definition enforces a limitation on the temporal variation of the transition kernel. Consequently, all stationary adversarial policies are constrained by this stipulation. In turn, this guarantees that (under the hypothesis of sa-rectangularity) there always exists a solution to the TC-RMDP that is also a solution to the original RMDP. In other words: optimizing policies for TC-RMDPs do not exclude optimal solutions to the underlying RMDP. This sheds an interesting light on the search for robust optimal policies, since TC-RMDPs shrink the search space of optimal adversarial policies. In practice, this is confirmed by the previous experimental results (Figure 2 ) where the optimal agent policy found by either Oracle-TC , Stacked-TC , or vanilla TC actually outperforms the one found by M2TD3 or RARL in the non time-constrained setting.6.2 Some Lipchitz-properties for non-stationary TC-RMPDS.In this subsection we slightly depart from the framework defined in Section 2 and study the smoothness of the robust objective for vanilla TC or Stacked-TC . Th. 2.1 is no longer applicable as ψ is not observed. However, we can still give smoothness of the objective starting from Lipchichz conditions on the evolution of the parameter that leads to smoothness on reward and transition kernel in the following definition 6.1. Definition 6.1 (Reward/Kernel Lipchitz TC-RMDPs [13] ). We say that a parametric RDMPs is time constrained if the parameter change is bounded through time ie. ∥ψ t+1 -ψ t ∥ ≤ L. Moreover, we assume that this variation in parameter implies a variation in the reward and transition kernel of∀s ∈ S, ∀a ∈ A, ∥P t (• | s, a) -P t+1 (• | s, a)∥ 1 ≤ L P ; |r t (s, a) -E [r t+1 (s, a)]| ≤ L rFrom a theoretical point of view, a TC-RMDP can be seen as a sequence of stationary MDPs with time indexed reward and transition kernel r t , P t that have continuity. More formally for M t = (S, A, Ψ, p ψt , r t , L = (L p , L r )), we can then define the sequence of stationary MDPs with Lipchitz variation :EQUATIONDefining r k t as the random variable corresponding to the reward function at time step t for stationary MDPs, but iterating with index k, the stationary rollout return at time t is G(π, M t ) = k≥0 γ k r k t . Assuming that at a fixed t the reward and transition kernel r t , P t are fixed, the robust objective function is:J R (π, t) := min m={m ′ t } t t ′ =t 0 ∈M L t E [G (π, m)]This leads to the following guarantee for vanilla TC and Stacked-TC algorithms. Theorem 6.2. Assume TC-RMPDS with L = (L r , L P ) smoothness. Then ∀t ∈ N,r t ∈ [0, 1], ∀t ∈ N + , ∀t 0 ∈ N + , |J R (π, t 0 ) -J R (π, t 0 + t)| ≤ L ′ t.withL ′ := γ (1-γ) 2 L P + 1 1-γ L rThis theorem states that a small variation of the Kernel and reward function will not affect too much the robust objective. In other terms, despite the fact that the TC Bellman operator may not admit a fixed point and yield a non-stationary sequence of value functions, variations of the expected return remain bounded. Proof of the Th. 6.2 can be found in Appendix C.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We provide the code with clear instruction, to reproduce the experiments.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We provides all hyperparameters, optimizer and reinforcement learning algorithm. All those details is provided in the appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 This paper presents a novel framework for robust reinforcement learning, which addresses the limitations of traditional methods that rely on rectangularity assumptions. These assumptions often result in overly conservative policies, which are not suitable for real-world applications where environmental disturbances are multifactorial, correlated, and time-constrained. In order to overcome these challenges, we proposed a new formulation, the Time-Constrained Robust Markov Decision Process (TC-RMDP). The TC-RMDP framework is capable of accurately capturing the dynamics of real-world environments, due to its consideration of the temporal continuity and correlation of disturbances. This approach resulted in the development of three algorithms: The three algorithms, Oracle-TC , Stacked-TC , vanilla TC which differ in the extent to which environmental information is incorporated into the decision-making process. A comprehensive evaluation of continuous control benchmarks using MuJoCo environments has demonstrated that the proposed TC-RMDP algorithms outperform traditional robust RL methods and domain randomization techniques. These algorithms achieved a superior balance between performance and robustness in both time-constrained and static settings. The results confirmed the effectiveness of the TC-RMDP framework in reducing the conservatism of policies while maintaining robustness. Moreover, we provided theoretical guaranties for Oracle-TC in Th. 2.1 and for Stacked-TC and vanilla TC in Th. 6.2. This study contributes to the field of robust reinforcement learning by introducing a time-constrained framework that more accurately reflects the dynamics observed in real-world settings. The proposed algorithms and theoretical contributions offer new avenues for the development of more effective and practical RL applications in environments with complex, time-constrained uncertainties.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We profide our computers resources to runs all experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We have a section in the appendix discussing of the computer resources for our work.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss on the broader impact in the conclusion and the appendix Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification:Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The authors cite the original paper that produced the code package or dataset.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: The Appendix is structured as follow :• In Appendix B, proof for fix point of Oracle-TC algorithm for can be found.• In Appendix C, proof for algorithm Vanilla TC and Stacked-TC can found about robust objective.• In Appendix F, the adversary training was sanity-checked within the time-constrained evaluation.• In Appendix E, all implementation details are provided.• In Appendix H, all raw results are presented.• In Appendix I, the computer resources and training wall clock time are detailed.• In Appendix J, the broader impact and limitations are discussed.• In Appendix K, a heuristic for choosing the radius L B Proof of Theorem 2.1Proof. The Proof is similar to [1] , using the fact that p ψ+b belongs to the simplex, we get contraction of the operator and convergence to a fix point v * B . Not that to converge to the fix point, there is no need of rectangularity.EQUATIONFirst we prove that the TC Robust OperatorT π B is a contraction. Let V 1 , V 2 ∈ R n . Fix s ∈ S, and assume that T π B V 1 (s, ψ) ≥ T π B V 2 (s, ψ).Then fix ϵ > 0 and pick π s.t given s ∈ S,EQUATIONFirst we pick a probability measure p ′ such that p ′ = p ψ+b , b ∈ B, such thatEQUATIONThen it lead toEQUATIONEQUATION)where last inequality is Holder's inequality between L 1 and L ∞ norms, use probability measure in the simplex such as ∥p ′ ∥ 1 = 1. Doing the same thing but in the case whereEQUATIONi.e.∥T π B V 1 -T π B V 2 ∥ ∞ ≤ γ∥V 1 -V 2 ∥ ∞ + 2ϵ.As we can choose ϵ arbitrary small, this establishes that the TC Bellman operator is a γ-contraction. Since T π B is a contraction operator on a Banach space, the Banach fixed point theorem implies that the operator equationT π B V = V has a unique solution V = v π B .A similar proof can be done for optimal operator T * B . The only difference is the maximum operator which is 1-Lipschitz. So T * B is also a contraction. Then, once proved that operators are γcontraction, following [1] (Th. 5), we have that the fixed point of this recursion is exactly :EQUATIONEQUATION)for (optimal) TC Bellman Operator.Recall that we represent a non-stationary robust MDPs (NS-RMDP) as a stochastic sequence,{M = {M i } ∞ t=t0, of stationary MDPs M t ∈ M, where M is the set of all stationary MDPs. Each M t is a tuple, S, A, P t , r t , γ, ρ 0 , where S The set of possible states is denoted by S, the set of actions by A, the discounting factor by γ, the start-state distribution by ρ 0 , and the reward distribution by r t . The reward distribution, denoted by r t : S × A → ∆(R), is the probability distribution of rewards. The transition function, represented by P t : S × A → ∆(S), is the probability distribution of transitions between states. The symbol ∆ denotes the simplex. For all M t ∈ M, we assume that the state space, action space, discount factor, and initial distribution remain fixed. A policy is represented as a function π : S → ∆(A). In general, we will use subscripts t to denote the time evolution during an episode and superscripts k to denote the time step assuming reward or kernel t which is stationary, assuming that the reward function is not changing as it is at time step t stationary. That r k t is the random variables corresponding to the state, action, and reward at time step t for stationary, but iterating with index k. Definition C.1 ( Lipschitz of sequence of MDPs). We denote the sequence of kernel and reward functionP = {P t } ∞ t=t0 and R = {r t } ∞ t=t0 . We define a sequence of MDP is L = (L r , L P )-Lipchitz if m = {m t } ∞ t=t0 ∈ M L with M L t = {M t } t t ′ =t0 ; ∃(L r , L P ) ∈ R 2 + ∀t ∈ N, ∀s ∈ S, ∀a ∈ A, ∥P t ′ (• | s, a) -P t ′ +1 (• | s, a)∥ 1 ≤ L P ; |r ′ t (s, a) -r t ′ +1 (s, a)| ≤ L rAssuming that for a time steps the reward function is stationary, we can compute the average return as: Definition C.2. Non-robust objective function, assuming that G(π, M t ) = k≥0 γ k r k t , the return is we assume stationary with reward function r tEQUATION)with d π the state occupancy measure defined in (16) . Definition C.3 (Robust (optimal) Return of NS-RMDPs). Let a return of π for any m t ∈ M t be G(π, M t ) := ∞ k=0 γ k r k t with kernel transition P t following π, with ∀k, t, r k t ∈ [0, 1], and the Robust non-stationary expected return with variation of kernel Let the robust performance of π for episode t be J R (π, t) := minm={m ′ t } t t ′ =t 0 ∈M L t E [G (π, m)] D Proof Theom 6.2 ∀t ∈ N + , ∀t 0 ∈ N + , |J R (π, t 0 ) -J R (π, t 0 + t)| ≤ L ′ t.withL ′ := γ (1-γ) 2 L P + 1 1-γ L rProof of Theorem 6.2. First, this difference can be upper bounded in the non robust case as:By definition, we can rewrite non-robust objective function and occupancy measure as.EQUATIONEQUATIONFirst, we can decompose the problem into sub-problems such thatEQUATIONusing triangular inequality. Looking at differences between two time steps:(1 -γ)|J(π, M t ) -J(π, M t+1 )| = s∈S d π (s, M t ) a∈A π(a | s)r t (s, a) - s∈S d π (s, M t+1 ) a∈A π(a | s)r t+1 (s, a) = s∈S a∈A π(a | s) (d π (s, M t ) r t (s, a) -d π (s, M t+1 ) r t+1 (s, a)) = s∈S a∈A π(a | s) (d π (s, M t ) (r t+1 (s, a) + (r t (s, a) -R t+1 (s, a))) -d π (s, M t+1 ) r t+1 (s, a)) = s∈S a∈A π(a | s) (d π (s, M t ) -d π (s, M t+1 )) r t+1 (s, a) + s∈S a∈A π(a | s)d π (s, M t ) (r t (s, a) -r t+1 (s, a)) (a) ≤ s∈S a∈A π(a | s) |d π (s, M t ) -d π (s, M t+1 )| |r t+1 (s, a)| + s∈S a∈A π(a | s)d π (s, M t ) |r t (s, a) -r t+1 (s, a)| (b) ≤ s∈S a∈A π(a | s) |d π (s, M t ) -d π (s, M t+1 )| + L R s∈S a∈A π(a | s)d π (s, M t ) = s∈S |d π (s, M t ) -d π (s, M t+1 )| + L rwhere (a) is triangular inequality, (b) is definition of of supremum of reward in the assumptions and reward bounded by 1. Then, let P π t ∈ R |S|×|S| be the transition matrix ( s ′ in rows and s in columns) resulting due to π and P t , i.e., ∀t, P π t (s ′ , s) := Pr (S t+1 = s ′ | S t = s, π, M t ), and let d π (•, M t ) ∈ R |S| denote the vector of probabilities for each state, then Finally we can easily bound the difference of occupation measure as :EQUATIONEQUATION≤ γ(1 -γ) -1EQUATIONEQUATIONEQUATIONwhich gives regrouping all terms:EQUATION)where the stationary MDP M t+1 can be chosen as the minimum over the previous MDPs at time step t such as |Pr (s ′ | s, a, M t ) -Pr (s ′ | s, a, M t+1 )| ≤ L p . Rewriting previous equation (27) , it holds thatEQUATION)Now considering non robust objective :EQUATION= minEQUATION= minEQUATION≤ maxEQUATIONwhere first equality is the definition of the robust objective, second equality is decomposition of minimum across time steps and final inequality is simply a property of the min such as | min amin b| ≤ sup |a -b|.Finally plugging 28 in (32) , it holds thatEQUATION= minEQUATION)Combining t times the previous equation gives the result:∀t ∈ N + , ∀t 0 ∈ N + , |J R (π, t 0 ) -J R (π, t 0 + t)| ≤ L ′ t.with L ′ := γ (1-γ) 2 L P + 1 1-γ L r E Implementation details E.i , a i , r(s i , a i ), ψ i , ψ i+1 , s i+1 } i∈[1,N ] ∼ B // Sample a mini-batch of transitions θ c ← θ c -α∇ θc L Q (θ c ) // Critic update phase θ a ← θ a -α∇ θa L π (θ a ) // Actor update ϕ c ← ϕ c + α∇ ϕc L Q(ϕ c ) // Adversary Critic update phase ϕ a ← ϕ a + α∇ ϕa L π (ϕ a )// Adversary update Note that in Time-constrained robust training Algorithm in section E.1, L Q and L π are as defined by [31] double critics and target network updates are omitted here for clarityIn Table 4 , for the stack algorithm, s i is defined as s i ← s i ∪ s i-1 ∪ a i-1 for Stacked-TC , and for the Oracle-TC version, s i ← s i ∪ ψ i .L Qθ c (TC-RARL) E [Q θc (s i , a i ) -r(s i , a i ) + γ min j=1,2 Q θc (s i+1 , π(s i+1 ))] L π (θ a ) (TC-RARL) -E [Q θc (s i , π θa (s i ))] L π (θ a ) (TC-RARL) E Qθc (s i , a i , π(s i , a i ), ψ i ) L Q(θ c ) (TC-RARL) E Qθc (s i , a i ) -r(s i , a i ) + γ min j=1,2 Qθc (s i+1 , π θa (s i+1 ), πθa (s i+1 , a i+1 , ψ i+1 )) L Qθ c Shared (TC-M2TD3) E [Q θc (s i , a i ) -r(s i , a i ) + γ min j=1,2 Q θc (s i+1 , π θa (s i+1 ), πθa (s i+1 , a i+1 , ψ i+1 ))] L π (θ a ) (TC-M2TD3) E [Q θc (s i , a i , πθa (s i , a i ), ψ i )] L π (θ a ) (TC-M2TD3) -E Qθc (s i , a i , πθa (s i , a i , ψ i ))Table 4 : Summary of Loss Functions for TD3 in TC-RARL and TC-M2TD3We employ a consistent neural network architecture for both the baseline and our proposed methods for the actor and the critic components. The architecture's design ensures uniformity and comparability across different models.The critic network is structured with three layers, as depicted in Figure 3a , the critic begins with an input layer that takes the state and action as inputs, which then passes through two fully connected linear layers of 256 units each. The final layer is a single linear unit that outputs a real-valued function, representing the estimated value of the state-action pair.The actor neural network, shown in Figure 3b , also utilizes a three-layer design. It begins with an input layer that accepts the state as input. This is followed by two linear layers, each consisting of 256 units. The output layer of the actor neural network has a dimensionality equal to the number of dimensions of the action space. We utilized the official M2TD3 [18] implementation provided by the original authors, accessible via the GitHub repository for M2TD3 and Oracle M2TD3.For the TC-M2TD3 or variants, we implemented the M2TD3 algorithm as specified. To simplify our approach, we omitted the implementation of the multiple ψ network and the system for resetting ψ. We replace with an adversary which π : S × A × Ψ → Ψ which minimize Q(s, a, ψ).We adopted the TD3 implementation from the CleanRL library, as detailed in [32] .A natural question arises regarding the worst time-constrained perturbation. Whether we adequately trained the adversary in the loop, or its suboptimal performance might lead to overestimating the trained agent reward against the worst-case time-constrained perturbation. We monitored the adversary's performance during its training against a fixed agent to address this. The attached figure shows the episodic reward (from the agent's perspective) during the adversary's training over 5 million timesteps, with a perturbation radius of L = 0.001. Each curve is an average of over 10 seeds. The plots show a rapid decline in reward during the initial stages of training, followed by quick stabilization. The episodic reward stabilizes early in the Ant (Figure 4a 4b ) environment, the reward shows a sharp initial decline and stabilizes, suggesting effective training. For Hopper (Figure 4c ), the reward decreases and then levels off, reflecting adversary convergence. Although the reward is more variable in the HumanoidStandup (Figure 4d ) environment, it ultimately reaches a steady state, confirming adequate training. Finally, in the Walker environment, the reward pattern demonstrates a quick drop followed by stabilization, indicating convergence. These observations confirm that the adversaries were not undertrained. The rapid convergence to a stable performance across all environments ensures the accuracy of the worst time-constrained perturbations estimated during training.The experiments of Section 5 follow the evaluation protocol proposed by [18] and based on MuJoCo environments [5] . These environments are designed with a 3D uncertainty sets. Table 7 lists all environments evaluated and their uncertainty sets. The uncertainty sets column defines the ranges of Table 8 reports the non-normalized time-constrained (with a radius of L = 0.001) worst-case scores, averaged across 10 independent runs for each benchmark. Table 9 reports the static worst case score obtained by each agent across a grid of environments, also averaged across 10 independent runs for 10 reports the static average case score obtained by each agent across a grid of environments, also averaged across 10 independent runs for each benchmark.At the beginning of each episode, ψ 0 ∼ U(Ψ) is selected for every fixed adversary. The episode length is 1000 steps. To begin with, the random fixed adversary simulates stochastic changes. It selects a parameter ψ t at each timestep within a radius of L = 0.1, which is 100 times larger than in our training methods. This tests the agents' adaptability to unexpected changes. In contrast, the cosine fixed adversary introduces deterministic changes using a cosine function. The radius 11 : Avg. performance against time-constrained fixed random adversary with a radius L = 0.1 over 10 seeds for each method of L = 0.1 scales the frequency of the cosine function, ensuring smooth and periodic variations. Additionally, a phase shift at the start of each episode ensures different starting points. Meanwhile, the linear fixed adversary employs a linear function. The parameters change linearly from the initial value to either one of a vertex of the uncertainty set Ψ over 1000 steps. Furthermore, the exponential fixed adversary uses an exponential function. Parameters change exponentially from the initial value to either of a vertex of the uncertainty set Ψ over 1000 steps. This ensures smooth and predictable variations. Similarly, the logarithmic fixed adversary uses a logarithmic function. Parameters change logarithmically from the initial value to either of a vertex of the uncertainty of the uncertainty set Ψ over 1000 steps, ensuring smooth and predictable variations. Agents trained under the time-constrained framework outperform all baselines in all environments for each fixed adversary, except when compared to the oracle TC method, which has access to ψ. In this case, the stacked-TC or TC methods outperform all baselines in all environments for the cosine, logarithmic, and exponential adversaries and outperform the fixed adversary baseline in 4 out of 5 instances for the random and linear fixed adversaries.We conducted training for each agent over a duration of 5 million steps, closely monitoring the cumulative rewards obtained over a trajectory spanning 1,000 steps. To enhance the reliability of our results, we averaged the performance curves across 10 different seeds. The graphs in All experiments were run on a desktop machine (Intel i9, 10th generation processor, 64GB RAM) with a single NVIDIA RTX 4090 GPU. Averages and standard deviations were computed from 10 independent repetitions of each experiment.This paper aims to advance robust reinforcement learning. It addresses general mathematical and computational challenges. These challenges may have societal and technological impacts, but we do not find it necessary to highlight them here.While our proposed Time-Constrained Robust Markov Decision Process (TC-RMDP) framework significantly advances robust reinforcement learning by addressing multifactorial, correlated, and time-dependent disturbances, several limitations must be acknowledged. The TC-RMDP framework assumes that the parameter vector ψ that governs environmental disturbances is known during training. In real-world applications, obtaining such detailed information may not always be feasible. This reliance on precise parameter knowledge limits the practical deployment of our algorithms in environments where ψ cannot be accurately measured or inferred. Our approach assumes that the environment's dynamics can be accurately parameterized and that these parameters remain within a predefined uncertainty set Ψ. This assumption might not hold in more complex or highly dynamic environments where disturbances are not easily parameterized or when the uncertainty set Ψ cannot comprehensively capture all possible variations. Consequently, the robustness of the learned policies might degrade when facing disturbances outside the considered parameter space. Addressing these limitations in future work. We used a simple heuristic to select the radius L in our experiments. First, we normalized the parameters of the uncertainty set, scaling them to lie within the range [0, 1] for each dimension. We then divided this range by the number of steps in an episode, denoted as ep, yielding L = 1 ep . Our experiments, with ep = 1000, resulted in a radius of L = 0.001. This choice ensures that any initial parameter ψ 0 can move to any point within the uncertainty set's parameters for an episode. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best Justification: Our contribution aims to safer RL algorithms, and we conducted our experiment of open source benchmarks.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: Most existing works on robust MDP make the (s,a) rectangularity assumption, which inherently assumes that the transition dynamics from a given current state to the next states are independent even for current states closer to each other. This assumption is strong, and may not be applicable in real-world problems. This paper proposes a new time-constrained robust MDP (TC-RMDP) formulation that considers correlated and time-dependent disturbances. The paper proposes three algorithms to solve this problem, each using varying levels of environmental information. The paper provides an extensive simulation-based evaluation of the proposed algorithms using standard benchmarks. 


We received four expert reviews, with the scores, 7, 6, 5, 5, and the average score is 5.75. Reviewers gave positive comments about the novelty of presenting the TC-RMDP, the novelty of three algorithms, and the extensive simulation-based evaluations of the algorithms. The reviewers are also happy about the quality of the presentation. 

The reviewers have also provided some very useful comments to improve the paper. Reviewer 1Xhb has asked about the complexity of proposed algorithms, selecting the parameter L, and providing a visualization/figure of the traditional (s,a) rectangular uncertainty set and uncertainty considered in this paper. Reviewer ZKxz has asked to include some specific examples of the practical applications where such uncertainty sets are relevant. I think all these are important points, and I recommend authors to include these details in their final manuscript. .

Reviews: review: paper_summary: paper_summary: The paper defines a robust training method for MDPs under uncertainty in the environment dynamics. Current methods assume sa-rectangularity, where the transition dynamics from consecutive states are independent of one another. The authors argue that this assumption is unrealistic and leads to overly conservative policies. Alternatively, this paper breaks the independence assumption in the environment dynamics to use a time-step evolution approach. This paper presents a Time-Constrained TC-MDP where the Time-Constrained Parameterized MDP kernels evolve each time-step constrained to be Lipschitz continuous. The model is trained in an adversarial manner, where the adversarial policy is learned to evolve the model dynamics but is limited to be L-close to the previous dynamics by definition. This learning method is tested under different variant conditions (cosine, exponential, linear, and logarithmic) with positive results. Robustly trained agents that learn under the worst TC adversities outperform vanilla robust algorithms that assume time independence in the model dynamics.
strength_weakness: strength_weakness: - Reporting that the worst-case TC variants outperform vanilla adversarial models as M2TD3 or RARL is expected. As the decisions of the adversary are rather limited when compared to the vanilla algorithms. Perhaps highlighting the evaluation results (fixed or static setting) in the conclusions would help in emphasizing this.
questions: questions: - In Algorithm 1, should is be: $b_{t+1} = \bar{\pi}(s_t, a_t, \psi_t)$ ?
- Typo in Page 7 Line 276: "Appendix G and G"
limitations: limitations: As the authors mention, this method assumes that the environment dynamics can be correctly parametrized in a confident set $Psi$. 

Additionally, selecting the radius L of the proximal adversarial set of actions is not trivial. Fortunately, the authors tested the environment with a large enough L (100 times larger than the one used during training) with positive results, which validates their claim.
paper_summary: paper_summary: The paper introduces Time-Constrained Robust MDPs (TC-RMDPs) as a novel formulation to address the overly conservative nature of traditional robust RL under sa-rectangularity assumptions. The authors propose three algorithms to handle time-dependent and correlated disturbances in different situations. Extensive evaluations on continuous control benchmarks show that these algorithms outperform traditional robust RL methods in terms of balancing performance and robustness.
strength_weakness: strength_weakness: # Major
1. The complexity of the proposed algorithms, especially Oracle-TC, may limit their practical applicability in real-world scenarios where complete environmental information is unavailable.

2. How to select the Lipschitz constant $L$ in practical scenarios? Some ablations studies and analysis regarding $L$ would be beneficial.

3. It would be better if the authors could provide some visualizations of the difference between the traditional independent transitions with the time-constrained parametric transitions.

4. What is the number of worst-case episodes in the experiments? If it is 1, maybe it would be better to display the results as the average of the worst 10% of episodes for better robustness. This is because having only 1 episode might introduce stochasticity.

# Minor
1. It is suggested to show the equation number for better illustration and reference.
2. There are some typos in the paper. For instance, in line 102, $B=\mathcal{B}(0_{\Psi},L)$ should be corrected to $B=\mathcal{B}({\Psi}_0,L)$,  the right-most column in Table 1 should be Avg. The authors should carefully review the paper for such errors.
questions: questions: please see above
limitations: limitations: The framework assumes that the transition parameter vector $\psi$ is known during training. However, in real-world applications, it could be challenging. Also, the assumption on the uncertainty set $\Psi$ is also not practical in practice.
paper_summary: paper_summary: This paper aims to develop a new framework of robust RL addressing the over-conservability issue of rectangularity and dynamic uncertainty set assumption. This framework allows for time-dependent, correlated and multifactorial disturbance to the dynamics. Three distinct algorithms are developed depending on the available information to the policy. Extensive numerical experiments are further provided to demonstrate the performance of the proposed algorithms.
strength_weakness: strength_weakness: - Thm 2.1 only applies to the case where the policy has the exact information of \psi, which in practice is usually unavailable. 
- The usefulness of this framework remains questionable, as it poses a significant challenge of constructing such complex uncertainty set with time and state-action correlation.
- The training algorithm has the same flavor of the adversarial training as those in the literature, e.g., RARL. The novelty seems rather limited as this is mostly an  experimental paper.
questions: questions: See weaknesses above.
limitations: limitations: na
paper_summary: paper_summary: This paper proposed a novel concept, time-constrained robust MDP, to address the conservativeness issue of the rectangular assumption. They assume the transition depends on an underlying parameter, and the parameter can be adversarially chosen from an uncertainty set. Several algorithms are proposed to solve the time-constrained robust MDP. Extensive numerical experiments show the effectiveness of the proposed framework and algorithms.
strength_weakness: strength_weakness: My concern is that the methods proposed in this paper might be too heuristic, given the non-stationarity and possibly non-existence of the optimal policy. As I am not very familiar with the experiment side, I am not sure if the evaluation in this paper is sufficient. See my questions for details.
questions: questions: 1. In the experiments, the oracle-TC is not always the best method. It seems that the performances are not that stable. Also, the vanilla TC outperforms in almost half cases. It is kind of anti-intuitive.

2. Compared to baseline methods, the TC-based methods show robustness to some extent, but how robust are they? What are the expected best (robust) performances?

3. In algorithm 1, what is "sample a mini-batch of transitions" for? How are the agent and adversary updated?

4. On lines 174-175, the authors claim that the rectangularity assumptions are rarely met in real-world scenarios. I think more evidence or references are required to support this claim.
limitations: limitations: yes

Comments: comment: Thank you for your thoughtful review. We appreciate your feedback and the opportunity to clarify our work. We hope that our explanations regarding the preservation of the optimal policy and the interpretation of empirical results have helped to address your concerns. We believe these points underscore the value of our contributions. With this in mind, we humbly ask if you might reconsider the scores, given also that the rebuttal seems to have addressed most of your concerns.
The reviewer thank the authors for the feedback. They addressed my concerns. I raised the score to 5.
Thanks for the authors' rebuttal. It addressed most of my concerns.
Thanks to the authors for the clarification, It addresses my concerns.
Thanks for the authors' response. It addressed most of my concerns, and I believe the paper meets the acceptance threshold.