Title: Time-Constrained Robust MDPs
Venue: NeurIPS 2024
Accepted: yes
Keywords: Robust reinforcement learning
Abstract: Robust reinforcement learning is essential for deploying reinforcement learning algorithms in real-world scenarios where environmental uncertainty predominates.
Traditional robust reinforcement learning often depends on rectangularity assumptions, where adverse probability measures of outcome states are assumed to be independent across different states and actions. 
This assumption, rarely fulfilled in practice, leads to overly conservative policies. 
To address this problem, we introduce a new time-constrained robust MDP (TC-RMDP) formulation that considers multifactorial, correlated, and time-dependent disturbances, thus more accurately reflecting real-world dynamics. This formulation goes beyond the conventional rectangularity paradigm, offering new perspectives and expanding the analytical framework for robust RL.
We propose three distinct algorithms, each using varying levels of environmental information, and evaluate them extensively on continuous control benchmarks. 
Our results demonstrate that these algorithms yield an efficient tradeoff between performance and robustness, outperforming traditional deep robust RL methods in time-constrained environments while preserving robustness in classical benchmarks.
This study revisits the prevailing assumptions in robust RL and opens new avenues for developing more practical and realistic RL applications.

Time-Constrained Robust MDPs

Abstract

Robust reinforcement learning is essential for deploying reinforcement learning algorithms in real-world scenarios where environmental uncertainty predominates. Traditional robust reinforcement learning often depends on rectangularity assumptions, where adverse probability measures of outcome states are assumed to be independent across different states and actions. This assumption, rarely fulfilled in practice, leads to overly conservative policies. To address this problem, we introduce a new time-constrained robust MDP (TC-RMDP) formulation that considers multifactorial, correlated, and time-dependent disturbances, thus more accurately reflecting real-world dynamics. This formulation goes beyond the conventional rectangularity paradigm, offering new perspectives and expanding the analytical framework for robust RL. We propose three distinct algorithms, each using varying levels of environmental information, and evaluate them extensively on continuous control benchmarks. Our results demonstrate that these algorithms yield an efficient tradeoff between performance and robustness, outperforming traditional deep robust RL methods in time-constrained environments while preserving robustness in classical benchmarks. This study revisits the prevailing assumptions in robust RL and opens new avenues for developing more practical and realistic RL applications.

1 Introduction

Robust MDPs capture the problem of finding a control policy for a dynamical system whose transition kernel is only known to belong to a defined uncertainty set. The most common framework for analyzing and deriving algorithms for robust MDPs is that of sa -rectangularity [1, 2], where probability measures on outcome states are picked independently in different source states and actions (in formal notation, P ( s ′ | s, a ) and P ( s ′ | ¯ s, ¯ a ) are independent of each other). This provides an appreciable decoupling of worst transition kernel search across time steps and enables sound algorithms like robust value iteration (RVI). But policies obtained for such sa -rectangular MDPs are by nature very conservative [3, 4], as they enable drastic changes in environment properties from one time step to the next, and the algorithms derived from RVI tend to yield very conservative policies even when applied to nonsa -rectangular robust MDP problems.

In this paper, we depart from the rectangularity assumption and turn towards a family of robust MDPs whose transition kernels are parameterized by a vector ψ . This parameter vector couples together the outcome probabilities in different ( s, a ) pairs, hence breaking the independence assumption that is problematic, especially in large dimension [3]. This enables accounting for the notion of transition

model consistency across states and actions: outcome probabilities are not picked independently anymore but are rather set across the state and action spaces by drawing a parameter vector. In turn, we examine algorithms for solving such parameter-based robust MDPs when the parameter is constrained to follow a bounded evolution throughout time steps. Our contributions are the following.

We introduce a formal definition for parametric robust MDPs and time-constrained robust MDPs, discuss their properties and derive a generic algorithmic framework (Sec. 2).

We propose three algorithmic variants for solving time-constrained MDPs, named vanilla TC , Stacked-TC and Oracle-TC (Sec. 4), which use different levels of information in the state space, and come with theoretical guaranties (Sec. 6).

These algorithms are extensively evaluated in MuJoCo [5] benchmarks, demonstrating they lead to non-conservative and robust policies (Sec. 5).

2 Problem statement

(Robust) MDPs. A Markov Decision Process (MDP) [6] is a model of a discrete-time, sequential decision making task. At each time step, from a state s t ∈ S of the MDP, an action a t ∈ A is taken and the state changes to s t +1 according to a stationary Markov transition kernel p ( s t +1 | s t , a t ) , while concurrently receiving a reward r ( s t , a t ) . S and A are measurable sets and we write ∆ S and ∆ A the set of corresponding probability distributions. A stationary policy π ( ·| s ) is a mapping from states to distributions over actions, prescribing which action should be taken in s . The value function v π p of policy π maps state s to the expected discounted sum of rewards E p,π [ ∑ t γ t r t ] when applying π from s for an infinite number of steps. An optimal policy for an MDP is one whose value function is maximal in any state. In a Robust MDP (RMDP) [1, 2], the transition kernel p is not set exactly and can be picked in an adversarial manner at each time step, from an uncertainty set P . Then, the pessimistic value function of a policy is v π P ( s ) = min p ∈P v π p ( s ) . An optimal robust policy is one that has the largest possible pessimistic value function v ∗ P in any state, hence yielding an adversarial max π min p optimization problem. Robust Value Iteration (RVI) [1, 7] solves this problem by iteratively computing the one-step lookahead best pessimistic value:

The T ∗ P operator is called the robust Bellman operator and the sequence of v n functions converges to the robust value function v ∗ P as long as the adversarial transition kernel belongs to the simplex of ∆ S .

Zero-sum Markov Games. Robust MDPs can be cast as zero-sum two-players Markov games [8, 9] where B is the action set of the adversarial player. Writing ¯ π : S × A → ∆ B the policy of this adversary, the robust MDP problem turns to max π min ¯ π v π, ¯ π , where v π, ¯ π ( s ) is the expected sum of discounted rewards obtained when playing π (agent actions) against ¯ π (transition models) at each time step from s . This enables introducing the robust value iteration sequence of functions

where T π, ¯ π := E a ∼ π ( s ) [ r ( s, a ) + γ E s ′ ∼ ¯ π ( s,a ) v n ( s ′ )] is a zero-sum Markov game operator. These operators are also γ -contractions and converge to their respective fixed point v π, ¯ π and v ∗∗ = v ∗ P [9]. This formulation will be useful to derive a practical algorithm in Section 4.

Often, this convergence is analyzed under the assumption of sa -rectangularity, stating that the uncertainty set P is a set product of independent subsets of ∆ S in each s, a pair. Quoting [1], rectangularity is a sort of independence assumption and is a minimal requirement for most theoretical results to hold. Within robust value iteration, rectangularity enables picking ¯ π ( s t , a t ) completely independently of ¯ π ( s t -1 , a t -1 ) . To set ideas, let us consider the robust MDP of a pendulum, described by its mass and rod length. Varying this mass and rod length spans the uncertainty set of transition models. The rectangularity assumption induces that ¯ π ( s t , a t ) can pick a measure in ∆ S corresponding to a mass and a length that are completely independent from the ones picked in the previous time step. While this might be a good representation in some cases, in general it yields policies that are very conservative as they optimize for adversarial configurations which might not occur in practice.

We first step away from the rectangularity assumption and define a parametric robust MDP as an RMDP whose transition kernels are spanned by varying a parameter vector ψ (typically the mass

and rod length in the previous example). Choosing such a vector couples together the probability measures on successor states from two distinct ( s, a ) and ( s ′ , a ′ ) pairs. The main current robust deep RL algorithms actually optimize policies for such parametric robust MDPs but still allow the parameter value at each time step to be picked independently of the previous time step.

Parametric MDPs. A parametric RMDP is given by the tuple ( S, A, Ψ , p ψ , r ) where the transition kernel p ψ ( s, a ) ∈ ∆ S is parameterized by ψ , and Ψ is the set of values ψ can take, equipped with an appropriate metric. This yields the robust value iteration update :

A parametric RMDP remains a Markov game and the Bellman operator remains a contraction mapping as long as p ψ can reach only elements in the simplex of ∆ S , where the adversary's action set is the set of parameters instead of a (possibly sa -rectangular) set of transition kernels.

Time-constrained RMDPs (TC-RMDPs). We introduce TC-RMDPs as the family of parametric RMDPs whose parameter's evolution is constrained to be Lipschitz with respect to time. More formally a TC-RMDP is given by the tuple ( S, A, Ψ , p ψ , r, L ) , where ∥ ψ t +1 -ψ t ∥ ≤ L , that is the parameter change is bounded through time. In the previous pendulum example, this might represent the wear of the rod which might lose mass or stretch length. Similarly, and for a larger scale illustration, TC-RMDPs enable representing the possible evolutions of traffic conditions in a path planning problem through a busy town. Starting from an initial parameter value ψ -1 , the pessimistic value function of a policy π is non-stationary, as ψ 0 is constrained to lay at most L -far away from ψ -1 , ψ 1 from ψ 0 , and so on. Generally, this yields non-stationary value functions as the uncertainty set at each time step depends on the previous uncertainty parameter. To regain stationarity without changing the TC-RMDP definition, we first change the definition of the adversary's action set. The adversary picks its actions in the constant set B = B (0 Ψ , L ) , which is the ball of radius L centered in the null element in Ψ . In turn, the state of the Markov game becomes the pair s, ψ and the Markov game itself is given by the tuple (( S × Ψ) , A, B, p ψ , r ) , where the Lipschitz constant L is included in B . Thus, given an action b t ∈ B and a previous parameter value ψ t -1 , the parameter value at time t is ψ t = ψ t -1 + b t . Then, we define the pessimistic value function of a policy as a function of both the state s and parameter ψ :

In turn, an optimal robust policy is a function of s and ψ and the TC robust Bellman operators are:

This iteration scheme converges to a fixed point according to Th. 2.1.

Theorem 2.1. The time-constrained (TC) Bellman operators T π B and T ∗ B are contraction mappings. Thus the sequences v n +1 = T π B v n and v n +1 = T ∗ B v n , converge to their respective fixed points v π B and v ∗ B .

Proof of Th. 2.1 can be found in Appendix B. We refer to this formulation as algorithm Oracle-TC (see Section 4 for implementation details) since an oracle makes the current parameter ψ visible to the agent. Therefore, it is possible to derive optimal policies for TC-RMDPs by iterated application of this TC Bellman operator. These policies have the form π ( s, ψ ) . In the remainder of this paper, we extend state-of-the-art robust deep RL algorithms to the TC-RMDP framework. In particular, we compare their performance and robustness properties with respect to classical robust MDP formulations, we also discuss their relation with the π ( s ) robust policies of classical robust MDPs.

If the agent is unable to observe the state variable ψ , it is not possible to guarantee the existence of a stationary optimal policy of the form π ( s ) . Similarly, there is no guarantee of convergence of value functions to a fixed point. Nonetheless, this scenario, in which access to the ψ parameter is not available, is more realistic in practice. It turns the two-player Markov game into a partially observable

Markov game, where one can still apply the TC Bellman operator but without these guarantees of convergence. We call vanilla TC the repeated application of the TC Bellman operator in this partially observable case. Vanilla TC will be tested in practice, and some theoretical properties of the objective function will be derived using the Lipschitz properties (Sec 6).

3 Related works

Since our method is a non-rectangular, Deep Robust RL algorithm, (possibly non-stationary for Stacked-TC and TC ), we discuss the following related work.

Non-stationary MDPs. First, non-stationarity has been studied in the Bandits setting in [10]. Then, for episodic, non-stationary MDPs [11, 12, 13] have explored and provided regret bounds for algorithms that use oracle access to the current reward and transition functions. More recently [14, 15] have facilitated oracle access by performing a count-based estimation of the reward and transition functions based on the recent history of interactions. Finally, for tabular MDPs, past data from a non-stationary MDP can be used to construct a full Bayesian model [16] or a maximum likelihood model [17] of the transition dynamics. We focus on the setting not restricted to tabular representations

Non-rectangular RMDPs. While rectangularity in practice is very conservative, it can be demonstrated that, in an asymptotic sense, non-rectangular ellipsoidal uncertainty sets around the maximum likelihood estimator of the transition kernel constitute the smallest possible confidence sets for the ground truth transition kernel, as implied by classical Cramér-Rao bounds. This is in accordance with the findings presented in § 5 and Appendix A of [7]. More recently, [3] extends the rectangular assumptions using a factored uncertainty model, where all transition probabilities depend on a small number of underlying factors denoted w 1 , . . . , w r ∈ R S , such that each transition probability P sa for every ( s, a ) is a linear (convex) combination of these r factors. Finally, [4] use policy gradient algorithms for non-rectangular robust MDPs. While this work presents nice theoretical guarantees of convergence, there is no practical Deep RL algorithms for learning optimal robust policies.

Deep Robust RL Methods. Many Deep Robust algorithms exist such as M2TD3 [18], M3DDPG [19], or RARL [20], which are all based on the two player zero-sum game presented in 2. We will compare our method against these algorithms, except [19] which is outperformed by [18] in general. We also compare our algorithm to Domain randomization (DR) [21] that learns a value function V ( s ) = max π E p ∼U ( P ) V π p ( s ) which maximizes the expected return on average across a fixed (generally uniform) distribution on P . As such, DR approaches do not optimize the worst-case performance but still have good performance on average. Nonetheless, DR has been used convincingly in applications [22, 23]. Finally, the zero-sum game formulation has lead to the introduction of action robustness [9] which is a specific case of rectangular MDPs, in scenarios where the adversary shares the same action space as the agent and interferes with the agent's actions. Several strategies based on this idea have been proposed. One approach, the Game-theoretic Response Approach for Adversarial Defense (GRAD) [24] builds on the Probabilistic Action Robust MDP (PR-MDP) [9]. This method introduces time-constrained perturbations in both the action and state spaces and employs a game-theoretic approach with a population of adversaries. In contrast to GRAD, where temporal disturbances affect the transition kernel around a nominal kernel, our method is part of a broader setting in which the transition kernel is included in a larger uncertainty set. Robustness via Adversary Populations (RAP) [25] introduces a population of adversaries. This approach ensures that the agent develops robustness against a wide range of potential perturbations, rather than just a single one, which helps prevent convergence to suboptimal stationary points. Similarly, State Adversarial MDPs [26, 27, 28, 24] address adversarial attacks on state observations, effectively creating a partially observable MDP. Finally, using rectangularity assumptions, [29, 30] use Wasserstein and χ 2 balls respectively for the uncertainty set in Robust RL.

4 Time-constrained robust MDP algorithms

The TC-RMDP framework addresses the limitations of traditional robust reinforcement learning by considering multifactorial, correlated, and time-dependent disturbances. Traditional robust reinforcement learning often relies on rectangularity assumptions, which are rarely met in real-

Environment

+p

+p

Yt+1

元

T

Adversary

Agent

St t St-1 at-1

St 4t

world scenarios, leading to overly conservative policies. The TC-RMDP framework provides a more accurate reflection of real-world dynamics, moving beyond the conventional rectangularity paradigm.

We cast the TC-RMDP problem as a two-player zero-sum game, where the agent interacts with the environment, and the adversary (nature) changes the MDP parameters ψ . Our approach is generic and can be derived within any robust value iteration scheme, performing max π ( s ) ∈ ∆ A min ψ ∈ Ψ E a ∼ π ( s ) [ r ( s, a ) + γ E s ′ ∼ p ψ ( s,a ) v n ( s ′ )] updates, by modifying the adversary's action space and potentially the agent's state space to obtain updates of the form max π ( s,ψ ) ∈ ∆ A min b ∈ B E a ∼ π ( s ) [ r ( s, a ) + γ E s ′ ∼ p ψ + b ( s,a ) v n ( s ′ )] . In Section 5, we will introduce time constraints within two specific robust value iteration algorithms, namely RARL [20] and M2TD3 [18] by simply limiting the search space for worst-case ψ at each step. This specific implementation extends the original actor-critic algorithms. For the sake of conciseness, we refer the reader to Appendix E.1 for details regarding the loss functions and algorithmic details.

Three variations of the algorithm are provided (illustrated in Figure 1) but all fall within the training loop of Algorithm 1.

Algorithm 1 Time-constrained robust training

Oracle-TC . As discussed in Section 2, the Oracle-TC version includes the MDP state and parameter value as input, π : S × Ψ → A . This method assumes that the agent has access to the true parameters of the environment, allowing it to make the most informed decisions and possibly reach the true robust value function. However, these parameters ψ are sometimes non-observable in practical scenarios, making this method not always feasible.

Stacked-TC . Since ψ might not be observable but may be approximately identified by the last transitions, the Stacked-TC policy uses the previous state and action as additional inputs in an attempt to replace ψ , π : S ×A×S → A . This approach leverages the information in the transitions, even though it might be insufficient for a perfect estimate of ψ . It aims to retain (approximately) the convergence properties of the Oracle-TC algorithm.

Vanilla TC . Finally, the vanilla TC version takes only the state, π : S → A , as input, similar to standard robust MDP policies. This method does not attempt to infer the environmental parameters or the transition dynamics explicitly. Instead, it relies on the current state information to guide the agent's actions. While this version is the most straightforward and computationally efficient, it may not perform as robustly as the Oracle-TC or Stacked-TC versions in environments with significant temporal disturbances, since it attempts to solve a partially observable Markov game, for which there may not exist a stationary optimal policy based only on the observation. Despite this, it remains a viable option in scenarios where computational simplicity and quick decision-making are prioritized.

5 Results

Experimental settings . This section evaluates the robust time-constrained algorithm's performance under severe time constraints and in the static settings. Experimental validation was conducted in continuous control scenarios using the MuJoCo simulation environments [5]. The approach was categorized into three variants. The Oracle-TC , where the agent accessed environmental parameters π ( s t , ψ ) ; the Stacked-TC , where the agent took in input π ( s t , s t -1 , a t -1 ) ; and the vanilla TC , which did not receive any additional inputs π ( s ) . For each variant of the time-constrained algorithms, we applied them to RARL [20], and M2TD3 [18], renaming them TC-RARL and TC-M2TD3, respectively. The algorithms were tested against two state-of-the-art robust reinforcement learning algorithms, M2TD3 and RARL. Additionally, the Oracle versions of M2TD3 and RARL, where the agent's policy included ψ in the input π : S × Ψ →A , were evaluated for a more comprehensive assessment. Comparisons were also made with Domain Randomization (DR) [21] and vanilla TD3. [31] to ensure a thorough analysis. A 3 D uncertainty set is defined in each environment P normalized between [0 , 1] 3 . Appendix G provides detailed descriptions of uncertainty parameters. Performance metrics were gathered after five million steps to ensure a fair comparison. All baselines were constructed using TD3, and a consistent architecture was maintained across all TD3 variants. The results presented below were obtained by averaging over ten distinct random seeds. Appendices E.4, E.3, E.2, and H.2 discuss further details on hyperparameters, network architectures, and implementation choices, including training curves for our methods and baseline comparisons. In the following tables 1, 2, 3, the best performances are shown in bold. Oracle methods, with access to optimal information, are shown in black. Items in bold and green represent the best performances with limited information on ψ , making them more easily usable in many scenarios. When there is only one element in bold and green, this implies that the best overall method is a non-oracle method.

Performance of TCRMDPs in worst-case time-constrained . Table 1 reports the worst-case time-constrained perturbation. To address the worst-case time-constrained perturbations for each trained agent π ∗ , we utilized a time-constrained adversary using TD3 algorithm ¯ π ∗ = min b ∈ B E a ∼ π ∗ ( s ) ,b ∼ ¯ π ( s,a,ψ ) [ r ( s, a ) + γ E s ′ ∼ p ψ + b ( s, a ) v n ( s ′ )] within a perturbation radius of L = 0 . 001 for a total of 5 million steps. The sum of episode rewards was averaged over 10 episodes. A heuristic to choose the perturbation radius is explained in Appendix K. To compare metrics across different environments, each method's score v was standardized relative to the reference score of TD3. TD3 was trained on the environment using default transition function parameters, with its score denoted as v TD 3 . The M2TD3 score, v M 2 TD 3 , was used as the comparison target. The formula applied was ( v -v TD 3 ) / ( | v M 2 TD 3 -v TD 3 | ) . This positioned v TD 3 as the minimal baseline and v M 2 TD 3 as the target score. This standardisation provides a metric that quantifies the

improvement of each method over TD3 in relation to the improvement of M2TD3 over TD3. In each evaluation environment, agents trained with the time-constrained framework (indicated by TC in the method name) demonstrated significantly superior performance compared to those trained using alternative robust reinforcement learning approaches, including M2TD3 and RARL. Furthermore, they outperformed those trained through domain randomisation (DR). Notably, even without directly conditioning the policy with ψ , the time-constrained trained policies excelled against all baselines, achieving up to a 2.9-fold improvement. The non-normalized scores are reported in Appendix H. Additionally, when policies were directly conditioned by ψ and trained within the robust reinforcement learning framework, they tended to be overly conservative in the time-constrained framework. This is depicted in Table 1, comparing the performances of Oracle RARL, Oracle M2TD3, Oracle TC-RARL, and Oracle TC-M2TD3. Both policies also observe ψ . The only difference is that Oracle RARL and Oracle M2TD3 were trained in the robust reinforcement learning framework, while Oracle TC-RARL and Oracle TC-M2TD3 were trained in the time-constrained framework. The performance differences under worst-case time-coupled perturbation are as follows: for Oracle RARL (resp. M2TD3) and Oracle TC-RARL (resp. M2TD3), the values are -0 . 86 ( 1 . 39 ) vs. 2 . 54 ( 3 . 07 ). This observation highlights the need for a balance between robust training and flexibility in dynamic conditions. A natural question arises regarding the worst-case time-constrained perturbation. Was the adversary in the loop adequately trained, or might its suboptimal performance lead to overestimating the trained agent's reward against the worst-case perturbation? The adversary's performance was monitored during its training against all fixed-trained agents. The results in Appendix F show that our adversary converged.

Robust Time-Constrained Training under various time fixed adversaries . The method was evaluated against various fixed adversaries, focusing on the random fixed adversary shown in Figure 2. This evaluation shows that robustly trained agents can handle dynamic and unpredictable conditions. The random fixed adversary simulates stochastic changes by selecting a parameter ψ t at each timestep within a radius of L = 0 . 1 . This radius is 100 times larger than in our training methods. At the start of each episode, ψ 0 is uniformly sampled from the uncertainty set ψ 0 ∼ U ( P ) . This tests the agents' adaptability to unexpected changes. Figures 2a through 2e show our agents' performance. Agents trained with our robust framework consistently outperformed those trained with standard methods. The policy was also assessed against five other fixed adversaries: cosine, exponential, linear, and logarithmic. Detailed results are provided in the Appendix. H.1.

(a) Ant

(b) HalfCheetah

(c) Hopper

(d) Humanoid

(e) Walker

Mean episode reward

8000

Mean episode reward

8000

Mean episode reward

2000

60000

400

4000

1000

2000

2000

TD3

SracleTC-

Algorithm

Algorithm

Algorithm

14000

Mean episode reward

80-

10

6-

00

4000

4000

200

TC-M2TD3

TC-RARL

acleTo

Algorithm

Algorithm

Performance of Robust Time-Constrained MDPs in the static setting . In static environments, the Robust Time-Constrained algorithms were evaluated for worst-case and average performance metrics, shown in Tables 2 and 3. A fixed uncertainty set P was used, dividing each dimension of Ψ into ten segments, creating a grid of 1000 points (10 3 ) . Each agent ran five episodes at each grid point, and the rewards were averaged. The scores were normalized as described for the

time-constrained adversary analysis in Table 1. The raw data is provided in Appendix 9 and 10. Performance scores were adjusted relative to the baseline v TD 3 and v M 2 TD 3 . As a result, normalized results reveal distinct trends among agent configurations within the TC-RMDP framework. The Oracle TC-M2TD3 variant achieved an average score of 3.12 3, while the Stacked TC-M2TD3 scored 2.23, indicating its resilience. Furthermore, in the worst-case scenario, the TC-RARL and Stacked TC-RARL variants demonstrated adaptability, with TC-RARL scoring 0 . 92 and TC-M2TD3 scoring 1 . 02 2. This performance highlights its reliability in challenging static environments.

6 Some Theoretical properties of TC-MDPS.

6.1 On the optimal policy of TC

Following Lemma 3.3 of [1], it is known that in the rectangular case, there exists an optimal policy of the adversary that is stationary, provided that the actor policy is stationary. The TC-RMDP definition enforces a limitation on the temporal variation of the transition kernel. Consequently, all stationary adversarial policies are constrained by this stipulation. In turn, this guarantees that (under the hypothesis of sa -rectangularity) there always exists a solution to the TC-RMDP that is also a solution to the original RMDP. In other words: optimizing policies for TC-RMDPs do not exclude optimal solutions to the underlying RMDP. This sheds an interesting light on the search for robust optimal policies, since TC-RMDPs shrink the search space of optimal adversarial policies. In practice, this is confirmed by the previous experimental results (Figure 2) where the optimal agent policy found by either Oracle-TC , Stacked-TC , or vanilla TC actually outperforms the one found by M2TD3 or RARL in the non time-constrained setting.

6.2 Some Lipchitz-properties for non-stationary TC-RMPDS.

In this subsection we slightly depart from the framework defined in Section 2 and study the smoothness of the robust objective for vanilla TC or Stacked-TC . Th. 2.1 is no longer applicable as ψ is not observed. However, we can still give smoothness of the objective starting from Lipchichz conditions

on the evolution of the parameter that leads to smoothness on reward and transition kernel in the following definition 6.1.

Definition 6.1 (Reward/Kernel Lipchitz TC-RMDPs [13]) . We say that a parametric RDMPs is time constrained if the parameter change is bounded through time ie. ∥ ψ t +1 -ψ t ∥ ≤ L . Moreover, we assume that this variation in parameter implies a variation in the reward and transition kernel of

From a theoretical point of view, a TC-RMDP can be seen as a sequence of stationary MDPs with time indexed reward and transition kernel r t , P t that have continuity. More formally for M t = ( S, A, Ψ , p ψ t , r t , L = ( L p , L r )) , we can then define the sequence of stationary MDPs with Lipchitz variation :

Defining r k t as the random variable corresponding to the reward function at time step t for stationary MDPs, but iterating with index k , the stationary rollout return at time t is G ( π, M t ) = ∑ k ≥ 0 γ k r k t . Assuming that at a fixed t the reward and transition kernel r t , P t are fixed, the robust objective function is:

This leads to the following guarantee for vanilla TC and Stacked-TC algorithms.

Theorem 6.2. Assume TC-RMPDS with L = ( L r , L P ) smoothness. Then ∀ t ∈ N , r t ∈ [0 , 1] ,

This theorem states that a small variation of the Kernel and reward function will not affect too much the robust objective. In other terms, despite the fact that the TC Bellman operator may not admit a fixed point and yield a non-stationary sequence of value functions, variations of the expected return remain bounded. Proof of the Th. 6.2 can be found in Appendix C.

7 Conclusion

This paper presents a novel framework for robust reinforcement learning, which addresses the limitations of traditional methods that rely on rectangularity assumptions. These assumptions often result in overly conservative policies, which are not suitable for real-world applications where environmental disturbances are multifactorial, correlated, and time-constrained. In order to overcome these challenges, we proposed a new formulation, the Time-Constrained Robust Markov Decision Process (TC-RMDP). The TC-RMDP framework is capable of accurately capturing the dynamics of real-world environments, due to its consideration of the temporal continuity and correlation of disturbances. This approach resulted in the development of three algorithms: The three algorithms, Oracle-TC , Stacked-TC , vanilla TC which differ in the extent to which environmental information is incorporated into the decision-making process. A comprehensive evaluation of continuous control benchmarks using MuJoCo environments has demonstrated that the proposed TC-RMDP algorithms outperform traditional robust RL methods and domain randomization techniques. These algorithms achieved a superior balance between performance and robustness in both time-constrained and static settings. The results confirmed the effectiveness of the TC-RMDP framework in reducing the conservatism of policies while maintaining robustness. Moreover, we provided theoretical guaranties for Oracle-TC in Th. 2.1 and for Stacked-TC and vanilla TC in Th. 6.2. This study contributes to the field of robust reinforcement learning by introducing a time-constrained framework that more accurately reflects the dynamics observed in real-world settings. The proposed algorithms and theoretical contributions offer new avenues for the development of more effective and practical RL applications in environments with complex, time-constrained uncertainties.

Acknowledgments

The authors acknowledge the support of the DEEL and ENVIA projects, the funding of the AI Interdisciplinary Institute ANITI funding, through the French 'Investing for the Future - PIA3' program under grant agreement ANR-19-PI3A-0004. This work benefited from computing resources from CALMIP under grant P21001. Pierre Clavier has been supported by a grant from Region Ile-de-France; DIM Math Innov and also by Fondation Mathématique Jacques Hadamard.

References

Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research , 30(2):257-280, 2005.

Arnab Nilim and Laurent El Ghaoui. Robust control of markov decision processes with uncertain transition matrices. Operations Research , 53(5):780-798, 2005.

Vineet Goyal and Julien Grand-Clement. Robust markov decision process: Beyond rectangularity. arXiv preprint arXiv:1811.00215 , 2018.

Mengmeng Li, Tobias Sutter, and Daniel Kuhn. Policy gradient algorithms for robust mdps with non-rectangular uncertainty sets. arXiv preprint arXiv:2305.19004 , 2023.

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 5026-5033. IEEE, 2012.

Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming . John Wiley & Sons, 2014.

Wolfram Wiesemann, Daniel Kuhn, and Berç Rustem. Robust markov decision processes. Mathematics of Operations Research , 38(1):153-183, 2013.

Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. In Machine learning proceedings 1994 , pages 157-163. Elsevier, 1994.

Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applications in continuous control. In International Conference on Machine Learning , pages 6215-6224. PMLR, 2019.

Aurélien Garivier and Eric Moulines. On upper-confidence bound policies for non-stationary bandit problems. arXiv preprint arXiv:0805.3415 , 2008.

Eyal Even-Dar, Sham M Kakade, and Yishay Mansour. Experts in a markov decision process. Advances in neural information processing systems , 17, 2004.

Yasin Abbasi Yadkori, Peter L Bartlett, Varun Kanade, Yevgeny Seldin, and Csaba Szepesvári. Online learning in markov decision processes with adversarially chosen transition probability distributions. Advances in neural information processing systems , 26, 2013.

Erwan Lecarpentier and Emmanuel Rachelson. Non-stationary markov decision processes, a worst-case approach using model-based reinforcement learning. Advances in neural information processing systems , 32, 2019.

Pratik Gajane, Ronald Ortner, and Peter Auer. A sliding-window algorithm for Markov decision processes with arbitrarily changing rewards and transitions. arXiv preprint arXiv:1805.10066 , 2018.

Wang Chi Cheung, David Simchi-Levi, and Ruihao Zhu. Reinforcement learning under drift. arXiv preprint arXiv:1906.02922 , 2019.

Nicholas K Jong and Peter Stone. Bayesian models of nonstationary Markov decision processes. Planning and Learning in A Priori Unknown or Dynamic Domains , page 132, 2005.

Melkior Ornik and Ufuk Topcu. Learning and planning for time-varying mdps using maximum likelihood estimation. arXiv preprint arXiv:1911.12976 , 2019.

Takumi Tanabe, Rei Sato, Kazuto Fukuchi, Jun Sakuma, and Youhei Akimoto. Max-min off-policy actor-critic method focusing on worst-case robustness to model misspecification. In Advances in Neural Information Processing Systems , 2022.

Shihui Li, Yi Wu, Xinyue Cui, Honghua Dong, Fei Fang, and Stuart Russell. Robust multi-agent reinforcement learning via minimax deep deterministic policy gradient. In Proceedings of the AAAI conference on artificial intelligence , volume 33, pages 4213-4220, 2019.

Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. In International Conference on Machine Learning , pages 2817-2826. PMLR, 2017.

Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS) , pages 23-30. IEEE, 2017.

Bhairav Mehta, Manfred Diaz, Florian Golemo, Christopher J Pal, and Liam Paull. Active domain randomization. In Conference on Robot Learning , pages 1162-1176. PMLR, 2020.

Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with a robot hand. arXiv preprint arXiv:1910.07113 , 2019.

Yongyuan Liang, Yanchao Sun, Ruijie Zheng, Xiangyu Liu, Tuomas Sandholm, Furong Huang, and Stephen Marcus McAleer. Game-theoretic robust reinforcement learning handles temporallycoupled perturbations. In The Second Workshop on New Frontiers in Adversarial Machine Learning , 2023.

Eugene Vinitsky, Yuqing Du, Kanaad Parvate, Kathy Jang, Pieter Abbeel, and Alexandre Bayen. Robust reinforcement learning using adversarial populations. arXiv preprint arXiv:2008.01825 , 2020.

Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and ChoJui Hsieh. Robust deep reinforcement learning against adversarial perturbations on state observations. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 21024-21037, 2020.

Huan Zhang, Hongge Chen, Duane S Boning, and Cho-Jui Hsieh. Robust reinforcement learning on state observations with learned optimal adversary. In International Conference on Learning Representations , 2021.

Samuel Stanton, Rasool Fakoor, Jonas Mueller, Andrew Gordon Wilson, and Alex Smola. Robust reinforcement learning for shifting dynamics during deployment. In Workshop on Safe and Robust Control of Uncertain Systems at NeurIPS , 2021.

Mohammed Amin Abdullah, Hang Ren, Haitham Bou Ammar, Vladimir Milenkovic, Rui Luo, Mingtian Zhang, and Jun Wang. Wasserstein robust reinforcement learning. arXiv preprint arXiv:1907.13196 , 2019.

Pierre Clavier, Stéphanie Allassonière, and Erwan Le Pennec. Robust reinforcement learning with distributional risk-averse formulation. arXiv preprint arXiv:2206.06841 , 2022.

Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning , pages 1587-1596. PMLR, 2018.

Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research , 23(274):1-18, 2022.

A Appendix

The Appendix is structured as follow :

In Appendix B, proof for fix point of Oracle-TC algorithm for can be found.

In Appendix C, proof for algorithm Vanilla TC and Stacked-TC can found about robust objective.

In Appendix F, the adversary training was sanity-checked within the time-constrained evaluation.

In Appendix E, all implementation details are provided.

In Appendix H, all raw results are presented.

In Appendix I, the computer resources and training wall clock time are detailed.

In Appendix J, the broader impact and limitations are discussed.

In Appendix K, a heuristic for choosing the radius L

B Proof of Theorem 2.1

Proof. The Proof is similar to [1], using the fact that p ψ + b belongs to the simplex, we get contraction of the operator and convergence to a fix point v ∗ B . Not that to converge to the fix point, there is no need of rectangularity.

Recall the recursion

First we prove that the TC Robust Operator T π B is a contraction. Let V 1 , V 2 ∈ R n . Fix s ∈ S , and assume that T π B V 1 ( s, ψ ) ≥ T π B V 2 ( s, ψ ) . Then fix ϵ > 0 and pick π s.t given s ∈ S ,

First we pick a probability measure p ′ such that p ′ = p ψ + b , b ∈ B , such that

Then it lead to

where last inequality is Holder's inequality between L 1 and L ∞ norms, use probability measure in the simplex such as ∥ p ′ ∥ 1 = 1 . Doing the same thing but in the case where T π B V 1 ( s ) ≤ T π B V 2 ( s ) , it holds

i.e. ∥ T π B V 1 -T π B V 2 ∥ ∞ ≤ γ ∥ V 1 -V 2 ∥ ∞ +2 ϵ . As we can choose ϵ arbitrary small, this establishes that the TC Bellman operator is a γ -contraction. Since T π B is a contraction operator on a Banach space, the Banach fixed point theorem implies that the operator equation T π B V = V has a unique solution V = v π B . A similar proof can be done for optimal operator T ∗ B . The only difference is the maximum operator which is 1 -Lipschitz. So T ∗ B is also a contraction. Then, once proved that operators are γ -contraction, following [1] (Th. 5), we have that the fixed point of this recursion is exactly :

for (optimal) TC Bellman Operator.

C Guaranties for non-stationary Robust MDPS

Recall that we represent a non-stationary robust MDPs (NS-RMDP) as a stochastic sequence, {M = { M i } ∞ t = t 0 , of stationary MDPs M t ∈ M , where M is the set of all stationary MDPs. Each M t is a tuple, ( S , A , P t , r t , γ, ρ 0 ) , where S The set of possible states is denoted by S , the set of actions by A , the discounting factor by γ , the start-state distribution by ρ 0 , and the reward distribution by r t . The reward distribution, denoted by r t : S × A → ∆( R ) , is the probability distribution of rewards. The transition function, represented by P t : S × A → ∆( S ) , is the probability distribution of transitions between states. The symbol ∆ denotes the simplex. For all M t ∈ M , we assume that the state space, action space, discount factor, and initial distribution remain fixed. A policy is represented as a function π : S → ∆( A ) . In general, we will use subscripts t to denote the time evolution during an episode and superscripts k to denote the time step assuming reward or kernel t which is stationary, assuming that the reward function is not changing as it is at time step t stationary. That r k t is the random variables corresponding to the state, action, and reward at time step t for stationary, but iterating with index k .

Definition C.1 ( Lipschitz of sequence of MDPs) . We denote the sequence of kernel and reward function P = { P t } ∞ t = t 0 and R = { r t } ∞ t = t 0 . We define a sequence of MDP is L = ( L r , L P ) -Lipchitz if m = { m t } ∞ t = t 0 ∈ M L with

Assuming that for a time steps the reward function is stationary, we can compute the average return as:

Definition C.2. Non-robust objective function, assuming that G ( π, M t ) = ∑ k ≥ 0 γ k r k t , the return is we assume stationary with reward function r t

with d π the state occupancy measure defined in (16) .

Definition C.3 (Robust (optimal) Return of NS-RMDPs) . Let a return of π for any m t ∈ M t be G ( π, M t ) := ∑ ∞ k =0 γ k r k t with kernel transition P t following π , with ∀ k, t, r k t ∈ [0 , 1] , and the Robust non-stationary expected return with variation of kernel

Let the robust performance of π for episode t be

D Proof Theom 6.2

Proof of Theorem 6.2. First, this difference can be upper bounded in the non robust case as: By definition, we can rewrite non-robust objective function and occupancy measure as.

First, we can decompose the problem into sub-problems such that

using triangular inequality. Looking at differences between two time steps:

where (a) is triangular inequality, (b) is definition of of supremum of reward in the assumptions and reward bounded by 1 . Then, let P π t ∈ R |S|×|S| be the transition matrix ( s ′ in rows and s in columns) resulting due to π and P t , i.e., ∀ t, P π t ( s ′ , s ) := Pr ( S t +1 = s ′ | S t = s, π, M t ) , and let d π ( · , M t ) ∈ R |S| denote the vector of probabilities for each state, then Finally we can easily bound the difference of occupation measure as :

which gives regrouping all terms:

where the stationary MDP M t +1 can be chosen as the minimum over the previous MDPs at time step t such as | Pr( s ′ | s, a, M t ) -Pr( s ′ | s, a, M t +1 ) | ≤ L p . Rewriting previous equation (27), it holds that

Now considering non robust objective :

where first equality is the definition of the robust objective, second equality is decomposition of minimum across time steps and final inequality is simply a property of the min such as | min a -min b | ≤ sup | a -b | .

Finally plugging 28 in (32), it holds that

Combining t times the previous equation gives the result:

E Implementation details

E.1 Algorithm

Algorithm 2 Time-constrained robust training

Input: Time-constrained MDP: ( S, A, Ψ , p ψ , r, L ) , Agent π , Adversary ¯ π

for each interaction time step t do

Note that in Time-constrained robust training Algorithm in section E.1, L Q and L π are as defined by [31] double critics and target network updates are omitted here for clarity

In Table 4, for the stack algorithm, s i is defined as s i ← s i ∪ s i -1 ∪ a i -1 for Stacked-TC , and for the Oracle-TC version, s i ← s i ∪ ψ i .

E.2 Neural network architecture

Weemploy a consistent neural network architecture for both the baseline and our proposed methods for the actor and the critic components. The architecture's design ensures uniformity and comparability across different models.

The critic network is structured with three layers, as depicted in Figure 3a, the critic begins with an input layer that takes the state and action as inputs, which then passes through two fully connected linear layers of 256 units each. The final layer is a single linear unit that outputs a real-valued function, representing the estimated value of the state-action pair.

The actor neural network, shown in Figure 3b, also utilizes a three-layer design. It begins with an input layer that accepts the state as input. This is followed by two linear layers, each consisting of 256 units. The output layer of the actor neural network has a dimensionality equal to the number of dimensions of the action space.

(a) Critic neural network architecture

(b) Actor neural network architecture

Linear 256

Linear 256

Linear 1

R

S

Linear 256

Linear 256

Linear 1

×

→

IR /AI

1

(V)

E.3 M2TD3

We utilized the official M2TD3 [18] implementation provided by the original authors, accessible via the GitHub repository for M2TD3 and Oracle M2TD3.

For the TC-M2TD3 or variants, we implemented the M2TD3 algorithm as specified. To simplify our approach, we omitted the implementation of the multiple ˆ ψ network and the system for resetting ˆ ψ . We replace with an adversary which ¯ π : S × A × Ψ → Ψ which minimize Q ( s, a, ψ ) .

E.4 TD3

We adopted the TD3 implementation from the CleanRL library, as detailed in [32].

F Sanity check on the adversary training in the time-constrained evaluation

A natural question arises regarding the worst time-constrained perturbation. Whether we adequately trained the adversary in the loop, or its suboptimal performance might lead to overestimating the trained agent reward against the worst-case time-constrained perturbation. We monitored the adversary's performance during its training against a fixed agent to address this. The attached figure shows the episodic reward (from the agent's perspective) during the adversary's training over 5 million timesteps, with a perturbation radius of L = 0 . 001 . Each curve is an average of over 10 seeds. The plots show a rapid decline in reward during the initial stages of training, followed by quick stabilization. The episodic reward stabilizes early in the Ant (Figure 4a) environment, indicating

quick convergence. Similarly, in the HalfCheetah (Figure 4b) environment, the reward shows a sharp initial decline and stabilizes, suggesting effective training. For Hopper (Figure 4c), the reward decreases and then levels off, reflecting adversary convergence. Although the reward is more variable in the HumanoidStandup (Figure 4d) environment, it ultimately reaches a steady state, confirming adequate training. Finally, in the Walker environment, the reward pattern demonstrates a quick drop followed by stabilization, indicating convergence. These observations confirm that the adversaries were not undertrained. The rapid convergence to a stable performance across all environments ensures the accuracy of the worst time-constrained perturbations estimated during training.

G Uncertainty set in MuJoCo environments

The experiments of Section 5 follow the evaluation protocol proposed by [18] and based on MuJoCo environments [5]. These environments are designed with a 3D uncertainty sets. Table 7 lists all environments evaluated and their uncertainty sets. The uncertainty sets column defines the ranges of

Ant

8000

Mean episode reward

6000

4000

2000

U

0

1

2

3

4

5

step

1e6

(a) Ant: Episodic reward of the trained agent during adversary training

Hopper

3000

2500

Mean episode reward

2000

1500

1000

500

Q

1

2

3

4

5

step

1e6

(c) Hopper: Episodic reward of the trained agent during adversary training

Walker

6000

5000

Mean episode reward

4000

3000

2000

1000

0

1

2

3

4

5

step

1e6

(e) Walker: Episodic reward of the trained agent during adversary training

HalfCheetah

10000

8000

Mean episode reward

6000

4000

2000

Q

1

2

3

4

5

step

1e6

HumanoidStandup

140000

120000

100000

80000

60000

Q

1

2

3

4

5

step

1e6

DR

RARL

M2TD3

Oracle RARL

TD3

TC-M2TD3

TC-RARL

(f) Legend for algorithm

variation for the parameters within each environment. The reference parameters column indicates the nominal or default values. The uncertainty parameters column describes the physical meaning of each parameter.

H Raw results

Table 8 reports the non-normalized time-constrained (with a radius of L = 0 . 001 ) worst-case scores, averaged across 10 independent runs for each benchmark. Table 9 reports the static worst case score obtained by each agent across a grid of environments, also averaged across 10 independent runs for

each benchmark. Table 10 reports the static average case score obtained by each agent across a grid of environments, also averaged across 10 independent runs for each benchmark.

H.1 Fixed adversary evaluation

At the beginning of each episode, ψ 0 ∼ U (Ψ) is selected for every fixed adversary. The episode length is 1000 steps. To begin with, the random fixed adversary simulates stochastic changes. It selects a parameter ψ t at each timestep within a radius of L = 0 . 1 , which is 100 times larger than in our training methods. This tests the agents' adaptability to unexpected changes. In contrast, the cosine fixed adversary introduces deterministic changes using a cosine function. The radius

Table 11: Avg. performance against time-constrained fixed random adversary with a radius L = 0 . 1 over 10 seeds for each method of L = 0 . 1 scales the frequency of the cosine function, ensuring smooth and periodic variations. Additionally, a phase shift at the start of each episode ensures different starting points. Meanwhile, the linear fixed adversary employs a linear function. The parameters change linearly from the initial value to either one of a vertex of the uncertainty set Ψ over 1000 steps. Furthermore, the exponential fixed adversary uses an exponential function. Parameters change exponentially from the initial value to either of a vertex of the uncertainty set Ψ over 1000 steps. This ensures smooth and predictable variations. Similarly, the logarithmic fixed adversary uses a logarithmic function. Parameters change logarithmically from the initial value to either of a vertex of the uncertainty of the uncertainty set Ψ over 1000 steps, ensuring smooth and predictable variations. Agents trained under the time-constrained framework outperform all baselines in all environments for each fixed adversary, except when compared to the oracle TC method, which has access to ψ . In this case, the stacked-TC or TC methods outperform all baselines in all environments for the cosine, logarithmic, and exponential adversaries and outperform the fixed adversary baseline in 4 out of 5 instances for the random and linear fixed adversaries.

H.2 Agents training curve

We conducted training for each agent over a duration of 5 million steps, closely monitoring the cumulative rewards obtained over a trajectory spanning 1,000 steps. To enhance the reliability of our results, we averaged the performance curves across 10 different seeds. The graphs in Figures 5 to 15 illustrate how different training methods, including Domain Randomization, M2TD3, RARL, Oracle RARL ,Oracle M2TD3, TC RARL, TC M2TD3, Stacked TC RARL and Stacked TC M2TD3, impact agent performance across various environments.

I Computer ressources

All experiments were run on a desktop machine (Intel i9, 10th generation processor, 64GB RAM) with a single NVIDIA RTX 4090 GPU. Averages and standard deviations were computed from 10 independent repetitions of each experiment.

J Broader impact

This paper aims to advance robust reinforcement learning. It addresses general mathematical and computational challenges. These challenges may have societal and technological impacts, but we do not find it necessary to highlight them here.

J.1 Limitations

While our proposed Time-Constrained Robust Markov Decision Process (TC-RMDP) framework significantly advances robust reinforcement learning by addressing multifactorial, correlated, and time-dependent disturbances, several limitations must be acknowledged. The TC-RMDP framework assumes that the parameter vector ψ that governs environmental disturbances is known during training. In real-world applications, obtaining such detailed information may not always be feasible. This reliance on precise parameter knowledge limits the practical deployment of our algorithms in environments where ψ cannot be accurately measured or inferred. Our approach assumes that the environment's dynamics can be accurately parameterized and that these parameters remain within a predefined uncertainty set Ψ . This assumption might not hold in more complex or highly dynamic environments where disturbances are not easily parameterized or when the uncertainty set Ψ cannot comprehensively capture all possible variations. Consequently, the robustness of the learned policies might degrade when facing disturbances outside the considered parameter space. Addressing these limitations in future work.

Ant

8000

6000

episodes_reward

4000

2000

0

0

1

2

3

5

1e6

step

Hopper

2000

1500

episodes_reward

1000

500

0

0

1

2

3

step

1e6

HalfCheetah

6000

5000

episodes_reward

4000

3000

2000

1000

0

0

1

2

3

5

1e6

step

HumanoidStandup

120000

episodes_rewara

100000

80000

60000

40000

0

1

2

m

5

step

1e6

(c) Training curve on Hopper with Domain Randomization

(d) Training curve on HumanoidStandup with Domain Randomization

Walker

5000

4000

episodes_reward

3000

2000

1000

0

0

1

2

3

4

5

step

1e6

(e) Training curve on Walker with Domain Randomization

K A heuristic for choosing the radius L

We used a simple heuristic to select the radius L in our experiments. First, we normalized the parameters of the uncertainty set, scaling them to lie within the range [0 , 1] for each dimension. We then divided this range by the number of steps in an episode, denoted as ep , yielding L = 1 ep . Our experiments, with ep = 1000 , resulted in a radius of L = 0 . 001 . This choice ensures that any initial parameter ψ 0 can move to any point within the uncertainty set's parameters for an episode.

(a) Training curve on Ant with M2TD3

(b) Training curve on HalfCheetah with M2TD3

(c) Training curve on Hopper with M2TD3

(d)

Training

curve

on HumanoidStandup with

M2TD3

(e) Training curve on Walker with M2TD3

Ant

HalfCheetah

6000

6000

5000

5000

4000

4000

isodes_reward

3000

3000

2000

1000

1000

0

2

3

step

1e6

step

1e6

HumanoidStandup

Hopper

120000

2500

100000

2000

episodes_reward

80000

1500

1000

60000

500

40000

step

1e6

0

2

3

D

step

1e6

Walker

5000

4000

0000

2000

1000

D

step

1e6

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We propose three distinct time-constrained robust mdp algorithm algorithms, each using varying levels of environmental information, and evaluate them extensively on continuous control benchmarks. Our results demonstrate that these algorithms yield an efficient tradeoff between performance and robustness, outperforming traditional deep robust RL methods in time-constrained environments while preserving robustness in classical benchmarks.

Guidelines:

(a) Training curve on Ant with RARL

(b) Training curve on HalfCheetah with RARL

(c) Training curve on Hopper with RARL

Ant

HalfCheetah

3000

2500

-1000

episodes_reward

2000

episodes_reward

-2000

1500

-3000

1000

-4000

500

-5000

0

-6000

0

2

3

4

0

1

5

step

1e6

step

1e6

HumanoidStandup

Hopper

120000

1000

110000

100000

800

pisodes_rewara

90000

episodes_reward

600

80000

70000

400

60000

200

50000

40000

step

1e6

0

2

3

step

1e6

Walker

4000

0OSE

00OE

2500

2000-

1500 -

1000-

500

0

5

step

1e6

(e) Training curve on Walker with RARL

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

(a) Training curve on Ant with TD3

(b) Training curve on HalfCheetah with TD3

(c) Training curve on Hopper with TD3

(d) Training curve on HumanoidStandup with TD3

Ant

HalfCheetah

14000

6000

12000

5000

10000

episodes_reward

episodes_reward

4000

8000

3000

6000

2000

4000

1000

2000

0

1

2

3

4

5

step

1e6

step

1e6

Hopper

HumanoidStandup

3500

110000

3000

100000

reward

2500

episodes_rewara

90000

2000

80000

1500

70000

60000

1000

50000

500

40000

0

step

1e6

step

1e6

Walker

5000

4000

_reward

3000

isodes_

2000

1000-

2

3

5

step

1e6

(e) Training curve on Walker with TD3

Justification: We have a section in the appendix discussing on the limitations of our work. Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

HalfCheetah

(c) Training curve on Hopper with Oracle RARL

(e) Training curve on Walker with Oracle RARL

Ant

3500

-1000

3000

episodes_reward

2000

reward

2500

2000

-3000

1500

-4000

1000

500

-5000

0

5

step

1e6

step

1e6

HumanoidStandup

Hopper

120000

110000

1000

100000

800

90000

episodes_reward

80000

600

70000

400

60000

50000

200

40000

0

step

1e6

step

1e6

Walker

4000

3500

3000

2500

2000-

1500

1000

500

3

5

step

1e6

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best

HalfCheetah

(a) Training curve on Ant with Oracle M2TD3

(c) Training curve on Hopper with Oracle M2TD3

(d) Training curve on HumanoidStandup with Ora- cle M2TD3

Ant

8000

7000

6000

6000

_reward

5000

episodes_reward

4000

episodes_

4000

3000

2000

2000

1000

0

5

step

1e6

step

1e6

HumanoidStandup

Hopper

140000

3500

120000

3000

episodes_reward

2500

100000

_reward

2000

80000

episodes_r

1500

60000

1000

40000

500

step

1e6

1

2

3

4

step

1e6

Walker

6000

5000

4000

3000

2000

1000

0

3

step

1e6

judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Hypotheis of proof are well stated in Th.6.2 and 2.1

Guidelines:

The answer NA means that the paper does not include theoretical results.

HalfCheetah

(e) Training curve on Walker with Oracle M2TD3

(c) Training curve on Hopper with Oracle M2TD3

Ant

8000

7000

6000

6000

_reward

5000

episodes_reward

4000

episodes_

4000

3000

2000

2000

1000

0

5

step

1e6

step

1e6

HumanoidStandup

Hopper

140000

3500

120000

3000

episodes_reward

2500

100000

_reward

2000

80000

episodes_r

1500

60000

1000

40000

500

step

1e6

1

2

3

4

step

1e6

Walker

6000

5000

4000

3000

2000

1000

0

3

step

1e6

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

(a) Training curve on Ant with Oracle TC-RARL

(c) Training curve on Hopper with Oracle TC-

RARL

(d) Training curve on HumanoidStandup with Ora- cle TC-RARL

(e) Training curve on Walker with Oracle TC-

Ant

10000

8000

8000

episodes_reward

6000

6000

episodes_reward

4000

4000

2000

2000

5

step

1e6

step

1e6

Hopper

HumanoidStandup

3500

120000

3000

2500

reward

episodes_reward

100000

2000

1500

80000

1000

60000

500

40000

0

0

2

3

5

0

2

5

step

1e6

step

1e6

Walker

6000

5000

episodes_reward

4000

3000

2000

1000

0

0

1

2

3

4

5

step

1e6

RARL

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide the algorithm, all detail implementations, hyperparameters.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

HalfCheetah

(c) Training curve on Hopper with Oracle TC-

M2TD3

(e) Training curve on Walker with Oracle TC-

Ant

10000

8000

8000

episodes_reward

6000

episodes_reward

6000

4000

4000

2000-

2000

0

5

step

1e6

step

1e6

Hopper

HumanoidStandup

OOSE

0000

120000

2500

Diemarsanosida

100000

2000

1500

80000

1000

60000

500

40000

0

0

2

3

0

2

5

step

1e6

step

1e6

Walker

5000

4000

episodes_reward

3000

2000

1000

0

0

1

2

3

4

5

step

1e6

M2TD3

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

HalfCheetah

Ant

8000

7000

6000

episodes_reward

5000

4000

3000

2000

1000

0

0

1

2

3

5

1e6

step

Hopper

3500

3000

2500

episodes_reward

2000

1500

1000

500

0

0

2

3

step

1e6

HalfCheetah

10000

8000

episodes_reward

6000

4000

2000

0

0

1

2

3

5

1e6

step

HumanoidStandup

110000

A

100000

90000

episodes_rewara

80000

70000

60000

50000

40000

0

1

2

3

5

step

1e6

Walker

6000

5000

episodes_reward

4000

3000

2000

1000

0

0

1

2

3

4

5

step

1e6

(e) Training curve on Walker with Stacked TC-

M2TD3

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

Ant

0008

7000

6000

episodes_reward

5000

4000

3000

2000

1000

1e6

step

Hopper

2500

2000

episodes_reward

1500

1000

500

0

3

1e6

HalfCheetah

7000

6000

5000

4000

3000

2000

1000

0

5

1e6

step

HumanoidStandup

120000

100000

episodes_rewara

80000

60000

40000

2

5

1e6

step step

Walker

5000

4000

episodes_reward

3000

2000

1000

0

0

1

2

3

4

5

step

1e6

(e) Training curve on Walker with Stacked TC-

RARL

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide the code with clear instruction, to reproduce the experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provides all hyperparameters, optimizer and reinforcement learning algorithm. All those details is provided in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We profide our computers resources to runs all experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have a section in the appendix discussing of the computer resources for our work.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our contribution aims to safer RL algorithms, and we conducted our experiment of open source benchmarks.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss on the broader impact in the conclusion and the appendix

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The authors cite the original paper that produced the code package or dataset.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a novel concept, time-constrained robust MDP, to address the conservativeness issue of the rectangular assumption. They assume the transition depends on an underlying parameter, and the parameter can be adversarially chosen from an uncertainty set. Several algorithms are proposed to solve the time-constrained robust MDP. Extensive numerical experiments show the effectiveness of the proposed framework and algorithms.\n\nSTRENGTHS:\nGiven that the field of robust MDP has limited large-scale experiments due to the intractability of the rectangular assumption, the idea in this paper is very nice. I agree that the rectangular assumption can be too general and lead to conservative policies, due to its inefficiency in leveraging the inherent structure of the uncertainty set. The proposed time constraint robust MDP is interesting, especially since the experiment results seem to be promising.\n\nWEAKNESSES:\nMy concern is that the methods proposed in this paper might be too heuristic, given the non-stationarity and possibly non-existence of the optimal policy. As I am not very familiar with the experiment side, I am not sure if the evaluation in this paper is sufficient. See my questions for details.\n\nQUESTIONS:\n1. In the experiments, the oracle-TC is not always the best method. It seems that the performances are not that stable. Also, the vanilla TC outperforms in almost half cases. It is kind of anti-intuitive.\n\n2. Compared to baseline methods, the TC-based methods show robustness to some extent, but how robust are they? What are the expected best (robust) performances?\n\n3. In algorithm 1, what is \"sample a mini-batch of transitions\" for? How are the agent and adversary updated?\n\n4. On lines 174-175, the authors claim that the rectangularity assumptions are rarely met in real-world scenarios. I think more evidence or references are required to support this claim.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-07"}

comments: SUMMARY:
This paper aims to develop a new framework of robust RL addressing the over-conservability issue of rectangularity and dynamic uncertainty set assumption. This framework allows for time-dependent, correlated and multifactorial disturbance to the dynamics. Three distinct algorithms are developed depending on the available information to the policy. Extensive numerical experiments are further provided to demonstrate the performance of the proposed algorithms.

STRENGTHS:
- The proposed framework addresses the major drawbacks of existing robust RL approach: overly conservative due to the use of dynamic and rectangular uncerainty sets.
- Extensive experiments are provided to demonstrate the performance of the three algorithms
- The approaches can be used with existing robust value iteration approaches.

WEAKNESSES:
- Thm 2.1 only applies to the case where the policy has the exact information of \psi, which in practice is usually unavailable. 
- The usefulness of this framework remains questionable, as it poses a significant challenge of constructing such complex uncertainty set with time and state-action correlation.
- The training algorithm has the same flavor of the adversarial training as those in the literature, e.g., RARL. The novelty seems rather limited as this is mostly an  experimental paper.

QUESTIONS:
See weaknesses above.

LIMITATIONS:
na
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Time-Constrained Robust MDPs (TC-RMDPs) as a novel formulation to address the overly conservative nature of traditional robust RL under sa-rectangularity assumptions. The authors propose three algorithms to handle time-dependent and correlated disturbances in different situations. Extensive evaluations on continuous control benchmarks show that these algorithms outperform traditional robust RL methods in terms of balancing performance and robustness.\n\nSTRENGTHS:\n1. The introduction of TC-RMDPs addresses a significant limitation of overly pessimistic in traditional robust RL, making the approach more applicable to real-world scenarios with time-dependent disturbances, which is interesting and novel. In addition, the paper presents three distinct algorithms with varying levels of information usage, expanding its usage. \n\n2. The authors provide formal theoretical guarantees for the proposed algorithms, which is appreciated.\n\n3. Extensive experiments on continuous control benchmarks demonstrate the efficacy of the proposed methods.\n\nWEAKNESSES:\n# Major\n1. The complexity of the proposed algorithms, especially Oracle-TC, may limit their practical applicability in real-world scenarios where complete environmental information is unavailable.\n\n2. How to select the Lipschitz constant $L$ in practical scenarios? Some ablations studies and analysis regarding $L$ would be beneficial.\n\n3. It would be better if the authors could provide some visualizations of the difference between the traditional independent transitions with the time-constrained parametric transitions.\n\n4. What is the number of worst-case episodes in the experiments? If it is 1, maybe it would be better to display the results as the average of the worst 10% of episodes for better robustness. This is because having only 1 episode might introduce stochasticity.\n\n# Minor\n1. It is suggested to show the equation number for better illustration and reference.\n2. There are some typos in the paper. For instance, in line 102, $B=\\mathcal{B}(0_{\\Psi},L)$ should be corrected to $B=\\mathcal{B}({\\Psi}_0,L)$,  the right-most column in Table 1 should be Avg. The authors should carefully review the paper for such errors.\n\nQUESTIONS:\nplease see above\n\nLIMITATIONS:\nThe framework assumes that the transition parameter vector $\\psi$ is known during training. However, in real-world applications, it could be challenging. Also, the assumption on the uncertainty set $\\Psi$ is also not practical in practice.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper defines a robust training method for MDPs under uncertainty in the environment dynamics. Current methods assume sa-rectangularity, where the transition dynamics from consecutive states are independent of one another. The authors argue that this assumption is unrealistic and leads to overly conservative policies. Alternatively, this paper breaks the independence assumption in the environment dynamics to use a time-step evolution approach. This paper presents a Time-Constrained TC-MDP where the Time-Constrained Parameterized MDP kernels evolve each time-step constrained to be Lipschitz continuous. The model is trained in an adversarial manner, where the adversarial policy is learned to evolve the model dynamics but is limited to be L-close to the previous dynamics by definition. This learning method is tested under different variant conditions (cosine, exponential, linear, and logarithmic) with positive results. Robustly trained agents that learn under the worst TC adversities outperform vanilla robust algorithms that assume time independence in the model dynamics.\n\nSTRENGTHS:\n- The evaluation is fair. The model trained under worst-case time-constrained adversarial dynamics is then evaluated under different fixed evolving dynamics (cosine, exponential, linear, and logarithmic) that range in stochastic changes much larger than the used in learning (L=0.1 in evaluation vs L=0.001 in training). \n- The definition of TC-RMDP is smartly selected to preserve stationarity, defining the state as the tuple ($S \\times \\Psi$) \n- The TC strategy can be straightforwardly implemented in previous algorithms by limiting the search space of the adversarial $psi$ at every time-step.\n\nWEAKNESSES:\n- Reporting that the worst-case TC variants outperform vanilla adversarial models as M2TD3 or RARL is expected. As the decisions of the adversary are rather limited when compared to the vanilla algorithms. Perhaps highlighting the evaluation results (fixed or static setting) in the conclusions would help in emphasizing this.\n\nQUESTIONS:\n- In Algorithm 1, should is be: $b_{t+1} = \\bar{\\pi}(s_t, a_t, \\psi_t)$ ?\n- Typo in Page 7 Line 276: \"Appendix G and G\"\n\nLIMITATIONS:\nAs the authors mention, this method assumes that the environment dynamics can be correctly parametrized in a confident set $Psi$. \n\nAdditionally, selecting the radius L of the proximal adversarial set of actions is not trivial. Fortunately, the authors tested the environment with a large enough L (100 times larger than the one used during training) with positive results, which validates their claim.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}