Adversarially Robust Decision Transformer

Authors: xiaohang  tang
afonso  marques
parameswaran  kamalaruban
ilija  bogunovic
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 22:09:15

Abstract: Decision Transformer (DT), as one of the representative Reinforcement Learning via Supervised Learning (RvS) methods, has achieved strong performance in offline learning tasks by leveraging the powerful Transformer architecture for sequential decision-making. However, in adversarial environments, these methods can be non-robust, since the return is dependent on the strategies of both the decision-maker and adversary. Training a probabilistic model conditioned on observed return to predict action can fail to generalize, as the trajectories that achieve a return in the dataset might have done so due to a suboptimal behavior adversary. To address this, we propose a worst-case-aware RvS algorithm, the Adversarially Robust Decision Transformer (ARDT), which learns and conditions the policy on in-sample minimax returns-to-go. 
ARDT aligns the target return with the worst-case return learned through minimax expectile regression, thereby enhancing robustness against powerful test-time adversaries. In experiments conducted on sequential games with full data coverage, ARDT can generate a maximin (Nash Equilibrium) strategy, the solution with the largest adversarial robustness. In large-scale sequential games and continuous adversarial RL environments with partial data coverage, ARDT demonstrates significantly superior robustness to powerful test-time adversaries and attains higher worst-case returns compared to contemporary DT methods.

Main paper: 
 introduction: 
 Reinforcement Learning via Supervised Learning (RvS) [8] , has garnered attention within the domain of offline Reinforcement Learning (RL). Framing RL as a problem of outcome-conditioned sequence modeling, these methodologies have showcased strong performance in offline benchmark tasks [50, 53, 13, 37, 17] . One of the representative RvS methods is Decision Transformer (DT) [2] , which simply trains a policy conditioned on target return via behavior cloning loss. Given the apparent efficacy and simplicity of RvS, our attention is drawn to exploring its performance in adversarial settings. In this paper, we aim to use RvS to achieve adversarial robustness, a critical capability for RL agents to manage environmental variations and adversarial disturbances [34, 42, 14, 46, 5, 36] . Such challenges are prevalent in real-world scenarios, e.g., changes in road conditions and decision-making among multiple autonomous vehicles.In offline RL with adversarial actions, the naive training of action-prediction models conditioned on the history and observed returns such as DT, or expected returns such as ESPER and DoC [32, 54, 53] , which we refer to as Expected Return-Conditioned DT (ERC-DT), may result in a non-robust policy. Figure 1 : LHS presents the game where decision-maker P is confronted by adversary A. In the worst-case scenario, if P chooses action a 0 , A will respond with ā0 , and if P chooses a 1 , A will counter with ā4 . Consequently, the worst-case returns for actions a 0 and a 1 are 0 and 1, respectively. Therefore, the robust choice of action for the decision-maker is a 1 . RHS displays tables of action probabilities and the worst-case returns for the Decision Transformer (DT), Expected Return-Conditioned DT (ERC-DT) methods and our algorithm, when conditioned on the largest return-to-go 6. After training using uniformly collected data that covered all possible trajectories, the results reveal that DT fails to select the robust action a 1 , whereas our algorithm manages to do so.This issue arises from the potential distributional shifts in the policy of the adversary during offline learning. Specifically, if the behavior policies used for data collection are suboptimal, a well-trained conditional policy overfitted to the training dataset may fail to achieve the desired target return during testing, if the adversary's policy at test time has changed or become more effective.We illustrate this using a sequential game with data collected by a uniform random policy with full coverage shown in Figure 1 . The standard RvS methods (DT and ERC-DT) when conditioned on high target return, show poor worst-case performance. DT's failure stems from training data containing the sequences (a 0 , ā2 ) and (a 1 , ā3 ), inaccurately indicating both actions a 0 and a 1 as optimal. ERC-DT missteps because the adversary's uniform random policy skews the expected return in favor of a 0 over a 1 , wrongly favoring a 0 . A robust strategy, however, exclusively selects a 1 against an optimal adversary to guarantee a high reward. Moreover, in more complex multi-decision scenarios, the adversarial strategy should not only minimize but also counterbalance with maximized returns by the decisionmaker at the subsequent state, aligning with a minimax return approach for robust policy development.Main Contributions. Consequently, there is a need for data curation to indicate the potential worstcase returns associated with each action in the Decision Transformer (DT), improving the robustness against adversarial actions. Building on this concept, we introduce the first robust training algorithm for Decision Transformer, the Adversarially Robust Decision Transformer (ARDT). To the best of our knowledge, this paper represents the first exploration of the robustness of RvS methods from an adversarial perspective. In ARDT, we leverage Expectile Regression [29, 1] to transform the original returns-to-go to in-sample minimax returns-to-go, and train DT with the relabeled returns. In the experiments, we illustrate the robustness of ARDT in three settings (i) Games with full data coverage (ii) A long-horizon discrete game with partial data coverage (iii) Realistic continuous adversarial RL problems. We provide evidence showcasing the robustness of ARDT to more powerful adversary compared to the behavior policy and the distributional shifts of the adversary's policy in the test time. Furthermore, in the adversarial MuJoCo tasks, namely Noisy Action Robust MDP [42] , ARDT exhibits better robustness to a population of adversarial perturbations compared to existing DT methods.Related work. Reinforcement Learning via Supervised Learning is a promising direction for solving general offline RL problems by converting the RL problem into the outcome-conditioned prediction problem [17, 37, 10, 31, 8] . Decision Transformer (DT) is one of the RvS instance [2, 15] , leveraging the strength of the sequence model Transformer [45] . The variants of DT have been extensively investigated to address the trajectory stitching [50, 53] , stochasticity [31, 30, 32, 54] , goal-conditioned problems [9] , and generalization in different environments [26, 51, 21, 52] .For different problems, it is critical to select an appropriate target as the condition of the policy. Q-Learning DT [53] relabels the trajectories with the estimated Q-values learned to achieve trajectory stitching [18] . ESPER [32] , aims at solving the environmental stochasticity via relabeling the trajectories with expected returns-to-go. Although our method also transform the returns-to-go to estimated values, we instead associate worst-case returns-to-go with trajectories, aiming at improving its robustness against adversarial perturbations.Adversarial RL can be framed as a sequential game; with online game-solving explored extensively in literature [19, 11, 40, 24, 20, 39, 38, 41] . In the offline setting, where there is no interaction with the environment, solving games is challenging due to distributional shifts in the adversary's policy, necessitating adequate data coverage. Tabular methods, such as Nash Q-Learning-a decades-old approach for decision-making in competitive games [12] -and other similar methods [4, 56] address these challenges through pessimistic out-of-sample estimation. While value-based methods are viable for addressing adversarial problems, our focus is on enhancing the robustness of RvS methods in these settings. Note that while some offline multi-agent RL methods, including DT-based approaches, are designed for cooperative games [26, 48, 44, 47] , our research is oriented towards competitive games. 
problem setup: 
 We consider the adversarial reinforcement learning problem involving a protagonist and an adversary, as described by Pinto et al. [34] . This setting is rooted in the two-player Zero-Sum Markov Game framework [23, 33] . This game is formally defined by the tuple (S, A, Ā, T, R, p 0 ). Here, S represents the state space, where A and Ā denote the action spaces for the protagonist and adversary, respectively. The reward function R : S × A × Ā → R and the transition kernel T : S × A × Ā → ∆(S) depend on the state and the joint actions of both players. The initial state distribution is denoted by p 0 . At each time-step t ≤ H, both players observe the state s t ∈ S and take actions a t ∈ A and āt ∈ Ā. The adversary is considered adaptive if its action āt depends on s t and a t . The protagonist receives a reward r t = R(s t , a t , āt ), while the adversary receives a negative reward -r t .Denote trajectory τ = (s t , a t , āt , r t ) H t=0 and its sub-trajectory τ i:j = (s t , a t , āt , r t ) j t=i for 0 ≤ i ≤ j ≤ H. The return-to-go refers to the sum of observed rewards from state s t onwards: R(τ t:H ) = H t ′ =t r t ′ . The protagonist policy π and the adversary policy π can be history-dependent, mapping history (τ 0:t-1 , s t ) to the distribution over protagonist and adversary's actions, respectively.In offline RL, direct interaction with the environment is unavailable. Instead, we rely on an offline dataset D, which consists of trajectories generated by executing a pair of behavioral policies, (π D , πD ). Our objective in this adversarial offline RL setting is to utilize this dataset D to learn a protagonist policy that seeks to maximize the return, while being counteracted by an adversary's policy π:EQUATIONwhereρ π,π (τ ) = p 0 (s 0 ) • t π(a t | τ 0:t-1 , s t ) • π(ā t | τ 0:t-1 , s t , a t ) • T (s t+1 | s t , a t , āt ).The maximin solution to this problem and its corresponding optimal adversary are denoted as π * and π * , respectively.We extend the RvS framework by incorporating adversarial settings within the Hindsight Information Matching (HIM) paradigm [9] . In adversarial HIM, the protagonist receives a goal (denoted by z) and acts within an adversarial environment. The protagonist's performance is evaluated based on how well it achieves the presented goal. Mathematically, the protagonist aims to minimize a distance metric D(•, •) between the target goal z and the information function I(•) applied to a trajectory τ . This sub-trajectory follows a distribution ρ dependent on both the protagonist's policy π z = π(a t | τ 0:t-1 , s t , z) and the test-time adversarial policy πtest . Formally, the information distance is minimized as follows:EQUATIONwhere ρ πz,πtest represents the trajectory distribution obtained by executing rollouts in a Markov Game with the protagonist's policy π z and the test-time adversarial policy πtest . Under HIM framework, Decision Transformer [2] uses adopt return-to-go value as the information function, while ESPER [32] employs the expected return-to-go. The well-trained RvS policy, including DT and ESPER, is derived from behavior trajectories filtered based on the condition I(τ ) = z:EQUATIONIf the conditional policy ensures that the information distance in Eq. ( 2) is minimized to 0, the generated policy π z is robust against πtest by simply conditioning on a large target return, or even the maximin optimal protagonist π * = lim z→+∞ π z if the test-time adversary is optimal. Consequently, our objective shifts from learning a maximin policy in Eq. ( 1) to minimizing the distance in Eq. ( 2).For a given target goal z, the trajectories where the information function I(τ ) = z can be considered as optimal demonstrations of achieving that goal, i.e. information distance minimization, D(I(τ ), z) = 0. However, in adversarial environments, this information distance is hard to be minimized to 0. In adversarial offline RL, a trajectory in the dataset might achieve a goal due to the presence of a weak behavior adversary, rather than through the protagonist's effective actions.Filtering datasets for trajectories with high return-to-go could select the ones from encounters with suboptimal adversaries. Consequently, an agent trained through behavioral cloning on these trajectories is not guaranteed to achieve a high return when facing a different and powerful adversary at test time. We show this formally through a theorem: Theorem 1. Let π D and πD be the data collecting policies used to gather data for training an RvS protagonist policy π. Assume T (st+1 | s t , a t , āt ) = ρ π D ,π D (s t+1 | τ 0:t-1 , s t , a t , āt , I(τ ) = z) 2 ,for any goal z such that ρ π D ,π D (I(τ ) = z | s 0 ) > 0, the information distance can be minimized:E τ ∼ρ πz ,π test [D(I(τ t ), z)] = 0 if πtest (ā t | τ 0:t-1 , s t , a t ) = ρ π D ,π D (ā t | τ 0:t-1 , s t , a t , I(τ ) = z).Theorem 1 suggests that the protagonist policy π z , defined in Eq. ( 3), can minimize the information distance when the test-time adversarial policy is πtest = ρ π D ,π D (ā t | τ 0:t-1 , s t , a t , I(τ ) = z). In other words, π z , conditioned on a large target return, is guaranteed to be robust against ρ π D ,π D (ā t | τ 0:t-1 , s t , a t , I(τ ) = z). Consequently, the policies of DT and ESPER are only guaranteed to perform well against behavior adversarial policies and will be vulnerable to powerful test-time adversaries since their information function is dependent on behavior adversary. This is a concern because, in practice, offline RL datasets are often collected in settings with weak or suboptimal adversaries.In this context, to enhance robustness, we consider using an information function I(τ ) to train the protagonist policy π z by simulating the worst-case scenarios we might encounter during test time. A natural choice for I(τ ) is the minimax return-to-go, which we formally introduce in Section 3 and use as a central component of our method. With this I(τ ) and sufficient data coverage, ρ π D ,π D (ā t | τ 0:t-1 , s t , a t , I(τ ) = z) can be approximated to the optimal adversarial policy, implying that π z is robust against the optimal adversary. Furthermore, even without sufficient data coverage, π z remains more robust compared to previous DT methods, which tend to overfit to the behavior adversarial policy. 
adversarially robust decision transformer: 
 To tackle the problem of suboptimal adversarial behavior in the dataset, we propose Adversarially Robust Decision Transformer (ARDT), a worst-case-aware RvS algorithm. In ARDT, the information function is defined as the expected minimax return-to-go:EQUATIONIntuitively, this information function represents the expected return when an adversary aims to minimize the value, while the protagonist subsequently seeks to maximize it. To train a protagonist policy as described in Eq. (3), ARDT first relabels the trajectory with the minimax returns-to-go in Eq. (4). Secondly, the protagonist policy is then trained similarly to a standard Decision Transformer (DT). This two-step overview of ARDT is provided in Figure 2 , where the left block includes the minimax return estimation with expectile regression for trajectory relabeling, and the right block represents the DT training conditioned on the relabeled returns-to-go. We relabel the trajectories in our dataset with the worst-case returns-to-go by considering the optimal in-sample adversarial actions. In practice, we approximate the proposed information function using an (in-sample) minimax returns-to-go predictor, denoted by Q. We simplify the notation of the appearance of actions in the dataset a t ∈ A : π D (a t |τ 0:t-1 , s t ) > 0 and āt ∈ Ā : πD (ā t |τ 0:t-1 , s t , a t ) > 0 to a t ∈ D and āt ∈ D, respectively. The optimal predictor Q * is a history-dependent state-action value function satisfying the following condition that at any time t = 1, • • • , T :EQUATIONWe adopt Expectile Regression (ER) [29, 1] to approximate Q. This choice is particularly advantageous for RL applications because it helps avoid out-of-sample estimation or the need to query actions to approximate the minimum or maximum return [16] . Specifically, the loss function for ER is a weighted mean squared error (MSE):EQUATION)Suppose a random variable y follows a conditional distribution y ∼ ρ(•|x), function g α (x) := arg min g(x) L α ER (g(x) -h(x, y)) can serve as an approximate minimum or maximum operator for the function h(x, y) over all possible value of y, i.e.,EQUATIONOur algorithm uses coupled losses for maximum and minimum operators to approximate the in-sample minimax returns-to-go. Formally, given an offline dataset D, we alternately update the minimax and maximin returns-to-go estimators: Q ν (s, a) and Q ω (s, a, ā). In each iteration, we fix ω (or ν), denoted as ω (or ν), and then update ν (or ω) based on the following loss functions:EQUATIONEQUATIONThe minimizers (ν * and ω * ) of the two losses mentioned above (with α → 0), combined with Eq. ( 7), satisfy the following condition for all s t , a t , āt ∈ D:EQUATIONEQUATIONBy substituting Eq. ( 11) into Eq. ( 10), the minimax returns-to-go predictor Q ν * satisfies Eq. ( 5).The formal algorithm of ARDT is presented in Algorithm 1, where from line 3-6 is the minimax return estimation diagrammed as the left block in Figure 2 , and line 7-10 is the Transformer training on Algorithm 1 Adversarially Robust Decision Transformer (ARDT)1: Inputs and hyperparameters: offline dataset D containing collection of trajectories; α = 0.01 2: Initialization: Initialize the parameters ω and ν for in-sample returns-to-go networks Q ω and Q ν with the original returns-to-go, and parameter θ of the protagonist policy π θ . // expectile regression updates for in-sample returns-to-go networks 3:for iteration k = 0, 1, • • • do 4:for trajectory τ ∈ D do // update ω while keeping ν fixed 5:ω ← ω -∇ ω ℓ 1-α (ω), where ℓ 1-α (ω) is given in (9) . // update ν while keeping ω fixed 6:ν ← ν -∇ ν ℓ α (ν), where ℓ α (ν) is given in (8) . // relabeling dataset D using in-sample returns-to-go7: Create D worst = τ = (• • • , R t-1 , s t-1 , a t-1 , R t , s t , a t , • • • ) | τ ∈ D, R t = Q ν (s t , a t ) // protagonist policy update 8: L(θ) = -τ ∈Dworst t log π θ (a t | τ 0:t-1 , s t , R t ) 9: for iteration k = 0, 1, • • • do 10: θ ← θ -∇ θ L(θ) 11: Output: Policy π θthe right of Figure 2 . Initially, we warm up two returns-to-go networks with the original returns-to-go values from the dataset. This step ensures the value function converges to the expected values, facilitating faster maximin value estimation and guaranteeing accurate value function approximation at terminal states. Subsequently, we estimate the minimax returns-to-go by alternately updating the parameters of the two networks based on the expectile regression losses in Eq. ( 9) and Eq. ( 8). Once the minimax expectile regression converges, we replace the original returns-to-go values in the trajectories with the values predicted by Q ν . Finally, we train the Decision Transformer (DT) conditioned on the states and relabeled returns-to-go to predict the protagonist's actions. Then the ARDT protagonist policy is ready to be deployed for evaluation in the adversarial environment directly. 
experiments: 
 In this section, we conduct experiments to examine the robustness of our algorithm, Adversarially Robust Decision Transformer (ARDT), in three settings: (i) Short-horizon sequential games, where the offline dataset has full coverage and the test-time adversary is optimal (Section 4.1), (ii) A long-horizon sequential game, Connect Four, where the offline dataset has only partial coverage and the distributional-shifted test-time adversary (Section 4.2), and (iii) The standard continuous Mujoco tasks in the adversarial setting and a population of test-time adversaries (Section 4.3). We compare our algorithm with baselines including Decision Transformer and ESPER. The implementation details are in Appendix C. Notably, since the rewards and states tokens encompass sufficient information about the adversary's actions, all DT models are implemented not conditioned on the past adversarial tokens to reduce the computational cost.The efficacy of our solution is first evaluated on three short-horizon sequential games with adaptive adversary: (1) a single-stage game, (2) an adversarial, single-stage bandit environment depicting a Gambling round [32] , and (3) a multi-stage game. These are depicted in Figure 1 , Figure 7 , and Figure 9 , respectively. The collected data consists of 10 5 trajectories, encompassing all possible trajectories. The online policies for data collection employed by the protagonist and adversary were both uniformly random. Figure 3 illustrates the efficacy of ARDT in comparison to vanilla DT and ESPER. Across all three environments, ARDT achieves the return of maximin (Nash Equilibrium) against the optimal adversary, when conditioned on a large target return. As illustrated in Figure 1 , ARDT is aware of the worst-case return associated with each action it can take due to learning to condition on the relabeled Protagonist: 50%Figure 4 : Average return of ARDT and vanilla DT on Connect Four when trained on suboptimal datasets collected with different levels of optimality for both the online protagonist's policy (30%, 40% and 50% optimal) and the adversary's policy (10%, 30%, 50% optimal), over 10 seeds. We test against a fixed adversary that acts optimally 50% of the time, and randomly otherwise.minimax returns-to-go, and successfully takes robust action a 1 . In single-stage game (Figure 3 ), ESPER is vulnerable to the adversarial perturbations, and DT fails to generate robust policy when conditioned on large target return. While, it is worth noting that DT has achieved worst-case return 1 when conditioning on target returns around 1. This imposes the question: Can we learn the robust policy by simply conditioning on the largest worst-case return in testing with vanilla DT training?We show via the Gambling environment in Figure 3 that vanilla DT can still fail even conditioning on the largest worst-case return. In this environment, the worst-case returns of three protagonist actions are -15, -6 and 1. At the target return 1, DT is unable to consistently reach a mean return of 1. Similar to DT in the single-stage game, it tends to assign equal likelihood to a 1 and a 2 to reach return 1, leading to a drop of performance to less than 1 under adversarial perturbations. Therefore, simply conditioning on the robust return-to-go in testing is insufficient to generate robust policy with vanilla DT. Moreover, as the target return approaches 5, DT's performance drops more since DT tends to choose a 0 more often to hopefully achieve a return of 5, but instead get negative return against the adversarial perturbations.Conversely, ARDT's policy is robust since it is trained with worst-case returns-to-go. ESPER also performs well in Gambling due to that in this game the robust action is also the one with the largest expected return (Figure 9 ). ARDT also attains the highest return in a multi-stage game when conditioned on large target return, while other two algorithms fail. Consequently, in the context of full data coverage and an adaptive adversary, ARDT is capable of approximating the maximin (Nash Equilibrium) robust policy against the optimal adversarial policy, whereas DT is unable to do so.Connect Four is a discrete sequential two-player game with deterministic dynamics [32] , which can also be viewed as a decision-making problem when facing an adaptive adversary. There is no Test-time adversary 30% optimal 50% optimal 70% optimal 100% optimal DT 0. intermediate reward; rather, there is a terminal reward of either -1, 0, or 1, meaning lose, tie or win for the first-moving player, respectively. Therefore we fix the target return in Connect Four as 1. We fix the protagonist to be the first-move player. This game has a maximum horizon of 22 steps. Since we have a solver of this game, we manage to collect the data with ϵ-greedy policy. We define the optimality of an ϵ-greedy policy of protagonist or adversary as (1 -ϵ) • 100%. Each dataset includes a total of 10 6 number of steps for variable-length trajectories.The results of training ARDT and DT on suboptimal datasets are presented in Figure 4 . To demonstrate the stitching ability of the two methods, datasets collected by suboptimal protagonist and adversarial policies were chosen, namely at 30%, 40% and 50% optimality for the protagonist and 10%, 30%, 50% optimality for the adversary. The figures show that ARDT outperforms DT learning from suboptimal datasets, and more clearly so when tested against more powerful adversaries than the behavior adversary.We hypothesise that one important factor accounting for the negative returns of DT in the setting in 4 may be the lack of long, winning trajectories in the dataset. To exclude this factor, in Table 1 we mix near-random and near-optimal datasets for training, and test our algorithms against different levels of adversary. Compared to ESPER and DT, ARDT still significantly outperforms DT, which suggests that even in the presence of trajectories with high returns in the data, training DT with original returns-to-go in this setting will limit its ability to extract a powerful and robust policy. In addition, the results show that ESPER outperforms DT, indicating relabeling trajectories with expected return can help the extracted policy be robust to distributionaly-shifted and more powerful test-time adversarial policies. ESPER conditioned on the expected return generates policy overfitted to the behavioral adversarial policy, which can fail under distributional shift. However, ARDT as a worst-case-aware algorithm, can achieve greater robustness at test time.Therefore, in discrete environment with partial data coverage and adaptive adversary, ARDT is more robust to distributional shift and a more powerful test-time adversary.In this section, we test the algorithms on a specific instance of Adversarial Reinforcement Learning problems, namely Noisy Action Robust MDP (NR-MDP), which was introduced by Tessler et al. [42] . This includes three continuous control problems of OpenAI Gym MuJoCo, Hopper, Walker2D and Halfcheetah [43] involving the adversarial action noise. In NR-MDP, the protagonist and adversary act jointly. Specifically, they exact a total force over one or more parts of the body of the agent in a Mujoco environment, corresponding to the weighted sum of their individual forces (adversary's weight δ). Despite no longer being in the discrete setting, the return minimax expectile regression in ARDT is implemented with the raw continuous states and actions input, without any discretization.To collect the data, we first trained Action Robust Reinforcement Learning (ARRL) online policies on NR-MDP tasks [42] to create the online protagonist policy and adversary policy for evaluation. Subsequently, the pre-collected data consist of the replayed trajectories removing the ones with low cumulative rewards. We collect the suboptimal trajectories with the ϵ-greedy of the saved ARRL agents, i.e. online protagonist and adversary policies. We then mix them with the pre-collected datasets to create the final training datasets. The proportion of random trajectories to pre-collected trajectories are 0.01 : 1, 0.1 : 1 and 1 : 1, respectively, for datasets categorised as being low, medium, and high randomness. At test time, we evaluate each of our trained ARDT protagonist policies against a population of 8 adversarial online policies, each across 5 seeds. We sweep over the same set of target returns for all algorithms, and pick the best results.As shown in Table 4 , ARDT has overall better worst-case performance than DT and ESPER. In Hopper, ARDT has significantly higher worst-case return than both ESPER and DT. In Halfcheetah, ARDT has significantly higher worst-case return than DT and competitive results with ESPER. This is despite the much lower data coverage than in the previous settings due to the continuous environment. Low data coverage is in theory a greater limitation for ARDT given that it attempts to learn a single best worst-case outcome, rather than an expectation over outcomes. Even so, our proposed solution outperforms the baselines in most settings across all three environments. In addition, ARDT also has better performance when varying the adversary's weight δ and relative mass (Figure 5 ), implying the robustness of ARDT to test-time environmental changes. Therefore, in the context of a continuous state and action space, and when facing a diverse population of adversaries or environmental changes, ARDT trained with minimax returns-to-go can still achieve superior robustness than our baselines.Finally, we offer abalation study on the expectile level α in discrete environments. Specifically, we varied α for relabeling in Single-stage Game, Multi-stage Game and Connect Four. Then we tested against the optimal adversary in the first two environments and an ϵ-greedy adversary (ϵ = 0.5) for the third environment.According to the first two environments, we can confirm a pattern consistent with our theory that smaller alpha leads to more robustness. According to the Equation ( 6)- (9) in the paper, it should In connect four, we set the target return to be 1, and the adversary to be ϵ-greedy (ϵ = 0.5), i.e. taking uniform random policy with probability 0.5. On Connect Four, ARDT with α = 0.01 acts too conservative.be that when the α is closer to 0, our estimate of minimax return is more accurate, and thus our algorithm is more robust. When the alpha is increased to 0.5, our method is converted to Expected-Return-Conditioned Decision Transformer (e.g. ESPER) since the expectile loss in this case reduces to Mean Square Error. This can be confirmed by comparing the results of Single-stage game and Multi-stage game in Figure 6 and 3 .In the third environment, Connect Four, the performance initially increases as decreases, but eventually drops. This is due to the weak stochastic adversary. When the value is too small, the algorithm becomes overly conservative. However, ARDT still outperforms DT significantly. Additionally, it also implies that we can tune the parameter alpha to adjust the level of conservativeness 
conclusions and limitations: 
 This paper introduces a worst-case-aware training algorithm designed to improve the adversarial robustness of the Decision Transformer, namely Adversarially Robust Decision Transformer (ARDT). By relabeling trajectories with the estimated in-sample minimax returns-to-go through expectile regression, our algorithm is demonstrated to be robust against adversaries more powerful than the behavior ones, which existing DT methods cannot achieve. In experiments, our method consistently exhibits superior worst-case performance against adversarial attacks in both gaming environments and continuous control settings.Limitations. In our experiments, our practical algorithm estimates the minimax returns-to-go without accounting for stochasticity, as the transitions in all tested environments, as well as many real-world game settings, are deterministic when conditioned on the actions of both players. Future work should explore both stochastic and adversarial environments to further evaluate the performance of our proposed solution.Broader Impact. Adversarial RvS holds potential for improving the robustness and security of autonomous agents in dynamic and potentially hostile environments. By training agents to withstand adversarial actions and unpredictable conditions, our work can improve the reliability and safety of technologies such as autonomous vehicles, cybersecurity defense systems, and robotic operations.  
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: [NA] Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Our implementation is based on some other codebases, which are cited in section 4 and Appendix C.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Theorem. 1 Let π D and πD be the data collecting policies used to gather data for training an RvS protagonist policy π. Assume T (s t+1 | s t , a t , āt ) = ρ π D ,π D (s t+1 | τ 0:t-1 , s t , a t , āt , I(τ ) = z), for any goal z such that ρ π D ,π D (I(τ ) = z | s 0 ) > 0, the information distance can be minimized:E τ ∼ρ πz ,π test [D(I(τ ), z)] = 0 if πtest (ā t | τ 0:t-1 , s t , a t ) = ρ π D ,π D (ā t | τ 0:t-1 , s t , a t , I(τ ) = z).Proof. According to Eq. ( 3):EQUATIONIf the sufficient condition is satisfied:πtest (ā t | τ 0:t-1 , s t , a t ) = ρ π D ,π D (ā t | τ 0:t-1 , s t , a t , I(τ ) = z),combined with our assumption, we haveEQUATIONThen, we have:E τ ∼ρ πz ,π test (•|s0) [D(I(τ ), z)] = τ ρ πz,πtest (τ | s 0 ) • D(I(τ ), z) = τ t π(a t | τ 0:t-1 , s t , z) • πtest (ā t | τ 0:t-1 , s t , a t ) • T (s t+1 | s t , a t , āt ) • D(I(τ ), z) = τ t ρ π D ,π D (a t | τ 0:t-1 , s t , I(τ ) = z) • ρ π D ,π D (s t+1 | τ 0:t-1 , s t , a t , I(τ ) = z) • D(I(τ ), z) = τ ρ π D ,π D (τ | s 0 , I(τ ) = z) • D(I(τ ), z) = E τ ∼ρ π D ,π D (•|s0,I(τ )=z) [D(I(τ ), z)] = 0.The second equation satisfies by simply decomposing ρ as in section 2. The third equation satisfies due to Eq. ( 13) and Eq. ( 3). The fourth equation satisfies by simply merging all ρ terms. The final equation satisfies since the expectation in the last but one equation is over the probability distribution conditional on I(τ ) = z, leading to that for any sampled trajectory τ , I(τ ) = z, and thus D(I(τ ), z).Multi-Stage Game Apart from single-stage game as motivating example, we also conduct experiments also on a multi-stage game in order to highlight the importance of multi-step minimax instead of a single-step minimum by adversary. This is because the optimal adversary will also consider the optimal protagonist strategy in the subsequent states. Considering a multi-step minimax return is more accurate to describe the value of states and sub-trajectories.Gambling Gambling [32] is originally a simple stochastic bandit problem with three arms and horizon one. When taking the first arm, the reward will be either 5 or -15 with 50% probability each. Similarly, when taking the second arm, the reward will be 1 or -6 with equal probabilities. The third arm has deterministic reward 1. In this paper, we extend it to adversarial setting by considering an adversary controlling the outcomes, shown in Figure 9 .Connect Four Connect Four has a long horizon 22 in the worst case. An example of the dynamics of this game is displayed in Figure 8 . The outcomes only consist of win, tie, and lose, leading to the reward for the protagonist 1, 0, and -1. We leverage the solver in https://github.com/ PascalPons/connect4 to generate stochastic policy for data collection.Figure 7 : Multi-stage Game. LHS is the tree representation of the problem, where P and A are decision maker (protagonist) and adversary, respectively. RHS is the same game tree where we attach the learned minimax return and the action probabilities of the well-trained ARDT of a single run when conditioning on the target return 7 to the adversarial node and the branches, respectively. For example, the highlighted node has minimax return-to-go 4.9. 0.99 on the branch lead to the highlited node indicates the action probability of taking a 0 at the initial state. Thick lines represent the optimal adversarial actions. Our implementation is based on the implementation of Decision Transformer [2] , and ESPER [32] . Our data collection and part of the evaluation against online adversaries are based on the implementation in Noisy Action Robust Reinforcement Learning [42] .The hyperparameters of the Minimax Expectile Regression are in (30, 30) 0.13 (0.99) (30, 50) -0.23 (0.97) (40, 10) 0.56 (0.83) (40, 30) 0.24 (0.97) (40, 50) -0.13 (0.99) (50, 10) 0.61 (0.79) (50, 30) 0.32 (0.95) (50, 50) -0.02 (1) Table 5 : Data profile of Connect Four. The tuples represent the optimalities percentages of protagonist and adversary.π. Subsequently, the trajectory is generated in an auto-regressive manner, utilizing both the actions sampled by protagonist's strategy π and the adversarial strategy π, along with the transitions and rewards provided by the online environments, till the environment terminates or the maximum trajectory length is reached for evaluation. Notably, the target return will be subtracted by the observed immediate reward once received any from the environment.Notably, given that the rewards and states tokens encompass sufficient information about the adversary's action, we train the Transformer model with sequences without adversarial tokens, as it ensures that the model capacity remains consistent across algorithms. As usual, vanilla DT's is conditioned on the observed returns-to-go, which are contaminated by the adversarial actions, and states. Similarly, ESPER policy is conditional on the expected returns-to-go and states. Since the adversary only influences the transition and rewards, ESPER treats the adversary implicitly part of the environment who causes stochasticity, and addresses it with expected return as the information function.Tabular value-based works have been studied to solving zero-sum games offline by leveraging pessimism for out-of-sample estimation [4, 56] . In contrast, we did not incorporate pessimism due to the difficulty of tuning its associated hyperparameter. We employ expectile regression to approximate the in-sample minimax return, inspired by the way of Implicit Q Learning [16] to approximate the Bellman optimal Q-values leveraging expectile regression (ER). ER is efficient since it doesn't rely on querying the actions during minimax return estimation, thus reducing complexity. Learning value functions in this way has much less complexity to learn strategy from the data directly. Thus, the protocol of learning values and then extracting policy from them is reasonable. In addition, training Transformer based on relabeled returns-to-go is similar to do policy extraction from the learned values (returns-to-go) [28] . Transformer which demonstrates the strong reasoning in sequence modeling, should be exploited for improving Reinforcement Learning.Within the broad setting of Robust Reinforcement Learning [27] , our experiments focus on actionrobustness.We conducted experiments on Noisy Action Robust-MDP as a representative environment, following the previous works [7, 42, 14, 34] . According to the different types of attacks proposed in [25, 35] , our adversarial perturbation can be viewed as Man In The Middle (MITM) attack formulation proposed in [35] , specifically actions attack, as noted in the last paragraph of the Related Work section of [35] . The actions attack we considered in our work can influence all elements in RL from the learner's perspective, including the next state observation, immediate reward, and dynamics, thus potentially causing all types of perturbation introduced in [25, 35] . Therefore, it is reasonable to select actions attack as a representative problem, although our architecture can be extended well to other formulations of robustness to the attacks.Robust Reinforcement Learning can be roughly categorized into addressing training-time robustness to poisoning attacks, and test-time robustness to evasion attacks [49] . Poisoning attacks are defined as the manipulation of elements in the training trajectory, including state, actions, and rewards in the data or online environment [25, 35] . The adversary in this paper is similar to the Man In The Middle (MITM) attack formulation in [35] . Robust IQL [55] and RL agent with survival instinct [22] have demonstrated robustness to data poisoning in an offline setting. However, they only aim at the training-time robustness, while testing the algorithms without any attack. Our method is closer to ROLAH [7] and other related works [42, 14, 34] , considering adversary appearing in both training-time and test-time serving evasion attacks. Despite the similarity,these methods are all online.Our method learns to achieve the robustness offline, which is more challenging due to potential distribution shift, and the lack of good behavior policy. In this paper, We didn't select offline robust RL as baseline since we only focus on the Reinforcement Learning via Supervised Learning (RvS) methods. We study the robustness of Decision Transformer (DT), and thus provide more analysis on DT-based algorithms including vanila DT and ESPER in various offline learning settings and environments.We publish our datasets along with the codebase via https://github.com/xiaohangt/ardt. There is no data access restrictions. The Mujoco data profiles are in Table 5 and 4 . The MuJoCo data has 1000 number of trajectories, each with 1000 steps of interactions. The Connect Four datasets have also 10 6 number of steps of interaction in total, where each trajectory has a length at most 22.We conduct experiments on GPUs: a GeForce RTX 2080 Ti with memory 11GB, and a NVIDIA A100 with memory 80GB. The memory for the entire computing is around 100GB when parallelizing the running over random seeds in Connect Four game, and MuJoCo.ARDT can empirically address stochastic environments. Intuitively, this is because by tuning the expectile level , we can balance between expected return and minimax return, as discussed above, to achieve robustness against the distributional environmental changes. When deploying reinforcement learning (RL) methodologies in real-world, safety-critical applications, it is crucial to to account for the worst-case scenarios to prevent unexpected performance drops or safety risks. This approach ensures the system's robustness and reliability under a range of adverse conditions, effectively mitigating the potential for catastrophic failures.One example is as discussed in the introduction of our paper, in the context of autonomous vehicles, considering worst-case scenarios is essential for safe navigation. The algorithm must be capable of handling sudden changes in road conditions, such as unexpected obstacles or extreme weather, as well as making safe decisions in complex interactive scenarios involving multiple (autonomous) vehicles. There are many other real-world control system utilizing RL requires safe and effective operations.In medical applications, such as surgical and assistive robots for patients treatment, preparing for worst-case scenarios-like unexpected drug interactions or patient non-compliance-enhances safety and efficacy. Similarly, in aerospace application, perturbations, such as turbulence, ensures safer aircraft control. Additionally, in the management of plasma states in nuclear fusion reactors using RL methods [6] , it is critical to consider the worst-case outcomes of actions to maintain safe and stable operations.Question • The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).This condition can be readily met in environments formulated as a game with deterministic transitions, which are prevalent in many sequential games (e.g., Go and Chess[38]), as well as in frequently considered MuJoCo environments involving adversarial action noise[34,3]. 


Decision: Interesting piece of work that presents a new algorithm, Adversarial Robust Decision Transformer (ARDT), for scalable offline RL in two-player zero-sum games. Empirical evaluations in full data coverage setting demonstrate the soundness of the proposed method. Good paper.

Reviews: review: paper_summary: paper_summary: This paper consider zero-sum two player Markov game that involves a protagonist and an adversary. The protagonist aims to maximize reward while the adversary aims to minimize it. This paper proposes Adversarial Robust Decision Transformer(ARDT), which is an algorithm that is based on Decision Transformer architecture, and learns the minimax return to go through expectile regression, hence guarantee robustness. Conduct various experiments to validate its algorithm.
strength_weakness: strength_weakness: Lack of ablation study. 
e.g. how \alpha can effect the performance? 
It seems when \alpha is closer to 1, the learned return to go is closer to the maximum conditioned value,  the more robustness the algorithms should present. It would be more solid to say expetile regression is valid if the experiment can show this phenonmenon.

Like what the author stated in the paper, the environment is all deterministic. It would be great if the experiment can show the algorithm can handle stochastic environment.

line 156 "estiation" to estimation
line 196 "deatils" to "details"
questions: questions: Updating \omega and \nu happens when another is frozen. Is there any theoretical guarantee this type of update rule can converge to optimal value?
limitations: limitations: The author stated one of their limitations regarding experiments. add experiment that has stochastic transtions if time permit. 
Add one more experiment that varies the hyperparameter \alpha to show the effectiveness of expectile regression.
paper_summary: paper_summary: This paper introduces Adversarial Robust Decision Transformer (ARDT), a novel approach enhancing robustness in sequential decision-making. ARDT aligns policies with worst-case scenarios learned through minimax expectile regression, outperforming DT in robustness against powerful adversaries across different data coverage scenarios, including sequential games.
strength_weakness: strength_weakness: 1 Several attacks on reinforcement learning have been proposed, such as [1][2], why have these not been applied in experimental settings?

2 There have been some related works on robust reinforcement learning, for example [3][4][5]. The authors should provide a comprehensive literature review on the topic of robust reinforcement learning, including similarities, differences, and reasons why it has not been used as a baseline for experimental comparisons.


[1] Optimal Attack and Defense for Reinforcement Learning, AAAI 2024

[2] Understanding the Limits of Poisoning Attacks in Episodic Reinforcement Learning, IJCAI 2022

[3] Towards Robust Offline Reinforcement Learning under Diverse Data Corruption, ICLR 2024

[4] Survival Instinct in Offline Reinforcement Learning, NeurIPS 2024

[5] Robust Reinforcement Learning through Efficient Adversarial Herding, arXiv
questions: questions: see weaknesses
limitations: limitations: This paper does not consider randomness, whereas random attacks are common in the real world. The authors have already clarified this limitation in the paper.
paper_summary: paper_summary: The paper tackles the worst-case-aware RL problem, revises the conventional DT formulation via minimax returns, and proposes the adversarial robust DT to enhance robustness against test-time adversaries.
strength_weakness: strength_weakness: - Two additional returns-to-go networks are needed for return re-labeling, which increases the computation burden. 
- The trade-off between robustness to adversaries and the policy conservativeness is not discussed.
questions: questions: - Can the ARDT be compared to baselines regarding the normal case, not only the worst case. It could be interesting to see the performance drop in the normal case, and judge where it is worth sacrificing the performance in conventional cases for the improvement in the worst case.
- Are there any real-world applications where it is necessary to consider a worst-case-aware RL algorithm? It could further highlight the significance of the proposed method if some examples could be provided.
limitations: limitations: - The application to real-world scenarios could be limited to some extent, as the learned policy might be too conservative.

Comments: comment: Thank you for your positive response. We are glad that the concerns have been addressed.
Thank you very much for the reply. We are glad that we managed to clarify and answer all the concerns.
Thank you for your response and increasing the score. We are pleased that we have been able to clarify and answer all your concerns.
The rebuttal clarifies my questions on random attacks. My recommendation is based on my overall evaluation of the work.
Thanks for your explanation and further experiments. It address my concerns and I have add one more point to the rating.
Thank you for acknowledging our rebuttal! Could you please confirm whether we have addressed all your concerns? Specifically, we would appreciate your feedback on whether the misunderstanding regarding random attacks has been clarified, and the related additional experimental results (Figure 3) in the rebuttal PDF is clear and sensible to you.
I have read the rebuttal and I will keep my score.
Thank the authors for their detailed response, which has addressed most of my concerns.