Adversarially Robust Decision Transformer
Authors: Xiaohang Tang
Afonso Marques
Parameswaran Kamalaruban
Ilija Bogunovic
Venue: NeurIPS 2024 poster (2024)
Keywords: Offline Reinforcement Learning
Reinforcement Learning via Supervised Learning
Decision Transformer
Robust Adversarial Reinforcement Learning
Primary area: reinforcement_learning
TL;DR: This paper investigate the adversarial robustness of Decision Transformer (DT), and develop a new algorithm to transform the original returns-to-go to in-sample minimax returns-to-go to better address the robustness of DT in adversarial settings.

Abstract: Decision Transformer (DT), as one of the representative Reinforcement Learning via Supervised Learning (RvS) methods, has achieved strong performance in offline learning tasks by leveraging the powerful Transformer architecture for sequential decision-making. However, in adversarial environments, these methods can be non-robust, since the return is dependent on the strategies of both the decision-maker and adversary. Training a probabilistic model conditioned on observed return to predict action can fail to generalize, as the trajectories that achieve a return in the dataset might have done so due to a suboptimal behavior adversary. To address this, we propose a worst-case-aware RvS algorithm, the Adversarially Robust Decision Transformer (ARDT), which learns and conditions the policy on in-sample minimax returns-to-go. 
ARDT aligns the target return with the worst-case return learned through minimax expectile regression, thereby enhancing robustness against powerful test-time adversaries. In experiments conducted on sequential games with full data coverage, ARDT can generate a maximin (Nash Equilibrium) strategy, the solution with the largest adversarial robustness. In large-scale sequential games and continuous adversarial RL environments with partial data coverage, ARDT demonstrates significantly superior robustness to powerful test-time adversaries and attains higher worst-case returns compared to contemporary DT methods.

Adversarially Robust Decision Transformer
Xiaohang Tang" University College London xiaohang tang www.cl.ac.c.....
Afonso Marques
University College London afonso marques. 22@ucl ac.uk
Parameswaran Kamalaruban" Featurespace kamal haaal.pprraeesaatrfffetueeeep..
Ilija Bogunovic
University College London
bogunovic@ucl. ac uk
Abstract
Decision Transformer (DT), one of the representative Reinforcement Learning via Supervised Learning (RvS) methods, has achieved strong performance in offline learning tasks by leveraging the powerful Transformer architecture for sequential becision-making. However, in adversarial environments, these methods can be non-robust, since the return is dependent on the strategies of both the decision- maker and adversary. Training a probabilistic model conditioned on observed return to predict action can fail to generalize, return in the dataset might have done so due to a suboptimal behavior adversary. To address this, propose worst-case-awe Robust Decision Transformer (ARDT), which learns and conditions the policy in-sample minimax returns-to-go. ARDT aligns the target return with the worst-case return learned through minimax expectile regression, thereby enhancing robustness against powerful test-time adversaries. In experiments conducted on sequential games with full data coverage, ARDT can generate a maximin (Nash Equilibrium) strategy, the solution with the largest adversarial robustness. In large-scale sequential games and continuous adversarial RL environments with partial data coverage, ARDT demonstrates significantly superior robustness powerful test-time adversaries and attains higher worst-case returns compared to contemporary DT methods.
the trajectories that achieve
RvS algorithm, the Adversarially
1 Introduction
Reinforcement Learning via Supervised Learning (RvS) 8), has garnered attention within the domain of offline Reinforcement Learning (RL). Framing RL problem of outcome-conditioned sequence modeling, these methodologies have showcased strong performance in offline benchmark tasks [50 53
13 37 17]. One of the representative RyS methods is Decision Transformer (DT) (2). which simply trains policy conditioned on target return via behavior cloning loss. Given the apparent efficacy and simplicity of RvS, settings. In this paper, we aim to use RvS to achieve adversarial robustness, a critical capability for RL agents manage environmental variations and adversarial disturbances [4 42 [4] 46 36) Such
attention is drawn exploring its performance in adversarial
challenges are prevalent in real-world scenarios, e.g., changes in road conditions and decision-making among multiple autonomous vehicles.
In offline RL with adversarial actions, the naive training action prreicttreiitionn models conditioned on the history and observed returns such as DT, or expected returns such as ESPER and DoC [32 [5 (3). which we refer to as Expected Return- Iiturn-Conditional DT (ERC-DT), may result a non-robust policy.
Equal Contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

P
Algorithms
Policy (R = 6): P(ao), P(11)
DT
0.54, 0.46
ERC-DT ARDT (ours)
0.9998, 0.0002
A
0.0006, 0.9994
Algorithms
Expected Worst-case Return
dg
0g
DT
0.46
ERC-DT
0.0002
4
6
2
ARDT (ours)
0.9994
Figure 1: LHS presents the game where ciciision-maker P is confronted by adversary A. In the worst-case scenario, if chooses action ag. A will respond with ao. and P chooses di, A will counter with d4. Consequently, the worst-case returns for actions an and aj are O and 1, respectively. Therefore, the robust choice of action for the decision-maker is a₁- RHS displays tables of action probabilities and the worst-case returns for the Decision Transformer (DT), Expected Return-Conditioned DT (ERC-DT) methods and our algorithm, when conditioned on the largest return-to-go 6. After training using uniformly collected data that covered all possible trajectories, the results reveal that DT fails to select the robust action aj, whereas our algorithm manages to do so.
This issue arises from the potential distributional shifts in the policy of the adversary during offline learning. Specifically, if the behavior policies used for data collection are suboptimal, well-trained conditional policy overfitted to the training dataset may fail to achieve the desired target return during testing, if the adversary's 's policy at test time has changed or become more effective.
We illustrate this using sequential game with data collected by a uniform random policy with full coverage shown in Figure [ The standard RvS methods (DT and ERC-DT) when conditioned on high target return, show poor worst-case performance. DT's failure stems from training data containing the sequences (4o, ā2) and (01, ā3), inaccurately indicating both actions ag and as optimal. ERC-DT missteps because the adversary's 's uniform random policy skews the expected return in favor of ag over a1, wrongly favoring ag. A robust strategy, however, exclusively selects aj against an optimal adver- sary to guarantee high reward. Moreover, in more complex multi-decision scenarios, the adversarial strategy should not only minimize but also counterbalance with maximized returns by the decision- maker at the subsequent state, aligning with a minimax return approach for robust policy development.
Main Contributions, Consequently, there is need for data curation to indicate the potential worst- case returns associated with each action in the Decision Transformer (DT), improving the robustness against adversarial actions. Building on this concept, we introduce the first robust training algorithm for Decision Transformer, the Adversarially Robust Decision Transformer (ARDT). To the best of our knowledge, this paper represents the first exploration of the robustness of RyS methods from an adversarial perspective. In ARDT, we leverage Expectile Regression 22 to transform the original returns-to-go to in-sample minimax returns-to-ge and train DT with the relabeled returns. In the experiments, we illustrate the robustness of ARDT three settings (i) Games with full data coverage (ii) A long-horizon discrete game with partial data coverage (iii) Realistic continuous adversarial RL problems. We provide evidence showcasing the robustness of ARDT more powerful adversary compared to the behavior policy and the distributional shifts of the adversary's policy in the test time. Furthermore, the adversarial MuJoCo tasks, namely Noisy Action Robust MDP 42, ARDT ex- hibits better robustness to population of adversarial perturbations compared to existing DT methods.
Related work. Reinforcement Learning via Supervised Learning is a promising direction for solving general offline RL problems by converting the RL problem into the outcome-conditioned
prediction problem 17 37 1O 30 Decision Transformer (DT) is one of the RvS instance 15).
leveraging the strength of the sequence model Transformer [45] The variants of DT have been
extensively investigated to address the trajectory stitching [50] (3), stochasticity [1 30 32 54]
coaltconditioned problems [9, and generalization in different environments 226 51 21 52
For different problems, it is critical to select an appropriate target as the condition of the policy. Q-Learning DT (33 relabels the trajectories with the estimated Q-values learned to achieve trajectory stitching [18]. ESPER (32), aims at solving the environmental stochasticity via relabeling the
trajectories with expected returns-to-go Although method also transform the returns-togo to

---

estimated values, we instead associate worst-case returns-to-ge with trajectories, aiming at improving its robustness against adversarial perturbations.
Adversarial RL can be framed as a sequential game; with online game-solving explored extensively 41). In the offline setting, where there is interaction with the environment, solving games is challenging due to distributional shifts in the adversary's policy, necessitating adequate data coverage. Tabular methods, such as Nash Q-Learning- a decades-old approach for docision-making in competitive games and other similar methods + [56] address these challenges through pessimistic for addressing adversarial problems, our focus is on enhancing the robustness of RvS methods in these settings. Note that while some offline multi multi-agent RL methods, including DT-based approaches, are
in literature 19
20 39
38
sample estimation. While value-based methods are viable
designed for cooperative games 226 48
our research is oriented towards competitive games.
Problem Setup
We consider the adversarial reinforcement learning problem involving protagonist and an adversary, as described by Pinto et al. [34]. This setting is rooted in the two-player Sem Markov Game framework [23] [3]. This game formally defined by the tuple AS, T, R. po). Here, S represents the state space, where A and A denote the action spaces for the protagonist and adversary. respectively. The reward function R: S
and the transition kernel T : S x Á - A (SS) depend on the state and the joint actions of both players. The initial state distribution is denoted by Po- At each time-step S H, both players observe the state 8 E S and take actions at E and a E A. The adversary is considered adaptive if its action a. depends on 8t and a,. The protagonist receives a reward ri = R(s;, at, az), while the adversary receives a negative reward -Tz.
x A
Denote trajectory T = (st, a, āe, Tı ri) and its sub-trajectory Te: = (st, ae, āe. re) for 0 Osišis < H. The return-to to-n--ttoooooooo refers to the sum of observed rewards from state onwards: R(TH)) = -v Tr. The protagonist policy æ and the adversary policy can be history-dependent, mapping history (Tu:-1, 8t) to the distribution over protagonist and adversary's 's actions, respectively,
In offline RL. direct interaction with the environment is unavailable. Instead, rely dataset D. which consists of trajectories generated by executing a pair of behavioral policies, (Tp, Rp). Our objective in this adversarial offline RI. setting is to utilize this dataset 'D to learn a protagonist policy that seeks to maximize the return, while being counteracted by an adversary's policy म:
offline
max min E
Σ
Th
where (r) = -Po(so) II. [a;
T0:t-1, St, az) T(s+11 8t, at, at). The maximin solution to this problem and its corresponding optimal adversary are denoted as x² and n², respectively.
To.-11, 8t) # (a,
2.1 RyS in Adversarial Environments
We extend the RyS framework by incorporating adversarial settings within the Hindsight Information Matching (HIM) paradigm (9] In adversarial HIM, the protagonist receives goal (denoted by 2) and acts within an adversarial environment. The protagonist's performance evaluated based on how well it achieves the presented goal. Mathematically, the protagonist aims to minimize distance metric D(..)) between the target goal x and the information function I(.) applied to a trajectory r. This sub-trajectory follows distribution p dependent on both the protagonist's policy
πz = xaa Toa-1, St, 2) and the test-time adversarial policy Fest- Formally, the information distance
minimized as follows:
min E, mai (D((T)),,)
where f= represents the trajectory distribution obtained by executing rollouts in Markov Game with the protagonist's policy and the test-time adversarial policy Test- Under HIM framework, Decision Transformer (2 uses adopt return-to-go value as the information function, while ESPER [32 employs the expected return-to-go. The well-trained RvS policy, including DT and ESPER, is derived
3

---

from behavior trajectories filtered based on the condition [(r)
(a To:--1, 2) = p 20 (at 10:1-1, 82, [(r)= = 2).
(3)
If the conditional policy ensures that the information distance in Eq. (2) is minimized to 0, the generated policy æ; robust against lest by simply conditioning on a large target return, or even the maximin optimal protagonist म¹ = -im Rz if the test-time adversary is optimal. Consequently, our objective shifts from learning a maximin policy in Eq. T to minimizing the distance in Eq. ②
For a given target goal 2, the trajectories where the information function I(T) = 2 can be considered as optimal demonstrations of achieving that goal, i.e. information distance minimization, =)) 0. However, in adversarial environments, this information distance is hard to be minimized to O. In adversarial offline RL, a trajectory in the dataset might achieve a goal due to the presence of a weak behavior adversary, rather than through the protagonist's 's effective actions. Filtering datasets for trajectories with high return-to-go could select the ones from encounters with suboptimal adversaries. Consequently, an agent trained through behavioral cloning on these trajectories not guaranteed to achieve a high return when facing different and powerful adversary at test time. We show this formally through a theorem:
Theorem 1. Let and TD be the data collecting policies used to gather data for training an R
protagonist policy x. Assume T(s+11 St, Gz, 0z) = -p= (8 To, at Qt, I(7) = 2)
for any goal such that p D= (I(T)= = so) > 0. the information distance can be minimized:
E, E ((I(Tt). =)] = 0 if Frstt (ee To:t-1, St, a+) p p=D (āe
To:t-1, St, at [(T) 2).
Theorem suggests that the protagonist policy π2, defined in Eq. 3 minimize the information distance when the test-time adversarial policy is Fest = p= ag To:t- St, at [[T] 2). In other words, Z.. conditioned on a large target return, is guaranteed to be robust against p= (a, 70:-11 8t, ae, [(r) = 2). Consequently, the policies of DT and ESPER are only guaranteed to perform well against behavior adversarial policies and will be vulnerable to powerful test-time adversaries since their information function is dependent on behavior adversary. This a concern because, in practice, offline RL datasets are often collected in settings with weak or suboptimal adversaries.
In this context, to enhance robustness, we consider using an information function 1(T) to train the protagonist policy nN by simulating the worst-case scenarios we might encounter during test time. A natural choice for ((T) is the minimax return-to-go. which we formally introduce in Section 3 and use as central component of our method. With this I(T) and sufficient data coverage, pT [ag] To St: 0,, [(T) = 2) can be approximated to the optimal adversarial policy, implying that æ. is robust against the optimal adversary. Furthermore, even without sufficient data coverage, Tz remains more robust compared to previous DT methods, which tend to overfit to the behavior adversarial policy.
Adversarially Robust Decision Transformer
To tackle the problem of suboptimal adversarial behavior in the dataset, we propose Adversari- ally Robust Decision Transformer (ARDT), a worst-case-aware RvS algorithm. In ARDT, the information function is defined the expected minimax returnttogo::
H
I(T) = min max E Bee Beees-T(eeysae ae) Br: 0:41:
Σ
R(ser, ar, āx) To:t-1, 8t, Ct
Intuitively, this information function represents the expected return when an adversary aims to minimize the value, while the protagonist subsequently seeks maximize it. To train a protagonist policy as described in Eq. 33, ARDT first relabels the trajectory with the minimax returns-to-ge in Eq. (4). Secondly, the protagonist policy is then trained similarly to a standard Decision Transformer (DT). This two-step overview of ARDT is provided in Figure [ where the left block includes the minimax return estimation with expectile regression for trajectory relabeling, and the right block represents the DT training conditioned on the relabeled returns-to-go.
"This condition can be readily met in environments formulated as game with deterministic transitions, which are prevalent in many sequential games (e.g.. Go and Chess (38), as well as in frequently considered MuJoCo environments involving adversarial action noise [34]] 3)

---

Minimax Expectile Regression:
Transformer Training
92
G2
max
dt
(t))
(o)
Causal Caseeeee
min
( des ax)
Gc+l
max
( 0e
Figure 2: Training of Adversarially Robust Decision Transformer. We adopt Expectile Regression for estimator Q to approximate the in-sample minimax. In the subsequent protagonist DT training, we replace the original returns-to-go with the learned values Q* to train policy.
We relabel the trajectories in our dataset with the worst-case returns-to-ge by considering the optimal in- un-sample adversarial actions. In practice, we approximate the proposed information function using an (in-sample) minimax returns-to toornss-t--goo predictor, denoted by Q. We simplify the notation of the appear- ance of actions in the dataset a, AA (ax/aa/o s. st) > 0 and ā, E A: (ā To $t, a.) > 0 a E DD and a E D, respectively. The optimal predictor Q* is a history- dependent state-action value function satisfying the following condition that at any time t = -,...,, TT:
②' (To:t-1, St, 0z) = min rz ((ctr=(((e)rrae))) a
max QT S141, 0011) ac+1ED
(5)
We adopt Expectile Regression (ER) 29 approximate Q. This choice is particularly advan- tageous for RL applications because it helps avoid out-of-sample estimation the need to query actions to approximate the minimum or maximum return [16]. Specificallyy the loss function for ER is weighted mean squared error (MSE):
(u)
E. [. 1(u 0) -2
(6)
Suppose random variable y follows conditional distribution p(-[x:,, function go (z) arg ming(s) (g())(()))))))))))))))(())) - (x, y)) can serve as an approximate minimum or maximum operator for the function h(x, y) over all possible value of y. i.e.,
lim (x(x) a0
min 1,,,, y), or lim (x( )P(()))>0
max M(x, v). ysp()==)>0
a-11
Our algorithm uses coupled losses for maximum and minimum operators to approximate the in-sample minimax returns-togoo. Formally, given an offline dataset D, we alternately update the minimax and maximin returns-to-go estimators: Qu (s, a) and Qu (s. a, a). In each iteration, we fix (or v), denoted as @ (or D). and then update v (or w) based on the following loss functions:
(v) = ETnD ( S1, az) - Qu (To.-1,, St, a, az))
(8)
e (w) = EAAD (Q 81, at ae)
Qp(To:, St+1, a(+1) re)
The minimizers (v" and w²) of the two losses mentioned above (with - 0), combined with Eq. satisfy the following condition for all 8t, at, āt D:
7
Qu- (Tu a.) min Qu (Jor-1. St, az a), RED
(10)
Qu-(700-1, ar, a.) = E ee Seriippp (ro. (.
Qu- (To.s, $141, + + a'ED
max
(11)
By substituting Eq. (1) into Eq. 100), the minimax returns-togoo predictor Q.- satisfies Eq. 5)
The formal algorithm of ARDT presented Algorithm T where from line 3-6 the minimax return estimation diagrammed the left block in Figure 2 and line 7-10 is the Transformer training
5

---

Algorithm Adversarially Robust Decision Transformer (ARDT)
1: Inputs and hyperparameters: offline dataset D containing collection of trajectories; a - 0.01
2: Initialization: Initialize the parameters and for in-sample returns-to-go networks Qu and
Qu with the original returns-to-go, and parameter of the protagonist policy ng.
I/ expectile regression updates for in-sample returns-to-ge networks
3: for iteration = 0, 1, do
4: for trajectory D do
update while keeping v fixed f- (wl"==(w), where (*=((w) is given in (9)
5:
update y while keeping w fixed 6: + V,("(,,, where ex (z) is given in (8)
// relabeling dataset D using in-sample returns-to-go
7: Create wors 7 (.... 1, -2-1, RA, St. oe, ...) E D, (s6,
// protagonist policy update
8: L(0) EEEDame Σ, log To(al To:--1, R))
9: for iteration k = 0. 1, do
+ 0 (0)
11: Output: Policy T
the right of Figure 2 Initially, we warm up returns-to-go networks with the original returns-to-go values from the dataset. This step ensures the value function converges to the expected values, facilitating faster maximin value estimation and guaranteeing accurate value function approximation at terminal states. Subsequently, we estimate the minimax returns-to-go by alternately updating the parameters of the two networks based on the expectile regression losses Eq. ( and Eq. (8) Once the minimax expectile regression converges, we replace the original returns-to-go values in the trajectories with the values predicted by Qu. Finally, we train the Decision Transformer (DT) conditioned on the states and relabeled returns-to-go to predict the protagonist's actions. Then the ARDT protagonist policy is ready to be deployed for evaluation in the adversarial environment directly,
Experiments
In this section, we conduct experiments to examine the robustness of our algorithm, Adversarially Robust Decision Transformer (ARDT), in three settings: (i) Short-horizon sequential games, where offline dataset has full coverage and the test-time adversary is optimal (Section 4. (ii) A long-horizon sequential game, Connect Four, where the offline dataset has only partial coverage and the distributional-sshiteddd test-time adversary (Section (.2), and (iii) The standard continuous Mujoco tasks in the adversarial setting and population of test-time adversaries (Section 4.3). We compare
our algorithm with baselines including Decision Transformer and ESPER. The implementation details in Appendix C Notably, since the rewards and states tokens encompass sufficient information
about the adversary's actions, all DT models are implemented not conditioned on the past adversarial tokens to reduce the computational cost.
4.1 Full Data Coverage Setting
The efficacy of our solution is first evaluated on three short-horizon sequential games with adaptive adversary: (I) single-stage game, (2) adversarial, single-stage bandit environment depicting a Gambling round [32], and (3) a multi-stage game. These are depicted in Figure [ Figure 7 and Figure 9 respectively. The collected data consists of 10 trajectories, encompassing all possible trajectories. The online policies for data collection employed by the protagonist and adversary were both uniformly random.
Figure 3 illustrates the efficacy of ARDT in comparison to vanilla DT and ESPER. Across all three environments, ARDT achieves the return of maximin (Nash Equilibrium) against the optimal adversary, when conditioned on a large target return. illustrated Figure ARDT is aware the worst-case return associated with each action it can take due to learning condition the relabeled

---

Nash
ARDT (ours)
Single-slage Game
Gambling
Multi-stage Game
Worst-case Return
1.0
0.5
0.0
Target Return
Target Return
larget Return
Figure 3: Worst-case return versus target return plot comparing the proposed ARDT algorithm against vanilla DTT, our Single-stage Game (left), Gambling (centre) and our Multi-stage Game (right), over 10 seeds.
Adv: 10%
Adv: 30%
Adv. 50%
Protagonist: 30%
Protagonist: 40%
Protagonist: 50%
0.4
Average Return
0.0
0.0
0.0
0.2
ARDT
ARDT
ARDT
Figure 4: Average return of ARDT and vanilla DT on Connect Four when trained on suboptimal datasets collected with different levels of optimality for both the online protagonist's policy (30%, 40% and 50% optimal) and the adversary's 's policy (10%, 30%, 50% optimal), over 10 seeds. We test against fixed adversary that acts optimally 50% of the time, and randomly otherwise.
minimax returns-to-go, and successfully takes robust action In single-stage game (Figure 3 ESPER is vulnerable to the adversarial perturbations, and DT fails generate robust policy when conditioned on large target return. While, it is worth noting that DT has achieved worst-case return 1 when conditioning on target returns around 1. This imposes the question: Can learn the robust policy by simply conditioning on the largest worst-case return in testing with vanilla DT training?
We show via the Gambling environment in Figure 3 that vanilla DT can still fail even conditioning on the largest worst-case return. In this environment, the worst-case returns of three protagonist actions 15, -6 and 1. At the target return 1, DT unable consistently reach mean return of 1. Similar to DT in the single-stage game, it tends to assign equal likelihood to aj and d2 to reach return 1, leading
drop of performance less than under adversarial perturbations. Therefore, simply conditioning on the robust return-to-go in testing is insufficient to generate robust policy with vanilla DT. Moreover, as the target return approaches 5. DT's performance drops more since DT tends to choose do more often to hopefully achieve a return of 5, but instead get negative return against the adversarial perturbations.
Conversely, ARDT's policy is robust since trained with worst-case returns-to-go. ESPER also performs well in Gambling due to that in this game the robust action is also the with the largest expected return (Figure 9). ARDT also attains the highest return in a multi-stage game when conditioned large target return, while other algorithms fail. Consequently, in the context of full data coverage and an adaptive adversary, ARDT is capable of approximating the maximin (Nash Equilibrium) robust policy against the optimal adversarial policy, whereas DT unable to do so.
4.2 Discrete Game with Partial Data Coverage
Connect Four is discrete sequential two-player game with deterministic dynamies 32 which also be viewed decision decision-making problem when facing an adaptive adversary. There is no
7

---

Test-time adversary
30% optimal
50% optimal
70% optimal
100% optimal
DT
0.18 (0.09)
-0288 (0.05)
-0.57 (0.06)
-1 (0.0)
ESPER ARDT (ours)
0.44 (0.09)
0 (0.1)
-0.42 (0.05)
-0.98 (0.01)
0.55 (0.11)
0.11 (0.22)
0.02 (0.09)
-1 (0.0)
Table 1: Average returns of ARDT, vanilla DT and ESPER Connect Four when trained on data mixed from both a near-random (30%, 50%) dataset and a near-optimal (90%, 90%) dataset. Evaluation is done against different optimality levels of adversary.
Halfcheetah
Hopper
Hopper
Werst-cab par--cade Return
DT
Worst case Return
DT
DT
2002
ESPER
ESPER
Average Rotur
1000
ESPER
ARDT
ARDT
ARDT
300
1002
3.85 Test time Noise Weight
Test time Noise Weight
Relative Mass
Figure 5: From left right, (1) the worst-case return under adversarial perturbations with different weights in Halfcheetah, (2) in Hopper, and (3) average returns of algorithms in environments with different relative mass. The initial target returns in the environments Halfcheetah and Hopper 2000 and 500, respectively.
intermediate reward; rather, there is terminal reward of either 1,, or meaning lose, tie win for the first-moving player, respectively. Therefore we fix the target return in Connect Four as 1. We fix the protagonist be the first-move player. This game has maximum horizon of 22 steps. Since we have a solver of this game, we manage to collect the data with e-greedy policy. We define the optimality of an e-greedy policy of protagonist adversary as (1 - e) 100%. Each dataset includes a total of 106 number of steps for variable-length trajectories.
The results of training ARDT and DT suboptimal datasets presented in Figure To demonstrate the stitching ability of the two methods, datasets collected by suboptimal protagonist and adversarial policies chosen, namely at 30%, 40% and 50% optimality for the protagonist and 10%, 30%, 50% optimality for the adversary. The figures show that ARDT outperforms DT learning from suboptimal datasets, and more clearly when tested against more powerful adversaries than the behavior adversary.
We hypothesise that one important factor accounting for the negative returns of DT in the setting in may be the lack of long, winning trajectories in the dataset. To exclude this factor, in Table [ mix near-random and near-optimal datasets for training, and test algorithms against different levels of adversary. Compared to ESPER and DT, ARDT still significantly outperforms DT,, which suggests that even in the presence of trajectories with high returns in the data, training DT with original returns-to-ge to-go in this setting will limit its ability to extract powerful and robust policy. In addition, the results show that ESPER outperforms DT, indicating relabeling trajectories with expected return can help the extracted policy be robust to distributionall--hifeee and more powerful test-time adversarial policies. ESPER conditioned on the expected return generates policy overfitted to the behavioral adversarial policy, which can fail under distributional shift. However, ARDT as worst-case-and algorithm, can achieve greater robustness test time.
Therefore, in discrete environment with partial data coverage and adaptive adversary, ARDT is more robust distributional shift and more powerful test-time adversary.
4.3 Continuous Adversarial Environments
In this section, we test the algorithms on a specific instance of Adversarial Reinforcement Learning problems, namely Noisy Action Robust MDP (NR-MDP), which was introduced by Tessler al. [42 This includes three continuous control problems of OpenAl Gym MuJoCo, Hopper, Walker2D and

---

Worst-case Return
Datasets
DT
ESPER
ARDT
Hopper-Ir
112.1 (63.4)
66.8 (5.5)
477.9 (219.9)
Hopper-mr
330.2 (286.8)
103.2 (11.7)
482.2 (83.7)
Hopper-hr
298.62 (319.72)
63.0 (31.5))
331.69 (187.18)
Ialker2D-Ir
429.6 (31.9)
380.6 (156.9)
405.6 (39.8)
Walker2D-mr
391.0 (20.6)
426.9 (29.7)
508,4 (61.8)
Walker2D-hr
413.22 (54.69)
502.6 (31.3)
492.63 (109.28)
Halfcheetah-Ir
1416.0 (48.5)
1691.8 (63.0)
1509.2 2 (33.8)
Halfcheetah-mr
1229.3 (171.1)
1725.6 (111.8)
1594.6 (137.5)
Halfcheetah-hr
1493.03 (32.87)
1699.8 (32.8)
1765.71 (150.63)
Table 2: Results of DT, ESPER. and our algorithm ARDT with and without conditioned on past adversarial tokens: the Worst-case Return against different online adversarial policies on MuJoCo Noisy Action Robust MDP tasks across 5 seeds. We tested with the same set of target returns and select the best target for each method. To expand the data coverage, we inject random trajectories collected by 0. online policies into the pre-collected online datasets with high robust returns. We use low (suffix -Ir), medium (suffix -mr) and high randomness (suffix -hr) to indicate the proportion of random trajectories.
Halfcheetah <3] involving the adversarial action noise. In NR-MDP, the protagonist and adversary act jointly. Specifically, they exact total force over one more parts of the body of the agent in a Mujoco environment, corresponding to the weighted sum of their individual forces (adversary's s weight 6). Despite longer being in the discrete setting, the return minimax expectile regression in ARDT is implemented with the raw continuous states and actions input, without any discretization.
To collect the data, we first trained Action Robust Reinforcement Learning (ARRL) online policies on NR-MDP tasks [42] to create the online protagonist policy and adversary policy for evaluation. Subsequently, the pre-collected data consist of the replayed trajectories removing the ones with low cumulative rewards. We collect the suboptimal trajectories with the e-greedy of the saved ARRL agents, i.e. online protagonist and adversary policies. We then mix them with the pre-collected datasets to create the final training datasets. The proportion of random trajectories to pre-collected trajectories are 0.01: 1, 0.1 1 and 1 1, respectively, for datasets categorised as being low, medium, and high randomness. At test time, we evaluate each of our trained ARDT protagonist policies against population of 8 adversarial online policies, each across 5 seeds. We sweep over the same set of target returns for all algorithms, and pick the best results.
As shown in Table ARDT has overall better worst-case performance than DT and ESPER. In Hopper, ARDT has significantly higher worst-case return than both ESPER and DT. In Halfcheetah, ARDT has significantly higher worst-case return than DT and competitive results with ESPER. This is despite the much lower data coverage than in the previous settings due to the continuous environment. Low data coverage is in theory greater limitation for ARDT given that it attempts to learn single best worst-case outcome, rather than an expectation over outcomes. Even so, our proposed solution outperforms the baselines in most settings across all three environments. In addition. ARDT also has better performance when varying the adversary's 's weight and relative mass (Figure 5). implying the robustness of ARDT test-time environmental changes. Therefore, in the context of continuous state and action space, and when facing a diverse population of adversaries or environmental changes, ARDT trained with minimax returns-to-go can still achieve superior robustness than our baselines.
4.4 Ablation Study
Finally, we offer abalation study on the expectile level a in discrete environments. Specifically, we varied for relabeling in Single-stage Game, Multi-stage Game and Connect Four. Then tested against the optimal adversary in the first two environments and an c-greedy adversary (c = 0.5) for the third environment.
According to the first two environments, we can confirm pattern consistent with our theory that smaller alpha leads to more robustness. According to the Equation (6)-(9) in the paper, it should

---

ARDT
Single-stage Game: Worst-case Adv.
Muiti-stage Game: Worst-case Adv.
Connect Four: e-greedy Adv.
6.6
Average Return
8.5
2.5
10
-0.
0.3
0.2
U.1
0.45
0.00
Target Return
Target Return
Expectile Level
Figure 6: Ablation study on expectile level 10 random seeds. In connect four, we set the target return to be 1, and the adversary to be e-greedy (e = 0.5), i.c. taking uniform random policy with probability 0.5. On Connect Four, ARDT with 0.01 acts too conservative.
be that when the is closer to O, our estimate of minimax return is more accurate, and thus our algorithm is more robust. When the alpha increased 0.5, our method is converted Expected- Decision Transformer (e.g. ESPER) since the expectile loss in this case reduces to Mean Square Error. This can be confirmed by comparing the results of Single-stage game and Multi-stage game in Figure and 3
In the third enviromment, Connect Four, the performance initially increases decreases, but eventually drops. This is due to the weak stochastic adversary. When the value is too small, the algorithm becomes overly conservative. However, ARDT still outperforms DT significantly. Additionally, also implies that we can tune the parameter alpha to adjust the level of conservativeness
Conclusions and Limitations
This paper introduces a worst-case-aware training algorithm designed to improve the adversarial robustness of the Decision Transformer, namely Adversarially Robust Decision Transformer (ARDT). By relabeling trajectories with the estimated in-sample minimax returns-to-go through expectile regression, our algorithm is demonstrated to be robust against adversaries more powerful than the behavior ones, which existing DT methods cannot achieve. In experiments, our method consistently exhibits superior worst-case performance against adversarial attacks in both gaming environments and continuous control settings.
Limitations. In our experiments, our practical algorithm estimates the minimax returns-to toouns-to-ggo without accounting for stochasticity, as the transitions in all tested environments, as well as many real-world game settings, are deterministic when conditioned on the actions of both players. Future work should explore both stochastic and adversarial environments further evaluate the performance of our proposed solution.
Broader Impact. Adversarial RyS holds potential for improving the robustness and security of autonomous agents in dynamic and potentially hostile environments. By training agents to withstand adversarial actions and unpredictable conditions, our work can improve the reliability and safety of technologies such as autonomous vehicles, cybersecurity defense systems, and robotic operations.
Acknowledgments
Ilija Bogunovic was supported by the EPSRC New Investigator Award EP/X03917X/1; the En- gineering and Physical Sciences Research Council EP/S021566// and Google Research Scholar award. Xiaohang Tang was supported by the Engineering and Physical Sciences Research Council [grant number EP/T5 1779/17. EP/W524335/1), The authors would like to thank the Department of Statistical Science, particular Chakkapas Visavakul, for co-ordinating the computer resources, and the Department of Electronic and Electrical Engineering, and Department of Computer Science at University College London for providing the computer clusters.
10

---

References
[1] Dennis Aigner, Takeshi Amemiya, and Dale Poirier. On the estimation of production frontiers: maximum likelihood estimation of the parameters of a discontinuous density function. International economic review, pages 377-396, 1976.
[2] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling, Advances in neural information processing systems, 15097,
2021.
[3] Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning. In International Conference on Machine Learning, pages 3852-3878. PMLR, 2022.
[4] Qiwen Cui and Simon S Du. When are offline two-player zero-sum markov games solvable? Advances in Neural Information Processing Systems, 011257799257911 2022.
[5] Sebastian Curi, Ilija Bogunovic, and Andreas Krause. Combining pessimism with optimism for robust and efficient model-based deep reinforcement learning. In International Conference on Machine Learning, pages 2254-2264. PMLR, 2021.
[6] Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897): 414-419, 2022.
[7] Juncheng Dong, Hao-Lun Hsu, Qitong Gao, Vahid Tarokh, and Miroslav Pajic. Robust rein- forcement learning through efficient adversarial herding. arXiv preprint arXiv:2306.07400. 2023.
[8] Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rys: What is essential for offline rl via supervised learning? arXiv preprint arXiv:2112.10731, 2021.
[9] Hiroki Furuta, Yutaka Matsuo, and Shixiang Shane Gu. Generalized decision transformer for offline hindsight information matching. arXiv preprint arXiv:2111.10364 2021.
[10] Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals via iterated supervised learning. arXiv preprint 1rXiv:1912.06088, 2019.
[11] Johannes Heinrich, Marc Lanctot, and David Silver. Fictitious self-play in extensive-form games. In International conference on machine learning, pages 805-813. PMLR, 2015.
[12] Junling Hu and Michael P Wellman. Nash q-laarning for general-sum stochastic games. Journal of machine learning research, 4(Nov): 2003.
[13) Michael Janner, Qiyang I.i, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. In Advances Neural Information Processing Systems, 2021.
[14] Parameswaran Kamalaruban. Yu-Ting Huang, Ya-Ping Hsich, Paul Rolland, Cheng Shi. and Volkan Cevher. Robust reinforcement learning via adversarial training with langevin dynamics. Advances in Neural Information Processing Systems, 33:8 33:8127-8138, 2020.
[15] Jeonghye Kim, Suyoung Lee, Woojun Kim, and Youngchul Sung. Decision convformer: Local filtering in metaformer is sufficient for decision making. arXiv preprint 10.00023 2023.
[16] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-laaming. arXiv preprint a 2021.
[17] Aviral Kumar, Xue Bin Peng, and Sergey Levine. Reward-conditioned policies. arXiv preprint 10X1v:191..13465.0 2019.

---

[18] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances Neural Information Processing Systems, 33: 1179-1191, 2020.
[19] Mare Lanctot, Kevin Waugh, Martin Zinkevich, and Michael Bowling. Monte carlo sampling for regret minimization in extensive games. Advances in neural information processing systems, 22, 2009.
[20] Mare Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. A unified game game-theoree theoretic approach to multiagent reinforcement learning. Advances in neural information processing systems, 30, 2017.
[21] Kuang-Huei Lee, Ofir Nachum, Mengjiao Sherry Yang. Lisa Lee, Daniel Freeman, Sergio Guadarrama, Ian Fischer, Winnie Xu, Eric Jang, Henryk Michalewski, et al. Multi-game decision transformers. Advances Neural Information Processing Systems, 35:27921-27936, 2022.
[22] Anqi Li, Dipendra Misra, Andrey Kolobov, and Ching-An Cheng, Survival instinct in offline reinforcement learning. Advances in neural information processing systems, 36, 2024.
[23] Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pages 157-163. Elsevier, 1994.
[24] Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competiiiee environments. Advances in neural information processing systems, 30, 2017.
[25] Jeremy McMahan, Young Wu, Xiaojin Zhu, and Qiaomin Xie. Optimal attack and defense for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 14332- 14340, 2024.
[26] Linghui Meng. Muning Wen, Yaodong Yang, Chenyang Le, Xiyun Li, Weinan Zhang, Ying Wen, Haifeng Zhang, Jun Wang, and Bo Xu. Offline pre-trained multi-agent decision transformer: One big sequence model tackles all smac tasks. arXiv preprint arXiv:2112.00 12.02845, 2021.
[27] Janosch Moos, Kay Hansel, Hany Abdulsamad, Svenja Stark, Debora Clever, and Jan Peters. Robust reinforcement learning: A review of foundations and recent advances. Machine Learning and Knowledge Extraction, 4(1):276-315, 2022.
[28] Gerhard Neumann and Jan Peters. Fitted q- q-iteration by advantage weighted regression, Advances in neural information processing systems, 21, 2008.
[29] Whitney K Newey and James L Powell. Asymmetric least squares estimation and testing. Econometrica: Journal of the Econometric Society, pages 819-847, 1987.
[30] Pedro A Ortega, Markus Kunesch, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Joel Veness, Jonas Buchli, Jonas Degrave, Bilal Piot, Julien Perolat, al. Shaking the foundations: delusions in sequence models for interaction and control. arXiv preprint 2rXiv:21 2021.
[31] Keiran Paster, Sheila A Mcllraith, and Jimmy Ba. Planning from pixels using inverse dynamics models. arXiv preprini arXiv:2012.02419, 2020.
[32] Keiran Paster, Sheila Mcllraith, and Jimmy Ba. You can't count on luck: Why decision transformers and rvs fail in stochastic environments. Advances Neural Information Processing Systems, 35:38966-38979, 2022.
[33] Julien Perolat, Bruno Scherrer, Bilal Piot, and Olivier Pietquin. Approximate dynamic pro- gramming for two-player zero-sum markov games. In International Conference on Machine Learning, pages 1321-1329. PMLR, 2015.
[34] Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. In International Conference on Machine Learning, pages 2817-2826.
PMLR, 2017.
12

---

[35] Anshuka Rangi. Haifeng Xu, Long Thanh, and Massimo Franceschetti. Understanding the limits of poisoning attacks in episodic reinforcement learning. arXiv preprint arXivv2208. 2022.
[36] Mare Rigter, Bruno Lacerda, and Nick Hawes. Rambo-rl: Robust adversarial model-based offline reinforcement learning. Advances in neural information processing systems, 35: 66022- 16097, 2022.
[37] Juergen Schmidhuber. Reinforcement learning upside down: Don't predict rewards- just map them to actions. arXiv preprint arXiv: a 1otv::922.000 1912.08875, 2019.
[38] Julian Schritwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Si- mon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604 609- 2020.
[39] Sriram Srinivasan, Mare Lanctot, Vinicius Zambaldi, Julien Pérolat, Karl Tuyls, Rémi Munos, and Michael Bowling. Actor-critic policy optimization in partially observable multiagent environments. Advances in neural information processing systems, 31, 2018.
[40] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Leibo, Karl Tuyls, et al. Value- decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv: 1006/70666552200 05296, 2017.
[41] Xiaohang Tang, Stephen Marcus McAleer, Yaodong Yang, et al. Regret-minimizing double oracle for extensive-form games. In International Conference on Machine Learning, pages 33599-33615. PMLR, 2023.
[42] Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applications in continuous control. In International Conference on Machine Learning, pages 6215-6224, PMLR, 2019.
[43] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012. doi: N0.1109/RROS
[44] Wei-Cheng Tseng, Huun- Johnson Wang, Yen-Chen Lin, and Phillip Isola Offline multi- agent reinforcement learning with knowledge distillation. Advances in Neural Information Processing Systems, 2577, 2022.
[45] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neurul information processing systems, 30, 2017.
[46] Eugene Vinitsky, Yuqing Du, Kanaad Parvate, Kathy Jang, Pieter Abbeel, and Alexandre Bayen. Robust reinforcement learning using adversarial populations. arXiv preprini a2Xiv:2008.01825, 2020.
[47] Xiangsen Wang, Haoran Xu, Yinan Zheng, and Xianyuan Zhan. Offline multi-agent reinforce- ment learning with implicit global-to-looe value regularization. Advances in Neural Information Processing Systems, 36, 2024.
[48] Muning Wen, Jakub Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang. Multi-agent reinforcement learning is sequence modeling problem. Advances in Neural Information Processing Systems, 35: 16509-19115,2, 2022.
[49] Fan Wu, Linyi Li, Chejian Xu, Huan Zhang, Bhavya Kailkhura, Krishnaram Kenthapadi, Ding Zhao, and Bo Li. Copa: Certifying robust policies for offline reinforcement learning against poisoning attacks. arXiv preprint S20i::2203008999 2022.
[50] Yueh-Hua WU, Xiaolong Wang, and Masashi Hamaya. Elastic decision transformer. Advances in Neural Information Processing Systems, 36, 2024.
13

---

[51] Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua Tenenbaum, and Chuang Gan. Prompting decision transformer for few-shot policy generalization. In international conference on machine learning, pages 24631-24645. PMLR, 2022.
[52] Mengdi Xu, Yuchen Lu, Yikang Shen, Shun Zhang, Ding Zhao, and Chuang Gan. Hyper- decision transformer for efficient online policy adaptation. arXiv preprint arXiv:2304.08887, 2023.
[53] Taku Yamagata, Ahmed Khalil, and Raul Santos-Rodrigun Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling in offline rl. In Interna- lional Conference on Machine Learning, pages 38989-39007. PMLR, 2023.
[54] Mengjiao Yang, Dale Schuurmans, Pieter Abbeel, and Ofir Nachum. Dichotomy of control: Separating what you can control from what you cannot. arXiv preprint arXiv:2210.13445,, 2022.
[55] Rui Yang, Han Zhong, Jiawei Xu, Amy Zhang, Chongjie Zhang, Lei Han, and Tong Zhang. Towards robust offline reinforcement learning under diverse data corruption. arXiv preprint vrXiv:2310.12955 2023.
[56] Han Zhong, Wei Xiong, Jiyuan Tan, Liwei Wang, Tong Zhang, Zhaoran Wang, and Zhuoran Yang. Pessimistic minimax value iteration: Provably efficient equilibrium learning from offline datasets. In International Conference Machine Learning, pages 271 17117227142. PMLR, 2022.
14

---

Appendix
A Proofs
Theorem. Let TD and TDD be the data collecting policies used to gather data for training an RvS protggniist policy и. Assume T(3+11 St. at, αt) /T T0:-11, St, at, Re [(T) 2), for any goal such that p= (I(T) = so) > 0, the information distance can be minimized: Er- te (T)((), 0 if Fest (ūt To:--1: 1, St, as) = (ōe Tot-11 1, St, at (T) = =2)
Proof. According Eq.
π (Ot. Tod-11, Sc, 2) =p0 p0 (at
70:--1, 1, St: I(T) = 2).
(12)
If the sufficient condition is satisfied: mea (ā; combined with our assumption, we have
1000000 (ā: 10, I(r) 2).
00
uas (āg To:--1, St, ar) T(sc-1
p T0:-11 S1, au [(T) = z). (13)
Then, we have:
E Prpupes (- [D][T),=)]
E p mm (T (s)--D((TT))2)
EII m(ae 11, 2) Feest (ār. TD:-11, a+) T(8441 (r,0,,,a)) =(
EII TD. (a+ Toa-1, 81, I(T) = : p (84+1 To ae [(T) = ()(((())),
Σ pro (T) so. (T) -))(()))
Erap' (1))0) (ox
0.
The second equation satisfies by simply decomposing in section The third equation satisfies due to Eq. (13) and Eq. (3). The fourth equation satisfies by simply merging all terms. The final equation satisfies since the expectation in the last but one equation is over the probability distribution conditional on ((T) z, leading to that for any sampled trajectory т, [(T) = z, and thus 2)
B Environments
Multi-Stage Game Apart from single-stage game motivating example, we also conduct experi- ments also on multi-stage game in order to highlight the importance multi-step minimax instead of single-step minimum by adversary. This is because the optimal adversary will also consider the optimal protagonist strategy the subsequent states. Considering a multi-step minimax return is more accurate to describe the value of states and sub-trajectories.
Gambling Gambling 33 is originally a simple stochastic bandit problem with three arms and horizon one. When taking the first arm, the reward will be either 5 -15 with 50% probability each. Similarly, when taking the second arm, the reward will be 1 or -6 with equal probabilities. The third arm has deterministic reward In this paper, we extend adversarial setting by considering an adversary controlling the outcomes, shown in Figure
Connect Four Connect Four has a long horizon 22 in the worst case. An example the dynamics of this game is displayed in Figure g The outcomes only consist of win, tie, and lose, leading to the reward for the protagonist 1, O. and 1. We leverage the solver in witttt:://// htap::://githulll com/ PascalPonss/coneect to generate stochastic policy for data collection.
15

---

0.11
4,9
P
Ry
R2
0.87
0.42
A
A
Rg.
Rp
Rg
Rg P1
Big
Rg.
Bg Pi
Figure Mulli-stage Game. LHS is the tree representation of problem, where and A are decision maker (protagonist) and adversary, respectively. RHS is the same game tree where we attach the learned minimax return and the action probabilities the well-trained ARDT of single run when conditioning the target return 7 to the adversarial node and the branches, respectively, For example, the highlighted node has minimax return-to-go 4.9, 0.99 on the branch lead the highlited node indicates the action probability of taking do the initial state. Thick lines represent the optimal adversarial actions.
A
R
6
dg
Figure 8: Connect Four game, Two players put pieces to specific columns in turns. The pieces will drop to the final empty row.
Figure 9: Two-player zero-sum game Gambling: P: protagonist, and A: adversary. The payoff at the leaves are for protagonist.
C Implementation Details
Our implementation is based the implementation of Decision Transformer 2 and ESPER 322. Our data collection and part of the evaluation against online adversaries are based on the implementation in Noisy Action Robust Reinforcement Learning 42
The hyperparameters the Minimax Expectile Regression are Table 3 practical version of ARDT's minimax regression, we adopt a leaf regularization to ensure the accuracy of predicting outcomes at terminal state (leaf node). We record our training iterations of minimax regression in the hyperparameters table. The number of training iterations in full coverage examples and Connect Four are larger than twice of the maximum trajectory length, which guarantees the minimax return propagated from the leaves every node. According to the prediction in Figure 7 the predicted values are relatively accurate. While MuloCo environment, the trajectory length are too long for minimax regression to make full propagation, partial propagate already brings robustness based the MuloCo results 4
Decision Transformer is trained with cross entroy loss in discrete environment and MSE in continuous environment, as in Decision Transformer (2]. During action inference at test time, all DT- based methods are prompted with desired initial target return and state to generate the strategy
16

---

Process
Hyperparameters
Values (Full coverage game/Connect Four/MuloCo)
Number of training steps
50000/000000010000000
Number of testing iterations
100
Context length
4/20/22
Learning rate
0.0001
Transformer training
Weight decay
0.0001
Warm up steps
1000
Drop out
Batch size
128/128/512
Optimizer
Adam
Number training iterations
6/44/50
Learning
0.001
Weight decay
0,0001
Minimax expectile regression
Model
LSTM MLP
Batch size
128/128/511
Ieaf weight
0.9
Expectile level
0.01
Optimizer
Adam
Table 3: Hyperparameters ARDT.
Datasets
Returns of Behavior Policy
Dataset
Returns of Behavior Policy
Hrpper-Ir
942.8 (417.4)
(30, 10)
0.49 (0.87)
Hopper-mr
870.5 (467.4)
(30, 30)
0.13 (0.99)
Hopper-hr
505.4 (532.2)
(0)
-0.23 (0.97)
(40, 10)
0.56 (0.83)
Walker2D-Ir
1330.2 (654.2)
(40,
0.24 (0.97)
Walker2D-mr
1277.4 (653.9)
(40, 50)
-0.13 (0.99)
Walker2D-hr
1013.1 (581.1)
(50, 10)
0.61 (0.79)
Halfcheetah-Ir
1516.4 (215.8)
(50, 30)
0.32 (0.95)
Halfcheetah-me mr Halfcheetah-B hr
1395.1 (461.2)
(50, 50)
-0.02 (1)
795.3 (764.4)
Table 5: Data profile of Connect Four.
The tuples represent the optimalities per- centages of protagonist and adversary.
Table 4: Data profile of MuJoCo NR-MDP.
т. Subsequently, the trajectory is generated in an auto-regressive manner, utilizing both the actions sampled by protagonist's strategy and the adversarial strategy म, along with the transitions and rewards provided by the online environments, till the environment terminates or the maximum trajectory length reached for evaluation. Notably, the target return will be subtracted by the observed immediate reward once received any from the environment.
Notably, given that the rewards and states tokens encompass sufficient information about the adver- sary's s action, we train the Transformer model with sequences without adversarial tokens, as it ensures that the model capacity remains consistent across algorithms. As usual, vanilla DT's is conditioned on the observed returns-to-go, which are contaminated by the adversarial actions, and states. Similarly, ESPER policy is conditional the expected returns-to-go and states. Since the adversary only influences the transition and rewards, ESPER treats the adversary implicitly part of the environment who causes stochasticity, and addresses it with expected return as the information function.
D Additional Related Works
D.1 Algorithmically Related Works
Tabular malue-based works have been studied to solving zero-sum games offline by leveraging pessimism out-of-sample estimation 56). In contrast, we not incorporate pessimism due to the difficulty of tuning its associated hyperparameter. We employ expectile regression approximate the in-sam minimax return, inspired by the way of Implicit Q Learning [16] to approximate the
Bellman optimal Q-values leveraging expectile regression (ER). ER is efficient since doesn't 't rely
17

---

on querying the actions during minimax return estimation, thus reducing complexity. Learning value functions this way has much less complexity to learn strategy from the data directly. Thus, the protocol of learning values and then extracting policy from them is reasonable. In addition, training Transformer based on relaheled returns-to-go is similar do policy extraction from the learned values (returns-to-ge) [28]. Transformer which demonstrates the strong reasoning in sequence modeling, should be exploited for improving Reinforcement Learning.
D.2 Related Adversarial Attacks
Within the broad setting of Robust Reinforcement Learning 27 our experiments focus on action- robustness, We conducted experiments on Noisy Action Robust-MDP
representative environment, following the previous works 7 14 34]. According to the different types attacks proposed in our adversarial perturbation can be viewed as Man In The Middle (MITM) attack formulation proposed in [5], specifically actions attack, as noted in the last paragraph of the Related Work section of 35 The actions attack considered in work
25 35
influence all elements in RL from the learner's ' perspective, including the next state observation, immediate reward, and dynamics, thus potentially causing all types of perturbation introduced in 25 (3). Therefore, it is reasonable to select actions attack as representative problem, although our architecture can be extended well to other formulations of robustness to the attacks.
D.3 Related Robust RL Methods
Robust Reinforcement Learning be roughly categorized into addressing training-time robustness to poisoning attacks, and test-time robustness to evasion attacks [49]. Poisoning attacks are defined the manipulation of elements in the training trajectory, including state, actions, and rewards in the data or online environment [25 35 The adversary in this paper is similar to the Man In The Middle (MITM) attack formulation 335 Robust IQL (55) and RL agent with survival instinct [22] have demonstrated robustness to data poisoning in an offline setting, However, they only aim the training-time robustness, while testing the algorithms without any attack. Our method is closer to ROLAH [ and other related works 42 14 34), considering adversary appearing in both training-time and test-time serving evasion attacks. Despite the similarity, methods are all online. Our method learns to achieve the robustness offline, which is more challenging due to potential distribution shift, and the lack of good behavior policy. In this paper, We didn't select offline robust RL as baseline since we only focus the Reinforcement Learning via Supervised I Learning (RvS) methods. We study the robustness of Decision Transformer (DT), and thus provide more analysis on D'T-based algorithms including vanila DT and ESPER in various offline learning settings and environments.
E Data and Computing Resources
We publish our datasets along with the codebase via hatttpaas/////hhhoooooooooaaaaar ardt There is no data access restrictions. The Mujoco data profiles are in Table and 4 has 1000 number of trajectories, each with 1000 steps of interactions. The Connect Four datasets have also 10 number of steps of interaction in total, where each trajectory has a length at most 22.
The MuloCo data
We conduct experiments GPUs: GeForce RTX 2080 Ti with memory 11GB, and NVIDIA 100 with memory 80GB. The memory for the entire computing is around 100GB when parallelizing the running over random seeds in Connect Four game, and MuloCo.
F Additional Experimental Results
ARDT can empirically address stochastic environments. Intuitively, this is because by tuning the expectile level, we can balance between expected return and minimax return, as discussed above, to achieve robustness against the distributional environmental changes.
18

---

With Adversarial Token
Without Adversarial Token
Datasets
DT
ESPER
ARDT
ESPER
ARDT
Hopper-Ir
175.9 (7.5)
210.6 (21.5)
302.0 (221.0)
112.1 (63.4)
66.8 (5.5)
477.9 (219.9)
Hopper-mm
361.3 (278.0)
302.8 (159.4)
454.7 (402.5)
330.2 (286.8)
103.2 (11.7)
482.2 (83.7)
Hopper-hr
322.8 (225.9)
48.8 (0.3)
158.5 (2.8)
298.62 (319.72)
63.0
331.69 (187.18)
Walker2D-Ir
475.4 (8488
406.1 (108.2) 2)
528.6 (85.7)
429.6 (31.9)
380.6 (156.9)
405.6 (39.8)
Walker2D-mr
505.1 (39.5)
455,2 (47.0)
485.5 (67.2)
391.0 (20.6)
426.9 (29.7)
508.4 (61.8)
Walker2D-hr
511.9 (94.6)
618.8 (100.5)
475.2 (84.2)
413.22 (54.69)
502.6 (31.3)
492.63 (109.28)
Halfcheetah--r
1170.6 (94.9)
1368.6 (58.1)
1323.2 (76.8)
1416.0 (48.5)
1691.8 (63.0)
1509.2 (33.8)
Halfchectab-mr
1599.9 (464.2)
1433.5 (64.7)
1299,9 (115.00
1229.3 (171.1)
1725.6 (111.8)
1594.6 (137.5)
Halfchoetah-hr
1140.8 (251.4)
1558.5 (345.5)
974.3 (52.3)
1493.03 (32.87)
1699.8 (32.8)
1765.71 (150.63)
Table Ablation study on training policy conditioning on adversary tokens with average Worst-case returns against 8 adversaries including algorithms with and without past adversary tokens. ARDT trained without adversarial token has better performance in general.
Halfcheetah
Hopper
Walker2d
500
Worst-case Return
DT
Worstccase Return
DT
Worst-cast Return
DT
2000
ARDT
600
ARDT
600
ARDT
3500
300
2000
500
0.02
0.05
0.05
0.23
0.05
Test-time Noise Weight 6
Test-time Noise Weight 6
Test-time Noise Weight 6
Figure 10: Average results in enviromments with different test-time weight of adversarial noise. The target return of the three environments are set to 2000, 500 and 800.
P
Algorithm
Average Return
Go...
DT
0.49 (0.0)
ESPER
0.11 (0.2)
0.1
0.9
ARDT (0.3)
0.23 (0.2)
ARDT (0.2)
0.82 (0.08)
A
A
ARDT (0.1)
0.89 (0.01)
ARDT (0.05)
0.91 (0.04)
ARDT (0.01)
0.9 (0.04)
0
4
6
6
1
2
(b) Results of DT-based methods Stochastic Game conditioned large target return. We vary expectile level alpha ARDT and observe that ARDT with a well-tuned alpha can perform the best.
(a) Stochastic Enviromment revised from Single- stage game, where the protagonist taking action ai might have probability 0.1 to reach the left branch and 0.9 to reach the right branch.
Figure 11: Results on stochastic environment.
G Additional Broader Impact
When deploying reinforcement learning (RL) methodologics in real-world, safety-critical applications, it is crucial to account for the worst-case scenarios to prevent unexpected performance drops or safety risks. This approach ensures the system's 's robustness and reliability under a range of adverse conditions, effectively mitigating the potential for catastrophic failures.
One example is as discussed in the introduction our paper, the context of autonomous vehicles, considering worst-case scenarios is essential for safe navigation. The algorithm must be capable of handling sudden changes in road conditions, such as unexpected obstacles or extreme weather, as well making safe decisions in complex interactive scenarios involving multiple (autonomous) vehicles.
19

---

DT
ESPER
ARDT
Single-Stage Game: Stochastic Adversary
Algorithm
Average Return
2.5
DT
5.19 (0.03)
Expected Return
ESPER
4.67 (0.07)
ARDT (0.3)
4.51 (0.1)
1.5
ARDT (0.2)
4.75 (0.32)
ARDT (0.1)
5.2 (0.02)
ARDT (0.05)
5.19 (0.02)
ARDT (0.01)
5.21 (0.04)
0.0
0.1
02
0.3
0.4
0.5
0.6
0.7
0.8
(b) Average retur conducted on Multi-stage Game, where the adversary has e-greedy policy (c 0.2).
(a) Expected return computed based the policy single random seed in Table in the paper.
ARDT with tuned expectile level achieved the best average return.
Figure 12: Results against stochastic adversary (e-greedy policy).
Environment
Algorithm
Normal Return
Worst-case Return
Return Drop
Hopper-hr
DT
809.44 (173.21)
298.62 (319.72)
510.82
Hopper-hr
ESPER
658.78 (331.68)
63.0 (11.5)
595.78
Hopper-hr
ARDT
746.87 (345.3)
331.69 (187.18)
415.18
Table Methods tested normal setting, i.c. without adversarial perturbation.
There many other real-world control system utilizing RL requires safe and effective operations. In medical applications, such as surgical and assistive robots for patients treatment, preparing for worst-case scenarios-lis unexpected drug interactions patient www.-oolccceeeeaaaneeee safety and efficacy. Similarly, in aerospace application, perturbations, such as turbulence, ensures safer aireraft control. Additionally, the management of plasma states in nuclear fusion reactors using RL methods [6], is critical to consider the worst-case outcomes of actions to maintain safe and stable operations.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation in the final section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
21

---

Justification:
We have a theorem in Section 2.1 and the proofs in Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide detailed demonstration on the algorithm in section D with pipeline diagram in Figure 2 as well as implementation details including hyperparameters Ap-
pendix C
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
22

---

reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the access to our dataset in Appendix The code is attached as the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
Cotee//ddeeeSbmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide detailed explanation of the environmental settings in Appendix B as well as the hyperparmeters in Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
23

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttps/s/horris...cpppitt///tei
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Our implementation is based on some other codebases, which are cited in section 4 and Appendix C
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
25

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The new dataset in Appendix
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "xriQutJGQd", "reviewer": "Reviewer_JwFn", "summary": "This paper consider zero-sum two player Markov game that involves a protagonist and an adversary. The protagonist aims to maximize reward while the adversary aims to minimize it. This paper proposes Adversarial Robust Decision Transformer(ARDT), which is an algorithm that is based on Decision Transformer architecture, and learns the minimax return to go through expectile regression, hence guarantee robustness. Conduct various experiments to validate its algorithm.", "questions": "Updating \\omega and \\nu happens when another is frozen. Is there any theoretical guarantee this type of update rule can converge to optimal value?", "limitations": "The author stated one of their limitations regarding experiments. add experiment that has stochastic transtions if time permit. \nAdd one more experiment that varies the hyperparameter \\alpha to show the effectiveness of expectile regression.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "First to explore the robusteness of reinforcement learning via supervised learning. Great presentation, great novelty. Exhaustive experiement details.", "weaknesses": "Lack of ablation study. \ne.g. how \\alpha can effect the performance? \nIt seems when \\alpha is closer to 1, the learned return to go is closer to the maximum conditioned value,  the more robustness the algorithms should present. It would be more solid to say expetile regression is valid if the experiment can show this phenonmenon.\n\nLike what the author stated in the paper, the environment is all deterministic. It would be great if the experiment can show the algorithm can handle stochastic environment.\n\nline 156 \"estiation\" to estimation\nline 196 \"deatils\" to \"details\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments and helpful suggestions. We address the concerns as follows:\n\n------\n\n**Q: Lack of ablation study. e.g. how \\alpha can effect the performance? It seems when \\alpha is closer to 1, the learned return to go is closer to the maximum conditioned value, the more robustness the algorithms should present. It would be more solid to say expectile regression is valid if the experiment can show this phenomenon.**\n\n**A:** Thank you for pointing this out. We have added an ablation study on $\\alpha$ in the rebuttal PDF (Figure 1) and will include these results and analyses in the paper. We varied $\\alpha$ for relabeling in Single-stage Game, Multi-stage Game and Connect Four. We tested against the optimal adversary in the first two environments and an $\\epsilon$-greedy adversary ($\\epsilon=0.5$) for the third environment. \n\n- According to the first two environments, we can confirm a pattern consistent with our theory that smaller alpha leads to more robustness. According to the Equation (6)-(9) in the paper, it should be that when the $\\alpha$ is closer to 0, our estimate of minimax return is more accurate, and thus our algorithm is more robust. When the alpha is increased to 0.5, our method is converted to Expected-Return-Conditioned Decision Transformer  (e.g. ESPER) since the expectile loss in this case reduces to Mean Square Error.\n- In the third environment, Connect Four, the performance initially increases as $\\alpha$ decreases, but eventually drops. This is due to the weak stochastic adversary. When the $\\alpha$ value is too small, the algorithm becomes overly conservative. However, ARDT still outperforms DT significantly. Additionally, it also implies that we can tune the parameter alpha to adjust the level of conservativeness (related to the 2nd and 5th question by reviewer p4QE). \n\n------\n\n**Q: Like what the author stated in the paper, the environment is all deterministic. It would be great if the experiment can show the algorithm can handle stochastic environment.**\n\n**A:** Thank you for your insightful feedback. We have added a stochastic environment in the rebuttal PDF (Figure 2) and demonstrated that with an appropriate $\\alpha$, ARDT can empirically address stochastic environments. Intuitively, this is because by tuning the expectile level $\\alpha$, we can balance between expected return and minimax return, as discussed above, to achieve robustness against the distributional environmental changes.\n\nFurther Clarification:\n\n- In reinforcement learning (RL) problems with adversarial perturbation, deterministic environments are common. For instance, MuJoCo continuous control, one of the most widely used and standard environments, is employed for experiments in our paper. Even with adversarial perturbation, many MuJoCo tasks still have deterministic transitions. These environments have been extensively used to examine the robustness of algorithms [1-5]. \n\n- Moreover, it is natural to adapt our work to directly address stochastic environment by adding an extra value network $V_{\\phi}(\\tau_{0:t-1}, s)$ in the minimax expectile regression. Instead of minimizing the loss in Eq. (8) and (9) alternatively, we would minimize the loss in Eq. (8) and the following two losses alternatively: \n$$\\ell^{1-\\alpha}(\\omega) = \\mathbb{E}\\_{\\tau \\sim \\mathcal{D}}[{{Q}\\_{\\omega}(\\tau\\_{0:t-1}, {s\\_t}, a_t, \\bar{a}\\_t) - V\\_{\\widehat{\\phi}}(\\tau\\_{0:t-1}, s\\_{t+1}) - r\\_t}]^2,\\  \\ell(\\phi) = \\mathbb{E}\\_{\\tau \\sim \\mathcal{D}} [L^{1-\\alpha}\\_{\\text{ER}}(\\widetilde{Q}\\_{\\widehat{\\nu}}(\\tau\\_{0:t}, {s\\_{t+1}},a\\_{t+1}) - V\\_{\\phi}(\\tau\\_{0:t-1}, s\\_{t+1}))].$$ However, this method is just a variant of our minimax expectile regression, so we leave it as a future work.\n\n------\n\n**Q: Updating \\omega and \\nu happens when another is frozen. Is there any theoretical guarantee this type of update rule can converge to optimal value?**\n\n**A:** In general, for non-convex-concave optimization problem, the alternative gradient update (fixing \\omega and update \\mu, vice versa) approach doesn't come with any convergence related theoretical guarantee. However, this is the standard approach used in adversarial ML and robust RL literature, such as GAN for image generation, Robust Adversarial Reinforcement Learning [2,3], and etc [2]. When considering additional structural assumptions on the training objective, some convergence guarantees can be provided [7].\n\n------\n\n**We believe we have successfully conducted all the additional experiments and addressed all the questions raised by the reviewer. In light of the strengths of our work, as acknowledged by the reviewer, and our detailed rebuttal to the questions posed, we kindly ask the reviewer to reconsider their score.**\n\n------\n\n[1] Pinto, Lerrel, et al. \"Robust adversarial reinforcement learning.\" *International conference on machine learning*. PMLR, 2017.\n\n[2] Tessler, Chen, Yonathan Efroni, and Shie Mannor. \"Action robust reinforcement learning and applications in continuous control.\" *International Conference on Machine Learning*. PMLR, 2019.\n\n[3] Kamalaruban, Parameswaran, et al. \"Robust reinforcement learning via adversarial training with langevin dynamics.\" Advances in Neural Information Processing Systems 33 (2020): 8127-8138.\n\n[4] Yang, Rui, et al. \"Rorl: Robust offline reinforcement learning via conservative smoothing.\" *Advances in neural information processing systems* 35 (2022): 23851-23866.\n\n[5] Rigter, Marc, Bruno Lacerda, and Nick Hawes. \"Rambo-rl: Robust adversarial model-based offline reinforcement learning.\" *Advances in neural information processing systems* 35 (2022): 16082-16097.\n\n[6] Panaganti, Kishan, et al. \"Robust reinforcement learning using offline data.\" *Advances in neural information processing systems* 35 (2022): 32211-32224.\n\n[7] Mescheder, Lars, Andreas Geiger, and Sebastian Nowozin. \"Which training methods for GANs do actually converge?.\" *International conference on machine learning*. PMLR, 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3Vb0rXTwkN", "reviewer": "Reviewer_18ip", "summary": "This paper introduces Adversarial Robust Decision Transformer (ARDT), a novel approach enhancing robustness in sequential decision-making. ARDT aligns policies with worst-case scenarios learned through minimax expectile regression, outperforming DT in robustness against powerful adversaries across different data coverage scenarios, including sequential games.", "questions": "see weaknesses", "limitations": "This paper does not consider randomness, whereas random attacks are common in the real world. The authors have already clarified this limitation in the paper.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1 The paper's motivation seems good and well-founded, and the idea is considered novel and presented clearly and intriguingly. \n\n2 The experimental results demonstrate better robustness compared to existing DT and ESPER methods.", "weaknesses": "1 Several attacks on reinforcement learning have been proposed, such as [1][2], why have these not been applied in experimental settings?\n\n2 There have been some related works on robust reinforcement learning, for example [3][4][5]. The authors should provide a comprehensive literature review on the topic of robust reinforcement learning, including similarities, differences, and reasons why it has not been used as a baseline for experimental comparisons.\n\n\n[1] Optimal Attack and Defense for Reinforcement Learning, AAAI 2024\n\n[2] Understanding the Limits of Poisoning Attacks in Episodic Reinforcement Learning, IJCAI 2022\n\n[3] Towards Robust Offline Reinforcement Learning under Diverse Data Corruption, ICLR 2024\n\n[4] Survival Instinct in Offline Reinforcement Learning, NeurIPS 2024\n\n[5] Robust Reinforcement Learning through Efficient Adversarial Herding, arXiv", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments and suggestions. We address the concern as follows:\n\n------\n\n**Q: Several attacks on reinforcement learning have been proposed, such as [1,2], why have these not been applied in experimental settings?**\n\n**A:** Thanks for the suggested related works. We will include these in the paper along with the next response.\n\nWithin the broad setting of Robust Reinforcement Learning [10], our experiments focus on action-robustness.We conducted experiments on Noisy Action Robust-MDP as a representative environment, following the previous works [5-9].  According to the different types of attacks proposed in [1,2], our adversarial perturbation can be viewed as Man In The Middle (MITM) attack formulation proposed in [2], specifically actions attack, as noted in the last paragraph of the Related Work section of [2]. The actions attack we considered in our work can influence all elements in RL from the learner's perspective, including the next state observation, immediate reward, and dynamics, thus potentially causing all types of perturbation introduced in [1,2]. Therefore, it is reasonable to select actions attack as a representative problem, although our architecture can be extended well to other formulations of robustness to the attacks.\n\n------\n\n**Q: There have been some related works on robust reinforcement learning, for example [3,4,5]. The authors should provide a comprehensive literature review on the topic of robust reinforcement learning, including similarities, differences, and reasons why it has not been used as a baseline for experimental comparisons.**\n\n**A:** Thank you for the great suggestion. We will add the following paragraph in the related work section in the paper to cover the suggested literature:\n\nRobust Reinforcement Learning can be roughly categorized into addressing training-time robustness to poisoning attacks, and test-time robustness to evasion attacks [11]. Poisoning attacks are defined as the manipulation of elements in the training trajectory, including state, actions, and rewards in the data or online environment [1,2]. The adversary in this paper is similar to the Man In The Middle (MITM) attack formulation in [2]. Robust IQL [3] and RL agent with survival instinct [4] have demonstrated robustness to data poisoning in an offline setting. However, they only aim at the training-time robustness, while testing the algorithms without any attack. Our method is closer to ROLAH [5] and other related works [6-9], considering adversary appearing in both training-time and test-time serving evasion attacks. Despite the similarity,these methods are all online. Our method learns to achieve the robustness offline, which is more challenging due to potential distribution shift, and the lack of good behavior policy. In this paper, We didn't select offline robust RL as baseline since we only focus on the Reinforcement Learning via Supervised Learning (RvS) methods. We study the robustness of Decision Transformer (DT), and thus provide more analysis on DT-based algorithms including vanila DT and ESPER in various offline learning settings and environments.\n\n------\n\n**Q: This paper does not consider randomness, whereas random attacks are common in the real world. The authors have already clarified this limitation in the paper.** \n\n**A:** We do consider random attacks in our paper by modeling an adversary who can have a stochastic policy (see texts below Eq. (1)). We only claim the limitation of deterministic transition, rather than the deterministic policy of adversary. \n\n- In our experiment section 4.2 of our paper, we have experiments against random attack. The test-time adversary attack in our game environment Connect Four is an $\\epsilon$-greedy policy (demonstrated in the first paragraph), which is a stochastic policy. The results have shown superior performance than existing DT-based methods. \n\n- To further examine this, we added additional results in the rebuttal PDF Figure 3 against stochastic adversary. According to the figure, ARDT in this single-stage and multi-stage game outperforms the baselines at different levels of stochasticity.\n\n------\n\n**We appreciate the reviewer's questions and suggestions, and we believe we have addressed all of them, as well as any misunderstandings raised by the reviewer. In light of the reviewer's recognition of the strengths of our work and our detailed rebuttal to their questions, we kindly ask the reviewer to reconsider their score.**\n\n------\n\n[1] Optimal Attack and Defense for Reinforcement Learning, AAAI 2024\n\n[2] Understanding the Limits of Poisoning Attacks in Episodic Reinforcement Learning, IJCAI 2022\n\n[3] Towards Robust Offline Reinforcement Learning under Diverse Data Corruption, ICLR 2024\n\n[4] Survival Instinct in Offline Reinforcement Learning, NeurIPS 2023\n\n[5] Robust Reinforcement Learning through Efficient Adversarial Herding, arXiv\n\n[6] Tessler, Chen, Yonathan Efroni, and Shie Mannor. \"Action robust reinforcement learning and applications in continuous control.\" International Conference on Machine Learning. PMLR, 2019.\n\n[7] Kamalaruban, Parameswaran, et al. \"Robust reinforcement learning via adversarial training with langevin dynamics.\" Advances in Neural Information Processing Systems 33 (2020): 8127-8138.\n\n[8] Pinto, Lerrel, et al. \"Robust adversarial reinforcement learning.\" *International conference on machine learning*. PMLR, 2017.\n\n[9] Vinitsky, Eugene, et al. \"Robust reinforcement learning using adversarial populations.\" *arXiv preprint arXiv:2008.01825* (2020).\n\n[10] Moos, Janosch & Hansel, Kay & Abdulsamad, Hany & Stark, Svenja & Clever, Debora & Peters, Jan. (2022). Robust Reinforcement Learning: A Review of Foundations and Recent Advances. Machine Learning and Knowledge Extraction. 4. 276-315. 10.3390/make4010013.\n\n[11] Wu, Fan, et al. \"Copa: Certifying robust policies for offline reinforcement learning against poisoning attacks.\" *arXiv preprint arXiv:2203.08398* (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Jg6RZ5ZitN", "reviewer": "Reviewer_p4QE", "summary": "The paper tackles the worst-case-aware RL problem, revises the conventional DT formulation via minimax returns, and proposes the adversarial robust DT to enhance robustness against test-time adversaries.", "questions": "- Can the ARDT be compared to baselines regarding the normal case, not only the worst case. It could be interesting to see the performance drop in the normal case, and judge where it is worth sacrificing the performance in conventional cases for the improvement in the worst case.\n- Are there any real-world applications where it is necessary to consider a worst-case-aware RL algorithm? It could further highlight the significance of the proposed method if some examples could be provided.", "limitations": "- The application to real-world scenarios could be limited to some extent, as the learned policy might be too conservative.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The formulation of adversarial robust DT with minimax return is sound and clear. \n- Improving the robustness against test-time adversaries is critical for real-world deployment of RL.\n- The highlight and superiority of the proposed method is verified by comprehensive experiments.", "weaknesses": "- Two additional returns-to-go networks are needed for return re-labeling, which increases the computation burden. \n- The trade-off between robustness to adversaries and the policy conservativeness is not discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments and suggestions. We address the concern as follows:\n\n------\n\n**Q: Two additional returns-to-go networks are needed for return re-labeling, which increases the computation burden.**\n\n**A:** The increase in computation is relatively low. And it is worthwhile for the significant benefits it brings, as demonstrated by the performance improvements throughout the experiment section. \n\n- Training returns-to-go networks is a one-time process. Once relabeling is done, the new dataset can be used for training different large models (e.g., Transformer) for all return-conditioned RvS methods. Additionally, since these returns-to-go networks are small models such as MLPs, the GPU memory cost is relatively low.\n- Training small models (e.g., MLPs) to guide the training of large models (e.g., Transformers) is a well-established approach. As we discussed in our related work, both ESPER and Q-Learning DT train small neural networks (e.g. MLP) to predict expected return, and then train DT conditioned on the predicted returns. These methods have shown significant improvements over vanilla Decision Transformer. Apart from Decision Transformer, Diffusion models for control problems also requires additional value networks to guide the training [2,3].\n\n------\n\n**Q: The trade-off between robustness to adversaries and the policy conservativeness is not discussed.**\n\n**A:** Thanks for this great suggestion! Please refer to our 1st response to reviewer JwFn. We have shown that in Figure 1(c) that  a stochastic adversary and in Figure 2 that a stochastic environment, too much conservativeness does lead to suboptimal performance. In this case, such a trade-off is crucial, while can be addressed by tunning the expectile level $\\alpha$. \n\n------\n\n**Q: Can the ARDT be compared to baselines regarding the normal case, not only the worst case. It could be interesting to see the performance drop in the normal case, and judge where it is worth sacrificing the performance in conventional cases for the improvement in the worst case.**\n\n**A:** Thank you for this suggestion. It is crucial to consider the trade-off between normal performance and achieved robustness with ARDT. We must first clarify the setting in our paper. The assumed performance drop in the normal (non-adversarial) case can only possibly occur when our method is trained with non-perturbed (clean) dataset and tested without attack. While our study assumes that adversarial perturbations are present both during training and testing.\n\nThus here we only study the performance change of algorithms trained with our adversarial dataset on Hopper, and tested in the normal setting. We provided the experimental results in Table 1 in the rebuttal PDF. To recover the normal case in testing, we set the test-time adversary always taking action zero, i.e. no perturbation on actions. The results show that although ARDT has a lower average return in the normal setting, it demonstrates a smaller performance drop compared to the baselines.\n\n------\n\n**Q: Are there any real-world applications where it is necessary to consider a worst-case-aware RL algorithm? It could further highlight the significance of the proposed method if some examples could be provided.**\n\n**A:** Thanks for the question! We provide a few examples and will include them in the paper.\n\nWhen deploying reinforcement learning (RL) methodologies in real-world, safety-critical applications, it is crucial to to account for the worst-case scenarios to prevent unexpected performance drops or safety risks. This approach ensures the system's robustness and reliability under a range of adverse conditions, effectively mitigating the potential for catastrophic failures.\n\nOne example is as discussed in the introduction of our paper, in the context of autonomous vehicles, considering worst-case scenarios is essential for safe navigation. The algorithm must be capable of handling sudden changes in road conditions, such as unexpected obstacles or extreme weather, as well as making safe decisions in complex interactive scenarios involving multiple (autonomous) vehicles. There are many other real-world control system utilizing RL requires safe and effective operations. In medical applications, such as surgical and assistive robots for patients treatment, preparing for worst-case scenarios—like unexpected drug interactions or patient non-compliance—enhances safety and efficacy. Similarly, in aerospace application, perturbations, such as turbulence, ensures safer aircraft control. Additionally, in the management of plasma states in nuclear fusion reactors using RL methods [1], it is critical to consider the worst-case outcomes of actions to maintain safe and stable operations. \n\n------\n\n**Q: The application to real-world scenarios could be limited to some extent, as the learned policy might be too conservative.**\n\n**A:** As we discussed above, we can tune the expectile level parameter alpha to adjust the level of conservativeness. Besides, we also had empirical evidence in our paper indicating that this issue isn't serious. Conservativeness is only problematic when the test-time adversary is weaker than the training-time adversary, causing our method to act suboptimally due to over-conservativeness. While according to Table 1 in our paper, where ARDT is tested under different adversarial perturbation, we can observe that when the adversary is weak as 30% optimal, we can still outperform the baselines.\n\n------\n\n[1] Degrave, Jonas, et al. \"Magnetic control of tokamak plasmas through deep reinforcement learning.\" *Nature* 602.7897 (2022): 414-419.\n\n[2] Psenka, Michael, et al. \"Learning a diffusion model policy from rewards via q-score matching.\" *arXiv preprint arXiv:2312.11752* (2023).\n\n[3] Wang, Zhendong, Jonathan J. Hunt, and Mingyuan Zhou. \"Diffusion policies as an expressive policy class for offline reinforcement learning.\" *arXiv preprint arXiv:2208.06193* (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their thoughtful comments and insights. We have revised our manuscript based on your comments and suggestions, and we have responded to each of your individual comments.

## Manuscript Revision Summary

- Typo fixed suggested by reviewer JwFn.
- Added related works suggested by reviewer 18ip.
- Added real-world examples that require considering the worst-case performance, suggested by reviewer p4QE.
- Included all further experiments.

## Further Experiments

- Ablation studies of expectile level $\alpha$ in single-stage game, multi-stage game and Connect Four.
- New results in stochastic environment.
- New results in deterministic games with stochastic adversarial perturbation.
- New results in Hopper tested with no adversarial perturbation .

Meta-review: Interesting piece of work that presents a new algorithm, Adversarial Robust Decision Transformer (ARDT), for scalable offline RL in two-player zero-sum games. Empirical evaluations in full data coverage setting demonstrate the soundness of the proposed method. Good paper.
Decision: Accept (poster)