Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modeling
Authors: Jiawei Xu
Rui Yang
Shuang Qiu
Feng Luo
Meng Fang
Baoxiang Wang
Lei Han
Venue: ICLR 2025 Poster (2025)
Keywords: Offline Reinforcement Learning
Data Corruption
Robust Reinforcement Learning
Primary area: reinforcement learning

Abstract: Learning policy from offline datasets through offline reinforcement learning (RL) holds promise for scaling data-driven decision-making while avoiding unsafe and costly online interactions. However, real-world data collected from sensors or humans often contains noise and errors, posing a significant challenge for existing offline RL methods, particularly when the real-world data is limited. Our study reveals that prior research focusing on adapting predominant offline RL methods based on temporal difference learning still falls short under data corruption when the dataset is limited. In contrast, we discover that vanilla sequence modeling methods, such as Decision Transformer, exhibit robustness against data corruption, even without specialized modifications. To unlock the full potential of sequence modeling, we propose **R**obust **D**ecision **T**ransformer (**RDT**) by incorporating three simple yet effective robust techniques: embedding dropout to improve the model's robustness against erroneous inputs, Gaussian weighted learning to mitigate the effects of corrupted labels, and iterative data correction to eliminate corrupted data from the source. 
Extensive experiments on MuJoCo, Kitchen, and Adroit tasks demonstrate RDT's superior performance under various data corruption scenarios compared to prior methods. Furthermore, RDT exhibits remarkable robustness in a more challenging setting that combines training-time data corruption with test-time observation perturbations. These results highlight the potential of sequence modeling for learning from noisy or corrupted offline datasets, thereby promoting the reliable application of offline RL in real-world scenarios.
Our code is available at https://github.com/jiawei415/RobustDecisionTransformer。

Published as a conference paper at ICLR 2025
TACKLING DATA CORRUPTION IN OFFLINE REIN- FORCEMENT LEARNING VIA SEQUENCE MODELING
Jiawei Xu Rui Yang** Shuang Qiu Feng Luo" Meng Fang² Baoxiang Wang' Lei Han
The Chinese University of Hong Kong, Shenzhen Uniiversity of Illinois Champpan 3 City University of Hong Kong *Rice University University of Liverpool Teencett Robotics X
ABSTRACT
Learning policy from offline datasets through offline reinforcement learning (RL) holds promise for scaling data-driven decision-making while avoiding unsafe and costly online interactions. However, real-world data collected from sen- humans often contains noise and errors, posing significant challenge for existing offline RI. methods, particularly when the real-world data is lim- ited. Our study reveals that prior research focusing on adapting predominant offline RL methods based on temporal difference learning still falls short un- der data corruption when the dataset limited. In contrast, we discover that vanilla sequence modeling methods, such as Decision Transformer, exhibit ro- bustness against data corruption, even without specialized modifications. To unlock the full potential of sequence modeling, we propose Robust Decision Transformer (RDT) by incorporating three simple yet yet effective robust techniques: embedding dropout to improve the model's robustness against erroneous inputs, Gaussian weighted learning to mitigate the effects corrupted labels, and iter- ative data correction to climinate corrupted data from the source. Extensive periments on MuJoCo, Kitchen, and Adroit tasks demonstrate RDT's superior performance under various data corruption scenarios compared to prior meth- ods. Furthermore, RDT exhibits remarkable robustness in a more challenging setting that combines training-tinne data corruption with test-time observation perturbations. These results highlight the potential of sequence modeling for learning from noisy or corrupted offline datasets, thereby promoting the reli- able application of offline RL in real-world scenarios. Our code is available at https///ghhhub.oooojjaee/4 oenisasesn
INTRODUCTION
Offline reinforcement learning (RL) aims to derive near-optimal policies from fully offline datasets (Levine al.) 2020 Kumar et al.) 2020 Fujimoto al. [2019] Wang et al.] [2018), thereby reducing the need for costly and potentially unsafe online interactions with the environment. However, offline
RL encounters significant challenge known distribution shift (Levine al.)
(020)), which can
lead to performance degradation. To address this challenge, several offline RL algorithms impose
policy constraints (Wang al. 2018 Fujimoto et al. 2019 Fujimoto & Gu| 2021 Kostrikov al.
2021
Park et al. (224)) or maintain pessimistic values for out-of out-of-distribution (OOD) actions (Kumar
al. 2020 An et al. 2021 Bai et al., 2022 Yang et al.) 2022a Ghasemipour al. 2022). ensuring
that the learned policy aligns closely with the training distribution. Although the majority of tradi-
tional offline RL methods rely on temporal difference learning, an alternative promising paradigm
for offline RL emerges in the form of sequence modeling (Chen et al.) 2021 2 Janner et al. [021 Shi
ct al.
2023 Wu et al., 2024). Unlike traditional RL methods, Decision Transformer (DT) (Chen et al.
2021) representative sequence modeling method, treats offline RL as supervised learn- ing task, predicting actions directly from sequences of reward-to-gos, states, and actions. Prior work (Bhargava al. 2023) suggests that DT excels at handling tasks with sparse rewards and suboptimal quality data, showcasing the potential of sequence modeling for real-world applications.
"Equal Contribution. Correspondence to Jiawei Xu(jiawar 20201 60gma .com) and Rui
Yanglyangrui thu200 10:0015550 58gmai 1ccmm.

---

Published as a conference paper at ICLR 2025
When deploying offline RL in practical settings, dealing with noisy corrupted data resulting from data collection or malicious attacks is inevitable (Zhang et al.
2020: 2021 Liang et al. 2024 Yang
jtt al., 2024a). Consequently, robust policy learning from such data essential for the successful de-
ployment of offline RI. A series of prior works (Zhang et al. 2022 Ye et al., 2024b Wu et al. 2022
Chen et al. 2024 Ye al.] 2024a) focus on the theoretical properties and certification of off RT.
under data corruption. Notably, Ye et al. (2024b) propose an uncertaintywwiigheed offline RL al- gorithm with Q ensembles address reward and dynamics corruption. while Yang et al. (2024c)
enhance the robustness of Implicit Q-Learning (IQL) (Kostrikov et al. (202T) against corruptions on
all elements using Huber loss and quantile Q estimators. These advancements have primarily con- centrated on temporal difference learning, dedicating significant effort to learning robust Q functions from corrupted data. This approach brings up an intriguing question: by bypassing the Q function, can sequence modeling methods effectively handle data corruption in offline RI.?
In this study, we conduct a comparative analysis between prior robust offline RL methods and DT (without any robust modifications) under various data corruption settings. Surprisingly, we find that traditional methods relying on temporal difference learning struggle in settings with limited data and significantly underperform compared to DT. Additionally, DT demonstrates superior performance over previous methods in scenarios involving state attacks. These findings highlight the promising potential of sequence modeling for addressing data corruption challenges in offline RI. To further unlock the capabillitess of sequence modeling methods under data corruption, inspired by successes in the supervised learning domain (Song et al., 2022] Chang et al., 2017 Reed et al.. 2014 Gal & Ghahramani 2016), we propose Robust Decision Transformer (RDT). RDT aims to boost the ro- bustness of DT in three aspects: model structure, loss function, and data refinement. To achieve this, RDT incorporates three simple yet highly effective techniques to mitigate the impact of corrupted data: embedding dropout, Gaussian weighted learning, and iterative data correction. These simple modifications are designed to address the fundamental limitations of the standard DT and unlock the potential of sequence modeling for data corruption.
To comprehensively study RDT's robustness under data corruption with limited data, we constructed a variety of small datasets from MuJoCo, Kitchen, and Adroit, creating a challenging benchmark for current robust offline RL methods. Through our experiments, we demonstrate that RDT outper- forms conventional temporal difference learning and sequence modeling approaches under both ran- dom and adversarial data corruption scenarios, achieving substantial 29% improvement in overall performance over DT. Furthermore, we show that after learning from corrupted data, RDT exhibits remarkable robustness against testing-time observation perturbations, indicating its potential to ad- dress both training-tinn tamein and test-time attacks simultaneously. Our study emphasizes the significance of robust sequence modeling and offers valuable insights for the trustworthy deployment of offline RL in real-world applications.
2 PRELIMINARIES
RL and Offline RL. RL is generally formulated as a Markov Decision Process (MDP) de- fined by tuple (S, A, P.r.Y). This tuple comprises state space S, action space A, a transition function P, a reward function r, and a discount factor 7 E [0. The ob- jective of RL is to learn a policy ([as)) that maximizes the expected cumulative return: max,
TT >/r((s, at) where Po denotes the distribution of Li
initial states and T is the trajectory length. In offline RL, the objective is to optimize the RL objec- tive with a previously collected dataset D = {(0])) (4) (s) (4) (6) 7- J,N-1 which contains a total ; aa .rt #(41)
of N trajectories. The agent cannot directly interact with the environment during the offline phase.
Decision Transformer (DT). DT models decision-making from offline datasets sequence modeling problem. The i-th trajectory (() of length T in dataset D is reorganized into a sequence of return-to-go 12(0) state (4) action at (6).
T (
Here, the retum-to-go R(()) is defined as the sum of rewards from the current step to the end of
the trajectory: R².") Dr= rg. DT employs three linear projection layers to project the return-
2

---

Published as a conference paper at ICLR 2025
to-gos, states, and actions to the embedding dimension, with additional learned embedding for each timestep added to each token. A GPT model adopted by DT to autoregressively predict the actions di
(t) with imput sequences of length K:
Ke ,/ 1
CDr WWD
where (6) T indicates the segment of 7(0) from timestep t K to
Data Corruption in Offline RL. In prior works (Ye et al. 2024b Yang et al. 2024c), the data is stored in transitions, and data corruption is performed on individual elements (state, action, reward, next-state) of each transition. This approach does not align well with trajectory- sequence modeling methods like DT. this paper, we consider a unified trajectory- corrupting next-state in transition corresponds corrupting state in the subsequent transition, while corruption for rewards and actions is consistent with prior works. To elaborate, an original
storage, where
trajectory denoted Vorigin
(80 00
(5) (8) (8)
(8)
(8) TT-1 which can be reorganized
for MDP-based methods. Note that here only three independent elements (i.e., states, actions, and rewards) under the trajectory- based storage formulation.
into the sequence data of DT in Eq. or split into T transitions (s)
(2) (6)
t=
Data corruption injects random adversarial noise into the original states, actions, and rewards. Random corruption adds random noise to the affected elements the datasets, resulting a 5(4) s a- -f-.) For instance,
corrupted trajectory denoted as Tcorrupt
(i)
80 an
(8) ((f)
8 = + std(s), À Uniform([c,c]*" (d, is the dimensions of state, and is the corruption scale) and std(s) is the d--dimensional standard deviation of all states in the offline dataset. In contrast, adversarial corruption uses Projected Gradient Descent attack (Madry al. 2017) with pretrained value functions. Specifically, we introduce learnable noise to the states or actions and then optimize this noise by minimizing the pretrained value functions through gradient descent. More details about data corruption are provided in Appendix C.I
3 SEQUENCE MODELING FOR OFFLINE RL WITH DATA CORRUPTION
We aim to answer the question of whether sequence modeling methods can effectively handle data corruption in offline RL in Section 3.1 To achieve this, we compare DT with prior offline RL methods in the context of data corruption. Based on the insights gained from the motivating example, we then propose enhancements for improving the robustness DT in Section 3.2
State Attack
Action
Howard Attack
00
Nomalooe
60
100% Dotsset
10% Donaset
100% Docacct
10% Dotaest
100% Detaset
10% Detaset
100% Dataset
10% Dataset
MBC
LOE
UMMSG
Figure 1: Average normalized scores of offline RL algorithms under random data corruption across three MuloCo tasks (halfcheetah, walker2d, and hopper) using mmeiium-replay-v2 datasets. Many offline RL algorithms experience substantial performance declines when subjected data comup- tion. In contrast, DT demonstrated remarkable robustness, particularly in the 10% data regime.
MOTIVATING EXAMPLE
As illustrated in Figure compare DT with various offline RL algorithms under data corruption. We apply random corruption introduced in Section (2 on states, actions, and rewards. Specifically, perturb 30% of the transitions in the dataset and introduce random noise scale of 1.0 standard
3

---

Published as a conference paper at ICLR 2025
Offliie Datasee
Iterative Data Correction
RCEx
LROT
Robust Decision Transformer
Dropout
H
R
R
d
REL
fc+11
Framework of Robust Decision Transformer (RDT). RDT enhances the robustness of DT against data coruptiio by incorporating three components on top of DT: embedding dropout, Gaussian weighted learning, and iterative data correction.
Figure 2:
deviation. In addition to the conventional full dataset setting employed in previous studies (Ye et al. 2024b Yang et al. 2024c), we also explore scenario with a reduced dataset, comprising only 10% of the original trajectories. Our results reveal that most offline RL methods are highly vulnerable to data corruption, partieularly in more challenging. limited data settings. In contrast, DT exhibits remarkable robustness to data corruption, especially in scenarios involving state corruption and lim- ited dataset conditions. This resilience can be attributed DT's formulation based on sequence modeling and its supervised training paradigm. We will delve deeper into the critical components affecting DT's robustness in Appendix(D.I D.I
Overall, the results demonstrate DT can outperform prior imitation-basee and temporal difference based methods without any robustness enhancement. This observation raises an interesting question:
How can we further unleash the potential of sequence modeling in addressing data corruption with limited dataset in offline RL?
3.2 ROBUST DECISION TRANSFORMER
To improve the robustness DT against various data corruptions, we draw inspiration from suc- cesses in the supervised learning domain (Song al. 2022 Chang et al., 2017, Reed et al.) 2014 Gal & Ghahramani] 2016) to introduce the Robust Decision Transformer (RDT). RDT enhances DT across three key aspects: model structure, loss function, and data refinement. This is achieved through the incorporation of three simple yet effective components: embedding dropout (Section 3.2.1. Gaussian weighted learning (Section (.2.2), and iterative data correction (Section 3.2.3 These simple modifications are designed to tackle the fundamental limitations of the standard DT and help unlock the potential of sequence modeling for data corruption. The overall framework is illustrated in Figure Notably, RDT predicts both actions and rewards (instead of reward- -oraardtt-gee to-gos). Given the typically high dimensionality of states, we avoid predicting states to mitigate potential negative impacts as discussed in Appendix D.14] Additionally, rewards provide more direct super- vision for the policy compared to reward-to-gos. which also depend on future actions.
3.2.1 EMBEDDING DROPOUT
In the context of data corruption, corrupted states, actions, and rewards can introduce shifted inputs errоneous features to the model. This can lead to performance drop if the model overfits these harmful features. Therefore, developing a robust representation is crucial to enhance the model's resilience against data corruption.
4

---

Published as a conference paper at ICLR 2025
walker2d medium v2
2d medium replay x2
MSE between Corrected and Orginal Data
vallee 23 rnd raplay
Eprade Rewndd
kinchen ste-e)
D Comptes Dets
OT Las cn Gtn Date
MSE
expert.vd
(TT Gif Trese Comptee fute
0.4
0" GI Lee oe Clas Dya
ic. ton shint copport 13
OF ic, soete entedding mg dropoct
200
Fpoch
Epoch
Epoch
(a)
(b)
(c)
Figure 3: (a) Comparing dropout methods under state attack: Embedding dropout outperforms
directly dropping the entire state (DeFog) or dropping dimensions on the raw state. (b) Gaussian
weighted learning under action attack: Gaussian weighted learning (DT w. GWL) alleviates overfitting to the corrupted data and slightly minimizes the loss on clean data. (c) Iterative data correction (DT w. IDC) ) under action attack: The MSE between corrected and oracle data grad- ually decreases to near zero.
Several studies (Carroli et al. 2022 Hu et al. 2023) utilize element-wise masking to learn robust representations by randomly masking state, return-to-go, or action elements in trajectory, lever- aging the redundancy in the trajectory's information. However, we discovered that corrupted trajectories contain less redundant information. Discarding raw elements directly can lead to notable information loss, ultimately resulting in decreased performance (see Figure ()). To address this, we propose randomly dropping dimensions in the feature space rather than masking raw elements. Specifically, we employ embedding dropout, which encourages the model to learn more robust embedding representations while preventing overfitting (Merity et al. 2017).
To clarify, we define three hidden emheddings after the linear projection layer as hng hac and hap formulated as:
hR. = ÓR(RR) + (t)), ha = (.(ss + (t(,), ha, = (a(a)) + @t()).
(3)
Here, dn. os and oh denote linear projection layers on different elements, while (x(() represents the time-step embedding. Subsequently, we apply randomized dimension dropping on these feature embeddings, resulting in the corresponding masked feature embeddings as follows:
hR. = M()) O hR. h. = M(pp) @ he:: ha, =M(p) 3 har
(4)
The function M(-) takes probability p E 1) input and outputs a binary mask with the same shape as embeddings to determine the dimension inclusion. We apply the same drop probabilities p within moderate range (0.1,0.3) for different element embeddings.
3.2.2 GAUSSIAN WEIGHTED LEARNING
In addition to mitigating the negative impact of erroneous inputs, it is also essential to address the influence of corrupted labels. In DT, actions serve as the most crucial supervised signals, acting as the labels. Therefore, erroneous actions can directly influence the model through backpropagation. In RDT, we predict both actions and rewards, with rewards helping to reduce overfitting to corrupted action labels. To further diminish the impact of corrupt data, we focus on minimizing the influence of unconfident action and reward labels that could misguide policy learning.
To identify uncertain labels, we adopt a simple but effective method: use the value of the sample- wise loss to adjust the weight for the DT loss. The underlying insight is that a corrupted label would typically lead a larger loss (see Figure 3b)). To softly reduce the effect of potentially corrupted labels, we use the Gaussian weight, i.e., a weight that decays exponentially in accordance with the sample's 's loss. This is mathematically formulated as:
(6) war e Bs 82, w - 67 where da, no-grad (I| TTo (1) (1 1+ ((0) 2.10) -a..0 (12)
12),
8,, = no grad (1) (-K+1=-1 (2(00 (0) a. +100 (22)
(5) In 5 8a, and 6, represent prediction errors at step t with detached gradients. The variables Ba 2 0, 3, _ 0 act as the temperature coefficients, providing flexibility to control the "blurring" effect of the Gaussian weights. Different tasks exhibit preferences for different temperature coefficients, as
5

---

Published as a conference paper at ICLR 2025
demonstrated in Appendix D33 loss function of RDT expressed as follows:
With larger value, more samples would be down- The
(RD Epiti
1
K
Σ t=
w
[T(
())) al
K
(6)
(1) T
||
R
Gaussian weighted learning enables us mitigate the detrimental effects of corrupted labels, thereby enhancing the algorithm's 's robustness.
3.2.3 ITERATIVE DATA CORRECTION
While the first two techniques have significantly reduced the impact of corrupted data, these erro- neous data can still affect the policy as they remain the datasets. We propose iteratively correcting corrupted data in the dataset using the model's predictions to bring the data closer to their true val- ues in the next iteration. This method can further minimize the detrimental effects of corruption and can be implemented to correct reward-to-go, state, and action elements. In RDT, it is straight- forward to correct the actions and rewards in the dataset and recalculate the reward-to-gos using corrected rewards. Therefore, our implementation focuses correcting actions and reward-to-gos in the datasets, leaving better data correction methods for states for future work.
Initially, we store the distribution information of prediction error å in Eq throughout the learning phase to preserve the mean μ4 and variance o3 of actions and rewards, updating them with every batch of samples. The hypothesis is that the prediction error 8 between predicted and clean labels should exhibit consistency after sufficient training. Therefore, erroneous label actions are encoun- tered, ó will deviate from the mean μ4s, behaving like outliers. This deviation essentially enables us to detect and correct the corrupted data using the information of 6.
Taking actions (1) 8(") for each sampled action ( If the condition (1) Ç- is met for any given action ds
example, detect corrupted actions, calculate the z-score, denoted by
(00), we infer that action (00 has been corrupted. We then permanently replace (()) in the dataset with the predicted action. This helps eliminate corrupted actions from the dataset (see Figure B(c)). Correcting rewards follows a similar process, with the additional step of recalculating the reward- to-gos. The hyperparameter Ç determines the detection thresholds: smaller Ç will classify more samples as corrupted data, while a larger Ç results in fewer modifications to the dataset. Empirically, achieve performance improvement by setting Ç approximately 5. More implementation details of RDT can be found in Appendix C.3
4 EXPERIMENTS
In this section, conduct comprehensive empirical evaluation of RDT, focusing the following key questions: 1) How does RDT perform under different data corruption scenarios? 2) Is RDT robust observation perturbations during the testing phase? 3) What is the contribution of each component of RDT to its overall performance? 4) How do baselines compare to RDT when equipped with transformer backbone?
4.1 EXPERIMENTAL SETUPS
We evaluate RDT using various offline RI. benchmarks, such as MuJoCo, KitChen, and Adroit (Fu et al. 2020). Two types of data corruption during the training phase are simulated: random and adversarial corerupion.... attacking states, actions, and rewards. Specifically, the corruption rate is set to 30%, and the corruption scale is 1.0 for the main results, similar to previous work (Yang et al. 2024c). Data corruption is introduced in Section 2, and more details can found in Appendix C.I We find that the data corruption problem be exacerbated when the data is limited. To simulate such a scenario, we down-sample on both MuJoCo and Adroit tasks, referred to as MuJoCo (10%) and Adroit (1%), Specifically, randomly select 10% (and 1 %) of the trajectories from MuJoCo

---

Published as a conference paper at ICLR 2025
Table 1: Detailed comparative results under random data corruption.
Attack
Task
BC
RBC
DeFog
CQL
UWMSG
RIQL
DT
RDT
halicheelah (10%)
1.9.0.2
1.710.5
6.710..5
9.7
3.910.5
4.410.9
6.3. 0.4
8.311.5
hopper (10%)
1..4
13.0-2.33
23.1
2.2
16.3174
15.545.4
36.1-7.6
40.8 3.5
walker2d (10%)
7,5+0.9
10.3+1.4
9.043.33
12-118
-04+002
9.244
18.0-2.5
20.3+2.8
kitshen-complete
15.9=5.2
21.1+3.1
37.1=1344
129=4.3
0.0+0.0
37,5+6.4
37.0=6.2
52.8=1.8
kitchen-partial
17.1=2.6
20.242..8
9.846.6
0.0=0.0
0.0::000
25.943.4
31.0=8.1
36.8=5.8
State
kitchen-mixes
17.4+4.9
29.245.6
10.6+55.0
0.0-0.0
0.0+0.00
21.6+3.7
31.8-34
41,8-4.3
door (1%)
15.7=117
73.349.2
101.3
-0.3=000
-0.20022
39.0
94.6=4.2
102.8:24
bammer (1%)
90.5=7.8
90.6+3.5
93.0=15.8
0.2=0.0
-0.0=0.1
70.0=12.6
91.8=12.3
113,8=1.6
relocate (1%)
7.224.5
12.246.88
14.713.7
-0.200.0
-0.31:00
5.245.0
61.6-5.6
65.0
Average
27.2
30.2
33,9
2.8
2.2
25.4
46.0
53.6
balfcheetah (10%)
0.8=0.3
0.7+0.1
10.3+18
19.8=5.1
8,0+0.6
2.340.6
3,6=0.3
10.4=1.7
hopper (10%)
18.3=2.4
17.742.3
32.8+3.9
18=0.
33.02:43
26.342288
32.1=6.0
37.625.7
walker2d (10%)
5.1=1.4
6.143.55
13,919.6
3.1=24
6.410.5
9,81199
22.2-4.0
2.,4
kitchen complete
3.5120
4.6.11.2
23.0114.5
12.6.4.2
0.0.0.00
20.116.1
12.6 5.2
44.0
kitchen-partial
29.9=3.0
34.0#3.55
5,922.5
5,4
0.0=0.0
33.4+5.6
26.2=0.8
40.8=3.7
Action
kitchen-mixed
34.923.8
39.514.55
17.017.7
0.0:0.0
41.0:10.4
30.4-2.2
44.4
dour (1%)
45.0
614119.1
100.713.1
0.3.0.1
0,2.0.11
29.61129.1
84.3
103.311.
hammer (1%)
15.4
67.8-66.9
99.7+-17.8
0.2-0.0
0,14-0.1
68.74-29.4
72.0 5.8
116.3-0.8
relocate (156
26.0=10.4
44.347.44
44,94266
002-0.11
-0,3+-0.0
20.4+-9.6
63.2-4.4
69.9-9.77
Average
26.6
30.7
38.7
5.2
28.0
38.5
56.3
halicheetah (10%)
2.410.2
2.9.0.33
14.313.6
1.94
2.210.7
6.1
9.3_ 0.9
21.7.4.0
hopper (10%)
19.742.8
19.342233
22.84-42
1.80.00
18.24-2.8
40.2442.66
36.0
42.8 3.4
walker2d (10%)
9.7+1.5
8.343.8
14,24-2.1
12-007
7,9+0088
9.64-2.5
27.4-3.4
30.1-4.5
kitchen complete
36.0111.5
38.2.18.4
43,215.7
14.24
0.0.1..
52.8.6.8
43.9_4.3
65.6
kitchen partial
34.1+114
39.1+-3.2
7.9+-6.4
0.0-0.0
0.0+-0.0
36.4+-222
47.1-6.9
51.4-2.0 2.0
Reward
kitchen-mixed
38.9+144
47.1+1.99
14,947.2
0.0=0.0
0.0+-0.0
49.8++4.3
42.8-1.9
47.8++4.3
door (1%)
76.==5.9
75.0=332
102.3#27
-0.3=000
-0.2=0.0
56.129.3
99.0=2.3
103.7=08
hammer (1%)
97.128.3
99.0:1..1
101.8110.2
0.2=0.0
0.410.4
$2.24:22.7
80.7=11.2
123.8=2.2
relocate (1%)
36.128.6
32.246.7
53,113.1
-0.3=0.1
-0,3::000
9.14:5.3
71.8-7.7
84.9
Average
38.9
40.1
41.6
3.8
3.1
34.7
$0.9
63.5
Average over all tasks
31.1
33.7
38.1
3.8
3.5
29.3
45.1
57.8
(and Adroit) tasks and conduct data corruption on the sampled data. We refrain from down-sampling the Kitchen dataset due to already limited size. Detailed information about the dataset sizes can be found in Appendix C.4
We compare RDT with several SOTA offline RL algorithms and corruption-robust methods, namely BC. RBC (Sasaki & Yamashina 2020), DeFog (Hu et al.) 2023). CQL (Kumar et al., 2020 UWMSG (Ye et 20244)), RIQL Yang et al. 2024c), and DT (Chen al. 2021). BC and RBC employ behaviour cloning loss within an --se-ased model for policy learning, while DeFog and DT utilize a Transformer architecture. We implement RBC with our Gaussian-
learning as
an instance of (Sasaki & Yamashina 2020). In Appendix D.I we investigate the robustness of DT across various critical parameters. Drawing from these findings on DT's robustness, we establish a default implementation of DT, DeFog, RDT with sequence length of 20 and block number of 3. To ensure the validity of the findings, each experiment is repeated using 4 different random seeds, and we also report the standard variance over these random seeds.
4.2 EVALUATION UNDER VARIOUS DATA CORRUPTION
Results under Random Corruption. To address the first question, we first evaluate the RDT and baselines under the random data corruption scenario. The mean of normalized scores across differ- ent seeds is calculated as the evaluation criterion for each task. As shown in Table RDT demon- strates superior performance in handling data corruption, achieving the highest score in 24 out of 27 settings. Notably, across all tasks, RDT consistently outperforms DT, with significant overall improvement of 28.2%. underscoring its effectiveness in reducing DT's sensitivity to data corrup- tion. Moreover, RBC slightly outperforms BC, further highlighting the effectiveness of the Gaussian weighted learning approach. However, prior offline RL methods, such as RIQL and UWMSG, fail to yield satisfactory results and even underperform BC, indicating temporal difference methods' weakness in the limited data domain. We further evaluate RDT under varying corruption scales and ratios in Appendix DD4 where RDT demonstrates superior robustness compared to other baselines.

---

Published as a conference paper at ICLR 2025
Table Results under adversarial data corruption. Each score is averaged across the task group.
Attack
Task
BC
RBC
DeFog
CQL
UWMSG
RIQL
DT
RDT
MuloCo (10%)
9.9
9,4
12.9
4.2
9.0
11.0
21.9
23.6
KitChen
23.4
28.9
20.0
2.2
0.0
40.4
37.9
49.1
State
Adroit (1%)
60.2
53.6
75.6
-0.1
-0.1
44.5
85.3
95.4
Average
31.2
30.6
36.2
2.1
32.0
48.4
56.0
MuloCo (10%)
4.2
4.3
10.3
7.3
16.9
7.1
13.4
21.6
KitChen
6.2
11.5
3.9
0.9
8.0
5.4
34.0
Action
Adroit (1%6)
8.4
20.7
51.8
-0.1
-0.1
42.4
47.4
80.3
Average
6.3
12.2
22.0
2.7
5.6
19.2
22.1
45.3
MuJoCo (10%)
10.6
10.2
11.5
11.8
15.2
18.3
25.2
31.9
KitChen
36.3
41.5
20.3
1.1
9.7
45.9
48.1
56.0
Reward
Adroit (1%)
69.7
68.7
80.8
-0.1
0.0
53.6
90.9
96.5
Average
38.9
40.1
37.6
4.3
-0.1
39.2
54.7
61.5
Average over all tasks
25.5
27.7
31.9
3.0
4.4
30.1
41.7
54.3
Results under Adversarial Corruption. We further extend the analysis to examine the robustness of the RDT under adversarial data corruption scenario. As illustrated in Table calculate the mean score across the task group. RDT consistently demonstrates robust performance, achieving the highest average scores under various attacked elements. Notably, RDT improves the average score by 105% % compared to DT in the adversarial action corruption scenario. Intriguingly, temporal- difference methods like CQL and UWMSG perform significantly worse than both BC and sequence modeling methods such as DT and DeFog, underscoring the potential of sequence modeling ap- proaches. Detailed results for each task are provided in Appendix D for comprehensive analysis.
Results under Mixed Corruption. To present a more challenging scenario for assessing the ro- bustness of RDT, conduct experiments under mixed data corruption settings. In this setting, all three elements (states, actions, and rewards) are corrupted at a rate of 30% and a scale of 1.0. As shown in Figure 4 RDT consistently outperforms other baselines across all tasks, highlighting its superior stability even when faced with simultaneous and diverse data corruptions. Notably, in the challenging Kitchen and Adroit tasks with mixed adversarial settings, RDT surpasses DT and RIQI. by an impressive margin of approximately 100%.
BC
UWMSG
RIQU
DT
RDT
30
80
Normalized Score
60)
(8)
40
40
20
20
MuloCo(10%)
Kitchen Mixed Random Acttack
Adrait(1%)
MulaCo(10%)
Kitchen
Adrait(1%)
(b) Mixed Adversarial Attack
Figure 4: Results under (a) mixed random corruption and (b) mixed adversarial corruption.
4.3 EVALUATION UNDER OBSERVATION PERTURBATION DURING THE TESTING PHASE
We investigate the robustness of RDT when deployed in perturbed environments after being trained on corrupted data, a challenging setting that includes both training-time and testing-time attacks.
To address this, evaluate RDT under two types of observation perturbations during the testing phase: Random and Action Difr. following prior works (Yang et al. 2022a Zhang et al. 2020). Among these, Action Diff involves sampling multiple random noises to maximize the impact on the difference policy output. The perturbation scale is used to control the extent of influence on the observation. Details of observation perturbations are provided in Appendix C22
The comparison results are presented in Figure S where all algorithms are trained on the offline dataset with mixed random corruption and evaluated under observation perturbation. These results
8

---

Published as a conference paper at ICLR 2025
A1ro(155)
Airok1179)
Norre tood Sooe.
(a) Random observation perturbation.
(b) Action Diff' observation perturbation.
Figure 5: Performance under various observation perturbation scales during the testing phase. All
the algorithms are trained under mixed random corruption during the training phase.
demonstrate the superior robustness of RDT under the two types of observation perturbations, main- taining stability even at high perturbation scale 0.5. Notably, RORL (Yang et al., 2022a), a SOTA offline RL method designed tackle testing-time observation perturbation, fails these tasks due to data corruption. Additionally, DT and RIQL experience significant performance drops as the per- turbation scale increases. Additional results comparing algorithms trained on offline datasets with corrupted states, actions, or rewards can be found in Appendix D.7
4.4 ABLATION STUDY
We conduct comprehensive ablation studies to analyze the impact of each component on RDT's ro- bustness. For evaluation, we use the "ollker2-mcdduu--eepaa""" "kitchen-compete,, and "relocate- expert" datasets. Specifically, we compare several variants of RDT: (1) DT(RP), which incorporates reward prediction in addition the original DT; (2) DT(RP) w. ED, which adds embedding dropout to DT(RP); (3) DT(RP) w. GWL, which applies Gaussian weighted learning on top of DT(RP); and (4) DT(RP) w. IDC, which integrates only the iterative data correction method.
We evaluate the performance of these variants under different data corruption scenarios, depicted in F In summary, all variants demonstrate improvements over DT(RP), proving the effective- ness of the individual components. Notably, Gaussian weighted learning appears to provide the most significant contribution, particularly under reward attack. However, it is important to note that none of these tailored models outperform RDT, indicating that the integration of all proposed techniques is crucial for achieving optimal robustness. Additionally, while reward prediction could potentially lead to performance drops under reward attacks, it brings obvious improvements under action and state attacks for specific tasks. More ablation results about three individual component and reward prediction are provided in Appendix D.9 and Appendix D22
Stare Attadk
Artice
Drworr
witchon comalete
relocate
2d
kitchn complete
reoocate
kischon complete
relocate
(TTAP))
BT(RP)
DT(RP]
Figure 6: Ablation study on the impact of different proposed components of RDT.
4.5 IMPACT OF TRANSFORMER BACKBONE ON BASELINES
To study the impact of the transformer backbone on baselines, we adapted temporal difference learn- ing baselines such as UWMSG and RIQL by incorporating the same transformer backbone used in DT and RDT. Specifically, we modified UWMSG and RIQL by equipping their policy networks with the transformer backbone while keeping the value networks unchanged. These modified algorithms are referred to as UWMSG w. TB and RIQL w. TB.
As shown in Figure 7 while the transformer backbone does not improve UWMSG, it significantly enhances RIQL's 's overall performance, particularly in the challenging Adroit task. The ineffective- ness of the sequence model UWMSG is likely due the inherent difficulties of Q-value learning.

---

Published as a conference paper at ICLR 2025
State Attack
oction Attack
Reward Artack
80
Nomaleed Score
20
welker2d
kicchn-coomlltee
relocats
2e
kiccen-comppen
reocaaee
walker2d
kichen-complete
relocate
UWMSS
UMMSG
TB
RIQL
RIOL w.
DT
RDT
Figure 7: Ablation study on the impact of transformer backbone on baselines.
These comparative results suggest that merely adopting a transformer structure does not directly able these baselines to outperform RDT; in some cases, may even decrease performance.
5 RELATED WORK
Robust Offline RL. Several works have focused on testing-time robustness against environment
shifts (Shi & Chi 2022 Yang et al. 2022a; Panaganti et al., 2022 YANG & Xu 2024). Regarding training-time robustness, Li et al. (2023) explores various types reward attacks offline RL and finds that certain biases can inadvertently enhance the robustness of offline RL methods to reward corruption. From a theoretical perspective, Zhang et al. (2022) propose robust offline RL algo- rithm utilizing robust supervised learning oracles. et (2024b) employ uncertainty weighting to address reward and dynamics corruption, providing theoretical guarantees. |Ackerman et al. (2024) investigates a similar setting, focusing on trajectory-level non-stationarity both rewards and dy- namics. The most relevant research by Yang et al. (2024c) employs the Huber loss to handle heavy- tailedness and utilizes quantile estimators balance penalization for corrupted data, Mandal et al. 2024); [Liang et al. (2024) enhance the resilience of offline algorithms within the RLHF framework. It is important to note that these studies primarily focus on enhancing temporal difference methods, with no emphasis on leveraging sequence modeling techniques to tackle data corruption.
Transformers for RL. Recent research has redefined offline RL decision-making as a sequence modeling problem using Transformer architectures (Chen et al. 202 Janner et al. (021). A seminal work, Decision Transformer (DT) (Chen et al. 2021). uses trajectory sequences to predict subse- quent actions. Trajectory Transformer (Janner al. 2021) discretizes input sequences into tokens and employs beam search to predict the next action. These efforts have led to subsequent advance- ments. For instance, Prompt DT (Xu al., 2022) integrates demonstrations for better generaliza- tion, while Xie et al. (2023) introduces pre-training with future trajectory information. Q-learning DT (Yamagata al. 2023) refines the return-to-go using Q-values, while the Agentic Transformer
(Liu & Abbeel (023) employs hindsight to relabel target returns. Additionally, several other works
(Hu et al. [2024] Zhuang et al. (0224) utilize learned value functions to improve policy optimization.
RIC ( Yang et al. 2024b) extends DT to handle multiple objectives and applies it to large foundation models. LaMo Shi et al., 2023) leverages pre-trained language models for offline RL, and DeFog (Hu et al. 2022) addresses robustness in specific frame-dropping scenarios. Our work deviates from
these approaches by focusing on improving robustness against data corruption in offline RL.
CONCLUSION
In this study, we investigate the robustness of offline RL algorithms under various data corruption scenarios, with a specific focus on sequence modeling methods. Our empirical evidence suggests that current offline RL. algorithms based on temporal difference learning are significantly suscepti- ble to data corruption, especially in scenarios with limited data. To address this issue, we introduce the Robust Decision Transformer (RDT), a novel robust offline RL algorithm developed from the perspective of sequence modeling. Our comprehensive experiments highlight RDT's excellent ro- bustmess against various types of data corruption. Furthermore, we demonstrate RDT's s superiority in handling both training-time and resting-time attacks. We hope that our findings will inspire further research into using sequence modeling methods to address data corruption challenges in increasingly complex and realistic scenarios.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
Baoxiang Wang partially supported by the National Natural Science Foundation of China (62106213, 72394361) and an extended support project from the Shenzhen Science and Technol- ogy Program. Shuang Qiu acknowledges the support of GRF 16209124.
ETHICS STATEMENT
This research conducted in compliance with all applicable ethical guidelines and institutional regulations, Since the study did not involve human participants, animals, or sensitive data, no spe- cific ethical approvals were required. All data used in this research were obtained from publicly available sources, ensuring full transparency and reproducibility of the results.
REPRODUCIBILITY STATEMENT
We have taken several measures to ensure the reproducibility of our results. All datasets used in experiments publicly available and be accessed through https://githu/ com/ Farama-Foour www.noountaiooo on DARL and
com po xxir/NeRRo, The imple- mentation of all baselines and RDT is based the codebase available at httpss////: com/tinkoff-ot aiRCORL] Detailed instructions for setting the environment and running the experiments are provided Appendix CC] Our implementation code is available Intpps: //github. cwm jiwe / RRobusttDe.. Tonn rans ransffrreee We encourage researchers refer to Appendix for more detailed information.

---

Published as a conference paper at ICLR 2025
REFERENCES
Johannes Ackermann, Takayuki Osa, and Masashi Sugiyama. Offline reinforcement learning from datasets with structured statiitttonn.t... arXiv preprint arXiv:2405. 14/ 2024.
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-bant offline re- inforcement learning with diversified g-ensemble. Advances in neural information processing systems, 34:7436-7447, 2021.
Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhihong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-drivet offline reinforcement learning. arXiv preprint arXiv:2202./1566, 2022.
Prajjwal Bhargava, Rohan Chitmis, Alborz Geramifard, Shagun Sodhani, and Amy Zhang. Se- quence modeling is robust contender for offline reinforcement learning. arXiv preprint arXiv:2305. 14550, 2023.
Micah Carroll, Orr Paradise, Jessy Lin, Raluca Georgescu, Mingfei Sun, David Bignell, Stephanie Milani, Katja Hofmann, Matthew Hausknecht, Anea Dragan, et al. Uni [mask]] Unified inference in sequential decision problems. Advances in neural information processing systems, 35:35365- 35378, 2022.
Haw-Shiuan Chang, Erik Learned-Miller, and Andrew McCallum. Active bias: Training more accurate neural networks by emphasizing high variance samples. Advances Neural Information Processing Systems, 30, 2017.
Yevgen Chebotar, Quan Vuong, Alex Irpan, Karol Hausman, Fei Xia, Yao Lu, Aviral Kumar, Tianhe Yu, Alexander Herzog, Karl Pertsch, et al. Q-transformer: Scalable offline reinforcement learning via autoregressive q- f-uncctions. arXiv preprint arXiv:2309. 10150, 2023.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learnng via sequence modeling. Advances in neural information processing systems, 14:1444-510997, 2021.
Yiding Chen, Xuezhou Zhang, Qiaomin Xie, and Xiaojin Zhu. Exact policy recovery in offline rl with both heavy-tailed rewards and data corruption. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 11416-11424 2024.
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline rl via supervised learning? arXiv preprint arXiv:2112.107511 2021.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arxiv:2004.072/99, 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 2052-2062. PMLR, 2019.
Yarin Gal and Zoubin Ghahramani. A theoretically grounded application of dropout in recurrent neural networks. Advances neural information processing systems, 29, 2016.
Kamyar Ghasemipour, Shixiang Shane Gu. and Ofir Nachum. Why so pessimistic? estimating uncertainties for offline rl through ensembles. and why their independence matters. Advances in Newral Information P Processing Systems, 13:18877-18281. 2022.
Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint 1rXi5:2304.10573. 2023.
Kaizhe Hu, Ray Chen Zheng, Yang Gao, and Huazhe Xu. Decision transformer under random frame dropping. In The Eleventh International Conference Learning Representations, 2022.
12

---

Published as a conference paper at ICLR 2025
Kaizhe Hu, Ray Chen Zheng, Yang Gao, and Huazhe Xu. Decision transformer under random frame dropping. arXiv preprint arXiv:2303.03331, 2023.
Shengchao Hu, Ziqing Fan, Chaoqin Huang, Li Shen, Ya Zhang, Yanfeng Wang, and Dacheng Tao. Q-value regularized transformer for offline reinforcement learning. arXiv preprint arXiv:2405. 2024.
Longyang Huang, Botao Dong, Wei Xic, and Weidong Zhang. Offline reinforcement learning with behavior value regularization. IEEE Transactions on Cybernetics, 2024.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as big sequence modeling problem. Advances in neural information processing systems, 34:1273-1286, 2021.
Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine Planning with diffusion for flexible behavior synthesis. In International Conference Machine Learning, pp. 9902-99111 PMLR, 2022.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. arXiv preprint 10.0616 2021.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179-1191, 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0164) 2020.
Anqi Li, Dipendra Misra, Andrey Kolobov, and Ching-An Cheng. Survival instinct in offline rein- forcement learning. arXiv preprint 2.3.::230000328866 2023.
Lanqing Li, Rui Yang, and Dijun Luo. Focal: Efficient fully-offline meta-reiifforcement learning via distance metric learning and behavior regularization. arXiv preprint arXiv:2010.01112, 2020.
Xize Liang, Chao Chen, Shuang Qiu, Jie Wang, Yue Wu, Zhihang Fu, Zhihao Shi, Feng Wu, and Jieping Ye. Ropo: Robust preference optimization for large language models, 2024. URL https: /ats:///rrxivi ats/abss////ooooo.ooe..../a..a.a
abs 04404.04100
Hao Liu and Pieter Abbeel. Emergent agentic transformer from chain of hindsight experience. In International Conference Machine Learning, pp. 21362-21374. PMLR, 2023.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 06083, 2017.
Debmalya Mandal, Andi Nika, Parameswaran Kamalaruban, Adish Singla, and Goran Radanović. Corruption robust offline reinforcement learning with human feedback. arXiv preprint arXiv:2402.06634, 2024.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing 1stm lan- guage models. arXiv preprint arXiv:
2017.
Alexander Nikulin, Vladislav Kurenkov, Denis Tarasov, and Sergey Kolesnikov. Anti-exploration by random network distillation. arXiv preprint 2rX1v:2301.136166 2023.
Kishan Panaganti, Zaiyan Xu, Dilcep Kalathil, and Mohammad Ghavamzadeh. Robust reinforce- ment learning using offline data. Advances in neural information processing systems, 35:32211- 32224, 2022.
Scohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. Higl: Offline goal- conditioned with latent states as actions. Advances in Neural Information Processing Systems, 36, 2024.
Rong-Jun Qin, Xingyuan Zhang, Songyi Gao, Xiong-Hu Hui Chen, Zewen Li, Weinan Zhang, and Yang Yu. Neorl: A near real-world benchmark for offline reinforcement learning. Advances in Neural Information Processing Systems, 25.24553-24475. 2022.
13

---

Published as a conference paper at ICLR 2025
Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. arXiv preprint arXiv:1412.6596, 2014.
Fumihiro Sasaki and Ryota Yamashina. Behavioral cloning from noisy demonstrations. In Interna- lional Conference on Learning Representations, 2020.
Laixi Shi and Yuejie Chi. Distributionally robust model-based offline reinforcement learning with near-optimal sample complexity. arXiv preprini vrXiv:2208.05766 2022.
Ruizhe Shi, Yuyao Liu, Yanjie Ze, Simon Shaolei Du, and Huazhe Xu. Unleashing the power of pre-trained language models for offline reinforcement learning. In The Twelfth International Conference on Learning Representations, 2023.
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey, IEEE Transactions on Neural Networks and Learning Systems, 2022.
Hao Sun, Lei Han, Rui Yang, Xiaoteng Ma, Jian Guo, and Bolei Zhou. Exploit reward shifting in value-based deep-rl: Optimistic curiosity-basee exploration and conservative exploitation via linear reward shaping. Advances in Neural Information Processing Systems, 35:37719-37734, 2022.
Hao Sun, Alihan Hüyük, Daniel Jarrett, and Mihaela van der Schaar. Accountability offline reinforcement learning: Explaining decisions with a corpus of examples. Advances in Neural Information Processing Systems, 36, 2024.
Quan Vuong, Aviral Kumar, Sergey Levine, and Yevgen Chebotar. Dasco: Dual-generator ad- versarial support constrained offiin reinforcement learning. Advances Neural Information Processing Systems, 35:38937-38949 2022.
Mianchu Wang, Rui Yang. Xi Chen, and Meng Fang. Goplan: Goal-conditioned offline reinforce- ment learning by planning with learned models. arXiv preprint arXiv:2310.20025, 2023.
Qing Wang, Jiechao Xiong, Lei Han, Han Liu, Tong Zhang, et al. Exponentially weighted imitation learning for batched historical data. Advances in Neural Information Processing Systems, 31, 2018.
Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as expressive policy class for offline reinforcement learning, arXiv preprint arXiv:2208.06199 066993 2022.
Fan Wu, Linyi Li, Chejian Xu, Huan Zhang, Bhavya Kailkhura, Krishnaram Kenthapadi, Ding Zhao, and Bo Li. Copa: Certifying robust policies for offline reinforeement learning against poisoning attacks. In International Conference on Learning Representations, 2022.
Yueh-Hua Wu, Xiaolong Wang, and Masashi Hamaya. Elastic decision transformer. Advances in Neural Information Processing Systems, 36, 2024.
Zhihui Xie, Zichuan Lin, Deheng Ye, Qiang Fu, Yang Wei, and Shuai Li. Future-conditioned unsu- pervised pretraining for decision transformer. International Conference on Machine Learning, pp. 38187-38203. PMLR, 2023.
Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xi- anyuan Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization. arXiv preprint arXiv:2303.188/1 2023.
Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao Joshua Tenenbaum, and Chuang Gan. Prompting decision transformer for few-shot policy generalization. In international confer- ence on machine learning, pp. 24631-24645 PMLR, 2022.
Taku Yamagata, Ahmed Khalil, and Raul Santos-Rodrigus Rodriguez. Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling in offline rl. Inter- national Conference on Machine Learning, pp. 18989-33001) PMLR, 2023.
14

---

Published as a conference paper at ICLR 2025
Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. Rorl: Ro- bust offline reinforcement learning via conservative smoothing. Advances in Neural Information Processing Systems, 35:23851 -22866, 2022a.
Rui Yang, Yiming Lu, Wenzhe Li, Hao Sun, Meng Fang, Yali Du, Xiu Li, Lei Han, and Chongjie Zhang. Rethinking goal-conditioned supervised learning and its connection to offline rl. In Inter- national Conference on Learning Representations. 2022b.
Rui Yang, Lin Yong, Xiaoteng Ma, Hao Hu, Chongjie Zhang, and Tong Zhang. What is essential for unseen goal generalization of offline goal-conditioned ri? In International Conference Machine Learning, pp. 39543-39571. PMLR, 2023.
Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang, and Tong Zhang. Regularizing hidden states enables learning generalizable reward model for llms. arXiv preprint arXiv:2406.. 10216, 2024a.
Rui Yang. Xiaoman Pan, Feng Luo. Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. Rewards- in-context: Multi-objective alignment of foundation models with dynamic preference adjustment. arXiv preprint arXiv:2402. 10207, 2024b.
Rui Yang, Han Zhong, Jiawei Xu, Amy Zhang, Chongjie Zhang, Lei Han, and Tong Zhang. Towards robust offline reinforcement learning under diverse data corruption. In The Twelfth International Conference on Learning Representations, 2024c.
Zhihe YANG and Yunjian Xu. DMBP: Diffusion model-based predictor for robust offline rein- forcement learning against state observation perturbations. In The Twelfth International Confer- ence on Learning Representations, 2024. URL httpss://ppenreeeeww.eeet net www.rroriwieeetnuuumt forum?id
BULjCYLAKe
Chenlu Ye, Jiafan He, Quanquan Gu, and Tong Zhang. Towards robust model-based reinforcement learning against adversarial corruption. arXiv preprint arXiv:2402.08991, 2024a.
Chenlu Ye, Rui Yang, Quanquan Gu, and Tong Zhang. Corruption-role coouution-robust offline reinforcement learning with general function approximation. Advances Neural Information Processing Sys- fems, 36, 2024b.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: foodel-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129- 14142, 2020.
Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations state observations. Ad- vances Neural Information Processing Systems, 33:21024-2100 2020.
Huan Zhang, Hongge Chen, Duane Boning, and Cho-Jui Hsieh. Robust reinforcement learning on state observations with learned optimal adversary. arXiv preprint wwxiv:2200.084555 2021.
Xuezhou Zhang, Yiding Chen, Xiaojin Zhu, and Wen Sun. Corruption-robust offline reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pp. 5757-5773. PMLR, 2022.
Zifeng Zhuang, Dengyun Peng, Ziqi Zhang, Donglin Wang, et al. Reinformer: Max-return sequence modeling for offline rl. arXiv preprint 20X.v:2405.08740. 2024.
15

---

Published as a conference paper at ICLR 2025
CONTENTS
Introduction
2
Preliminaries
3
Sequence Modeling for Offline RL with Data Corruption
3.1
Motivating Example
Robust Decision Transformer
3.2.1
Embedding Dropout
3.2.2
Gaussian Weighted Learning
3.2.3
Iterative Data Correction
Experiments
Experimental Setups
4.2
Evaluation under Various Data Corruption
Evaluation under Observation Perturbation during the Testing Phase
4.4
Ablation Study
4.5
Impact of Transformer Backbone on Baselines)
Related Work
10
Conclusion
10
Algorithm Pseudocode
18
Additional Related Works
18
Implementation Details
18
Data Corruption Details during Training Phase
18
C2
Observation Perturbation Details during the Testing Phase
................................
19
Implementation Details of RDT
19
Detailed Information on Dataset
20
Additional Experiments
D.1
Impact of Decision Transtormer's 's Hyperparameters
21
D.2
Ablation Study on Reward Prediction
................................
D.3
Ablation Study Hyperparameters of RDT
................................
22
D.4
Varying Corruption Rates and Scales
22
D.5
Detailed Results for the Experiments Adversarial Attack
23
D.6
Comparison Results on MuJoCo Tasks with Full Dataset
23
D.7
Additional Results under Observation Perturbation during Testing Phase
24
16

---

Published as a conference paper at ICLR 2025
D.8
Training Time
25
D.9
Additional Ablation Study on Three Robust Techniques
26
D 10 Evaluation under Various Dataset Scales
26
D. |1 Performance under MuJoCo Dataset with Different Quality Levels
................................
27
D 12 Performance under NeoRL Dataset
27
13 Additional Evaluation on Iterative Data Correction
28
D 14 Discussion on State Prediction and Correction
28
D.15 15 Motivating Example under Difterent Data Corruption
29
imitation and Discussion
29
17

---

end if Published as a conference paper at ICLR 2025
A ALGORITHM PSEUDOCODE
To provide overview and better understanding, we detail the implementation of the Robust Deci- sion Transformer (RDT) in Algorithm I
Algorithm Robust Decision Transformer (RDT)
Require: Offline dataset D. sequence model ma, initialed mean jas and variance o3. 1: for training step= 1, 2, ...,, T do
2: Extract batch +t:t+-11 from the offline dataset D.
3: Update sequence model Te based on Eq.
4: Compute prediction errors da, and o,, in batch data.
5: Update corresponding mean jáa and variance og.
6: end for
7: if correction phase begins then
8: Evaluate the z-score to identify the corrupted action at and reward f(")
9: Substitute actions and rewards with the predicted actions and rewards in dataset D.
B ADDITIONAL RELATED WORKS
Offline RL. Maintaining proximity between the policy and data distribution is essential in offline RL., as distributional shifts can lead to erroneous estimations (Levine et al. (020). To counter this,
offline RI. algorithms are primarily divided into two categories. The first category focuses on policy
constraints on the learned policy (Wang et al., 2018 Fujimoto et al. 2019 Li et al. 2020 Fujimoto
al. 1021 Emmons al. [021 Yang al. 2022b; 2023 Sun et al., 2024 u et al.) 2023 Park et al.]
& Gu 2021; Kostrikov
The other category learns pessimistic value functions
to penalize OOD actions (Kumar et al. [2000 Yu al.,
2020 An et al., 2021 Bai al., 2022
Yang et al. 2022a] Ghasemipour et al., 2022; Sun et al.
2022; Nikulin et al. [023]] Huang et al.
2024j. To enhance the potential of offline RL in handling more complex tasks, recent research
has integrated advanced techniques like GAN (Vuong et al. 2022, Wang et al. 2023). transformers
(Chen et al., 2021; Chebotar et al. 2023 Yamagata et al. 2023) and diffusion models (Janner et al.
2022 Hansen-Estruuh et al. 2023 Wang et al. 2022).
C
IMPLEMENTATION DETAILS
C.I DATA CORRUPTION DETAILS DURING TRAINING PHASE
Our study utilizes both random and adversarial corruption across three elements: states, actions, and rewards. We consider range of tasks including MuJoCo, Kitchen, and Adroit. Particularly, we utilize the "medium-replay-v2" datasets in the MuloCo tasks with sampling ratios 10%, the "expert-vO" datasets in the Adroit tasks with sampling ratio of 1%, and we employ full datasets for the tasks in the Kitchen due to their already limited data size. These datasets (Fu et al. 2020) are collected either during the training process of an SAC agent from expert demonstrations, thereby providing highly diverse and representative tasks of the real world. To control the overall level of corruption within the datasets, we introduce two parameters e and e following previous work (Ye et al., 2024b] Yang et al. 2024c). The parameter signifies the rate of corrupted data within dataset, whilst represents the scale corruption observed across each dimension. We outline three types of random data corruption and present comprehensive overview of mixed corruption approach as follows. Note that in our setting, only three independent elements (i.c., states, actions, and rewards) considered under the trajectory- -based storage approach.
states from all trajectories, where N refer to the number of trajectories and T represents the number of steps in trajectory. We then modify the selected state to = s + std(s), À ~ Sniform|-c,d cjd. Here, d, represents the dimension of states, and "std(s)" is the d, dimensional standard deviation of
Random state attack: We randomly sample c
18

---

Published as a conference paper at ICLR 2025
all states in the offline dataset. The noise is scaled according to the standard deviation of each dimension and independently added to each respective dimension.
Random action attack: We randomly select c. -N- T actions from all given trajectories, and modify the action to å = a A- -td((a), X ~ Uniform -c, <!"r, where d, represents the dimension of actions and "std(a)" is the di-diimensoona standard deviation of all actions in the offline dataset.
Random reward attack: We randomly sample N. rewards from all given trajectories, and modify the reward to Uniform[ [iifform--3--.. 6,30 c]. We multiply by 30 because we have noticed that offline RL algorithms tend be resilient to small-scale random reward corruption (also observed in Li el al. (2023),, but would fail when faced with large-scale random reward corruption.
In addition, three types of adversarial data corruption are detailed as follows:
Adversarial state attack: We first pretrain IQL agents with a Q function Qp and policy function 7p on clean datasets. Then, we randomly sample c N T states, and modify them mingeBa(s.. @p(5, a). Here, cB)(s,))
[$||s s] I e std(ss) regularizes the maximum difference for each state dimension. The optimization is implemented through Projected Gradient Descent similar to prior works (Madry et al. 2017 Zhang et al. 2020
Yang et al. 2024c). Specifically, We first initialize learnable vector
c and then conduct a 100-step gradient descent with a step size of 0.01 for = 8 2 std(s), and clip each dimension of 2 within the range [-c, after each update.
Adversarial action attack: We use the pretrained IQL agent with function Qp and a policy function Tp. Then, we randomly sample e N T actions, and modify them to std(a)) regularizes the
à minaae @p(s, â). Here, e) {āllä al maximum difference for each action dimension. The optimization is implemented through Projected Gradient Descent, as discussed above.
Adversarial reward attack: We randomly sample c. N rewards, and directly modify them to: = -E T.
C.2 OBSERVATION PERTURBATION DETAILS DURING THE TESTING PHASE
We evaluate RDT and other baselines under two types of observation perturbations during the testing phase: Random and Action Diff perturbations, as described in prior works (Yang et al. 2022a Zhang et al.] 2020). Generally, offline RL algorithms are sensitive to observation perturbations, often resulting in a significant performance drop. The detailed implementations are as follows:
Random:
We sample perturbed states within an lx0 ball of norm centering the original state. Specifically, we create the perturbation set (s e) = (8 d(s 8) S <} for state 8, where d(-) the Loo norm, and sample one perturbed state to return to the agent.
Action Diff: This an adversarial attack based on pretrained IQL deterministic policy ((s). We first sample 50 perturbed states within an 1.0 ball of norm e and then find the one that maximizes the difference in actions: maxaeBa(s.e) [ln(s) 21)
The parameter controls the scale observation perturbations. We consider {0.0, 0.1, 0.3, 0.5) for in our experiments. In this setup, first train offline RL algorithms under various data corruption settings, and then evaluate their performance in environments with observation perturbations.
C.3 IMPLEMENTATION DETAILS OF RDT
We implement RDT and other baselines using the existing code base Specifically. we build the network with 3 Transformer blocks, incorporating one MLP embedding layer for each key element: state, action, and return-to-go. We update the neural network using the Adam optimizer, with a learning rate set sequence length of 20, to ensure effective and efficient training. To maintain stability during training,
10-1 and weight decay of 1 10-1 The batch size is set to 64, with
19

---

Published as a conference paper at ICLR 2025
adopt the state normalization in Yang et al. (2024c) During the training phase, we train DeFog, DT and RDT for 100 epochs and other baselines with 1000 epochs. Each epoch is characterized by 1000 gradient steps. For evaluative purposes, trajectories, each with a maximum length of 1000, and we average the returns from these rollouts for comparison. We ensure the consisteney and reliability of our reported results by averaging them over 4 unique random seeds. All experiments are conducted on P40 GPUs.
rollout each agent in environment across 10
As for the hyperparameters of RDT, we configure the values for the embedding dropout probability as p. Gaussian Weighted coefficient 3, and 3,, and iterative data correction threshold Ç through the sweeping mechanism. We start the iterative data correction the 50th epoch for all tasks. We use iterative action correction exclusively for action corruption and iterative reward correction exclu- sively for reward corruption: therefore, only single Ç value is shown Table 3 It's noteworthy that we do not conduct a hyperparameter search for individual datasets; instead, we select hyperparameters that apply to the entire task group. As such, RDT doesn't require extensive hyperparameter tuning previous methods. The exact hyperparameters used for the random corrup- tion experiment are detailed in 3 We employ almost same hyperparameters for the adversarial corruption experiment, further attesting to the robustness and stability of RDT.
Table 3: Hyperparameters used for RDT under the random corruption.
Tasks
Attack Element
p
(9u. 8.)
0.2
(1.0, 1.0)
MuloCo (10%)
action
(1.0, 1.0)
reward
0.2
(1.0, 1.0)
6.0
state
0.1
(30.0, 30.0)
Kitchen
action
0.1
(30.0,
5.0
reward
0.1
(30.0, 30.0)
state
0.1
0..1,
Adroit (1%)
action
0.1
(10.0, 0.0)
6.0
reward
0.1
(0.1, 0.1)
6.0
C.4 DETAILED INFORMATION ON DATASET
Since we conduct experiments on both the full and down-sampled datasets, we provide detailed information about the number of transitions and trajectories. As seen from Table g after down- sampling, the number of transitions remains of the same order of magnitude for most datasets, except for the Kitchen. Due the small number of transitions in the original full "kitchen-complete" dataset, we opted not to perform down-sampling on the Kitchen.
Dataset
halfcheetah
hopper
walker2d
# Transitions
202000
402000
302000
# Trajectories
202
2041
1093
Dataset
halfcheetah(10%) 20000
hopper(10%) 32921
walker2d (10%)
# Transitions
27937
# Trajectories
20
204
109
Dataset
kitchen-complete
kitchen-partial
kitchen-mixed
Transitions
3680
136950
136950
# Trajectories
19
613
613
Dataset
door(1 1 %)
hammer(1 %)
relocate( %)
# Transitions
10000
10000
10000
# Trajectories
50
50
50
Table 4: Detailed information about the number of transitions and trajectories.
20

---

Published as a conference paper at ICLR 2025
D ADDITIONAL EXPERIMENTS
To provide a comprehensive understanding of our work, we conduct several additional analyses. First, we investigate the key hyperparameters that influence the robustness of DT. Next, we discuss the impact of reward prediction on DT. Following this, we demonstrate how these hyperparameters affect RDT and assess the robustness of RDT under various scales of data corruption. We then compare the training time between RDT and baseline models. Finally, we present detailed results of RDT under both training-tine time corruption and testing-time perturbation.
D.1 IMPACT OF DECISION TRANSFORMER 'S HYPERPARAMETERS
We aim to understand the robustness characteristic of sequence modeling. The key feature of the DT is its utilization of sequence modeling, with the incorporation of the Transformer model. Therefore, we identify two crucial factors that may influence the performance of DT in the context of sequence modeling: the model structure and the input data.
State /ifark
Number of Blocks
Numberot Blocks
Sequence Length
Sequence Langth
Rushhnn
Pageto D0M
strrer
Kturer
(a)
(b)
(c)
Figure 8: Ablation study on the robustness of DT. (a) Analyzing the influence of Transformer block number while keeping sequence length with 20. (b) The effect of sequence length while maintaining 3 Transformer blocks. (c) Investigation on how the return-to-ge impacts DT. We create the variant DT w/o. Return, which excludes the use of return-to-go as inputs.
We explore the influence of the model's structure by modifying the number of Transformer blocks in DT. As depicted in Figure a)). the performance on both MuloCo and Kitchen tasks consistently improves with increased number of blocks. Therefore, the enhancement can be partially attributed to the increased capabilities of the larger model.
We also examine the impact of the inputs for the sequence model. There are two primary differences between the input of DT and that of conventional BC. DT predicts actions based historical data and return-to-go elements, whereas BC relies solely on the current states. To assess the influence of these factors, we independently evaluate the impact of the length of history and return-to-go elements. The results in Figure (b) and (c) reflect that Kitchen tasks favour longer historical input and are less reliant returntoogo elements. Conversely, MuJoCo tasks demonstrated different trend. This discrepancy could be due to the sparse reward structure of Kitchen tasks, where return- to-go elements do not provide sufficient information, leading the agent to rely more heavily on historical data for policy learning. In contrast, MuJoCo tasks. with their denser reward structures, rely more on return- to-go elements for optimal policy learning.
In summary, the robustness of DT can be attributed to the model capacity, history length, and return- to-go conditioning. Based on these findings on the robustness of DT, we adopt a sequence length of 20 and block number of 3 as the default implementation for all sequence modeling methods, including DT, DeFog, and RDT.
D.2 ABLATION STUDY ON REWARD PREDICTION
The nature of the sequence model allows us to predict not only actions but also other elements, such as state and return-to-go. Previous studies have shown that predicting these additional elements does not significantly enhance performance Chen et al. (202I). However, we find that predicting rewards provide advantages in the context of data corruption. We introduce DTTRP), variant of the original DT, which predicts rewards in addition to actions. As illustrated in Figure [ predicting rewards indeed improves performance under random state and action corruption conditions. How-
21

---

Published as a conference paper at ICLR 2025
DT(PP) performs poorly under reward corruption due the degraded quality of reward labels. This issue can be mitigated via embedding dropout, Gaussian weighted learning, and iterative data correction in RDT.
Action
80
80
80
Nomakzel Sore
60
63
60
80
43
400
20
20
Kitchn
Adrot(13%)
MuloCo(105)
Kichen
Adroi([15]
MuloCo(10%
Kitchn
Adro
DT
DI(RP))
Figure 9: Ablation study on the impact of reward prediction.
D.3 ABLATION STUDY ON HYPERPARAMETERS OF RDT
We conduct experiments investigate the impact on hyperparameters of RDT under scenarios of random action corruption.
In the study of the embedding dropout technique, depicted in Figure T0 T0Ja), a), discover that applying
a moderate dropout probability (c.g., g., 0.1) to embeddings enhances resilience against data corruption.
However, larger dropout rate (2 0.5) can lead performance degradation.
As shown in Figure b)),,, regarding the Gaussian weighted coefficient Sa, the walker2d task favours lower values 3g such 0. I, while the crohhee-complte and relocate tasks favour larger 8a values, suggesting that these tasks are less impacted by Gaussian weight learning.
In terms the iterative data correction threshold Ç. the results are highlighted inFigure (0(e). The walker2d and kitchen-complete tasks favour lower values of whereas the relocate task exhibits the opposite preference. Notably, the relocate task, RDT shows markedly enhanced performance after correcting outliers (with Ç 2 5) in the dataset.
Morsd
(a) Embedding drop p.
Gaussian weight coefficient Ba. (c) Iterative correction threshold Ç
Figure 10: Ablation study the impact of hyperparameters under action corruption scenario.
D.4 VARYING CORRUPTION RATES AND SCALES
We have evaluated the effectiveness of RDT under a data corruption rate of 0.3 and scale of 1.0 in Section 4.2 We further examine the robustness of RDT under different corruption rates from (0.0, 0.1, 0.3, 0.5) and scales from {0.0, 1.0, 2.0}. As illustrated in Figure 11 RDT consistently de- livers superior performance compared to other baselines across different corruption rates and scales. However, we also observed decline in RDT's performance when it encountered state corruption with high rates. This decline can attributed to the distortion of the state's 's distribution caused by the high corruption rates, resulting in significant deviation from the clean test environment the corrupted datasets, Enhancing the robustness of RDT against state corruption, especially under high corruption rates and scales, holds potential for future advancements.
22

---

Published as a conference paper at ICLR 2025
State Amtek
Action Attsck
Reward Amtak
State Artuct
Action Altac
Dound Amszk
Nerolas
Comption Rate
Corrptine Rec
Comption Raiz
Comptice
Cerpptinn
Comption
(a)
(b)
Figure 11: Results under various corruption rates (a) and scales (b) on the "walker2d" task.
D.5 DETAILED RESULTS FOR THE EXPERIMENTS ON ADVERSARIAL ATTACK
The detailed outcomes of each task under adversarial data corruption are presented in Table No- tably, RDT outperforms baseline models in the majority of tasks, achieving the best average per- formance. In particular, RDT consistently outperforms DT, demonstrating the effectiveness of our three robust techniques.
Table Detailed comparative results under adversarial data corruption.
Attack
Task
BC
RBC
DeFog
COL
UWMSG
RIQI.
DT
RDT
halfcheetahh (10%)
2.140..6
2.41-0.3
4.8+0.77
10.0 1.6
7.242.6
3.6+-0.5
5.410.7
7.5+0.9
hopper (10%)
19.1+3.2
18.7+4..
21.8=74
1.8-088
14.743.4
19.146.9
38.9+2.9
39.1=4.6
walker2d (10%)
8.7+1.8
7.140.2
12.0=3.6
0.7=1.5
5.142.
10.4+19
21.3:115
24.2:+3.2
kitchen-complete
20.849..
43.545.5
37.643..2
kitchen-partial
27.14442
30.8+3.8
8.955.2
0.000.0
0.000.0
35,4+5.8
37,946.2
41.6+5,1
State
kitchen-mixed
24.1+3.1
35,2+3.7
7.6=66.3
2.=33.5
0,0+0.00
33.94:1.2
38.1=6.0
44.5=5.1
door (1%)
71.148.8
66.1+10.2
102.2+144
-.3--0.0
-0.340.1
47.34-24.8
98.24448
104.0+1.0
hammer (1%))
82.5+1.11
79.348.4
102.44888
0.2-0.0
0.1+0.11
69.1+:2344
90.1+1119
110.949.0
relocate
27.042.
15.547.2
22.1=3355
-0.3=0.0
-0.3+0.00
17.149.5
67.642.6
71.3+2.
Average
31.2
30.6
36.2
2.1
2.9
32.0
48.4
56,0
halfcheetah (10%)
00.01022
89210
19.0-2.2
6.71111
17.1233
hopper (10%)
10.913.7
11.1122
16.1110.4
1.5
37.41.00
17.1114
28.8133
31.6188
walker2d (10%)
1.920.7
1.940.5
5.822.4
1.41.00
6.74100
3.3412
9.0:1.4
16.02:17
cilchen-complete
6.610.6
5,61:3.0
115:1111
18219
0.020.0
20.61:45
11.12.
25.
kitchen kitchen-partill
6.912..3
30.8
Action
kitchen mixed
6.477.4
22.1
0.1400.2
0.0-0.0
0.040.0
3.41455
2.813344
45.9
door (1%)
6.0113
11885.77
102821.1
0,3-0.1
-02:0.1
38.7121.0
62/217.4
97.6:15.5
hummer (1%)
16.3110.2
44.0112.2
37.91..99
0.2.0.0
0.2.1.11
88.1
62.2.116.7
107.7.1.3
relocate (1%)
2,7427
Average
6.3
12.2
22.0
2.7
5.6
19.2
22.1
45.3
halfcheetah (10%)
2,4022
31.2-1.6
1.9+0.2
10.941.6
9.540.66
23.3455.
hopper (10%)
19,71128
19.313.1
15.6.8.44
1.8.0.0
28.6.1999
34.515.11
37.2.6..5
16,2
walker2d (10%)
9,741.5
8.34-177
4.0++199
2.2-2.4
9.743.11
9.34-1.2
29.0+-3.6
35.2+2.9
Kitchen-complete
36.0E11..
38.243.0
48.0=2.44
3.4-3.8
0.0:50.0
$1.5426
45.0+E5.0
65.9-12.0
Reward
kitchen-partial
34.111.4
39.1+188
9.5+7.9
0.0=0.0
0.0+00.0
37.24-8.5
45.11:6.1
46.1+1166
kitchen-mixed
38.9+1.4
47.1420
3.5-3.33
0.0-0.0
0.0+0.00
48.9+-4.7
54.14223
56.1+-288
door (15)
76.0559
15.0+9.0
1022=1.0
-03=0.11
-0220.1
73.4111.
98.9-2.8
101.420.6
hammer (1%)
97.148.3
99.021118
92.1=14.00
0.2=0.0
0.1++0.1
69.1+:777
97.1=:11.3
110.9=1..6
relocate (1%)
-0.3-0.1
77.2+44.2
Average
38.9
40.1
37.6
4.3
4.4
39.2
54.7
61.5
Average
25.5
3.0
30.1
54.3
D.6 COMPARISON RESULTS ON MuJoCo TASKS WITH FULL DATASET
We further evaluate the robustness of RDT on MuloCo tasks using the full dataset. As shown in Table RIQL is comparable to DT in this setting, while RDT achieves the highest overall average score. These findings further demonstrate the effectiveness of our proposed robust techniques across different dataset scales.
Our reported results for RIQL on MuJoCo tasks using the full 100% dataset show some discrepan- cies compared to the original paper (Yang al. 2024c). To investigate this, plotted the learning curve of RIQL under an action attack, as shown in Figure [2] In comparison to Figure in the RIQL paper, found that RIQI may require more training epochs to converge when using the
23

---

Published as a conference paper at ICLR 2025
Table 6: Results under random data corruption on MuJoCo tasks with 100% dataset.
Attack
Task
BC
RBC
DeFig
CQL
UWMSG
RIQL
DT
RDT
halfcheetah
3..41199
33.0=0.9
14.917.1
23.744.4
2.2+0.66
19.9=2.1
27.5-2.5
30.8=1.8
hopper
27.118.6
26.6#2299
21.166.8
34.4+2288
21.62800
34.0=13.4
51.3=1.0
56.6=299
State
walker2d
19.214.7
16.826.0
17.422.4
33.7
0.8
14.2+1.2
47.624.9
53.4=4.0
Average
26.2
25.4
17.8
30.6
8.2
22.7
42.1
46.9
halfcheetah
34,2+1.4
34.6+-1.9
34.2+1.88
43.9+0.4
4.29
41.3+
35.1+2.4
37.4+1.4
hopper
22.5+4.8
28.8:10.1
40.8211.6
29.848.8
39.0:668
19.6
63.2-5.6
63.2=5.8
Action
walker2d
18.342.6
21.343.6
50.7.6.0
1843.6
53.3+16.8
12.6=17.3
59.0=5.5
6..41177
Average
25.0
28.2
41.9
25.2
47.4
52.4
halfcheetah
35.920.5
34.7=11.4
32.6#2.3
43,3=0.3
37.215.88
42.7=0.9
38,4=033
39.6=0.8
Reward
hopper
26.6+-6.9
29.5
19.3
29.9+7.22
65.4++26.0
59.6
54.3+10.7
70.9-8.7
walker2d
23.549.8
22.916.9
40.119.44
38.3422.99
51.3
79.0=5.88
64.2=4.0
66.7=440
Averuge
28.7
29.0
34.0
37.2
51.3
60.4
52.3
59.0
Average over all tasks
26.6
27.6
31.2
31.0
35.6
47.0
48.9
53.3
full dataset, which is reasonable. Given our focus on the limited data setting (10% for MuJoCo), observed that 1,000 epochs are sufficient for both RDT and baselines like RIQL to converge, as detailed Appendix D.8] Therefore, we use 1,000 training epochs by default, which, however, may not be optimal for the full dataset setting.
x hallcheetah-medin---eaaa--
X12 hoppe--medium-eppaa-22
X102 wallker2--mediun-rrlaa-
Eplcde Rowad
13
0-
2000
1111
1300
800
1000
0
2011
2001
6D0
9001
10000
200
400
0.01
800
1010
Epoch
Epoch
Epoch
100%
10%
Figure 12: Learning curve RIQL under action attack.
D.7 ADDITIONAL RESULTS UNDER OBSERVATION PERTURBATION DURING TESTING PHASE
To further demonstrate the robustness of RDT, we conduct additional evaluations under observation perturbations during the testing phase. Specifically, we compare different algorithms trained with offline datasets under conditions of random state, action, or reward corruption environments with observation perturbations. As shown in Figures 13 to 15 RDT consistently maintains superior performance and robustness across observation perturbations both the Kitchen and Adroit tasks. We leave the investigation of robustness improvements over MuJoCo tasks for future work.
Kirheo
Advi((()
Portebation Scale
Potarlation Scale
Poturiation Stale
Portarlatoo Stale
RD
RCB
(a) Random observation perturbation.
Action Diff observation perturbation.
Figure 13: Performance under varying observation perturbation scales during the testing phase. All the algorithms are trained under random state corruption during the training phase.
24

---

Published as a conference paper at ICLR 2025
Narskted
Pottebacion Scat
Paturbation Stale
Patur seltrn Stale
Potarbebor Sule
99
101
(a) Random observation perturbation.
(b) Action Diff observation perturbation.
Figure 14: Performance under varying observation perturbation scales during the testing phase. All the algorithms are trained under random action corruption during the training phase.
KinChen
Adve((S))
Keribe
A
Seor
horroland
Pottrbation
(a) Random observation perturbation.
(b) Action Diff observation perturbation.
Figure 15: Performance under varying observation perturbation scales during the testing phase. All the algorithms are trained under random reward corruption during the training phase.
D.8 TRAINING TIME
We train all algorithms on the same P40 GPU, and the training times are presented in Table 7 As expected, BC has the shortest epoch time due its simplicity. Notably, RDT requires only slightly more time per epoch than DT because incorporates three robust techniques. Although RIQI. requires less time per epoch than RDT, it necessitates 10 times does not achieve satisfactory performance in the limited data regime. In conclusion, RDT achieves superior performance and robustness without imposing significant computational costs in terms of total training time.
epochs to converge and still
Table 7: Training time on the "walkerrdd-meddum-rrplye -2" dataset.
Method
BC
CQL
UWMSG
RIQL
DT
RDT
Epoch Num
1000
1000
1000
1000
100
100
Epoch Time (s)
33.8
22.2
14.9
44.0
Total Time (h)
1.03
9.39
6.17
4.14
1.22
1.28
25

---

Published as a conference paper at ICLR 2025
D.9 ADDITIONAL ABLATION STUDY ON THREE ROBUST TECHNIQUES
We investigate the impact of each individual technique in Section 4.4 .4 and demonstrate that inte- grating all techniques is essential for achieving optimal robustness. To provide a more comprehen- sive analysis of our proposed techniques, we create the variants RDT w/o ED, RDT w/o GWL, and RDT w/o IDC, which eliminate the components Embedding Dropout, Gaussian Weighted Learning, and Iterative Data Correction from RDT, respectively. As illustrated in Figure 16 the results are consistent with the findings in Section This consistency reinforces our initial conclusion that integration of all proposed techniques is crucial for achieving optimal robustness.
Stase Attack
Roward
Store
walkerds
Kechen-compees
relocate
walkerdd
kitch
Malke
kitchen complete
20T
RDI
Figure 16: Additional ablation study on the impact of proposed techniques.
D.10 EVALUATION UNDER VARIOUS DATASET SCALES
We conducted experiments across various dataset scales to understand the impact of dataset scales. Specifically, we evaluate RDT on the MuJoCo tasks with 20% and 50% of the dataset and Adroit tasks with 5% and 10% of the "expert-v0" dataset. The comparison results are shown in Figure 17 Our empirical findings indicate that: (1) The overall performance of all methods improves as the dataset size increases. (2) Morcover,
methods like RIQL comparable to vanilla sequence modeling methods like DT when the dataset is large, but they are less preferable when the dataset size is limited. (3) Notably, RDT consistently outperforms other baselines across all dataset sizes, validating the effectiveness and importance of proposed robustness techniques.
cmppora--diffrrncee
State Attack
Action Attack
Reward Attack
Normllzzd Score
80
40
30
30
30
30
20
20
10
10
10
10%
20% Dataset Scale
50%
10%
20%
50%
10%
20%
50%
Dataset Scale
Dataset Scale
DeFag
RIQL
DT
RDT
(a) Results on MuloCo tasks.
State Attack
Action Attack
Reward Attack
Normalized Score
100
100
100
80
80
80
60
60
40
136
5% Dataset Scale
10%
1%
5%
10%
1%
5% Dataset Scale
10%
Dataset Scale
Defog
(b) Results Adroit tasks.
Figure 17: Performance of different algorithms across various dataset scales.
26

---

Published as a conference paper at ICLR 2025
D.11 PERFORMANCE UNDER MUJoCo DATASET WITH DIFFERENT QUALITY LEVELS
To further validate the robustness of the RDT across varying dataset quality levels, we conduct evalu- ations using both the "expert-v2" and "medium-v2" datasets on MuJoCo tasks. We randomly sample 2% for both datasets, respectively, resulting in both datasets having a similar size of approximately 10a transitions. We include RIQL, DeFog, and DT as baselines, and the results are presented in Tables [ and / Notably, RDT consistently outperforms the other baselines across datasets of different quality levels. This superior performance reinforces RDT's capability to efficiently handle range of dataset qualities.
Table 8: Results on "medium-v2 -2 dataset under random data corruption.
Attack
Task
DeFog
RIQI
RDT
hallcheetah (2%)
7.5
18.011.5
15.7
1..3:1.11
hopper (2%)
20.418.3
45.747.3
48.6.13.2
52.21:5.7
State
walker2d (2%)
1..2.1166
25.41:5.0
20.516.00
28.0
Average
14.0
29.7
28.3
34.2
halfcheetah (2%)
24.3+:1.4
13.9+-1.9
16.2++0.8
3.2
hopper (2%)
36.7-19..6
$1,9+
37.5++6.0
45,3+332
Action
walker2d (2%)
30.7+7.00
27.5++5.3
38.9442.9
43.8
Average
30.5
31,1
30,9
40.6
halfcheetah (2%)
27.4:22.9
36.111133
33.311.0
37.5
hopper (2%)
32.5::7.55
55.516.8
51.612.7
53,91:466
Reward
walker2d (2%)
28.2+18.4
42.1+13.4
50.5+5.1
62.1+4477
Average
29.3
44.6
45.1
51.2
Average over tasks
24.6
35.1
34.8
42.0
Table 9: Results on "expert-v2" dataset under random data corruption.
Attack
Task
DeFog
RIQL
DT
RDT
halfcheetah (2%)
3.8+2.3
0,5+1.2
2.9+0.88
4,4-0.2 0.2
hopper (2%)
15.1+44.2
32.0+-4.4
38.5+7.4
48.6-7.0
State
walker2d (2%)
15.343.1
21.71:4.6
40.7114.8
41.6-4.2
Average
11.4
18.1
27.4
31.5
halfcheetah (2%)
2.7+1..0
11.3
3.24:2.4
5.4-0.7
hopper (2%)
18.014.7
32.418.7
21.2.13.3
36.0-4.8
Action
walker2d (2%)
25.6113.8
7.11100
33.116..8
73.8
Average
15.5
12.7
19.2
38.4
halfcheetah (2%)
2.5:10.8
2.7.10.9
4.9-10.7
17.2 5.5
hopper (2%)
21.2:16.3
89.5:14.66
49.5+13.3
67.3=6.6
Reward
walker2d (2%)
36.019..
64.2+11..0
96.3+12.66
103.1-3.6
Average
19.9
52.1
50.2
62.5
Average over all tasks
15.6
27.6
32.3
44.1
D.12 PERFORMANCE UNDER NEORL DATASET
To further demonstrate the robustness of the RDT, we evaluate on the NeoRL benchmark (Qin et al., 2022). Specifically, we select the testheds. The Finance dataset enables the construction of a trading simulator, while the CityLearn dataset reshapes the aggregation curve of electricity demand by controlling energy storage. We randomly sample 20% and 10% of the Finance and CityLearn datasets, respectively, resulting in both datasets having similar size of approximately 2 x a transitions. We consider RIQL, DeFog, and DT to be comparable baselines. The performance is measured using the response reward criterion, detailed in Table I0) Notably, RDT achieves the best average performance compared to the other baselines, outperforming DT by 12. 13% under Finance dataset. These results underscore RDT's robustness and potential for real-world applications.
and "finance-medium" datasets as the
27

---

Published as a conference paper at ICLR 2025
Table 10: Results on NeoRL dataset under random data corruption.
Task
Attack
DeFog
RIQI.
DT
RDT
State
39138.9
30235.3
37878.9
39124.2
citylearn (20%)
Action
38970.6
29093.2
35589.0
39066.4
Reward
38156.4
29634.8
37626.9
38909.8
Average
38755.3
29654.4
37031.6
39033.5
State
402.2
339.4
411.4
443.0
finance (10%)
Action
386.6
400.6
352.6
452,
Reward
330,3
376.3
329.5
417.8
Average
373.0
372.1
347.5
437.6
D.13 ADDITIONAL EVALUATION ON ITERATIVE DATA CORRECTION
We have demonstrated the effectiveness of iterative data correction by illustrating the MSE loss between corrected and original data in Figure c). The MSE loss gradually decreases training proceeds, indicating that RDT can accurately predict the correct data. To further investigate the effectiveness of iterative data correction, Specifically, DT w. IDC can detect N data points as corrupted within batch, and M out of these data points are truly corrupted. We record the ratio of M/N under action attack as shown in Figure 18 As observed, the detection precision increases with the training process and can reach 80% to 90% at the 50th epoch, demonstrating the effectiveness of IDC. It is worth noting that precision is low at the start of training; therefore, we begin data correction at the 50th training epoch in our default implementation.
record the precision of detection using the z-score.
Precision of IDC Detection
1.00
0.80
Precision
0.60
walker2d walker2d-medium mediimm----
0.40
kicheen-omplete-vv
relacats-oxpertvv
0.20
Epoch
Figure 18: Accuracy of iterative data correction detection via the s-score.
D.14 DISCUSSION ON STATE PREDICTION AND CORRECTION
In the implementation of RDT, we predict both actions and rewards to tackle data corruption. We justify that reward prediction can provide benefits under state and action attacks (see Appendix(D.) Here, we explore the impact of state prediction. We create a variant, DT w. SP, based on the original DT,, which additionally predicts states. We compare DT and DT w. SP under state and action attacks, shown in Figure a). As observed, state prediction does not consistently benefit all tasks. The performance drops on the Kitchen and Adroit tasks, perhaps due to the larger state dimensions. For example, the state dimension of Kitchen is 60, which is larger than its action dimension of 9.
Based on DT w. SP. we evaluate state correction via iterative data correction under different Ç, shown in Figure (b). The results indicate that state correction still causes performance drops. Indeed, state prediction involves learning the environment's 's dynamics (i.e., transition probabilities), which is inherently challenging, prediction errors in state prediction negatively impact policy
28

---

Published as a conference paper at ICLR 2025
learning. Additionally, in experimental setting, data comuptiin exacerbates prediction errors, leading to further performance degradation. We plan to draw inspiration from model-based RL work to address this issue in future research.
Stars
Kichar
(105))
Kidhen
Pitshar
Adwr
(a) Ablation on state prediction.
(b) Ablation on state correction.
Figure 19: Ablation study on state prediction and correction. (a) Comparison of DT and DT with State Prediction (DT w. SP), showing that DT SP does not consistently bring improvements. (b) Evaluation of state correction under different values. Results indicate that state prediction can cause performance drops.
D.15 MOTIVATING EXAMPLE UNDER DIFFERENT DATA CORRUPTION
To demonstrate the performance drop under varying dataset sizes from another perspective, we mod- ify Figure to create Figure 20 In Figure 20 we place the performances of different algorithms in separate boxes to better illustrate their sensitivity to different types of data corruption. As ob- served, UWMSG and RIQL are very sensitive state attacks. This sensitivity arises because we do not separate state and next-state attacks but apply state attacks within single trajectory, repre- senting more general setting. UWMSG and RIQL are algorithms based on temporal difference learning, which learns the Q-value dependent on the next state. Hence, they experience signifi- cant performance drops under state attacks in our settings. In contrast, DeFog and DT, which are based sequence modeling, comparatively robust to all types of data corruption. Furthermore, DT demonstrates better robustness than other methods, even with limited datasets. Therefore, we propose three robustness techniques to further improve robustness against data corruption.
UMMSC
RIQU
DaFeg
60
60
Nomakard Scoee
20
100%
100%
10%
100%
10%
No Attack
State Attack
Action fstuck
Rovard Atack
Figure 20: An alternative version of the motivating example, where the performances of different algorithms are placed in separate boxes to better illustrate their sensitivity to various data corruption.
E LIMITATION AND DISCUSSION
One limitation of RDT is decision to avoid state prediction and correction within the robust sequence modeling framework. The high dimensionality of states poses significant challenges to achieving accurate predictions. Additionally, data corruption leading to potential declines in performance, as shown in Appendix D.14. Despite this, we have validated the feasibility and effectiveness of robust sequence modeling this paper, and we leave state prediction and correction as promising directions for future exploration.
further complicate state prediction,
29


Reviews:
[{"review_id": "JMJdJvouKH", "reviewer": "Reviewer_4tSr", "summary": "The work deals with the challenges posed by data corruption in offline RL and proposes a novel approach RDT. The primary contribution is the application of sequence modeling, specifically through a decision transformer, which has shown greater robustness against corrupted data compared to conventional RL methods that rely on temporal difference learning. The RDT model incorporates three key techniques—embedding dropout, Gaussian weighted learning, and iterative data correction—to enhance robustness. Empirical results indicate that RDT achieves superior performance under various corruption scenarios and provides resilience to both training-time data corruption and testing-time observation perturbations.", "questions": "Could the authors provide more clarity on the choice of dropout probabilities for different embeddings (states, actions, rewards) and its sensitivity to these values? How were these values chosen in the experiments?\n\nThe paper states that actions and rewards are corrected iteratively but leaves out state correction. What challenges or limitations prevented state correction? Would the authors consider incorporating state correction as future work, and how would it theoretically affect robustness?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Novel Method: The use of sequence modeling as a robust alternative to traditional temporal-difference-based methods for offline RL under corrupted data conditions is a significant and novel contribution.\n- The paper introduces a robust architecture combining embedding dropout, Gaussian weighted learning, and iterative data correction, addressing different aspects of the corruption challenge.\n- The experiments are extensive, covering several benchmarks and a variety of corruption scenarios, demonstrating consistent outperformance of RDT over baselines.\n- The study highlights the potential of sequence modeling to handle real-world data corruption in offline RL, an area with increasing practical relevance.", "weaknesses": "-The datasets used are predominantly standard RL benchmarks. Incorporating more complex, real-world datasets could provide stronger support for the robustness claims in real-world applications.\n\n- While the paper corrects actions and rewards through iterative data correction, it leaves out the correction of states, which might impact robustness in tasks with high state complexity.\n\n- The study lacks a detailed comparison with other robust sequence modeling approaches outside of decision transformers, such as those incorporating value functions or other regularization techniques.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5Qx88SN8pI", "reviewer": "Reviewer_bfdB", "summary": "This paper focus on invesgating the robustness of offline RL to various data corruption, including states, actions, rewards.  The problem getting more challengable as the dataset is limited. To tackle this problem, the paper proposed Robust Decision Transformer (RDT) by incorporating three effective components, embedding dropout, gaussian weighted learning and iterative data correction. The experiment results demonstrates RDT consistently outperforms the baselines on most tasks under limited corrupt dataset settings.", "questions": "1. In the gaussian weight learning section, the paper proposes using the exponential form of the prediction loss as a weight for the optimization terms in RDT. This design seems questionable, as early in training, the model is likely to be inaccurate, resulting in high loss values. However, this approach down-weights these losses, potentially reducing gradient information and slowing the learning process. Could the authors provide more intuition behind the design of these weighted terms?\n2. Based on the experimental results, RDT appears to provide greater benefits for Adroit tasks compared to the other two task settings. Could you explain the reason for this difference?\n3. I reviewed the full dataset results in RIQL, and they seem to conflict with the scores listed in the paper’s appendix. Could you clarify why this discrepancy occurs?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written, and the main idea is easy to follow.\n2. The problem setting is interesting, and the proposed method is straightforward, effectively demonstrating its efficiency in handling corrupted, limited datasets.", "weaknesses": "1. There is no discussion regarding corrupted data at different scales. An analysis of how dataset scale affects the challenges of data corruption would be beneficial.\n2. The proposed method is only evaluated on the medium-replay dataset for MuJoCo tasks and the expert dataset for Adroit. A discussion on the impact of corrupted data with varying quality levels is needed.\n3. No discussion of the limitations of RDT, which would help in fairly assessing the proposed approach\n4. There is a typographical error on line 452: \"relocate-exper\" should be corrected to \"relocate-expert.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5CCaLzqNWw", "reviewer": "Reviewer_jkcs", "summary": "This paper attacks the problem of learning a policy in an offline RL setting, in the specific case of having access to a lossy dataset, where data can be perturbed. While the literature has focused on algorithms leveraging the Q function to handle this problem, this paper shows that sequence-based models such as the Decision Transformer (DT) perform competitively, and are surprisingly robust compared to baselines when the size of the dataset is significantly reduced. The authors then propose 3 modifications to the DT algorithm that lead to algorithm even more robust than DT. They evaluate the performance of their proposed algorithm in the D4RL benchmark, and perform several ablations to get a better understanding of their method.", "questions": "- In Section 3.2.2, you add a reward prediction term that predicts the immediate reward. Unless I am mistaken, you tested your model on dense reward environments, where such a prediction makes sense and might be more aligned with the environment goal. Do you expect your method to work well in non-dense reward environment? If not, do you think “dense reward signal” should be added as a constraint of your method? \n- In Section 3.2.2 still, what happens at the beginning of training, when all examples are unknown? How do you differentiate corrupted examples from natural ones, when we expect the model’s prediction to be poor for all inputs?\n- Section 3.2.3 : Have you tested how frequently your assumption was true? I am curious to know the proportion of times the z score actually detects the perturbed transitions. \n- In Table 1, only RDT gets an uncertainty measure. Why is that? \n- In Figure 3, different environments are chosen to showcase the effects of the elements forming the robust DT method (which are also different from the ones of the ablation in Fig. 6). How were these environments chosen? Why not use the same environment? If there is a difference in behavior of the RDT additions across environments, is there a pattern? \n- In Figure 6, could you add DT without any modification as a baseline?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper has several strengths.\n- I found the introduction did a good job of introducing the problem and motivating the research direction. This is particularly true for the introduction, where the overarching research question was clearly stated. Overall, the paper was well written and pleasant to read. The structure of the document is logical and progressively builds the proposed method. The different contributions were mostly clearly presented, using equations when necessary to support the text. Finally, Fig. 2 was a welcome addition to summarize the modifications brought by the algorithm to the DT architecture.\n- Offline RL is an important problem, with many applications in the real-world; I think the community benefits of works improving the performance of policies in that setting, and even more since it includes using a new policy architecture. I find it very sensible to wonder how sequence models fare in that setting, especially since their influence and importance in the literature has grown steadily over the past years. The modules to add to the DT are well motivated and sensible (especially the first 2). In addition, the modules seem rather easy to implement, which makes the paper easily applicable.\n- I found the paper did a good job at studying comprehensively the use of DT-like methods in the offline RL literature. Several ablations are done to isolate the contribution of the different elements of RDT (Fig. 3 and Fig. 6). Moreover, there is a large variety of additional experiments in the appendix that naturally answered some of my questions as I read the main text, such as the influence of the value of $\\zeta$ and $\\beta$ (Fig. 10) or the variation of performance for different corruption scale values (Fig. 11).\n- The proposed method delivers, as it gets robust performance in the small scale version of the D4RL dataset, improving over the baselines considered (Table 1), sometimes significantly.\n- Ultimately, I think the paper asks an interesting question in the use sequence-based methods for robust offline learning. I believe this can generate interesting discussions at the conference around the usage of sequence-based models.", "weaknesses": "There are a couple of axes where I found weaknesses in this work, which I present below.\n- Several of the methodological details are pushed to the appendix without being explicitly referred to in the main text. For instance, it is hinted in the main text that the choice of the $\\zeta$ parameter (L301) or when the data iterative process starts (L292, “after sufficient training”) is important, but you have to go to the appendix to discover that this information is indeed provided. Adversarial corruption is mentioned quickly (L132) but without giving an intuition about how it would be used in this setting; the action diff perturbation (L426-428) could also have used some intuition (generic is OK) about it before being pushed to the appendix. Overall, I find the authors could make the main text more self-contained by giving the conclusions (maybe broadly) of their different experiments in the main text and referring to the appendix only for the in-depth analysis of the results. Methodological details can be pushed to the appendix, but as the reader goes through the paper, they should know that these details do, in fact, exist.\n- While the three proposed additions to DT presented in Sec. 3 make sense, I found their concrete impact on performance difficult to evaluate. For instance, in the ablation of Fig. 6, the addition of the different modules to DT do not drastically change the performance — in fact, in several settings the confidence interval of RDT intersects with the DT(RP). I would be also curious to know how these ablated methods compare with unmodified DT in Fig. 6. Moreover, it seems that different modules contribute more in different settings (ED on action attack, GWL on reward attack …); I think some analysis about which environment should benefit from which module the most, if possible, would have helped framing the modules. The fact that the different ablations of Fig. 6 and 3 are evaluated on different environments also makes it difficult to get a clear sense of the general behavior of each module.\n- I find the setting in which this method is expected to perform well to be relatively narrow, as it is mostly valid for datasets that are simultaneously small and corrupted. I do not think this makes this submission less valuable, but it raises questions that I would have liked to see addressed — for instance, what aspect exactly of the low data regime favors sequence models over Q-function based models? Is there any hope for sequence models to overcome Q-function models, even with bigger datasets (especially since transformer models have been shown to work so great in large data regimes)? On this point, while the problem is well-motivated, I found the conclusion that DT worked better than Q-function based models could have used more insights. It is explained at the end of Sec 3.1 (L187-189) but I found these two sentences did not bring much besides repeating the definition of a sequence model. \n- Only RDT has an uncertainty measure in Table 1, and there is no uncertainty measure in Table 2. Moreover, I did not find a description of the nature of the uncertainty measure in the main text.\n\nIn addition, I list below other minor weaknesses which did not impact my rating, but that I think could be addressed to improve the document:\n- The text of the figures is small and makes reading the paper difficult on paper. (Especially true for Fig. 3 that was very hard to read). It would be useful to increase the size of the captions to make the figure more readable.\n- L. 98: “Different from the MDP framework, DT [..]”. I would argue that DT still operates within the MDP framework, it is just a different algorithm to learn a policy.\n- L246, Eq 4: you did not define the circle-cross operator. If this is supposed to be an elementwise product, I would argue that this is not the most appropriate symbol (\\odot would, I think, be more appropriate). \n- L535: I found the term “exceptional” to be rather out of place.\n- Typo: MoJoCo, while it should be MuJoCo (several places).\n- Table 1: Lacking information about the meaning of +- in the RDT column. What exact uncertainty measure did you use? \n- In the experiments section, I would refer to the papers that introduced the experiments parameters setting (for example, you set the corruption scale to 1., but unless I am mistaken, we see in the appendix only that this value is set the same as previous works).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hdbFHkd1Ig", "reviewer": "Reviewer_GDNb", "summary": "This paper introduces a novel method, the Robust Decision Transformer (RDT), designed to address data corruption in offline reinforcement learning. The study finds that traditional offline reinforcement learning methods based on temporal difference learning perform poorly under data corruption, while the DT, a sequence modeling approach, demonstrates greater robustness in such scenarios. RDT enhances DT’s robustness by incorporating three simple yet effective techniques: embedding dropout, Gaussian-weighted learning, and iterative data correction. Experimental results show that RDT significantly outperforms traditional methods across various tasks in the presence of data corruption, especially in complex scenarios where both training and testing phases are affected by attacks.", "questions": "1. Figure 1 is not as intuitive as it could be; it might be more effective to directly display the performance decline under various attacks.\n2. In the ablation study, the current approach is to compare RDT with DT(RP) that incorporates three different improvements. A more rational method would be to compare the performance of RDT with RDT w/o (ED, GWL and IDC).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Innovation: This approach introduces sequence modeling to address data corruption, moving beyond traditional frameworks based on temporal difference learning.\n2. Enhanced Robustness: Embedding dropout, Gaussian-weighted learning, and iterative data correction significantly improve the algorithm’s performance under data corruption.\n3. Extensive Validation: The method has been validated across various data corruption scenarios and with limited datasets, demonstrating its robustness.\n4. High Adaptability: RDT performs exceptionally well in limited data settings, showcasing the potential of sequence modeling across diverse tasks and data conditions.", "weaknesses": "No obvious weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)