Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modeling

## Abstract

Abstract Learning policies from offline datasets through offline reinforcement learning (RL) holds promise for scaling data-driven decision-making and avoiding unsafe and costly online interactions. However, real-world data collected from sensors or humans often contains noise and errors, posing a significant challenge for existing offline RL methods. Our study indicates that traditional offline RL methods based on temporal difference learning tend to underperform Decision Transformer (DT) under data corruption, especially when the amount of data is limited. This suggests the potential of sequential modeling for tackling data corruption in offline RL. To further unleash the potential of sequence modeling methods, we propose Robust Decision Transformer (RDT) by incorporating several robust techniques. Specifically, we introduce Gaussian weighted learning and iterative data correction to reduce the effect of corrupted data. Additionally, we leverage embedding dropout to enhance the model’s resistance to erroneous inputs. Extensive experiments on MoJoCo, KitChen, and Adroit tasks demonstrate RDT’s superior performance under diverse data corruption compared to previous methods. Moreover, RDT exhibits remarkable robustness in a challenging setting that combines training-time data corruption with testing-time observation perturbations. These results highlight the potential of robust sequence modeling for learning from noisy or corrupted offline datasets, thereby promoting the reliable application of offline RL in real-world tasks.

## 1 Introduction

Offline reinforcement learning (RL) aims to derive near-optimal policies from fully offline datasets (Levine et al.,, ; Wang et al.,, ; Fujimoto et al.,, ; Kumar et al.,, ), thereby reducing the need for costly and potentially unsafe online interactions with the environment. However, offline RL encounters a significant challenge known as distribution shift (Levine et al.,, ), which can lead to performance degradation. To address this challenge, several offline RL algorithms impose policy constraints (Wang et al.,, ; Fujimoto et al.,, ; Fujimoto and Gu,, ; Kostrikov et al.,, ; Park et al.,, ) or maintain pessimistic values for out-of-distribution (OOD) actions (Kumar et al.,, ; An et al.,, ; Bai et al.,, ; ; Ghasemipour et al.,, ), ensuring that the learned policy aligns closely with the training distribution. Although the majority of traditional offline RL methods rely on temporal difference learning, an alternative promising paradigm for offline RL emerges in the form of sequence modeling (Chen et al.,, ; Janner et al.,, ; Furuta et al.,, ; Shi et al.,, ; Wu et al.,, ). Unlike traditional RL methods, Decision Transformer (DT) (Chen et al.,, ), a representative sequence modeling method, treats RL as a supervised learning task, predicting actions directly from sequences of reward-to-gos, states, and actions. Prior work (Bhargava et al.,, ) suggests that DT excels at handling tasks with sparse rewards and suboptimal quality data, showcasing the potential of sequence modeling for real-world applications.

When deploying offline RL in practical settings, dealing with noisy or corrupted data resulting from data collection or malicious attacks is inevitable (Zhang et al.,, ; Liang et al.,, ; ). Therefore, robustly learning policies from such data becomes crucial for the deployment of offline RL. A series of prior works (Zhang et al.,, ; ; Wu et al.,, ; Chen et al.,, ; ) focus on the theoretical properties and certification of offline RL under data corruption. Notably, a recent study by proposes an uncertainty-weighted offline RL algorithm with Q ensembles to address reward and dynamics corruption, while enhance the robustness of Implicit Q-Learning (IQL) (Kostrikov et al.,, ) against corruptions on all elements using Huber loss and quantile Q estimators.
These advancements, predominantly based on temporal difference learning without utilizing sequence modeling methods, prompt the intriguing question: Can sequential modeling method effectively handle data corruption in offline RL?

In this study, we conduct a comparative analysis of DT with traditional offline RL methods under different data corruption settings. Surprisingly, we observe that traditional methods relying on temporal difference learning struggle in limited data settings and notably underperform DT. Moreover, DT outperforms prior methods under state attack scenarios. These observations highlight the promising potential of sequential modeling for addressing data corruption challenges in offline RL. To further unlock the capabilities of sequential modeling methods under data corruption, we propose a novel algorithm called Robust Decision Transformer (RDT). RDT incorporates three robust techniques for DT, including Gaussian weighted learning, iterative data correction, and embedding dropout to mitigate the impact of corrupted data.

Through comprehensive experiments conducted on a diverse set of tasks, including MoJoCo, KitChen, and Adroit, we demonstrate that RDT outperforms conventional temporal difference learning and sequence modeling approaches under both random and adversarial data corruption scenarios. Additionally, RDT exhibits remarkable robustness in a challenging setting that combines training-time data corruption with testing-time observation perturbations. Our study emphasizes the significance of robust sequential modeling and offers valuable insights that would contribute to the trustworthy deployment of offline RL in real-world applications.

## 2 Preliminaries

### 2.1 RL and Offline RL

RL is generally formulated as a Markov Decision Process (MDP) defined by a tuple $(S,A,P,r,\gamma)$. This tuple comprises a state space $S$, an action space $A$, a transition function $P$, a reward function $r$, and a discount factor $\gamma\in[0,1)$. The objective of RL is to learn a policy $\pi(a|s)$ that maximizes the expected cumulative return:

$$ $\max_{\pi}\mathbb{E}_{s_{0}\sim\rho_{0},a_{t}\sim\pi(\cdot|s_{t}),s_{t+1}\sim P (\cdot|s_{t},a_{t})}\left[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t})\right],$ $$

where $\rho_{0}$ denotes the distribution of initial states.
In offline RL, the objective is to optimize the RL objective with a previously collected dataset $\mathcal{D}=\{(s_{t}^{(i)},a_{t}^{(i)},r_{t}^{(i)},s_{t+1}^{(i)})_{t=0}^{T-1}
\}_{i=0}^{N-1}$. which consists of trajectories of length $T$ and a total of $N$ trajectories. The agent cannot directly interact with the environment during the offline phase.

### 2.2 Decision Transformer (DT)

Different from the MDP framework, DT models decision making from offline datasets as a sequential modeling problem. The $i$-th trajectory $\tau^{(i)}$ of length $T$ in dataset $\mathcal{D}$ is reorganized into a sequence of return-to-go $R_{t}^{(i)}$, state $s_{t}^{(i)}$, action $a_{t}^{(i)}$:

$$ $\tau^{(i)}=\left(R_{0}^{(i)},s_{0}^{(i)},a_{0}^{(i)},\ldots,R_{T-1}^{(i)},s_{T -1}^{(i)},a_{T-1}^{(i)}\right)$ (1) $$

Here, the return-to-go $R_{t}^{(i)}$ is defined as the sum of rewards from the current step to the end of the trajectory: $R_{t}^{(i)}=\sum_{t^{\prime}=t}^{T}r_{t^{\prime}}^{(i)}$.
DT employs three linear projection layers to project the return-to-gos, states, and actions to the embedding dimension, with an additional learned embedding for each timestep added to each token. A GPT model is adopted by DT to autoregressively predict the actions $a_{t}^{(i)}$ with input sequences of length $K$:

$$ $\mathcal{L}_{DT}(\theta)=\mathbb{E}_{\tau^{(i)}\sim\mathcal{D}}\left[\frac{1}{ K}\sum_{t=0}^{K-1}\|\pi_{\theta}(\tau_{t-K+1:t-1}^{(i)},R_{t}^{(i)},s_{t}^{(i) })-a_{t}^{(i)}\|_{2}^{2}\right]$ (2) $$

where $\tau_{t-K+1:t-1}^{(i)}$ indicates the segment of $\tau^{(i)}$ from timestep $t-K+1$ to $t-1$.

### 2.3 Data Corruption in Offline RL

In prior works (; ), the data is stored in transitions, and data corruption is performed on individual elements (state, action, reward, next-state) of each transition. This approach does not align well with trajectory-based sequential modeling methods like DT.
In this paper, we consider a unified trajectory-based storage, where corrupting a next-state in a transition corresponds to corrupting a state in the subsequent transition, while corruption for rewards and actions is consistent with prior works. To elaborate, an original trajectory is denoted as $\tau_{\rm{origin}}^{(i)}=\left(s_{0}^{(i)},a_{0}^{(i)},r_{0}^{(i)},\ldots,s_{T
-1}^{(i)},a_{T-1}^{(i)},r_{T-1}^{(i)}\right)$, which can be reorganized into the sequence data of DT in Eq. 1 or split into $T-1$ transitions $\left(s_{t}^{(i)},a_{t}^{(i)},r_{t}^{(i)},s_{t+1}^{(i)}\right)_{t=0}^{T-2}$ for MDP-based methods. Note that here only are three independent elements (i.e., states, actions, and rewards) under the trajectory-based storage formulation.

Data corruption injects random or adversarial noise into the original states, actions, and rewards. Random corruption adds random noise to the affected elements in the datasets, resulting in a corrupted trajectory denoted as $\tau_{\rm{corrupt}}^{(i)}=\left(\hat{s}_{0}^{(i)},\hat{a}_{0}^{(i)},\hat{r}_{0
}^{(i)},\ldots,\hat{s}_{T-1}^{(i)},\hat{a}_{T-1}^{(i)},\hat{r}_{T-1}^{(i)}\right)$. For instance, $\hat{s}_{0}^{(i)}=s_{0}^{(i)}+\lambda\cdot\textrm{std}(s),\lambda\sim\rm{
Uniform}[-\epsilon,\epsilon]^{d_{s}}$ ($d_{s}$ is the dimensions of state, and $\epsilon$ is the corruption scale) and $\textrm{std}(s)$ is the $d_{s}$-dimensional standard deviation of all states in the offline dataset.
In contrast, adversarial corruption uses Projected Gradient Descent attack (Madry et al.,, ) with pretrained value functions. More details about data corruption are provided in Appendix C.1.

## 3 Sequence Modeling for Offline RL with Data Corruption

We aim to answer the question of whether sequential modeling methods can effectively handle data corruption in offline RL in Section 3.1. To achieve this, we compare DT with prior offline RL methods in the context of data corruption. Based on the insights gained from the motivating example, we then propose enhancements for improving the robustness of DT in Section 3.2.

Figure: Figure 1: Average normalized scores of offline RL algorithms under random data corruption across three MoJoCo tasks (Halfcheetah, Walker and Hopper) using "medium-replay-v2" datasets. Many offline RL algorithms experience a substantial performance decline when subjected to data corruption. Notably, DT demonstrates superior robustness in the face of state corruption and limited data regimes.
Refer to caption: x1.png

### 3.1 Motivating Example

As illustrated in Figure 1, we compare DT with various offline RL algorithms under data corruption. We apply random corruption introduced in Section 2.3 on states, actions, and rewards. Specifically, we perturb $30\%$ of the transitions in the dataset and introduce random noise at a scale of 1.0 standard deviation. In addition to the conventional full dataset setting employed in previous studies (; ), we also explore a scenario with a reduced dataset, comprising only $10\%$ of the original trajectories. Our results reveal that most offline RL methodologies are susceptible to data corruption, particularly in the limited data setting. In contrast, DT exhibits remarkable robustness to data corruption, particularly in cases of state corruption and all limited dataset settings. This resilience can be attributed to DT’s formulation based on sequence modeling and its supervised training paradigm. We will later discuss more in-depth about the important components impacting the robustness of DT in Section D.1.
In the full dataset setting, it is worth noting that DT exhibits slightly lower performance compared to RIQL and UWMSG when subjected to action and reward attacks, indicating room for further improvement.

Overall, the results demonstrate DT can outperform prior imitation-based and temporal-difference based methods without any robustness enhancement. This observation raises an intriguing question:

How can we further unleash the potential of sequential modeling in addressing data corruption with limited dataset in offline RL?

### 3.2 Robust Decision Transformer

To enhance the robustness of DT against various data corruption, we propose Robust Decision Transformer (RDT) by incorporating three components: embedding dropout (Section 3.2.1), Gaussian weighted learning (Section 3.2.2), and iterative data correction (Section 3.2.3). The overall framework is depicted in Figure 2. Notably, RDT predicts actions and rewards (rather than reward-to-gos) on top of the DT. Given the typically high dimensionality of states, we avoid predicting states to mitigate potential negative impacts. Additionally, rewards provide more direct supervision for the policy compared to reward-to-gos, which are also dependent on future actions.

Figure: Figure 2: Framework of Robust Decision Transformer (RDT). RDT enhances the robustness of DT against data corruption by incorporating three components on top of DT: embedding dropout, Gaussian weighted learning, and iterative data correction.
Refer to caption: x2.png

#### 3.2.1 Embedding Dropout

In the setting of data corruption, corrupted states, actions, and rewards can supply shifted inputs or erroneous features to the model. This can lead to a performance drop if the model overfits to particular harmful features. Therefore, learning a robust representation is important to enhance the model’s resilience against data corruption scenario. To achieve this, we employ embedding dropout, which encourages the model to learn more robust embedding representations and avoid overfitting (Merity et al.,, ).

To clarify, we define three hidden embeddings after the linear projection layer as $h_{R_{t}}$, $h_{s_{t}}$, and $h_{a_{t}}$, represented as:

$$ $h_{R_{t}}=\phi_{R}(R_{t})+\phi_{t}(t),\quad h_{s_{t}}=\phi_{s}(s_{t})+\phi_{t} (t),\quad h_{a_{t}}=\phi_{a}(a_{t})+\phi_{t}(t).$ $$

Here, $\phi_{R}$, $\phi_{s}$, and $\phi_{a}$ denote linear projection layers on different elements, while $\phi_{t}(t)$ represents the time-step embedding. Subsequently, we apply randomized dimension dropping on these feature embeddings, resulting in the corresponding masked feature embeddings as follows:

$$ $\tilde{h}_{R_{t}}=\mathcal{M}(p)\otimes h_{R_{t}},\quad\tilde{h}_{s_{t}}= \mathcal{M}(p)\otimes h_{s_{t}},\quad\tilde{h}_{a_{t}}=\mathcal{M}(p)\otimes h _{a_{t}}.$ (3) $$

The function $\mathcal{M}(\cdot)$ takes a probability as input and outputs a binary mask to determine the dimension inclusion. We use the same drop probabilities $p$ for different elements embedding.

#### 3.2.2 Gaussian Weighted Learning

In DT, actions serve as the most crucial supervised information, acting as the labels. Consequently, erroneous actions can directly influence the model through backpropagation. In RDT, we predict both actions and rewards, with rewards helping to reduce overfitting to corrupted action labels. To further mitigate the impact of corrupt data, we aim to reduce the influence of unconfident action and reward labels that could erroneously guide policy learning.

To identify uncertain labels, we adopt a simple but effective method: we use the value of the sample-wise loss to adjust the weight for the DT loss. The underlying insight is that a corrupted label would typically lead to a larger loss. To softly reduce the effect of potentially corrupted labels, we use the Gaussian weight, i.e., a weight that decays exponentially in accordance with the sample’s loss. This is mathematically represented as:

$$ $\displaystyle w_{a_{t}}^{(i)}=e^{-\beta_{a}\cdot\delta_{a_{t}}^{2}},$ $\displaystyle w_{r_{t}}^{(i)}=e^{-\beta_{r}\cdot\delta_{r_{t}}^{2}},\text{ where }\delta_{a_{t}}=\text{no_grad}(\|\pi_{\theta}(\hat{\tau}_{t-K+1:t-1}^{( i)},\hat{R}_{t}^{(i)},\hat{s}_{t}^{(i)})-\hat{a}_{t}^{(i)}\|_{2}),$ (4) $\displaystyle\delta_{r_{t}}=\text{no_grad}(\|\pi_{\theta}(\hat{\tau}_{t-K+1:t -1}^{(i)},\hat{R}_{t}^{(i)},\hat{s}_{t}^{(i)},\hat{a}_{t}^{(i)})-\hat{r}_{t}^{ (i)}\|_{2}),$ $$

In this equation, $\delta_{a_{t}}$ and $\delta_{r_{t}}$ represent the detached prediction errors at step $t$. The variables $\beta_{a}\geq 0,\beta_{r}\geq 0$ act as the temperature coefficients, providing flexibility to control the "blurring" effect of the Gaussian weights. With a larger value, more samples would be down-weighted. The loss function of RDT is expressed as follows:

$$ $\displaystyle\mathcal{L}_{RDT}(\theta)=\mathbb{E}_{\hat{\tau}^{(i)}\sim \mathcal{D}}$ $\displaystyle\Bigg{[}\frac{1}{K}\sum_{t=0}^{K-1}\Big{[}w_{a_{t}}^{(i)}\|\pi_{ \theta}(\hat{\tau}_{t-K+1:t-1}^{(i)},\hat{R}_{t}^{(i)},\hat{s}_{t}^{(i)})-\hat {a}_{t}^{(i)}\|_{2}^{2}$ (5) $\displaystyle+w_{r_{t}}^{(i)}\|\pi_{\theta}(\hat{\tau}_{t-K+1:t-1}^{(i)},\hat{ R}_{t}^{(i)},\hat{s}_{t}^{(i)},\hat{a}_{t}^{(i)})-\hat{r}_{t}^{(i)}\|_{2}^{2} \Big{]}\Bigg{]}$ $$

Gaussian weighted learning enables us to mitigate the detrimental effects of corrupted labels, thereby enhancing the algorithm’s robustness.

#### 3.2.3 Iterative Data Correction

While Gaussian weighted learning has significantly reduced the impact of corrupted labels, these erroneous data can still influence the policy as they still serve as inputs for policy learning. We propose iteratively correcting corrupted data in the dataset using the model’s predictions to bring the data closer to their true values in the next iteration. This method can further minimize the detrimental effects of corruption and can be implemented to correct reward-to-gos, states, and actions in the sequence modeling framework. In RDT, it is straightforward to correct the actions and rewards in the dataset, and recalculate the reward-to-gos using corrected rewards. Therefore, our implementation focuses on correcting states and reward-to-gos in the datasets, leaving better data correction methods for states for future work.

Initially, we store the distribution information of prediction error $\delta$ in Eq.4 throughout the learning phase to preserve the mean $\mu_{\delta}$ and variance $\sigma^{2}_{\delta}$ of actions and rewards, updating them with every batch of samples. Our hypothesis is that the prediction error $\delta$ between predicted and clean label actions should exhibit consistency after sufficient training. Therefore, if erroneous label actions are encountered, $\delta$ will deviate from the mean $\mu_{\delta}$, behaving like outliers. This deviation essentially enables us to detect and correct the corrupted action using the distributional information of $\delta$.

Taking actions as an example, to detect corrupted actions, we calculate the $z$-score, denoted by $z^{(i)}=\frac{\delta^{(i)}-\mu_{\delta}}{\sigma_{\delta}}$, for each sampled action $\hat{a}^{(i)}$ and the current policy. If the condition $z^{(i)}>\zeta\cdot\sigma_{\delta}$ is met for any given action $\hat{a}^{(i)}$, we infer that the action $\hat{a}^{(i)}$ has been corrupted. We then permanently replace $\hat{a}^{(i)}$ in the dataset with the predicted action. Correcting rewards follows a similar process, with the additional step of recalculating the reward-to-gos. The hyperparameter $\zeta$ determines the detection thresholds: a smaller $\zeta$ will classify more samples as corrupted data, while a larger $\zeta$ results in fewer modifications to the dataset.

## 4 Experiments

In this section, we conduct a comprehensive empirical assessment of RDT to address the following questions: 1) How does RDT perform under several data corruption scenarios during the training phase? 2) Is RDT robust to different data corruption rates and scales? 3) What is the impact of each component of RDT on its overall performance? 4) How does RDT perform when faced with observation perturbations during the testing phase?

### 4.1 Experimental Setups

We evaluate RDT across a variety of tasks, such as MoJoCo ("medium-replay-v2" datasets), KitChen, and Adroit (Fu et al.,, ). Two types of data corruption during the training phase are simulated: random corruption and adversarial corruption, attacking states, actions, and rewards, respectively.
The corruption rate is set to $0.3$ and the corruption scale is $\epsilon=1.0$ for the main results.
Data corruption is introduced in Section 2.3 and more details can be found in Section C.1.
Additionally, we find that the data corruption problem can exacerbate when the data is limited. To simulate this scenario, we down-sample on both MuJoCo and Adroit tasks, referred to as MuJoCo (10%) and Adroit (1%), Specifically, we randomly select 10% (and 1%) of the trajectories from MuJoCo (and Adroit) tasks and conduct data corruption on the sampled data. We do not down-sample the Kitchen dataset because it already has a limited dataset size. All considered tasks have a similar dataset size of about 20,000 transitions, except for the original MuJoCo task, which includes a larger dataset.

We compare RDT with several SOTA offline RL algorithms and robust methods, namely BC, RBC (Sasaki and Yamashina,, ), DeFog (Hu et al.,, ), CQL (Kumar et al.,, ), UWMSG (), RIQL (), and DT (Chen et al.,, ). BC and RBC employ behavior cloning loss within an MLP-based model for policy learning, while DeFog and DT utilize a Transformer architecture. We implement RBC with our Gaussian-weighted learning as an instance of  (Sasaki and Yamashina,, ). In Appendix D.1, we investigate the robustness of DT across various critical parameters. Drawing from these findings on DT’s robustness, we establish a default implementation of DT, DeFog, RDT with a sequence length of 20 and a block number of 3. To ensure the validity of the findings, each experiment is repeated using 4 different random seeds.

Table: Table 1: Average normalized performance under random data corruption.

### 4.2 Evaluation under Various Data Corruption

##### Random Data Corruption.

To address the first question, we evaluate the RDT and baselines on random data corruption across a diverse set of task groups. Each group consists of three datasets, and the mean normalized score was calculated to represent the final outcomes.
As shown in Table 1, RDT demonstrates superior performance in handling data corruption, achieving the highest score in 10 out of 12 settings, particularly in the challenging Kitchen and Adroit tasks. Notably, across all task groups, RDT outperforms DT, underscoring its effectiveness in reducing DT’s sensitivity to data corruption across all elements. Moreover, RBC significantly outperforms BC, further highlighting the effectiveness of Gaussian weighted learning. However, prior offline RL methods, such as RIQL and UWMSG, fail to yield satisfactory results and even underperform BC overall, despite showing enhanced performance on the full MuJoCo dataset. This indicates that temporal difference methods are dependent on large-scale datasets.

Figure: Figure 3: Results under mixed random data corruption.
Refer to caption: x3.png

##### Mixed Random Data Corruption.

To present a more challenging setting for the robustness of RDT, we conduct an experiment under the mixed random data corruption setting. In this setting, all three elements - states, actions, and rewards - are corrupted at a rate of $0.3$ and a scale of $\epsilon=1.0$. The experimental results are shown in Figure 3. Remarkably, RDT consistently outperforms other baseline models across all tasks, thereby highlighting its superior stability even when faced with simultaneous and diverse random data corruption. Especially on the challenging Kitchen task, RDT outperforms DT by an impressive margin of approximately 40$\%$.

##### Adversarial Data Corruption.

We further extend the analysis to examine the robustness of the RDT under adversarial data corruption. We employ the same hyperparameters for RDT as those utilized in the random data corruption scenarios, demonstrating RDT’s advantage of not requiring meticulous hyperparameter tuning.
As evidenced in Table 2, RDT consistently showcases a robust performance by attaining the highest scores in 8 out of 9 settings and sustaining the top average performance. Notably, RDT stands out for enhancing the average score by $28.9\%$ compared to DT in the adversarial action corruption scenario. Intriguingly, temporal-difference methods like CQL, UWMSG, and RIQL perform notably worse than BC and sequence modeling methods such as DT and DeFog, highlighting the promise of sequence modeling approaches.
We provide detailed results for each task in Appendix D for comprehensive analysis.

Table: Table 2: Average normalized performance under adversarial data corruption.

### 4.3 Varying Corruption Rates and Scales

In the above experiments, we evaluate the effectiveness of RDT under a data corruption rate of $0.3$ and a scale of $1.0$. We further examine the robustness of RDT under different corruption rates from $\{0.0,0.1,0.3,0.5\}$ and scales from $\{0.0,1.0,2.0\}$. As illustrated in Figure 4, RDT consistently delivers superior performance compared to other baselines across different corruption rates and scales. However, we did observe a decline in RDT’s performance when it encountered state corruption with high rates. This decline can be attributed to the distortion of the state’s distribution caused by the high corruption rates, resulting in a significant deviation from the clean test environment to the corrupted datasets. Enhancing the robustness of RDT against state corruption, especially under high corruption rates and scales, holds potential for future advancements.

Figure: Figure 4: Results under various data corruption rates (a) and scales (b) on "walker2d-medium-replay".
Refer to caption: x4.png

### 4.4 Ablation Study

We conduct comprehensive ablation studies to analyze the impact of each component on RDT’s robustness. For evaluation, we use the "walker2d-medium-replay", "kitchen-compete", and "relocate-exper"’ datasets. Specifically, we compare several variants of RDT: (1) DT(RP), which incorporates reward prediction in addition to the original DT; (2) DT(RP) w. ED, which adds embedding dropout to DT(RP); (3) DT(RP) w. GWL, which applies Gaussian weighted learning on top of DT(RP); and (4) DT(RP) w. IDC, which integrates only the iterative data correction method.

We evaluate the performance of these variants under different data corruption scenarios, as depicted in Figure 5. In summary, all variants demonstrate improvements over DT(RP), proving the effectiveness of the individual components. Notably, Gaussian weighted learning appears to provide the most significant contribution, particularly under reward attack. However, it is important to note that none of these tailored models outperform RDT, indicating that the integration of all proposed techniques is crucial for achieving optimal robustness. Additionally, we provide ablation results on reward prediction in Section D.2, which demonstrate that reward prediction also brings improvements under state and action corruption compared to DT.

Figure: Figure 5: Ablation study on the impact of different proposed components.
Refer to caption: x6.png

### 4.5 Evaluation under Observation Perturbation during the Testing Phase

In this section, we further investigate the robustness of RDT when deployed in perturbed environments after being trained on corrupted data, a challenging setting that includes both training-time and testing-time attacks. To address this, we evaluate RDT under two types of observation perturbations during the testing phase: random and action diff, following prior works (; Zhang et al.,, ). The perturbation scale is used to control the extent of influence on the observation. Detailed implementation of observation perturbations during the testing phase is provided in Section C.2.

The comparison results are presented in Figure 6, where all algorithms are trained on the offline dataset with action corruption and evaluated under observation perturbation. These results demonstrate the superior robustness of RDT under the two types of observation perturbations, maintaining stability even at a high perturbation scale of 0.5. Notably, RORL (), a SOTA offline RL method designed to tackle testing-time observation perturbation, fails to perform effectively on these tasks. Additionally, DT and RIQL experience significant performance drops as the perturbation scale increases. Further results comparing different algorithms trained on offline datasets with state or reward corruption can be found in Section D.4.

Figure: (a) Random observation perturbation.
Refer to caption: x7.png

## 5 Related Work

##### Robust Offline RL.

In offline RL, several works have focused on testing-time robustness against environment shifts (Shi and Chi,, ; ; Panaganti et al.,, ; Zhihe and Xu,, ). Regarding training-time robustness, Li et al., () explore various types of reward attacks in offline RL and find that certain biases can inadvertently enhance the robustness of offline RL methods to reward corruption. From a theoretical perspective, Zhang et al., () propose a robust offline RL algorithm utilizing robust supervised learning oracles. Additionally, employ uncertainty weighting to address reward and dynamics corruption, providing theoretical guarantees. The most relevant research by employs the Huber loss to handle heavy-tailedness and utilizes quantile estimators to balance penalization for corrupted data. Furthermore, Mandal et al., (); Liang et al., () enhance the resilience of offline algorithms within the RLHF framework.
It is important to note that these studies primarily focus on enhancing temporal difference methods, with no emphasis on leveraging sequence modeling techniques to tackle data corruption.

##### Transformers for RL.

Recent research has redefined offline RL decision-making as a sequence modeling problem using Transformer architectures (Chen et al.,, ; Janner et al.,, ). Unlike traditional RL methods, these studies treat RL as a supervised learning task at a trajectory level. A seminal work, Decision Transformer (DT) (Chen et al.,, ), uses trajectory sequences to predict subsequent actions. Trajectory Transformer (Janner et al.,, ) discretizes input sequences into tokens and employs beam search to predict the next action. These efforts have led to subsequent advancements. For instance, Prompt DT (Xu et al.,, ) integrates demonstrations for better generalization, while Xie et al., () introduce pre-training with future trajectory information. Q-learning DT (Yamagata et al.,, ) refines return-to-go using Q-values, and Agentic Transformer (Liu and Abbeel,, ) uses hindsight to relabel target returns. LaMo (Shi et al.,, ) leverages pre-trained language models for offline RL, and DeFog (Hu et al.,, ) addresses robustness in specific frame-dropping scenarios. Our work deviates from these approaches by focusing on improving robustness against data corruption in offline RL.

## 6 Conclusion

In this study, we investigate the robustness of offline RL algorithms under various data corruptions, with a specific focus on sequence modeling methods. Our empirical evidence suggests that current offline RL algorithms based on temporal difference learning exhibit significant susceptibility to data corruption, particularly in scenarios with limited data. To address this issue, we introduce the Robust Decision Transformer (RDT), a novel robust offline RL algorithm developed from the perspective of sequence modeling. Our comprehensive experiments highlight RDT’s exceptional robustness against both random and adversarial data corruption, across different corruption ratios and scales. Furthermore, we demonstrate RDT’s superiority in handling both training-time and testing-time attacks. We hope that our findings will inspire further research into the exploration of using sequence modeling methods to address data corruption challenges in increasingly complex and realistic scenarios.

## Appendix A Algorithm Pseudocode

To provide an overview and better understanding, we detail the implementation of Robust Decision Transformer (RDT) in  Algorithm 1.

Figure: Algorithm 1 Robust Decision Transformer (RDT)

## Appendix B Additional Related Works

Offline RL. Maintaining proximity between the policy and data distribution is essential in offline RL, as distributional shifts can lead to erroneous estimations (Levine et al.,, ). To counter this, offline RL algorithms are primarily divided into two categories.
The first category focuses on policy constraints on the learned policy (Wang et al.,, ; Fujimoto et al.,, ; Li et al.,, ; Fujimoto and Gu,, ; Kostrikov et al.,, ; Emmons et al.,, ; ; Sun et al.,, ; Xu et al.,, ; Park et al.,, ). The other category learns pessimistic value functions to penalize OOD actions (Kumar et al.,, ; Yu et al.,, ; An et al.,, ; Bai et al.,, ; ; Ghasemipour et al.,, ; Sun et al.,, ; Nikulin et al.,, ; Huang et al.,, ).
To enhance the potential of offline RL in handling more complex tasks, recent research has integrated advanced techniques like GAN (Vuong et al.,, ; Wang et al.,, ), transformers (Chen et al.,, ; Chebotar et al.,, ; Yamagata et al.,, ) and diffusion models (Janner et al.,, ; Hansen-Estruch et al.,, ; Wang et al.,, ).

## Appendix C Implementation Details

### C.1 Data Corruption Details during Training Phase

Our study utilizes both random and adversarial corruption across three elements: states, actions, and rewards. We consider a range of tasks including MuJoCo, Kitchen, and Adroit.
Particularly, we utilize the “medium-replay-v2” datasets in the MuJoCo tasks with sampling ratios of 100% and 10%, the “expert-v0” datasets in the Adroit tasks with a sampling ratio of 1%, and we employ full datasets for the tasks in the Kitchen due to their already limited data size. These datasets (Fu et al.,, ) are collected either during the training process of an SAC agent or from expert demonstrations, thereby providing highly diverse and representative tasks of the real world.
To control the overall level of corruption within the datasets, we introduce two parameters $c$ and $\epsilon$ following previous work (). The parameter $c$ signifies the rate of corrupted data within a dataset, whilst $\epsilon$ represents the scale of corruption observed across each dimension. We outline three types of random data corruption and present a comprehensive overview of a mixed corruption approach as follows. Note that in our setting, only three independent elements (i.e., states, actions, and rewards) are considered under the trajectory-based storage approach.

- •
Random state attack: We randomly sample $c\cdot N\cdot T$ states from all trajectories, where $N$ refer to the number of trajectories and $T$ represents the number of steps in a trajectory. We then modify the selected state to $\hat{s}=s+\lambda\cdot\text{std}(s),\lambda\sim\text{Uniform}[-\epsilon,
\epsilon]^{d_{s}}$. Here, $d_{s}$ represents the dimension of states, and “$\text{std}(s)$” is the $d_{s}$-dimensional standard deviation of all states in the offline dataset. The noise is scaled according to the standard deviation of each dimension and is independently added to each respective dimension.
- •
Random action attack: We randomly select $c\cdot N\cdot T$ actions from all given trajectories, and modify the action to $\hat{a}=a+\lambda\cdot\text{std}(a),\lambda\sim\text{Uniform}[-\epsilon,
\epsilon]^{d_{a}}$, where $d_{a}$ represents the dimension of actions and “$\text{std}(a)$” is the $d_{a}$-dimensional standard deviation of all actions in the offline dataset.
- •
Random reward attack: We randomly sample $c\cdot N\cdot T$ rewards from all given trajectories, and modify the reward to $\hat{r}\sim\text{Uniform}[-30\cdot\epsilon,30\cdot\epsilon]$. We multiply by 30 because we have noticed that offline RL algorithms tend to be resilient to small-scale random reward corruption (also observed in (Li et al.,, )), but would fail when faced with large-scale random reward corruption.

In addition, three types of adversarial data corruption are detailed as follows:

- •
Adversarial state attack: We first pretrain IQL agents with a Q function $Q_{p}$ and policy function $\pi_{p}$ on clean datasets. Then, we randomly sample $c\cdot N\cdot T$ states, and modify them to $\hat{s}=\min_{\hat{s}\in\mathbb{B}_{d}(s,\epsilon)}Q_{p}(\hat{s},a)$. Here, $\mathbb{B}_{d}(s,\epsilon)=\{\hat{s}||\hat{s}-s|\leq\epsilon\cdot\text{std}(s)\}$ regularizes the maximum difference for each state dimension. The optimization is implemented through Projected Gradient Descent similar to prior works (Madry et al.,, ; Zhang et al.,, ; ). Specifically, We first initialize a learnable vector $z\in[-\epsilon,\epsilon]^{d_{s}}$, and then conduct a 100-step gradient descent with a step size of 0.01 for $\hat{s}=s+z\cdot\text{std}(s)$, and clip each dimension of $z$ within the range $[-\epsilon,\epsilon]$ after each update.
- •
Adversarial action attack: We use the pretrained IQL agent with a Q function $Q_{p}$ and a policy function $\pi_{p}$. Then, we randomly sample $c\cdot N\cdot T$ actions, and modify them to $\hat{a}=\min_{\hat{a}\in\mathbb{B}_{d}(a,\epsilon)}Q_{p}(s,\hat{a})$. Here, $\mathbb{B}_{d}(a,\epsilon)=\{\hat{a}||\hat{a}-a|\leq\epsilon\cdot\text{std}(a)\}$ regularizes the maximum difference for each action dimension. The optimization is implemented through Projected Gradient Descent, as discussed above.
- •
Adversarial reward attack: We randomly sample $c\cdot N\cdot T$ rewards, and directly modify them to: $\hat{r}=-\epsilon\times r$.

### C.2 Observation Perturbation Details during the Testing Phase

We evaluate the Robust Decision Transformer (RDT) under two types of perturbations during the testing phase: random and action diff perturbations, as described in prior works (; Zhang et al.,, ). Both perturbation types target the observations during the testing phase. The detailed implementations are as follows:

- •
Random: We sample perturbed states within an $l_{\infty}$ ball of norm $\epsilon$. Specifically, we create the perturbation set $\mathbb{B}_{d}(s,\epsilon)=\{\hat{s}:d(s,\hat{s})\leq\epsilon\}$ for state $s$, where $d(\cdot)$ is the $l_{\infty}$ norm, and sample one perturbed state to return to the agent.
- •
Action Diff: This is an adversarial attack based on the pretrained IQL deterministic policy $\mu(s)$. We first sample 50 perturbed states within an $l_{\infty}$ ball of norm $\epsilon$ and then find the one that maximizes the difference in actions: $\max_{\hat{s}\in\mathbb{B}_{d}(s,\epsilon)}||\mu(s)-\mu(\hat{s})||^{2}$.

The parameter $\epsilon$ controls the extent of the observation perturbation. In this experiment, we first train RDT under different data corruption scenarios and then evaluate its performance in environments subjected to observation perturbation.

### C.3 Implementation Details of RDT

We implement the DT and RDT algorithms using the existing code base(^*^**[https://github.com/tinkoff-ai/CORL](https://github.com/tinkoff-ai/CORL)). Specifically. we build the network with 3 Transformer blocks, incorporating one MLP embedding layer for each key element: state, action, and return-to-go.
We update the neural network using the AdamW optimizer, with a learning rate set at $1\times 10^{-4}$ and a weight decay of $1\times 10^{-4}$. The batch size is set to 64, with a sequence length of 20, to ensure effective and efficient training. To maintain stability during training, we adopt the state normalization as in ().
During the training phase, we train DeFog, DT and RDT for 100 epochs and other baselines with 1000 epochs. Each epoch is characterized by 1000 gradient steps. For evaluative purposes, we rollout each agent in the clean environment across 10 trajectories, each with a maximum length of 1000, and we average the returns from these rollouts for comparison. We ensure the consistency and reliability of our reported results by averaging them over 4 unique random seeds. All experiments are conducted on 8 P40 GPUs.

As for the hyperparameters of RDT, we configure the values for the embedding dropout probability as $p$, Gaussian Weighted coefficient $\beta_{a}$ and $\beta_{r}$, and iterative data correction threshold $\zeta$ through the sweeping mechanism. We start the iterative data correction at the 50th epoch for all tasks. We use iterative action correction exclusively for action corruption and iterative reward correction exclusively for reward corruption; therefore, only a single $\zeta$ value is shown in Table 3. It’s noteworthy that we do not conduct a hyperparameter search for individual datasets; instead, we select hyperparameters that apply to the entire task group. As such, RDT doesn’t require extensive hyperparameter tuning as previous methods. The exact hyperparameters used for the random corruption experiment are detailed in Table 3. We employ the same hyperparameters for the adversarial corruption experiment, further attesting to the robustness and stability of RDT.

**Table 3: Hyperparameters used for RDT under the random corruption benchmark.**
| Tasks | Attack Element | $p$ | ($\beta_{a}$, $\beta_{r}$) | $\zeta$ |
| --- | --- | --- | --- | --- |
| MuJoCo | state | 0.0 | (1.0, 10.0) | $-$ |
| action | 0.0 | (0.1, 0.0) | 3.0 |  |
| reward | 0.0 | (0.1, 0.1) | 6.0 |  |
| MuJoCo (10%) | state | 0.1 | (0.1, 0.1) | $-$ |
| action | 0.1 | (0.1, 0.1) | 5.0 |  |
| reward | 0.2 | (0.1, 0.1) | 6.0 |  |
| Kitchen | state | 0.1 | (10.0, 10.0) | $-$ |
| action | 0.1 | (10.0, 10.0) | 1.5 |  |
| reward | 0.0 | (10.0, 10.0) | 3.0 |  |
| Adroit (1%) | state | 0.1 | (0.1, 0.0) | $-$ |
| action | 0.1 | (10.0, 0.0) | 6.0 |  |
| reward | 0.1 | (0.1, 0.1) | 6.0 |  |

## Appendix D Additional Experiments

### D.1 Impact of Decision Transformer’s Hyperparameters

We aim to understand the robustness characteristic of sequence modeling. The key feature of the DT is its utilization of sequence modeling, with the incorporation of the transformer model. Consequently, we identify two crucial factors that may influence the performance of DT in the context of sequence modeling: the model structure and the input data.

Figure: Figure 7: Ablation study on the robustness of DT. (a) Analyzing the influence of block number of DT while keeping sequence length with 20. (b) The effect of sequence length while maintaining 3 transformer blocks. (c) Investigation on how the return-to-go impacts DT. We create the variant DT w/o. Return which excludes the use of return-to-go as inputs.
Refer to caption: x9.png

We explore the influence of the model’s structure by modifying the number of Transformer blocks in DT. As depicted in Figure 7(a), the performance on both MuJoCo and Kitchen tasks consistently improves with an increase in the number of blocks. Therefore, the enhancement can be partially attributed to the increased capabilities of the larger model.

We also examine the impact of the inputs for the sequence model. There are two primary differences between the input of DT and that of conventional BC. DT predicts actions based on historical data and return-to-go elements, whereas BC relies solely on the current states. To assess the influence of these factors, we independently evaluate the impact of the length of history and return-to-go elements.
The results in Figure 7(b) and (c) reflect that Kitchen tasks favor longer historical input and are less reliant on return-to-go elements.
Conversely, MuJoCo tasks demonstrated a different trend.
This discrepancy could be due to the sparse reward structure of Kitchen tasks, where return-to-go elements do not provide sufficient information, leading the agent to rely more heavily on historical data for policy learning. In contrast, MuJoCo tasks, with their denser reward structures, rely more on return-to-go elements for optimal policy learning.

In summary, the robustness of DT can be attributed to the model capacity, history length, and the return-to-go conditioning. Based on this findings on the robustness of DT, we adopt a sequence length of 20 and a block number of 3 as the default implementation for all sequence modeling methods, including DT, DeFog, and RDT.

### D.2 Ablation Study on Reward Prediction

The nature of the sequential model allows us to predict not only actions but also other elements such as state and return-to-go. Previous studies have shown that predicting these additional elements does not significantly enhance performance Chen et al., (). However, we find that predicting rewards can provide advantages in the context of data corruption. We introduce DT(RP), a variant of the original Decision Transformer (DT), which predicts rewards in addition to actions. As illustrated in Figure 8, predicting rewards indeed improves performance under state and action corruption conditions. However, DT(RP) performs poorly under reward corruption due to the degraded quality of reward labels. This issue can be mitigated via embedding dropout, Gaussian weighted learning, and iterative data correction in RDT.

Figure: Figure 8: Ablation study on the impact of reward prediction.
Refer to caption: x12.png

### D.3 Ablation Study on Hyperparameters

We conducted experiments to investigate the impact of hyperparameters of the Robust Decision Transformer (RDT) under scenarios of action corruption.

In our study of the embedding dropout technique, depicted in Figure 9(a), we discovered that applying a moderate dropout probability (e.g., 0.1) to embeddings enhances resilience against data corruption. However, a larger dropout rate ($\geq$ 0.5) can lead to performance degradation.

As shown in Figure 9(b), regarding the Gaussian weighted coefficient $\beta_{a}$, the walker2d task favors lower values of $\beta_{a}$ such as 0.1, while the kitchen and relocate tasks favor larger $\beta_{a}$ values, suggesting that these tasks are more impacted by Gaussian weight learning.

In terms of the iterative data correction threshold $\zeta$, the results are highlighted in Figure 9(c). The walker2d and kitchen tasks favor lower values of $\zeta$, whereas the relocate task exhibits the opposite preference. Notably, in the relocate task, RDT shows markedly enhanced performance after correcting outliers (with $\zeta\geq 5$) in the dataset.

Figure: (a) Embedding drop $p$
Refer to caption: x13.png

### D.4 Additional Results under Observation Perturbation during the Testing Phase

To further demonstrate the robustness of RDT, we conducted additional evaluations under observation perturbation during the testing phase. Specifically, we compared different algorithms trained with offline datasets under conditions of state or reward corruption in perturbed environments. As shown in Figures 10 and 11, RDT consistently maintained superior performance and robustness across observation perturbations in both the Kitchen and Adroit tasks.

Figure: (a) "Random" observation perturbation.
Refer to caption: x16.png

Figure: (a) "Random" observation perturbation.
Refer to caption: x18.png

### D.5 Training Time

We trained all algorithms on the same P40 GPU, and the training times are presented in Table 4. As expected, BC has the shortest epoch time due to its simplicity. Notably, RDT requires only slightly more time per epoch than DT because it incorporates three robust techniques. Although RIQL requires less time per epoch than RDT, it necessitates 10 times more epochs to converge and still does not achieve satisfactory performance in the limited data regime. In conclusion, RDT achieves superior performance and robustness without imposing significant computational costs in terms of total training time.

**Table 4: Training time on the "walker2d-medium-replay" dataset.**
| Method | BC | CQL | UWMSG | RIQL | DT | RDT (ours) |
| --- | --- | --- | --- | --- | --- | --- |
| Epoch Num | 1000 | 1000 | 1000 | 1000 | 100 | 100 |
| Epoch Time (s) | 3.7 | 33.8 | 22.2 | 14.9 | 44.0 | 46.1 |
| Total Time (h) | 1.03 | 9.39 | 6.17 | 4.14 | 1.22 | 1.28 |

### D.6 Detailed Results for the Main Experiments

The detailed outcomes of each task, under both random and adversarial data corruption, can be found in Table 5 and Table 6 respectively. For HalfCheetah, Hopper, and Walker2D tasks, we have utilized the "medium-replay-v2" datasets, while the "expert-v0" datasets have been used for Door, Hammer, and Relocate tasks. Notably, RDT consistently outperforms baselines in the majority of tasks, achieving the best average performance and demonstrating its robustness and superiority, particularly under the adversarial data corruption.

Table: Table 5: Detailed performance under random data corruption.

Table: Table 6: Detailed performance under adversarial data corruption.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Could the authors provide more clarity on the choice of dropout probabilities for different embeddings (states, actions, rewards) and its sensitivity to these values? How were these values chosen in the experiments?

The paper states that actions and rewards are corrected iteratively but leaves out state correction. What challenges or limitations prevented state correction? Would the authors consider incorporating state correction as future work, and how would it theoretically affect robustness?
soundness: 3
strengths: - Novel Method: The use of sequence modeling as a robust alternative to traditional temporal-difference-based methods for offline RL under corrupted data conditions is a significant and novel contribution.
- The paper introduces a robust architecture combining embedding dropout, Gaussian weighted learning, and iterative data correction, addressing different aspects of the corruption challenge.
- The experiments are extensive, covering several benchmarks and a variety of corruption scenarios, demonstrating consistent outperformance of RDT over baselines.
- The study highlights the potential of sequence modeling to handle real-world data corruption in offline RL, an area with increasing practical relevance.
summary: The work deals with the challenges posed by data corruption in offline RL and proposes a novel approach RDT. The primary contribution is the application of sequence modeling, specifically through a decision transformer, which has shown greater robustness against corrupted data compared to conventional RL methods that rely on temporal difference learning. The RDT model incorporates three key techniques—embedding dropout, Gaussian weighted learning, and iterative data correction—to enhance robustness. Empirical results indicate that RDT achieves superior performance under various corruption scenarios and provides resilience to both training-time data corruption and testing-time observation perturbations.
weaknesses: -The datasets used are predominantly standard RL benchmarks. Incorporating more complex, real-world datasets could provide stronger support for the robustness claims in real-world applications.

- While the paper corrects actions and rewards through iterative data correction, it leaves out the correction of states, which might impact robustness in tasks with high state complexity.

- The study lacks a detailed comparison with other robust sequence modeling approaches outside of decision transformers, such as those incorporating value functions or other regularization techniques.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. In the gaussian weight learning section, the paper proposes using the exponential form of the prediction loss as a weight for the optimization terms in RDT. This design seems questionable, as early in training, the model is likely to be inaccurate, resulting in high loss values. However, this approach down-weights these losses, potentially reducing gradient information and slowing the learning process. Could the authors provide more intuition behind the design of these weighted terms?
2. Based on the experimental results, RDT appears to provide greater benefits for Adroit tasks compared to the other two task settings. Could you explain the reason for this difference?
3. I reviewed the full dataset results in RIQL, and they seem to conflict with the scores listed in the paper’s appendix. Could you clarify why this discrepancy occurs?
soundness: 2
strengths: 1. This paper is well-written, and the main idea is easy to follow.
2. The problem setting is interesting, and the proposed method is straightforward, effectively demonstrating its efficiency in handling corrupted, limited datasets.
summary: This paper focus on invesgating the robustness of offline RL to various data corruption, including states, actions, rewards.  The problem getting more challengable as the dataset is limited. To tackle this problem, the paper proposed Robust Decision Transformer (RDT) by incorporating three effective components, embedding dropout, gaussian weighted learning and iterative data correction. The experiment results demonstrates RDT consistently outperforms the baselines on most tasks under limited corrupt dataset settings.
weaknesses: 1. There is no discussion regarding corrupted data at different scales. An analysis of how dataset scale affects the challenges of data corruption would be beneficial.
2. The proposed method is only evaluated on the medium-replay dataset for MuJoCo tasks and the expert dataset for Adroit. A discussion on the impact of corrupted data with varying quality levels is needed.
3. No discussion of the limitations of RDT, which would help in fairly assessing the proposed approach
4. There is a typographical error on line 452: "relocate-exper" should be corrected to "relocate-expert."

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - In Section 3.2.2, you add a reward prediction term that predicts the immediate reward. Unless I am mistaken, you tested your model on dense reward environments, where such a prediction makes sense and might be more aligned with the environment goal. Do you expect your method to work well in non-dense reward environment? If not, do you think “dense reward signal” should be added as a constraint of your method? 
- In Section 3.2.2 still, what happens at the beginning of training, when all examples are unknown? How do you differentiate corrupted examples from natural ones, when we expect the model’s prediction to be poor for all inputs?
- Section 3.2.3 : Have you tested how frequently your assumption was true? I am curious to know the proportion of times the z score actually detects the perturbed transitions. 
- In Table 1, only RDT gets an uncertainty measure. Why is that? 
- In Figure 3, different environments are chosen to showcase the effects of the elements forming the robust DT method (which are also different from the ones of the ablation in Fig. 6). How were these environments chosen? Why not use the same environment? If there is a difference in behavior of the RDT additions across environments, is there a pattern? 
- In Figure 6, could you add DT without any modification as a baseline?
soundness: 3
strengths: This paper has several strengths.
- I found the introduction did a good job of introducing the problem and motivating the research direction. This is particularly true for the introduction, where the overarching research question was clearly stated. Overall, the paper was well written and pleasant to read. The structure of the document is logical and progressively builds the proposed method. The different contributions were mostly clearly presented, using equations when necessary to support the text. Finally, Fig. 2 was a welcome addition to summarize the modifications brought by the algorithm to the DT architecture.
- Offline RL is an important problem, with many applications in the real-world; I think the community benefits of works improving the performance of policies in that setting, and even more since it includes using a new policy architecture. I find it very sensible to wonder how sequence models fare in that setting, especially since their influence and importance in the literature has grown steadily over the past years. The modules to add to the DT are well motivated and sensible (especially the first 2). In addition, the modules seem rather easy to implement, which makes the paper easily applicable.
- I found the paper did a good job at studying comprehensively the use of DT-like methods in the offline RL literature. Several ablations are done to isolate the contribution of the different elements of RDT (Fig. 3 and Fig. 6). Moreover, there is a large variety of additional experiments in the appendix that naturally answered some of my questions as I read the main text, such as the influence of the value of $\zeta$ and $\beta$ (Fig. 10) or the variation of performance for different corruption scale values (Fig. 11).
- The proposed method delivers, as it gets robust performance in the small scale version of the D4RL dataset, improving over the baselines considered (Table 1), sometimes significantly.
- Ultimately, I think the paper asks an interesting question in the use sequence-based methods for robust offline learning. I believe this can generate interesting discussions at the conference around the usage of sequence-based models.
summary: This paper attacks the problem of learning a policy in an offline RL setting, in the specific case of having access to a lossy dataset, where data can be perturbed. While the literature has focused on algorithms leveraging the Q function to handle this problem, this paper shows that sequence-based models such as the Decision Transformer (DT) perform competitively, and are surprisingly robust compared to baselines when the size of the dataset is significantly reduced. The authors then propose 3 modifications to the DT algorithm that lead to algorithm even more robust than DT. They evaluate the performance of their proposed algorithm in the D4RL benchmark, and perform several ablations to get a better understanding of their method.
weaknesses: There are a couple of axes where I found weaknesses in this work, which I present below.
- Several of the methodological details are pushed to the appendix without being explicitly referred to in the main text. For instance, it is hinted in the main text that the choice of the $\zeta$ parameter (L301) or when the data iterative process starts (L292, “after sufficient training”) is important, but you have to go to the appendix to discover that this information is indeed provided. Adversarial corruption is mentioned quickly (L132) but without giving an intuition about how it would be used in this setting; the action diff perturbation (L426-428) could also have used some intuition (generic is OK) about it before being pushed to the appendix. Overall, I find the authors could make the main text more self-contained by giving the conclusions (maybe broadly) of their different experiments in the main text and referring to the appendix only for the in-depth analysis of the results. Methodological details can be pushed to the appendix, but as the reader goes through the paper, they should know that these details do, in fact, exist.
- While the three proposed additions to DT presented in Sec. 3 make sense, I found their concrete impact on performance difficult to evaluate. For instance, in the ablation of Fig. 6, the addition of the different modules to DT do not drastically change the performance — in fact, in several settings the confidence interval of RDT intersects with the DT(RP). I would be also curious to know how these ablated methods compare with unmodified DT in Fig. 6. Moreover, it seems that different modules contribute more in different settings (ED on action attack, GWL on reward attack …); I think some analysis about which environment should benefit from which module the most, if possible, would have helped framing the modules. The fact that the different ablations of Fig. 6 and 3 are evaluated on different environments also makes it difficult to get a clear sense of the general behavior of each module.
- I find the setting in which this method is expected to perform well to be relatively narrow, as it is mostly valid for datasets that are simultaneously small and corrupted. I do not think this makes this submission less valuable, but it raises questions that I would have liked to see addressed — for instance, what aspect exactly of the low data regime favors sequence models over Q-function based models? Is there any hope for sequence models to overcome Q-function models, even with bigger datasets (especially since transformer models have been shown to work so great in large data regimes)? On this point, while the problem is well-motivated, I found the conclusion that DT worked better than Q-function based models could have used more insights. It is explained at the end of Sec 3.1 (L187-189) but I found these two sentences did not bring much besides repeating the definition of a sequence model. 
- Only RDT has an uncertainty measure in Table 1, and there is no uncertainty measure in Table 2. Moreover, I did not find a description of the nature of the uncertainty measure in the main text.

In addition, I list below other minor weaknesses which did not impact my rating, but that I think could be addressed to improve the document:
- The text of the figures is small and makes reading the paper difficult on paper. (Especially true for Fig. 3 that was very hard to read). It would be useful to increase the size of the captions to make the figure more readable.
- L. 98: “Different from the MDP framework, DT [..]”. I would argue that DT still operates within the MDP framework, it is just a different algorithm to learn a policy.
- L246, Eq 4: you did not define the circle-cross operator. If this is supposed to be an elementwise product, I would argue that this is not the most appropriate symbol (\odot would, I think, be more appropriate). 
- L535: I found the term “exceptional” to be rather out of place.
- Typo: MoJoCo, while it should be MuJoCo (several places).
- Table 1: Lacking information about the meaning of +- in the RDT column. What exact uncertainty measure did you use? 
- In the experiments section, I would refer to the papers that introduced the experiments parameters setting (for example, you set the corruption scale to 1., but unless I am mistaken, we see in the appendix only that this value is set the same as previous works).

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. Figure 1 is not as intuitive as it could be; it might be more effective to directly display the performance decline under various attacks.
2. In the ablation study, the current approach is to compare RDT with DT(RP) that incorporates three different improvements. A more rational method would be to compare the performance of RDT with RDT w/o (ED, GWL and IDC).
soundness: 2
strengths: 1. Innovation: This approach introduces sequence modeling to address data corruption, moving beyond traditional frameworks based on temporal difference learning.
2. Enhanced Robustness: Embedding dropout, Gaussian-weighted learning, and iterative data correction significantly improve the algorithm’s performance under data corruption.
3. Extensive Validation: The method has been validated across various data corruption scenarios and with limited datasets, demonstrating its robustness.
4. High Adaptability: RDT performs exceptionally well in limited data settings, showcasing the potential of sequence modeling across diverse tasks and data conditions.
summary: This paper introduces a novel method, the Robust Decision Transformer (RDT), designed to address data corruption in offline reinforcement learning. The study finds that traditional offline reinforcement learning methods based on temporal difference learning perform poorly under data corruption, while the DT, a sequence modeling approach, demonstrates greater robustness in such scenarios. RDT enhances DT’s robustness by incorporating three simple yet effective techniques: embedding dropout, Gaussian-weighted learning, and iterative data correction. Experimental results show that RDT significantly outperforms traditional methods across various tasks in the presence of data corruption, especially in complex scenarios where both training and testing phases are affected by attacks.
weaknesses: No obvious weaknesses.

Meta-review: This paper proposes to use sequence models to tackle data corruption in offline RL with a limited dataset. It introduces Robust Decision Transformer with three componenets embedding dropout, Gaussian weighted learning, and iterative data correction. Extensive experiments on diverse datasets show the robustness and superior results of the proposed method against TD-based approaches.

Strengths:
- Clear presentation
- Novel use of decision transformers as a robust modeling method in offline RL
- Comprehensive empirical evaluation across various datasets, data scale and corruption scenarios.

Weaknesses:
- Limited scope to offline RL with a limited dataset and data corruption. The performance is comparable with TD-based methods at the full dataset scale.
- Challenges in applying state correction. This is left as a future direction.

Based on the novelty and robust improvement of the RDT method in a diverse setting, the reviewers reach unanimous agreement of accepting this submission.

Decision: Accept (Poster)