Towards Robust Offline Reinforcement Learning under Diverse Data Corruption
Authors: Rui Yang
Han Zhong
Jiawei Xu
Amy Zhang
Chongjie Zhang
Lei Han
Tong Zhang
Venue: ICLR 2024 spotlight (2024)
Keywords: Offline RL
robust RL
data corruption
training-time attack
Primary area: reinforcement learning
TL;DR: We investigate the robustness of offline RL algorithms under diverse data corruption, and introduce a novel algorithm RIQL that exhibits significantly improved robustness.

Abstract: Offline reinforcement learning (RL) presents a promising approach for learning reinforced policies from offline datasets without the need for costly or unsafe interactions with the environment. However, datasets collected by humans in real-world environments are often noisy and may even be maliciously corrupted, which can significantly degrade the performance of offline RL. In this work, we first investigate the performance of current offline RL algorithms under comprehensive data corruption, including states, actions, rewards, and dynamics. Our extensive experiments reveal that implicit Q-learning (IQL) demonstrates remarkable resilience to data corruption among various offline RL algorithms. Furthermore, we conduct both empirical and theoretical analyses to understand IQL's robust performance, identifying its supervised policy learning scheme as the key factor. Despite its relative robustness, IQL still suffers from heavy-tail targets of Q functions under dynamics corruption. To tackle this challenge, we draw inspiration from robust statistics to employ the Huber loss to handle the heavy-tailedness and utilize quantile estimators to balance penalization for corrupted data and learning stability. By incorporating these simple yet effective modifications into IQL, we propose a more robust offline RL approach named Robust IQL (RIQL). Extensive experiments demonstrate that RIQL exhibits highly robust performance when subjected to diverse data corruption scenarios.

Published as a conference paper at ICLR 2024
TOWARDS ROBUST OFFLINE
REINEORCEMENT
LEARNING UNDER DIVERSE DATA CORRUPTION
Rui Yang': : Han Z Jiawei 'HKUST, "Peking University, SCUHK (SZ), *University of Texas at Austin
Amy Zhang', "
Chongjie Zhang Lei Han Tong Zhang'
Washington University (St. Louis), Tencent Roboties X
ABSTRACT
Offline reinforcement learning (RL) presents a promising approach for learning reinforced policies from offline datasets without the need for costly unsafe in- teractions with the environment. However, datasets collected by humans in real- world environments are often noisy and may even be maliciously corrupted, which can significantly degrade the performance of offline RL. In this work, we first in- vestigate the performance of current offline RL algorithms under comprehensive data corruption, including states, actions, rewards, and dynamics. Our extensive experiments reveal that implicit Q-learning (IQL) demonstrates remarkable re- silience to data corruption among various offline RL algorithms. Furthermore, we conduct both empirical and theoretical analyses to understand IQL's robust performance, identifying its supervised policy learning scheme as the key factor. Despite its relative robustness, IQL still suffers from heavy-tail targets of Q func- draw inspiration tions under dynamics corruption. To tackle this challenge, heavy-tailednness and from robust statistics to employ the Huber loss to handle the heavy- utilize quantile estimators balance penalization for corrupted data and learn- ing stability. By incorporating these simple yet effective modifications into IQL, we propose more robust offline RL approach named Robust IQL. (RIQL). Exten- sive experiments demonstrate that RIQL exhibits highly robust performance when subjected diverse data corruption scenarios.
INTRODUCTION
Offline reinforcement learning (RL) is an increasingly prominent paradigm for learning decision- making from offline datasets (Levine et al. 2020 Fujimoto et al., 2019 Kumar et al.) 2019 2020). Its primary objective is to address the limitations of RL, which traditionally necessitates a large amount of online interactions with the environment. Despite its advantages, offline RL suffers from the challenges of distribution shift between the learned policy and the data distribution (Levine al. 2020). To address such issue, offline RL algorithms either enforce policy constraint (Wang et al.
2018 Fujimoto al. 2019 Fujimoto & Gu 2021 Kostrikov et al., 2021) penalize values of out-
of-distribution (OOD) actions (Kumar et al. 2020 An et al. 2022 Bai et al. [202]] Ghasemipour
al. 2022) to ensure the learned policy remains closely aligned with the training distribution.
Most previous studies on offline RL have primarily focused on simulation tasks, where data col- lection is relatively accurate. However, in real-world environments, data collected by humans or sensors can be subject to random noise or even malicious corruption. For example, during RLHF data collection, annotators may inadvertently or deliberately provide incorrect responses or assign higher rewards for harmful responses. This introduces challenges when applying offline RI. algo- rithms, as constraining the policy to the corrupted data distribution can lead to a significant decrease in performance
deviation from the original objectives. This limitation cability of offline RL to many real-world scenarios (Wang et al. [0224] Sun 2023). To the best of our knowledge, the problem of robustly learning an RL policy from corrupted offline datasets
restrict the appli-
mains relatively unexplored. It is important to note that this setting differs from many prior works
on robust offline RL (Shi & Chi 2022 Yang al. 2022a Blanchet al. 2023 which mainly
Equal Contribution. Correspondence to Rui Yang (yangrui... thw201 50gma com) and Han Zhong (han Chanzhong@stu.co .pt.ssttt............. edu .cn). Our code is available https//githhh.com//agguu//111111

---

Published as a conference paper at ICLR 2024
No Attack
Osservation Attack
Action Attack
Reward Attack
Dynamics Attack
Epiiode Bewwnd
3-
Scale 0.0
Scale 1,0
Scale 2.0
Scai 1.0
Scale 2.0
Scal 1,0
Scale 2.0
Scain 1.0
Scale 2.0
BC
EDAC
MSG
CQL
SQL
AWAC
IOL
Figure I: Performance of offline RL algorithms under random attacks on the Hopper task. Many of- fline RL algorithms are susceptible to different types of data corruption. Notably, IQL demonstrates superior resilience to 3 out of 4 types data corruption.
focus on testing-time robustness, i.e., learning from clean data and defending against attacks during evaluation. The most closely related works are Zhang al. (2022) Ye al. (2023b); Wu et al. (2022), which focus on the statistical robustness and stability certification of offline RL under data corruption, respectively. Different from these studies, we aim to develop an offline RL algorithm that is robust to diverse data corruption scenarios.
In this paper, we initially investigate the performance of various offline RL algorithms when con- fronted with data corruption of all four elements in the dataset: states, actions, rewards, and dynam- ics. As illustrated in Figure [ our observations reveal two findings: (1) current SOTA pessimism- based offline RL algorithms, such as EDAC (An al. (02I) and MSG (Chasemipour et al. 2022), experience a significant performance drop when subjected to data corruption. (2) Conversely, IQL (Kostrikov al. 2021) demonstrates greater robustness against diverse data corruption. Further analysis indicates that the supervised learning scheme the key to its resilience. This intriguing finding highlights the superiority of weighted imitation learning (Wang al. 2018 Peng et al.
2019 Kostrikov et al. 202T) for offline RI. under data corruption. From a theoretical perspective, prove that the corrupted dataset impacts IQL. by introducing imitation errors, which are the generalization errors of supervised learning objectives and typically diminish as the dataset size N increases; and (ii) corruption errors at a rate of ( ( </N), where the cumulative corruption level (see Assumption D This result underscores that, under the mild assumption that o(N), IQL exhibits robustness in the face of diverse data corruption scenarios.
types of errors: (i) intrinsic
Although IQL outperforms other algorithms under many types of data corruption, it still suffers sig- nificant performance degradation, especially when faced with dynamics corruption. To enhance its overall robustness, we propose three improvements: observation normalization, the Huber loss, and the quantile Q estimators. We note that the heavy-tailedness in the value targets under dynamics cor- ruption greatly impacts IQL's performance. To mitigate this issue, we draw inspiration from robust statisties and employ the Huber loss (Huber, 1992. 2004) to robustly learn value functions. More- over, we find the Clipped Double Q-learning (CDQ) trick (Fujimoto et al.) (2088. while effective in penalizing corrupted data, lacks stability under dynamics corruption. To balance penalizing cor- rupted data and enhancing learning stability, we introduce quantile Q estimators with an ensemble of Q functions. By integrating all of these modifications into IQL., we present Robust IQL (RIQL), a simple yet remarkably robust offline RL algorithm. Through extensive experiments, we demon- strate that RIQL consistently exhibits robust performance across various data corruption scenarios. We believe our study not only provides valuable insights for future robust offline RL research but also lays the groundwork for addressing data corruption in more realistic situations.
2 RELATED WORKS
Offline RL. To address the distribution shift problem, offline RL algorithms typically fall into policy constraints- based (Wang et al. 2018 Peng et al. 2019; Fujimoto & Gu 2021: Kostrikov
et al. 2021 [Sun et al. 2024 Xu et al. (023). and pessimism based approaches (Kumar al. 2020
An et al. 2021 121::00 Bai et al. 2022] Ghasemipour et al. 2022 Nikulin et al.] (023). Among these algorithms, ensemble-based approaches (An et al. 2021 Ghasemipour
al. 2022 demonstrate superior performance by estimating the lower-contidence bound (LCB) of Q values for OOD actions.
Additionally, weighted imitation-based algorithms (Kostrikov al. 2021 Xu al.) 2023 enjoy
better simplicity and stability compared to pessimism-basee approaches.
Robust Offfine RL. In the robust offline setting, a number of works focus on testing-time or the
distributional robustness (Zhou et al. 2021 Shi & Chil 2022 Yang et al. 2022a Panaganti al.
2

---

Published as a conference paper at ICLR 2024
2022 Blanchet et al. 2023). Regarding the training- time robustness of offline RL, Li et al. (2023)
investigate various reward attacks in offline RL. From theoretical perspective, Zhang et al. (0022 study offline RL under data contamination. Different from these works, propose algorithm that is both provable and practical under diverse data corruption on all elements. More comprehensive related works deferred to Appendix
PRELIMINARIES
Reinforcement Learning (RL). RI. is generally represented as a Markov Decision Process (MDP) defined by tuple (S, P,r, r.). The tuple consists of a state space S, an action space A. tran- sition function P. a reward function r, and a discount factor y E [0, 1). For simplicity, we assume that the reward function is deterministic and bounded |r(s,a) a)l I Rmax for any (s, a) S x A. The objective of an RL agent is to learn a policy T(als) that maximizes the expected cumula- tive return: max, (s00/00aa-==(((1 (1rl)))ss__PPPaPaPaa (~)(())),)a)) 1 ?"r(s,, az)], where Po is the distribution of initial states. The value functions are defined as V*(s) (s) Ex,P EE ('r(st, as)]so and Q* a) = r(s, a) + + P [V*(s')]
Offine RL and IQL. We focus on offline RL, which aims to learn near-optimal policy from static dataset DD. IQL (Kosrrkovv et al.
2021) employs expectile regression to learn the value functions:
(0) = (s, JDD
[(rss a) (V' - Qo(s,a))))
(v(V) E (s.a)> (Qo a) V=)) (x)x) |r 1(x < 03|2².
(2)
IQL further extracts the policy using weighted imitation learning with a hyperparameter 8: (x) _le.o) A(s, a)) log ma(a[s)], A(s,a) Qo(s, a) Vo(s).
Clean Data and Corrupted Data. We assume that the uncorrupted data (s. a, r, s') follows the distribution (s, a) ((..-),. r as) and s' ~ P(- s, s. Here u(--)) the behavior distribu- tion. Besides, we use (a 8) to denote the conditional distribution. For the corrupted dataset DD we assume that D = [(5:, ar, Ti, s(), consists of N samples, where we allow (si, a;) not to be sampled from the behavior distribution )( r; F(si, a;), and sf ~ P(- az). Here 7 and f are the corrupted reward function and transition dynamics for the i-th data, respectively. For case of presentation, denote the empirical state distribution and empirical state-action distribution as a) respectively, while the conditional distribution is represented by (als))) i.c.
dp (s) and d >. 1(5,=8,0,=u)
(a(aas) = max [I. 1(s==s)) Also, we introduce the following notations:
[TV] a) r(s,a) + Be-e((ea)(((s)) for any () E S x A and V : S + 10, Rmax/ /1 7)]-
[TV](s, f(x, a) + P( [V()), (4)
(V)))
OFFLINE RL UNDER DIVERSE DATA CORRUPTION
In this section, first compare various offline RL algorithms in the context of data corruption. Subsequently, drawing on empirical observations, we delve into a theoretical understanding of the provable robustness of weighted imitation learning methods under diverse data corruption.
4.1 EMPIRICAL OBSERVATION
Initially, we develop random attacks on four elements: states, actions, rewards, and next-states (or dynamics). We sample 30% of the transitions from the dataset and modify them by incorporating random noise at a scale of standard deviation. Details about data corruption are provided in Ap- pendix D.I] As illustrated in Figure D1 current SOTA offline RL algorithms, such EDAC and MSG, are vulnerable to various types of data corruption, with particular susceptibility to observa- tion and dynamics attacks. In contrast, IQI. exhibits enhanced resilience to observation, action, and reward attacks, although it still struggles with dynamic attack. Remarkably, Behavior Cloning (BC) undergoes a mere 0.4% average performance degradation under random corruption. However, BC sacrifices performance without considering future returns. Trading off between performance and robustness, weighted imitation learning, to which IQL belongs, is thus more favorable. In Appendix E.1 perform ablation studies IQL and uncover evidence indicating that the supervised pol- icy learning scheme contributes more to IQL's robustness than other techniques, such as expectile regression. This finding motivates to theoretically understand such learning scheme.
3

---

Published as a conference paper at ICLR 2024
4.2 THEORETICAL ANALYSIS
We define the cumulative corruption level as follows.
Assumption (Cumulative Corruption). Lel Ç - 1 (2C; + log (i) denote the cumulative cor- ruption level, where Si and C are defined
TD((a s;) Tu(a [ si)
[[TV] a) [TV] a)l Gts max
S Co Va eaek A.
T ( si) T si) Here || 1lso means taking supremum over V S [0, Rmax/
7
We remark that { Assumption Cacnmppptoonnuaana quantifies the corruption level of reward functions and tran- sition dynamics, and { represents the corruption level of observations and actions. Although {GH could be relatively large, our final results only have logarithmic dependency on them. In particular, if there is no observation and action corruptions, log ç = log 1 = 0 for any - 1, N.
To facilitate our theoretical analysis, we assume that we have obtained the optimal value function (I) and (2. This has been demonstrated in Kostrikov et al. (202T) under the uncor- rupted setting. We provide justification for this under the corrupted setting the discussion part of
V by solving
Appendix C.2 Then the policy update rule in (3) takes the following form:
TIQL = arg max (s))) (3
(expp [TV* - V*] a)) log x(a s)]
(5)
arg min E. Ex (TE() 88) (- ())],
where (a 8) (a s) exp(3 [TV - (s, a)) is the policy that IQL. imitates. However, the learner only has access to the corrupted dataset D. With D, IQL finds the following policy
iio arg min Bg- (f 1s,
(- [s)]]
where Te s) x (a s) exp(3 (řv- v*](s,
To bound the value difference between TIQU. and नiq.. result relies on the coverage assumption.
Assumption 2 (Coverage). There exists an M > 0 satisfying max (a.() d' any (s, a) E S x A.
} S M for ((5,0) do(s,a)
Assumption requires that the corrupted dataset and the clean data μ have good coverage of fgg and TE respectively, similar to the partial coverage condition in (Jin et al. 2021 Rashidinejad al.) 20021. The following theorem shows that IQL robust to corrupted data. For simplicity, we choose in (5) and ( Our analysis is ready to be extended to the case where 3 takes on any constant.
Theorem (Robustness of IQL). Fix B ==
Under Assumptions) and 2
it holds that
VIIQL VRIQL < 22M Rmax
2R 2Rmax
MC N
(1 (1- [Ve1 + Ve2) + (1 7)
where = Eggu [T[(T(- 8), TIQu( 8)]] and = BEND (TT s), #Iqu( [s)]]
See Appendix C.I for detailed proof. The é and e2 are standard imitation errors under clean data and corrupted data, respectively. These imitation errors can also be regarded as the generalization error of supervised learning objectives (5) and (6). As N increases, this type of error typically di- minishes (Janner et al. [2119] Xu et al. 2020). Besides, the corruption error term decays to zero with the mild assumption that Ç = o(N). Combining these two facts, we can conclude that IQL. exhibits robustness in the face of diverse data corruption scenarios. Furthermore, we draw a com- parison with a prior theoretical work (Zhang et al. 2022) in robust offline RL (refer to Remark 7 Appendix C.I
5 ROBUST IQL FOR DIVERSE DATA CORRUPTION
Our theoretical analysis suggests that the adoption of weighted imitation learning inherently offers robustness under data corruption. However, empirical results indicate that IQL still remains suscep- tible to dynamics attack. To enhance its resilience against all forms data corruption, we introduce three improvements below.
4

---

Published as a conference paper at ICLR 2024
Papper
Water
276
140
100
+100
3040
Alter Maher
102
162
2000
(a)
(b)
(c)
(d)
Figure 3: (a) Heavy tailedness of Q target distributions on the Hopper task. The target values are normalized by subtracting the mean. (b) Kurtosis values of Q target distribution Hopper and Walker tasks. The gray dashed line represents the kurtosis value of Gaussian distribution. (c) Average final returns of IQL with different datasets. (d) Comparison of IQL w/ and w/o Huber loss.
5.1 OBSERVATION NORMALIZATION
In the context of offline RI., previous studies have suggested that normalization does not yield significant improvements (Fujimoto find that normalization is highly ad-
& Gu 202T. However, vantageous in data corruption settings. Normalization is likely to help ensure that the algorithm's 's performance is not unduly affected
Average Relum
10
by features with large-scale values. As depicted in Figure 2 it is evident that normalization enhances the performance of IQL across diverse attacks. More details regarding the ablation experiments on normalization deferred Appendix E.2 Given that s and s' in (s, r, s') can be corrupted independently, the normalization is conducted by calculating the mean 1lo and variance o, of all states and next-states in 'D. Based on Mo and do. the states and next- states are normalized as 8;
IQL
(norm)
observation action atlack
reward attack
dynamics
attack
attack
Figure 2: Average perfor-
mance of IQL w/ and w/o nor- malization over four datasets.
(#1 (#i-Ho)
.8 (s( (d(- where t
do
do
2N ΣN Ex=1(s, (s; + s)) and o² 2N Dmsl mo)² + (s) -p))
5.2
HUBER LOss FOR ROBUST VALUE FUNCTION LEARNING
noted Section IQL, along with other offline RL algorithms, is susceptible to dynamics attack. The question arises: why are dynamics attacks challenging to handle? While numerous in- fluential factors may contribute, we have identified an intriguing phenomenon: dynamics corruption introduces heavy-tailed targets for learning the Q function, which significantly impacts the perfor- the Q target, i.e., r(s, a) + VV,
mance of IQL. In Figure 3 (a), we visualize the distribution which is essentially influenced by the rewards and the transition dynamics. Additionally,
col- lected 2048 samples to calculate statistical values after training IQL for 3 x 10 steps. As depicted in Figure (a), the reward attack does significantly impact the target distribution, whereas the dy- namics attack markedly reshapes the target distribution into a heavy-tailed one. In Figure 3 (b), we
MN -X:-x // which measures the heavy-tailedness relative employ the metric Kurtosis K S (X.
x X
to normal distribution (Mardial 1970: Garg al. 2021). The results demonstrate that dynamics
corruption significantly increases the Kurtosis value compared to the clean dataset and the reward- attacked datasets. Correspondingly, shown in Figure (c), higher degree of heavy-tailedness generally correlates with significantly lower performance. This aligns with our theory, as the heavy- tailed issue leads to a substantial increase in { making the upper bound in Theorem 3 very loose. This observation underscores the challenges posed by dynamics corruption and the need for robust strategies to mitigate its impact.
The above finding motivates us to utilize Huber loss (Huber 1973, employed to mitigate the issue of heavy-tailedness, (Sun et al. 2020 Roy al. (02T). specific, given Vr,, replace the quadratic loss in (I) by Huber loss 14 (-) and obtain the following objective
1992), which is commonly
if | S 6
Lo = EbannNND [1 + +V'(') - Q(s,a))) where (x)
26
(7)
|ar|
if
r > 8
2
5

---

Published as a conference paper at ICLR 2024
ey
GL
2.5
K2
19000
2:5
1900
Arsrag uslu
+180
Aoorago
KGL jingl QD
0540 (5L0
wilon
073015
1RD0 TIE 2000 SHID 2010 1030
(a)
(b)
(d)
Figure 4: Comparison of IQL. and IQL. with single Q function the (a) Hopper and (b) Walker tasks. (c) Average returns and (d) average values in the Hopper task under dynamics attack. IQL. can be unstable due to the minimum operator.
Here trades off the resilience to outliers from & loss and the rapid convergence of &g loss. The & loss result in geometric median estimator that is robust outliers and heavy-tailed distribu- tions. As illustrated in Figure [ (d), upon integrating Huber loss as in (7). we observe a significant enhancement the robustness against dynamic attack. Next, we provide a theoretical justification for the usage of Huber loss. We first adopt the following heavy-tailed assumption.
Assumption 4. Fix the value function parameter th, for any (s,a, r')) ~ D, it holds that r + VV r(s, a) + - P(-Is,) VVaaaaVVVVV (5)) Es,o,s', where Es,as' is the heavy-tailed noise satisfying ElEs,a,8' s, a, s'] and E|,,,, 1 1'-y] 0o for some
Assumption 4 only assumes the existence of the (1 w)-th moment of the noise Es,a,s' which is strictly weaker than the standard boundedness or sub-Gaussian assumption. This heavy-tailed assumption is widely used in heavy-tailed bandits (Bubeck et al. 2013 Shao et al. 2018 and heavy- tailed RL (Zhuang & Sui [222] Huang et al. (023)). Notably, the second moment of Sa,a,a' may not exist when v < 1, leading to the square loss inapplicable in the heavy-tailed setting. In contrast, by adopting the Huber regression, we obtain the following theoretical guarantee:
Lemma 5. Under Assumption 7 if we solve the Huber regression problem in and obtain the estimated Q-function Q, then we have
[[}(s,a) rs, -BE. (Vo('))ll S N min(v/(L+0).12)))
where -|/ means taking supremum over (s, E S x A and N is the size of dataset D. Further- more, if N + oc, we have Q(s. a) + r(s, + (s')) for any (s. E S X A.
The detailed proof is deferred to Appendix C.2 Thus. by utilizing the Huber loss, we can recover the nearly unbiased value function even in the presence of heavy-tailed target distribution.
5.3 PENALIZING CORRUPTED DATA VIA IN-DATASET UNCERTAINTY
We have noted key technique within IQL's implementation the Clipped Double Q-learning (CDQ) (Fujimoto et al. 2018). CDQ learns two Q functions and utilizes the minimum one for both value and policy updating. To verify its effectiveness, we set another variant of IQL without the CDQ trick, namely "sinle Q)", which only learns single Q function for IQL updates. As shown in Figure (a) and (b), "IQglsingle Q)" notably underperforms IQL on most corruption settings except the dynamics corruption. The intrigue lies within the minimum operator's 's correlation to the lower confidence bound (LCB) (An al., 202T). Essentialll. it penalizes corrupted data, corrupted data typically exhibits greater uncertainty, consequently reducing the influence of these data points on the policy. Further results Figure (c) and (d) explain why CDQ decreases the performance under dynamics corruption. Since the dynamics corruption introduces heeav-tailednnes, in the Q target as explained in Section 5.2 the minimum operator would persistently reduce the value esti- mation, leading negative value exploding. Instead, "sQglsingle Q)" learns average Q value, as opposed to the minimum operator, resulting in better performance under dynamics corruption.
The above observation inspires us to the CDQ trick safely by extending it to quantile Q timators using an ensemble of Q functions [Qu,]K The a-quantile value Qo is defined as Pr(Q Qa) a, where E [0, 1]. A detailed calculation of the quantiles is deferred to the

---

Published as a conference paper at ICLR 2024
2 dean
daat aBacood
0- cein
0- daa1
6:001
X stskke
a- arwakd
- v33
0.35
61011
0-05
6:0.5
u:0.59
0.0
Poudf
0.15
0.10
Eooda
Ezocha
(a)
(b)
(c)
Figure 5: (a) Quantiles normal distribution. In-dataset penalty for attacked data and clean data
across (b) different number of K and (c) different quantile value
Appendix D.3
The quantiles of a normal distribution are depicted in Figure (a). When a corre-
sponds to an integer index and the Q value follows Gaussian distribution with mean m(s, a) and
variance o (s, a), it is well-known (Blom 1958 Royston 1982) that the q-quantile is correlated to
the LCB estimation of variable: E [Qa a)] m(s, a) &
csss a).
2-2x0.37+11
Hence, we can control the degree penalty by controlling a and the ensemble size K. In Figure 5 (b) and (c), we illustrate the penalty, defined m(s. a) (s, a), for both clean and corrupted parts of the dataset. It becomes evident that (1) the attacked portion of data can incur a heavier penalty compared to the clean part, primarily due to the larger uncertainty and higher estimation variance o3 a) associated with these data points, and (2) the degree of penalty can be adjusted flexibly be larger with a larger K or a smaller a. Notably, when K 2 and 0, the quantile Q estimator recovers the CDQ trick in IQL. We will discuss the impact of the quantile Q estimator on the equations of RIQL in Section 5.4
5.4 RIQL ALGORITHM
We summarize the overall RIQL algorithm, whose pseudocode is provided in Algorithm ! (Ap- pendix[B], First, RIQL normalizes the states and next states according to Section The ensemble Q value functions (Qu, (s, K learned by solving the Huber regression
(0,) E B(s...:))= + (V' Qe, (s,a))].
where (+) is the Huber loss defined 7). The value function (s((s) is learned minimize
(v [E (Q. (s)), (x) = 1(x < 0)]a².
Here (s, a) is defined by the a-quantile value among (Qa, (s, a) )K The policy is learned to maximize the a-quantile advantage-weighte imitation learning objective:
L (0) (s,,)~ (exp(3AA a)) log (a(s)))
(10)
where the advantage (s,s, a) = Q. Vo(s), Intuitively, the quantile Q estimator serves dual purpose: (I) underestimating the Q values with high uncertainty in (9), and (2) downweighting these samples to reduce their impact the policy in [O).. An essential difference between our quantile Q estimator and those in prior works (An et al. 202T Bai et al. 2022 Ghasemipour et al, 2022) is that penalize in-dataset data with high uncertainty due to the presence of corrupted
data, whereas conventional offline RI. leverages uncertainty to penalize out-of-distributiin actions.
EXPERIMENTS
In this section, we empirically evaluate the performance of RIQL across diverse data corruption scenarios, including both random and adversarial attacks states, actions, rewards, and next-states.
Experimental Setup. Two parameters, corruption rate E [0. 1] and corruption scale e are used to control the cumulative corruption level. Random corruption is applied by adding random noise to the attacked elements c portion the datasets. In contrast, adversarial corruption is in- troduced through Projected Gradient Descent attack (Madry et al. 2017 Zhang et al., 2020) with
7

---

Published as a conference paper at ICLR 2024
Table I: Average normalized performance under random data corruption.
Environment
Attack Element
EDAC
MSG
SQL
IQL
RIQL (ours)
observation
33.41188
2.1=0.5
-02212.2
9.027.5
4.141.4
21.41:19
27.3122.
Halfchectah
action
36.2+0.3
47,4113
52,010.9
19.9=21.3
42.940.4
42.24119
42,910066
35.810.99
38.640.3
17.5=16.4
32.6=19.6
41.710.8
42.310.4
43.6.10.6
dynamics
35.8+-0.9
1.5-0.2
1,7:0.4
29.244.00
35.5+10.4
36.74:1.8
43.1
observation
9.6:13.9
0.240..3
00410.11
19.421.6
0.621.0
27.245.1
28.4117.7
Walker2d
action
18.122.1
83.24:99
25.3=10.6
62.747.2
76.04-42
71.347.8
84.6+23.3
reward
16.027.4
3.3
18.499.5
69.4:7.4
33.8
65.3:18.4
83.2:12.66
dynamics
16.01:7.4
0.111.00
7.43377
15.3:12.2
17.74.7.3
78.2.1
observation
21.3
1.00..5
6.955.0
42.847.0
5,241.9
12.0
62.41:8
Hopper
action
22.8++7.0
100.8:0.55
37.6+6..5
69.8+44.5
73.4+7.3
76.3+15.4
90.6:t55.6
19,543.4
2.6=0.7
24.9114.3
70.3:3.9
52.3+117
18.7
84.8=13.1
dynamics
19,543344
0.8=0.0
12.44-199
0.8
24.34:5.6
1.3
51.5.18.1
Average score
23.7
23.5
17.0
35.5
33.8
43.6
60.0
Average degrudation percentage +
0.4%
68.5%
61.5%
42.3%
45.0%
31.2%
17.0%
the pretrained value functions. A unique case is the adversarial reward corruption, which directly multiplies -c to the original rewards instead of performing gradient optimization. We utilize the dataset from (Fu et al.] 2020). which better represents real scenarios because it collected during the training of a SAC agent. The corruption rate is set to c = 0.3 and the corrup- tion scale is for the main results. More details about all forms of data corruption used in experiments are provided in Appendix(D.I We compare RIQL with SOTA offline RL algorithms, in- cluding ensemble- 2022). as well as ensemble-free free baselines like IQL (Kostrikov et al. 2021). CQL (Kumar et al., 2020 and SQL (Xu al. 2023). To evaluate the performance, we test each trained agent in a clean environment and average their performance over four random seeds. Implementation details and additional empirical results are provided Appendix aDa Appendix respectively.
algorithms such as EDAC (An al. (021) and MSG (Ghasemipour al,
6.1 EVALUATION UNDER RANDOM AND ADVERSARIAL CORRUPTION
Random Corruption We begin by evaluating various algorithms under random data corruption. [ the average normalized performance (calculated as 100 x scare random score
random of different algorithms. Overall, RIQL demonstrates superior performance compared to other base- lines, achieving an average score improvement of 37.6% over IQL.. In out of 12 settings, RIQL achieves the highest results, particularly excelling in the dynamics corruption settings, where it outperforms other methods by a large margin. Among the baselines, periences the least average degradation percentage compared its score the clean dataset. This can be attributed to its supervised learning scheme, which avoids incorporating future information but also results in relatively low score. Besides, offline RL methods like EDAC and MSG exhibit performance degradation exceeding 60%, highlighting their unreliability under data corruption.
expert
noteworthy that BC ex-
Adversarial Corruption To further investigate the robustness of RIQL under a more challeng- ing corruption setting, we consider an adversarial corruption setting. As shown in Table 2 RIQL consistently surpasses other baselines by significant margin. Notably, RIQL improves the average score by almost 70% over IQI.. When compared to the results of random corruption, every algorithm experiences a larger performance drop. All baselines, except for BC, undergo a performance drop of more than or close half. In contrast, RIQL's performance only diminishes by 22% compared to its score on the clean data. This confirms that adversarial attacks present a more challenging setting and further highlights the effectiveness of RIQL against various data corruption scenarios.
6.2 EVALUATION WITH VARYING CORRUPTION RATES
In the above experiments, we employ a constant corruption rate of 0.3. In this part, we assess RIQL and other baselines under varying corruption rates ranging from 0.0 to 0.5. The results are depicted in Figure 6 RIQL exhibits the most robust performance compared to other baselines, particularly under reward and dynamics attacks, due to its effective management of heavy-tailed targets using the Huber loss. Notably, EDAC and MSG 0.1 under observation, reward, and dynamics attacks. Furthermore, among the attacks on the four elements, the observation attack presents the greatest challenge for RIQL. This can be attributed to
highly sensitive to a minimal corruption rate of
8

---

Published as a conference paper at ICLR 2024
Table 2: Average normalized performance under adversarial data corruption.
Environment
Attack Element
BC
EDAC
MSG
CQL
SQL
IQL
RIQL (ours)
observation
34,3+:1.5
1.120.3
1.1=02
5.0=11.6
8.34099
32.6+2.77
35.744.2
action
14.0+11
32.740.7
37.310.7
-2,3+122
32.7=100
27.540.3
31.7+177
Halfcheetah
reward
35.8+09
-1,740.3
42.6+004
44.1+0.8
dynamics
35.8+0.9
-1.340.1
-1.50000
-1.6+0.0
10.4=2.6
26.74:0.7
38.8+2.1
observation
12.7239
-0.010.1
2922.7
618E74
18219
37.7113.0
70.013.3
5.40.44
41.9=24.0
5.4=0.9
27.01:7.5
31.3=288
27.51:0.6
66.1114.6
Walker2d
reward
16.047.4
57,3=33,2
9.624.99
67.026.11
78.1=2.0
73,5=44.85
85.0=:1.5
dynamics
16.0:7.4
4.3:00.9
0.1:0.2
3.9:1:44
2.74:19
-0.14..11
60.61:21.8
observation
21.6:17.1
7..0
32.8166.4
5..8.17.6
Hopper
action
15.5-42.22
25.7433.8
23.0+-2.1
32.247..6
30.0=0.4
37.9444.8
63.6:77.3
reward
19.5::3.4
21.2::199
22.6:1..88
49.6:112.3
57.9=
57.31.9.7
65.8:1 9.8
dynamics
19.5+-3.4
0.6+0.00
0.6-0.0
0.6++0.0
18.94-22.6
1,3411
65,74-21.1
Average score
20.5
21.7
13.7
26.6
25.8
33.1
56.2
Average degradation percentage
13.4%
71.2%
69.9%
66.8%
57.5%
46.0%
22.0%
60
80
00
8,3
Comustion Rate
Comuation Rate
Comuption Rate
Corruption Rane
Figure 6: Varying the corruption rate of random attacks on the Walker2d task.
the additional distribution shift introduced by the observation corruption, highlighting the need for further enhancements.
6.3 ABLATIONS
We assess the individual contributions of RIQL's components un- der the mixed attack scenario, where all four elements are cor- rupted independently, each with corruption rate of 0.2 and a corruption range 1.0. We consider the following RIQL variants: RIQL without observation normalization (RIQL w/o norm), RIQL without the quantile Q estimator (RIQL w/o quan- tile), and RIQI without the Huber loss (RIQL w/o Huber). The average results for the Walker and Hopper tasks on the "medium- replay-v2" dataset dicate that each component contributes to RIQL's performance, with the Huber loss standing out the most impactful factor due to its effectiveness in handling the dynamics corruption. The best performance is achieved when combining all three components into RIQL.
Avetag Retum
202
2.0
2024
presented in Figure 7 These results in-
Figure 7: Ablation RIQL un- der mixed attack.
7 CONCLUSION
In this paper, we investigate the robustness of offline RL algorithms in the presence of diverse data corruption, including states, actions, rewards, and dynamics. Our empirical observations reveal that current offline RI. algorithms are notably vulnerable to different forms of data corruption, particu- larly dynamics corruption, posing challenges for their application real-world scenarios. To tackle this issue, we introduce Robust IQL (RIQL), offline RL algorithm that incorporates three sim- ple yet impactful enhancements: observation normalization, Huber loss, and quantile Q estimators. Through empirical evaluations conducted under both random and adversarial attacks, demon- strate RIQL's exceptional robustness against various types of data corruption. We hope that our work will inspire further research aimed at addressing data corruption in more realistic scenarios.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
The work done when Rui Yang and Tong Zhang were at HKUST. The authors would like thank Chenlu Ye for insightful discussions and the anonymous reviewers for their valuable feedback.
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective offline reinforcement learning. In International Conference on Machine Learning, pp. 104-114, PMLR, 2020.
Gaon An, Seungyong Moon, Jang- Hyng-Hyun Kim, and Hyun Oh Song. Uncertainty- based offline re- inforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34:7436-7447, 2021.
Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhihong Deng, Animesh Garg. Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. arXiv preprint arXiv: 1566, 2022.
Jose Blanchet, Miao Lu, Tong Zhang, and Han Zhong. Double pessimism is provably efficient for distributionally robust offline reinforcement learning: Generic algorithm and robust partial coverage. arXiv preprint arXiv:2305.09653 2023.
Gunnar Blom. Statistical estimates and transformed beta-variables. PhD thesis, Almqvist & Wik- sell, 1958.
Sébastien Bubeck, Nicolo Cesa-Bianchi, and Gábor Lugosi. Bandits with heavy tail. IEEE Trans- actions on Information Theory, 59(11):7711-7717 2013.
Johan Samir Obando Ceron and Pablo Samuel Castro. Revisiting rainbow: Promoting more insight- ful and inclusive deep reinforcement learning research. International Conference on Machine Learming. pp. 1373-1388 PMLR, 2021.
Yevgen Chebotar, Quan Vuong, Alex Irpan, Karol Hausman, Fei Xia, Yao Lu, Aviral Kumar, Tianhe Yu, Alexander Herzog, Karl Pertsch, et al. Q-transformer: Scalable offline reinforcement learning via autoregressive q-functions. arXiv preprint arXiv:2309.10152 2023.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 14:15084-15097, 2021.
Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning, arXiv preprint arXiv:2202.024446 2022.
Qiwen Cui and Simon Du. When are offline two-player zero-sum markov games solvable? Ad- vances in Neural Information P Processing Systems, 15:25779-257911 2022.
Will Dabney, Mark Rowland, Mare Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 32, 2018
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline rl via supervised learnin?? arXiv preprint arXiv:2112.107511 2021.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20155, 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference machine learning, pp. 1587-1596. PMLR, 2018.
10

---

Published as a conference paper at ICLR 2024
Scott Fujimoto, David Meger, and Doina Precup. Off Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 2052-2061 PMLR, 2019.
Saurabh Garg, Joshua Zhanson, Emilio Parisotto, Adarsh Prasad, Zico Kolter, Zachary Lipton, Sivaraman Balakrishnan, Ruslan Salakhutdinov, and Pradeep Ravikumar. proximal policy optimization's s heavy-tailed gradients. In International Conference on Machine Learning, pp- 3610-3619. PMLR, 2021.
Kamyar Ghasemipour, Shixiang Shane Gu, and Ofir Nachum. Why so pessimistic? estimating uncertainties for offline through ensembles, and why their independence matters. Advances in Neural Information Processing Systems, 00118267-12888 2022.
Philippe -attsuch Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304. 10573, 2023.
Chin Pang Ho, Marek Petrik, and Wolfram Wiesemann. Fast bellman updates for robust mdps. In International Conference on Machine Learning, pp. 1979-1988. PMLR, 2018.
Jiachen Hu, Han Zhong, Chi Jin, and Liwei Wang. Provable sim-to-real transfer in continuous domain with partial observations. arXiv preprint arXiv:2210.15598. 2022.
Jiatai Huang, Yan Dai, and Longbo Huang. Adaptive cestoo-bothhworldd algorithm for heavy-tailed multi-armed bandits. In International Conference on Machine Learning, pp. 9173-9200, PMLR, 2022.
Jiayi Huang, Han Zhong, Liwei Wang, and Lin F Yang. Tackling heavy-tailed rewards in rein- forcement learning with function approximation: Minimax optimal and instance-dependent regret bounds. arXiv preprint arXiv:2306.068366 2023.
Peter Huber. Robust regression: asymptotics, conjectures and monte carlo. The annals of statisties, pp. 799-8299111 1973.
Peter Huber. Robust estimation of a location parameter. In Breakthroughs in statistics: Methodol- ogy and distribution, pp. 492-518. Springer, 1992.
Peter Huber. Robust statistics, volume 523. John Wiley & Sons, 2004.
Hisham Husain, Kamil Ciosek, and Ryota Tomioka. Regularized policies are reward robust. In International Conference on Artificial Intelligence and Statistics, pp. 64-72. PMLR, 2021.
Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2): 257-280, 2005.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. Advances in neural information processing systems, 32, 2019.
Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for ficxible behavior synthesis. In International Conference on Machine Learning, pp. 9902-9915. PMLR, 2022.
Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline rl? In International Conference Machine Learning, pp. 5084-5096. PMLR, 2021.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning, pp. 267-274, 2002.
Minhyun Kang and Gi-Soo Kim. Hcavy-tailed linear bandit with huber regression. In Uncertainty in Artificial Intelligence, pp. 1027-1036. PMLR. 2023,
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. arXiv preprint arXiv:2 10.06/69, 2021.

---

Published as a conference paper at ICLR 2024
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- policy q- learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 19111799111,, 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
Anqi Li, Dipendra Misra, Andrey Kolobov, and Ching-An Cheng. Survival instinct in offline rein- forcement learning. arXiv preprint arXiv:2306.0322 2023.
Gen Li, Laixi Shi, Yuxin Chen, Yuejie Chi, and Yuting Wei. Settling the sample complexity of model-based offline reinforcement learning. arXiv preprint arXiv:2204.05225, 2022.
Lanqing Li, Rui Yang, and Dijun Luo. Focal: Efficient fully- meta-reiiffrreement learning via
distance metric learning and behavior regularization. arXiv preprint arXhv:2010.01112, 2020.
Liu Liu, Ziyang Tang, Lanqing Li, and Dijun Luo. Robust imitation learning from corrupted demon- strations. arXiv preprint arXiv:2201. 125944 2022.
Antonio Loquercio, Mattia Segu, and Davide Scaramuzza. A general framework for uncertainty estimation in deep learning. IEEE Robotics and Automation Letters, 5(2):3153-3160, 2020.
Thodoris Lykouris, Max Simchowitz, Alex Slivkins, and Wen Sun. Corrutiion-obuutt exploration in episodic reinforcement learning. In Conference Learning Theory, pp. 3242 3245. PMLR, 2021.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint /rXiv:/706.00 06083, 2017.
Daniel J Mankowitz, Nir Levine, Rae Jeong, Yuanyuan Shi, Jackie Kay, Abbas Abdolmaleki, Jost Tobias Springenberg, Timothy Mann, Todd Hester, and Martin Riedmiller. Robust re- inforcement learning for continuous control with model misspecification. arXiv preprint 16X.v:1906.07516. 2019.
Kanti V Mardia. Measures of multivariate skewness and kurtosis with applications. Biometrika, 57 (3)5519-530, 1970.
Janosch Moos, Kay Hansel, Hany Abdulsamad, Svenja Stark, Debora Clever, and Jan Peters. Robust reinforcement learning: A review of foundations and recent advances. Machine Learning and Knowledge Extraction, 4(1):276-315, 2022.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online rein- forcement learning with offline datasets. arXiv preprint MXXiv:2006609352 2020.
Alexander Nikulin, Vladislav Kurenkov, Denis Tarasov, and Sergey Kolesnikov. Anti-exploration by random network distillation. arXiv preprint arXiv:2301. 2023.
Arnab Nilim and Laurent Ghaoui. Robustness in markov decision problems with uncertain transition matrices. Advances in neural information processing systems, 16, 2003.
Kishan Panaganti, Zaiyan Xu, Dilcep Kalathil, and Mohammad Ghavamzadeh. Robust reinforce- ment learning using offline data. Advances in neural information processing systems, 35:32211- 32224, 2022.
Andrew Patterson, Victor Liao, and Martha White. Robust losses for learning value functions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):6157-6167, 2022.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off- off-policy reinforcement learning. arXiv preprint 1rXi: 2019 2019.
12

---

Published as a conference paper at ICLR 2024
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. In International Conference on Machine Learning, Pp. 2817-2826. PMLR, 2017.
Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline rein- forcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems, 34:11702-11716, 2021.
Abhishek Roy, Krishnakumar Balasubramanian, and Murat A Erdogdu. On empirical risk minimiza- tion with dependent and heavy-tailed data. Advances in Neural Information Processing Systems, 34:8913-899 2021.
JP Royston. Algorithm as 177: Expected normal order statistics (exact and approximate). Journal of the royal statistical society. Series C (Applied statistics), 31()):61-166,, 1982.
Fumihiro Sasaki and Ryota Yamashina. Behavioral cloning from noisy demonstrations. In Interna- fional Conference on Learning Representations, 2020.
Han Shao, Xiaotian Yu, Irwin King, and Michael R Lyu. Almost optimal algorithms for linear stochastic bandits with heavy-tailed payoffs. Advances in Neural Information Processing Systems, 31, 2018.
Laixi Shi and Yuejie Chi. Distributionally robust model-based offline reinforcement learning with near-optimal sample complexity. arXiv preprint arXiv:2208.05766 2022.
Laixi Shi, Gen Li, Yuting Wei, Yuxin Chen, and Yuejic Chi. Pessimistic q-learning for offline reinforcement learning: Towards optimal sample complexity. arXiv preprint arXiv:2202.1 12022222..899. 13890, 2022.
Hao Sun. Reinforcement learning in the of llss: What is essential? what is needed? rl perspective on rlhf, prompting, and beyond. arXiv preprint arXiv:2310.06147, 2023.
Hao Sun. Lei Han, Rui Yang, Xiaoteng Ma, Jian Guo, and Bolei Zhou. Exploit reward shifting in value-based deep-rl: Optimistic curiosity-basede exploration and conservative exploitation via linear reward shaping. Advances in Neural Information Processing Systems, 35:37719-37734, 2022a.
Hao Sun, Boris van Breugel, Jonathan Crabbe, Nabeel Seedat, and Mihaela van der Schaar. Daux: density-based approach for uncertainty explanations. arXiv preprint 1xXi::207.001611 2022b.
Hao Sun, Alihan Hüyüik, Daniel Jarrett, and Mihaela van der Schaar. Accountability in offline reinforcement learning: Explaining decisions with a corpus of examples. Advances in Neural Information Processing Systems, 36, 2024.
Qiang Sun, Wen-Xin Zhou, and Jianqing Fan. Adaptive huber regression. Journal of the American Statistical Association, 115(529):254 265, 2020.
Voot Tangkaratt, Nontawat Charoenphakdee, and Masashi Sugiyama. Robust imitation learning from noisy demonstrations. arXiv preprint arXiv:2010. 10181, 2020a.
Voot Tangkaratt, Bo Han, Mohammad Emtiyaz Khan, and Masashi Sugiyama. Variational imitation learning with diverse-quality demonstrations. In Proceedings the 37th International Conference on Machine Learning, pp.
2020b.
Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. CORL: Research-oriented deep offline reinforcement learning library. In 3rd Offline RL Work- shop: Offline RL as a "Launchpad", 2022. URL
net httpfformmmiim
SyAS49bBcv
Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and appli- cations in continuous control. In International Conference Machine Learning, pp. 6215-6224. PMLR, 2019.
Masatoshi
Uehara and Wen Sun. Pessimistic model-based offline reinforcement learning under partial coverage. arXiv preprint arXiv:2107.06626. 2021.
13

---

Published as a conference paper at ICLR 2024
Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, et al. Secrets of rlhf in large language models part ii: Reward modeling. arXiv preprint (rXXi:24010060800 2024.
Jingkang Wang, Yang Liu, and Bo Li. Reinforcement learning with perturbed rewards. In Proceed- ings of the AAAI conference on artificial intelligence, volume 34, pp. 6202-6209, 2020.
Qing Wang, Jiechao Xiong, Lei Han, Han Liu, Tong Zhang, et al. Exponentially weighted imitation learning for batched historical data. Advances in Neural Information Processing Systems, 31, 2018.
Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou. Diffusion policies as expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06192 2022.
Chen-Yu Yu Wei, Christoph Dann, and Julian Zimmert. A model selection approach for corruption robust reinforcement learning. In International Conference on Algorithmic Learning Theory, pp- 1043-1096. PMLR, 2022.
Xiaoyu Wen, Xudong Yu, Rui Yang, Chenjia Bai, and Zhen Wang. Towards robust offline-to-online reinforcement learning via uncertainty and smoothness. arXiv preprint urXiv:2309.69973 2023.
Fan Wu, Linyi Li, Chejian Xu, Huan Zhang, Bhavya Kailkhura, Krishnaram Kenthapadi, Ding Zhao, and Bo Li. Copa: Certifying robust policies for offline reinforcement learning against poisoning attacks. In International Conference on Learning Representations, 2022.
Tianhao Wu, Yunchang Yang. Simon Du, and Liwei Wang. On reinforcement learning with ad- versarial corruption and its application to block mdp. In International Conference on Machine Learning, pp. 11296-11306. PMLR, 2021.
Yueh-Hua Wu, Nontawat Charoenphakdee, Han Bao, Voot Tangkaratt, and Masashi Sugiyama. Imi- tation learning from imperfect demonstration. International Conference on Machine Learning, pp. 2311-6687.. PMLR, 2019.
Tengyang Xie, Ching-An An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman-consisten pessimism for offline reinforcement learning. Advances in neural information processing systems, 34:6683-6694, 2021a
Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridg- ing sample-efficient offline and online reinforcement learning. Advances neural information processing systems, 34:27395-27407. 2021b.
Wei Xiong, Han Zhong, Chengshuai Shi, Cong Shen, Liwei Wang, and Tong Zhang. Nearly mini- max optimal offline reinforcement learning with linear function approximation: Single-agent mdp and markov game. arXiv preprint arXiv:2205. 15512, 2022.
Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xi- anyuan Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization. arXiv preprint 233. 2023.
Tian Xu, Ziniu Li, and Yang Yu. Error bounds imitating policies and environments. Advances in Neural Information Processing Systems, 33:15737-15749, 2020.
Bo Xue, Guanghui Wang, Yimu Wang, and Lijun Zhang. Nearly optimal regret for stochastic linear bandits with heavy-tailed payoffs. arXiv preprint 2344::2200443445. 2020.
Taku Yamagata, Ahmed Khalil, and Raul Santos- Rodriguez. Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling in offline rl. In Inter- national Conference on Machine Learning, pp. 38989-39007.. PMLR, 2023.
Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. Rorl: Ro- bust offline reinforcement learning via conservative smoothing. Advances in Neural Information Processing Systems, 15:23851-23866 2022a.
14

---

Published as a conference paper at ICLR 2024
Rui Yang, Yiming Lu. Wenzhe Li, Hao Sun, Meng Fang, Yali Du, Xiu Li, Lei Han, and Chongjie Zhang. Rethinking goal-conditionas supervised learning and its connection to offline rl. In Inter- national Conference Learning Representations. 2022b.
Rui Yang, Lin Yong, Xiaoteng Ma, Hao Hu, Chongjie Zhang, and Tong Zhang. What is essential for unseen goal generalization of offline goal-conditioned ri? In International Conference on Machine Learning. pp. 39543-39571 PMLR. 2023.
Chenlu Ye, Wei Xiong, Quanquan Gu, and Tong Zhang. Corruption-robust algorithms with uncer- tainty weighting for nonlinear contextual bandits and markov decision processes. In International Conference on Machine Learning, pp. 39834-39863. PMLR, 2023a.
Chenlu Ye, Rui Yang, Quanquan Gu, and Tong Zhang. Corruption-rol coruptttoo-robust offline reinforcement learning with general function approximation. arXiv preprint, 2023b.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129-1444 14142, 2020.
Andrea Zanette, Martin Wainwright, and Emma Brunskill. Provable benefits of actor-critic meth- ods for offline reinforcement learning. Advances in neural information processing systems, 34: 16626-13640, 2021.
Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations state observations. Ad- vances in Neural Information Processing Systems, 33:21024-21037, 2020.
Huan Zhang, Hongge Chen, Duane Boning, and Cho-Jui Hsieh. Robust reinforcement learning on state observations with learned optimal adversary. arXiv preprint arXiv:2101.0844 2021.
Xuezhou Zhang, Yiding Chen, Xiaojin Zhu, and Wen Sun. Corruption-robust offline reinforcement learning. In International Conference Artificial Intelligence and Statistics, pp. 5757-5773. PMLR, 2022.
Han Zhong, Jiayi Huang, Lin Yang, and Liwei Wang. Breaking the moments condition barrier: No-regret algorithm for bandits with super heavy-tailed payoffs. Advances in Neural Information Processing Systems, 11:117100-57200 2021.
Han Zhong, Wei Xiong, Jiyuan Tan, Liwei Wang, Tong Zhang, Zhaoran Wang. and Zhuoran Yang. Pessimistic minimax value iteration: Provably efficient equilibrium learning from offline datasets. In International Conference on Machine Learning, pp. 27117-27142. PMLR, 2022.
Zhengqing Zhou, Zhengyuan Zhou, Qinxun Bai, Linhai Qiu, Jose Blanchet, and Peter Glynn. Finite- sample regret bound for distributionally robust offline tabular reinforcement learning. In Interna- tional Conference on Artificial Intelligence and Statistics, pp. 3331-3339. PMLR, 2021.
Vincent Zhuang and Yanan Sui. No-regret reinforcement learning with heavy-tailed rewards. In International Conference on Artificial Intelligence and Statistics, pp. 3385-3393, PMLR, 2021.
15

---

Published as a conference paper at ICLR 2024
A RELATED WORKS
Offline RL. Ensuring that the policy distribution remains close to the data distribution is crucial for
offline RL, as distributional shift can lead to unreliable estimation (Levine et al.
2020). Offline RL.
algorithms typically fall into two categories to address this: those that enforce policy constraints on
2019 Li et al.] 2020; Fuji- the learned policy (Wang et al., 2018 Fujimoto et al. 2019 Peng et al.
moto & Gu 202 Kostrikov et 2021 Chen et al., 202 Emmons et al. [022] 1 Sun et al. 2004 Xu
itt al. 2023), and those that learn pessimistie value functions to penalize OOD actions (Kumar et al.
2020 Yu et al. 2020; An et al.
202 Bai et al. [022; Yang et al., 2022a Ghasemipour al. 2022
Sun ct al. 2022a| Nikulin et al. 2023)
). Among these algorithms, ensemble- approaches (An)
et al.
2021 Ghasemipour et al. 2022 demonstrate superior performance by estimating the lower-
confidence bound (LCB) of Q values for OOD actions based on uncertainty estimation (Loquercio
jet al.) 2020 Sun al., 2022b)) Additionally, mmitation-based (Emmons et al.] (202T) weighted
imitation-based algorithms (Wang et al., 2018: Nair et al. [020 Kostrikov et al., (021]) generally enjoy better simplicity and stability compared to other pessimism-basel approaches. Studies in
offline goal-conditioned RL (Yang et al. 2023: 2022b) further demonstrate that weighted imita- tion learning offers improved generalization compared to pessimism-
offline RL methods. In addition to traditional offlin RI. methods, recent studies employ advanced techniques such as trans- former (Chen et al. 2021 Chebotar et al. [023]] Yamagata et al. (0033) and diffusion models (Janner et al., 2022 Hansen- Estruch et al. 2023 Wang al. (022) for sequence modeling or function class enhancement, thereby increasing the potential of offline RL to tackle more challenging tasks.
Offiine RL Theory. There is large body of literature (Jin et al. 2021 Rashidinejad et al. 2021
Zanette et al. 2021 Xie
Xie et al. [202aa] [|] Uehara & Sun 2001: Shi et al. 2022 Zhong et 2022 Cui & Dul 2022 Xiong et al. 2022 Li et al.] 2022; Cheng et al., 2022) dedicating to the development of pessimism based algorthhms for offline RL. These algorithms can provably efficiently find near- optimal policies with only the partial coverage condition. However, these works do not consider the corrupted data. Moreover, when the cumulative corruption level is sublinear, i.e., C = o(N), our algorithm can handle the corrupted data under similar partial coverage assumptions.
Robust RL. One type of robust RL is the distributionally robust RL, which aims to learn a policy that optimizes the worst-case performance across MDPs within an uncertainty set, typically framed as a Robust MDP problem (Nilim & Ghaoui] 2003 Lyengar 2005 Ho et al., 2018 Moos et al. 2022). Hu et al. (2022) prove that distributionally robust RL can effectively reduce the sim-to-real gap. Besides, numerous studies in the online setting have explored robustness to perturbations on observations (Zhang et al. 2020 2021), actions (Pinto et al. 2017 Tessler et al. 2019), rewards [Wang et al. 2020 Husain et al.) (02T), and dynamics (Mankowitz et al. [019)) )). There is also
a line of theory works (Lykouris et al. 2021 Wu et al. [021] Wei et al. 2022; Ye et al. 2023a)
studying online corruption-robust RL. In the offline setting, number of works Tocus testing-ting
robustness or distributional robustness in offline RL (Zhou et al. 2021 Shi & Chil 2022
Hu et al.
2022 Yang al.) 2022a Panaganti et al. 2022 Wen et al. 2023 Blanchet et al.. 2023 Regarding
the training-time robustness of offline RL, Li et al. (2023) investigated reward attacks in offline
RL., revealing that certain dataset biases can implicitly enhance offline RL's resilience to reward corruption. Wu et al. (2022) propose certification framework designed to ascertain the number of tolerable poisoning trajectories in relation to various certification criteria. From a theoretical perspective, Zhang et al. (2022) studied offline RL under contaminated data. One concurrent work Ye et al., 2023b) leverages uncertainty weighting to tackle reward and dynamics corruption with theoretical guarantees. Different from prior work, we propose an algorithm that is both provable and practical under diverse data corruption on all elements.
Robust Imitation Learning. Robust imitation learning focuses imitating the expert policy using corrupted demonstrations (Liu et al. (022) or mixture of expert and non-expert demonstrations (Wu et al. 2019 Tangkaratt et | 20200b Sasaki & Yamashina 2020). These approaches primarily concentrate on noise or attacks on states and actions, without considering the future return. In contrast, robust offline RL faces the intricate challenges associated with corruption in rewards and dynamics.
taileedess in RL. In the realm of RL, Zhuang & Sui (2021): Huang al. (0223 delved into the issue of heavy-tailed rewards in tabular Markov Decision Processes (MDPs) and function approximation, respectively. There is also a line of works (Bubeck et al., 2013 Shao et al. 2018
16

---

Published as a conference paper at ICLR 2024
Xue et al. 2020
Zhong et al. 2021: Huang et al. 2022 Kang & Kim 2023) studying the heavy-
tailed bandit, which is a special case of MDPs. Besides, Garg et al. (2021) investigated the heavy- tailed gradients in the training of Proximal Policy Optimization. In contrast, our work addresses the heavy-tailed target distribution that emerges from data corruption.
Huber Loss in RL. The Huber loss, known for its robustness to outliers, has been widely employed in the Deep Q-Network (DQN) literature, (Dabney et al.] 20188] Agarwal et al.) 2020; Patterson al. 2022). However, Ceron Castro (2021) reevaluated the Huber loss and discovered that fails to outperform the MSE loss on MinAtar environments. In our study, we leverage the Huber loss address the heavy-tailedness in Q targets caused by data corruption, and we demonstrate its remarkable effectiveness.
B ALGORITHM PSEUDOCODE
The pseudocode of RIQL and IQL can be found in Algorithm I and Algorithm 2 respectively:
Algorithm 1: Robust IQL algorithm
Initialize policy a and value function Ve. Qo, i [1, K]
Normalize the observation according to Section 5.1 for training step= 1, 2, ..... do Sample mini-batch from the offline dataset: ((s,,,T,s')) ~ D:
Update value function VP minimize
(v( (e.a) (45 V+(+))]
Update {Qo, 1K independently to minimize
(Q((,)
(s, as))
Update policy TΦ to maximize
(x( (( [exp(BA. log oo (a|s)]
end for
Algorithm 2: IQL. algorithm (Kostrikov et al. (202)) (for comparison)
Initialize policy a and value function Vi. Qo for training step= 1, 2.......7 do
Sample mini-batch from the offline dataset: a(,,,,,,)') D:
Update value function Vi to minimize (2)
Update Qo minimize (I:
Update policy T to maximize
end for
C THEORETICAL ANALYSIS
C.I PROOF OF THEOREM 3
Before giving the proof of Theorem 3 we first state the performance difference lemma.
Lemma (Performance Difference Lemma (Kkkade & Langford 2002)). For any and ni' it holds
that
T Va' Eysar Ea (s.a)].
Proof. See Kakade & Langford (2002) for detailed proof.
17

---

Published as a conference paper at ICLR 2024
Proof of Theorem First, we have
V
V
V
V V
RiQu
imitation error
corruption error
imilation error 2
We then analyze these three error terms respectively. Corruption Error. By the performance difference lemma (Lemma
we have
VTB
V
an (a (1 (s),,)))
E snndi LE Ba (1)) (V QTT(s,
1
1- E Rnn B BENTE( (Q)
1Q (s) Bu (QTH
Rmax
7)2 TE s [|
8) = 8)]11},
(11)
where the second inequality uses the fact that ( cquality obtained by Hölder's ' inequality and the fact that || loo S Runax/ / (1- By Pinsker's inequality, further have
(s. Q (s, V= and the last in-
E dt [1( (s) - TF(. [s]]]]] S Exnat [V
2KL ( 8), Tpl. |s)]]
T 8) TE (s)
2E (a,a)radi log
( |s)
2ME (3.a)~dpp
log
Tp
2M N
#Ø s;)
log
(12)
TE TT (a: s;)
where the second inequality follows from Jensen's inequality and the definition KL- divergence, the third inequality is obtained by the coverage assumption (Assumption 2 that sup,,, s.a 1d (s, (d, s M, and the last inequality uses the definition of dp. Recall that Te and He take the following forms:
TE(a s) x (a(a 8) exp(8. [TV' V))(,,)),,
#g s) x TD(a s) exp(B 'TV* V*((s,))).
We have
( s:)
TDaa; si) exp(3 TT - V*
Tg (a; s;)
TT (a; si) exp(B. (3. [TV- V*[(s,, a:))
(13)
E (n 8i) exp(3 3: TV - (* a))
EocA (a 8;) exp(3 [TV+ - V*]
By the definition of corruption levels in Assumption
we have
max TD(a 8i) Ta 8i)
S = IITTV - [TV] - alllo,
1 S.
Va e A.
T si) TD(a st)
This implies that
<D 8:) exp(p [Tv* [s 0z))
< < exp(((()
n (a; 8y) e3 [TV [' a:))
(14)
(a 8:) exp(B [TV* V*] a))
< c exp(
Va 4.
TD(a s:) exp(3. [TV* V*I a))
18

---

Published as a conference paper at ICLR 2024
Plugging 14 into 13
we have
mg 04 s;)
- -xx((3()) Enes TD(a (.)) exp(3 TTV* V=((s.)))
< Ç exp()(i)
TE(a, [; si)
ΣeEs TD(a 8;) exp(B [TV* V-s(ss,, 0))
exp 2.()).
(15)
Combining 11 12 and
15) we have
V V
1 (1 7)3
2Rmax
M
Σ (log Ç 28Ç,)
2Rmax
MC
(16)
(1 2}-
N
where the last equality uses 3 = and the definition of Ç in Assumption
1
Imitation Error 1. Following the derivation of [1 we have
Rmax
rIiqL VER
Randrr [| 8) T
[|]]]]
7)2
22R 2 E ESndrE KL(TT((
s), TIQ(( s))]
(1 7)2
v /2Rmaxx
(1 7)2 /Laad'E [KL(TE(
s), --QU(- s))]
22 Rimax
22 Pmax
71- Ta (KT(((
8), (- {))]
(1 7) Vel,
(17)
where the second inequality uses Pinsker's inequality, the third inequality is obtained by Jensen's inequality, the fourth inequality uses Assumption and the final equality follows from the definition of é1.
Imitation Error 2. By the performance difference lemma (Lemma and the same derivation of 11 we have
Rmax E, Exnd+e Enwūg(())) (A a))
VdE Vfla
1-7
Rmax E Bondtell l|| #Iqu( s)
< (1 7)2
By the same derivation of 117 we have
/2M V2M Rmax
22M Rmax
V faQu <
(EaDD(KL(TEE 8), Ā. |s)]] s
(1 (1-7)2 VE2.
(18)
(1 - 7)2
Putting Together. Combining
16
17 and 18 we obtain that
22 Rinxx
2 2Rmax Wa + Ve2)+
MC
VIQQL ViQQL
<
(1 (1- 7)2
(1 21- N
which concludes the proof of Theorem 3
We would like to make a comparison with Zhang et al. (2022). a theory work about offline RL with corrupted dutu. They assume that N data points are corrupled and propose least- squares value iteration (LSVI) type algorithm that achieves an ((ve) optimality gap by ignoring other parameters. When applying
Remark 7.
results to their setting, have Ç S eN, which implies that our corruption error term ((/C/N) O(VE). This matches the result in Zhang al. 2022)) which combines LSVI with a robust regression oracle. However, their result may not readily apply to scenario, given that permit complete data corruption (c 1). Furthermore, from the empir- ical side, IQL does not require any robust regression oracle and outperforms LSVI-rype algorithms, such EDAC and MSG.
19

---

Published as a conference paper at ICLR 2024
C.2 PROOF OF LEMMA 5
Proof of Lemma
5
We first state Theorem of
Sf Sun et al. (2020) as follows.
Lemma 8 (Theorem Sun al. 2020)). Consider the following statistical model:
yi = (2i, +
with E (c; m;) = O, E(jel + < oc, v1 s
By solving the Huber regression problem with a proper parameter ö
3 arg min Σ (y (x B)).
the obtained B satisfies
|| 82 2 N min min(s/((40))111))
Back to our proof, for any (ss, a,.))))) ~ D. we take (i) 2, = 1ss a+} E R|SA as the one-hot vector (st, az) and (ii) Y piEss (V()))) E RISA as Sr. Hence, by Lemma 8 and the fact that 11 S || 11112, we obtain
(s)). Moreover, treat {r(s, +
(Q(ssa) r((s, P(-s,)) (b')ll S N min(v/(1+0)112))
where N is the size of dataset D. This concludes the proof of Lemma
Discussion about Recovering the Optimal Value Function denote the optimal solutions to and by VA and Qr, respectively. When N + we know
(s) E ()) 1Q, [s, (r, r(s,a) P(saa)) [V (s')),
where -T denotes the 7-th expectile of the random variable X. following the analy-
sis of IQL (Kostrikov et al. 2021 Theorem 3), we can further show that lim,-11 Vi(s) max.cAst s.. =p(als)>0 Q" a.. With the Huber loss, we can further recover the optimal value func- tion even the presence heavy-tailed target distribution.
D IMPLEMENTATION DETAILS
D.1 DATA CORRUPTION DETAILS
We apply both random and adversarial corruption to the four elements, namely states, actions, re- wards, and dynamics (or "next-states"). In experiments, replay-v2" and "medium-expert-x2" datasets from (Fu et al.] 2020). These datasets are gathered either during the training of SAC agent or by combining equal proportions of expert demonstra- tions and medium data, making them more representative of real-world scenarios. To control the cumulative corruption level, we introduce two parameters, and Here, c represents the corruption rate within the dataset of size N, while denotes the corruption scale for each dimension. We detail four types of random data corruption and a mixed corruption below:
primarily utilize the "medium-
Random observation attack: We randomly sample e- transitions (s,, a, r, s'), and mod- ify the state to å = + X std, Uniform[ -e, e)d,, Here, d, represents the dimension of states and "std(s)" dataset. The noise is scaled according to the standard deviation of each dimension and is independently added to each respective dimension.
the d.-dimensional standard deviation of all states in the offline
Random action attack: We randomly select e- transitions (s, as,0, r, s'), and modify the action to à = + X std(a), À Uniformi-ed e],, where d, represents the dimension of actions and "std(a)" " is the d, dimeensnii standard deviation of all actions in the offline dataset.
20

---

Published as a conference paper at ICLR 2024
Random reward attack: We randomly sample N transitions (s, a.) from D, and modify the reward to UUniform [-30- c, 30 - e/. We multiply by 30 because we have noticed that offline RL algorithms tend to be resilient to small-scale random reward corrup- tion (as observed (Li et al. 2023)), but would fail when faced with large-scale random reward corruption.
Random dynamics attack: We randomly sample c N transitions (s, a, s'),, and modify the next-step state 8' s' + A std(s'), X Uniform[-c, c).. Here, dg indicates the dimension of states and "sttdss""" is the dg fe-dimensional standard deviation of all next-states in the offline dataset.
Random mixed attack: We randomly select e- N of the transitions and execute the random observation attack. Subsequently, we again randomly sample c N of the transitions and carry out the random action attack. The same process is repeated for both reward and dynamics attacks.
In addition, four types of adversarial data corruption are detailed as follows:
Adversarial observation attack: We first pretrain an EDAC agent with set of Qp functions and policy function Tp using clean dataset. Then, we randomly sample c N transitions (s, a, r, s'), and modify the states to = minseRa(s.e) @p(s Here, Ba(s, c) = {š|ls s| S e std(s)} regularizes the maximum difference for each state di- mension. The Q function in the objective the average of the Q functions in EDAC. The optimization is implemented through Projected Gradient Descent similar to prior works (Madry et al. 2017) Zhang et al.] 2020). Specifically, We first initialize a learnable vector 2 E c Th and then conduct a 100-step gradient descent with a step size of 0.01 for 8 = 8 + z std(s), and clip each dimension of = within the range [-e, after each update.
Adversarial action attack: We use the pretrained EDAC agent with group Op functions and policy functiion Tp. Then, we randomly sample c N transitions (s, a, r, s'), and modify the actions
mingeB.(e.e)) Qp a). Here, BBa(a,)) e) {āllà al S std(a)) regularizes the maximum difference for each action dimension. The optimization is implemented through Projected Gradient Descent, as discussed above.
Adversarial reward attack: We randomly sample e- transitions (8, r, 8'), and directly
modify the rewards to: f - X r.
Adversarial dynamics attack: We use the pretrained EDAC agent with a group of Op functions and a policy function Tp. Then, we randomly select N transitions (s, a, r,,,'s), and modify the next-step states to 8' mingre Qp (8))). Here, Ba(s'se) = < std(s')). The optimization the same as discussed above. {
Table 3: Hyperparameters used for RIQL under the random corruption benchmark.
Environments
Attack Element
N
8
observation
5
0.1
0.1
action
3
0.25
0.5
Halfcheetah
reward
5
0.25
3.0
dynamics
5
3.0
observation
5
0.25
0.1
action
5
0.1
0.5
Walker2d
reward
5
0.1
3.0
dynamics
0.25
1.0
observation
3
0.25
0.1
action
5.
0.25
0.1
Hopper
reward
3
0.25
1.0
dynamics
5
1.0
D.2 IMPLEMENTATION DETAILS OF IQL AND RIQL
For the policy and value networks of IQL and RIQL, we utilize an MLP with 2 hidden layers, each consisting of 256 units, and ReLU activations. These neural networks are updated using the Adam
21

---

Published as a conference paper at ICLR 2024
Table 4: Hyperparameters used for RIQI. under the adversarial corruption benchmark.
Environments
Attack Element
N
8
observation
5
0.1
0.1
action
5
0.1
1.0
Halfcheetah
reward
5
0.1
1.0
dynamics
5
0.1
1.0
observation
5
0.25
1.0
action
5
0.1
1.0
Walker2d
reward
5
0.1
3.0
dynamics
5
0.25
1.0
observation
5
0.25
1.0
action
5
0.25
1.0
Hopper
reward
5
0.25
0.1
dynamics
5
0.5
1.0
optimizer with learning rate of 3 10-4. We set the discount factor yy 0.99, the target networks are updated with a smoothing factor of 0.005 for soft updates. The hyperparameter 3 and T for IQL and RIQL are to 3.0 and 0.7 across all experiments. In terms of policy parameterization, we argue that RIQL is robust to different policy parameterizations, such as deterministic policy and diagonal Gaussian policy. For the main results, IQL and RIQL employ deterministic policy, which means that maximizing the weighted log- ligeliildhood is equivalent to minimizing weighted l2 loss on the policy output: (o) (<s.a)>D(eexp (B,(ss, a)) To (s) We also include discussion about the diagonal Gaussian parameterization in Appendix E.44 In the training phase, we train IQL., RIQL, and other baselines for 3 x 10 steps following (An al. 2021). which corresponds to 3000 epochs with 1000 steps per epoch. The training is performed using batch size of 256. For evaluation, we rollout each agent in the clean environment for 10 trajectories (maximum length equals 1000) and average the returns. All reported results are averaged over four random seeds. As for the specific hyperparameters of RIQL, we search E {3. 5) and quantile a E {0.1, 0.25, 0.5] for the quantile Q estimator, and 5 E (0.1, 0.5, 1.0, 3.0} for the Huber loss. In most settings, we find that the highlighted hyperparameters often yield the best results. The specific hyperparameters used for the random and adversarial corruption experiment in Section 6.. are listed in Table and Table 4 respectively. Our code is available at https://githu..com/ /ut.uuuccooaaaaggiii1////00 which is based on the open-source library of CORL (Taassvv 2022)
D.3 QUANTILE CALCULATION
To calculate the a-quantile for group of Q function (Qo, ) we can map E [0, 1] to the range of indices 1, K] in order determine the location of the quantile the sorted input. the quantile
lies between two data points Qo, < Qe, with indices i.j in the sorted order, the result is computed according to the linear interpolation: Q. = Qo, + (Qa, Qa,) ) * fraction(a x (K - 1) + 1), where the "fraction" represents the fractional part of the computed index. As special case, when K = 2, Qa = (1 (1--oh a) min Q2, Q8,) + a max Qe, Qo,), and Qa recovers the Clipped Double Q-learning trick when a
E ADDITIONAL EXPERIMENTS
E. ABLATION OF IQL.
As observed in Figure | IQL demonstrates notable robustness against some types of data corruption. However, it raises the question: which component of IQL contributes most to its robustness? IQL can be interpreted as combination of expectile regression and weighted imitation learning. To understand the contribution of each component, we perform ablation study IQL under mixed corruption in the Hopper and Walker tasks, setting the corruption rate to 0.1 and the corruption scale 1.0. Specifically, we consider the following variants:
IQL = 0.7: This is the standard IQL baseline.
IQL. 0.5: This variant sets
0.5 for IQL.
22

---

Published as a conference paper at ICLR 2024
IQL w/o Expectile: This variant removes the expectile regression that learns the value function and instead directly learns the Q function to minimize (s..a..,''') +>Q x Q Qs, and updating the policy to maximize
(s, π (8))) log (a]s))
ER w. Q gradient: This variant retains the expectile regression to learn the Q function and function, then uses only the learned Q function to perform a deterministic policy gradient: (Qx, (
110
Ferock
The results are presented in Figure From these, we can con- clude that while expectile regression enhances performance, it is not the key factor for robustness. On the one hand, IQL r 0.7 outperforms IQL 0.5 and IQL w/o Expectile, confirming that expectile regression indeed an enhancement factor. On the other hand, the performance of ER nearly zero, significantly lower than BC, indicating that expectile regression not the crucial component for robustness. Instead, the supervised policy learning scheme is proved to be the key to achieving better robustness.
20-
15-
gradient drops
0.5
Flopee
Walker
Figure Ablation results of IQL under mixed attack.
E.2 ABLATION OF OBSERVATION NORMALIZATION
Figure 9 illustrates the comparison between IQL and "IQL (norm)" the medium-reple and medium-exper datasets under various data corruption scenarios. In most settings, IQL with nor- malization surpasses the performance of the standard IQL.. This finding contrasts with the general offline Reinforcement Learning (RI) setting, where normalization does not significantly enhance performance, noted by (Fujimoto & Gu 202T). These results further justify decision in- corporate observation normalization into RIQL.
tegeo-mnetarnnng www.orrttesddeerrooo
wedov mwc.immmmmmorrrrr
3833
artuck
artic
(a)
(b)
(d)
Figure Comparison of IQL w/ and w/o observation normalization.
E.3 EVALUATION UNDER THE MIXED CORRUPTION
We conducted experiments under mixed corruption settings, where corruption independently oc- curred on four elements: state, actions, rewards, and next-states (or dynamics). The results for corruption rates of 1/0.0 and corruption scales of 1.0/2.0 the "medium-reppla"" datasets presented in Figure T0 and Figure II From the figures, performs other baselines by significant margin in the mixed attack settings with varying corruption rates and scales. Among the baselines, EDAC and MSG continue to struggle this corruption setting. Besides, CQL also serves as a reasonable baseline, nearly matching the perfor- mance of IQL in such mixed corruption settings. Additionally, SQL also works reasonably under small corruption rate 0.1 but fails under corruption rate of 0.2.
evident that RIQL consistently out-
E.4 EVALUATION OF DIFFERENT POLICY PARAMETERIZATION
In the official implementation of IQL Kostrikov et al. 2021), the policy is parameterized diagonal Gaussian distribution with a inte-inndeenndent standard deviation. However, in the data corrptiion setting, findings indicate that this version of IQL generally underperforms IQL with a deterministic policy. This observation supported by Figure 12, where we compare the perfor- the deterministic
mance under the mixed attack with corruption rate of 0.1. Consequently,
23

---

Published as a conference paper at ICLR 2024
1e3
Hopper
1e3
Wa lker
3.0
2.5
2.5
Episode Reward
2.0
2.0
1.0
1.0
0.5
0.5
0.0
Scale
Scale
BC
EDAC
MSG
COL
SQL
IQL
RIQL
Figure 10: Results under random mixed attack with corruption rate of 0.
1e3
Hopper
1e3
Wa lee
2.5
2.5
Bolsode Reward
2.0
2.0
0.5
0.5
0.0
0.0
Scale 2.0
Scale 1.0
2.0
BC
EDAC
MSG
CQL
SQL
IQL
RIQL
Figure 11: Results under random mixed attack with corruption rate of 0.2.
Hopper
Wal ker
2.5
2.5
1.5
1.5
0.5
0.5
Scale 1.0
Scale 2.0
Scale 1.0
Scale 2.0
Deterministic
Deterministic
Figure 12: Comparison of different policy parameterizations under random mixed attack with corruption rate of 0.1.
policy for both IQL and RIQL by default. Intriguingly, RIQL demonstrates greater robustness to the policy parameterization. In 12, "RIQL- Gaul-Gaa can even surpass "RIQL. Determin- istic" and exhibits lower performance variance. We also note the consistent improvement of RIQL over IQL under different policy parameterizations. which serves as additional advantage of RIQL. Based on our extensive experiments, we have noticed that "RIQL- Gaussian" offers greater stabil- ity and necessitates less hyperparameter tuning than "RIQL- Deterministic" Conversely, "RIQL Deterministic" requires more careful hyperparameter tuning but can yield better results on average in scenarios with higher corruption rates.
24

---

Published as a conference paper at ICLR 2024
Table 5: Average normalized performance under random data corruption (scale 1.0) using "medium-expers datasets, Results are averaged over 4 random seeds.
Environment
Attack Element
BC
EDAC
MSG
CQL
SQL
IQL
RIQL (ours)
observation
51.611299
4.0=332
.4.914455
1.4
30.51:5.1
71.01:34
79.7:1..2
61.245.6
45.4=3.2
328444.1
67.549.7
89.810.4
88.611.2
92.9:10.6
Halfcheetah
reward
59.91:3.3
6.88:666
28.7:16.0
87.34.5.1
92.91007
93.41122
93.2:10.5
dynamics
59.943.3
25.41:1.3
2.542244
54.0##1.9
78.343..
804420
92.2:10.9
observation
106.2=1.7
-0.=000
-0.340.1
13.042..6
17.544433
71.1423.5
101.3411.8
action
89,3+17.3
48.41:20.5
3.8
109.240.33
109.7=0.7
112.740.5
113.5:::0.5
Walker2d
reward
97.9+15.4
-0.44033
110.741..3
112.7:4-0.6
dynamics
97.9=15.4
0.2+00.5
3,6+6,7
82.0=
93.5443.7
100,3:7.55
112.8:10.2
observation
52.74116
03::0000
0.910.1
46.2 12.9
1,3::0.1
23.2:133.6
24.222242
Hopper
50.41:.5
23.5:6.0
18.8110.6
83.51
59.11:32.0
77.8116.4
92.6=38899
reward
52.41155
0.7400.0
3.945.44
90.8+33
15.4
78.6++31.0
91.4-25.9
dynamics
52.411.5
5.6:16.5
2.14:244
26.7:17.5
75.8:19.3
69.741.5..3
94.6-15.4
Average
69.3
19.4
7.6
64.2
70.3
81.5
95.3
E.5 EVALUATION ON THE MEDIUM-EXPERS DATASETS
In addition to the "medium-replay" dataset, which closely resembles real-world environments, also present results using the "medium-expert" dataset under random corruption Table [5] The "medium-expert" dataset is a mixture of 50% expert and 50% medium demonstrations. sult of the inclusion of expert demonstrations, the overall performance of each algorithm is much better compared to the results obtained from the "medium-reeply"" datasets. Similar the main results, EDAC and MSG exhibit very low average scores across various data corruption scenarios. Additionally, CQL and SQL only manage match the performance of BC. IQL, on the other hand, outperforms BC by 17.6%, confirming its effectiveness. Most notably. RIQL achieves the highest score in 10 out of 12 settings and improves by 16.9% over IQL. These results align with our findings in the main paper, further demonstrating the superior effectiveness of RIQL across different types of datasets.
a re-
Table 6: Average normalized performance under random corruption of scale 2.0 using the "medium-replay" datasets. Results are averaged over random seeds.
Environment
Attack Element
BC
EDAC
CQL
SQL
IQL
RIQL (ours)
observation
16.115.5
-0440.00
-0.410.1
12.5:2214
0.74:1.2
27.4::5.7
$0.3=11.5
Walker2d
action
13.342.5
77.5433
16.323.7
154199.1
79.0=15.2
69.9112.8
82.9433.8
reward
16.0+7.4
1,2+1.6
9.3+5.9
48.444.6
0,2+-1.9
56.0+-7.4
81.5-14.66
dynamics
16.0:77.
-0.120.0
4,243.88
0.1::0.4
18.5433.6
12.0::5.6
77.8-16.3
17.6+1.2
2.740.8
29.21119
13.743.11
69.7:17.00
17.4
Hopper
action
16.0+3.1
31.9+3.22
35.9=8.3
19.2:10.5
38.8
18.1
92.7=11.0
reward
19.5+3.44
6.4-15.0
42. 42.14:142
51.7:+14.8
41.5:1122
62.1118.3
85.0=17.3
dynamics
19.5+:3.4
0.8+H0.1
14.4+22.7
45.8-19.3
Average
16.8
15.0
14.8
22.2
25.9
46.6
71.6
Table 7: Average normalized performance under adversarial corruption of scale 2.0 using the "medium-replay" datasets. Results are averaged over 4 random seeds.
Environment
Attack Element
BC
EDAC
MSG
CQL
SQL
IQL
RIQL (ours)
observation
13.0+1..5
3.966.7
3.54399
63.345..1
2.7+3300
47.0+18.3
71.5-16..4
Walker2d
action
0.3++0.5
9.2+244
4.8+0.4
1.7:+3,5
5.6
3.=+0.9
7.9-0.5
reward
16.0:7.4
-0.110.0
9.12338
28.4111922
-0.311.00
22.9-12.5
81.11:2.4
dynamics
16.0+7.4
2.21..88
1.920.2
4.2422,2
3.4+-199
1.7+10
66.0-17.77
obscrvation
17.023.4
239110.6
19.9:6.6
$2.618.
14.0=E12
44.3:22.9
57.0117.6
action
9,944.6
23.3:14.7
22.8411.7
14.7409
19.3413.1
21.8:1499
23.8-144.0
Hopper
reward
19.5+33.4
1.2+:0.5
22.9+11.1
22.9
0.6+:0.0
24.7+:1.6
35.5
dynamics
19.5+3.4
0.6:10.0
0.6:0.00
0.6:10.0
24.3:13.2
0.7-0.0
29,9-11.5
Average score
13.9
8.0
10.7
23.6
8.7
20.8
46.6
25

---

Published as a conference paper at ICLR 2024
E.6 EVALUATION UNDER BARGE-SCCALE CORRUPTION
In our main results, we focused on random and adversarial corruption with a scale of 1.0. In this subsection, we present empirical evaluation under corruption scale of 2.0 using the "medium- replay" datasets. The results for random and adversarial corruptions are presented in Table 6 and Table respectively.
In both tables, RIQL achieves the highest performance in out of settings, surpassing IQL by 53.5% and 124.0% respectively. Most algorithms experience significant decrease in performance under adversarial corruption at the same scale, with the exception of CQL. However, it is still not comparable to our algorithm, RIQL. These results highlight the superiority RIQL handling large-scale random and adversarial corruptions.
E.7 TRAINING TIME
We report the average epoch time on the Hopper task as measure of computational cost in Table From the table, it is evident that BC requires the least training time, as has the simplest algo- rithm design. IQL requires more than twice the amount of time compared to BC, as it incorporates expectile regression and weighted imitation learning. Additionally, RIQI. requires comparable computational cost to IQL., introducing the Huber loss and quantile Q estimators. On the other hand, EDAC, MSG, and CQL require significantly longer training time. This is primarily due to their re- liance larger number of Q ensembles and additional computationally intensive processes, such as the approximate logsumexp via sampling in CQL. Notably, DT necessitates the longest epoch time, due to its extensive transformer-bast based architecture. The results indicate that RIQL achieves significant gains robustness without imposing heavy computational costs.
Table 8: Average Epoch Time.
Algorithm
BC
DT
EDAC
MSG
CQL
IQL
RIQL
Time (s)
3.8:E0.1
28.9
14.1
12.0.0.88
22.8
8.7.10.3
9.2-10.4
RIQL 06
RIOL (6=
RIOL (6 1.0)
RIOL 36=3.0)
IQL
x16²
hopper-mediun-replay
x10"
wallker2ddmeddlu--eplayy
Average Return
1.5
1.5
1.0
0.5
0.0
1000
2000
3000
1000
2000
3000
Epochs
Epochs
Figure 13: Hyperparameter study for 8 in the Huber loss under the mixed attack.
E.8 HYPERPARAMETER STUDY
In this subsection, we investigate the impact of key hyperparameters in RIQL, namely ō in the Huber loss, and a. K in the quantile Q estimators. We conduct our experiments using the mixed attack, where corruption is independently applied to each element (states, actions, rewards, and next-states) with corruption rate of 0.2 and corruption scale of 1.0. The dataset used for this evaluation the "medium-replay" dataset of the Hopper and Walker environments.
Hyperparameter In this evaluation, we set a = 0.1 and = for RIQL. The value directly affects the position of the boundary between the l loss and the la loss. As 8 approaches
26

---

Published as a conference paper at ICLR 2024
RIQL (a= 0.1)
RIQL (a= 0.25)
RIQL (a 0.5)
IQL
hopper-medium-replay
x10"
wallkerdd-mddiu-eepla
2.0
2.0
Average Retum
1.5
1.0
0.5
0.5
0.0
1000
2000
3000
1000
2000
3000
Epochs
Epochs
Figure 14: Hyperparameter study for the quantile estimators under the mixed attack.
RIQL(K=3)
RIQL(K
RIQL(K = 10)
x16
hopper-medium-replay
x10"
walker2d-meeiuu--en
2.0
2.0
Average Retum
1.0
1.0
0.5
0.0
2000
3000
1000
2000
3000
Epochs
Epochs
Figure 15: Hyperparameter study for in the quantile estimators under the mixed attack.
the boundary moves far from the origin of coordinates, making the loss function similar to the squared loss and resulting in performance closer IQL. This can be observed in Figure 13] where 8 0.1 the closest to JQL. As increases, Huber loss becomes more similar to the 4 Joss and exhibits greater robustness to heavy-tail distributions. However, excessively large values of 8 can decrease performance, as the 4 loss can also impact the convergence. This is evident in the results, where 5 = 3.0 slightly decreases performance. Generally, 5 1.0 achieves the best nearly the best performance in the mixed corruption setting. However, it is important to note that the optimal value of also depends on the corruption type when only one type corruption (e.g., state, action, reward, and next-state) takes place. For instance, we find that ó 0.1 generally yields the best results for state and action corruption, while 1 is generally optimal for the dynamics corruption.
Hyperparameter a In this evaluation, we set = 1.0 and K = 5 for RIQL. The value of a determines the quantile used for estimating Q values and advantage values. A small a results higher penalty for the corrupted data, generally leading to better performance. This is evident in achieves the best performance in the mixed corruption setting. However,
Figure [4] where a as mentioned in Section 5.3 when only dynamics corruption is present, it is advisable to use slightly larger a, such as 0.25, to prevent the excessive pessimism the face of dynamics corruption.
Hyperparameter K In this evaluation, we set 0.1 and 5 = 1.0 for RIQL. Hyperparameter K is used to adjust the quantile Q estimator for in-dataset penalty. From Figure we can observe that the impact of K is more pronounced in the walker2d task compared to the hopper task, whereas the above hyperparameter a has greater influence on the hopper task. For the walker2d task, a larger value of K leads to better performance. Overall, K achieves the best nearly the best performance in both settings. Therefore, we set K 5 by default, considering both computational cost and performance.
27

---

Published as a conference paper at ICLR 2024
Table Comparison with additional baselines under random data corruption. Average normal- ized scores are reported and the highest score is highlighted.
Environment
Attack Element
DT
MSG
UWMSG
RIQL (ours)
observation
33.71:199
-0.2
2.1
27.
action
36.3+1.5
52.0
53.8
42.9:1..6
Halfcheetah
reward
39.2+t1.0
17.5+16.4
39.9
43.6+0066
dynamics
33.7+:1.9
1.740.4
3.6+2..0
43.1
observation
54.5+15.0
-0.40.11
1.5+2.0
28.447.7
action
10.3+3.1
25.3+10.6
61.2:t9.9
84.6:13.3
Walker2d
reward
65.5+:.9
18.449.5
65.9+13.9
83.2:+2.66
dynamics
54.54:5.0
7.43.77
6.5+2.8
78.
observation
65,24:1.1
6.945.0
11.3+4.0
1.8
action
15.5++0.7
37.6+6..5
82.0+17.6
90.6
Hopper
reward
78.0+-5.3
24.9+
61.0+
84.8++1.J 13.1
dynamics
65.2112.1
12.414.9
18.714.6
51.518..
Average score 1
46.0
17.0
34.0
60.0
E.9
ADDITIONAL BASELINES
In addition to the baselines compared the main paper, we compare RIQL to a sequence-man sequence-modeling baseline, Decision Transformer (DT) (Chen et al. 2021). and recently proposed robust offline RI. algorithm for data corruption, UWMSG (Ye et al. 2023b). The results are presented in Table 9 and
Table Overall, RIQL demonstrates robust performance compared the two additional baselines.
In comparison to BC, DT demonstrates notable improvement, highlighting the effectiveness of sequence modeling. Furthermore, as DT employs distinct sequence modeling approach from TD- based methods, it does not store data in independent transitions (st, Qg Ti, s() Instead, it utilizes a trajectory sequence {st, at, rt] for i-th trajectory. Consequently, the observation corruption identical for DTT Analyzing Table Iaieeaand and Table TO]
and dynamics corruption DT holds an advantage under observation corruption due to its ability to condition on historical infor- mation rather than relying solely on single state. Despite leveraging trajectory history information and requiring 3.1 times the epoch time compared to RIQL, DT still falls significantly behind RIQL performance under both random and adversarial data corruption. Exploring the incorporation of trajectory history, similar to DT, into RIQL could be promising avenue for future research.
observe that
With regard to UWMSG, we observe its improvement over MSG, confirming its effectiveness with uncertainty weighting. However, it is still more vulnerable to data corruption compared to RIQL. We speculate that the explicit uncertainty estimation used UWMSG is still unstable.
Table 10: Comparison with additional baselines under adversarial data corruption. Average normalized scores are reported and the highest score is highlighted.
Environment
Attack Element
DT
MSG
UWMSG
RIQL (ours)
observation action
35.5
1.1:0.22
1.3.0.88
35.7
16.6
37.3
39.4
31.7+11.7
Halfcheetah
reward
37.0++2.6
47.7+0.4
43.8+
44.1
dynamics
35.5+4..3
-1.540.00
10.9
35.8+2.1
observation
57,413.2
2.942.7
9.9+
70.0
action
9.5+1.6
5,440.9
7.422.4
66.1+4.66
Walker2d
reward
64.94:5.4
9.6:44.9
52.9+11.1
85.0+1.5
dynamics
57,4+3.2
0.1+0.2
2.1+0.2
60.6+21.88
observation
57.7+15.1
16.0.2.8
13.620..7
50.817.6
action
14.0+t1.1
23.0+:2.1
34.0
63,617.3
Hopper
reward
68.6+10.5
22.6+22.8
23.6+22.1
65.8+99.8
dynamics
57.7+15.1
0.6+0.0
0.8+0.0
65.7+21.1
Average score 1
42.7
13.7
20.0
56.2
28

---

Published as a conference paper at ICLR 2024
Obserwaion
Addon Atback
Taward Abracc
Dynam Araac
0.0
0.0
066
06
6.
AT3 HUBle
10.
KOL mam
Nd.
quarch wider
HUGK
quartl
Figure 16: Ablations of RIQL under four types of data corruption.
E.10 ADDITIONAL ABLATION STUDY
this subsection, we analyze the contributions of RILQ's components the presence of observa- tion, action, reward, and dynamics corruption. Figure T6 illustrates the average normalized perfor- mance of Walker and Hopper tasks on the medium-replay datasets. We consider three variants of RIQL: RIQL without observation normalization (RIQL w/o norm), RIQL without the quantile Q estimator (RIQL. w/o quantile), and RIQI. without the Huber loss (RIQL w/o Huber). In RIQL. w/o quantile, the Clipped Double Q trick used in IQL, is employed to replace the quantile Q esti- mator in RIQL. Based on the figure, we draw the following conclusions for each component: (1) normalization is beneficial for all four types of corruption, (2) quantile estimator is more effective for observation and action corruption but less so for reward and dynamics corruption, (3) conversely, the Huber loss is more useful for reward and dynamics corruption but less effective for observation and action corruption. Overall, all three components contribute to the performance of RIQL.
F EVALUATION ON ANTMAZE TASKS
In this section, we present the evaluation of RIQL and baselines on more challenging benchmark, AntMaze, which introduces additional difficulties due to its sparse-reward setting. Specifically, we assess four "medium" and "large" level tasks. Similar to the MuJoCo tasks, we consider random data corruption, but with corruption rate of 0.2 for all AntMaze tasks. We observe that algorithms in AntMaze tasks are more sensitive to data noise, particularly when it comes to observation and dynamics corruption. Therefore, reward (30.0), and dynamics (0.3). The hyperparameters used for RIQL are list in Talbe [2]
smaller corruption scales for observation (0.3), action (1.0),
As shown in Table 11 methods such BC, DT, EDAC, and CQL struggle to learn meaningful policies in the presence of data corruption. In contrast, IQL exhibits relatively higher robustness, particularly under action corruption. RIQL demonstrates significant improvement of 77.8% over IQL, successfully learning reasonable policies even in the presence of all types of corruption. These results further strengthen the findings our paper and highlight the importance of RIQL.
Table Average normalized performance under random data corruption on AntMaze tasks.
Environment
Attack Element
BC
DT
EDAC
CQL
IQL
RIQL (ours)
observation
0.0:0..0
0.0:10.0
0.0-0.00
0.811:4
7.41500
18.0 4.7
-tmazz-ncdium-plaa--222
0.0+00.0
0.0++0.0
0.0=0.0
0.8+1.4
reward
0.0400.0
0.0+10.0
0.0=0.0
0.0+0.0
12.48811
61.0 5,8
dynamics
0.0+10.0
0.0+10.0
0.0=0.0
37.5+:28.6
32.7 + 8.4
63.2 5.4
observation
0.0+0.00
0.0+0.00
0.0=0.1
0.0+0.00
11.9 + 6.9
12.5
ummaaze-nediim-diveeee-->
action
0.0-10.0
15.0.:22.3
65.6
reward
0.0++0.0
0.0++0.0
0.0=0.0
0.0+0.00
8.3 4.5
22.9 + 17.9
dynamics
0.0-+0.0
0.0+0.00
0.0+0.0
20.8+14.2
35.5 4 8.5
41.8 7.2
observation
0.0:10.0
0.0:10.0
0.0=0..0
0.0:0.00
0.0:10.0
27.3 5.1
0.0++0.0
0.0:+0.0
0.0=0.0
0.0+0.00
33.1
untmaze-lage-play--2
reward
0.0-10.0
0.0+10.0
0.0=0.0
0.0+0.0
0.6 +: 0.9
27.7 8.0
dynamics
0.0:10.00
0.0:20.0
0.0=0.0
4.2+:5.5
1.6 1.5
25.0 6.4
observation
0.0:10.0
0.0-10.0
0.0=0.0
0.0+0..0
0.0:10.00
14.9
0.0-10.0
0.0+0.0
antmazo-large-ivvreee..
reward
0.0-10.00
0.0:10.0
0.0=0.0
0.0.0.00
1.
20.2 17.0
dynamics
0.0++0.0
0.0+0.00
0.0+0.0
4.2+7.2
2.5 +
16.5
Average score
0.0
0.0
0.0
5.2
19.4
34.5
29

---

Published as a conference paper at ICLR 2024
Table 12: Hyperparameters used for RIQL under the random corruption the AntMaze Tasks.
Environments
Attack Element
N
observation
0.25
0.5
action
3
0,5
0.1
antrnazeemmcddin--laay-v2
reward
3
0.5
0.5
dynamics
0.5
0.5
observation
0.25
0.1
action
3
0.5
0.1
antmaze memmzze-mddummdiiverss...
reward
3
0.25
0.1
dynamics
0.5
0.1
observation
0.25
0.5
action
3
0.25
0.1
antmaze-argge-layyy
reward
3
0.5
0.5
dynamics
0.5
observation
0.25
0.5
action
5
0.25
0.05
autmaze-lage-ddivrree
reward
0.25
0.5
dynamics
0.5
0.5
F.1 LEARNING CURVES OF THE BENCHMARK EXPERIMENTS
In Figure 17 and Figure 18 we present the learning curves of RIQL and other baselines the "medium-replay" datasets, showcasing their performance against random and adversarial corrup- tion, respectively. The corruption rate is results correspond to Section in the main paper. From these figures. it is evident that RIQL not only attains top-level performance but also exhibits remarkable learning stability compared to other baselines that can experience significant performance degradation during training. We postulate that this is attributed the inherent characteristics of the Bellman backup, which can accumulate errors along the trajectory throughout the training process, particularly under the data corruption setting. On the contrary, RIQL can effectively defend such effeet and achieves both exceptional performance and remarkable stability.
0.3 and the corruption scale is
1.0. These
G DISCUSSION OF THE CORRUPTION SETTING
In our corruption setting, the states and next-states independently corrupted. However, work does not consider the scenario where next-states are not explicitly saved the dataset, meaning that corrupting next-state would correspond corrupting another state in the same trajectory.
The motivations behind our chosen corruption setting are as follows:
Our work aims address a comprehensive range of data corruption types. Consequently, conducting an independent analysis of each element's vulnerability can effectively high- light the disparities in their susceptibility.
When data saved into a,,,,,,)) for offline RL, each element in the tuple can be subject to corruption independently.
Our corruption method aligns with the concept of c-Contamination in offline RL theo- retical analysis (Zhang et al. (022). which considers arbitrary corruption the 4-tuples (8, s')..
Data corruption can potentially occur at any stage, such as storage, writing, reading, trans- mission, or processing of data. When data is loaded into memory, there is also possibility memory corruption and even intentional modifications by hackers. the case of TD- based offline RL methods, it is common practice load 4-tuples (s, a, s') into memory and each element has the potential to be corrupted independently.
In the future, exploring the effects of data corruption in scenarios considering the dependence be- tween states and next-states, or in the context of POMDP, holds promise for further research.
30

---

Published as a conference paper at ICLR 2024
1e3
Observation Attack
1e3
Action Attack
1e3
Reward Attack
1e3
Dynamics Attack
Average Return
2
2
2
Epochs
1e3
Epochs
1e3
Epochs
1e3
Epochs
1e3
BC
EDAC
MSG
CQL
IQL
RIQL
HalfChectah
1e3
Observation Attack
1e3
Action Attack
1e3
Reward Attack
1e3
Dynamics Attack
Average Return
2
2
2
Epochs
1e3
Epochs
1e3
Epochs
1e3
Epochs
1e3
BC
EDAC
MSG
CQL
IQL
RIQL
(b) Walker
1e3
Observation
1e3
1e3
Attack
1e3
Dynamics
Average Return
3
Epochs
1e3
Epachs
Epochs
Epochs
1e3
BC
EDAC
MSG
COL
IQL
RIQL
(c) Hopper
Figure 17: Learning curves under random data corruption on the "medium-replay" datasets.
31

---

Published as a conference paper at ICLR 2024
1e3 Observation Attack
1e3
Action Attack
1e3
Reward Attack
1e3
Dynamics Attack
Average Return
2
2
Epochs
1e3
Epochs
1e3
Epochs
1c3
Epochs
1e3
BC
EDAC
MSG
CQL
IQL
RIQL
HalfCheetah
1e3
Observation Attack
1e3
Action Attack
1e3
Reward Attack
1e3
Dynamics Attack
Average Return
2
2
Epochs
1e3
Epochs
1e3
Epochs
1e3
Epochs
1e3
BC
EDAC
MSG
CQL
IQL
RIQL
(b) Walker
1e3
1e3
1e3
1e3
Dynamics
Average Return
Epochs
1e3
Epachs
Epochs
Epochs
1e3
BC
EDAC
MSG
COL
IQL
RIQL
(c) Hopper
Figure 18: Learning curves under adversarial data corruption on the "medium-replay" datasets.
32


Reviews:
[{"review_id": "LKQgnkSU1C", "reviewer": "Reviewer_t4o4", "summary": "This paper concerns the robustness of different offline reinforcement learning methods to comprehensive data corruption, including states, actions, rewards, and dynamics. Their empirical results show that IQL method has better robustness to all types of data corruptions except noisy dynamic. The authors first give a theoretical explanation about IQL's robust performance, and according to empirical evidence, attribute this exceptional phenomenon to the heavy-tailed Q-targets. To verify this assumption and address this issue, this paper adopts observation normalization and Huber loss function for robust value function learning. Besides, this paper finds the potential negative value exploding issue of the clipped double Q-learning technique used in the original IQL and adopts quantile Q estimators instead of the LCB estimation. All the above modifications are verified to improve the performence of IQL under the data corruption of enviorment dynamic.", "questions": "In Fig.1 only two scales are illustrated, would you please make it more clear about the underlying mechanism that generates the corrupted data in each cases,  and how to ensure that the generated noisy data is realistic and representative in the sense that they do represent typical noisy data we encountered in practice? Intuitively, different level of noise would lead to different robustness outcomes of various algorithms - in this sense, how do you ensure that the empirical evidence is conclusive?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper provides comprehensive analysis and interesting findings on different types of data corruption in the offline RL;\n2. Sufficient empirical evidence on the possible reason of the failure setting, and efficient solutions to address these issues.", "weaknesses": "1. Lack theoretical evidence on heavy-tailed Q-target issue;\n2. Heavy work on parameter finetuning;", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m0mcgeOV3i", "reviewer": "Reviewer_41NY", "summary": "This work concentrates on dealing with diverse types of data corruption associated with state, action, reward, and transition kernel in offline RL history datasets. In particular, it first did data corruption test on some existing offline RL algorithms to show their vulnerable behaviors against data corruption. A theoretical result for the robustness of IQL has been shown w.r.t. the data corruption ratio. Then a robust variant of IQL has been proposed and outperform offline RL baselines, which consist of three key parts: state normalization, Huber loss, and $\\alpha$-quantile Q ensemble.", "questions": "1. Is there any study on ablation study for the proposed RIQL that is similar to Figure 1. It will be helpful to see how can RIQL handle different types of data corruption.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. It shows interesting testing of the effect of data corruption for current offline RL algorithms.\n2. Some theoretical relationship between the data corruption ratio and the performance of IQL is presented.\n3. A new robust IQL algorithm has been proposed and outperforms conducted baselines when data corruption appears.", "weaknesses": "1. Except for the traditional offline RL algorithms based on TD learning or the Bellman operator, there exists some other new baselines using a transformer or a diffusion model. It is helpful to involve the discussion about such algorithms at least.\n2. Since the new algorithm (RIQL) involves in ensemble, which is a very powerful trick in RL algorithms, it is better to add some baselines that also use ensemble Q, while not appearing in the experiments if I didn't miss something.\n3. There does not exist comparisons between the proposed algorithm RIQL to other existing robust RL algorithms, but only to non-robust counterparts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DOofHR1d7a", "reviewer": "Reviewer_dDco", "summary": "This paper proposes Robust IQL (RIQL), an offline RL algorithm that works decently even when the dataset is corrupted. It first experimentally and theoretically shows that IQL is robust to dataset noise. Adding upon the discovery, it presents three heuristics that can further improve IQL's performance on the corrupted dataset: (1) observation normalization, (2) Huber loss for value function learning, (3) using the $\\alpha$-quantile of an ensemble of Q functions as the target value. The authors conducted experiments on varying the degree and type of corruption and showed that RIQL outperforms other baselines in most of the settings.", "questions": "1. The main theorem holds for any algorithm that uses AWR to learn the optimal policy. Is it possible to add experimental results of other algorithms based on AWR?\n\n2. Does observation normalization also improve the performance of other algorithms?\n\n3. The sentence \"This is likely because normalization can help to ensure that the algorithm's performance is not unduly influenced by larger-scale features.\" from §4.1 is difficult to understand.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper tackles a novel problem where the offline dataset is corrupted. It provides an exact error bound of IQL under the data contamination setting supported by a mathematically sound argument. The paper also presents an interesting discovery that the Q target distribution has heavy tails and suggests a simple but clever solution, which is to use the Huber loss instead of an MSE loss. To prove the effectiveness of RIQL, multiple experiments were conducted, together with a thorough ablation study. Finally, the paper is overall well-written and easy to understand.", "weaknesses": "1. The paper provides no plausible scenarios where a malicious attack on the offline dataset would occur.\n\n2. The definition of $\\pi_{\\mathcal{D}}(a\\mid s)$ is unclear.\n\n3. The paper assumes that IQL can learn the optimal value function $V^*$ from the corrupted dataset without justification.\n\n4. The random corruption setting used in the experiments is a bit unrealistic.\n\n    * If environmental noise exists, the entire dataset would be corrupted, not just a tiny portion as assumed in the experiments.\n\n    * Most offline RL datasets are collections of trajectories not $(s, a, r, s')$ 4-tuples. Adding random noise just to $s$ or $s'$ does not seem to make much sense.\n\n5. The adversarial corruption setting is not really \"adversarial\" towards algorithms other than EDAC since the adversarial noise was computed via projected gradient descent with respect to the Q functions learned by EDAC. To see whether RIQL is robust to malicious attacks, the adversarial noise should be computed with respect to $Q_\\alpha$ learned by RIQL.\n\n6. The paper does not contain experiments for sequence-modeling-based algorithms such as Decision Transformer (Chen et al., 2021) or Diffuser (Janner et al., 2022). As the sequence modeling approach is one of the main branches of offline RL, I believe they should be included.\n\n### Minor comments\n\n1. §4.3 first paragraph \"This is evidenced in Figure 5(b), where the **penalty** for attacked data is larger...\" ⇒ The term \"Penalty\" is used before being defined.\n\n2. §C.1 Eq. (11)  third equality ⇒ swap $\\tilde{\\pi}_E$ and $\\pi_E$\n\n3. §C.1 \"and the last inequality is obtained by **Cauchy–Schwarz inequality**\" ⇒ Cauchy–Schwarz inequality → Hölder's inequality\n\n4.  §C.1 Eq. (15)  ⇒ $\\pi_\\mu(a\\mid s_i)$ → $\\pi_{\\mathcal{D}}(a\\mid s_i)$\n\n5. §C.1 Eq. (16)   ⇒ $\\zeta$ → $\\zeta_i$\n\n### References\n\n1. Chen, Lili, et al. \"Decision transformer: Reinforcement learning via sequence modeling.\" Advances in neural information processing systems 34 (2021): 15084-15097.\n\n2. Janner, Michael, et al. \"Planning with diffusion for flexible behavior synthesis.\" arXiv preprint arXiv:2205.09991 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7KJwbSg4M5", "reviewer": "Reviewer_wKTe", "summary": "In this paper, the authors investigate the robustness of offline RL algorithms when dealing with data corruption, including states, actions, rewards, and dynamics. \nThey then introduce Robust IQL (RIQL), an offline RL algorithm that enhances robustness through observation normalization, Huber loss, and quantile Q estimators. \nEmpirical evaluations demonstrate RIQL's exceptional resistance to various data corruption types, both random and adversarial. \nThis research sheds light on the vulnerability of current offline RL algorithms and provides a promising solution to enhance their robustness in real-world scenarios.", "questions": "- As I pointed above, Have the authors evaluated how RIQL performs in any of the environments I mentioned above?\n- Why hasn't the new model RIQL been tested on the COPA protocol in (Wu, 2022)?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ This study illuminates the susceptibility of existing offline RL algorithms while offering a promising approach to bolster their resilience in real-world environments. The proposed analysis is unquestionably relevant to the offline RL community. While there have been previous works (e.g., Zhang et al. (2022)) that offer theoretical analyses regarding the impact of using contaminated data in offline RL algorithms, this work appears to take it a step further by examining the impact of various forms of contamination (actions, states, etc.) and providing a highly detailed analysis for the offline IQL model. As a conclusion, it suggests a robust alternative for IQL, addressing how to mitigate its identified weaknesses.\n\n\n+ The theoretical and experimental analysis presented in Section 3 is elegantly and clearly articulated. The authors' discovery about the robustness of using a weighted imitation learning strategy is intriguing, but for the specific case of dynamics corruption, this doesn't seem to hold true. This observation leads them to propose up to three modifications to the IQL model to enhance its robustness. Each of these modifications (observation normalization, Huber loss, and quantile Q estimators) is well-founded and justified.\n\n+ The experimental evaluation is comprehensive and very detailed. First, all state-of-the-art models in offline-RL are assessed to see how they perform in perturbation scenarios. Once the analysis is completed, the article focuses on improvements for the IQL model. The environments used for the experiments are well-known within the community (Halfcheetah, Halfcheetah, Halfcheetah). The results presented are conclusive. I also appreciate the ablation study in section 5.3, which allows the reader to understand the impact of the modifications applied to IQL until achieving the robust version (RIQL).", "weaknesses": "- Some particularly interesting, and I would say more challenging, environments have been left out of the experimental analysis, especially in terms of the potential impact of perturbations, and in which the original IQL model was tested. It would be interesting to hear the authors' opinion on this. I'm referring to the following environments: locomotion-v2, antmaze-xx, kitchen, adroit.\n\n- Somewhat, the theoretical analysis and the experimental evidence detailed in the section appear contradictory. The paper explicitly states this as follows: \"Our theoretical analysis suggests that the adoption of weighted imitation learning inherently offers robustness under data corruption. However, empirical results indicate that IQL still remains susceptible to dynamics attacks\".  This contradiction is not further examined in the manuscript. On the contrary, what is proposed is to address IQL's issue with dynamic attacks.\n\n- Previous works on certification protocols for offline RL against attacks has not been considered for the novel RIQL model proposed. This is an important weaknesses that should be addressed in the rebuttal. \n\n- Minor comments:\n(Wu, 2022) Update the reference as it is not anymore an ArXiv manuscript but an ICLR22 paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)