Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptions
Authors: Rui Yang
Jie Wang
Guoping Wu
Bin Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Robust Offline Reinforcement Learning
Variational Bayesian Inference
Diverse Data Corruptions
Uncertainty
Primary area: reinforcement_learning

Abstract: Real-world offline datasets are often subject to data corruptions (such as noise or adversarial attacks) due to sensor failures or malicious attacks. Despite advances in robust offline reinforcement learning (RL), existing methods struggle to learn robust agents under high uncertainty caused by the diverse corrupted data (i.e., corrupted states, actions, rewards, and dynamics), leading to performance degradation in clean environments. To tackle this problem, we propose a novel robust variational Bayesian inference for offline RL (TRACER). It introduces Bayesian inference for the first time to capture the uncertainty via offline data for robustness against all types of data corruptions. Specifically, TRACER first models all corruptions as the uncertainty in the action-value function. Then, to capture such uncertainty, it uses all offline data as the observations to approximate the posterior distribution of the action-value function under a Bayesian inference framework. An appealing feature of TRACER is that it can distinguish corrupted data from clean data using an entropy-based uncertainty measure, since corrupted data often induces higher uncertainty and entropy. Based on the aforementioned measure, TRACER can regulate the loss associated with corrupted data to reduce its influence, thereby enhancing robustness and performance in clean environments. Experiments demonstrate that TRACER significantly outperforms several state-of-the-art approaches across both individual and simultaneous data corruptions.

Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptions
Rui Yang! 1.2, Jie Wangl.2, Guoping Wu', Bin Li
UUniversity of Science and Technology of China MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition fyr0013, guoping) ustc. edu. cn {jiewangz, binli)@uster edu.
Abstract
Real-world offline datasets are often subject to data corruptions (such as noise or adversarial attacks) due sensor failures or malicious attacks. Despite advances in robust offline reinforcement learning (RL), existing methods struggle to learn robust agents under high uncertainty caused by the diverse corrupted data (i.e., corrupted states, actions, rewards, and dynamics), leading to performance degradation clean environments. To tackle this problem. we propose novel robust variational Bayesian inference for offline RL (TRACER). It introduces Bayesian inference for the first time capture the uncertainty via offline data for robustness against all types of data corruptions. Specifically, TRACER first models all corruptions the uncertainty the action-value function. Then, capture such uncertainty, it uses all offline data as the observations to approximate the posterior distribution of the action-van value function under a Bayesian inference framework. An appealing feature of TRACER is that it can distinguish corrupted data from clean data using an entropy-bast based uncertainty measure, since corrupted data often induces higher uncertainty and entropy. Based the aforementioned measure, TRACER can regulate the loss associated with corrupted data to reduce its influence, thereby enhancing robustness and performance in clean environments. Experiments demon- strate that TRACER significantly outperforms several state-of-the-art approaches across both individual and simultaneous data corruptions.
Introduction
Offline reinforcement learning (RL) aims to learn an effective policy from fixed dataset without direct interaction with the environment 2 This paradigm has recently attracted much attention in scenarios where real-time data collection is expensive, risky, impractical, such as in healthcare (3]. autonomous driving [4. and industrial automation [5]. Due to the restriction of the dataset, offline RI. confronts the challenge of distribution shift between the policy represented in the offline dataset and the policy being learned, which often leads to the overestimation for -uttr--distribution (OOD) actions [I 6 7 To address this challenge, estimation techniques, such as using the ensemble of action-value functions or Bayesian inference to measure the uncertainty of the dynamics model [8-I or the action-value function [2 regarding the rewards and transition dynamics. Therefore, they can constrain the learned policy to remain close the policy represented in the dataset, guiding the policy to be robust against OOD actions.
of the promising approaches introduce uncertainty
Corresponding author. Email: jiewangx wwwraanx@uutc.edu.... eus.c.edu.cn.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Figure 1: Graphical model of pocision-making process. Nodes connected by solid lines denote data points in the offline dataset, while the Q values (i.e., action values) connected by dashed lines are not part of the dataset. These Q values are often objectives that offline algorithms aim to approximate.
Nevertheless, in the real world, the dataset collected by sensors or humans may be subject to extensive and diverse corruptions [16 e.g., random noise from sensor failures or adversarial attacks during RLHF data collection. Offline RL methods often assume that the dataset is clean and representative of the environment. Thus, when the data is corrupted, the methods experience performance degradation in the clean environment, as they often constrain policies close to the corrupted data distribution.
Despite advances in robust offline RL []. these approaches struggle address the challenges posed by diverse data corruptions [18). Specifically, many previous methods on robust offline RL aim to enhance the testing-time robustness, learning from clean datasets and defending against attacks during testing [1) However, they cannot exhibit robust performance using offline dataset with perturbations while evaluating the agent in a clean environment. Some related works for data corruptions (also known as corruption-robus offline RL methods) introduce statistical robustness and stability certification improve performance, but they primarily focus on enhancing robustness against adversarial attacks [16 22 [3] Other approaches focus on the robustness against both random states, rewards, or transition dynamics (24 17 Based on these methods, recent work (18) extends the data corruptions to all four elements in the dataset, including states, actions, rewards, and dynamics. This work demonstrates the superiority of the supervised policy learning scheme [6) for the data corruption of each element in the dataset. However, as it does not take into account the uncertainty in decision- making caused by the simultaneous presence of diverse corrupted data, this work still encounters difficulties in learning robust agents, limiting its applications real-world scenarios.
noise and adversarial attacks, but they often aim to address only corruptions
In this paper, we propose to use offline data as the observations, thus leveraging their correlations to capture the uncertainty induced by all corrupted data. Considering that (1) diverse corruptions may introduce uncertainties into all elements in the offline dataset, and (2) each element is correlated with the action values (see dashed lines Figure] T]. there high uncertainty approximating the action-vall value function by using various corrupted data. To address this high uncertainty. propose to leverage all elements in the dataset as observations, based on the graphical model in Figure I By using the high correlations between these observations and the action values 27]. identify the uncertainty of the action-value function.
accurately
Motivated by this idea, we propose a robust variational Bayesian inference for offline RL (TRACER) to capture the uncertainty via offline data against all types of data corruptions. Specifically, TRACER first models all data corruptions such uncertainty, it introduces variational Bayesian inference (28), which uses all offline data as observations to approximate the posterior distribution of the action-value function. Moreover, the corrupted observed data often induce higher uncertainty than clean data, resulting in higher entropy uncertainty
uncertainty in the action-value function. Then, to capture
in the distribution of action-value function. Thus, TRACER can the entropy measure to effectively distinguish corrupted data from clean data. Based on the entropy- based uncertainty measure, it can regulate the loss associated with corrupted data in approximating the action-value distribution. This approach effectively reduces the influence of corrupted samples, enhancing robustness and performance in clean environments.
This study introduces Bayesian inference into offline RL for data corruptions. It significantly captures the uncertainty caused by diverse corrupted data, thereby improving both robustness and performance in offline RI. Moreover, it is important to note that, unlike traditional Bayesian online and offline RI. methods that only model uncertainty from rewards and dynamics (2 35), our approach identifies the

---

uncertainty of the action-valus function regarding states, actions, rewards, and dynamics under data corruptions. We summarize our contributions as follows.
To the best of our knowledge, this study introduces Bayesian inference into corruption-robust offline RL for the first time. By leveraging all offline data as observations, it can capture uncertainty the action-value function caused by diverse corrupted data.
By introducing an entropy-bane -based uncertainty measure, TRACER can distinguish corrupted from clean data, thereby regulating the loss associated with corrupted samples to reduce its influence for robustness.
Experiment results show that TRACER significantly outperforms several state-of-the offline RL methods across range of both individual and simultaneous data corruptions.
2 Preliminaries
Bayesian RL. We consider a Markov decision process (MDP), denoted by tuple M (S. R. P. Po.7), where S is the state space, A is the action space, R. is the reward space, a() E (S)) the transition probability distribution over next states conditioned on a state- action pair (s, a). Po(+) E P(S) is the probability distribution of initial states, and 7 E [0, 1) is the discount factor. Note that P(S) and P(A) denote the sets of probability distributions on subsets of S and A, respectively. For simplicity, throughout the paper, we use uppercase letters to refer to random variables and lowercase letters to denote values taken by the random variables. Specifically, R(s, a) denotes the random variable of one-step reward following the distribution p(r(s, a), and r(s, a) represents a value of this random variable. We assume that the random variable of one-step rewards and their expectations are bounded by Rima and max for any (s, a) S x A, respectivcly.
Our goal is to learn a policy that maximizes the expected discounted cumulative return:
R(s ai)
arg max (s) (e. ((1ss,,)) (AA)
0c Based on the return, can define the value function V = E Ex the action-value function as (=(saa) a) (e.) [R(s, a) value distribution (36]
E 7 *((s St: uz) s).
+ 7
(s')),
and the action-
(= = of R(s =,,00= a), with 8141 P(-|st, at), a+t1 ~ (-118111)- (1)
Note that (*(x)= Easn (x, = Ea (s,aa)
Variational Inference. Variational inference is a powerful method for approximating complex posterior distributions, which is effective for RL to handle the parameter uncertainty and deal with modelling errors (36]. Given an observation X and latent variables Z, Bayesian inference aims to compute the posterior distribution p( p(( Direct computation of this posterior is often intractable due to the high- -igh-dimensional integrals involved. To approximate the true posterior, Bayesian inference introduces a parameterized distribution (Z; ø) and minimizes the Kullback- -uibback-Leibler (KL) divergence ( Z; (x))). Note that minimizing the KL divergence is equivalent maximizing the EL)O(Ø) = (Z;))) [log p(X, Z) log q Z: Φ)].
evidence lower bound (ELBO) 37 38
Offline RL under Diverse Data Corruptions. In the real world, the data collected by sensors or humans may be subject to diverse corruption due to sensor failures or malicious attacks. Let b and B denotes the uncorrupted and corrupted dataset with samples {(5{, of rf. 1) respectively. Each data in B may be corrupted. We assume that an uncorrupted state follows a state distribution pb(-), corrupted state follows (-).). an uncorrupted action follows a behavior policy Tb (-((ss)), corrupted action is sampled from TB( (ra(116 (-]sf), a corrupted reward is sampled from p=s( af), and corrupted next state is drawn from Pa[-Is (o(lst,,t)) We also denote the uncorrupted and corrupted empirical state-action distributions as (s(s), at) and pa(sf, af), respectively. Moreover, we introduce the notations [18] 39) as follows.
()) = f(s,a) + E. Pa((sa)) [(s').. F(s,a) = Er-pe((sa))())
(2)
(s( _ R(s,a) a) + +s''s',)) s'~ Po(-ls, a), a'~ xB(+)s),
(3)
3

---

for any (s,a) a) E S A and Q: A 10, Tmax/ / ?)], where y denotes equality of probability laws, that is the random variable X is distributed according to the same law as Y.
To address the diverse data corruptions, based on IQL (26), RIQL [8]] introduces quantile estimators
with an ensemble of action-value functions {Qo, (s, a) } and employs a Huber regression [40]:
1 x²
if
æ S K
LQ (0:) = ((eenn)))) 1 (r + 7V (s') Q0, a))], (x) = ||| 1
2N
(4)
if c > K
(v (s.0)= (4% (Q.
(s)) (z(z)= = I(x 0)1 x²
(5)
Note that 12, is the Huber loss, and Qa is the a-quantile value among (Qu, (s. a)) RIQL then follows IQL [26] to learn the policy using weighted imitation learning with a hyperparameter B::
n (0) (s [exp( (s, log [a(ass), Aa(s, = (s. a) Vv(+).
(6)
3 Algorithm
We first introduce the Bayesian inference for capturing the uncertainty caused by diverse corrupted data in Section 3.1 Then, we provide our algorithm TRACER with the entropy- uncertainty measure in Section 3.2 Moreover, we provide the theoretical analysis for robustness, the architecture,
and the detailed implementation of TRACER in Appendices A.I B.I and B.2 respectively.
3.1 Variational Inference for Uncertainty induced by Corrupted Data
We focus on corruption-robust offline RL to learn an agent under diverse data corruptions, i.e., random or adversarial attacks on four elements of the dataset. We propose to use all elements as observations, leveraging the data correlations to simultaneously address the uncertainty. By introducing Bayesian inference framework, our aim is to approximate the posterior distribution of the action-value function.
At the beginning, based on the relationships between the action values and the four elements (i.e., states, actions, rewards, next states) in the offline dataset as shown in Figure [ we define Da Do(S, A, R)) (-1S,, A, R), parameterized by O. Building the action-value distribution, we can explore how to estimate the posterior of Do using the elements available in the offline data.
Firstly, start from the actions { following the corrupted distribution Ty and use them as observations to approximate the posterior of the action-value distribution under a variational inference. As the actions are correlated with the action values and all other elements in the dataset, the likelihood is Pp.. (A) D. S, R, S'). parameterized by fa- Then, under the variational inference framework, we maximize the posterior and derive minimize the loss function based on ELBO:
A Pa) = EB.ps [DKa. (p.p. Do, S, R,S') (A(S)) - EAmp,, (log po (D]s, A, R)]]
(7)
where S, R, and S" follow the offline data distributions ps: PS. and Ps, respectively.
Secondly, we apply the rewards {ri) drawn from the corrupted reward distribution Ps as the observations. Considering that the rewards related the states, actions, and action values,
model the likelihood as Pv., (R D. S, A), parameterized by fr. Therefore, we can derive loss function by following Equation 7
( pr) = -E,,0 [DEL (pe. (R) S, Ps (R), ERope, (Do 8
where S and A follow the offline data distributions P5 and TB, respectively.
Finally, we employ the state {sf i= in the offline dataset following the corrupted distribution PB as the observations. Due to the relation of the states, we can model the likelihood as Pv. (S) A, R). parameterized by kpr We then have the loss function:
(e, pp.) Es,po DKL (pp. (S) A, R) || PB(()) Es-pp. [log po (DoSS, A, R)]]
(9)
where A and R follow the offline data distributions Ty and PB. respectively. We present the detailed derivation process in Appendix A.2
The goal of first terms in Equations
and 9 is to estimate TH( (A()) Ps R A), and ps (S) using P.., (A Do. S, R, S'), Pe. (R) Do S. A), and P.. (S) A, R), respectively. As we do not

---

have the explicit expression of distributions TB. PB. and Ps, we cannot directly compute the KL divergence in these first terms. To address this issue, based on the generalized Bayesian inference [4T] we can exchange two distributions in the KI. divergence. Then, we model all the aformentioned distributions as Gaussian distributions, and use the mean 14p and standard deviation Enp to represent the corresponding Pp- For implementation, we directly employ MLPs to output each (M4p Exp) using the corresponding conditions of pv- Then, based on the KL divergence between two Gaussian distributions, we can derive the loss function as follows.
1
Corst(0, Poi ias (r))
2 BBsaar))BBD - Borpp 1(pp - -a)" 20)2ipp po (up. - a) + (Mip - r)² 21)"2/pp00 (Me. - -)
2
(Hpp, +1pp, 8)" (Hip, 5) log [Dppll Exp.l
-E
(10)
Moreover, the goal of second terms in Equations (8) and (9) is to maximize the likelihoods of Do given samples á ~ P.p.. P.... or Pp.. Thus, with (s, a, r) B, we propose minimizing the distance between (ø(5, r) and D(s,a,1) D(s, a, r), where ~ pep,, ~ P and f ~ Pe.. Then, based on [41], we can derive the following loss with any metric l to maximize the log probabilities:
(s à, r) and ((s, a, r), and Dø f) and
Leccond (0, Pas Pas (r)
P.s..ar))BB.sppp. d dr toPer Drop
+ (D((s, a + ((D(s,,,,)) Do(6, (( (11)
3.2 Corruption-Robut Algorithm with the Entropy-based Uncertainty Measure
We focus on developing tractable loss functions for implementation in this subsection.
Learning the Action- Value Distribution based on Temporal Difference (TD). Based on 42 43
we introduce the quantile regression 44 to approximate the action-value distribution in the offine dataset B using ensemble model (De; Ie We use Equation (4) to derive the loss
N
N'
)
LD (0.) -Eaa,,
Σ
Σ
Pr
00
8T = (6') Do, (s,
(6)--D;
N
(12)
where 13 (6) 11 < 031 2/ (6) with the threshold k, Z denotes the value distribution, %% is the sampled TD error based on the parameters 0,, 7 and 7' are two samples drawn from a uniform distribution ([0, 11). (s, a, r) == FD (T) is the sample drawn from po(-|s, a, r). (T(s) :=
2(0) (T) is sampled from p(-1s). Fx (r) is the inverse cumulative distribution function (also known as quantile function) [45 at T for the random variable X, and N and N' represent the respective number of iid samples and r'. Notably, based on 43 we have Qo, (s, a) >m D5; (s, r,
In addition, if learn the value distribution Z, the action-value distribution can extract the informa- tion from the next states based on Equation 12) which is effective for capturing the uncertainty. On the contrary. if we directly use the next states the offline dataset as the observations, in practice. the parameterized model of the action-value distribution needs to take (s, a, r, s', a', r', s") as the input data. Thus, the model can compute the action values and values for the sampled TD error in Equation (22). To avoid the changes in the input data caused by directly using next states as observations Bayesian inference, we draw inspiration from IQL and RIQL to learn a parameterized we derive a new objective as:
value distribution. Based on Equations 5 and (2),
Σ 2 (s, Z 23 (o)) n=
(z( = E(
(13)
where DJ is the a-quantile value among (DT, (s, H) and (s(s)) = EN (s). (9)...M, More details are shown in Appendix(B.. Furthermore, we provide the theoretical analysis in Appendix A. to give value bound between the value distributions under clean and corrupted data.
Updating the Action- Value Distribution based on Variational Inference for Robustness. We discuss the detailed implementation of Equations (0) and based on Equations
12 and 13) As the data
5

---

corruptions may introduce heavy-tailed targets [8] we apply the Huber loss to replace all quadratic loss Equation (TO) and the metric in Equation (T. mitigating the issue caused by heavy-tailes targets [6 for robustness. We rewrite Equation as follows.
(sccona(e, ((s.a.)) Saper Drmu
Σ H (s, (s,,,,,
(s,ū,r)
(s, (B), (s, (s, + 1 DT (s,, D D (3. (14)
Thus, we have the whole loss function CDis.A.R (ora (Po, Pax (pr) + Esccond (0s: Pax Pa: (P)) in the generalized variational inference framework. Moreover, based on the assumption of heavy-tailed noise in [8] we have a upper bound of action- value distribution by using the Huber regression loss.
Entropy-based Uncertainty Measure for Regulating the Loss associated with Corrupted Data. To further address the challenge posed by diverse data corruptions, we consider the problem: how to exploit uncertainty to further enhance robustness?
Considering that our goal is to improve performance in clean environments, we propose to reduce the influence of corrupted data, focusing on using clean data to learn agents. Therefore, we provide a two-step plan: (1) distinguishing corrupted data from clean data: (2) regulating the loss associated with corrupted data to reduce its influence, thus enhancing the performance in clean environments.
For (1), as the Shannon entropy for the measures of aleatoric and epistemic uncertainties provides important insight [4), and the corrupted data often results in higher uncertainty and entropy of the action-value distribution than the clean data, we use entropy [50] to quantify uncertainties of corrupted and clean data. Furthermore, by considering that the exponential function can amplify the numerical difference in entropy between corrupted and clean data, we propose the use of exponential metric of extent a distribution-tor design our uncertainty measure.
entropy 551
Specifically, based on Equation 12 we can use the quantile points {Tn [Tn]]r to learn the corresponding quantile function values (DTT } drawn from the action-value distribution pe. We sort the quantile points and their corresponding function values in ascending order based on the values. Thus, we have the sorted sets (sn N (D Jm and the estimated PDF values [5m] 1 where 51 Si and
5n Sn-1 for 1 < I N. Then, we can further estimate differential entropy following [52 (see Appendix(A.3 for detailed derivation).
N Σ ==1
H(po, =
Sn DD (s, a, log Sn,
(15)
where &n denotes (In-1 for 1 < n S N, and D denotes D D5 for 1 < n S N.
For (2), TRACER employs the reciprocal value of exponential entropy 1/ exp(H((pe))) )) to weight the corresponding loss of 0, in our proposed whole loss function LDIS.A,R. Therefore, during the learning process, TRACER can regulate the loss associated with corrupted data and focus minimizing the loss associated with clean data, enhancing robustness and performance in clean environments. Note that we normalize entropy values by dividing the mean of samples (i.e., quantile function values) drawn from action-value distributions for each batch. Figure 3 we show the relationship of entropy values of corrupted and clean data estimated by Equation (15) during the learning process. The results illustrate the effectiveness of the entropy- weighted technique for data corruptions.
Updating the Policy based on the Action-Valle Distribution. We directly applies the weighted imitation learning technique in Equation (6) to learn the policy. As Qa a) the a-quantile value
among (Qo,(s, (s, [))K = {IN D (s. a.r)) (s.
K
and (s(s)= = EN Z (s). we have
N Cx(0) = Biaaay exp B. (s, - -3. Σ n=1
2 (x)( -log loggo(als))
(16)

---

Table 1: Average scores and standard errors under random and adversarial simultancous corruptions.
Eny
Corrupt
BC
EDAC
MSG
UWMSG
CQL
IQL
RIQL
TRACER (ouns)
Hallcheetah
midoo
23.17
0.43
1,70 + 0.50
9.57 3.44
8.31 1.25
14,25
1.39
24.82
0.57
20.94
1.00
33.04 + 0.42
advers
16.37
0.32
0.50 0.30
3.60 0.89
3.13 0.85
5.61 2:21
11.06
17.86 2: 1.30
19.72 +: 2.80
Walker2d
rundom
13.77
1.06
-0.13 6.01
-0.15 0.11
4.36 1.55
0.63 - 0.36
12.35 2.03
17.42
23.62 2.33
advers
8.75
0.33
000
3.77
2.82
4.23 1.35
16.61 2.73
9.20
1.40
Hopper
random
18.49
0.52
0.50 0.01
15.84
2.47
12:22 1. 2.11
3.16 1.07
25.28
15.34
22.50
10.01
28.83
7.06
advers
17.34 + 1.00
0.80 + 0.01
12.14 + 0.71
10:43 + 0.94
0.10 0.34
19.56 = 1.08
24.71
6.20
24.80 + 7.14
Average soore
13.98
0.65
7.53
7.11
4.86
18.28
20.27
24.54
Table 2: Average score under diverse random corruptions.
Env
Comupted Element
BC
EDAC
MSG
UWMSG
CQL
IQL
RIQL
TRACER (ouns)
observation
$3.4 1:
0.5
-0.2
29 0.1
9.0 = 7.5
21.4
27.3
34.2 1.
Halfcheetah
+
4..4
+-.6
17.5 164
5.0++09
35.6+00
0.4
32.6
19.9 - 21.3
1.9
40.0+
dynamics
25.8 0.9
1.5 0.2
04
= 0.1
20.2 4.0
36.7 1.8
43.1
43.8 +
observation
2.5
0.2 0.3
0.4 0.1
6.2 0.5
19.4 1.6
27.2
29.4
32.7 + 2.8
Walker
action
18.1 2.1
83.2 1.9
25.3 10.6
1844+99.
31.3 10.6
3.7
69.4 7.4
62.7 7.2
71.3
T..3 7.8
8.4
$4.6 3.3
N6.7 1: 6.2
dynamics
0.2
observation
21.5
1.0 + 0.5
= DJD
12.8=0044
42.8 7.0
52.0 16.6
62.4
62.7 +
Hopper
action
22.8 a: 7.0
100.8 +: 0.5
37.6 6.3
53.4 d 5.4
09.8 d: 4.5
76.3 2: 15.4
90.0 5.6
92.8 4: 2.5
19.5
0.7
219 4.3
2.5
8.9
09.7 18.8
81. 13.1
85.7 1.
dynaniics
19.5
Avenae score
23.7
23.5
17.0
27.5
35.5
43.6
60.0
Table 3: Average score under diverse adversarial corruptions.
Fav
Compptd Flemen
BC
EDAC
MSG
UWMSG
CQI
IQ1.
RIQU
TRACER (ours)
observation
34.5 1 1.5
1.1 0.3+
1.1+000 0.2
1.9 0.1
5. 11.6
32.6 2.7
35.7
36.8 + 2.1
Halfeheetah
action
0.77
37.3
1.0
1.2
33.4 +
dynamics
0.9
-133..1
1.55 0.0
5.0 2.2
-1. 0.0
26.7 11
35.8
36.2 1:
observation
5.9
0.01
2.
6.7
61.8
37.7 1: 13.0
70.0
70.0 1: 6.7
Walker2d
action
0.
41.9 24.0
3.4 0.3
5.0 +0-+00 0.4
27.0 + 7.5
27.5 + 0.6
66.1 + 4.6
69.3 + 4.6
16.0
S7.3 33.2
35.1 10.5
4.9
1.5
88.9
dynamics
observation
21.8
36.2
16.2
16.0 2.8
15.0 2: 1.3
78.0
32.8
50.8
64.5
action
15.5.12.2
25.7
1 3.8
23.0 2.1
27.7 1.3
32.2 7.6
37.9
63.8 7.5
67.2 1. 3.8
Hopper
roward
1.4
21.2 1.9
22.6 2.8
$0.5 6.2
69.8 12.3
57.5 9.7
65.8 9.8
66.3 1.
19.5
0.6:000
Average score
20.5
21.7
13.7
17.5
26.6
33.1
56.2
4 Experiments
In this section, show the effectiveness of TRACER across various simulation tasks using diverse corrupted offline datasets. Firstly, we provide our experiment setting, focusing on the corruption settings for offline datasets. Then, we illustrate how TRACER significantly outperforms previous state- of-the-art approaches under a range of both individual and simultaneous data corruptions. Finally, we conduct validation
experimentt experiments and ablation studies to show the effectiveness of TRACER.
4.1 Experiment Setting
Building upon RIQL [18. we use two hyperparameters, i.c., corruption rate c e [0, 1] and corruption scale e, to control the corruption level. Then, we introduce the random corruption and adversarial corruption in four elements (i.e., states, actions, rewards, next states) of offline datasets. The implementation of random corruption is to add random noise to elements of portion of the offline datasets, and the implementation of adversarial corruption follows the Projected Gradient Descent attack [53] 54] using pretrained value functions. Note that unlike other adversarial corruptions, the adversarial reward corruption multiplies -e to the clean rewards instead of using gradient optimization. We also introduce the random adversarial simultaneous corruption, which refers to random or adversarial corruption simultaneously present four elements of the offline datasets. We apply the corruption rate 0.3 and corruption scale 1.0 in experiments.
7

---

CARLA- Lane-v0
Hopper-Medium Medium- Medium-Replaye Replay-v2
80
LQL
RIQU
RIQL TRACER
70
Average Score
20
Average Score
15-
40
10
30
5-
10
0.22, -1.000000000 Corrupt Level
Random Simultaneous Coruuptions
Figure In the left, we report the means and standard deviations on CARLA under random simultancous corruptions. In the right, we report the results with random simultaneous corruptions against different corruption levels.
We conduct experiments on D4RL benchmark [55]. Referring to RIQL, train all agents for 3000 epochs on the 'medium-replay-v dataset, which closely mirrors real-world applications as it is collected during the training of SAC agent. Then, we evaluate agents in clean environments, reporting the average normalized performance over four random seeds. See Appendix C for detailed information. The algorithms compare include: (1) CQL 77 and IQL [26). offline RL algorithms using a twin Q networks. (2) EDAC [56] and MSG [57], offline RL algorithms using ensemble Q net-
works (number of ensembles 2). (3) UWMSG [17 and RIQI,, state-of-the-arr in comuption-robust offline RI. Note that EDAC, MSG, and UWMSG are all uncertainty-bant bassdd offline RI. algorithms.
4.2
Main results under Diverse Data Corruptions
We conduct experiments on MuJoCo 58
(see Tables and which highlight the highest results) and CARLA [59] (see the left of Figure tasks from D4RL under diverse corupptions to show the superiority of TRACER. In Table I we report all results under random or adversarial simultaneous data corruptions. These results show that TRACER significantly outperforms other algorithms in all tasks, achieving an average score improvement of +21.1 In the left of Figure 2 results on "CARLA-lane v0' under random simultaneous corruptions also illustrate the superiority of TRACER. See Appendix C.2 for details.
Random Corruptions. We report the results under random simultaneous data corruptions of all algorithms in TT. Such results demonstrate that TRACER achieves average score gain of +22.4% under the setting of random simultaneous corruptions. Based on the results, it clear that many offline RL algorithms, such as EDAC, MSG, and CQL, suffer the performance degradation under data corruptions. Since UWMSG is designed to defend the corruptions in rewards and dynamics, its performance degrades when faced with the stronger random simultaneous corruption. Moreover, we report results across range of individual random data corruptions in Table 2 where TRACER outperforms previous algorithms out of 12 settings. We then explore hyperparameter tuning on Hopper task and further improve TRACER's results, demonstrating its potential for performance gains. We provide details in Appendix C33.1
Adversarial Corruptions. We construct experiments under adversarial simultaneous corruptions to evaluate the robustness of TRACER. The results in Table [ show that TRACER surpasses others by significant margin, achieving an average score improvement of +19.3%. In these simultaneous corruption, many algorithms experience more severe performance degradation compared to the random simultaneous corruption, which indicates that adversarial attacks are more damaging to the reliability of algorithms than random noise. Despite these challenges, TRACER consistently achieves significant performance gains over other methods. Moreover, we provide the results across range of individual adversarial data corruptions in Table 3 where TRACER outperforms previous algorithms in out of settings. We also explore hyperparameter tuning on Hopper task and further improve TRACER's 's results, demonstrating its potential for performance gains. See Appendix C.3.1 for details.

---

Walker2d Medium Replay-v2
Eatro of Corrupt Clean (%)
Adversarial Simulumeeus Corruptions
Simultaneuus Corruptions
3
TRACUR Using Lhne
Eatro Corrupt Clean
2 4
TRACER Usin Entro
Averggo Scare
TRACER wing Eato
TRACER ming Ento
20
-2
-2
TRACER Using Enro
TRACER usig Eatr
Adversriial
Random
Mixed Compptions
Walker2d- Epoch
WalkerZd Epoch
Halfchoctah--ee Meei--MMeiiiue-eeeeeee
Entro of Corrupe Cleam (%)
Adversarial Simultaan cous Comuptions
Enntro of Corrupe > Clean (%)
Random Simultaneous Compttions
TRACER Uning Engo
TRACER Uning Enuo
Average Soore
d
nsing Latro
usng
-1-
12
TRACER Using Extro
TRACER NOT using
Adversoial
Kandom
316
Mixed Cormuptions
Halfchectah
Epoch
Halfcheetah
Epech
Figure 3: In the first column, we report the and standard deviation show the superiority of using the entropy-based uncertainty measure. In the second and third columns, we report the results over three seeds to show the higher entropy of corrupted data compared clean data during training.
4.3 Evaluation of TRACE under Various Corruption Levels
Building upon RIQL, we further extend our experiments to include Mujoco datasets with various corruption levels, using different corruption rates and scales e. We report the average scores and using batch sizes of 256.
standard deviations over four random seeds in the right of Figure 2
Results in the right of Figure 2 demonstrate that TRACER significantly outperforms baseline algo- rithms in all tasks under random simultaneous corruptions with various corruption levels. It achieves an average score improvement of +3.66% Moreover, as the corruption levels increase, the slight decrease in TRACER'S 's results indicates that while TRACER is robust to simultaneous corruptions, its performance depends on the extent of corrupted data it encounters. We also evaluate TRACER in different scales of corrupted data and provide the results in Appendix C.4.1
4.4 Evaluation of the Entropy-based Uncertainty Measure
We evaluate the entropy-based uncertainty measure in action-value distributions show: (1) is the uncertainty caused by corrupted data higher than that of clean data? (2) is regulating the loss associated with corrupted data effective in improving performance?
For (1), we first introduce labels indicating whether the data is corrupted. Importantly, these labels not used by agents during the training process. Then, estimate entropy values of labelled corrupted and clean data in each batch based on Equation (5)). Thus, we can compare entropy values to compute results, showing how many times the entropy of the corrupted data is higher than that of clean data. Specifically, we evaluate the accuracy every 50 epochs over 3000 epochs. For each evaluation, we sample 500 batches to compute the average entropy of corrupted and clean data. Each batch consists of 32 clean and 32 corrupted data. We illustrate the curves over three seeds in the second and third columns of Figure where each point shows how many of the 500 batches have higher entropy for corrupted data than that of clean data.
Figure 3 indicates an oscillating upward trend of TRACER's measurement accuracy using entropy (TRACER Using Entro) under simultancous corruptions, demonstrating that using the entropy-based uncertainty measure can effectively distinguish corrupted data from clean data. These curves also reveal that even in the absence of any constraints on entropy (TRACER NOT using Entro), the entropy associated with corrupted data tends to exceed that of clean data.
For (2), the first column of Figure these results demonstrate that TRACER using the entropy- based uncertainty measure can effectively reduce the influence of corrupted data, thereby enhancing robustness and performance against all corruptions. We provide detailed information for this evalua- tion in Appendix C.4.2

---

5
Related Work
Robust RL. Robust RL can be categorized into two types: testing-time robust RL and training-time robust RL. Testing-time robust RL 119 20 refers to training a policy on clean data and ensuring its robustness by testing in an environment with random noise or adversarial attacks. Training-time robust RL [T. 17 aims to learn a robust policy in the presence of random noise or adversarial attacks during training and evaluate the policy in clean environment. In this paper, we focus on training- time robust RL under the offline setting. where the offline training data is subject to various data corruptions, also known as corruption robust offline RL.
Corruption- RI. Some theoretical work on corruption- online RL 60 63] aims to
analyze the sub-optimal bounds of learned policies under data corruptions. However, these studies primarily address simple bandits or tabular MDPs and focus on the reward corruption. Some further work [64 65] extends the modeling problem more general MDPs and begins investigate the corruption in transition dynamics.
It is worth noting that corruption- -oouuttttoooortttttt robust offline RL has not been widely studied. UWMSG [17] designs a value-based
eeeertainty-eighhtie technique, thus using the weight to mitigate the impact of corrupted data. RIQI. [18 further extends the data corruptions to all four elements in the offline dataset, including states, actions, rewards, and next states (dynamics). It then introduces quantile estimators with an ensemble of action-value functions and employs Huber regression based on IQL [26). alleviating the performance degradation caused by corrupted data.
Bayesian RL. Bayesian RL integrates the Bayesian inference with RL to create framework for decision decision-making under uncertainty (28). It is important to highlight that Bayesian RL is divided into two categories for different uncertainties: the parameter uncertainty in the learning of models 66 67
and the inherent uncertainty from the data/cnvironment in the distribution over returns 68 69 In this paper, focus capturing the latter.
explicitly model the transition dynamics and using Bayesian inference to update the model. It is useful when dealing with complex environments for sample efficiency.
For the latter uncertainty, in model-based Bayesian RL, many approaches 68
model-free Bayesian RL, value-based methods (69, (2) use the reward information to construct the posterior distribution of the action-value function. Besides, policy gradient methods [73] 74 use information of the return to construct the posterior distribution of the policy. They directly apply Bayesian inference to the value function or policy without explicitly modeling transition dynamics.
Offline Bayesian RL. offline Bayesian RL integrates Bayesian inference with offline RL to tackle the challenges of learning robust policies from static datasets without further interactions with the environment. Many approaches 77,77 use Bayesian inference to model the transition dynamics or guide action selection for adaptive policy updates, thereby avoiding overly conservative estimates in the offline setting. Furthermore, recent work (78) applies variational Bayesian inference to learn the model of transition dynamics, mitigating the distribution shift in offline RL.
Conclusion
In this paper, we investigate and demonstrate the robustness and effectiveness of introducing Bayesian inference into offline RL to address the challenges posed by data corruptions. By leveraging Bayesian techniques, our proposed approach TR TRACER captures the uncertainty caused by diverse corrupted data, Moreover, the use entropy-basee uncertainty measure in TRACER can distinguish corrupted data from clean data. Thus, TRACER can regulate the loss associated with corrupted data to reduce its influence, improving performance in clean environments. Our extensive experiments demonstrate the potential of Bayesian methods in developing reliable decision-makee maainng.
Regarding the limitations of TRACER, although it achieves significant performance improvement under diverse data corruptions. future work could explore more complex and realistic data corruption scenarios and related challenges, such as the noise the preference data for RLHF and adversarial attacks on safety-critical driving decisions. Moreover, we look forward to the continued development corrupted-robus offline RI., which could further enhance the
and optimization of uncertainty- effectiveness of TRACER and similar approaches for increasingly complex real-world scenarios.
10

---

Acknowledgments
We would like to thank all the anonymous reviewers for their insightful comments. This work was supported in part by National Key R&D Program of China under contract 2022ZD00 19801, National Nature Science Foundations of China grants U23A20388 and 62021001, and DiDi GAIA Collaborative Research Funds.
References
[1] Scott Fujimoto, David Meger, Doina Precup. Off-policy deep reinforcement learning without explo- ration. Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2052-2062. PMLR, 2019.
[2] Rafael Figueiredo Prudencio, Marcos R. O. A. Máximo, and Esther Luna Colombini. A survey offline reinforcement learning: Taxonomy, review, and open problems. CoRR, Abs/2203.01387, 2022.
[3] Shengpu Tang and Jenna Wiens. Model selection for offline reinforcement learning: Practical considerations healthcare settings. Ken Jung, Serena Yeung, Mark Sendak, Michael W. Sjoding, and Rajesh Ranganath, editors, Proceedings of the Machine Learning for Healthcare Conference, volume 149 of Proceedings of Machine Learning Research, pages 2-35, PMLR, 2021.
[4] Christopher Dichl, Timo Sievemich, Martin Krüger, Frank Hoffmann, and Torsten Bertram, Uncertainty- model-based offline reinforcement learning for automated driving. IEEE Robotics Autom. Lett., 167 2023.
[5] Tony 7. Zhao, Jianlan Luo, Oleg Sushkov, Rugile Pevceviciute, Nicolas Heess, Jon Scholz, Stefan Schaal, and Sergey Levine. Offline meta-reinforcement learning for industrial insertion. In 2022 International Conference on Robotics and Automation, pages 6386-6393, IEEE, 2022.
[6] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-BBBu Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32, pages 17761-11777 2019.
[7] Jiafei Lyu. Xiaoteng Ma, Li, and Zongging Mildly conservative q- q-learning for offline reinforcement learning. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and Oh, editors, Advances in Neural Information Processing Systems 35, 2022.
[8] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y. Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: model-based offline policy optimization. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances Neural Information Processing Systems 33, 2020.
[9] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria- Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33, 2020.
[10] Porter Jenkins, Hua Wei, J. Stockton Jenkins. and Zhenhui Li. Bayesian model-based offline reinforeement learning for product allocation. In Thirty-Sixih AAAI Conference Artificial Intelligence, pages 12531- 12537. AAAI Press, 2022.
[11] Sen Lin, Jialin Wan, Tengyu Xu, Yingbin Liang, and Junshan Zhang. Model-based offline meta- reinforcement learning with regularization. The Tenth International Conference on Learning Rep- resentations. OpenReview.net, 2022.
[12] Gaon An, Scungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty- offline reinforcement learning with diversified qeensemble. In Mare Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34, pages 7436-7447, 2021.
[13] Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor critic for offline reinforcement learning. In Marina Meila and Tong Zhang, editors. Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 11319 11328. PMLR, 2021.

---

[14] Chenjia Bai, Lingxiao Wang. Zhuoran Yang. Zhi-Hong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. In The Tenth International Conference on Learning Representations. OpenReview.neh, 2022.
[15] Filippo Valdettaro and A. Aldo Faisal. Towards offline reinforcement learning with pessimistic value priors. In Fabio Cuzzolin and Maryam Sultana, editors, Epistemic Uncertainty in Artificial Intelligence First International Workshop. volume 14523 of Lecture Notes in Computer Science, pages 89-100. Springer. 2024.
[16] Xuezhou Zhang, Yiding Chen, Xiaojin Zhu, and Wen Sun. Corruption robust offline reinforcement learning. In Gustau Camps-Valie Francisco J. R. Ruiz, and Isabel Valera, editors, International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pages 5757-5773. PMLR, 2022.
[17] Chenlu Ye, Rui Yang. Quanquan Gu, and Tong Zhang. Corruption-ron robust offline reinforcement learning with general function approximation. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz: Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36, 2023.
[18] Rui Yang. Han Zhong, Jiawei Xu, Amy Zhang, Chongjie Zhang, Lei Han, and Tong Zhang. Towards robust offline reinforcement learning under diverse data corruption. In The Eleventh International Conference on Learning Representations, 2023.
[19] Kishan Panaganti, Zaiyan Xu, Dileep Kalathil, and Mohammad Ghavamzadch. Robust reinforcement learning using offline data. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35, 2022.
[20] Rui Yang. Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjje Zhang, and Lei Han. RORL: robust offline reinforcement learning via conservative smoothing. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances Neural Information Processing Systems 35, 2022.
[21] Jose H. Blanchet, Miao Lu, Tong Zhang, and Han Zhong. Double pessimism is provably efficient for distributionally robust offiine reinforcement learning: Generic algorithm and robust partial coverage. In Alice Oh, Tristan Naumann. Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors. Advances in Neural Information Processing Systems 36, 2023.
[22] Yuzhe Ma, Xuezhou Zhang, Wen Sun, and Jerry Zhu. Policy poisoning in batch reinforcement learning and control. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence Alché- Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32, pages 14543-144 14553, 2019.
[23] Fan Wu, Linyi Li, Huan Zhang, Bhavya Kailkhura, Krishnaram Kenthapadi, Ding Zhao, and Bo Li. COPA: COPA: certifying robust policies for offline reinforcement learning against poisoning attacks. In The Tenth International Conference on Learning Representations. OpenReview.net, 2022.
[24] Zhihe Yang and Yunjian Xu. Dmbp: Diffusion model based predictor for robust offline reinforcement learning against state observation perturbations. In The Twelfth International Conference on Learning Representations, 2023.
[25] Xue Bin Peng. Aviral Kumar, Grace Zhang, and Sergey Levine. maantage-weighted regression: Simple and scalable off-policy reinforcement learning. CoRR, abs/1910,00177. 2019.
[26] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learming. In The Tenth International Conference on Learning Representations. OpenReview.net, 2022.
[27] Rens van de Schoot, Sarah Depaoli, Ruth King, Bianca Kramer, Kaspar Märtens, Mahlet Tadesse, Marina Vannucci, Andrew Gelman, Duco Veen, Joukje Willemsen, et al. Bayesian statistics and modelling. Nature Reviews Methods Primers, 1(1):1, 2021.
[28] Box George EP and Tiao George C. Bayesian inference in statistical analysis. John Wiley Sons, 2011.
[29] Esther Derman, Daniel J. Mankowitz, Timothy A. Mann, and Shie Mannor. A bayesian approach robust reinforcement learning. In Amir Globerson and Ricardo Silva, editors, Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence, volume 115 of Proceedings of Machine Learning Research, pages 648-658. AUAI Press, 2019.
[30] Matthew Fellows, Anuj Mahajan, Tim G. J. Rudner, and Shimon Whiteson. VIREL: variational inference framework for reinforcement learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer. Florence d' Alché-Bur Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32, pages 7120-7134, 2019.
12

---

[31]
Mattie Fellows, Kristian Hartikainen, and Shimon Whiteson. Bayesian bellman operators. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34, pages 13641-13656, 2021.
[32]
Brendan O' O'Donoguue. Variational bayesian reinforcement learning with regret bounds. In Marc Aurelio Ranzato, Alina Beygelzimer. Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34, pages 19208-2822)) 2021.
[33] Ron Dorfmaan Idan Shenfeld, and Aviv Tamar. Offline meta reinforcement learning identifiability challenges and effective data collection strategies. In Mare' Aurelio Ranzato, Alina Beygelzimer. Yann N. Dauphin, Perey Liang, and Jennifer Wortman Vaughan, editors, Advances Neural Information Processing Systems 34, pages 4607-4618, 2021.
[34] Dibya Chosh, Anurag Ajay. Pulkit Agrawal, and Sergey Levine. Offline RL policies should be trained be adaptive. In Kamalika Chaudhuri, Stefanie Jegelka, Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 7513-7530. PMLR, 2022.
[35] Yuhao Wang and Enlu Zhou. Bayesian risk-avers averse q-learning with streaming observations. Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36, 2023.
[36] Mohammad Ghavamzadch, Shic Mannor, Joelle Pineau, and Aviv Tamar. Bayesian reinforcement learning: A survey, Found. Trends Mach. Learn., 8(5-6):359 483, 2015.
[37] Carl Docrsch. Tutorial variational autoencoders. CoRR, 10W1605900.88 2016.
[38] Diederik P. Kingma and Max Welling. An introduction to variational autoencoders. Found. Trends Mach. Learn., 12(4):307-392, 2019.
[39] Mare G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 449 458, PMLR, 2017.
[40] Kotz Samuel and Johnson Norman Breakthroughs statisticx: methodology and distribution. Springer Science & Business Media, 2012.
[41] Jeremias Knoblauch, Jack Jewson, and Theodoros Damoulas. Generalized variational inference. CoRR, abs/1904.02063, 2019.
[42] Will Dabney, Mark Rowland, Mare G. Bellemare, and Rémi Munos. Distributional reinforvement learning with quantile regression. In Sheila A. Mcllraith and Kilian Q Weinberger, editors, Proceedings of the Thirty- AAAI Conference on Artificial Intelligence, pages 2892-2901. AAAI Press, 2018
[43] Will Dabney, Georg Ostrovski, David Silver. and Rémi Munos. Implicit quantile networks for distribu- tional reinforcement learning. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 Proceedings of Machine Learning Research. pages 1104-1113, PMLR, 2018
[44] Koenker Roger and Hallock Kevin F. Quantile regression. Jourss of economic perspectives, 15(4):143-15) 156. 2001.
[45] Müller Alfred. Integral probability metrics and their generating classes of functions. Advances in applied probability, 29(2):499 443, 1997.
[46] Abhishek Roy, Krishnakumar Balasubramanian, and Murat A. Erdogdu. On empirical risk minimization with dependent and heavy-tailed data. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang. and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems
34, pages 8913 8913-8926, 2021.
[47] Michele Caprio. Souradeep Dutta, Kuk Jin Jang, Vivian Lin, Radoslav Ivanov, Oleg Sokolsky. and Insup Lec. Credal bayesian deep learning, arXiv e-prints, pages arXiv-2300 2023.
[48] Michele Caprio, Yusuf Sale, Eyke Hüllermeier, and Insup Lee. A novel bayes' theorem for upper probabili- ties. In Fabio Cuzzolin and Maryam Sultana, editors. Epistemic Uncertainty in Artificial Intelligence - First International Workshop, volume 14523 of Lecture Notes in Computer Science, pages 1-12. Springer, 2024.
[49] Souradeep Dutta, Michele Caprio, Vivian Lin, Matthew Cleaveland, Kuk Jin Jang, Ivan Ruchkin, Oleg Sokolsky, and Insup Lee. Distributionally robust statistical verification with imprecise neural networks. CoRR, abs/2308.1
1100008111111111 14815, 2023.
13

---

[50] Jim W. Hall. Uncertainty-baser sensitivity indices for imprecise probability distributions. Reliub. Eng. Syst. Saf., 91(10-1111144--4551 2006.
[51] I. Lome Camphell. Exponential entropy measure of extent of distribution. Zeitsckrift für Wahrschein- lichkeitstheorie und verwandte Gebiete, 5(3):117-225, 1966.
[52] Sheri Edwards. Thomas m. cover and joy thomas, elements of information theory (2nd ed.). wiley & sons, inc. (2006). Inf. Process. Manag., 4 44(1):40004400 4011, 2008.
[53] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, 2018.
[54] Huan Zhang, Hongge Chen, Chaowci Xiao, Bo Li, Mingyan Liu, Duane S. Boning, and Cho-Jui Hsich. Robust deep reinforcement learning against adversarial perturbations on state observations. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33, 2020.
[55] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: datasets for deep data-driven reinforcement learning. CoRR, 2020.
[56] Gaon An, Seungyong Moon, Jang- Hyun Kim, and Hyun Oh Song. Uncertainty based offline reinforcement learning with diversified q-ensemble. In Mare Aurelio Ranzato, Alina Beygelzimer, Yann Dauphin. Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34, 2021.
[57] Seyed Kamyar Seyed Ghasemipour, Shixiang Shane Gu, and Ofir Nachum. Why so pessimistic? estimating uncertainties for offline RL through ensembles, and why their independence matters. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35, 2022.
[58] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model- control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2012, Vilamoura, Algarve, Portugal, October 7-12, 2012, pages 5026-5033. IEEE, 2012.
[59] Alexey Dosovitskiy, Germán Ros, Felipe Codevilla, Antonio M. López, and Vladlen Koltun. CARLA: an open urban driving simulator. In /st Anmual Conference on Robot Learning, volume 78 of Proceedings of Machine Learning Research, pages 1-16. PMLR, 2017.
[60] Chi Jin, Tiancheng Jin, Haipeng Luo, Suvrit Sra, and Tiancheng Yu. Learning adversarial markov decision processes with bandit feedback and unknown transition. In Proceedings of the 37th International Conference on Machine Learning, 2020.
[61] Thodoris Lykouris. Vahab S. Mirrokni, and Renato Paes Leme. Stochastic bandits robust adversarial corruptions. In Ilias Diakonikolas,, David Kempe, and Monika Henzinger, editors, Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, 2018.
[62] Aviv Rosenberg and Yishay Mansour. Online convex optimization in adversarial markov decision pro- cesses. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning. 2019.
[63] Anupam Gupta, Tomer Koren, and Kunal Talwar. Better algorithms for stochastic bandits with adversarial corruptions. In Alina Beygelzimer and Daniel Hsu, editors, Conference on Learning Theory, 2019.
[64] Chenlu Ye, Wei Xiong. Quanquan Gu, and Tong Zhang. Corruption-rouust algorithms with uncertainty weighting for nonlinear contextual bandits and markov decision processes. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, 2023.
[65] Thodoris Lykouris, Max Simchowitz, Alex Slivkins, and Wen Sun. Corruption-rbuust exploration in episodic reinforcement learning. In Mikhail Belkin and Samory Kpotufe, editors, Conference on Learning Theory, 2021.
[66] Matthicu Geist and Olivier Pictquin. Kalman temporal differences. J. Artif. Intell. Res., 39:483-532, 2010.
[67] lan Osband, Daniel Russo, and Benjamin Van Roy. (more) efficient reinforcement learning via posterior sampling. In Christopher J. C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger. editors, Advances in Neural Information Processing Systems 26, pages 3003-3011, 2013.
14

---

[68] Mare Peter Deisenroth and Carl Edward Rasmussen. PILCO: A model-based and data-efficient approach to policy search. In Lise Getoor and Tobias Scheffer, editors, Proceedings of the 28th International Conference on Machine Learning, 2011.
[69] Richard Dearden, Nir Friedman, and Stuart Russell. Bayesian q-learning. In Jack Mostow and Chuck Rich, editors, Proceedings of the Fifteenth National Conference on Artificial Intelligence and Tenth Innovative Applications of Artificial Intelligence Conference, 1998.
[70] Gal Yarin, McAllister Rowan, and Rasmussen Carl Edward. Improving pilco with bayesian neural network dynamics models. In Data-efficient machine learning workshop, ICML, volume 4, page 25, 2016.
[71] Zhihai Wang. Jie Wang. Qi Zhou, Bin Li, and Houqiang Li. Sample-efficient reinforcement learning via conservative model-based actor-critic, In Thirty-Sixth AAAI Conference on Artificial Intelligence, pages 8612 8620. AAAI Press, 2022.
[72] Yaakov Engel, Shie Mannor, and Ron Meir. Reinforcement lcarning with gaussian processes. In Luc De Raedt and Stefan Wrobel, editors, Machine Learning, Proceedings of the Twenty-Second International Conference (ICML 2005), 2005.
[73] Mohammad Ghavamzadeh and Yaakov Engel. Bayesian policy gradient algorithms. In Bernhard Schölkopf, John C. Platt, and Thomas Hofmann, editors, Advances in Neural Information Processing Systems 19, 2006.
[74] Mohammad Ghavamzadeh, Yaakov Engel, and Michal Valko. Bayesian policy gradient and actor-critic algorithms. J. Much. Learn. Rex., 2016.
[75] Dibya Ghosh, Anurag Ajay, Pulkit Agrawal, and Sergey Levine. Offline RL policies should be trained to be adaptive. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, 2022.
[76] Hao Hu, Yiqin Yang, Jianing Yc, Ziqing Mai, Yujing Hu, Tangjic Lv, Changjie Fan, Qianchuan Zhao, and Chongjic Zhang. Bayesian offline-to-online reinforcement learning: : A realist approach, 2024.
[77] Yuhao Wang and Enlu Zhou. Bayesian risk-averse g-learning with streaming observations. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36, 2023.
[78] Toru Hishinuma and Kei Senda. Importance- variational inference model estimation for offline bayesian model-basee reinforcement lcarning. IEEE Access, 2023.
[79] Sham M. Kakade and John Langford. Approximately optimal approximate reinforcement learnin.. In Claude Sammut and Achim G. Hoffmann, editors, Machine Learning. Proceedings of the Nineteenth International Conference (ICMI. 2002), pages 267-274. Morgan Kaufmann, 2002.
[80] Vallender SS. Calculation of the wasserstein distance between probability distributions on the line. Theory of Probability & Its Applications, 18:784-786, 1974.
[81] Anqi Li, Dipendra Misra, Andrey Kolobov, and Ching-An An Cheng, Survival instinct offline reinforcement learning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, 2023.
[82] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Advances in Neural Information Processing Systems 34, pages 15094, 2021.
[83] Tuomas Haamoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off Off-po maximum entropy deep reinforeement learning with stochastic actor, In International conference on machine learning, pages 1861-1870. PMIRR, 2018.
[84] Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, and Dmitry Vetrov. Controlling overestimation bias with truncated mixture of continuous distributional quantile critics. In Internatioonl Conference on Machine Learning, pages 5556-55566 5566. PMLR, 2020.
[85] Zhihai Wang, Taoxing Pan, Qi Zhou, and Jie Wang. Efficient exploration in resource-restritte rein- forcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 10279-102877 2023.
15

---

[86] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32, 2019.
[87] D Mankowitz, Andrea Michi, Anton Zhernov, Marco Gelmi, Marco Selvi, Cosmin Paduraru, Edouard Leurent, Shariq Iqbal, lean-Baptiste Lespiau. Alex Ahern, al. Faster sorting algorithms discovered using deep reinforcement learning. Nature, 61877964)2257-663 2023.
[88] Chris Gamble and Jim Gao. Safety-first ai for autonomous data centre cooling and industrial control. DeepMind, August, 17, 2018.
[89] Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healthcare: A survey. ACM Computing Surveys (CSUR). 55(1):1-36, 2021.
[90] Zhihai Wang. Jie Wang, Dongsheng Zuo, Yunjie Ji, Xinli Xia, Yuzhe Ma, Jianye Hao, Mingxuan Yuan, Yongdong Zhang. and Feng Wu. A hierarchical adaptive multi task reinforcement learning framework for multiplier circuit design. In Forty-first International Conference on Machine Learning. PMLR, 2024.
[91] Zhihai Wang, Xijun Li, Jie Wang, Yufei Kuang, Mingxuan Yuan, Jia Zeng. Yongdong Zhang, and Feng Wu, Learning cut selection for mixed-integer linear programming via hierarchical sequence model. In The Eleventh International Conference on Learning Representations, 2023.
[92] Jie Wang, Zhihai Wang. Xijun Li, Yufei Kuang, Zhihao Shi. Fangzhou Zhu, Mingxuan Yuan, Jia Zeng. Yongdong Zhang, and Feng Wu. Learning to cut via hierarchical sequence/set model for efficient mixed- integer programming. IEEE Transactions Pattern Analysis and Machine Intelligence, pages 1-17, 2024.
[93] Zhihai Wang. Zijie Geng, Zhaojie Tu, Jie Wang. Yuxi Qian, Zhexuan Xu, Ziyan Liu, Siyuan Xu, Zhentao Tang. Shixiong Kai, et al. Benchmarking end-to-end performance of ai basedd chip placement algorithms. arXiv preprint arXiv:2407, 15026, 2024.
16

---

Appendix supplemmnnal material
A Proofs
We provide proof for upper bound value distributions and derivations of loss functions. See Table all notations.
A.1 Proof for Value Bound
To prove an upper bound of the value distribution, we introduce an assumption from 18 below.
Assumption A.1. [8] Lel Ç Ei=1 (2G; + log (i) denote the cumulative corruption level, where So and C are defined as
TV (8t) TV (s))) SCn
max
TB (a 84)
Th (a 8;)
S S.
V.
Tb (a 8i)
TE (a 8:)
Here || Iloo means taking supremum over V S + [0, Tmax/ /1 - 7)]-
Note that {C}): = N can quantify the corruption level of states and actions, and {G) = 1 can quantify the corruption level of rewards and next states (transition dynamics).
Then, we provide the following assumption.
Assumption A.2.
There exists an > such that
maxt (s, (s, ( d= (s) d pb(s.a) pb(s,a) d (F)(s) dF
M, M, ((s,d)
where Tg(a 8) Ts(a 8) exp (3- V'I is the policy of clean data, TiQL in Equation (77) is the policy following IQL's supervised policy learning scheme under clean data,
TIQL arg minB,~8 [KL (#t(- s), T(- |s)s s))s the policy under corrupted data, and #g(a &
Ta(a 8) exp (3. [te v*] (s) is the policy of corrupted data.
moQ = arg max (s (exp (3. [TQ" - (8, log T(a
(17)
arg min Egwo [KL (TE( s), x(- s))]
(18)
As TRACER directly applies the weighted imitation learning technique from IQL learn the policy, we can use ĀIQL as the policy learned by TRACER under data corruptions, akin to RIQL. Assumption A.2 requires that each pair, including the policy TE and the clean data b, the policy Tg and the corrupted dataset B, Tg and ã g, TE and TIOL, and 7g and WoL.. has good coverage. It similar to the coverage condition in [8]]. Based on Assumptions A.I and A.2 we can derive the following theorem to show the robustness of our approach using the supervised policy learning scheme and learning the action-valus distribution.
Lemma A.3. (Performance Difference) For any and w. we have
1
(²(s) (s)
E ss.a di [D (s, (s)).
(19)
1-
17

---

Proof. Based on 79. for any म and I,, we have
(s) Σ 7 (. [R(S), (RG
(20)
(R(S, 7 (s) (R(S), At) + Z* - = s]
(21)
7 AS. (R(S At) + (S*11)+ (5 27())())
(22)
(* +
"B (s..A.)) (St, A,, Rτ) - (S())s =
(23)
t=
(²)
E ((.aa)dd as, (s).
(24)
1
Theorem A.4. (Robustness). Under Assumplions A.7
and A.2
we have
W1 ./)
2Mr max
(1 7)2 V1
e- -
MC
(25)
where of is the value distribution, (; is the Wasserstein distance (80) for measuring the distri- bution difference, e1 Egnt (DRr. (TT()))) (s) m1)x(+18))], and e2 Egg- (Dxz. (#E(-18) || #ror(-1s)))-
inf (er(P))) S. || wil-dd w)
Proof. The definition of Wasserstein metric is Wp(P.Q)
The value distribution q², also denoted by p(-(s) Section 3.2 of the main text, an expactation of action value distribution Bn-o-,--pppo0-ls,, r)..
Note that TRACER directly applies the weighted imitation learning technique from IQL to learn the policy (see Section we can use fIqL as the policy learned by TRACER under data corruptions, akin to RIQL.
Let S0 and I be the value distributions of learned policies following IQL's supervised policy learning scheme under corrupted and clean data, respectively. z qrraz and Z g Let f be the value distribution of the policy of clean data, * be the value distribution of the policy of corrupted data, E p and Z Rx 50, We have
( T < W WI(GTIOn ,F*)) + WF (pFF,
(26)
Wi(9 ( =*) )++ + W (p= =P) + W TpTH (1Ttt,TTT00)) m106)
(27)
18

---

Moreover, based on the Bretagnolle-Hude inequality, we have dIv(P,Q)
vI exp(-DxL (PI))). For Wi Wi(p R pT we have
inf xE Zā 7 1 - - 1dd $2ER)
(28)
inf >ET(d .d
(1 =(
(29)
S .M ( - -5*5(s))
(30)
M E snidio 4 (s) - (s))
(31)
= .M *
R Bevd" a~ (-10) [ (s) - D= (8)
(32)
M
E did (a as, (a (D (s,a,
(s,,,
(33)
7
Mrmax
B [T((|s)) - T
(34)
(1 7))
2M AMMmax
18
Bandt 11- (
(35)
(1 -71²
2Mrmaa
< (1 --
E Bandii (1 - c (M 9))
(36)
2Mrmax
< 7)
e E, BrNatE (1)
(37)
2M mmax
-E Liaamaat (log (g(als) aglald).
C
(38)
(1 7)
Moreover, we have
T (a.agg (82)
#B(ai(s;) exp(B. (TV ()) TB(ai|0)) (8)) exp(( [TV* V*](s, a))
(39)
Tg Tg(a;1s))
x EeEs ( exp(B. vT (s)
(40)
(B(((ss)) exp(B. (TV= (s)
By the definition of corruption levels, we have
Te [e(a, 82) < C exp(3()) Si exp(3C) EoEA TB exp(3 exp(3-17v - vv)
(ac(())))
(41)
TB exp(3 ( - v*)(
G exp(23G))
(42)
Thus, can derive
2M max
AMaa
V1-e 1 3/
(43)
(1 1 7)2
Then, for Wi ( (1 we have
Mrmax E E [1) 5 (1 7)²
2Mrmax v1 - -
Wi g 10 <
(44)
(1 7)2
where E = Eand [DKL (TE() ll mIQ (-1s))
Finally, for W Wi "n *E), we have
M Mrmax E. Exndie (| - -1)1(11)) 5 (1 7)2
2Mrmax vI - -2 (45)
W1 Wilg Me2 (0as,,=E) <
(1 - 72
19

---

Therefore, we have
(0) (1 - 7)2
2M max
(V1 (V1- e-- Mr
+ + v1 V1-
(46)
Note that in Theorem A.4 the major difference between TRACER and IQL/RIQL, is that TRACER uses the action-value and value distributions rather than the action-value and value functions in IQL/RIQL. Therefore, we provide this theorem to prove upper bound on the difference in value distributions TRACER to show its robustness, which also provides guarantee of how TRACER's performance degrades with increased data corruptions.
A.2 Derivation of Loss Functions
For Equations
we provide the detailed derivations. Firstly, for Equation we maximize the posterior and have
7 and (9)
max log D) max log Es [p(D)S,, Rt)] max Es,, R, D(D Sr, Rt)]
max Es,, [ Pv. Se, logp(D)S, Ri)
(47)
logp(D)S;,,i, Rτ) = =/ Per S.41) log
p(D, A, St, Rr S.+1) Pv. (Ad), St, S.+1) dAp
P(A)
Pv. [A,]D, St, S.+1) p(A, DD, S,, S(+1)
.PLAn Pdo (A, S,, llog
p(D. A.D.AsSS Rd)
Pv. (Ar)D, St, Sc+1) dA.
P(A)
Pas (A, Sr, Se+1) log Pv. (Ar)D, St, S.41) dA. p(A, D. Se+1)
P(A)
= =/ P(A)
Pdo (Ar D, S, S.11) log p(D, Ac Rd)
Pv. Se, dA.
+ DXL (po. (A,), S₁, Sp41) S(,
Lb DKL (pv. (Ar)D, S. ( S(41)
(48)
2 Lb.
(49)
Note that Lg is
Tb [ Pe. S4,1)) log
p(D, A Sr, Re)
dAt
(50)
JP(
Po. (Ai D, St, St+1)
= = -PPAA Ps. S,, S.41) (A (S)) (D(SD Ad, Ri)
dAA
Pv. (A,)D, SD,,S,,, Se41)
= = (po. (Ar)D, S,,
+ P(A)
Pv. (A, S1+1) (ogg (D)S, Ar, Re)dA,
20

---

Secondly, for Equation we have
max log PDD = max log R.,,,, [P(D St, A:)]
2 max A. Dlg P(D]St, A:)l
max Es
Pv. (Rr S,,Ai A.) -og
P(R)
max Es
DeL (ps. (BI1...55..AA411(R)55....)))
/ P(R)) Pa. St, Ae, Ra)dRa
(51)
Finally, for Equation
we have
max log (D) max log A. R. [p( D|A Re)
2 max EA, [log pl D Al, Re)
max R.
Pv. (S:) Ar, Rτ) lloglog logg(D)Ae, Re)dS:
P(S)
max B., Ri
DKL. (Nv, (S. A,, R.)llp(Ss)A.. Rc))
/ P(S)
Pe. D, (o) Ar, Ro)dS,
(52)
Therefore, have Equations 7 and 19
A.3 Estimation of Entropy
We estimate differential entropy following 522
We have differential entropy follows. Note that we omit the condition the following equations.
H(po(D)) El-po (D)) DD po (og
(53)
Then, we consider continuous function Po discretized into bins of size A. By the mean-value theorem there exists a action value D; in each bin such that
pe (D,) (++1)
pa (D)dD.
(54)
We can estimate the integral of po (in the Riemannian sense) by
OC
DD lim Σ
Po (D.)A.
(55)
A-0
Thus, we have
H(po(D)) = /
00
Po (D)logg loggl(p)dsee
lim (Ha(po(D)) log A)
8
lim A
Σ
po (D,))
(D.))
(56)
Based on Equation
56
we can derive Equation
15
where é, denotes the midpoint of po (D,), and
D denotes A.
21

---

Table 4: Notations used in our proofs.
Notations
Descriptions
Cumulative corruption level.
G
Metric quantifying the corruption level of rewards and next states (transition dynamics).
Metric quantifying the corrpption level of states and actions.
(-Is)
The behavior policy that is used to collect clean data,
T (+sss)
The behavior policy that used to collect corrupted data.
The policy that we want learn under clean data.
*g()s)
The policy that are learning under corrupted data.
rra((1ls)
The learned policy using IQL's weighted imitation learning under clean data. The learned policy using IQL's weighted imitation learning under corrupted data.
(-la)
d" (ssss a)
The probability density function associated with policy at state and action a.
Wi (p. g)
The Wasserstein-1 distance that measures the difference between distributions p and The value distribution of the policy æ.
q
nn (s)
The random variable the value distribution (-((ss)
61
KL divergence between Fg and FIQL, i.e., the standard imitation under clean data.
62
KL divergence between t and RiQu, i.c., the standard imitation error under corrupted data.
6n
The midpoint of the probability density
NEW
Ensemble Models
Φ MLPs (Do, r..
Pos
St
a. MLPs (De, St. Sc.
Pen
Ensemble Loss Larst + Leecond
S
y MLPs MLPs(Da,,. ap)
Pwr
D
Modified Distributional RL
Sc
ac
Critic Network (0:
(Do
Critic Loss
Sc+1
Value Network vi
Z
Value Loss
D et
Actor Network +
T
Actor Loss
Figure 4: Architecture of TRACER.
B TRACER Approach
B.1 Architecture of TRACER
On the actor-critic architecture based on IQL, our approach TRACER adds just one ensemble model (Pe.. : PP...P to reconstruct the data distribution and replace the function approximation in the critic with distribution estimation (see Figure Based on our experiments, this structure significantly improves the ability to handle both simultancous and individual corruptions.
22

---

B.2 Implementation Details for TRACER
TRACER consists of four components: the actor network, the critic network, the value network, and the observation model. We first implement the actor network for decision- with a 3-layer MLP, where the dimension of the hidden layers are 256 and the activation functions are ReLUs. Then, we utilize quantile regression A+ to design the critic networks and the value network, approximating the distributions of the action-valus function D and value function Z, respectively.
[0. - R to compute obtain the corresponding approximation by D (8, a) R I [g(s)) @ h(T))a- Here, S + Rd is the function computed by MI.ss and f: Rdd + RIAl is the subsequent fully-connected layers mapping g(s) to the estimated action-values Dss (g())) For the approximation of the value distribution Z" (s), we leverage the same architecture. Note that the function is implemented by fully-connected layer, and the output dimension set 256. We then use an ensemble model with K networks. Each network a 3-layer MIP, consisting of 256 units and ReLU activations, By using the ensemble model, we can construct the critic network, using the quantile regression for optimization. Moreover, the value network is constructed by 3-layer MLP. The dimension of the hidden layers are also 256 and activation functions are ReLUs.
Specifically, for the action- value distribution, we use function embedding for the sampling point Then,
The observation model uses an ensemble model with 3 neural networks. Each network is used to reconstruct different observations in the offline dataset. It consists of two fully connected layers, and the activation functions are ReLUs. We apply a masked matrix during the learning process for updating the observation model. Thus, each network in the observation model can different input data to compute Equations 7. and (9). respectively.
Following the setting of [18] we list the hyper- ayper-parameters of N. K, and for the approxmiation of valie functions, and in the Huber loss in TRACER under random and adversarial corruption in Table 5 and Table respectively. Here, N is the number of samples and K is the number of ensemble models.
Moreover, we apply Adam optimizer using a learning rate 1 x 10-3, 0.99, target smoothing coefficient - 0.05, batch size 256 and the update frequency for the target network is 2. The comrpption rate is = 0.3 and the corruption scale is 1.0 for all experiments. For the loss function of observation models, we use linear decay parameter r/ from 0.0001 to 0.01 to trade off the loss Lars and Lepondd We update each algorithm for 3000 epochs. Each epoch uses 1000 update times following [56). Then, we evaluate each algorithm in clean environments for 100 epochs and report the average normalized return (calculated by 100
conde-radom score
over four random seeds.
expert score random
Table 5: Hyper- paremeerrseetrrss used for TRACER under the random corruption benchmark.
Environments
Corruption Types
N
K
observation
32
5
0.25
0.1
action
32
5
0.25
0.1
Halfcheetah
reward
32
5
0.25
0.1
dynamics
32
5
0.25
0.1
simultaneous
32
0.25
0.1
observation
32
0.25
0.1
action
32
5
0.25
0.1
Walker2d
reward
32
5
0.5
0.1
dynamics
32
5
0.25
1.0
simultancous
32
5
0.25
0.1
observation
32
5
0.25
0.1
action
32
0.25
0.1
Hopper
reward
32
5
0.5
0.1
dynamics
32
0.25
0.1
simultaneous
32
5
0.25
0.1
CARLA
simultancous
32
5
0.25
0.1
23

---

Table 6: Hyper-parameters used for TRACER under the adversarial corruption benchmark.
Environments
Corruption Types
N
K
at
K
observation
32
5
0.1
0.1
action
32
0.1
0.1
Halfcheetah
reward
32
5
0.1
0.1
dynamics
32
0.1
0.1
simultaneous
32
5
0.25
0.1
observation
32
0.25
1.0
action
32
5
0.1
1.0
Walker2d
reward
32
0.1
0.1
dynamics
32
5
0.25
0.1
simultaneous
32
0.25
0.1
observation
32
0.25
0.5
action
32
5
0.25
0.1
Hopper
reward
32
0.5
0.1
dynamics
32
0.5
0.1
simultaneous
32
0.5
0.001
C Detailed Experiments
C.1 Details of Data Corruptions
We follow the corruption settings proposed by [18] applying either random adversarial corruptions to each element of the offline data, namely state, action, reward, and dynamics (or "next-state)) to simulate potential attacks that may occur during the offline data collection and usage process in real-world scenarios. We begin with the details of random corruption below.
Random observation corruption. We randomly sample e% transitions (s. a,, from the offline dataset, and for each these selected states s, we add noise to form 8 = s + std((s), where ~~ Uni form[-e,, e c]d,, is the corruption rate, the corruption scale, d, refers to the dimension of states and std( represents the d, dimeenonsal standard deviation of all states in the offline dataset.
Random action corruption. We randomly sample 2% transitions (s, a, r, r')) from the offline dataset, and for each of these selected actions a, add noise to form à = + X std(a), where ~ Uniformm form[-c, d, refers to the dimension of actions and std(()) represents the d, dimdemessional standard deviation of all actions the offline dataset.
Random reward corruption. We randomly sample c% transitions (s, a, s') from the offline dataset, and for each of these selected rewards r, we modify it to e, e,30 e). We adopt this harder reward corruption setting since [8] has found that offline RI. algorithms are often insensitive to small perturbations of rewards.
Random dynamies corruption. We randomly sample c% transitions (s, a, s') from the offline dataset, and for each of these selected next-step states s', we add noise to form á = s' + A- std((ss where Uni c)d,,, d, refers to the dimension of next-step states and sid(s') represents the dy-dimensional standard deviation of all next states in the offline dataset.
The harder adversarial corruption settings are detailed as follows:
Adversarial observation corruption. To impose adversarial attack on the offline dataset, first need to pretrain an EDAC agent with a set of Qp functions and policy function Tp using clean dataset. Then, we randomly sample e% transitions (s. s') from the offline dataset, and for each of these selected states s, we attack to form & = minser (5, a). Here, Ba,(s, <) = {álls s) e sld(s)) regularizes the maximum difference for each state dimension.

---

Adversarial action corruption. We randomly sample c% transitions (s. a, 7, s') from the offline dataset, and for each of these selected actions a. we use the pretrained EDAC agent to attack it to form â = mndrgeBa(a Qp(s, (,,)),,,))))))) a). Here, Ba((a,) c) {állā al S (a))
regularizes the maximum difference for each action dimension.
Adversarial reward corruption. We randomly sample e% transitions (s, a, s') from the offline dataset, and for each of these selected rewards r. we directly attack it to form
f - X without any adversarial model. This is because that the objective of adversarial reward corruption is f = mi.reB(r.e)) f > a')]. Here (B(r,)) {f f - rl S
(1 (Tmaxx regularizes the maximum distance for rewards. Therefore, we have f - Xr.
Adversarial dynamics corruption. We randomly sample %% transitions (s, a, r, r') from the offline dataset, and for each of these selected next-step states s', we use the pretrained EDAC agent to attack it to form s minge (pp Qp(s', (p(s))) Here, Ba(s', e) {&110> 8 S sld(s') regularizes the maximum difference for each dimension of the dynamics.
The optimization of the above adversarial corruptions are implemented through Projected Gradient Descent [53] 54]. Taking adversarial observation corruption for example, we first initialize learnable vector E |-c, C, cjd,, and then conduct 100-step gradient descent with a step size of 0.01 for 8= s + -- (s), and clip each dimension of within the range [-e, after each update. For action and dynamics corruption, we conduct similar operation.
Building upon the corruption settings for individual elements as previously discussed, we further intensify the corruption to simulate the challenging conditions that might be encountered in real-world scenarios. We present the details of the simultancous corruption below:
Random simultaneous corruptions. We sequentially conduct random observation corrup- tion, random action corruption, random reward corruption, and random dynamics corruption to the offline dataset in order. That is to say, we randomly select e% of the transitions each time and corrupt one element among them, repeating four times until states, actions, rewards, and dynamics of the offline dataset are all corrupted.
Adversarial simultaneous corruptions. We sequentially conduct adversarial observation corruption, adversarial action corruption, adversarial reward corruption, and adversarial dynamics corruption to the offline dataset order. That is to say, we randomly select c% of the transitions each time and attack one element among them, repeating four times until states, actions, rewards, and dynamics of the offline dataset are all corrupted.
C.2 Details for CARLA
We conduct experiments in CARLA from D4RL benchmark. We use the clean environment 'CARLA- Lane-v0' to evaluate IQL., RIQL, and TRACER (Ours). We report each mean result with the standard deviation in the left of Figure 2 over four random seeds for 3000 epochs. We apply the random simultaneous data corruptions, where each element in the offline dataset (including states, actions, rewards, and next states) may involve random noise. We follow the setting in Section 4.2 using the corrppiion rate c = 0.3 and scale = 1.0 in the CARLA task. The results in the left of Figure 2 show the superiority of TRACER in the random simultaneous corruptions. We provide the hyperparameters of TRACER in Table 5
C,3 Additional Experiments and Analysis
C.3.1 Results Comparison between TRACER and RIQL under Individual Corruptions
In Tables and of the main text, we adhered to commonly used settings for individual corruptions in corruption-robus offline RL. We directly followed hyperparameters from RIQL (i.e., K for huber loss, the ensemble number K, and in action-valus functions, see Tables 5 and 6). Results show that TRACER outperforms RIQL in 18 out of 24 settings, demonstrating its robustness even when aligned with RIQL'S hyperparameters.
Further, we explore hyperparameter tuning, specifically k. on Hopper task to improve TRACER's performance. This results in TRACER outperforming RIQL in 7 out of 8 settings on Hopper, up
25

---

Adversarial Mixed Corruption
Random Mixed Corruption
7.9
15.50
Entro Value
7.8-
Entro Value
15.25
7.7-
15:00
7.6
TRACER with Entro: Clean Data
14.75
TRACER with Entro: Clean Data
TRACER with Entro Corrupted Data
14.50
TRACER with Entro: Corrupted Data
7.5
TRACER without Entr:: Clean Data
TRACER without Entro: Clean Data
TRACER without Entr:: Corrupted Data
TRACER without Entro: Corrupted Data
7.4
1000 Walker2d Epoch
2500
1500 Walker2d Epoch
2500
Adversarial Mixed Corruption
Random Mixed Corruption
8:25
6.25
TRACER with Entro: Clean
with Entro: Costupted
Entro Value
6.00
Entro Value
without Entro:
5.75
7.75
TRACER without Entro: Corrupted
5.50
7.50
5.25
TRACER with Entro: Clean Data
7:25
TRACER with Entro: Corupted Data
5.00
TRACER without Eatro: Clean Data
7.00
TRACER without Entro: Corrupted Data
4.75
6.75
500
1000
1500
2000
2500
3000
500
1000
1500
2000
2500
3000
Halfchectah Epoch
Halfcheetah Epoch
Figure 5: We report the smoothed curves of mean of entropy values for each batch in 'Walker2d- peedium-replay-ve and 'Halfcheetah senccheetah-mdiium--ree medium-repllat reeduuuv-- under adversarial and random simultaneous data corruptions.
Table 7: Average results and standard errors with 2 seeds and 64 batch sizes in Hopper-medium: r2play-v2 task for hyperparameter tuning.
Random Dynamics
Random Dynamics
Random Dynamics
Random Dynamics
0.01
0.1
0.5
[.0
TRACER (Ours)
52.6 + 0.9
57.5 : 13.0
45.1 + 10.5
44.0 = 6.5
Adversarial Reward
Adversarial Reward
Adversarial Reward 0.5
Adversarial Reward 1.0
TRACER (Ours)
82.2 0.1
61.3 0.8
56.7 + 1.6
51.3 + 2.1
from settings (see Tables and (). The further improvement highlights TRACER's potential to achieve greater performance gains.
we find that TRACER requires low in Huber loss, using L1 loss for large errors. Thus, TRACER can linearly penalize corrupted data and reduce its influence on the overall model fit.
Based Table 7
C.3.2 Additional Experiments
We further conduct experiments two AntMaze datasets and two additional Mujoco datasets, presenting results under random simultancous corruptions in Table g
Each result represents the mean and standard error over four random seeds and 100 episodes in clean environments. For each experiment, the methods train agents using batch sizes of 64 for 3000 epochs. Building upon RIQL, we apply the experiment settings as follows. (1) For the two Mujoco datasets, we use a corruption rate of e = 0.3 and scale of 1.0. Note that simultaneous corruptions with = 0.3 implies that approximately 76.0% of the data is corrupted. (2) For the two AntMaze datasets, we use the corruption rate of 0.2, corruption scales for observation (0.3), action (1.0), reward (30.0), and dynamics (0.3).
26

---

Table 8: Average results and standard errors with 2 seeds and 64 batch sizes in Hopper-mediun replay-v2 task under individual corruptions.
Random Ohservation
Random Action
Random Reward
Random Dynamies
RIOL TR TRACER (Raw) TRACER (New)
62.4 62.7 = 8.2
90.6 92.8 = 2.5
84 85.7 + 1.4
51.5
49.8 + 5.3 53.8 13.5
Adversarial Observation
Adversarial Action
Adversarial Reward
Adversarial Dynamics
RIOL TRARER (Raw) TRACER (New)
50.8 = 7.6 64.5 + 3.7
63.6 + 7.3 67.2 + 3,8
65.8 + 9.8 64.3 + 1.5 71.7
65.7 + 21.1 61.1 +.2
Table The average scores and standard errors under random simultaneous corruptions.
AntMaze- Mediium Play-v2
AniMaze-Medium Diverse-v2 Mediuun-Diveeee2e2222
Walker2d
Expert-v2
Hopper-Mediunn MedieM- Expert---
IOL RIOL TRACER
20.6 3,4
1.4
21.6
4.3
3.7
6.6 1.4
47.0 6.6
55.5 2.9
The results in Table show that TRACER significantly outperforms other methods in all these tasks with the aforementioned AntMaze and Mujoco datasets.
show
C.4 Details for Evaluation and Ablation Studies
C.4.1 Evaluation for Robustness of TRACER across Different Scales of Corrupted Data
Theorem A.4 shows that the higher the scale of corrupted data, the greater the difference in action- value distributions and the lower the TRACER's performance. Thus, we further conduct experiments to evaluate TRACER across various corruption levels. Specifically, we apply different corruption rate in all four elements of the offline dataset. We randomly select e% of transitions from the dataset and corrupt one element in each selected transition. Then, we repeat this step four times until all elements are corrupted. Therefore, approximately 100 - c) 4% of data in the offline dataset is corrupted.
In Table 10 we evaluate TRACER using different c%, including 10%, 20%, 30%, 40%, and 50%. These rates correspond to approximately 34.4%, 59.0%, 76.0%, 87. .0%, and 93.8% of the data being corrupted. The results in Table I0 demonstrate that while TRACER robust simultancous corruptions, its scores depend on the extent of corrupted data it encounters.
C.4.2 Details for Evaluation of the Entropy-based Uncertainty Measure
In Figure we additionally report the entropy values of TRACER with and without using entropy-basee uncertainty measure, corresponding to F3] The curves illustrate that TRACER using entropy- based uncertainty measure can effectively regulate the loss associated with corrupted data, reducing the influ- ence of corrupted samples. Therefore, the estimated entropy values of corrupted data can be higher than those of clean data.
Walkerr2-MMediun--eplaa---
TRACER Using Entr
TRACER using Entro
25
DRIQL
Average Score
15
C.4.3 Ablation Study for Bayesian Inference
We conduct experiments of TRACER and TRACER without Bayesian inference, i..., RIQL combined with distributional RL methods, namely DRIQL. The experiments are on "Walker2d-medinne -eakkrrrd-mmdiuum--plaaa---2- under random simultaneous data corruptions. We employ the same hyperparameters of Huber loss for these methods, using a 0.25 and 1.0.
Figure 6: Results and standard errors under random simultaneous corruptions.
We report each mean result with the standard deviation in Figure 6 Each result is averaged over four seeds for 3000 epochs. These results show the effectiveness of our proposed Bayesian inference.
27

---

Table 10: Results in Hopper- under various random simultancous corruption levels.
Corrupt rate e corrupted data/ all data
0.1
0.2 2 59.0%
0.3 2 76.0%
0.4 2 87.0%
0.5 2 93.8%
Rs 34.4%
RIQL TRACER (Ours)
43.9 +: 10.3
55.9
5.7
22.5
10.0
21.6
6.2
15.6 + 1.8
79.6 3.5
64.0
3.0
28.8 7.1
25.9 = 2.2
19.7 = 1.2
D Compute Resource
In this subsection, we provide the computational cost of our approach TRACER.
For the MuloCo tasks, including Halfcheetah, Walker2d, and Hopper, the average training duration 40.6 hours. For the CARLA task, training extends to hours. We conduct all experiments on NVIDIA GeForce RTX 3090 GPUs.
To compare the computational cost, we report the average epoch time on Hopper in Table II
where
results of baselines (including DT (82) are from [18]. The formula for computational cost is:
avg_epoch_timee time_ttee______ of RIQL in [88]
evg_epoch time we_rum run TRACER.
avg_epoch time we rum RIQL
Note that TRACER requires long epoch time due to two main reasons:
1. Unlike RIQL and IQL, which learn oinedimnnsi action-valus functions, TRACER generates multiple samples for the estimation of action-value distributions. Following 43 generate 32 samples of action values for each state-action pair.
2. TRACER uses states, actions, and rewards as observations to update models, estimating the posterior of action-value distributions.
Table 11: Average epoch time.
Algorithm
BC
DT
EDAC
MSG
CQL
IQL
RIQL
TRACER (Ours)
Times (s)
3.8
28.9
14.1
12.0
22.8
8.7
9.2
19.4
E
More Related Work
Online RL. In general, standard online RL fall into two categories: model-free RL 83 85 and model-basee RL 86 [1]. In recent years, RL has achieved great success in many important real- world decision-making tasks [87]93] However, the online RL methods still typically rely on active data collection succeed, hindering their application in scenarios where real-time data collection is expensive, risky, and/or impractical. Thus, we focus on the offline RI, paradigm in this paper.
Code
We implement our codes in Python version 3.8 and make the code available online
G
Limitations and Negative Societal Impacts
TRACER's 's performance is related to the extent of corrupted data within the offline dataset. Although TRACER consistently outperforms RIQL even under conditions of extensive corrupted data (see Table its performance does degrade as the corruption rate (i.c., the extent/scale of corrupted data) increases. To tackle this problem, we look forward the continued development and optimization of corruption-roussi offline RL with large language models, introducing the prior knowledge of clean data against large-scale or near-total corrupted data. Thus, this corruption-robus offline RL can perform well even when faced with large-scale or near-total corrupted data in the offline dataset.
28

---

This paper proposes a novel approach called TRACER to advance the field of robustness in offline RL for diverse data corruptions, enhancing the potential of agents real-world applications. Although our primary focus is on technical innovation, we recognize the potential societal consequences of our work, as robustness offline RI for data corruptions has the potential to influence various domains such as autonomous driving and the large language model field. We are committed to ethical research practices and attach great importance to the social implications and ethical considerations in the development of robustness research in offline RL.
29

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We provide our main claims in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We provide the limitations of this work Appendix G and the second paragraph of Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
30

---

Justification:
We present all assumptions and proofs in Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide detailed information of the hyperparameters and implementation our approach (see Appendix (.2)) to facilitate the reproduction of our main results. We also provide the detailed experiment settings in Section 4.1 and Appendix C.I
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
31

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the data and code in Appendix F which can also be found in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc)
cublic/guuides/CodeSubbsss cosssiionooyyy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wwtps/cccppplgguuuiaaeeeae
www.ssssiicggoooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide detailed experiment settings in Section 4.1
Appendices C.1 and
B.2
Moreover, we provide details in Appendix C.4 for each result in Section 4.4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report standard deviations and standard errors for all results in the main text, including results in Tables 1 2
Figures 2 and
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
32

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the detailed information on the computer resources in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: httpp::////euriiee hww.rpsu...../. cwterips.ccpppuuueee ccc//ubllc/ttiiccccidde.l....
Answer: [Yes]
Justification: Our research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
33

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
All our experiment data is open source.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
34

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
35

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
36


Reviews:
[{"review_id": "TtWlxCBYZV", "reviewer": "Reviewer_V2Jx", "summary": "This paper seeks to conduct reinforcement learning from corrupted offline data. More specifically, they propose the TRACER algorithm, which uses bayesian inference to calculate the uncertainty in estimating the action-value function. The authors conduct experiments with diverse corruptions on CARLA and Mujoco environments.", "questions": "- Can you clarify the contribution of TRACER compared to RIQL? It seems like it is an orthogonal approach, but the two algorithms are not really compared in detail.\n- Can you show the hyperparameter tuning results? Is TRACER stable under different hyperparameter settings? How does it compare to RIQL in this regard?\n- How does the computational cost of TRACER compare to the other baselines?", "limitations": "The authors do discuss their method's limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper considers an important problem.\n- Their proposed algorithm is interesting and performs well.", "weaknesses": "- The contribution compared to other baselines (RIQL) is not really clear. In the introduction the authors claim that their method can handle simultaneous corruptions and RIQL cannot. However no experiments are done in this setting.\n- The experimental settings are fairly limited: only three Mujoco environments are considered and one carla experiment.\n- Different perturbations levels are not shown.\n- The authors do not discuss the computational cost of their method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's insightful and constructive comments and suggestions. We respond to each comment as follows and sincerely hope that our responses could properly address your concerns. If so, we would deeply appreciate it if you could raise your score. If not, please let us know your further concerns, and we will continue actively responding to the comments and enhancing our submission.\n- **Q1**. No experiments for simultaneous corruptions.\n  - **A1**. We apologize for the confusion caused by unclear descriptions in our main text. Specifically, the experiments presented in Table 1 on Page 7 use the simultaneous corruptions. The term \"**mixed corruptions**\" in the caption of Table 1 refers to the simultaneous corruptions (see the explanation on Line 233 of Section 4 in the main text). In our revision, we will replace \"mixed corruptions\" with \"simultaneous corruptions\" to avoid any ambiguity.\n- **Q2**. The contribution compared to other baselines (RIQL) is not really clear.\n  - **A2**. Thanks for the kind and insightful comment.\n    - **Relation between TRACER and RIQL**. Our approach TRACER applies the weighted imitation learning in RIQL for policy improvement (see Eq. (6) on Page 4 of the main text). This allows us to build and expand upon RIQL.\n    - **Contributions of TRACER**. We present TRACER's major contributions compared to corruption-robust offline RL methods [1,2] (especially RIQL):\n      1. To the best of our knowledge, TRACER introduces Bayesian inference into corruption-robust offline RL for **the first time**. Thus, it can capture uncertainty caused by diverse corrupted data to **simultaneously** handle the corruptions, unlike other corruption-robust offline RL methods that primarily focus on individual corruptions.\n      2. TRACER can **distinguish corrupted data from clean data** using an entropy-based uncertainty measure. Thus, TRACER can regulate the loss associated with corrupted data to reduce its influence. In contrast, existing corruption-robust offline RL methods lack the capability to identify which data is corrupt.\n      3. Based on Tables 1, 2, and 3 of the main text, TRACER significantly outperforms existing corruption-robust offline RL methods across a range of **both individual and simultaneous** data corruptions.\n    \n  [1] Corruption-robust offline reinforcement learning with general function approximation. NeurIPS 2023.\n\n  [2] Towards robust offline reinforcement learning under diverse data corruption. ICLR 2023.\n- **Q3**. The experimental settings are fairly limited.\n  - **A3**. Thanks for your insightful comment.\n    - Our experiments using Mujoco and CARLA datasets are consistent with standard practices in corruption-robust offline RL. Existing methods [1,2] often select these datasets to assess their effectiveness.\n    - Based on the corruption-robust offline RL methods, we further conduct experiments on two AntMaze datasets and two additional Mujoco datasets, presenting results under random simultaneous corruptions in Table G5 of the PDF of our global response.\n      - **Settings**. Each result represents the mean and standard error over four random seeds and 100 episodes in clean environments. For each experiment, the methods train agents using batch sizes of 64 for 3000 epochs. Building upon RIQL, we apply the experiment settings as follows.\n        1. For the two Mujoco datasets, we use a corruption rate of $c=0.3$ and scale of $\\epsilon=1.0$. Note that simultaneous corruptions with $c=0.3$ implies that approximately $76.0\\\\%$ of the data is corrupted.\n        2. For the two AntMaze datasets, we use the corruption rate of 0.2, corruption scales for observation (0.3), action (1.0), reward (30.0), and dynamics (0.3).\n      - **Results**. The results in Table G5 show that TRACER significantly **outperforms** other methods in **all these tasks** with the aforementioned AntMaze and Mujoco datasets.\n\n  [1] Corruption-robust offline reinforcement learning with general function approximation. NeurIPS 2023.\n\n  [2] Towards robust offline reinforcement learning under diverse data corruption. ICLR 2023.\n- **Q4**. Different perturbation/corruption levels are not shown.\n  - **A4**. Thanks for the kind and insightful comment.\n    - **Setting**. Building upon RIQL, we extend our experiments to include Mujoco datasets with different corruption levels, using different corruption rates and scales. We report the average scores and standard errors over four random seeds in Figure G2 of the PDF of our global response, using batch sizes of 256.\n    - **Results**. Figure G2 in the PDF shows that TRACER significantly outperforms other algorithms in **all tasks** under **random simultaneous corruptions**, achieving an average score improvement of ${\\bf +33.6\\\\%}$.\n- **Q5**. Computational cost of TRACER.\n  - **A5**. Please refer to A1 in our global response.\n- **Q6**. Can you show the hyperparameter tuning results?\n  - **A6**. Thank you for the meaningful comment.\n    - **Setting**. We conduct hyperparameter tuning experiments for both TRACER and RIQL, varying values of $\\kappa$ in the Huber loss and $\\alpha$ for action-value functions. We report results in Tables G6 and G7 of the PDF of our global response. Note that except the first column in Table G6, we use batch sizes of 64 and a learning rate of 0.0003 over four random seeds on Hopper task under adversarial simultaneous corruptions.\n    - **Results**. Tables G6 and G7 reveal that TRACER is more stable than RIQL and consistently outperforms RIQL, achieving an average performance gain of ${\\bf +43.3\\\\%}$ under adversarial simultaneous corruptions.\n\n      Moreover, we find that both TRACER and RIQL reach their respective highest performance at $\\kappa=0.1$ and $\\alpha=0.25$, and TRACER still achieves a substantial performance gain of ${\\bf +24.7 \\\\%}$ compared to RIQL.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cQ3Spja30k", "reviewer": "Reviewer_QZTE", "summary": "The paper introduces TRACER, a robust offline reinforcement learning (RL) algorithm designed to address the challenges posed by data corruptions in offline datasets. The corruptions can be\nrealized in form of states, actions, rewards, and dynamics corruption. The proposed methodology uses Bayesian inference to model and mitigate the impact of corrupted data on the action-value function.\nBy leveraging an entropy-based uncertainty measure, TRACER can distinguish between corrupted and clean data, thereby reducing the influence of the former and enhancing the robustness of the learned policies.\nThe paper provides thorough evaluation on MujoCo and Carla offline RL datasets.", "questions": "1) How relevant is the reward corruption given offline RL already is robust to noise in the reward function as it mimics the actions for datasets with length bias? Please refer to ref 1\n2) The authors aim to leverage all elements in the dataset as observations. Do the authors assume that complete traces of the system are available for which the offline policy is being learned? \nIn situations of dynamic corruption these traces might be different than the system. For example s_t in the corrupted dataset may not be observed in the real system. How are such cases \nhandled?\n3) As the method learns action-value function D from the data while reducing the influence of corrupted data. What amount of clean data needs to be present in the dataset for this method \nto work?\n4) What is the reason for not using projected gradient descent for the reward corruption? Also could the author provide some understanding of why the method outperforms RIQL under certain settings and why it does not in some cases?\n5) Please also respond to the weaknesses.\n\n1. Li, Anqi, et al. \"Survival instinct in offline reinforcement learning.\" Advances in neural information processing systems 36 (2024).", "limitations": "Limitations: The authors discuss the limitation in terms of handling realistic noises in conclusion. However, a limitation is also on the dependence on clean observations present in the dataset when some elements are corrupted.\nThe authors do not discuss any negative sociatal impact of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper proposes an interesting method of down weighting the contribution corrupted data while learning action value function from the offline dataset using entropy as an uncertainty measure.\n2. The paper uses a Bayesian approach to learn the action value function from the data which aids learning from uncorrupted elements in the dataset.\n3. The authors provide extensive experimental validation across multiple tasks and corruption types, showing mostly consistent performance improvements over other methods. The paper also \nreports ablation studies of using entropy showing potential improvement from the proposed methodology.", "weaknesses": "1. It is not clear how the final policy is extracted after learning $D_{\\theta}$.\n2. The paper relies on many hyperparameters. What kind of kernel is used in the gaussian distribution learning in Eq 10? Is there a upper bound on what amount of corruption the method will be able to handle? \n3. While the authors consider individual corruption of one element at a time data collected from in a real world system can have simultaneous corruptions. For\nexample corrupted action due to adversarial noise will lead the system to different state causing simultaneous corruptions.\n4. Notational clarity: The paper introduces a lot of notations which is difficult to keep track of. Specially in the following contexts :\na :  The paper introduces LQ (θi), LV(ψ), Lπ(ϕ) in Eq 4, 5 and 6 but does not discuss how they relate to Dθ. Is the θ used for learning Q function and D same?\nb : The paper introduces a new notation qπ in Eq 18 called value distribution. Is this the same as action value distribution? RIQL already proves the robustness of IQL policy with respect of data corruption. What is \nthe objective of Theorem A.3? It will be helpful to add a discussion on the theoretical analysis. Also Providing a table summarizing the notations in appendix will enhance readability of the text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful comments. We respond to each comment as follows and sincerely hope that our responses could properly address your concerns. If so, we would deeply appreciate it if you could raise your score. If not, please let us know your further concerns, and we will continue actively responding to the comments and enhancing our submission.\n- **Q1**. (1) The paper introduces LQ (θi), LV(ψ), Lπ(ϕ) in Eq 4, 5 and 6 but does not discuss how they relate to Dθ. Is θ for learning Q function and D same? (2) How is the final policy extracted after learning Dθ.\n  - **A1**. We apologize for any confusion caused by unclear descriptions in our main text.\n    1. Relation between $Q$, $D$, and $\\theta$.\n        - We use **the same $\\theta$** to learn $D$ and then estimate $Q$. Specifically, following [1], we have\n        $$Q_{\\theta_i}(s,a) = \\sum_{n=1}^N D_{\\theta_i}^{\\tau_n}(s,a,r).$$\n          See Lines 167-172 of the main text for more details regarding the notations.\n\n          Similarly, we also have\n          $$V_{\\psi}(s) = \\sum_{n=1}^N Z_{\\psi}^{\\tau_n}(s).$$\n        - With the relation between $Q$ and $D$, $V$ and $Z$, we use Eq. (4) to derive $\\delta$ in Eq. (12) and Eq. (5) to derive Eq. (13) in our main text.\n    2. Policy Learning.\n        - Following the weighted imitation learning in RIQL and IQL, we use Eq. (6) in our main text to learn the policy. Details for notations are shown in Lines 122-123 of the main text.\n\n  [1] Implicit quantile networks for distributional reinforcement learning.\n- **Q2**. The paper relies on many hyperparameters.\n  - **A2**. Compared to RIQL, one of SOTA methods, our approach TRACER only introduces additional hyperparameters associated with the approximation of action-value distributions: (1) the number $N$ of samples $\\tau$; (2) a linear decay parameter $\\beta$ used to trade off the losses $\\mathcal{L}\\_{\\text{first}}$ and $\\mathcal{L}\\_{\\text{second}}$.\n- **Q3**. What kind of kernel is used in Gaussian distribution of Eq 10?\n  - **A3**. We do **not** use a Gaussian kernel in Eq. (10). See the explanation of Eq. (10) in A2 of our global response for more details.\n- **Q4**. Is there a upper bound on what amount of corruption the method can handle?\n  - **A4**. Yes. Please refer to A4 in our global response.\n- **Q5**. While authors consider individual corruptions, data in the real world can have simultaneous corruptions.\n  - **A5**. We specifically design TRACER to handle the challenging simultaneous corruptions. Results in Table 1 of the main text show that TRACER outperforms several SOTAs in **all tasks under simultaneous corruptions**, achieving an average gain of ${\\bf +21.1\\\\%}$. Note that \"mixed corruptions\" in the main text refers to simultaneous corruptions (see Line 233 in the main text).\n- **Q6**. The paper introduces $q^{\\pi}$ in Eq 18. Is this the same as action value distribution?\n  - **A6**. No. The notation $q^\\pi$ in Eq. (18) of the Appendix denotes the value distribution, consistent with $p(\\cdot|s)$ used in Line 170 of the main text, which is an expactation of action value distribution $\\mathbb{E}_{a\\sim \\pi, r\\sim \\rho} [p\\_\\theta(\\cdot|s,a,r)]$.\n- **Q7**. RIQL already proves the robustness of IQL policy against data corruption. What is the objective of Theorem A.3?\n  - **A7**. Please refer to A4 in our global response.\n- **Q8**. Providing a table summarizing the notations in appendix will enhance readability.\n  - **A8**. Thanks for the insightful comment. We will include this table in our revision.\n- **Q9**. How relevant is the reward corruption given offline RL already robust to noise in the reward function?\n  - **A9**. While the offline RL is robust to small-scale random reward corruptions [1], it tends to struggle with large-scale reward corruptions (see Page 21 in RIQL).\n\n  [1] Survival instinct in offline reinforcement learning.\n- **Q10**. Do the authors assume that complete traces of the system are available?\n  - **A10**. No. We employ the commonly used offline RL setting [1], where the offline dataset consists of shuffled tuples and agents only use individual tuples rather than complete traces.\n\n  [1] Off-policy deep reinforcement learning without exploration.\n- **Q11**. In situations of dynamic corruption these traces might be different than the system. For example, $s_t$ in the corrupted dataset may not be observed in the real system. How are such cases handled?\n  - **A11**. Thanks for the insightful comment. In a scenario where an element (e.g., state) may be corrupted, TRACER captures uncertainty by using (1) other elements and (2) correlations between all elements and action values (see Lines 53-60 of the main text).\n- **Q12**. What amount of clean data needs to be present in the dataset for this method to work?\n  - **A12**. Please refer to A4 in our global response. Results in Table G4 show that while TRACER is robust to simultaneous corruptions, its performance depends on the extent of corrupted data it encounters.\n- **Q13**. What is the reason for not using projected gradient descent for reward corruptions?\n  - **A13**. Following RIQL, the objective of adversarial reward corruption is $\\hat{r} = \\min_{\\hat{r} \\in \\mathbb{B}(r, \\epsilon)} \\hat{r} + \\gamma \\mathbb{E}[Q(s', a')]$. Here $\\mathbb{B}(r, \\epsilon)=\\\\{\\hat{r}\\mid |\\hat{r} - r| \\leq (1+\\epsilon)\\cdot r_{\\max} \\\\}$ regularizes the maximum distance for rewards. Thus, we can directly compute $\\hat{r} =-\\epsilon\\times r$ without using projected gradient descent.\n- **Q14**. Could the author provide some understanding of why the method outperforms RIQL under certain settings and why it does not in some cases?\n  - **A14**. Please refer to A3 in our global response.\n- **Q15**. Limitations.\n  - **A15**. We look forward to developing corruption-robust offline RL with large language models, introducing the prior knowledge of clean data against large-scale or near-total corrupted data. Moreover, we plan to add potential negative societal impacts in our revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xk1H3TnJwf", "reviewer": "Reviewer_wmJN", "summary": "This paper presents a novel approach called TRACER, aimed at addressing the challenge of learning robust policies from offline datasets that are subject to various data corruptions. The key contribution of this work lies in the integration of Bayesian inference to capture uncertainty within the offline data. TRACER models data corruptions as uncertainty in the action-value function and approximates the posterior distribution of this function. Furthermore, it employs an entropy-based uncertainty measure to distinguish between corrupted and clean data, thereby regulating the loss associated with corrupted data to enhance robustness in clean environments. Experimental results indicate that TRACER outperforms SOTA methods in handling both individual and simultaneous data corruptions.", "questions": "- The proposed method is complicated. Can this method be simplified?\n\n- The results in the main tables (Table 2 and Table 3) appear similar to those of RIQL, raising concerns about the necessity of the included components.\n\n- Another issue is the readability and clarity of the writing discussed above.\n\n- Why does the accuracy of the measurement for corrupted data decreases during training in Figure 3?\n\n- Moreover, I suggest the authors provide the computational cost comparison with prior work.", "limitations": "This paper acknowledges its limitations; however, I identified an additional concern regarding the complexity of the proposed method.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The studied problem is important, and the paper offers a reasonable solution.\n- Theoretical guarantees are provided.\n- The empirical results show considerable improvement over prior work.", "weaknesses": "- Though I like the insight from Variational Bayesian inference, my major concern is that the method is quite complicated, incorporating numerous components and designs such as the distributional value function, variational inference, and entropy-weighted loss function. This complexity makes this method less preferable in practice.\n\n- Moreover, the results in the main tables (Table 2 and Table 3) appear similar to those of RIQL, raising concerns about the necessity of the included components.\n\n- Another issue is the readability and clarity of the writing. For example, Equations 10 and 11 lack necessary explanation and insight. Additionally, the implementation details for $\\phi_a, \\phi_b, \\phi_c$ are missing, leaving readers unsure of how the method works in practice. Furthermore, Equation 10 seems to include $\\theta$ as an input unnecessarily, as the right side does not depend on $\\theta$.\n\n- The analysis of the experiments is not thorough. For instance, the accuracy of the measurement for corrupted data decreases during training in Figure 3, which is not discussed and seems unreasonable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's insightful and constructive comments and suggestions. We respond to each comment as follows and sincerely hope that our responses could properly address your concerns. If so, we would deeply appreciate it if you could raise your score. If not, please let us know your further concerns, and we will continue actively responding to the comments and enhancing our submission.\n- **Q1**. The proposed method is complicated. Can this method be simplified?\n  - **A1**. Thanks for your meaningful comment.\n    - On the actor-critic architecture based on IQL, our approach TRACER adds just one ensemble model $(p_{\\varphi_a},p_{\\varphi_r},p_{\\varphi_s})$ to reconstruct the data distribution and repace the function approximation in the critic with a distribution estimation (see Figure G1 in the PDF of our global response). Therefore, the network structure is not complicated, while significantly improving the ability to handle both simultaneous and individual corruptions.\n    - It is worth noting that our primary aim is the development of corruption-robust offline RL techniques for diverse corruptions. To the best of our knowledge, this study introduces Bayesian inference into corruption-robust offline RL for the first time, prioritizing novelty and robust performance in challenging scenarios with simultaneous corruptions.\n    - We thank you for reminding us of the importance of the balance between complexity and usability. In future work, we plan to refine TRACER by estimating uncertainty directly within the representation space. Thus, we can simply learn action-value functions without using distributional RL. This advancement will potentially broaden the applications of TRACER in real-world scenarios.\n- **Q2**. The results in the main tables (Table 2 and Table 3) appear similar to those of RIQL, raising concerns about the necessity of the included components.\n  - **A2**. Thanks for the insightful comment. Please refer to A3 in our global response.\n- **Q3**. Another issue is the readability and clarity of the writing. For example, Equations 10 and 11 lack necessary explanation and insight. Additionally, the implementation details for $\\varphi_a$, $\\varphi_r$, $\\varphi_s$ are missing, leaving readers unsure of how the method works in practice.\n  - **A3**. Please refer to A2 in our global response.\n- **Q4**. Furthermore, Equation 10 seems to include $\\theta$ as an input unnecessarily, as the right side does not depend on $\\theta$.\n  - **A4**. Here is our explanation.\n    - The goal of Eq. (10) of our main text is to optimize $(\\theta, \\varphi_a,\\varphi_r,\\varphi_s)$, thus minimizing the difference between $p_{\\varphi_a}(A|D_\\theta,S,R,S')$ and $\\pi_{\\mathcal{B}}(A|S)$, $p_{\\varphi_r}(R|D_\\theta,S,A)$ and $\\rho_{\\mathcal{B}}(R|S,A)$, and $p_{\\varphi_s}(S|D_\\theta,A,R)$ and $p_{\\mathcal{B}}(S)$. More details are in A2 of the global response.\n    - In Eq. (10), the input of each $(\\mu_{\\varphi}, \\Sigma_{\\varphi})$ includes $D_\\theta$ with the parameter $\\theta$.\n- **Q5**. The analysis of experiments is not thorough. Why does the accuracy of the measurement for corrupted data decrease during training in Figure 3?\n  - **A5**. Thanks for your meaningful comment.\n    - We apologize for any confusion caused by the unclear description and presentation of the validation experiments in Figure 3 of our main text. To clearly illustrate the measurement accuracy of agents learned by TRACER with respect to uncertainty during training, we further conduct validation experiments on Walker2d-Medium-Replay-v2.\n\n      Specifically, we evaluate the accuracy every 50 epochs over 3000 epochs. For each evaluation, we sample 500 batches to compute the average entropy of corrupted and clean data. Each batch consists of 32 clean and 32 corrupted data. We illustrate the curves over three seeds in Figure G3 of the PDF of our global response, where each point shows how many of the 500 batches have higher entropy for corrupted data than that of clean data.\n    - Figure G3 illustrates an oscillating upward trend of TRACER's measurement accuracy using entropy under simultaneous corruptions, demonstrating that using the entropy-based uncertainty measure can effectively distinguish corrupted data from clean data.\n- **Q6**. Moreover, I suggest the authors provide the computational cost comparison with prior work.\n  - **A6**. Please refer to A1 in the global response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0NSt4CdXfI", "reviewer": "Reviewer_vWB5", "summary": "The authors introduce TRACER, a new Bayesian methodology to capture the uncertainty via offline data for robustness against all types of data corruptions. An appealing feature of TRACER is that it can distinguish corrupted data from clean data using an entropy-based uncertainty measure. Experiments are provided to prove the effectiveness of such a methodology.", "questions": "I'd like to politely ask the authors to expand the related work with the above references [1,2]. In addition, would it be fair to say that, on page 3, the expectation that finds $\\pi^\\star$ is an integral taken with respect to the joint probability given by all the sources of randomness $P_0, \\pi(\\cdot \\mid s_t), \\rho(\\cdot \\mid s_t,a_t), P(\\cdot \\mid s_t,a_t)$? \n\nIn equation (1), the closed parenthesis ) should go after $a_0=a$.", "limitations": "The limitations are explicitly acknowledged by the authors in the conclusion.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Very clearly stating the problem, and convincing the reader that's a relevant one. Compelling method to address such a problem.", "weaknesses": "The authors missed some important references: [1,2] introduce credal Bayesian deep learning, which is a robust way of performing Bayesian inference via Bayesian Neural Networks (where posterior and predictive distributions are approximated via variational inference) taking into account different types of uncertainty (using so-called credal sets, i.e. closed and convex sets of prior and likelihood probabilities), and quantifying them using entropy-based uncertainty measures. In the future, the authors may also look into [3], which seems like a compelling work for the research venue explored in the paper.\n\n[1] https://arxiv.org/abs/2302.09656\n\n[2] https://link.springer.com/chapter/10.1007/978-3-031-57963-9_1#:~:text=In%20their%20seminal%201990%20paper,bound%20to%20hold%20with%20equality\n\n[3] https://arxiv.org/abs/2308.14815", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s insightful and constructive feedback. We have carefully addressed these concerns and accordingly revised the manuscript.\nThese comments have not only facilitated significant improvements in our manuscript but have also inspired us for further in-depth studies in our future research.\n\n- **Q1**. The authors missed some important references [1,2]. In the future, the authors may also look into [3], which seems like a compelling work for the research venue explored in the paper.\n  - **A1**. Thank you for pointing out these crucial references. The Shannon entropy for the measures of aleatoric and epistemic uncertainties discussed in these works [1,2] provides important insight and support for our method. We will study these references thoroughly and consider incorporating their findings into our future research. We also appreciate the suggestion to explore the recent work in [3], which could further enhance our understanding and methodology. We will include these additions in our revised manuscript.\n\n  [1] Credal Bayesian Deep Learning. 2023.\n\n  [2] A Novel Bayes' Theorem for Upper Probabilities. Epi UAI 2024.\n\n  [3] Distributionally Robust Statistical Verification with Imprecise Neural Networks. 2023.\n\n- **Q2**. In addition, would it be fair to say that, on page 3, the expectation that finds $\\pi^*$ is an integral taken with respect to the joint probability given by all the sources of randomness $P_0$, $\\pi (\\cdot | s_t)$, $\\rho (\\cdot | s_t, a_t)$, $P(\\cdot |s_t, a_t)$?\n  - **A2**. Yes. On page 3, the expectation that determines $\\pi^*$ involves an integral taken with respect to the joint probability distribution influenced by all sources of randomness, including $P_0$, $\\pi(\\cdot|s_t)$, $\\rho(\\cdot|s_t,a_t)$, and $P(\\cdot|s_t,a_t)$ [1,2].\n\n  [1] Trust Region Policy Optimization. ICML 2015.\n\n  [2] Distributional reinforcement learning with quantile regression. AAAI 2018.\n\n- **Q3**. In equation (1), the closed parenthesis ) should go after $a_0=a$.\n  - **A3**. Thank you for catching the typo in Eq. (1). We will carefully correct this in our revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Global Response
We would like to thank reviewers for their insightful comments. We respond to the collective feedback below and hope that our responses could adequately address these general concerns. If so, we would deeply appreciate it if reviewers could raise the score. If not, please let us know the further concerns, and we will continue to refine our submission in response to the comments.
- **Q1** for **#R V2Jx and wmJN**. Computational cost comparison.
  - **A1**. We provided the average training duration of our approach TRACER for Halfcheetah, Walker2d, and Hopper in Section B.5 of our Appendix.
  - To compare the computational cost, we report the average epoch time on Hopper in Table G1 of the PDF, where results of baselines (including DT [3]) are from [1]. The formula for computational cost is:
    $$\frac{\text{avg epoch time of RIQL in [1]}}{\text{avg epoch time we run RIQL}} \times \text{avg epoch time we run TRACER}.$$
    Note that TRACER requires a long epoch time due to two main reasons:
    1. Unlike RIQL and IQL, which learn one-dimensional action-value functions, TRACER generates multiple samples for the estimation of action-value distributions. Following [2], we generate 32 samples of action values for each state-action pair.
    2. TRACER uses states, actions, and rewards as observations to update models, estimating the posterior of action-value distributions.

    In the future work, we plan to improve TRACER’s computational efficiency by optimizing codes to estimate the posterior in parallel using various observations.

  [1] Towards robust offline reinforcement learning under diverse data corruption.

  [2] Implicit quantile networks for distributional reinforcement learning.

  [3] Decision transformer: Reinforcement learning via sequence modeling.
- **Q2** for **#R QZTE and wmJN**. Explanation for Eqs. (10) and (11).
  - **A2**. We apologize for any confusion caused by unclear descriptions for Eqs. (10) and (11).
    - The goal of **Eq. (10)** is to estimate $\pi_{\mathcal{B}}(A|S)$, $\rho_{\mathcal{B}}(R|S,A)$, and $p_{\mathcal{B}}(S)$ using $p_{\varphi_a}(A|D_\theta,S,R,S')$, $p_{\varphi_r}(R|D_\theta,S,A)$, and $p_{\varphi_s}(S|D_\theta,A,R)$, respectively. We model all these distributions as Gaussian distributions, and use the mean $\mu_{\varphi}$ and standard deviation $\Sigma_{\varphi}$ to represent the corresponding $p_{\varphi}$. **For implementation**, we employ MLPs to output each $(\mu_{\varphi},\Sigma_{\varphi_r})$ using the corresponding conditions of $p_{\varphi}$. Then, based on the KL divergence between two Gaussian distributions, we can derive Eq. (10).
    - The goal of **Eq. (11)** is to maximize the likelihoods of $D_\theta$ given samples $\hat{s} \sim p_{\varphi_s}$, $\hat{a} \sim p_{\varphi_a}$, or $\hat{r} \sim p_{\varphi_r}$. Thus, with $(s,a,r)\sim \mathcal{B}$, we propose minimizing the distance between $D_\theta (\hat{s},a,r)$ and $D(s,a,r)$, $D_\theta (s,\hat{a},r)$ and $D(s,a,r)$, and $D_\theta (s,a,\hat{r})$ and $D(s,a,r)$, where $\hat{s} \sim p_{\varphi_s}$, $\hat{a} \sim p_{\varphi_a}$, and $\hat{r} \sim p_{\varphi_r}$, thus deriving Eq. (11).
- **Q3** for **#R QZTE and wmJN**. Results comparison between TRACER and RIQL.
  - **A3**. Thanks for the insightful comment.
  - **Simultaneous Corruptions**. Our method TRACER is specifically designed to address simultaneous corruptions for robustness. Results in Table 1 of the main text show that TRACER outperforms RIQL across **all tasks** under simultaneous corruptions, achieving an average gain of ${\bf +21.1\\%}$.

    This is because that TRACER captures uncertainty via offline data against simultaneous corruptions. Thus, it can use uncertainty to distinguish corrupted data from clean data and then reduce the influence of corrupted data.
  - **Individual Corruptions**. In Tables 2 and 3 of the main text, we adhered to commonly used settings for individual corruptions in corruption-robust offline RL. We directly followed hyperparameters from RIQL (i.e., $\kappa$ for huber loss, the ensemble number $K$, and $\alpha$ in action-value functions). Results show that TRACER outperforms RIQL in **18 out of 24** settings, demonstrating its robustness even when aligned with RIQL’s hyperparameters.

    Further, we explore hyperparameter tuning, specifically of $\kappa$, on Hopper to improve TRACER's performance. This results in TRACER outperforming RIQL in **7 out of 8** settings on Hopper, up from 5 settings (see Tables G2 and G3 in the PDF). The further improvement highlights TRACER’s potential to achieve greater performance gains.
    - Based on Table G2, we find that TRACER requires low $\kappa$ in Huber loss, using L1 loss for large errors. Thus, TRACER can linearly penalize the corrupted data and reduce its influence on the overall model fit.
- **Q4** for **#R QZTE**. (1) RIQL already proves the robustness of IQL. What is the objective of Theorem A.3? (2) Is there a upper bound on what amount of corruption the method will be able to handle?
  - **A4**.
    1. Theorem A.3 in our Appendix builds upon RIQL and extends it to TRACER. Specifically, RIQL first proves an upper bound on **the distance in value functions** that IQL can learn under clean and corrupted data. Then, we provide Theorem A.3 to prove an upper bound on **the difference in value distributions** that TRACER can learn under clean and corrupted data. This theorem not only supports the robustness claims of TRACER but also provides a guarantee of how TRACER's performance degrades with increased data corruptions.
    2. Yes. Theorem A.3 shows that the higher the scale of corrupted data, the greater the difference in action-value distributions and the lower the TRACER's performance. We also evaluate TRACER across various corruption levels. Table G4 in the PDF shows that while TRACER is robust to simultaneous corruptions, its scores depend on the extent of corrupted data it encounters.

Meta-review: The reviewers agreed that the paper addresses an important problem of robust reinforcement learning under corrupted data, proposes an interesting method based on Bayesian inference, and the extensive empirical evaluation demonstrates the effectiveness of the proposed method. However, the reviewers also raised several concerns and questions in their initial reviews. We want to thank the authors for their responses and active engagement during the discussion phase. The reviewers appreciated the responses, which helped in answering their key questions. The reviewers have an overall positive assessment of the paper, and there is a consensus for acceptance. The reviewers have provided detailed feedback, and we strongly encourage the authors to incorporate this feedback when preparing the final version of the paper.
Decision: Accept (poster)