Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptions

Authors: rui  yang
jie  wang
guoping  wu
bin  li
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 12:30:21

Abstract: Real-world offline datasets are often subject to data corruptions (such as noise or adversarial attacks) due to sensor failures or malicious attacks. Despite advances in robust offline reinforcement learning (RL), existing methods struggle to learn robust agents under high uncertainty caused by the diverse corrupted data (i.e., corrupted states, actions, rewards, and dynamics), leading to performance degradation in clean environments. To tackle this problem, we propose a novel robust variational Bayesian inference for offline RL (TRACER). It introduces Bayesian inference for the first time to capture the uncertainty via offline data for robustness against all types of data corruptions. Specifically, TRACER first models all corruptions as the uncertainty in the action-value function. Then, to capture such uncertainty, it uses all offline data as the observations to approximate the posterior distribution of the action-value function under a Bayesian inference framework. An appealing feature of TRACER is that it can distinguish corrupted data from clean data using an entropy-based uncertainty measure, since corrupted data often induces higher uncertainty and entropy. Based on the aforementioned measure, TRACER can regulate the loss associated with corrupted data to reduce its influence, thereby enhancing robustness and performance in clean environments. Experiments demonstrate that TRACER significantly outperforms several state-of-the-art approaches across both individual and simultaneous data corruptions.

Main paper: 
 introduction: 
 Offline reinforcement learning (RL) aims to learn an effective policy from a fixed dataset without direct interaction with the environment [1, 2] . This paradigm has recently attracted much attention in scenarios where real-time data collection is expensive, risky, or impractical, such as in healthcare [3] , autonomous driving [4] , and industrial automation [5] . Due to the restriction of the dataset, offline RL confronts the challenge of distribution shift between the policy represented in the offline dataset and the policy being learned, which often leads to the overestimation for out-of-distribution (OOD) actions [1, 6, 7] . To address this challenge, one of the promising approaches introduce uncertainty estimation techniques, such as using the ensemble of action-value functions or Bayesian inference to measure the uncertainty of the dynamics model [8] [9] [10] [11] or the action-value function [12] [13] [14] [15] regarding the rewards and transition dynamics. Therefore, they can constrain the learned policy to remain close to the policy represented in the dataset, guiding the policy to be robust against OOD actions.Figure 1 : Graphical model of decision-making process. Nodes connected by solid lines denote data points in the offline dataset, while the Q values (i.e., action values) connected by dashed lines are not part of the dataset. These Q values are often objectives that offline algorithms aim to approximate.Nevertheless, in the real world, the dataset collected by sensors or humans may be subject to extensive and diverse corruptions [16] [17] [18] , e.g., random noise from sensor failures or adversarial attacks during RLHF data collection. Offline RL methods often assume that the dataset is clean and representative of the environment. Thus, when the data is corrupted, the methods experience performance degradation in the clean environment, as they often constrain policies close to the corrupted data distribution.Despite advances in robust offline RL [2] , these approaches struggle to address the challenges posed by diverse data corruptions [18] . Specifically, many previous methods on robust offline RL aim to enhance the testing-time robustness, learning from clean datasets and defending against attacks during testing [19] [20] [21] . However, they cannot exhibit robust performance using offline dataset with perturbations while evaluating the agent in a clean environment. Some related works for data corruptions (also known as corruption-robust offline RL methods) introduce statistical robustness and stability certification to improve performance, but they primarily focus on enhancing robustness against adversarial attacks [16, 22, 23] . Other approaches focus on the robustness against both random noise and adversarial attacks, but they often aim to address only corruptions in states, rewards, or transition dynamics [24, 17] . Based on these methods, recent work [18] extends the data corruptions to all four elements in the dataset, including states, actions, rewards, and dynamics. This work demonstrates the superiority of the supervised policy learning scheme [25, 26] for the data corruption of each element in the dataset. However, as it does not take into account the uncertainty in decisionmaking caused by the simultaneous presence of diverse corrupted data, this work still encounters difficulties in learning robust agents, limiting its applications in real-world scenarios.In this paper, we propose to use offline data as the observations, thus leveraging their correlations to capture the uncertainty induced by all corrupted data. Considering that (1) diverse corruptions may introduce uncertainties into all elements in the offline dataset, and (2) each element is correlated with the action values (see dashed lines in Figure 1 ), there is high uncertainty in approximating the action-value function by using various corrupted data. To address this high uncertainty, we propose to leverage all elements in the dataset as observations, based on the graphical model in Figure 1 . By using the high correlations between these observations and the action values [27], we can accurately identify the uncertainty of the action-value function.Motivated by this idea, we propose a robust variational Bayesian inference for offline RL (TRACER) to capture the uncertainty via offline data against all types of data corruptions. Specifically, TRACER first models all data corruptions as uncertainty in the action-value function. Then, to capture such uncertainty, it introduces variational Bayesian inference [28] , which uses all offline data as observations to approximate the posterior distribution of the action-value function. Moreover, the corrupted observed data often induce higher uncertainty than clean data, resulting in higher entropy in the distribution of action-value function. Thus, TRACER can use the entropy as an uncertainty measure to effectively distinguish corrupted data from clean data. Based on the entropy-based uncertainty measure, it can regulate the loss associated with corrupted data in approximating the action-value distribution. This approach effectively reduces the influence of corrupted samples, enhancing robustness and performance in clean environments.This study introduces Bayesian inference into offline RL for data corruptions. It significantly captures the uncertainty caused by diverse corrupted data, thereby improving both robustness and performance in offline RL. Moreover, it is important to note that, unlike traditional Bayesian online and offline RL methods that only model uncertainty from rewards and dynamics [29] [30] [31] [32] [33] [34] [35] , our approach identifies the uncertainty of the action-value function regarding states, actions, rewards, and dynamics under data corruptions. We summarize our contributions as follows.• To the best of our knowledge, this study introduces Bayesian inference into corruption-robust offline RL for the first time. By leveraging all offline data as observations, it can capture uncertainty in the action-value function caused by diverse corrupted data. • By introducing an entropy-based uncertainty measure, TRACER can distinguish corrupted from clean data, thereby regulating the loss associated with corrupted samples to reduce its influence for robustness. • Experiment results show that TRACER significantly outperforms several state-of-the-art offline RL methods across a range of both individual and simultaneous data corruptions. 
preliminaries: 
 Bayesian RL. We consider a Markov decision process (MDP), denoted by a tuple M = (S, A, R, P, P 0 , γ), where S is the state space, A is the action space, R is the reward space, P (•|s, a) ∈ P(S) is the transition probability distribution over next states conditioned on a stateaction pair (s, a), P 0 (•) ∈ P(S) is the probability distribution of initial states, and γ ∈ [0, 1) is the discount factor. Note that P(S) and P(A) denote the sets of probability distributions on subsets of S and A, respectively. For simplicity, throughout the paper, we use uppercase letters to refer to random variables and lowercase letters to denote values taken by the random variables. Specifically, R(s, a) denotes the random variable of one-step reward following the distribution ρ(r|s, a), and r(s, a) represents a value of this random variable. We assume that the random variable of one-step rewards and their expectations are bounded by R max and r max for any (s, a) ∈ S × A, respectively.Our goal is to learn a policy that maximizes the expected discounted cumulative return:π * = arg max π∈P(A) E s0∼P0,at∼π(•|st),R∼ρ(•|st,at),st+1∼P (•|st,at) ∞ t=0 γ t R(s t , a t ) .Based on the return, we can define the value function asV π (s) = E π,ρ,P [ ∞ t=0 γ t R(s t , a t )|s 0 = s], the action-value function as Q π (s, a) = E R∼ρ(•|s,a),s ′ ∼P (•|s,a) [R(s, a) + γV π (s ′ )],and the actionvalue distribution [36] asD π (s, a) = ∞ t=0 γ t R(s t , a t |s 0 = s, a 0 = a), with s t+1 ∼ P (•|s t , a t ), a t+1 ∼ π(•|s t+1 ). (1) Note that V π (s) = E a∼π [Q π (s, a)] = E a∼π,ρ [D π (s, a)].Variational Inference. Variational inference is a powerful method for approximating complex posterior distributions, which is effective for RL to handle the parameter uncertainty and deal with modelling errors [36] . Given an observation X and latent variables Z, Bayesian inference aims to compute the posterior distribution p(Z|X). Direct computation of this posterior is often intractable due to the high-dimensional integrals involved. To approximate the true posterior, Bayesian inference introduces a parameterized distribution q(Z; ϕ) and minimizes the Kullback-Leibler (KL) divergence D KL (q(Z; ϕ)∥p(Z|X)). Note that minimizing the KL divergence is equivalent to maximizing the evidence lower bound (ELBO) [37, 38] :ELBO(ϕ) = E q(Z;ϕ) [log p(X, Z) -log q(Z; ϕ)].Offline RL under Diverse Data Corruptions. In the real world, the data collected by sensors or humans may be subject to diverse corruption due to sensor failures or malicious attacks. Let b and B denotes the uncorrupted and corrupted dataset with samples {(s i t , a i t , r i t , s i t+1 )} N i=1 , respectively. Each data in B may be corrupted. We assume that an uncorrupted state follows a state distribution p b (•), a corrupted state follows p B (•), an uncorrupted action follows a behavior policy π b (•|s i t ), a corrupted action is sampled from π B (•|s i t ), a corrupted reward is sampled from ρ B (•|s i t , a i t ), and a corrupted next state is drawn from P B (•|s i t , a i t ). We also denote the uncorrupted and corrupted empirical state-action distributions as p b (s i t , a i t ) and p B (s i t , a i t ), respectively. Moreover, we introduce the notations [18, 39] as follows.EQUATIONT D(s, a) :EQUATIONfor any (s, a) ∈ S × A and Q : S × A → [0, r max /(1 -γ)], where X : D = Y denotes equality of probability laws, that is the random variable X is distributed according to the same law as Y .To address the diverse data corruptions, based on IQL [26] , RIQL [18] introduces quantile estimators with an ensemble of action-value functions {Q θi (s, a)} K i=1 and employs a Huber regression [40] :EQUATIONEQUATION)Note that l κ H is the Huber loss, and Q α is the α-quantile value among {Q θi (s, a)} K i=1 . RIQL then follows IQL [26] to learn the policy using weighted imitation learning with a hyperparameter β:EQUATION)3 AlgorithmWe first introduce the Bayesian inference for capturing the uncertainty caused by diverse corrupted data in Section 3.1. Then, we provide our algorithm TRACER with the entropy-based uncertainty measure in Section 3.2. Moreover, we provide the theoretical analysis for robustness, the architecture, and the detailed implementation of TRACER in Appendices A.1, B.1, and B.2, respectively. 
variational inference for uncertainty induced by corrupted data: 
 We focus on corruption-robust offline RL to learn an agent under diverse data corruptions, i.e., random or adversarial attacks on four elements of the dataset. We propose to use all elements as observations, leveraging the data correlations to simultaneously address the uncertainty. By introducing Bayesian inference framework, our aim is to approximate the posterior distribution of the action-value function.At the beginning, based on the relationships between the action values and the four elements (i.e., states, actions, rewards, next states) in the offline dataset as shown in Figure 1 , we defineD θ = D θ (S, A, R) ∼ p θ (•|S, A, R), parameterized by θ. Building on the action-value distribution, we can explore how to estimate the posterior of D θ using the elements available in the offline data.Firstly, we start from the actions {a i t } N i=1 following the corrupted distribution π B and use them as observations to approximate the posterior of the action-value distribution under a variational inference. As the actions are correlated with the action values and all other elements in the dataset, the likelihood is p φa (A|D, S, R, S ′ ), parameterized by φ a . Then, under the variational inference framework, we maximize the posterior and derive to minimize the loss function based on ELBO:EQUATIONwhere S, R, and S ′ follow the offline data distributions p B , ρ B , and P B , respectively. Secondly, we apply the rewards {r i t } N i=1 drawn from the corrupted reward distribution ρ B as the observations. Considering that the rewards are related to the states, actions, and action values, we model the likelihood as p φr (R|D, S, A), parameterized by φ r . Therefore, we can derive a loss function by following Equation (7) :EQUATIONwhere S and A follow the offline data distributions p B and π B , respectively. Finally, we employ the state {s i t } N i=1 in the offline dataset following the corrupted distribution p B as the observations. Due to the relation of the states, we can model the likelihood as p φs (S|D, A, R), parameterized by φ r . We then have the loss function:EQUATIONwhere A and R follow the offline data distributions π B and ρ B , respectively. We present the detailed derivation process in Appendix A.2.The goal of first terms in Equations ( 7), (8) , and ( 9) is to estimate π B (A|S), ρ B (R|S, A), and p B (S) using p φa (A|D θ , S, R, S ′ ), p φr (R|D θ , S, A), and p φs (S|D θ , A, R), respectively. As we do not have the explicit expression of distributions π B , ρ B , and p B , we cannot directly compute the KL divergence in these first terms. To address this issue, based on the generalized Bayesian inference [41] , we can exchange two distributions in the KL divergence. Then, we model all the aformentioned distributions as Gaussian distributions, and use the mean µ φ and standard deviation Σ φ to represent the corresponding p φ . For implementation, we directly employ MLPs to output each (µ φ , Σ φ ) using the corresponding conditions of p φ . Then, based on the KL divergence between two Gaussian distributions, we can derive the loss function as follows.EQUATIONMoreover, the goal of second terms in Equations ( 7), (8) , and ( 9) is to maximize the likelihoods of D θ given samples ŝ ∼ p φs , â ∼ p φa , or r ∼ p φr . Thus, with (s, a, r) ∼ B, we propose minimizing the distance between D θ (ŝ, a, r) and D(s, a, r), D θ (s, â, r) and D(s, a, r), and D θ (s, a, r) and D(s, a, r), where ŝ ∼ p φs , â ∼ p φa , and r ∼ p φr . Then, based on [41] , we can derive the following loss with any metric ℓ to maximize the log probabilities:L second (θ, φ s , φ a , φ r ) = E (s,a,r)∼B,ŝ∼pφ s ,â∼pφ a ,r∼pφ r ,D∼p ℓ D(s, a, r), D θ (s, â, r)EQUATIONWe focus on developing tractable loss functions for implementation in this subsection.Learning the Action-Value Distribution based on Temporal Difference (TD). Based on [42, 43] , we introduce the quantile regression [44] to approximate the action-value distribution in the offline dataset B using an ensemble model {D θi } K i=1 . We use Equation ( 4) to derive the loss as:EQUATIONwhere ρ κ τ (δ) = |τ -I {δ < 0}| • l κ H (δ) with the threshold κ, Z denotes the value distribution, δ τ,τ ′ θi is the sampled TD error based on the parameters θ i , τ and τ ′ are two samples drawn from a uniform distribution U ([0, 1]), D τ θ (s, a, r):= F -1 D θ (s,a,r) (τ ) is the sample drawn from p θ (•|s, a, r), Z τ (s) := F -1 Z(s) (τ ) is sampled from p(•|s), F -1 X (τ )is the inverse cumulative distribution function (also known as quantile function) [45] at τ for the random variable X, and N and N ′ represent the respective number of iid samples τ and τ ′ . Notably, based on [43] , we have Q θi (s, a) = N n=1 D τn θi (s, a, r). In addition, if we learn the value distribution Z, the action-value distribution can extract the information from the next states based on Equation (12) , which is effective for capturing the uncertainty. On the contrary, if we directly use the next states in the offline dataset as the observations, in practice, the parameterized model of the action-value distribution needs to take (s, a, r, s ′ , a ′ , r ′ , s ′′ ) as the input data. Thus, the model can compute the action values and values for the sampled TD error in Equation (12) . To avoid the changes in the input data caused by directly using next states as observations in Bayesian inference, we draw inspiration from IQL and RIQL to learn a parameterized value distribution. Based on Equations ( 5) and (12) , we derive a new objective as:EQUATIONwhere D τ α is the α-quantile value among {D τ θi (s, a)} K i=1 , and V ψ (s) = N n=1 Z τn ψ (s). More details are shown in Appendix B.2. Furthermore, we provide the theoretical analysis in Appendix A.1 to give a value bound between the value distributions under clean and corrupted data.Updating the Action-Value Distribution based on Variational Inference for Robustness. We discuss the detailed implementation of Equations (10) and (11) based on Equations ( 12) and (13) . As the data corruptions may introduce heavy-tailed targets [18] , we apply the Huber loss to replace all quadratic loss in Equation (10) and the metric ℓ in Equation (11) , mitigating the issue caused by heavy-tailed targets [46] for robustness. We rewrite Equation (11) as follows.L second (θ i , φ s , φ a , φ r ) = E (s,a,r)∼B,ŝ∼pφ s ,â∼pφ a ,r∼pφ r ,D τ ∼p N n=1 l κ H D τn (s, a, r), D τn θi (s, â, r)+ l κ H D τn (s, a, r), D τn θi (s, a, r) + l κ H D τn (s, a, r), D τn θi (ŝ, a, r) . ( 14)Thus, we have the whole loss function L D|S,A,R = L first (θ i , φ s , φ a , φ r ) + L second (θ i , φ s , φ a , φ r ) in the generalized variational inference framework. Moreover, based on the assumption of heavy-tailed noise in [18] , we have a upper bound of action-value distribution by using the Huber regression loss.Entropy-based Uncertainty Measure for Regulating the Loss associated with Corrupted Data. To further address the challenge posed by diverse data corruptions, we consider the problem: how to exploit uncertainty to further enhance robustness?Considering that our goal is to improve performance in clean environments, we propose to reduce the influence of corrupted data, focusing on using clean data to learn agents. Therefore, we provide a two-step plan: (1) distinguishing corrupted data from clean data; (2) regulating the loss associated with corrupted data to reduce its influence, thus enhancing the performance in clean environments.For (1), as the Shannon entropy for the measures of aleatoric and epistemic uncertainties provides important insight [47] [48] [49] , and the corrupted data often results in higher uncertainty and entropy of the action-value distribution than the clean data, we use entropy [50] to quantify uncertainties of corrupted and clean data. Furthermore, by considering that the exponential function can amplify the numerical difference in entropy between corrupted and clean data, we propose the use of exponential entropy [51] -a metric of extent of a distribution-to design our uncertainty measure.Specifically, based on Equation 12, we can use the quantile points {τ n } N n=1 to learn the corresponding quantile function values {D τn } N n=1 drawn from the action-value distribution p θ . We sort the quantile points and their corresponding function values in ascending order based on the values. Thus, we have the sorted sets {ς n } N n=1 , {D ςn } N n=1 , and the estimated PDF values {ς n } N n=1 , where ς 1 = ς 1 and ς n = ς n -ς n-1 for 1 < n ≤ N . Then, we can further estimate differential entropy following [52] (see Appendix A.3 for a detailed derivation).EQUATIONwhere ςn denotes(ς n-1 + ς n )/2 for 1 < n ≤ N , and D ςn denotes D ςn -D ςn-1 for 1 < n ≤ N .For (2), TRACER employs the reciprocal value of exponential entropy 1/ exp(H(p θi )) to weight the corresponding loss of θ i in our proposed whole loss function L D|S,A,R . Therefore, during the learning process, TRACER can regulate the loss associated with corrupted data and focus on minimizing the loss associated with clean data, enhancing robustness and performance in clean environments. Note that we normalize entropy values by dividing the mean of samples (i.e., quantile function values) drawn from action-value distributions for each batch. In Figure 3 , we show the relationship of entropy values of corrupted and clean data estimated by Equation (15) during the learning process. The results illustrate the effectiveness of the entropy-weighted technique for data corruptions.Updating the Policy based on the Action-Value Distribution. We directly applies the weighted imitation learning technique in Equation ( 6) to learn the policy. As EQUATION 
experiments: 
 In this section, we show the effectiveness of TRACER across various simulation tasks using diverse corrupted offline datasets. Firstly, we provide our experiment setting, focusing on the corruption settings for offline datasets. Then, we illustrate how TRACER significantly outperforms previous stateof-the-art approaches under a range of both individual and simultaneous data corruptions. Finally, we conduct validation experiments and ablation studies to show the effectiveness of TRACER.Building upon RIQL [18] , we use two hyperparameters, i.e., corruption rate c ∈ [0, 1] and corruption scale ϵ, to control the corruption level. Then, we introduce the random corruption and adversarial corruption in four elements (i.e., states, actions, rewards, next states) of offline datasets. The implementation of random corruption is to add random noise to elements of a c portion of the offline datasets, and the implementation of adversarial corruption follows the Projected Gradient Descent attack [53, 54] using pretrained value functions. Note that unlike other adversarial corruptions, the adversarial reward corruption multiplies -ϵ to the clean rewards instead of using gradient optimization. We also introduce the random or adversarial simultaneous corruption, which refers to random or adversarial corruption simultaneously present in four elements of the offline datasets. We apply the corruption rate c = 0.3 and corruption scale ϵ = 1.0 in our experiments. We conduct experiments on D4RL benchmark [55] . Referring to RIQL, we train all agents for 3000 epochs on the 'medium-replay-v2' dataset, which closely mirrors real-world applications as it is collected during the training of a SAC agent. Then, we evaluate agents in clean environments, reporting the average normalized performance over four random seeds. See Appendix C for detailed information. The algorithms we compare include: (1) CQL [7] and IQL [26] , offline RL algorithms using a twin Q networks. ( 2) EDAC [56] and MSG [57] , offline RL algorithms using ensemble Q networks (number of ensembles > 2). (3) UWMSG [17] and RIQL, state-of-the-arts in corruption-robust offline RL. Note that EDAC, MSG, and UWMSG are all uncertainty-based offline RL algorithms.We conduct experiments on MuJoCo [58] (see Tables 1, 2 , and 3, which highlight the highest results) and CARLA [59] (see the left of Figure 2 ) tasks from D4RL under diverse corruptions to show the superiority of TRACER. In Table 1 , we report all results under random or adversarial simultaneous data corruptions. These results show that TRACER significantly outperforms other algorithms in all tasks, achieving an average score improvement of +21.1%. In the left of Figure 2 , results on 'CARLA-lane_v0' under random simultaneous corruptions also illustrate the superiority of TRACER. See Appendix C.2 for details.Random Corruptions. We report the results under random simultaneous data corruptions of all algorithms in Table 1 . Such results demonstrate that TRACER achieves an average score gain of +22.4% under the setting of random simultaneous corruptions. Based on the results, it is clear that many offline RL algorithms, such as EDAC, MSG, and CQL, suffer the performance degradation under data corruptions. Since UWMSG is designed to defend the corruptions in rewards and dynamics, its performance degrades when faced with the stronger random simultaneous corruption. Moreover, we report results across a range of individual random data corruptions in Table 2 , where TRACER outperforms previous algorithms in 7 out of 12 settings. We then explore hyperparameter tuning on Hopper task and further improve TRACER's results, demonstrating its potential for performance gains. We provide details in Appendix C.3.1.Adversarial Corruptions. We construct experiments under adversarial simultaneous corruptions to evaluate the robustness of TRACER. The results in Table 1 show that TRACER surpasses others by a significant margin, achieving an average score improvement of +19.3%. In these simultaneous corruption, many algorithms experience more severe performance degradation compared to the random simultaneous corruption, which indicates that adversarial attacks are more damaging to the reliability of algorithms than random noise. Despite these challenges, TRACER consistently achieves significant performance gains over other methods. Moreover, we provide the results across a range of individual adversarial data corruptions in Table 3 , where TRACER outperforms previous algorithms in 7 out of 12 settings. We also explore hyperparameter tuning on Hopper task and further improve TRACER's results, demonstrating its potential for performance gains. See Appendix C. In the first column, we report the mean and standard deviation to show the superiority of using the entropy-based uncertainty measure. In the second and third columns, we report the results over three seeds to show the higher entropy of corrupted data compared to clean data during training.Building upon RIQL, we further extend our experiments to include Mujoco datasets with various corruption levels, using different corruption rates c and scales ϵ. We report the average scores and standard deviations over four random seeds in the right of Figure 2 , using batch sizes of 256.Results in the right of Figure 2 demonstrate that TRACER significantly outperforms baseline algorithms in all tasks under random simultaneous corruptions with various corruption levels. It achieves an average score improvement of +33.6%. Moreover, as the corruption levels increase, the slight decrease in TRACER's results indicates that while TRACER is robust to simultaneous corruptions, its performance depends on the extent of corrupted data it encounters. We also evaluate TRACER in different scales of corrupted data and provide the results in Appendix C.4.1.We evaluate the entropy-based uncertainty measure in action-value distributions to show: (1) is the uncertainty caused by corrupted data higher than that of clean data? (2) is regulating the loss associated with corrupted data effective in improving performance?For (1), we first introduce labels indicating whether the data is corrupted. Importantly, these labels are not used by agents during the training process. Then, we estimate entropy values of labelled corrupted and clean data in each batch based on Equation (15) . Thus, we can compare entropy values to compute results, showing how many times the entropy of the corrupted data is higher than that of clean data. Specifically, we evaluate the accuracy every 50 epochs over 3000 epochs. For each evaluation, we sample 500 batches to compute the average entropy of corrupted and clean data. Each batch consists of 32 clean and 32 corrupted data. We illustrate the curves over three seeds in the second and third columns of Figure 3 , where each point shows how many of the 500 batches have higher entropy for corrupted data than that of clean data.Figure 3 indicates an oscillating upward trend of TRACER's measurement accuracy using entropy (TRACER Using Entro) under simultaneous corruptions, demonstrating that using the entropy-based uncertainty measure can effectively distinguish corrupted data from clean data. These curves also reveal that even in the absence of any constraints on entropy (TRACER NOT using Entro), the entropy associated with corrupted data tends to exceed that of clean data.For (2) , in the first column of Figure 3 , these results demonstrate that TRACER using the entropybased uncertainty measure can effectively reduce the influence of corrupted data, thereby enhancing robustness and performance against all corruptions. We provide detailed information for this evaluation in Appendix C.4.2. 
related work: 
 Robust RL. Robust RL can be categorized into two types: testing-time robust RL and training-time robust RL. Testing-time robust RL [19, 20] refers to training a policy on clean data and ensuring its robustness by testing in an environment with random noise or adversarial attacks. Training-time robust RL [16, 17] aims to learn a robust policy in the presence of random noise or adversarial attacks during training and evaluate the policy in a clean environment. In this paper, we focus on training-time robust RL under the offline setting, where the offline training data is subject to various data corruptions, also known as corruption-robust offline RL.Corruption-Robust RL. Some theoretical work on corruption-robust online RL [60] [61] [62] [63] aims to analyze the sub-optimal bounds of learned policies under data corruptions. However, these studies primarily address simple bandits or tabular MDPs and focus on the reward corruption. Some further work [64, 65] extends the modeling problem to more general MDPs and begins to investigate the corruption in transition dynamics.It is worth noting that corruption-robust offline RL has not been widely studied. UWMSG [17] designs a value-based uncertainty-weighting technique, thus using the weight to mitigate the impact of corrupted data. RIQL [18] further extends the data corruptions to all four elements in the offline dataset, including states, actions, rewards, and next states (dynamics). It then introduces quantile estimators with an ensemble of action-value functions and employs a Huber regression based on IQL [26] , alleviating the performance degradation caused by corrupted data.Bayesian RL. Bayesian RL integrates the Bayesian inference with RL to create a framework for decision-making under uncertainty [28] . It is important to highlight that Bayesian RL is divided into two categories for different uncertainties: the parameter uncertainty in the learning of models [66, 67] and the inherent uncertainty from the data/environment in the distribution over returns [68, 69] . In this paper, we focus on capturing the latter.For the latter uncertainty, in model-based Bayesian RL, many approaches [68, 70, 71] explicitly model the transition dynamics and using Bayesian inference to update the model. It is useful when dealing with complex environments for sample efficiency. In model-free Bayesian RL, value-based methods [69, 72] use the reward information to construct the posterior distribution of the action-value function. Besides, policy gradient methods [73, 74] use information of the return to construct the posterior distribution of the policy. They directly apply Bayesian inference to the value function or policy without explicitly modeling transition dynamics.Offline Bayesian RL. offline Bayesian RL integrates Bayesian inference with offline RL to tackle the challenges of learning robust policies from static datasets without further interactions with the environment. Many approaches [75] [76] [77] use Bayesian inference to model the transition dynamics or guide action selection for adaptive policy updates, thereby avoiding overly conservative estimates in the offline setting. Furthermore, recent work [78] applies variational Bayesian inference to learn the model of transition dynamics, mitigating the distribution shift in offline RL. 
conclusion: 
 In this paper, we investigate and demonstrate the robustness and effectiveness of introducing Bayesian inference into offline RL to address the challenges posed by data corruptions. By leveraging Bayesian techniques, our proposed approach TRACER captures the uncertainty caused by diverse corrupted data. Moreover, the use of entropy-based uncertainty measure in TRACER can distinguish corrupted data from clean data. Thus, TRACER can regulate the loss associated with corrupted data to reduce its influence, improving performance in clean environments. Our extensive experiments demonstrate the potential of Bayesian methods in developing reliable decision-making.Regarding the limitations of TRACER, although it achieves significant performance improvement under diverse data corruptions, future work could explore more complex and realistic data corruption scenarios and related challenges, such as the noise in the preference data for RLHF and adversarial attacks on safety-critical driving decisions. Moreover, we look forward to the continued development and optimization of uncertainty-based corrupted-robust offline RL, which could further enhance the effectiveness of TRACER and similar approaches for increasingly complex real-world scenarios.Appendix / supplemental material A ProofsWe provide a proof for the upper bound of value distributions and derivations of loss functions. See Table 4 for all notations. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We report standard deviations and standard errors for all results in the main text, including results in Tables 1, 2, 3, Figures 2, and 3 .Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: NA.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [NA] Justification: All our experiment data is open source.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: NA.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: NA.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: NA.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: To prove an upper bound of the value distribution, we introduce an assumption from [18] below.Assumption A.1. [18] Let ζ = N i=1 (2ζ i + log ζ ′ i )denote the cumulative corruption level, where ζ i and ζ ′ i are defined asT V (s i ) -T V (s i ) ∞ ≤ ζ i , max π B (a | s i ) π b (a | s i ) , π b (a | s i ) π B (a | s i ) ≤ ζ ′ i , ∀a ∈ A.Here ∥ • ∥ ∞ means taking supremum over V : S → [0, r max /(1 -γ)].Note that {ζ ′ i } i = 1 N can quantify the corruption level of states and actions, and {ζ i } i = 1 N can quantify the corruption level of rewards and next states (transition dynamics).Then, we provide the following assumption. (a | s) ∝ π B (a | s) • exp β • T Q * -V * (s, a)is the policy of corrupted data.EQUATION= arg minEQUATIONAs TRACER directly applies the weighted imitation learning technique from IQL to learn the policy, we can use πIQL as the policy learned by TRACER under data corruptions, akin to RIQL. Assumption A.2 requires that each pair, including the policy π E and the clean data b, the policy πE and the corrupted dataset B, π E and πE , π E and π IQL , and πE and πIQL , has good coverage. It is similar to the coverage condition in [18] . Based on Assumptions A.1 and A.2, we can derive the following theorem to show the robustness of our approach using the supervised policy learning scheme and learning the action-value distribution.Lemma A.3. (Performance Difference) For any π and π, we haveEQUATIONProof. Based on [79] , for any π and π, we haveEQUATIONEQUATIONTheorem A.4. (Robustness). Under Assumptions A.1 and A.2, we haveEQUATIONwhere q π is the value distribution, W 1 (•, •) is the Wasserstein distance [80] for measuring the distribution difference,ϵ 1 = E s∼b [D KL (π E (•|s) ∥ π IQL (•|s))],andϵ 2 = E s∼B [D KL (π E (•|s) ∥ πIQL (•|s))].Proof. The definition of Wasserstein metric is W p (P, Q) = inf γ∈Γ(P,Q) x y ||x -y|| p p dγ(x, y)1 p.The value distribution q π , also denoted by p(•|s) in Section 3.2 of the main text, is an expactation of action value distribution E a∼π,r∼ρ [p θ (•|s, a, r)].Note that as TRACER directly applies the weighted imitation learning technique from IQL to learn the policy (see Section 3.2), we can use πIQL as the policy learned by TRACER under data corruptions, akin to RIQL.Let q πIQL and q π IQL be the value distributions of learned policies following IQL's supervised policy learning scheme under corrupted and clean data, respectively. Z πIQL ∼ q πIQL and Z π IQL ∼ q π IQL . Let p π E be the value distribution of the policy of clean data, p πE be the value distribution of the policy of corrupted data, Z π E ∼ p π E , and Z πE ∼ p πE . We haveEQUATIONEQUATIONMoreover, based on the Bretagnolle-Huber inequality, we have d T V (P, Q) ≤ 1 -exp(-D KL (P ||Q)). For W 1 (p πE , p π E ), we haveEQUATIONEQUATIONA.3EQUATIONEQUATIONEQUATIONEQUATIONEQUATIONEQUATIONMoreover, we haveEQUATIONEQUATION)By the definition of corruption levels, we haveEQUATIONEQUATIONThus, we can deriveEQUATIONThen, for W 1 (p π E , q π IQL ), we haveEQUATIONwhereϵ 1 = E s∼b [D KL (π E (•|s) ∥ π IQL (•|s))].Finally, for W 1 (q πIQL , p πE ), we haveEQUATIONTherefore, we haveEQUATIONNote that in Theorem A.4, the major difference between TRACER and IQL/RIQL is that TRACER uses the action-value and value distributions rather than the action-value and value functions in IQL/RIQL. Therefore, we provide this theorem to prove an upper bound on the difference in value distributions of TRACER to show its robustness, which also provides a guarantee of how TRACER's performance degrades with increased data corruptions.For Equations ( 7), (8) , and ( 9), we provide the detailed derivations.Firstly, for Equation ( 7), we maximize the posterior and have EQUATIONEQUATION)Note that L b is The behavior policy that is used to collect clean data. π B (•|s)L b = P(A) p ψa (A t |D, S t , S t+1 ) • log p(D, A t |S t ,The behavior policy that is used to collect corrupted data. π E (•|s)The policy that we want to learn under clean data.The policy that we are learning under corrupted data. π IQL (•|s) The learned policy using IQL's weighted imitation learning under clean data. πIQL (•|s) The learned policy using IQL's weighted imitation learning under corrupted data.d π (s, a)The probability density function associated with policy π at state s and action a. W 1 (p, q) The Wasserstein-1 distance that measures the difference between distributions p and q.q πThe value distribution of the policy π. Z π (s)The random variable of the value distribution q π (•|s). ϵ 1 KL divergence between π E and π IQL , i.e., the standard imitation error under clean data. ϵ 2 KL divergence between πE and πIQL , i.e., the standard imitation error under corrupted data. ςnThe midpoint of the probability density p θ . On the actor-critic architecture based on IQL, our approach TRACER adds just one ensemble model (p φa , p φr , p φs ) to reconstruct the data distribution and replace the function approximation in the critic with a distribution estimation (see Figure 4 ). Based on our experiments, this structure significantly improves the ability to handle both simultaneous and individual corruptions.TRACER consists of four components: the actor network, the critic network, the value network, and the observation model. We first implement the actor network for decision-making with a 3-layer MLP, where the dimension of the hidden layers are 256 and the activation functions are ReLUs. Then, we utilize quantile regression [44] to design the critic networks and the value network, approximating the distributions of the action-value function D and value function Z, respectively.Specifically, for the action-value distribution, we use a function h : [0, 1] -→ R d to compute an embedding for the sampling point τ . Then, we can obtain the corresponding approximation by D τ (s, a) ≈ f (g(s) ⊙ h(τ )) a . Here, g : S -→ R d is the function computed by MLPs and f : R d -→ R |A| is the subsequent fully-connected layers mapping g(s) to the estimated action-values D(s, a) ≈ f (g(s)) a . For the approximation of the value distribution Z τ (s), we leverage the same architecture. Note that the function h is implemented by a fully-connected layer, and the output dimension is set to 256. We then use an ensemble model with K networks. Each network is a 3-layer MLP, consisting of 256 units and ReLU activations. By using the ensemble model, we can construct the critic network, using the quantile regression for optimization. Moreover, the value network is constructed by a 3-layer MLP. The dimension of the hidden layers are also 256 and activation functions are ReLUs.The observation model uses an ensemble model with 3 neural networks. Each network is used to reconstruct different observations in the offline dataset. It consists of two fully connected layers, and the activation functions are ReLUs. We apply a masked matrix during the learning process for updating the observation model. Thus, each network in the observation model can use different input data to compute Equations ( 7), (8) , and ( 9), respectively.Following the setting of [18] , we list the hyper-parameters of N , K, and α for the approxmiation of action-value functions, and κ in the Huber loss in TRACER under random and adversarial corruption in Table 5 and Table 6 , respectively. Here, N is the number of samples τ , and K is the number of ensemble models. Moreover, we apply Adam optimizer using a learning rate 1 × 10 -3 , γ = 0.99, target smoothing coefficient τ = 0.05, batch size 256 and the update frequency for the target network is 2. The corruption rate is c = 0.3 and the corruption scale is ϵ = 1.0 for all experiments. For the loss function of observation models, we use a linear decay parameter η from 0.0001 to 0.01 to trade off the loss L first and L second . We update each algorithm for 3000 epochs. Each epoch uses 1000 update times following [56] . Then, we evaluate each algorithm in clean environments for 100 epochs and report the average normalized return (calculated by 100 × score-random score expert score-random score ) over four random seeds.Table 5 : Hyper-parameters used for TRACER under the random corruption benchmark. We follow the corruption settings proposed by [18] , applying either random or adversarial corruptions to each element of the offline data, namely state, action, reward, and dynamics (or "next-state"), to simulate potential attacks that may occur during the offline data collection and usage process in real-world scenarios. We begin with the details of random corruption below.• Random observation corruption. We randomly sample c% transitions (s, a, r, s ′ ) from the offline dataset, and for each of these selected states s, we add noise to form ŝ = s+λ•std(s), where λ ∼ U nif orm[-ϵ, ϵ] ds , c is the corruption rate, ϵ is the corruption scale, d s refers to the dimension of states and std(s) represents the d s -dimensional standard deviation of all states in the offline dataset.• Random action corruption. We randomly sample c% transitions (s, a, r, s ′ ) from the offline dataset, and for each of these selected actions a, we add noise to form â = a + λ • std(a), where λ ∼ U nif orm[-ϵ, ϵ] da , d a refers to the dimension of actions and std(a) represents the d a -dimensional standard deviation of all actions in the offline dataset.• Random reward corruption. We randomly sample c% transitions (s, a, r, s ′ ) from the offline dataset, and for each of these selected rewards r, we modify it to r = U nif orm[-30• ϵ, 30 • ϵ]. We adopt this harder reward corruption setting since [81] has found that offline RL algorithms are often insensitive to small perturbations of rewards.• Random dynamics corruption. We randomly sample c% transitions (s, a, r, s ′ ) from the offline dataset, and for each of these selected next-step states s ′ , we add noise to form ŝ′ = s ′ +λ•std(s ′ ), where λ ∼ U nif orm[-ϵ, ϵ] d s ′ , d s ′ refers to the dimension of next-step states and std(s ′ ) represents the d s ′ -dimensional standard deviation of all next states in the offline dataset.The harder adversarial corruption settings are detailed as follows:• Adversarial observation corruption. To impose adversarial attack on the offline dataset, we first need to pretrain an EDAC agent with a set of Q p functions and a policy function π p using clean dataset. Then, we randomly sample c% transitions (s, a, r, s ′ ) from the offline dataset, and for each of these selected states s, we attack it to form ŝ = min ŝ∈B d (s,ϵ) Q p (ŝ, a). Here, B d (s, ϵ) = {ŝ||ŝ -s| ≤ ϵ • std(s)} regularizes the maximum difference for each state dimension.• Adversarial action corruption. We randomly sample c% transitions (s, a, r, s ′ ) from the offline dataset, and for each of these selected actions a, we use the pretrained EDAC agent to attack it to form â = min â∈B d (a,ϵ) Q p (s, â). Here, B d (a, ϵ) = {â||â -a| ≤ ϵ • std(a)} regularizes the maximum difference for each action dimension. • Adversarial reward corruption. We randomly sample c% transitions (s, a, r, s ′ ) from the offline dataset, and for each of these selected rewards r, we directly attack it to form r = -ϵ × r without any adversarial model. This is because that the objective of adversarial reward corruption is r = min r∈B(r,ϵ) r + γE[Q(s ′ , a ′ )]. Here B(r, ϵ) = {r | |r -r| ≤ (1+ϵ)•r max } regularizes the maximum distance for rewards. Therefore, we have r = -ϵ×r.• Adversarial dynamics corruption. We randomly sample c% transitions (s, a, r, s ′ ) from the offline dataset, and for each of these selected next-step states s ′ , we use the pretrained EDAC agent to attack it to form ŝ′ =min ŝ′ ∈B d (s ′ ,ϵ) Q p ( ŝ′ , π p (s ′ )). Here, B d (s ′ , ϵ) = { ŝ′ || ŝ′ -s ′ | ≤ ϵ • std(s ′)} regularizes the maximum difference for each dimension of the dynamics.The optimization of the above adversarial corruptions are implemented through Projected Gradient Descent [53, 54] . Taking adversarial observation corruption for example, we first initialize a learnable vector z ∈ [-ϵ, ϵ] ds , and then conduct a 100-step gradient descent with a step size of 0.01 for ŝ = s + z • std(s), and clip each dimension of z within the range [-ϵ, ϵ] after each update. For action and dynamics corruption, we conduct similar operation.Building upon the corruption settings for individual elements as previously discussed, we further intensify the corruption to simulate the challenging conditions that might be encountered in real-world scenarios. We present the details of the simultaneous corruption below:• Random simultaneous corruptions. We sequentially conduct random observation corruption, random action corruption, random reward corruption, and random dynamics corruption to the offline dataset in order. That is to say, we randomly select c% of the transitions each time and corrupt one element among them, repeating four times until states, actions, rewards, and dynamics of the offline dataset are all corrupted. • Adversarial simultaneous corruptions. We sequentially conduct adversarial observation corruption, adversarial action corruption, adversarial reward corruption, and adversarial dynamics corruption to the offline dataset in order. That is to say, we randomly select c% of the transitions each time and attack one element among them, repeating four times until states, actions, rewards, and dynamics of the offline dataset are all corrupted.We conduct experiments in CARLA from D4RL benchmark. We use the clean environment 'CARLA-Lane-v0' to evaluate IQL, RIQL, and TRACER (Ours). We report each mean result with the standard deviation in the left of Figure 2 over four random seeds for 3000 epochs. We apply the random simultaneous data corruptions, where each element in the offline dataset (including states, actions, rewards, and next states) may involve random noise. We follow the setting in Section 4.2, using the corruption rate c = 0.3 and scale ϵ = 1.0 in the CARLA task. The results in the left of Figure 2 show the superiority of TRACER in the random simultaneous corruptions. We provide the hyperparameters of TRACER in Table 5 .In Tables 2 and 3 of the main text, we adhered to commonly used settings for individual corruptions in corruption-robust offline RL. We directly followed hyperparameters from RIQL (i.e., κ for huber loss, the ensemble number K, and α in action-value functions, see Tables 5 and 6 ). Results show that TRACER outperforms RIQL in 18 out of 24 settings, demonstrating its robustness even when aligned with RIQL's hyperparameters.Further, we explore hyperparameter tuning, specifically of κ, on Hopper task to improve TRACER's performance. This results in TRACER outperforming RIQL in 7 out of 8 settings on Hopper, up The results in Table 9 show that TRACER significantly outperforms other methods in all these tasks with the aforementioned AntMaze and Mujoco datasets. Theorem A. 4 shows that the higher the scale of corrupted data, the greater the difference in actionvalue distributions and the lower the TRACER's performance. Thus, we further conduct experiments to evaluate TRACER across various corruption levels. Specifically, we apply different corruption rate c in all four elements of the offline dataset. We randomly select c% of transitions from the dataset and corrupt one element in each selected transition. Then, we repeat this step four times until all elements are corrupted. Therefore, approximately 100• (1 -(1 -c) 4)% of data in the offline dataset is corrupted.In In Figure 5 , we additionally report the entropy values of TRACER with and without using entropy-based uncertainty measure, corresponding to Figure 3 . The curves illustrate that TRACER using entropy-based uncertainty measure can effectively regulate the loss associated with corrupted data, reducing the influence of corrupted samples. Therefore, the estimated entropy values of corrupted data can be higher than those of clean data.We conduct experiments of TRACER and TRACER without Bayesian inference, i.e., RIQL combined with distributional RL methods, namely DRIQL. The experiments are on 'Walker2d-medium-replay-v2' under random simultaneous data corruptions. We employ the same hyperparameters of Huber loss for these methods, using α = 0.25 and κ = 1.0.We report each mean result with the standard deviation in Figure 6 . Each result is averaged over four seeds for 3000 epochs. These results show the effectiveness of our proposed Bayesian inference. In this subsection, we provide the computational cost of our approach TRACER.For the MuJoCo tasks, including Halfcheetah, Walker2d, and Hopper, the average training duration is 40.6 hours. For the CARLA task, training extends to 51 hours. We conduct all experiments on NVIDIA GeForce RTX 3090 GPUs.To compare the computational cost, we report the average epoch time on Hopper in Table 11 , where results of baselines (including DT [82] ) are from [18] . The formula for computational cost is:avg_epoch_time_of_RIQL_in_ [18] avg_epoch_time_we_run_RIQL × avg_epoch_time_we_run_TRACER.Note that TRACER requires a long epoch time due to two main reasons:1. Unlike RIQL and IQL, which learn one-dimensional action-value functions, TRACER generates multiple samples for the estimation of action-value distributions. Following [43] , we generate 32 samples of action values for each state-action pair. 2. TRACER uses states, actions, and rewards as observations to update models, estimating the posterior of action-value distributions. [86, 71] . In recent years, RL has achieved great success in many important realworld decision-making tasks [87] [88] [89] [90] [91] [92] [93] . However, the online RL methods still typically rely on active data collection to succeed, hindering their application in scenarios where real-time data collection is expensive, risky, and/or impractical. Thus, we focus on the offline RL paradigm in this paper.We implement our codes in Python version 3.8 and make the code available online2 .TRACER's performance is related to the extent of corrupted data within the offline dataset. Although TRACER consistently outperforms RIQL even under conditions of extensive corrupted data (see Table 10 ), its performance does degrade as the corruption rate (i.e., the extent/scale of corrupted data) increases. To tackle this problem, we look forward to the continued development and optimization of corruption-robust offline RL with large language models, introducing the prior knowledge of clean data against large-scale or near-total corrupted data. Thus, this corruption-robust offline RL can perform well even when faced with large-scale or near-total corrupted data in the offline dataset.This paper proposes a novel approach called TRACER to advance the field of in offline RL for diverse data corruptions, enhancing the potential of agents in real-world applications. Although our primary focus is on technical innovation, we recognize the potential societal consequences of our work, as robustness in offline RL for data corruptions has the potential to influence various domains such as autonomous driving and the large language model field. We are committed to ethical research practices and attach great importance to the social implications and ethical considerations in the development of robustness research in offline RL.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification: We provide the data and code in Appendix F, which can also be found in the supplemental material. Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material.https://github.com/MIRALab-USTC/RL-TRACER 


Decision: The reviewers agreed that the paper addresses an important problem of robust reinforcement learning under corrupted data, proposes an interesting method based on Bayesian inference, and the extensive empirical evaluation demonstrates the effectiveness of the proposed method. However, the reviewers also raised several concerns and questions in their initial reviews. We want to thank the authors for their responses and active engagement during the discussion phase. The reviewers appreciated the responses, which helped in answering their key questions. The reviewers have an overall positive assessment of the paper, and there is a consensus for acceptance. The reviewers have provided detailed feedback, and we strongly encourage the authors to incorporate this feedback when preparing the final version of the paper.

Reviews: review: paper_summary: paper_summary: This paper seeks to conduct reinforcement learning from corrupted offline data. More specifically, they propose the TRACER algorithm, which uses bayesian inference to calculate the uncertainty in estimating the action-value function. The authors conduct experiments with diverse corruptions on CARLA and Mujoco environments.
strength_weakness: strength_weakness: - The contribution compared to other baselines (RIQL) is not really clear. In the introduction the authors claim that their method can handle simultaneous corruptions and RIQL cannot. However no experiments are done in this setting.
- The experimental settings are fairly limited: only three Mujoco environments are considered and one carla experiment.
- Different perturbations levels are not shown.
- The authors do not discuss the computational cost of their method.
questions: questions: - Can you clarify the contribution of TRACER compared to RIQL? It seems like it is an orthogonal approach, but the two algorithms are not really compared in detail.
- Can you show the hyperparameter tuning results? Is TRACER stable under different hyperparameter settings? How does it compare to RIQL in this regard?
- How does the computational cost of TRACER compare to the other baselines?
limitations: limitations: The authors do discuss their method's limitations.
paper_summary: paper_summary: The paper introduces TRACER, a robust offline reinforcement learning (RL) algorithm designed to address the challenges posed by data corruptions in offline datasets. The corruptions can be
realized in form of states, actions, rewards, and dynamics corruption. The proposed methodology uses Bayesian inference to model and mitigate the impact of corrupted data on the action-value function.
By leveraging an entropy-based uncertainty measure, TRACER can distinguish between corrupted and clean data, thereby reducing the influence of the former and enhancing the robustness of the learned policies.
The paper provides thorough evaluation on MujoCo and Carla offline RL datasets.
strength_weakness: strength_weakness: 1. It is not clear how the final policy is extracted after learning $D_{\theta}$.
2. The paper relies on many hyperparameters. What kind of kernel is used in the gaussian distribution learning in Eq 10? Is there a upper bound on what amount of corruption the method will be able to handle? 
3. While the authors consider individual corruption of one element at a time data collected from in a real world system can have simultaneous corruptions. For
example corrupted action due to adversarial noise will lead the system to different state causing simultaneous corruptions.
4. Notational clarity: The paper introduces a lot of notations which is difficult to keep track of. Specially in the following contexts :
a :  The paper introduces LQ (θi), LV(ψ), Lπ(ϕ) in Eq 4, 5 and 6 but does not discuss how they relate to Dθ. Is the θ used for learning Q function and D same?
b : The paper introduces a new notation qπ in Eq 18 called value distribution. Is this the same as action value distribution? RIQL already proves the robustness of IQL policy with respect of data corruption. What is 
the objective of Theorem A.3? It will be helpful to add a discussion on the theoretical analysis. Also Providing a table summarizing the notations in appendix will enhance readability of the text.
questions: questions: 1) How relevant is the reward corruption given offline RL already is robust to noise in the reward function as it mimics the actions for datasets with length bias? Please refer to ref 1
2) The authors aim to leverage all elements in the dataset as observations. Do the authors assume that complete traces of the system are available for which the offline policy is being learned? 
In situations of dynamic corruption these traces might be different than the system. For example s_t in the corrupted dataset may not be observed in the real system. How are such cases 
handled?
3) As the method learns action-value function D from the data while reducing the influence of corrupted data. What amount of clean data needs to be present in the dataset for this method 
to work?
4) What is the reason for not using projected gradient descent for the reward corruption? Also could the author provide some understanding of why the method outperforms RIQL under certain settings and why it does not in some cases?
5) Please also respond to the weaknesses.

1. Li, Anqi, et al. "Survival instinct in offline reinforcement learning." Advances in neural information processing systems 36 (2024).
limitations: limitations: Limitations: The authors discuss the limitation in terms of handling realistic noises in conclusion. However, a limitation is also on the dependence on clean observations present in the dataset when some elements are corrupted.
The authors do not discuss any negative sociatal impact of their work.
paper_summary: paper_summary: This paper presents a novel approach called TRACER, aimed at addressing the challenge of learning robust policies from offline datasets that are subject to various data corruptions. The key contribution of this work lies in the integration of Bayesian inference to capture uncertainty within the offline data. TRACER models data corruptions as uncertainty in the action-value function and approximates the posterior distribution of this function. Furthermore, it employs an entropy-based uncertainty measure to distinguish between corrupted and clean data, thereby regulating the loss associated with corrupted data to enhance robustness in clean environments. Experimental results indicate that TRACER outperforms SOTA methods in handling both individual and simultaneous data corruptions.
strength_weakness: strength_weakness: - Though I like the insight from Variational Bayesian inference, my major concern is that the method is quite complicated, incorporating numerous components and designs such as the distributional value function, variational inference, and entropy-weighted loss function. This complexity makes this method less preferable in practice.

- Moreover, the results in the main tables (Table 2 and Table 3) appear similar to those of RIQL, raising concerns about the necessity of the included components.

- Another issue is the readability and clarity of the writing. For example, Equations 10 and 11 lack necessary explanation and insight. Additionally, the implementation details for $\phi_a, \phi_b, \phi_c$ are missing, leaving readers unsure of how the method works in practice. Furthermore, Equation 10 seems to include $\theta$ as an input unnecessarily, as the right side does not depend on $\theta$.

- The analysis of the experiments is not thorough. For instance, the accuracy of the measurement for corrupted data decreases during training in Figure 3, which is not discussed and seems unreasonable.
questions: questions: - The proposed method is complicated. Can this method be simplified?

- The results in the main tables (Table 2 and Table 3) appear similar to those of RIQL, raising concerns about the necessity of the included components.

- Another issue is the readability and clarity of the writing discussed above.

- Why does the accuracy of the measurement for corrupted data decreases during training in Figure 3?

- Moreover, I suggest the authors provide the computational cost comparison with prior work.
limitations: limitations: This paper acknowledges its limitations; however, I identified an additional concern regarding the complexity of the proposed method.
paper_summary: paper_summary: The authors introduce TRACER, a new Bayesian methodology to capture the uncertainty via offline data for robustness against all types of data corruptions. An appealing feature of TRACER is that it can distinguish corrupted data from clean data using an entropy-based uncertainty measure. Experiments are provided to prove the effectiveness of such a methodology.
strength_weakness: strength_weakness: The authors missed some important references: [1,2] introduce credal Bayesian deep learning, which is a robust way of performing Bayesian inference via Bayesian Neural Networks (where posterior and predictive distributions are approximated via variational inference) taking into account different types of uncertainty (using so-called credal sets, i.e. closed and convex sets of prior and likelihood probabilities), and quantifying them using entropy-based uncertainty measures. In the future, the authors may also look into [3], which seems like a compelling work for the research venue explored in the paper.

[1] https://arxiv.org/abs/2302.09656

[2] https://link.springer.com/chapter/10.1007/978-3-031-57963-9_1#:~:text=In%20their%20seminal%201990%20paper,bound%20to%20hold%20with%20equality

[3] https://arxiv.org/abs/2308.14815
questions: questions: I'd like to politely ask the authors to expand the related work with the above references [1,2]. In addition, would it be fair to say that, on page 3, the expectation that finds $\pi^\star$ is an integral taken with respect to the joint probability given by all the sources of randomness $P_0, \pi(\cdot \mid s_t), \rho(\cdot \mid s_t,a_t), P(\cdot \mid s_t,a_t)$? 

In equation (1), the closed parenthesis ) should go after $a_0=a$.
limitations: limitations: The limitations are explicitly acknowledged by the authors in the conclusion.

Comments: title: Thanks for your kind support!
Thanks again for your continued support!
Thank you for the discussion
Thanks for your kind support and for helping us improve the paper!
Thanks for your kind support and for helping us improve the paper!
Response
Thank you!
Thanks for your kind support and for helping us improve the paper!
Thanks for the response
Response to Additional Comments
Response to rebuttal
Table of Notations used in Appendix for Q8.
comment: Dear Reviewer vWB5,

Thanks for your kind support and further improvements you have suggested for our manuscript! We are committed to incorporating all your valuable suggestions in the final version, if accepted.

Thank you again for your valuable comments and guidance.

Best,

Authors
Dear Reviewer QZTE,

Thanks again for your continued support and the further improvements you have suggested for our manuscript. We sincerely appreciate your insightful feedback, which has significantly helped us in refining the explanations and enhancing the theoretical derivations. We remain committed to incorporating all your valuable suggestions into the final version, if accepted.

We are deeply grateful for your thoughtful comments and for the confidence you have shown in our work by raising your score.

Best,

Authors
Thank you for the discussion and the extra experiments. My concerns are addressed and I will raise my score.
Dear Reviewer V2Jx,

Thanks for your kind support and for helping us improve the paper! We are committed to including all these additional results and detailed explanations and will incorporate your suggestions in the final version, if accepted.

Thank you again for your valuable comments and guidance.

Best,

Authors
Dear Reviewer QZTE,

Thanks for your kind support and for helping us improve the paper! We are committed to including all these additional results, notation descriptions, and detailed explanations and will incorporate your suggestions in the final version, if accepted.

Thank you again for your valuable comments and guidance.

Best,

Authors
Thank you for addressing my concerns. I have raised my score. I would urge the authors to include the notation description, additional results and clear explanation of Theorem A3 in the revised version.
Thanks for your answers; I'm happy to keep my score.
Dear Reviewer wmJN,

Thanks for your kind support and for helping us improve the paper! We are committed to including all additional diagrams and clarifications and will incorporate your suggestions in the final version, if accepted.

Thank you again for your valuable comments and guidance.

Best,

Authors
My concerns have mostly been addressed, so I have decided to raise my score. The added diagram and clarification on the writing greatly improve understanding. The authors are expected to include these in the revised paper.
Thank you for your valuable comments. We respond to each of your comments as follows.

**Q16**. (1) How does $\tilde{\pi}_{\text{IQL}}$ help to prove the upper bound for TRACER?  (2) How do you equate $Z^\pi(s)$ with $D^\pi (s,a)$ in steps 24, 25, and 26 in Theorem A.3? (3) What is $D^\pi (s,a)$ as previously D is defined as D(s, a, r)?
  - **A16**.
    - For (1), as TRACER directly applies the weighted imitation learning technique from IQL to learn the policy, we can use $\tilde{\pi}_{\text{IQL}}$ as the policy learned by TRACER under data corruptions, akin to RIQL.
    In Theorem A.3 of the Appendix, the major difference between TRACER and IQL/RIQL is that TRACER uses the action-value and value distributions rather than the action-value and value functions in IQL/RIQL. Therefore, we further prove an upper bound on the difference in value distributions of TRACER to show its robustness.
    - For (2), we apologize for the missing reference in steps 24, 25, and 26 of Theorem A.3 (see Lemma 6.1 of [1]). We follow [1] to provide the detailed derivation below.
      - For any $\pi'$ and $\pi$, we have
  $$\begin{align}
Z^{\tilde{\pi}}(s) &= \sum_{t=0}^\infty \gamma^t \mathbb{E}\_{(S_t,A_t)\sim P,\tilde{\pi}} \left[R(S_t, A_t) | S_0 = s\right]\\\\
&= \sum_{t=0}^\infty \gamma^t \mathbb{E}_{(S_t,A_t)\sim P,\tilde{\pi}} \left[R(S_t, A_t)+Z^{\pi}(S_t)-Z^{\pi}(S_t) | S_0 = s\right]\\\\
&= \sum\_{t=0}^\infty \gamma^t \mathbb{E}\_{(S\_t,A\_t,S\_{t+1}) \sim P,\tilde{\pi}} \left[R(S\_t, A\_t)+\gamma Z^{\pi}(S\_{t+1})-Z^{\pi}(S\_t) | S\_0 = s\right] + Z^{\pi}(s)\\\\
&= Z^{\pi}(s)+\sum\_{t=0}^\infty \gamma^t \mathbb{E}\_{(S_t,A_t)\sim P,\tilde{\pi}} \left[D^{\pi}(S\_t,A\_t,R\_t)-Z^{\pi}(S\_t) | S\_0 = s\right]\\\\
&= Z^{\pi}(s)+ \frac{1}{1-\gamma} \mathbb{E}\_{(s,a)\sim d^{\tilde{\pi}},\tilde{\pi}} \left[D^{\pi}(s,a,r)-Z^{\pi}(s)\right].
\end{align}$$

        Thus, we can derive the step 25 from the step 24.
    - For (3), we apologize for any confusion caused by the notations. We will replace the notation $D^\pi (s,a)$ with $D^\pi (s,a,r)$ in Theorem A.3 of our revision.

    [1] Approximately optimal approximate reinforcement learning.

**Q17**. (1) What is the value of $c$ we use in experiments? (2) What is the effect on performance for varying $c$?
  - **A17**.
    - For (1): 
      - For each experiment with a random seed under individual corruptions in Tables 2 and 3 of our main text, we randomly select $c\\% = 30\\%$ of transitions from the offline dataset. Within these selected transitions, we replace one element per transition with corrupted element.
      - In Table 1 with simultaneous corruptions, we also apply $c\\% = 30\\%$ but extend the corruption process across all four elements of the offline dataset. Specifically, we randomly select $30\\%$ of transitions and corrupt one element in each selected transition. Then, we repeat this step four times until all elements are corrupted. Therefore, approximately $76.0\\%$ of data in the offline dataset is corrupted, calculated as $1 - (1 - c)^4$.

        In Table G4 of our attached PDF of the global response, we evaluate TRACER using different $c \\%$, including $10\\%, 20\\%, 30\\%, 40\\%,$ and $50\\%$. These rates correspond to approximately $34.4\\%, 59.0\\%, 76.0\\%, 87.0\\%$, and $93.8\\%$ of the data being corrupted.
    - For (2):
      - Results in Table G4 show that while TRACER is robust to simultaneous corruptions and significantly outperforms RIQL, its performance depends on the extent of corrupted data it encounters, degrading with increased data corruptions.

We hope our responses adequately address your concerns.  If you have further concerns, please let us know and we will continue actively responding to your comments, enhancing our submission. We would deeply appreciate it if you could raise your score based on these revisions.
Thank you for your efforts in answering the questions and conducting additional experiments. I specifically like Fig G1 which helps in understanding the proposed methodology. I still have the following concerns:

1. Theorem A.3 is unclear to me since it is derived in terms of $\tilde \pi_{iql}$ which as per my understand is iql learned on corrupted data. How does this prove the upper bound for TRACER? How do you equate $Z^\pi(s)$ with $D^\pi(s, a)$ is steps 24, 25 and 26. And what is $D^\pi(s, a)$ as previously D is defined as D(s, a, r)?
2. The authors in Appendix B state we randomly select c% of the transitions each time for corruption. I am interested in knowing what is the value of c experimentally and what the effect on performance for varying c.
For Q8, we provide the detailed table summarizing the notations used in our Appendix. See Table G8 as follows.

Table G8. Notations in our Appendix.

| Notations used in our Appendix | Descriptions                                                 |
| ------------------------------ | ------------------------------------------------------------ |
| $\mathbb{\zeta}$               | Cumulative corruption level.                                 |
| $\mathbb{\zeta}_i$             | Metric quantifying the corruption level of rewards and next states (transition dynamics). |
| $\mathbb{\zeta}_i^{'}$         | Metric quantifying the corruption level of states and actions. |
| $\pi_{b}(\cdot\|s)$            | The behavior policy that is used to collect clean data.      |
| $\pi_{\mathcal{B}}(\cdot\|s)$  | The behavior policy that is used to collect corrupted data.  |
| $\pi_{E}(\cdot\|s)$            | The policy that we want to learn under clean data.           |
| $\tilde{\pi}_{E}(\cdot\|s)$    | The policy that we are learning under corrupted data.        |
| $\pi_{\text{IQL}}(\cdot\|s)$          | The learned policy using IQL's weighted imitation learning under clean data. |
| $\tilde{\pi}_{\text{IQL}}(\cdot\|s)$  | The learned policy using IQL's weighted imitation learning under corrupted data. |
| $d^{\pi}(s,a)$                 | The probability density function associated with policy $\pi$ at state $s$ and action $a$. |
| $W_1(p,q)$                     | The Wasserstein-1 distance that measures the difference between distributions $p$ and $q$. |
| $q^{\pi}(\cdot \| s)$            | The value distribution of the policy $\pi$.                  |
| $Z^{\pi}(s)$                   | The random variable of the value distribution $q^{\pi}(\cdot\|s)$. |
| $\epsilon_1$                   | The KL divergence between policies $\pi_{E}$ and $\pi_{\text{IQL}}$, representing standard imitation error under clean data. |
| $\epsilon_2$                   | The KL divergence between policies $\tilde{\pi}\_{E}$ and $\tilde{\pi}\_{\text{IQL}}$, representing the standard imitation error under corrupted data. |
| $\hat{\varsigma}_n$            | The midpoint of the action-value distribution $p_{\theta}$.        |