Title: Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptions
Venue: NeurIPS 2024
Accepted: yes
Keywords: Robust Offline Reinforcement Learning

Variational Bayesian Inference

Diverse Data Corruptions

Uncertainty
Abstract: Real-world offline datasets are often subject to data corruptions (such as noise or adversarial attacks) due to sensor failures or malicious attacks. Despite advances in robust offline reinforcement learning (RL), existing methods struggle to learn robust agents under high uncertainty caused by the diverse corrupted data (i.e., corrupted states, actions, rewards, and dynamics), leading to performance degradation in clean environments. To tackle this problem, we propose a novel robust variational Bayesian inference for offline RL (TRACER). It introduces Bayesian inference for the first time to capture the uncertainty via offline data for robustness against all types of data corruptions. Specifically, TRACER first models all corruptions as the uncertainty in the action-value function. Then, to capture such uncertainty, it uses all offline data as the observations to approximate the posterior distribution of the action-value function under a Bayesian inference framework. An appealing feature of TRACER is that it can distinguish corrupted data from clean data using an entropy-based uncertainty measure, since corrupted data often induces higher uncertainty and entropy. Based on the aforementioned measure, TRACER can regulate the loss associated with corrupted data to reduce its influence, thereby enhancing robustness and performance in clean environments. Experiments demonstrate that TRACER significantly outperforms several state-of-the-art approaches across both individual and simultaneous data corruptions.

Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptions

Abstract

Real-world offline datasets are often subject to data corruptions (such as noise or adversarial attacks) due to sensor failures or malicious attacks. Despite advances in robust offline reinforcement learning (RL), existing methods struggle to learn robust agents under high uncertainty caused by the diverse corrupted data (i.e., corrupted states, actions, rewards, and dynamics), leading to performance degradation in clean environments. To tackle this problem, we propose a novel robus t va r iational B a yesian inferen c e for offlin e R L (TRACER). It introduces Bayesian inference for the first time to capture the uncertainty via offline data for robustness against all types of data corruptions. Specifically, TRACER first models all corruptions as the uncertainty in the action-value function. Then, to capture such uncertainty, it uses all offline data as the observations to approximate the posterior distribution of the action-value function under a Bayesian inference framework. An appealing feature of TRACER is that it can distinguish corrupted data from clean data using an entropy-based uncertainty measure, since corrupted data often induces higher uncertainty and entropy. Based on the aforementioned measure, TRACER can regulate the loss associated with corrupted data to reduce its influence, thereby enhancing robustness and performance in clean environments. Experiments demonstrate that TRACER significantly outperforms several state-of-the-art approaches across both individual and simultaneous data corruptions.

1 Introduction

Offline reinforcement learning (RL) aims to learn an effective policy from a fixed dataset without direct interaction with the environment [1, 2]. This paradigm has recently attracted much attention in scenarios where real-time data collection is expensive, risky, or impractical, such as in healthcare [3], autonomous driving [4], and industrial automation [5]. Due to the restriction of the dataset, offline RL confronts the challenge of distribution shift between the policy represented in the offline dataset and the policy being learned, which often leads to the overestimation for out-of-distribution (OOD) actions [1, 6, 7]. To address this challenge, one of the promising approaches introduce uncertainty estimation techniques, such as using the ensemble of action-value functions or Bayesian inference to measure the uncertainty of the dynamics model [8-11] or the action-value function [12-15] regarding the rewards and transition dynamics. Therefore, they can constrain the learned policy to remain close to the policy represented in the dataset, guiding the policy to be robust against OOD actions.

𝑎𝑡-1

𝑎𝑡

𝑠𝑡+1

𝑟 𝑡-1

𝑟 𝑡

𝑟 𝑡+1

𝑟 𝑡+2

𝑄𝑡+1

𝜋

𝑄𝑡

𝜋

𝑎𝑡+1

𝑎𝑡+2

𝜋

𝑠𝑡+2

𝑟 𝑡+3

𝑎𝑡+3

𝑄𝑡+3

𝜋

𝑠𝑡+3

…

…

𝑠𝑡-2

𝑎𝑡-2

𝑟 𝑡-2

𝑄𝑡-2

𝜋

𝑄𝑡-1

𝜋

𝑠𝑡-1

Nevertheless, in the real world, the dataset collected by sensors or humans may be subject to extensive and diverse corruptions [16-18], e.g., random noise from sensor failures or adversarial attacks during RLHF data collection. Offline RL methods often assume that the dataset is clean and representative of the environment. Thus, when the data is corrupted, the methods experience performance degradation in the clean environment, as they often constrain policies close to the corrupted data distribution.

Despite advances in robust offline RL [2], these approaches struggle to address the challenges posed by diverse data corruptions [18]. Specifically, many previous methods on robust offline RL aim to enhance the testing-time robustness, learning from clean datasets and defending against attacks during testing [19-21]. However, they cannot exhibit robust performance using offline dataset with perturbations while evaluating the agent in a clean environment. Some related works for data corruptions (also known as corruption-robust offline RL methods) introduce statistical robustness and stability certification to improve performance, but they primarily focus on enhancing robustness against adversarial attacks [16, 22, 23]. Other approaches focus on the robustness against both random noise and adversarial attacks, but they often aim to address only corruptions in states, rewards, or transition dynamics [24, 17]. Based on these methods, recent work [18] extends the data corruptions to all four elements in the dataset, including states, actions, rewards, and dynamics. This work demonstrates the superiority of the supervised policy learning scheme [25, 26] for the data corruption of each element in the dataset. However, as it does not take into account the uncertainty in decisionmaking caused by the simultaneous presence of diverse corrupted data, this work still encounters difficulties in learning robust agents, limiting its applications in real-world scenarios.

In this paper, we propose to use offline data as the observations, thus leveraging their correlations to capture the uncertainty induced by all corrupted data. Considering that (1) diverse corruptions may introduce uncertainties into all elements in the offline dataset, and (2) each element is correlated with the action values (see dashed lines in Figure 1), there is high uncertainty in approximating the action-value function by using various corrupted data. To address this high uncertainty, we propose to leverage all elements in the dataset as observations, based on the graphical model in Figure 1. By using the high correlations between these observations and the action values [27], we can accurately identify the uncertainty of the action-value function.

Motivated by this idea, we propose a robus t va r iational B a yesian inferen c e for offlin e R L (TRACER) to capture the uncertainty via offline data against all types of data corruptions. Specifically, TRACER first models all data corruptions as uncertainty in the action-value function. Then, to capture such uncertainty, it introduces variational Bayesian inference [28], which uses all offline data as observations to approximate the posterior distribution of the action-value function. Moreover, the corrupted observed data often induce higher uncertainty than clean data, resulting in higher entropy in the distribution of action-value function. Thus, TRACER can use the entropy as an uncertainty measure to effectively distinguish corrupted data from clean data. Based on the entropy-based uncertainty measure, it can regulate the loss associated with corrupted data in approximating the action-value distribution. This approach effectively reduces the influence of corrupted samples, enhancing robustness and performance in clean environments.

This study introduces Bayesian inference into offline RL for data corruptions. It significantly captures the uncertainty caused by diverse corrupted data, thereby improving both robustness and performance in offline RL. Moreover, it is important to note that, unlike traditional Bayesian online and offline RL methods that only model uncertainty from rewards and dynamics [29-35], our approach identifies the

uncertainty of the action-value function regarding states, actions, rewards, and dynamics under data corruptions. We summarize our contributions as follows.

To the best of our knowledge, this study introduces Bayesian inference into corruption-robust offline RL for the first time. By leveraging all offline data as observations, it can capture uncertainty in the action-value function caused by diverse corrupted data.

By introducing an entropy-based uncertainty measure, TRACER can distinguish corrupted from clean data, thereby regulating the loss associated with corrupted samples to reduce its influence for robustness.

Experiment results show that TRACER significantly outperforms several state-of-the-art offline RL methods across a range of both individual and simultaneous data corruptions.

2 Preliminaries

Bayesian RL. We consider a Markov decision process (MDP), denoted by a tuple M = ( S , A , R , P, P 0 , γ ) , where S is the state space, A is the action space, R is the reward space, P ( ·| s, a ) ∈ P ( S ) is the transition probability distribution over next states conditioned on a stateaction pair ( s, a ) , P 0 ( · ) ∈ P ( S ) is the probability distribution of initial states, and γ ∈ [0 , 1) is the discount factor. Note that P ( S ) and P ( A ) denote the sets of probability distributions on subsets of S and A , respectively. For simplicity, throughout the paper, we use uppercase letters to refer to random variables and lowercase letters to denote values taken by the random variables. Specifically, R ( s, a ) denotes the random variable of one-step reward following the distribution ρ ( r | s, a ) , and r ( s, a ) represents a value of this random variable. We assume that the random variable of one-step rewards and their expectations are bounded by R max and r max for any ( s, a ) ∈ S × A , respectively.

Our goal is to learn a policy that maximizes the expected discounted cumulative return:

.

Based on the return, we can define the value function as V π ( s ) = E π,ρ,P [ ∑ ∞ t =0 γ t R ( s t , a t ) | s 0 = s ] , the action-value function as Q π ( s, a ) = E R ∼ ρ ( ·| s,a ) ,s ′ ∼ P ( ·| s,a ) [ R ( s, a ) + γV π ( s ′ )] , and the actionvalue distribution [36] as

Variational Inference. Variational inference is a powerful method for approximating complex posterior distributions, which is effective for RL to handle the parameter uncertainty and deal with modelling errors [36]. Given an observation X and latent variables Z , Bayesian inference aims to compute the posterior distribution p ( Z | X ) . Direct computation of this posterior is often intractable due to the high-dimensional integrals involved. To approximate the true posterior, Bayesian inference introduces a parameterized distribution q ( Z ; ϕ ) and minimizes the Kullback-Leibler (KL) divergence D KL ( q ( Z ; ϕ ) ∥ p ( Z | X )) . Note that minimizing the KL divergence is equivalent to maximizing the evidence lower bound (ELBO) [37, 38]: ELBO( ϕ ) = E q ( Z ; ϕ ) [log p ( X , Z ) -log q ( Z ; ϕ )] .

Offline RL under Diverse Data Corruptions. In the real world, the data collected by sensors or humans may be subject to diverse corruption due to sensor failures or malicious attacks. Let b and B denotes the uncorrupted and corrupted dataset with samples { ( s i t , a i t , r i t , s i t +1 ) } N i =1 , respectively. Each data in B may be corrupted. We assume that an uncorrupted state follows a state distribution p b ( · ) , a corrupted state follows p B ( · ) , an uncorrupted action follows a behavior policy π b ( ·| s i t ) , a corrupted action is sampled from π B ( ·| s i t ) , a corrupted reward is sampled from ρ B ( ·| s i t , a i t ) , and a corrupted next state is drawn from P B ( ·| s i t , a i t ) . We also denote the uncorrupted and corrupted empirical state-action distributions as p b ( s i t , a i t ) and p B ( s i t , a i t ) , respectively. Moreover, we introduce the notations [18, 39] as follows.

for any ( s, a ) ∈ S × A and Q : S × A ↦→ [0 , r max / (1 -γ )] , where X : D = Y denotes equality of probability laws, that is the random variable X is distributed according to the same law as Y .

To address the diverse data corruptions, based on IQL [26], RIQL [18] introduces quantile estimators with an ensemble of action-value functions { Q θ i ( s, a ) } K i =1 and employs a Huber regression [40]:

Note that l κ H is the Huber loss, and Q α is the α -quantile value among { Q θ i ( s, a ) } K i =1 . RIQL then follows IQL [26] to learn the policy using weighted imitation learning with a hyperparameter β :

3 Algorithm

We first introduce the Bayesian inference for capturing the uncertainty caused by diverse corrupted data in Section 3.1. Then, we provide our algorithm TRACER with the entropy-based uncertainty measure in Section 3.2. Moreover, we provide the theoretical analysis for robustness, the architecture, and the detailed implementation of TRACER in Appendices A.1, B.1, and B.2, respectively.

3.1 Variational Inference for Uncertainty induced by Corrupted Data

We focus on corruption-robust offline RL to learn an agent under diverse data corruptions, i.e., random or adversarial attacks on four elements of the dataset. We propose to use all elements as observations, leveraging the data correlations to simultaneously address the uncertainty. By introducing Bayesian inference framework, our aim is to approximate the posterior distribution of the action-value function.

At the beginning, based on the relationships between the action values and the four elements (i.e., states, actions, rewards, next states) in the offline dataset as shown in Figure 1, we define D θ = D θ ( S, A, R ) ∼ p θ ( ·| S, A, R ) , parameterized by θ . Building on the action-value distribution, we can explore how to estimate the posterior of D θ using the elements available in the offline data.

Firstly, we start from the actions { a i t } N i =1 following the corrupted distribution π B and use them as observations to approximate the posterior of the action-value distribution under a variational inference. As the actions are correlated with the action values and all other elements in the dataset, the likelihood is p φ a ( A | D,S,R,S ′ ) , parameterized by φ a . Then, under the variational inference framework, we maximize the posterior and derive to minimize the loss function based on ELBO:

where S , R , and S ′ follow the offline data distributions p B , ρ B , and P B , respectively.

Secondly, we apply the rewards { r i t } N i =1 drawn from the corrupted reward distribution ρ B as the observations. Considering that the rewards are related to the states, actions, and action values, we model the likelihood as p φ r ( R | D,S,A ) , parameterized by φ r . Therefore, we can derive a loss function by following Equation (7):

where S and A follow the offline data distributions p B and π B , respectively.

Finally, we employ the state { s i t } N i =1 in the offline dataset following the corrupted distribution p B as the observations. Due to the relation of the states, we can model the likelihood as p φ s ( S | D,A,R ) , parameterized by φ r . We then have the loss function:

where A and R follow the offline data distributions π B and ρ B , respectively. We present the detailed derivation process in Appendix A.2.

The goal of first terms in Equations (7), (8), and (9) is to estimate π B ( A | S ) , ρ B ( R | S, A ) , and p B ( S ) using p φ a ( A | D θ , S, R, S ′ ) , p φ r ( R | D θ , S, A ) , and p φ s ( S | D θ , A, R ) , respectively. As we do not

have the explicit expression of distributions π B , ρ B , and p B , we cannot directly compute the KL divergence in these first terms. To address this issue, based on the generalized Bayesian inference [41], we can exchange two distributions in the KL divergence. Then, we model all the aformentioned distributions as Gaussian distributions, and use the mean µ φ and standard deviation Σ φ to represent the corresponding p φ . For implementation, we directly employ MLPs to output each ( µ φ , Σ φ ) using the corresponding conditions of p φ . Then, based on the KL divergence between two Gaussian distributions, we can derive the loss function as follows.

Moreover, the goal of second terms in Equations (7), (8), and (9) is to maximize the likelihoods of D θ given samples ˆ s ∼ p φ s , ˆ a ∼ p φ a , or ˆ r ∼ p φ r . Thus, with ( s, a, r ) ∼ B , we propose minimizing the distance between D θ (ˆ s, a, r ) and D ( s, a, r ) , D θ ( s, ˆ a, r ) and D ( s, a, r ) , and D θ ( s, a, ˆ r ) and D ( s, a, r ) , where ˆ s ∼ p φ s , ˆ a ∼ p φ a , and ˆ r ∼ p φ r . Then, based on [41], we can derive the following loss with any metric ℓ to maximize the log probabilities:

3.2 Corruption-Robust Algorithm with the Entropy-based Uncertainty Measure

We focus on developing tractable loss functions for implementation in this subsection.

Learning the Action-Value Distribution based on Temporal Difference (TD). Based on [42, 43], we introduce the quantile regression [44] to approximate the action-value distribution in the offline dataset B using an ensemble model { D θ i } K i =1 . We use Equation (4) to derive the loss as:

where ρ κ τ ( δ ) = | τ -I { δ < 0 }| · l κ H ( δ ) with the threshold κ , Z denotes the value distribution, δ τ,τ ′ θ i is the sampled TD error based on the parameters θ i , τ and τ ′ are two samples drawn from a uniform distribution U ([0 , 1]) , D τ θ ( s, a, r ) := F -1 D θ ( s,a,r ) ( τ ) is the sample drawn from p θ ( ·| s, a, r ) , Z τ ( s ) := F -1 Z ( s ) ( τ ) is sampled from p ( ·| s ) , F -1 X ( τ ) is the inverse cumulative distribution function (also known as quantile function) [45] at τ for the random variable X , and N and N ′ represent the respective number of iid samples τ and τ ′ . Notably, based on [43], we have Q θ i ( s, a ) = ∑ N n =1 D τ n θ i ( s, a, r ) .

In addition, if we learn the value distribution Z , the action-value distribution can extract the information from the next states based on Equation (12), which is effective for capturing the uncertainty. On the contrary, if we directly use the next states in the offline dataset as the observations, in practice, the parameterized model of the action-value distribution needs to take ( s, a, r, s ′ , a ′ , r ′ , s ′′ ) as the input data. Thus, the model can compute the action values and values for the sampled TD error in Equation (12). To avoid the changes in the input data caused by directly using next states as observations in Bayesian inference, we draw inspiration from IQL and RIQL to learn a parameterized value distribution. Based on Equations (5) and (12), we derive a new objective as:

where D τ α is the α -quantile value among { D τ θ i ( s, a ) } K i =1 , and V ψ ( s ) = ∑ N n =1 Z τ n ψ ( s ) . More details are shown in Appendix B.2. Furthermore, we provide the theoretical analysis in Appendix A.1 to give a value bound between the value distributions under clean and corrupted data.

Updating the Action-Value Distribution based on Variational Inference for Robustness. We discuss the detailed implementation of Equations (10) and (11) based on Equations (12) and (13). As the data

corruptions may introduce heavy-tailed targets [18], we apply the Huber loss to replace all quadratic loss in Equation (10) and the metric ℓ in Equation (11), mitigating the issue caused by heavy-tailed targets [46] for robustness. We rewrite Equation (11) as follows.

Thus, we have the whole loss function L D | S,A,R = L first ( θ i , φ s , φ a , φ r ) + L second ( θ i , φ s , φ a , φ r ) in the generalized variational inference framework. Moreover, based on the assumption of heavy-tailed noise in [18], we have a upper bound of action-value distribution by using the Huber regression loss.

Entropy-based Uncertainty Measure for Regulating the Loss associated with Corrupted Data. To further address the challenge posed by diverse data corruptions, we consider the problem: how to exploit uncertainty to further enhance robustness?

Considering that our goal is to improve performance in clean environments, we propose to reduce the influence of corrupted data, focusing on using clean data to learn agents. Therefore, we provide a two-step plan: (1) distinguishing corrupted data from clean data; (2) regulating the loss associated with corrupted data to reduce its influence, thus enhancing the performance in clean environments.

For (1), as the Shannon entropy for the measures of aleatoric and epistemic uncertainties provides important insight [47-49], and the corrupted data often results in higher uncertainty and entropy of the action-value distribution than the clean data, we use entropy [50] to quantify uncertainties of corrupted and clean data. Furthermore, by considering that the exponential function can amplify the numerical difference in entropy between corrupted and clean data, we propose the use of exponential entropy [51]-a metric of extent of a distribution-to design our uncertainty measure.

Specifically, based on Equation 12, we can use the quantile points { τ n } N n =1 to learn the corresponding quantile function values { D τ n } N n =1 drawn from the action-value distribution p θ . We sort the quantile points and their corresponding function values in ascending order based on the values. Thus, we have the sorted sets { ς n } N n =1 , { D ς n } N n =1 , and the estimated PDF values { ς n } N n =1 , where ς 1 = ς 1 and ς n = ς n -ς n -1 for 1 < n ≤ N . Then, we can further estimate differential entropy following [52] (see Appendix A.3 for a detailed derivation).

where ˆ ς n denotes ( ς n -1 + ς n ) / 2 for 1 < n ≤ N , and D ς n denotes D ς n -D ς n -1 for 1 < n ≤ N .

For (2), TRACER employs the reciprocal value of exponential entropy 1 / exp( H ( p θ i )) to weight the corresponding loss of θ i in our proposed whole loss function L D | S,A,R . Therefore, during the learning process, TRACER can regulate the loss associated with corrupted data and focus on minimizing the loss associated with clean data, enhancing robustness and performance in clean environments. Note that we normalize entropy values by dividing the mean of samples (i.e., quantile function values) drawn from action-value distributions for each batch. In Figure 3, we show the relationship of entropy values of corrupted and clean data estimated by Equation (15) during the learning process. The results illustrate the effectiveness of the entropy-weighted technique for data corruptions.

Updating the Policy based on the Action-Value Distribution. We directly applies the weighted imitation learning technique in Equation (6) to learn the policy. As Q α ( s, a ) is the α -quantile value

4 Experiments

In this section, we show the effectiveness of TRACER across various simulation tasks using diverse corrupted offline datasets. Firstly, we provide our experiment setting, focusing on the corruption settings for offline datasets. Then, we illustrate how TRACER significantly outperforms previous stateof-the-art approaches under a range of both individual and simultaneous data corruptions. Finally, we conduct validation experiments and ablation studies to show the effectiveness of TRACER.

4.1 Experiment Setting

Building upon RIQL [18], we use two hyperparameters, i.e., corruption rate c ∈ [0 , 1] and corruption scale ϵ , to control the corruption level. Then, we introduce the random corruption and adversarial corruption in four elements (i.e., states, actions, rewards, next states) of offline datasets. The implementation of random corruption is to add random noise to elements of a c portion of the offline datasets, and the implementation of adversarial corruption follows the Projected Gradient Descent attack [53, 54] using pretrained value functions. Note that unlike other adversarial corruptions, the adversarial reward corruption multiplies -ϵ to the clean rewards instead of using gradient optimization. We also introduce the random or adversarial simultaneous corruption, which refers to random or adversarial corruption simultaneously present in four elements of the offline datasets. We apply the corruption rate c = 0 . 3 and corruption scale ϵ = 1 . 0 in our experiments.

c

= 0

.

3

, /epsilon1

= 1

.

0

Random Simultaneous Corruptions

0

5

10

15

20

25

Average Score

CARLA-Lane-v0

IQL

RIQL

TRACER

c

= 0

.

1

, /epsilon1

= 1

.

0

c

= 0

.

1

, /epsilon1

= 2

.

0

c

= 0

.

2

, /epsilon1

= 1

.

0

c

= 0

.

2

, /epsilon1

= 2

.

0

Corrupt Level

0

10

20

30

40

50

60

70

80

Average Score

Hopper-Medium-Replay-v2

IQL

RIQL

TRACER

We conduct experiments on D4RL benchmark [55]. Referring to RIQL, we train all agents for 3000 epochs on the 'medium-replay-v2' dataset, which closely mirrors real-world applications as it is collected during the training of a SAC agent. Then, we evaluate agents in clean environments, reporting the average normalized performance over four random seeds. See Appendix C for detailed information. The algorithms we compare include: (1) CQL [7] and IQL [26], offline RL algorithms using a twin Q networks. (2) EDAC [56] and MSG [57], offline RL algorithms using ensemble Q networks (number of ensembles > 2 ). (3) UWMSG [17] and RIQL, state-of-the-arts in corruption-robust offline RL. Note that EDAC, MSG, and UWMSG are all uncertainty-based offline RL algorithms.

4.2 Main results under Diverse Data Corruptions

We conduct experiments on MuJoCo [58] (see Tables 1, 2, and 3, which highlight the highest results) and CARLA [59] (see the left of Figure 2) tasks from D4RL under diverse corruptions to show the superiority of TRACER. In Table 1, we report all results under random or adversarial simultaneous data corruptions. These results show that TRACER significantly outperforms other algorithms in all tasks, achieving an average score improvement of + 21 . 1 % . In the left of Figure 2, results on 'CARLA-lane_v0' under random simultaneous corruptions also illustrate the superiority of TRACER. See Appendix C.2 for details.

Random Corruptions. We report the results under random simultaneous data corruptions of all algorithms in Table 1. Such results demonstrate that TRACER achieves an average score gain of + 22 . 4 % under the setting of random simultaneous corruptions. Based on the results, it is clear that many offline RL algorithms, such as EDAC, MSG, and CQL, suffer the performance degradation under data corruptions. Since UWMSG is designed to defend the corruptions in rewards and dynamics, its performance degrades when faced with the stronger random simultaneous corruption. Moreover, we report results across a range of individual random data corruptions in Table 2, where TRACER outperforms previous algorithms in 7 out of 12 settings. We then explore hyperparameter tuning on Hopper task and further improve TRACER's results, demonstrating its potential for performance gains. We provide details in Appendix C.3.1.

Adversarial Corruptions. We construct experiments under adversarial simultaneous corruptions to evaluate the robustness of TRACER. The results in Table 1 show that TRACER surpasses others by a significant margin, achieving an average score improvement of + 19 . 3 % . In these simultaneous corruption, many algorithms experience more severe performance degradation compared to the random simultaneous corruption, which indicates that adversarial attacks are more damaging to the reliability of algorithms than random noise. Despite these challenges, TRACER consistently achieves significant performance gains over other methods. Moreover, we provide the results across a range of individual adversarial data corruptions in Table 3, where TRACER outperforms previous algorithms in 7 out of 12 settings. We also explore hyperparameter tuning on Hopper task and further improve TRACER's results, demonstrating its potential for performance gains. See Appendix C.3.1 for details.





















































































































































































4.3 Evaluation of TRACER under Various Corruption Levels

Building upon RIQL, we further extend our experiments to include Mujoco datasets with various corruption levels, using different corruption rates c and scales ϵ . We report the average scores and standard deviations over four random seeds in the right of Figure 2, using batch sizes of 256.

Results in the right of Figure 2 demonstrate that TRACER significantly outperforms baseline algorithms in all tasks under random simultaneous corruptions with various corruption levels. It achieves an average score improvement of + 33 . 6 % . Moreover, as the corruption levels increase, the slight decrease in TRACER's results indicates that while TRACER is robust to simultaneous corruptions, its performance depends on the extent of corrupted data it encounters. We also evaluate TRACER in different scales of corrupted data and provide the results in Appendix C.4.1.

4.4 Evaluation of the Entropy-based Uncertainty Measure

We evaluate the entropy-based uncertainty measure in action-value distributions to show: (1) is the uncertainty caused by corrupted data higher than that of clean data? (2) is regulating the loss associated with corrupted data effective in improving performance?

For (1), we first introduce labels indicating whether the data is corrupted. Importantly, these labels are not used by agents during the training process. Then, we estimate entropy values of labelled corrupted and clean data in each batch based on Equation (15). Thus, we can compare entropy values to compute results, showing how many times the entropy of the corrupted data is higher than that of clean data. Specifically, we evaluate the accuracy every 50 epochs over 3000 epochs. For each evaluation, we sample 500 batches to compute the average entropy of corrupted and clean data. Each batch consists of 32 clean and 32 corrupted data. We illustrate the curves over three seeds in the second and third columns of Figure 3, where each point shows how many of the 500 batches have higher entropy for corrupted data than that of clean data.

Figure 3 indicates an oscillating upward trend of TRACER's measurement accuracy using entropy ( TRACER Using Entro ) under simultaneous corruptions, demonstrating that using the entropy-based uncertainty measure can effectively distinguish corrupted data from clean data. These curves also reveal that even in the absence of any constraints on entropy ( TRACER NOT using Entro ), the entropy associated with corrupted data tends to exceed that of clean data.

For (2), in the first column of Figure 3, these results demonstrate that TRACER using the entropybased uncertainty measure can effectively reduce the influence of corrupted data, thereby enhancing robustness and performance against all corruptions. We provide detailed information for this evaluation in Appendix C.4.2.

5 Related Work

Robust RL. Robust RL can be categorized into two types: testing-time robust RL and training-time robust RL. Testing-time robust RL [19, 20] refers to training a policy on clean data and ensuring its robustness by testing in an environment with random noise or adversarial attacks. Training-time robust RL [16, 17] aims to learn a robust policy in the presence of random noise or adversarial attacks during training and evaluate the policy in a clean environment. In this paper, we focus on training-time robust RL under the offline setting, where the offline training data is subject to various data corruptions, also known as corruption-robust offline RL .

Corruption-Robust RL. Some theoretical work on corruption-robust online RL [60-63] aims to analyze the sub-optimal bounds of learned policies under data corruptions. However, these studies primarily address simple bandits or tabular MDPs and focus on the reward corruption. Some further work [64, 65] extends the modeling problem to more general MDPs and begins to investigate the corruption in transition dynamics.

It is worth noting that corruption-robust offline RL has not been widely studied. UWMSG [17] designs a value-based uncertainty-weighting technique, thus using the weight to mitigate the impact of corrupted data. RIQL [18] further extends the data corruptions to all four elements in the offline dataset, including states, actions, rewards, and next states (dynamics). It then introduces quantile estimators with an ensemble of action-value functions and employs a Huber regression based on IQL [26], alleviating the performance degradation caused by corrupted data.

Bayesian RL. Bayesian RL integrates the Bayesian inference with RL to create a framework for decision-making under uncertainty [28]. It is important to highlight that Bayesian RL is divided into two categories for different uncertainties: the parameter uncertainty in the learning of models [66, 67] and the inherent uncertainty from the data/environment in the distribution over returns [68, 69]. In this paper, we focus on capturing the latter.

For the latter uncertainty, in model-based Bayesian RL, many approaches [68, 70, 71] explicitly model the transition dynamics and using Bayesian inference to update the model. It is useful when dealing with complex environments for sample efficiency. In model-free Bayesian RL, value-based methods [69, 72] use the reward information to construct the posterior distribution of the action-value function. Besides, policy gradient methods [73, 74] use information of the return to construct the posterior distribution of the policy. They directly apply Bayesian inference to the value function or policy without explicitly modeling transition dynamics.

Offline Bayesian RL. offline Bayesian RL integrates Bayesian inference with offline RL to tackle the challenges of learning robust policies from static datasets without further interactions with the environment. Many approaches [75-77] use Bayesian inference to model the transition dynamics or guide action selection for adaptive policy updates, thereby avoiding overly conservative estimates in the offline setting. Furthermore, recent work [78] applies variational Bayesian inference to learn the model of transition dynamics, mitigating the distribution shift in offline RL.

6 Conclusion

In this paper, we investigate and demonstrate the robustness and effectiveness of introducing Bayesian inference into offline RL to address the challenges posed by data corruptions. By leveraging Bayesian techniques, our proposed approach TRACER captures the uncertainty caused by diverse corrupted data. Moreover, the use of entropy-based uncertainty measure in TRACER can distinguish corrupted data from clean data. Thus, TRACER can regulate the loss associated with corrupted data to reduce its influence, improving performance in clean environments. Our extensive experiments demonstrate the potential of Bayesian methods in developing reliable decision-making.

Regarding the limitations of TRACER, although it achieves significant performance improvement under diverse data corruptions, future work could explore more complex and realistic data corruption scenarios and related challenges, such as the noise in the preference data for RLHF and adversarial attacks on safety-critical driving decisions. Moreover, we look forward to the continued development and optimization of uncertainty-based corrupted-robust offline RL, which could further enhance the effectiveness of TRACER and similar approaches for increasingly complex real-world scenarios.

Acknowledgments

We would like to thank all the anonymous reviewers for their insightful comments. This work was supported in part by National Key R&D Program of China under contract 2022ZD0119801, National Nature Science Foundations of China grants U23A20388 and 62021001, and DiDi GAIA Collaborative Research Funds.

References

Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 2052-2062. PMLR, 2019.

Rafael Figueiredo Prudencio, Marcos R. O. A. Máximo, and Esther Luna Colombini. A survey on offline reinforcement learning: Taxonomy, review, and open problems. CoRR , abs/2203.01387, 2022.

Shengpu Tang and Jenna Wiens. Model selection for offline reinforcement learning: Practical considerations for healthcare settings. In Ken Jung, Serena Yeung, Mark P. Sendak, Michael W. Sjoding, and Rajesh Ranganath, editors, Proceedings of the Machine Learning for Healthcare Conference , volume 149 of Proceedings of Machine Learning Research , pages 2-35. PMLR, 2021.

Christopher Diehl, Timo Sievernich, Martin Krüger, Frank Hoffmann, and Torsten Bertram. Uncertaintyaware model-based offline reinforcement learning for automated driving. IEEE Robotics Autom. Lett. , 8(2):1167-1174, 2023.

Tony Z. Zhao, Jianlan Luo, Oleg Sushkov, Rugile Pevceviciute, Nicolas Heess, Jon Scholz, Stefan Schaal, and Sergey Levine. Offline meta-reinforcement learning for industrial insertion. In 2022 International Conference on Robotics and Automation , pages 6386-6393. IEEE, 2022.

Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32 , pages 11761-11771, 2019.

Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative q-learning for offline reinforcement learning. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35 , 2022.

Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y. Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: model-based offline policy optimization. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33 , 2020.

Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33 , 2020.

Porter Jenkins, Hua Wei, J. Stockton Jenkins, and Zhenhui Li. Bayesian model-based offline reinforcement learning for product allocation. In Thirty-Sixth AAAI Conference on Artificial Intelligence , pages 1253112537. AAAI Press, 2022.

Sen Lin, Jialin Wan, Tengyu Xu, Yingbin Liang, and Junshan Zhang. Model-based offline metareinforcement learning with regularization. In The Tenth International Conference on Learning Representations . OpenReview.net, 2022.

Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34 , pages 7436-7447, 2021.

Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua M. Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 11319-11328. PMLR, 2021.

Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhi-Hong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. In The Tenth International Conference on Learning Representations . OpenReview.net, 2022.

Filippo Valdettaro and A. Aldo Faisal. Towards offline reinforcement learning with pessimistic value priors. In Fabio Cuzzolin and Maryam Sultana, editors, Epistemic Uncertainty in Artificial Intelligence - First International Workshop , volume 14523 of Lecture Notes in Computer Science , pages 89-100. Springer, 2024.

Xuezhou Zhang, Yiding Chen, Xiaojin Zhu, and Wen Sun. Corruption-robust offline reinforcement learning. In Gustau Camps-Valls, Francisco J. R. Ruiz, and Isabel Valera, editors, International Conference on Artificial Intelligence and Statistics , volume 151 of Proceedings of Machine Learning Research , pages 5757-5773. PMLR, 2022.

Chenlu Ye, Rui Yang, Quanquan Gu, and Tong Zhang. Corruption-robust offline reinforcement learning with general function approximation. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36 , 2023.

Rui Yang, Han Zhong, Jiawei Xu, Amy Zhang, Chongjie Zhang, Lei Han, and Tong Zhang. Towards robust offline reinforcement learning under diverse data corruption. In The Eleventh International Conference on Learning Representations , 2023.

Kishan Panaganti, Zaiyan Xu, Dileep Kalathil, and Mohammad Ghavamzadeh. Robust reinforcement learning using offline data. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35 , 2022.

Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. RORL: robust offline reinforcement learning via conservative smoothing. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35 , 2022.

Jose H. Blanchet, Miao Lu, Tong Zhang, and Han Zhong. Double pessimism is provably efficient for distributionally robust offline reinforcement learning: Generic algorithm and robust partial coverage. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36 , 2023.

Yuzhe Ma, Xuezhou Zhang, Wen Sun, and Jerry Zhu. Policy poisoning in batch reinforcement learning and control. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32 , pages 14543-14553, 2019.

Fan Wu, Linyi Li, Huan Zhang, Bhavya Kailkhura, Krishnaram Kenthapadi, Ding Zhao, and Bo Li. COPA: certifying robust policies for offline reinforcement learning against poisoning attacks. In The Tenth International Conference on Learning Representations . OpenReview.net, 2022.

Zhihe Yang and Yunjian Xu. Dmbp: Diffusion model based predictor for robust offline reinforcement learning against state observation perturbations. In The Twelfth International Conference on Learning Representations , 2023.

Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. CoRR , abs/1910.00177, 2019.

Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In The Tenth International Conference on Learning Representations . OpenReview.net, 2022.

Rens van de Schoot, Sarah Depaoli, Ruth King, Bianca Kramer, Kaspar Märtens, Mahlet G Tadesse, Marina Vannucci, Andrew Gelman, Duco Veen, Joukje Willemsen, et al. Bayesian statistics and modelling. Nature Reviews Methods Primers , 1(1):1, 2021.

Box George EP and Tiao George C. Bayesian inference in statistical analysis . John Wiley & Sons, 2011.

Esther Derman, Daniel J. Mankowitz, Timothy A. Mann, and Shie Mannor. A bayesian approach to robust reinforcement learning. In Amir Globerson and Ricardo Silva, editors, Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence , volume 115 of Proceedings of Machine Learning Research , pages 648-658. AUAI Press, 2019.

Matthew Fellows, Anuj Mahajan, Tim G. J. Rudner, and Shimon Whiteson. VIREL: A variational inference framework for reinforcement learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32 , pages 7120-7134, 2019.

Mattie Fellows, Kristian Hartikainen, and Shimon Whiteson. Bayesian bellman operators. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34 , pages 13641-13656, 2021.

Brendan O'Donoghue. Variational bayesian reinforcement learning with regret bounds. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34 , pages 28208-28221, 2021.

Ron Dorfman, Idan Shenfeld, and Aviv Tamar. Offline meta reinforcement learning - identifiability challenges and effective data collection strategies. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34 , pages 4607-4618, 2021.

Dibya Ghosh, Anurag Ajay, Pulkit Agrawal, and Sergey Levine. Offline RL policies should be trained to be adaptive. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 7513-7530. PMLR, 2022.

Yuhao Wang and Enlu Zhou. Bayesian risk-averse q-learning with streaming observations. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36 , 2023.

Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, and Aviv Tamar. Bayesian reinforcement learning: A survey. Found. Trends Mach. Learn. , 8(5-6):359-483, 2015.

Carl Doersch. Tutorial on variational autoencoders. CoRR , abs/1606.05908, 2016.

Diederik P. Kingma and Max Welling. An introduction to variational autoencoders. Found. Trends Mach. Learn. , 12(4):307-392, 2019.

Marc G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 449-458. PMLR, 2017.

Kotz Samuel and Johnson Norman L. Breakthroughs in statistics: methodology and distribution . Springer Science & Business Media, 2012.

Jeremias Knoblauch, Jack Jewson, and Theodoros Damoulas. Generalized variational inference. CoRR , abs/1904.02063, 2019.

Will Dabney, Mark Rowland, Marc G. Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Sheila A. McIlraith and Kilian Q. Weinberger, editors, Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence , pages 2892-2901. AAAI Press, 2018.

Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 1104-1113. PMLR, 2018.

Koenker Roger and Hallock Kevin F. Quantile regression. Journal of economic perspectives , 15(4):143-156, 2001.

Müller Alfred. Integral probability metrics and their generating classes of functions. Advances in applied probability , 29(2):429-443, 1997.

Abhishek Roy, Krishnakumar Balasubramanian, and Murat A. Erdogdu. On empirical risk minimization with dependent and heavy-tailed data. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34 , pages 8913-8926, 2021.

Michele Caprio, Souradeep Dutta, Kuk Jin Jang, Vivian Lin, Radoslav Ivanov, Oleg Sokolsky, and Insup Lee. Credal bayesian deep learning. arXiv e-prints , pages arXiv-2302, 2023.

Michele Caprio, Yusuf Sale, Eyke Hüllermeier, and Insup Lee. A novel bayes' theorem for upper probabilities. In Fabio Cuzzolin and Maryam Sultana, editors, Epistemic Uncertainty in Artificial Intelligence - First International Workshop , volume 14523 of Lecture Notes in Computer Science , pages 1-12. Springer, 2024.

Souradeep Dutta, Michele Caprio, Vivian Lin, Matthew Cleaveland, Kuk Jin Jang, Ivan Ruchkin, Oleg Sokolsky, and Insup Lee. Distributionally robust statistical verification with imprecise neural networks. CoRR , abs/2308.14815, 2023.

Jim W. Hall. Uncertainty-based sensitivity indices for imprecise probability distributions. Reliab. Eng. Syst. Saf. , 91(10-11):1443-1451, 2006.

L Lorne Campbell. Exponential entropy as a measure of extent of a distribution. Zeitschrift für Wahrscheinlichkeitstheorie und verwandte Gebiete , 5(3):217-225, 1966.

Sheri Edwards. Thomas m. cover and joy a. thomas, elements of information theory (2nd ed.), john wiley &sons, inc. (2006). Inf. Process. Manag. , 44(1):400-401, 2008.

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations , 2018.

Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane S. Boning, and Cho-Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations on state observations. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33 , 2020.

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: datasets for deep data-driven reinforcement learning. CoRR , 2020.

Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34 , 2021.

Seyed Kamyar Seyed Ghasemipour, Shixiang Shane Gu, and Ofir Nachum. Why so pessimistic? estimating uncertainties for offline RL through ensembles, and why their independence matters. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35 , 2022.

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2012, Vilamoura, Algarve, Portugal, October 7-12, 2012 , pages 5026-5033. IEEE, 2012.

Alexey Dosovitskiy, Germán Ros, Felipe Codevilla, Antonio M. López, and Vladlen Koltun. CARLA: an open urban driving simulator. In 1st Annual Conference on Robot Learning , volume 78 of Proceedings of Machine Learning Research , pages 1-16. PMLR, 2017.

Chi Jin, Tiancheng Jin, Haipeng Luo, Suvrit Sra, and Tiancheng Yu. Learning adversarial markov decision processes with bandit feedback and unknown transition. In Proceedings of the 37th International Conference on Machine Learning , 2020.

Thodoris Lykouris, Vahab S. Mirrokni, and Renato Paes Leme. Stochastic bandits robust to adversarial corruptions. In Ilias Diakonikolas, David Kempe, and Monika Henzinger, editors, Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing , 2018.

Aviv Rosenberg and Yishay Mansour. Online convex optimization in adversarial markov decision processes. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , 2019.

Anupam Gupta, Tomer Koren, and Kunal Talwar. Better algorithms for stochastic bandits with adversarial corruptions. In Alina Beygelzimer and Daniel Hsu, editors, Conference on Learning Theory , 2019.

Chenlu Ye, Wei Xiong, Quanquan Gu, and Tong Zhang. Corruption-robust algorithms with uncertainty weighting for nonlinear contextual bandits and markov decision processes. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning , 2023.

Thodoris Lykouris, Max Simchowitz, Alex Slivkins, and Wen Sun. Corruption-robust exploration in episodic reinforcement learning. In Mikhail Belkin and Samory Kpotufe, editors, Conference on Learning Theory , 2021.

Matthieu Geist and Olivier Pietquin. Kalman temporal differences. J. Artif. Intell. Res. , 39:483-532, 2010.

Ian Osband, Daniel Russo, and Benjamin Van Roy. (more) efficient reinforcement learning via posterior sampling. In Christopher J. C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger, editors, Advances in Neural Information Processing Systems 26 , pages 3003-3011, 2013.

Marc Peter Deisenroth and Carl Edward Rasmussen. PILCO: A model-based and data-efficient approach to policy search. In Lise Getoor and Tobias Scheffer, editors, Proceedings of the 28th International Conference on Machine Learning , 2011.

Richard Dearden, Nir Friedman, and Stuart Russell. Bayesian q-learning. In Jack Mostow and Chuck Rich, editors, Proceedings of the Fifteenth National Conference on Artificial Intelligence and Tenth Innovative Applications of Artificial Intelligence Conference , 1998.

Gal Yarin, McAllister Rowan, and Rasmussen Carl Edward. Improving pilco with bayesian neural network dynamics models. In Data-efficient machine learning workshop, ICML , volume 4, page 25, 2016.

Zhihai Wang, Jie Wang, Qi Zhou, Bin Li, and Houqiang Li. Sample-efficient reinforcement learning via conservative model-based actor-critic. In Thirty-Sixth AAAI Conference on Artificial Intelligence , pages 8612-8620. AAAI Press, 2022.

Yaakov Engel, Shie Mannor, and Ron Meir. Reinforcement learning with gaussian processes. In Luc De Raedt and Stefan Wrobel, editors, Machine Learning, Proceedings of the Twenty-Second International Conference (ICML 2005) , 2005.

Mohammad Ghavamzadeh and Yaakov Engel. Bayesian policy gradient algorithms. In Bernhard Schölkopf, John C. Platt, and Thomas Hofmann, editors, Advances in Neural Information Processing Systems 19 , 2006.

Mohammad Ghavamzadeh, Yaakov Engel, and Michal Valko. Bayesian policy gradient and actor-critic algorithms. J. Mach. Learn. Res. , 2016.

Dibya Ghosh, Anurag Ajay, Pulkit Agrawal, and Sergey Levine. Offline RL policies should be trained to be adaptive. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning , 2022.

Hao Hu, Yiqin Yang, Jianing Ye, Ziqing Mai, Yujing Hu, Tangjie Lv, Changjie Fan, Qianchuan Zhao, and Chongjie Zhang. Bayesian offline-to-online reinforcement learning : A realist approach, 2024.

Yuhao Wang and Enlu Zhou. Bayesian risk-averse q-learning with streaming observations. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36 , 2023.

Toru Hishinuma and Kei Senda. Importance-weighted variational inference model estimation for offline bayesian model-based reinforcement learning. IEEE Access , 2023.

Sham M. Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Claude Sammut and Achim G. Hoffmann, editors, Machine Learning, Proceedings of the Nineteenth International Conference (ICML 2002) , pages 267-274. Morgan Kaufmann, 2002.

Vallender SS. Calculation of the wasserstein distance between probability distributions on the line. Theory of Probability & Its Applications , 18:784-786, 1974.

Anqi Li, Dipendra Misra, Andrey Kolobov, and Ching-An Cheng. Survival instinct in offline reinforcement learning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023 , 2023.

Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Advances in Neural Information Processing Systems 34 , pages 15084-15097, 2021.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR, 2018.

Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, and Dmitry Vetrov. Controlling overestimation bias with truncated mixture of continuous distributional quantile critics. In International Conference on Machine Learning , pages 5556-5566. PMLR, 2020.

Zhihai Wang, Taoxing Pan, Qi Zhou, and Jie Wang. Efficient exploration in resource-restricted reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 10279-10287, 2023.

Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in neural information processing systems , 32, 2019.

Daniel J Mankowitz, Andrea Michi, Anton Zhernov, Marco Gelmi, Marco Selvi, Cosmin Paduraru, Edouard Leurent, Shariq Iqbal, Jean-Baptiste Lespiau, Alex Ahern, et al. Faster sorting algorithms discovered using deep reinforcement learning. Nature , 618(7964):257-263, 2023.

Chris Gamble and Jim Gao. Safety-first ai for autonomous data centre cooling and industrial control. DeepMind, August , 17, 2018.

Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healthcare: A survey. ACM Computing Surveys (CSUR) , 55(1):1-36, 2021.

Zhihai Wang, Jie Wang, Dongsheng Zuo, Yunjie Ji, Xinli Xia, Yuzhe Ma, Jianye Hao, Mingxuan Yuan, Yongdong Zhang, and Feng Wu. A hierarchical adaptive multi-task reinforcement learning framework for multiplier circuit design. In Forty-first International Conference on Machine Learning . PMLR, 2024.

Zhihai Wang, Xijun Li, Jie Wang, Yufei Kuang, Mingxuan Yuan, Jia Zeng, Yongdong Zhang, and Feng Wu. Learning cut selection for mixed-integer linear programming via hierarchical sequence model. In The Eleventh International Conference on Learning Representations , 2023.

Jie Wang, Zhihai Wang, Xijun Li, Yufei Kuang, Zhihao Shi, Fangzhou Zhu, Mingxuan Yuan, Jia Zeng, Yongdong Zhang, and Feng Wu. Learning to cut via hierarchical sequence/set model for efficient mixedinteger programming. IEEE Transactions on Pattern Analysis and Machine Intelligence , pages 1-17, 2024.

Zhihai Wang, Zijie Geng, Zhaojie Tu, Jie Wang, Yuxi Qian, Zhexuan Xu, Ziyan Liu, Siyuan Xu, Zhentao Tang, Shixiong Kai, et al. Benchmarking end-to-end performance of ai-based chip placement algorithms. arXiv preprint arXiv:2407.15026 , 2024.

Appendix / supplemental material

A Proofs

We provide a proof for the upper bound of value distributions and derivations of loss functions. See Table 4 for all notations.

A.1 Proof for Value Bound

To prove an upper bound of the value distribution, we introduce an assumption from [18] below.

Assumption A.1. [18] Let ζ = ∑ N i =1 (2 ζ i +log ζ ′ i ) denote the cumulative corruption level, where ζ i and ζ ′ i are defined as

Here ∥ · ∥ ∞ means taking supremum over V : S ↦→ [0 , r max / (1 -γ )] .

Note that { ζ ′ i } i = 1 N can quantify the corruption level of states and actions, and { ζ i } i = 1 N can quantify the corruption level of rewards and next states (transition dynamics).

Then, we provide the following assumption.

Assumption A.2. There exists an M > 0 such that

where π E ( a | s ) ∝ π b ( a | s ) · exp( β · [ T Q ∗ -V ∗ ] ( s, a )) is the policy of clean data, π IQL in Equation (17) is the policy following IQL's supervised policy learning scheme under clean data, ˜ π IQL = arg min π E s ∼B [KL(˜ π E ( · | s ) , π ( · | s ))] is the policy under corrupted data, and ˜ π E ( a | s ) ∝

As TRACER directly applies the weighted imitation learning technique from IQL to learn the policy, we can use ˜ π IQL as the policy learned by TRACER under data corruptions, akin to RIQL. Assumption A.2 requires that each pair, including the policy π E and the clean data b , the policy ˜ π E and the corrupted dataset B , π E and ˜ π E , π E and π IQL, and ˜ π E and ˜ π IQL, has good coverage. It is similar to the coverage condition in [18]. Based on Assumptions A.1 and A.2, we can derive the following theorem to show the robustness of our approach using the supervised policy learning scheme and learning the action-value distribution.

Proof. Based on [79], for any ˜ π and π , we have

Theorem A.4. (Robustness). Under Assumptions A.1 and A.2, we have

where q π is the value distribution, W 1 ( · , · ) is the Wasserstein distance [80] for measuring the distribution difference, ϵ 1 = E s ∼ b [ D KL ( π E ( ·| s ) ∥ π IQL ( ·| s ))] , and ϵ 2 = E s ∼B [ D KL (˜ π E ( ·| s ) ∥ ˜ π IQL ( ·| s ))] .

Proof. The definition of Wasserstein metric is W p ( P, Q ) = ( inf γ ∈ Γ( P,Q ) ∫ x ∫ y || x -y || p p dγ ( x, y ) ) 1 p .

The value distribution q π , also denoted by p ( ·| s ) in Section 3.2 of the main text, is an expactation of action value distribution E a ∼ π,r ∼ ρ [ p θ ( ·| s, a, r )] .

Note that as TRACER directly applies the weighted imitation learning technique from IQL to learn the policy (see Section 3.2), we can use ˜ π IQL as the policy learned by TRACER under data corruptions, akin to RIQL.

Let q ˜ π IQL and q π IQL be the value distributions of learned policies following IQL's supervised policy learning scheme under corrupted and clean data, respectively. Z ˜ π IQL ∼ q ˜ π IQL and Z π IQL ∼ q π IQL . Let p π E be the value distribution of the policy of clean data, p ˜ π E be the value distribution of the policy of corrupted data, Z π E ∼ p π E , and Z ˜ π E ∼ p ˜ π E . We have

Moreover, based on the Bretagnolle-Huber inequality, we have d TV ( P, Q ) ≤ √ 1 -exp( -D KL ( P || Q )) . For W 1 ( p ˜ π E , p π E ) , we have

Moreover, we have

By the definition of corruption levels, we have

Thus, we can derive

Then, for W 1 ( p π E , q π IQL ) , we have

where ϵ 1 = E s ∼ b [ D KL ( π E ( ·| s ) ∥ π IQL ( ·| s ))] .

Finally, for W 1 ( q ˜ π IQL , p ˜ π E ) , we have

Therefore, we have

Note that in Theorem A.4, the major difference between TRACER and IQL/RIQL is that TRACER uses the action-value and value distributions rather than the action-value and value functions in IQL/RIQL. Therefore, we provide this theorem to prove an upper bound on the difference in value distributions of TRACER to show its robustness, which also provides a guarantee of how TRACER's performance degrades with increased data corruptions.

A.2 Derivation of Loss Functions

For Equations (7), (8), and (9), we provide the detailed derivations. Firstly, for Equation (7), we maximize the posterior and have

Note that L b is

Secondly, for Equation (8), we have

Finally, for Equation (9), we have

Therefore, we have Equations (7), (8), and (9).

A.3 Estimation of Entropy

We estimate differential entropy following [52]. We have differential entropy as follows. Note that we omit the condition in the following equations.

Then, we consider a continuous function p θ discretized into bins of size ∆ . By the mean-value theorem there exists a action value D i in each bin such that

We can estimate the integral of p θ (in the Riemannian sense) by

Thus, we have

Based on Equation (56), we can derive Equation (15), where ˆ ς n denotes the midpoint of p θ ( D i ) , and D ς n θ i denotes ∆ .

𝒔𝒕

𝒂𝒕

𝒓𝒕

𝒔𝒕+𝟏

𝒔𝒕

Critic Network

{𝜽𝒊 }

𝝍

𝝓

Ensemble Loss

ℒfirst

+ ℒsecond

Critic Loss

Value Loss

Actor Loss

𝒔𝒕

𝒂𝒕

𝒓𝒕

𝒔𝒕+𝟏

𝝋𝒔

:

, 𝑎

𝑡

, 𝑟

𝑡

Ensemble Models

𝝋𝒂

:

, 𝑠

𝑡

, 𝑟

𝑡

, 𝑠 𝑡+1

𝝋𝒓

:

, 𝑠

𝑡

, 𝑎

𝑡

NEW

Modified

𝒑𝝋𝒂

𝒑𝝋𝒓

𝑫𝜽𝒊

𝑫𝜽𝒊

}

𝑍𝜓

𝝅𝝓

B TRACER Approach

B.1 Architecture of TRACER

On the actor-critic architecture based on IQL, our approach TRACER adds just one ensemble model ( p φ a , p φ r , p φ s ) to reconstruct the data distribution and replace the function approximation in the critic with a distribution estimation (see Figure 4). Based on our experiments, this structure significantly improves the ability to handle both simultaneous and individual corruptions.

B.2 Implementation Details for TRACER

TRACER consists of four components: the actor network, the critic network, the value network, and the observation model. We first implement the actor network for decision-making with a 3-layer MLP, where the dimension of the hidden layers are 256 and the activation functions are ReLUs. Then, we utilize quantile regression [44] to design the critic networks and the value network, approximating the distributions of the action-value function D and value function Z , respectively.

Specifically, for the action-value distribution, we use a function h : [0 , 1] - → R d to compute an embedding for the sampling point τ . Then, we can obtain the corresponding approximation by D τ ( s, a ) ≈ f ( g ( s ) ⊙ h ( τ )) a . Here, g : S - → R d is the function computed by MLPs and f : R d - → R |A| is the subsequent fully-connected layers mapping g ( s ) to the estimated action-values D ( s, a ) ≈ f ( g ( s )) a . For the approximation of the value distribution Z τ ( s ) , we leverage the same architecture. Note that the function h is implemented by a fully-connected layer, and the output dimension is set to 256. We then use an ensemble model with K networks. Each network is a 3-layer MLP, consisting of 256 units and ReLU activations. By using the ensemble model, we can construct the critic network, using the quantile regression for optimization. Moreover, the value network is constructed by a 3-layer MLP. The dimension of the hidden layers are also 256 and activation functions are ReLUs.

The observation model uses an ensemble model with 3 neural networks. Each network is used to reconstruct different observations in the offline dataset. It consists of two fully connected layers, and the activation functions are ReLUs. We apply a masked matrix during the learning process for updating the observation model. Thus, each network in the observation model can use different input data to compute Equations (7), (8), and (9), respectively.

Following the setting of [18], we list the hyper-parameters of N , K , and α for the approxmiation of action-value functions, and κ in the Huber loss in TRACER under random and adversarial corruption in Table 5 and Table 6, respectively. Here, N is the number of samples τ , and K is the number of ensemble models.

Moreover, we apply Adam optimizer using a learning rate 1 × 10 -3 , γ = 0 . 99 , target smoothing coefficient τ = 0 . 05 , batch size 256 and the update frequency for the target network is 2. The corruption rate is c = 0 . 3 and the corruption scale is ϵ = 1 . 0 for all experiments. For the loss function of observation models, we use a linear decay parameter η from 0 . 0001 to 0 . 01 to trade off the loss L first and L second. We update each algorithm for 3000 epochs. Each epoch uses 1000 update times following [56]. Then, we evaluate each algorithm in clean environments for 100 epochs and report the average normalized return (calculated by 100 × score -random score expert score -random score ) over four random seeds.

C Detailed Experiments

C.1 Details of Data Corruptions

We follow the corruption settings proposed by [18], applying either random or adversarial corruptions to each element of the offline data, namely state, action, reward, and dynamics (or 'next-state'), to simulate potential attacks that may occur during the offline data collection and usage process in real-world scenarios. We begin with the details of random corruption below.

Random observation corruption. We randomly sample c % transitions ( s, a, r, s ′ ) from the offline dataset, and for each of these selected states s , we add noise to form ˆ s = s + λ · std ( s ) , where λ ∼ Uniform [ -ϵ, ϵ ] d s , c is the corruption rate, ϵ is the corruption scale, d s refers to the dimension of states and std ( s ) represents the d s -dimensional standard deviation of all states in the offline dataset.

Random action corruption. We randomly sample c % transitions ( s, a, r, s ′ ) from the offline dataset, and for each of these selected actions a , we add noise to form ˆ a = a + λ · std ( a ) , where λ ∼ Uniform [ -ϵ, ϵ ] d a , d a refers to the dimension of actions and std ( a ) represents the d a -dimensional standard deviation of all actions in the offline dataset.

Random reward corruption. We randomly sample c % transitions ( s, a, r, s ′ ) from the offline dataset, and for each of these selected rewards r , we modify it to ˆ r = Uniform [ -30 · ϵ, 30 · ϵ ] . We adopt this harder reward corruption setting since [81] has found that offline RL algorithms are often insensitive to small perturbations of rewards.

Random dynamics corruption. We randomly sample c % transitions ( s, a, r, s ′ ) from the offline dataset, and for each of these selected next-step states s ′ , we add noise to form ˆ s ′ = s ′ + λ · std ( s ′ ) , where λ ∼ Uniform [ -ϵ, ϵ ] d s ′ , d s ′ refers to the dimension of next-step states and std ( s ′ ) represents the d s ′ -dimensional standard deviation of all next states in the offline dataset.

The harder adversarial corruption settings are detailed as follows:

Adversarial observation corruption. To impose adversarial attack on the offline dataset, we first need to pretrain an EDAC agent with a set of Q p functions and a policy function π p using clean dataset. Then, we randomly sample c % transitions ( s, a, r, s ′ ) from the offline dataset, and for each of these selected states s , we attack it to form ˆ s = min ˆ s ∈ B d ( s,ϵ ) Q p (ˆ s, a ) . Here, B d ( s, ϵ ) = { ˆ s || ˆ s -s | ≤ ϵ · std ( s ) } regularizes the maximum difference for each state dimension.

Adversarial action corruption. We randomly sample c % transitions ( s, a, r, s ′ ) from the offline dataset, and for each of these selected actions a , we use the pretrained EDAC agent to attack it to form ˆ a = min ˆ a ∈ B d ( a,ϵ ) Q p ( s, ˆ a ) . Here, B d ( a, ϵ ) = { ˆ a || ˆ a -a | ≤ ϵ · std ( a ) } regularizes the maximum difference for each action dimension.

Adversarial reward corruption. We randomly sample c % transitions ( s, a, r, s ′ ) from the offline dataset, and for each of these selected rewards r , we directly attack it to form ˆ r = -ϵ × r without any adversarial model. This is because that the objective of adversarial reward corruption is ˆ r = min ˆ r ∈ B ( r,ϵ ) ˆ r + γ E [ Q ( s ′ , a ′ )] . Here B ( r, ϵ ) = { ˆ r | | ˆ r -r | ≤ (1+ ϵ ) · r max } regularizes the maximum distance for rewards. Therefore, we have ˆ r = -ϵ × r .

Adversarial dynamics corruption. We randomly sample c % transitions ( s, a, r, s ′ ) from the offline dataset, and for each of these selected next-step states s ′ , we use the pretrained EDAC agent to attack it to form ˆ s ′ = min ˆ s ′ ∈ B d ( s ′ ,ϵ ) Q p ( ˆ s ′ , π p ( s ′ )) . Here, B d ( s ′ , ϵ ) = { ˆ s ′ || ˆ s ′ -s ′ | ≤ ϵ · std ( s ′ ) } regularizes the maximum difference for each dimension of the dynamics.

The optimization of the above adversarial corruptions are implemented through Projected Gradient Descent [53, 54]. Taking adversarial observation corruption for example, we first initialize a learnable vector z ∈ [ -ϵ, ϵ ] d s , and then conduct a 100-step gradient descent with a step size of 0.01 for ˆ s = s + z · std ( s ) , and clip each dimension of z within the range [ -ϵ, ϵ ] after each update. For action and dynamics corruption, we conduct similar operation.

Building upon the corruption settings for individual elements as previously discussed, we further intensify the corruption to simulate the challenging conditions that might be encountered in real-world scenarios. We present the details of the simultaneous corruption below:

Random simultaneous corruptions. We sequentially conduct random observation corruption, random action corruption, random reward corruption, and random dynamics corruption to the offline dataset in order. That is to say, we randomly select c % of the transitions each time and corrupt one element among them, repeating four times until states, actions, rewards, and dynamics of the offline dataset are all corrupted.

Adversarial simultaneous corruptions. We sequentially conduct adversarial observation corruption, adversarial action corruption, adversarial reward corruption, and adversarial dynamics corruption to the offline dataset in order. That is to say, we randomly select c % of the transitions each time and attack one element among them, repeating four times until states, actions, rewards, and dynamics of the offline dataset are all corrupted.

C.2 Details for CARLA

We conduct experiments in CARLA from D4RL benchmark. We use the clean environment 'CARLALane-v0' to evaluate IQL, RIQL, and TRACER (Ours). We report each mean result with the standard deviation in the left of Figure 2 over four random seeds for 3000 epochs. We apply the random simultaneous data corruptions, where each element in the offline dataset (including states, actions, rewards, and next states) may involve random noise. We follow the setting in Section 4.2, using the corruption rate c = 0 . 3 and scale ϵ = 1 . 0 in the CARLA task. The results in the left of Figure 2 show the superiority of TRACER in the random simultaneous corruptions. We provide the hyperparameters of TRACER in Table 5.

C.3 Additional Experiments and Analysis

C.3.1 Results Comparison between TRACER and RIQL under Individual Corruptions

In Tables 2 and 3 of the main text, we adhered to commonly used settings for individual corruptions in corruption-robust offline RL. We directly followed hyperparameters from RIQL (i.e., κ for huber loss, the ensemble number K , and α in action-value functions, see Tables 5 and 6). Results show that TRACER outperforms RIQL in 18 out of 24 settings, demonstrating its robustness even when aligned with RIQL's hyperparameters.

Further, we explore hyperparameter tuning, specifically of κ , on Hopper task to improve TRACER's performance. This results in TRACER outperforming RIQL in 7 out of 8 settings on Hopper, up





























































































































































from 5 settings (see Tables 7 and 8). The further improvement highlights TRACER's potential to achieve greater performance gains.

Based on Table 7, we find that TRACER requires low κ in Huber loss, using L1 loss for large errors. Thus, TRACER can linearly penalize corrupted data and reduce its influence on the overall model fit.

C.3.2 Additional Experiments

We further conduct experiments on two AntMaze datasets and two additional Mujoco datasets, presenting results under random simultaneous corruptions in Table 9.

Each result represents the mean and standard error over four random seeds and 100 episodes in clean environments. For each experiment, the methods train agents using batch sizes of 64 for 3000 epochs. Building upon RIQL, we apply the experiment settings as follows. (1) For the two Mujoco datasets, we use a corruption rate of c = 0 . 3 and scale of ϵ = 1 . 0 . Note that simultaneous corruptions with c = 0 . 3 implies that approximately 76 . 0% of the data is corrupted. (2) For the two AntMaze datasets, we use the corruption rate of 0.2, corruption scales for observation (0.3), action (1.0), reward (30.0), and dynamics (0.3).

The results in Table 9 show that TRACER significantly outperforms other methods in all these tasks with the aforementioned AntMaze and Mujoco datasets.

C.4 Details for Evaluation and Ablation Studies

C.4.1 Evaluation for Robustness of TRACER across Different Scales of Corrupted Data

Theorem A.4 shows that the higher the scale of corrupted data, the greater the difference in actionvalue distributions and the lower the TRACER's performance. Thus, we further conduct experiments to evaluate TRACER across various corruption levels. Specifically, we apply different corruption rate c in all four elements of the offline dataset. We randomly select c % of transitions from the dataset and corrupt one element in each selected transition. Then, we repeat this step four times until all elements are corrupted. Therefore, approximately 100 · (1 -(1 -c ) 4 )% of data in the offline dataset is corrupted.

In Table 10, we evaluate TRACER using different c % , including 10% , 20% , 30% , 40% , and 50% . These rates correspond to approximately 34 . 4% , 59 . 0% , 76 . 0% , 87 . 0% , and 93 . 8% of the data being corrupted. The results in Table 10 demonstrate that while TRACER is robust to simultaneous corruptions, its scores depend on the extent of corrupted data it encounters.

C.4.2 Details for Evaluation of the Entropy-based Uncertainty Measure

In Figure 5, we additionally report the entropy values of TRACER with and without using entropy-based uncertainty measure, corresponding to Figure 3. The curves illustrate that TRACER using entropy-based uncertainty measure can effectively regulate the loss associated with corrupted data, reducing the influence of corrupted samples. Therefore, the estimated entropy values of corrupted data can be higher than those of clean data.

C.4.3 Ablation Study for Bayesian Inference

We conduct experiments of TRACER and TRACER without Bayesian inference, i.e., RIQL combined with distributional RL methods, namely DRIQL. The experiments are on 'Walker2d-medium-replay-v2' under random simultaneous data corruptions. We employ the same hyperparameters of Huber loss for these methods, using α = 0 . 25 and κ = 1 . 0 .























We report each mean result with the standard deviation in Figure 6. Each result is averaged over four seeds for 3000 epochs. These results show the effectiveness of our proposed Bayesian inference.

D Compute Resource

In this subsection, we provide the computational cost of our approach TRACER.

For the MuJoCo tasks, including Halfcheetah, Walker2d, and Hopper, the average training duration is 40.6 hours. For the CARLA task, training extends to 51 hours. We conduct all experiments on NVIDIA GeForce RTX 3090 GPUs.

To compare the computational cost, we report the average epoch time on Hopper in Table 11, where results of baselines (including DT [82]) are from [18]. The formula for computational cost is:

Note that TRACER requires a long epoch time due to two main reasons:

Unlike RIQL and IQL, which learn one-dimensional action-value functions, TRACER generates multiple samples for the estimation of action-value distributions. Following [43], we generate 32 samples of action values for each state-action pair.

TRACER uses states, actions, and rewards as observations to update models, estimating the posterior of action-value distributions.

E More Related Work

Online RL. In general, standard online RL fall into two categories: model-free RL [83-85] and model-based RL [86, 71]. In recent years, RL has achieved great success in many important realworld decision-making tasks [87-93]. However, the online RL methods still typically rely on active data collection to succeed, hindering their application in scenarios where real-time data collection is expensive, risky, and/or impractical. Thus, we focus on the offline RL paradigm in this paper.

F Code

We implement our codes in Python version 3.8 and make the code available online 2 .

G Limitations and Negative Societal Impacts

TRACER's performance is related to the extent of corrupted data within the offline dataset. Although TRACER consistently outperforms RIQL even under conditions of extensive corrupted data (see Table 10), its performance does degrade as the corruption rate (i.e., the extent/scale of corrupted data) increases. To tackle this problem, we look forward to the continued development and optimization of corruption-robust offline RL with large language models, introducing the prior knowledge of clean data against large-scale or near-total corrupted data. Thus, this corruption-robust offline RL can perform well even when faced with large-scale or near-total corrupted data in the offline dataset.

This paper proposes a novel approach called TRACER to advance the field of robustness in offline RL for diverse data corruptions, enhancing the potential of agents in real-world applications. Although our primary focus is on technical innovation, we recognize the potential societal consequences of our work, as robustness in offline RL for data corruptions has the potential to influence various domains such as autonomous driving and the large language model field. We are committed to ethical research practices and attach great importance to the social implications and ethical considerations in the development of robustness research in offline RL.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We provide our main claims in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We provide the limitations of this work in Appendix G and the second paragraph of Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We present all assumptions and proofs in Appendix A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide detailed information of the hyperparameters and implementation of our approach (see Appendix B.2) to facilitate the reproduction of our main results. We also provide the detailed experiment settings in Section 4.1 and Appendix C.1.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide the data and code in Appendix F, which can also be found in the supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide detailed experiment settings in Section 4.1, Appendices C.1 and B.2. Moreover, we provide details in Appendix C.4 for each result in Section 4.4.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report standard deviations and standard errors for all results in the main text, including results in Tables 1, 2, 3, Figures 2, and 3.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the detailed information on the computer resources in Appendix D Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: NA.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: All our experiment data is open source.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: NA.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: NA.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: NA.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce TRACER, a new Bayesian methodology to capture the uncertainty via offline data for robustness against all types of data corruptions. An appealing feature of TRACER is that it can distinguish corrupted data from clean data using an entropy-based uncertainty measure. Experiments are provided to prove the effectiveness of such a methodology.\n\nSTRENGTHS:\nVery clearly stating the problem, and convincing the reader that's a relevant one. Compelling method to address such a problem.\n\nWEAKNESSES:\nThe authors missed some important references: [1,2] introduce credal Bayesian deep learning, which is a robust way of performing Bayesian inference via Bayesian Neural Networks (where posterior and predictive distributions are approximated via variational inference) taking into account different types of uncertainty (using so-called credal sets, i.e. closed and convex sets of prior and likelihood probabilities), and quantifying them using entropy-based uncertainty measures. In the future, the authors may also look into [3], which seems like a compelling work for the research venue explored in the paper.\n\n[1] https://arxiv.org/abs/2302.09656\n\n[2] https://link.springer.com/chapter/10.1007/978-3-031-57963-9_1#:~:text=In%20their%20seminal%201990%20paper,bound%20to%20hold%20with%20equality\n\n[3] https://arxiv.org/abs/2308.14815\n\nQUESTIONS:\nI'd like to politely ask the authors to expand the related work with the above references [1,2]. In addition, would it be fair to say that, on page 3, the expectation that finds $\\pi^\\star$ is an integral taken with respect to the joint probability given by all the sources of randomness $P_0, \\pi(\\cdot \\mid s_t), \\rho(\\cdot \\mid s_t,a_t), P(\\cdot \\mid s_t,a_t)$? \n\nIn equation (1), the closed parenthesis ) should go after $a_0=a$.\n\nLIMITATIONS:\nThe limitations are explicitly acknowledged by the authors in the conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel approach called TRACER, aimed at addressing the challenge of learning robust policies from offline datasets that are subject to various data corruptions. The key contribution of this work lies in the integration of Bayesian inference to capture uncertainty within the offline data. TRACER models data corruptions as uncertainty in the action-value function and approximates the posterior distribution of this function. Furthermore, it employs an entropy-based uncertainty measure to distinguish between corrupted and clean data, thereby regulating the loss associated with corrupted data to enhance robustness in clean environments. Experimental results indicate that TRACER outperforms SOTA methods in handling both individual and simultaneous data corruptions.\n\nSTRENGTHS:\n- The studied problem is important, and the paper offers a reasonable solution.\n- Theoretical guarantees are provided.\n- The empirical results show considerable improvement over prior work.\n\nWEAKNESSES:\n- Though I like the insight from Variational Bayesian inference, my major concern is that the method is quite complicated, incorporating numerous components and designs such as the distributional value function, variational inference, and entropy-weighted loss function. This complexity makes this method less preferable in practice.\n\n- Moreover, the results in the main tables (Table 2 and Table 3) appear similar to those of RIQL, raising concerns about the necessity of the included components.\n\n- Another issue is the readability and clarity of the writing. For example, Equations 10 and 11 lack necessary explanation and insight. Additionally, the implementation details for $\\phi_a, \\phi_b, \\phi_c$ are missing, leaving readers unsure of how the method works in practice. Furthermore, Equation 10 seems to include $\\theta$ as an input unnecessarily, as the right side does not depend on $\\theta$.\n\n- The analysis of the experiments is not thorough. For instance, the accuracy of the measurement for corrupted data decreases during training in Figure 3, which is not discussed and seems unreasonable.\n\nQUESTIONS:\n- The proposed method is complicated. Can this method be simplified?\n\n- The results in the main tables (Table 2 and Table 3) appear similar to those of RIQL, raising concerns about the necessity of the included components.\n\n- Another issue is the readability and clarity of the writing discussed above.\n\n- Why does the accuracy of the measurement for corrupted data decreases during training in Figure 3?\n\n- Moreover, I suggest the authors provide the computational cost comparison with prior work.\n\nLIMITATIONS:\nThis paper acknowledges its limitations; however, I identified an additional concern regarding the complexity of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces TRACER, a robust offline reinforcement learning (RL) algorithm designed to address the challenges posed by data corruptions in offline datasets. The corruptions can be\nrealized in form of states, actions, rewards, and dynamics corruption. The proposed methodology uses Bayesian inference to model and mitigate the impact of corrupted data on the action-value function.\nBy leveraging an entropy-based uncertainty measure, TRACER can distinguish between corrupted and clean data, thereby reducing the influence of the former and enhancing the robustness of the learned policies.\nThe paper provides thorough evaluation on MujoCo and Carla offline RL datasets.\n\nSTRENGTHS:\n1. The paper proposes an interesting method of down weighting the contribution corrupted data while learning action value function from the offline dataset using entropy as an uncertainty measure.\n2. The paper uses a Bayesian approach to learn the action value function from the data which aids learning from uncorrupted elements in the dataset.\n3. The authors provide extensive experimental validation across multiple tasks and corruption types, showing mostly consistent performance improvements over other methods. The paper also \nreports ablation studies of using entropy showing potential improvement from the proposed methodology.\n\nWEAKNESSES:\n1. It is not clear how the final policy is extracted after learning $D_{\\theta}$.\n2. The paper relies on many hyperparameters. What kind of kernel is used in the gaussian distribution learning in Eq 10? Is there a upper bound on what amount of corruption the method will be able to handle? \n3. While the authors consider individual corruption of one element at a time data collected from in a real world system can have simultaneous corruptions. For\nexample corrupted action due to adversarial noise will lead the system to different state causing simultaneous corruptions.\n4. Notational clarity: The paper introduces a lot of notations which is difficult to keep track of. Specially in the following contexts :\na :  The paper introduces LQ (θi), LV(ψ), Lπ(ϕ) in Eq 4, 5 and 6 but does not discuss how they relate to Dθ. Is the θ used for learning Q function and D same?\nb : The paper introduces a new notation qπ in Eq 18 called value distribution. Is this the same as action value distribution? RIQL already proves the robustness of IQL policy with respect of data corruption. What is \nthe objective of Theorem A.3? It will be helpful to add a discussion on the theoretical analysis. Also Providing a table summarizing the notations in appendix will enhance readability of the text.\n\nQUESTIONS:\n1) How relevant is the reward corruption given offline RL already is robust to noise in the reward function as it mimics the actions for datasets with length bias? Please refer to ref 1\n2) The authors aim to leverage all elements in the dataset as observations. Do the authors assume that complete traces of the system are available for which the offline policy is being learned? \nIn situations of dynamic corruption these traces might be different than the system. For example s_t in the corrupted dataset may not be observed in the real system. How are such cases \nhandled?\n3) As the method learns action-value function D from the data while reducing the influence of corrupted data. What amount of clean data needs to be present in the dataset for this method \nto work?\n4) What is the reason for not using projected gradient descent for the reward corruption? Also could the author provide some understanding of why the method outperforms RIQL under certain settings and why it does not in some cases?\n5) Please also respond to the weaknesses.\n\n1. Li, Anqi, et al. \"Survival instinct in offline reinforcement learning.\" Advances in neural information processing systems 36 (2024).\n\nLIMITATIONS:\nLimitations: The authors discuss the limitation in terms of handling realistic noises in conclusion. However, a limitation is also on the dependence on clean observations present in the dataset when some elements are corrupted.\nThe authors do not discuss any negative sociatal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

comments: SUMMARY:
This paper seeks to conduct reinforcement learning from corrupted offline data. More specifically, they propose the TRACER algorithm, which uses bayesian inference to calculate the uncertainty in estimating the action-value function. The authors conduct experiments with diverse corruptions on CARLA and Mujoco environments.

STRENGTHS:
- This paper considers an important problem.
- Their proposed algorithm is interesting and performs well.

WEAKNESSES:
- The contribution compared to other baselines (RIQL) is not really clear. In the introduction the authors claim that their method can handle simultaneous corruptions and RIQL cannot. However no experiments are done in this setting.
- The experimental settings are fairly limited: only three Mujoco environments are considered and one carla experiment.
- Different perturbations levels are not shown.
- The authors do not discuss the computational cost of their method.

QUESTIONS:
- Can you clarify the contribution of TRACER compared to RIQL? It seems like it is an orthogonal approach, but the two algorithms are not really compared in detail.
- Can you show the hyperparameter tuning results? Is TRACER stable under different hyperparameter settings? How does it compare to RIQL in this regard?
- How does the computational cost of TRACER compare to the other baselines?

LIMITATIONS:
The authors do discuss their method's limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11