Title: Corruption-Robust Offline Reinforcement Learning with General Function Approximation
Venue: NeurIPS 2023
Accepted: yes
Keywords: offline RL; adversarial corruption; general function approximation
Abstract: We investigate the problem of corruption robustness in offline reinforcement learning (RL) with general function approximation, where an adversary can corrupt each sample in the offline dataset, and the corruption level $\zeta\geq0$ quantifies the cumulative corruption amount over $n$ episodes and $H$ steps. Our goal is to find a policy that is robust to such corruption and minimizes the suboptimality gap with respect to the optimal policy for the uncorrupted Markov decision processes (MDPs). Drawing inspiration from the uncertainty-weighting technique from the robust online RL setting \citep{he2022nearly,ye2022corruptionrobust}, we design a new uncertainty weight iteration procedure to efficiently compute on batched samples and propose a corruption-robust algorithm for offline RL. Notably, under the assumption of single policy coverage and the knowledge of $\zeta$, our proposed algorithm achieves a suboptimality bound that is worsened by an additive factor of $\mathcal O(\zeta \cdot (\text CC(\lambda,\hat{\mathcal F},\mathcal Z_n^H))^{1/2} (C(\hat{\mathcal F},\mu))^{-1/2} n^{-1})$ due to the corruption. Here $\text CC(\lambda,\hat{\mathcal F},\mathcal Z_n^H)$ is the coverage coefficient that depends on the regularization parameter $\lambda$, the confidence set $\hat{\mathcal F}$, and the dataset $\mathcal Z_n^H$, and $C(\hat{\mathcal F},\mu)$ is a coefficient that depends on $\hat{\mathcal F}$ and the underlying data distribution $\mu$. When specialized to linear MDPs, the corruption-dependent error term reduces to $\mathcal O(\zeta d n^{-1})$ with $d$ being the dimension of the feature map, which matches the existing lower bound for corrupted linear MDPs. This suggests that our analysis is tight in terms of the corruption-dependent term.

Corruption-Robust Offline Reinforcement Learning with General Function Approximation

Abstract

We investigate the problem of corruption robustness in offline reinforcement learning (RL) with general function approximation, where an adversary can corrupt each sample in the offline dataset, and the corruption level ζ ≥ 0 quantifies the cumulative corruption amount over n episodes and H steps. Our goal is to find a policy that is robust to such corruption and minimizes the suboptimality gap with respect to the optimal policy for the uncorrupted Markov decision processes (MDPs). Drawing inspiration from the uncertainty-weighting technique from the robust online RL setting [18, 55], we design a new uncertainty weight iteration procedure to efficiently compute on batched samples and propose a corruption-robust algorithm for offline RL. Notably, under the assumption of single policy coverage and the knowledge of ζ , our proposed algorithm achieves a suboptimality bound that is worsened by an additive factor of O ( ζ · (CC( λ , ̂ F , Z H n )) 1 / 2 ( C ( ̂ F , µ )) -1 / 2 n -1 ) due to the corruption. Here CC( λ , ̂ F , Z H n ) is the coverage coefficient that depends on the regularization parameter λ , the confidence set ̂ F , and the dataset Z H n , and C ( ̂ F , µ ) is a coefficient that depends on ̂ F and the underlying data distribution µ . When specialized to linear MDPs, the corruption-dependent error term reduces to O ( ζ dn -1 ) with d being the dimension of the feature map, which matches the existing lower bound for corrupted linear MDPs. This suggests that our analysis is tight in terms of the corruption-dependent term.

1 Introduction

Offline reinforcement learning (RL) has received tremendous attention recently because it can tackle the limitations of online RL in real-world applications, e.g., healthcare [45] and autonomous driving [35], where collecting online data is risky, expensive and even infeasible. By leveraging a batch of pre-collected datasets, offline RL aims to find the optimal policy that is covered by the dataset without further interaction with the environment. Due to the restriction of the offline dataset, the utilization of pessimism in the face of uncertainty is widespread [1, 2, 53, 40, 16] and plays a central role in

Quanquan Gu

providing theoretical guarantees for efficient learning [24, 36, 44, 48, 51, 56, 57, 63, 52]. Notably, these theoretical works demonstrate that a single-policy coverage is sufficient to guarantee sample efficiency.

In this paper, we study offline RL under adversarial corruption and with general function approximation. Adversarial corruption refers to adversarial attacks on the reward functions and transition dynamics on the data at each step before the learner can access the dataset. The learner only knows the cumulative corruption level and cannot tell whether the corruption occurs at each data point. Our corruption formulation subsumes the model misspecification [22] and the fixed fraction of data contamination [60] as special cases. Various real-world problems are under the threat of adversarial corruption, such as chatbots misled by discriminative or unethical conversations [32, 61], and autonomous vehicles tricked by hacked navigation instructions or deliberately contaminated traffic signs [12]. On the other hand, general function approximation (approximating the value function with a nonlinear function class, such as deep neural networks) plays a pivotal role in modern large-scale RL problems, such as large language model [10], robotics [26] and medical treatment [28]. Recent works have established different frameworks to explore the minimal structure condition for the function class that enables sample efficiency [19, 47, 9, 21, 14, 6, 64]. In particular, Wang et al. [47] leverage the concept of eluder dimension [39] and construct the least squares value iteration (LSVI) framework, which establishes optimism at each step for online RL.

For adversarial corruption and general function approximation, a significant amount of research has focused on the online setting. However, offline RL in this setting is still understudied due to restricted coverage conditions and lack of adaptivity. One notable exception is Zhang et al. [60], which assumes /epsilon1 -fraction of the offline dataset is corrupted, and their algorithm suffers from a suboptimal bound on the corruption term. Our work moves a step further and achieves corruption robustness under the LSVI framework [47] in offline RL with general function approximation by generalizing the uncertainty weighting technique [18, 55]. We also propose an algorithm robust to an additional known distribution shift. Due to space limit, we defer it to Appendix C. We summarize our contributions as follows:

We formally define the corruption level in offline RL. With knowledge of the corruption level, we design an algorithm that draws inspiration from the PEVI algorithm [24] and the uncertaintyweighting technique. The uncertainty for each data point, serving as the bonus function, is quantified by its informativeness with respect to the whole dataset. We propose the uncertainty weight iteration algorithm to calculate the weights efficiently and prove that the output, an approximation of the uncertainty, is sufficient to control the corruption term.

Theoretically, our proposed algorithm enjoys a suboptimality bound of ˜ O ( H · CC 1 / 4 (ln N ) 1 / 2 ( C ( ̂ F , µ )) -1 / 4 n -1 / 2 + ζ · CC 1 / 2 ( C ( ̂ F , µ )) -1 / 2 n -1 ) , where H is the episode length, n is the number of episodes, N is the covering number, CC is the coverage coefficient, and C ( ̂ F , µ ) is the coefficient depicting how well the underlying data distribution µ explores the feature space, and ̂ F is the confidence set. The corruption-dependent term reduces to O ( ζ dn -1 ) in the linear model of dimension d , thus matching the lower bound for corrupted linear MDPs. It is worth highlighting that our novel analysis enables us to eliminate the uncertainty-related weights from the coverage condition.

Motivated by our theoretical findings, we present a practical offline RL algorithm with uncertainty weighting and demonstrate its efficacy under diverse data corruption scenarios. Our practical implementation achieves a 104% improvement over the previous state-of-the-art uncertainty-based offline RL algorithm under data corruption, demonstrating its potential for effective deployment in real-world applications.

1.1 Related Work

Corruption-Robust Bandits and RL. There is an emerging body of theoretical literature on bandits and online RL with corruption. The adversarial corruption is first formulated in the multi-armed bandit problem by Lykouris et al. [30], where an adversary corrupts the reward in each round t by ζ t and the corruption level is measured by ζ = ∑ T t =1 | ζ t | . Then, a lower bound with a linear dependence on ζ is constructed by Gupta et al. [17], indicating that the ideal regret bound should achieve a 'parallel' relationship: Regret( T ) = o ( T ) + O ( ζ ) , and the corruption-independent term approximates the non-corrupted bound. When extending to linear contextual bandits, a line of

work [4, 8, 13, 27, 62, 25] propose various methods but either derive sub-optimal regrets or require particular assumptions. The gap is later closed by He et al. [18], which achieves the minimax lower bound using a novel sample-dependent weighting technique. Specifically, the weight for each sample is adaptive to its confidence, which is also called uncertainty. Beyond bandits, earlier works on MDPs [5, 20, 23, 29, 33, 37, 38] consider the setting where only the rewards are corrupted, and the transitions remain intact. Wu et al. [50] begin to handle corruption on both rewards and transitions for tabular MDPs. Wei et al. [48] establish a unified framework for RL with unknown corruption under a weak adversary, where the corruption happens before the decision is made in each round. Later, Ye et al. [55] extend the weighting technique [18] to corrupted RL with general function approximation and achieve a linear dependence on the cumulative corruption level ζ . Particularly, Wei et al. [48], Ye et al. [55] both impose corruption on the Bellman operator, which is the same as the corruption model considered in this paper.

Offline RL Against Attacks. The emergence of poisoning attacks in real-world scenarios poses new challenges for offline RL and necessitates improved defenses [49]. There are generally two types of attacks [3], namely test-time attacks and training-time attacks. In test-time attacks, the training data is clean, and the learned policy must contend with an attacker during test time. For example, Yang et al. [53] propose learning conservative and smooth policies robust to different test-time attacks. In contrast, our paper focuses on the training-time attack as another line of work [31, 49, 60], where part of the training data is corrupted maliciously. Wu et al. [49] propose two certification criteria and a new aggregation-based method to improve the learned policy from corrupted data. To the best of our knowledge, [60] is the only theoretical work on corrupted offline RL, which considers that an /epsilon1 -fraction ( /epsilon1 = ζ /nH ) of samples are corrupted on both rewards and transitions for linear MDPs and achieves an O ( ζ 1 / 2 dn -1 / 2 ) suboptimality bound. Notably, distinct from the setting in Zhang et al. [60] that clean data is first collected and then corrupted by an adversary, we consider the setting that data collection and corruption occur at the same time (thus corruption at one step affects the subsequent trajectory). Therefore, our setting is different from that of Zhang et al. [60].

2 Preliminaries

In this section, we formulate the episodic Markov decision process (MDP) with adversarial corruption and under general (nonlinear) function approximation. Before the formal introduction, we introduce some notations to facilitate our presentation.

Notations. Let [ n ] denote the set { 1 , . . . , n } . For spaces X and A and a function f : X × A → R , let f ( x ) = max a ∈ A f ( x, a ) . Given a semi-definite matrix M and a vector v , we define ‖ v ‖ M = √ v /latticetop Mv . For two positive sequences { f ( n ) } ∞ n =1 , { g ( n ) } ∞ n =1 , let f ( n ) = O ( g ( n )) if there exists a constant C > 0 such that f ( n ) ≤ Cg ( n ) for all n ≥ 1 , and f ( n ) = Ω ( g ( n )) if there exists a constant C > 0 such that f ( n ) ≥ Cg ( n ) for all n ≥ 1 . We use ˜ O ( · ) to omit polylogarithmic factors. Sometimes we use the shorthand notation z = ( x, a ) .

2.1 Episodic MDPs

We consider an episodic MDP ( X , A , H, P , r ) with the state space X , action space A , episode length H , transition kernel P = { P h } h ∈ [ H ] , and reward function r = { r h } h ∈ [ H ] . Suppose that the rewards are bounded: r h ≥ 0 for any h ∈ [ H ] , and ∑ H h =1 r h ( x h , a h ) ≤ 1 almost surely. Given any policy π = { π h : X → A} h ∈ [ H ] , we define the Q-value and V-value functions starting from step h as

where the expectation E π is taken with respect to the trajectory under the policy π . There exists an optimal policy π ∗ and optimal value functions V h ∗ ( x ) := V h π ∗ ( x ) = sup π V h π ( x ) and Q h ∗ ( x, a ) := Q h π ∗ ( x, a ) = sup π Q h π ( x, a ) that satisfy the Bellman optimality equation:

where T h is called the Bellman operator. Then we define the Bellman residual as

2.2 General Function Approximation

We approximate the Q-value functions by a function class F = F 1 × · · · × F H where F h : X × A → [0 , 1] for h ∈ [ H ] , and f H +1 ≡ 0 since no reward is generated at step H +1 . Generally, the following assumption is common for the approximation function class.

Assumption 2.1 (Realizability and Completeness) . For all h ∈ [ H ] , Q h ∗ ∈ F h . Additionally, for all g h +1 ( x h +1 ) ∈ [0 , 1] , ( T h g h +1 )( x h , a h ) ∈ F h .

The realizability assumption [21] ensures the possibility of learning the true Q-value function by considering the function class F . The Bellman completeness (adopted from Wang et al. [47] and Assumption 18.22 of Zhang [59]) is stronger than that in Jin et al. [21]. The former applies the least squares value iteration (LSVI) algorithm that establishes optimism at each step, while the latter proposes the GOLF algorithm that only establishes optimism at the first step. We use the standard covering number to depict the scale of the function class F .

Definition 2.1 ( /epsilon1 -Covering Number) . The /epsilon1 -covering number N ( /epsilon1 , F , ρ ) of a set F under metric ρ is the smallest cardinality of a subset F 0 ⊆ F such that for any f ∈ F , there exists a g ∈ F 0 satisfying that ρ ( f, g ) ≤ /epsilon1 . We say F 0 is an ( /epsilon1 , ρ ) cover of F .

2.3 Offline Data Collection Process

Offline Clean Data. Consider an offline clean dataset with n trajectories D = { ( x h i , a h i , r h i ) } n,H i,h =1 . We assume the dataset D is compliant with an MDP ( X , A , H, P , r ) with the value functions Q,V and the Bellman operator T : for any policy π ,

∣ where the realizability and completeness in Assumption 2.1 hold. The compliance assumption (4) is also made in Jin et al. [20], Zhong et al. [63], which means that D remains the Markov property and allows D to be collected by an adaptive behavior policy. The induced distribution of the state-action pair is denoted by µ = { µ h } h ∈ [ H ] .

Adversarial Corruption. During the offline dataset collection process, after observing the stateaction pair ( x h , a h ) chosen by the data collector, an adversary corrupts r h and x h +1 at each step h before they are revealed to the collector. For the corrupted dataset D , we define the corrupted value function Q D , V D , and the Bellman operator T D satisfying (4). To measure the corruption level, we notice that characterizing the specific modification on each tuple ( s, a, s ′ , r ) is hard and unnecessary since once one modifies a tuple, the subsequent trajectory changes. Therefore, it is difficult to tell whether the change is caused by the corruption at the current step or a previous step. In fact, we only care about the part of the change that violates the Bellman completeness. Therefore, following the online setting [55, 48], we measure the corruption level by the gap between T D and T as follows.

Definition 2.2 (Cumulative Corruption) . The cumulative corruption is ζ if at any step h ∈ [ H ] , for a sequence { ( x h i , a h i ) } n,H i,h =1 ⊂ X × A chosen by the data collector and a sequence of functions { g h : X → [0 , 1] } H h =1 , we have for all h ∈ [ H ] ,

The learning objective is to find a policy π that minimizes the suboptimality of π given any initial state x 1 = x : SubOpt( π , x ) = V 1 ∗ ( x ) -V 1 π ( x ) , where V ( · ) is the value function induced by the uncorrupted MDP.

3 Algorithm

In this section, we first highlight the pivotal role that uncertainty weighting plays in controlling the corruption-related bound. To extend the uncertainty weighting technique to the offline setting, we propose an iteration algorithm. With the proposed algorithm, the theoretical result for the suboptimality is presented.

3.1 Uncertainty-Related Weights

In this subsection, we discuss the choice of weight for a simplified model without state transition ( H = 1 ) and use the notation z i = ( x i , a i ) . Given a dataset { ( z i , y i ) } i ∈ [ n ] , we have y i = ¯ f ( z i ) + ζ i + /epsilon1 i for i ∈ [ n ] , where ¯ f ∈ F is the uncorrupted true value, the noise /epsilon1 i is zero-mean and conditional η -subGaussian, and the corruption level is ζ = ∑ n i =1 | ζ i | .

We begin with delineating the consequence caused by the adversarial corruption for the traditional least-square regression: ˆ f = min f ∈ F ∑ n i =1 ( f ( z i ) -y i ) 2 . Some calculations lead to the following decomposition:

The term I 1 ≤ 0 since ˆ f is the solution to the least-square regression. The term I 2 is bounded by ˜ O (ln N ) because of the η -subGaussainity of /epsilon1 i , where N is the covering number of F . The term I 3 is ruined by corruption: I 3 ≤ 2 ∑ n i =1 | ζ i | = O ( ζ ) . Hence, the confidence radius ( ∑ n i =1 ( ˆ f ( z i ) -¯ f ( z i )) 2 ) 1 / 2 = ˜ O ( √ ζ +ln N ) will explode whenever the corruption level ζ grows with n .

To control the corruption term, motivated by the uncertainty-weighting technique from online settings [55, 18, 65], we apply the weighted regression: ˆ f = min f ∈ F ∑ n i =1 ( f ( z i ) -y i ) 2 / σ 2 i , where ideally, we desire the following uncertainty-related weights:

where α , λ > 0 are pre-determined parameters. The uncertainty quantity in the above equation is the supremum of the ratio between the prediction error | f ( z i ) -f ′ ( z i ) | and the training error √ ∑ n j =1 ( f ( z j ) -f ′ ( z j )) 2 / σ 2 j over f, f ′ ∈ F . Intuitively, the quantity depicts the relative information of a sample z i against the whole training set { z 1 , . . . , z n } . We can use the linear function class as a special example to explain it. When the function space F h is embedded into a d -dimensional vector space: F h = { 〈 w ( f ) , φ ( · ) 〉 : z → R} , the uncertainty quantity becomes

where Λ = ∑ n j =1 φ ( z j ) φ /latticetop ( z j ) / σ 2 j . Moreover, ( φ /latticetop ( z i ) Λ -1 φ ( z i ) ) -1 represents the effective number of samples in the { z i } n i =1 along the φ ( z i ) 's direction. We discuss in Lemma B.3 that under mild conditions the linear and nonlinear uncertainty quantities are almost equivalent.

However, since the uncertainty also depends on weights, it is impossible to determine all the weights { σ i } i ∈ [ n ] simultaneously. Compared with the online setting where the weight in each round can be determined sequentially (iteratively in rounds), we face two challenges in the offline setting: (a) how to compute uncertainty-related weights iteratively? (b) will an approximate solution to the uncertainty play an equivalent role in controlling the corruption term?

To solve the first challenge, we propose the weight iteration algorithm in Algorithm 1. Moreover, we demonstrate the convergence of this algorithm by the monotone convergence theorem in the following lemma, which ensures that the output weights are sufficiently close to desired ones (5). The proof is provided in Appendix B.1.

Lemma 3.1. There exists a T such that the output of Algorithm 1 σ i := σ T +1 n

Algorithm 1 Uncertainty Weight Iteration

1:

2: Initialization: t = 0 , σ 0 i = 1 , i = 1 , . . . , n

Input: { ( x i , a i ) } n i =1 , F , α > 0

3: repeat

4: t t

6: until max i ∈ [ n ] ( σ t i / σ t -1 i ) 2 ≤ 2 7: Output: { σ t i } n i =1

For the second challenge, the weighted version L n := ∑ n i =1 ( ˆ f ( z i ) -¯ f ( z i )) 2 / σ 2 i can also be decomposed into three terms correspondingly. We can demonstrate that an approximate choice of weights satisfying (6) is sufficient to control the corruption term as

Since the corruption-unrelated terms (corresponding to I 1 , I 2 ) can still be bounded by ˜ O (ln N ) , we have L n = ˜ O (ln N +2 αζ √ L n ) , leading to an ˜ O ( αζ + √ ln N ) confidence radius. Therefore, with a sufficiently small α , the effect of corruption can be countered.

3.2 Corruption-Robust Algorithm

Algorithm 2 CR-PEVI

1: Input: D = { ( x h i , a h i , r h i ) } n,H i,h =1 , F

3: for step h = H,H -1 , . . . , 1 do

2: Initialization: Set f H +1 n ( · ) ← 0

4: Choosing weights { σ h i } n i =1 by proceeding Algorithm 1 with inputs { ( x h i , a h i ) } n i =1 , F h , α

5: Find the weighted least-squares solution in (7)

6: Find β h and construct confidence set

7: Construct bonus function as (8)

9: Set ˆ π h ( · ) = argmax a ∈ A f h n ( · , a )

8: Let f h n ( · , · ) = max ( 0 , ˆ f h ( · , · ) -β h b h ( · , · ) )

10: end for

11: Output: { ˆ π h } H h =1

Now, for the offline RL with general function approximation, we integrate the uncertainty weight iteration algorithm with the pessimistic value iteration (PEVI) algorithm [24], and propose a CorruptionRobust PEVI (CR-PEVI) in Algorithm 2. Our algorithm employs backward induction from step H to 1. Set estimated value function f H +1 n ( · ) = 0 . At each step h ∈ [ H ] , having obtained f h +1 n , we calculate f h n by solving the following weighted least-square regression:

where the weights are obtained via Algorithm 1. As opposed to online RL where the necessity of exploration stimulates optimistic estimation, the literature on offline RL [24, 63] is more inclined to pessimism due to the limitation of offline data coverage. Hence, we construct a confidence set ̂ F h = { f ∈ ̂ F h : λ + ∑ n i =1 ( f ( x h i , a h i ) -ˆ f h ( x h i , a h i )) 2 / ( σ h i ) 2 ≤ ( β h ) 2 } such that the uncorrupted Bellman operator T h converts the value function f h +1 n into the function class ̂ F h (i.e., T h f h +1 n ∈

̂ F h ) with high probability. For the bonus function, we follow [55] and choose it as which is seldom used in practical algorithms due to its unstability. Specifically, the covering number of the space containing (8) may be uncontrollable. According to Appendix E in [55], the issue of the covering number can be addressed under mild conditions by some techniques. Therefore, to maintain readability and consistency in this paper, we assume the corresponding bonus function space B h +1 of (8) has a bounded covering number. Then we introduce pessimism by subtracting b h from the estimated value function: f h n ( x, a ) = max(0 , ˆ f h ( x, a ) -β h b h ( x, a )) .

4 Theoretical Analysis

4.1 Coverage Condition

No guarantee for the suboptimality can be provided with insufficient data coverage. Based on pessimism, Jin et al. [20], Rashidinejad et al. [36] have demonstrated that the coverage over the optimal policy is sufficient for sample-efficient offline RL. The following condition covers the optimal policy under general function approximation.

Definition 4.1 (Coverage Coefficient) . Consider the offline dataset { x h i , a h i } n,H i,h =1 . For any initial state x 1 = x ∈ X , the coverage coefficient is:

In the face of corruption, we require single-policy coverage over the uncorrupted trajectory. This coefficient depicts the expected uncertainty of sample ( x h , a h ) induced by the optimal policy π ∗ compared to the n training samples. We use the linear MDP to interpret this condition, where the function space F h is embedded into a d -dimensional vector space: F h = { 〈 w ( f ) , φ ( · ) 〉 : z → R } , where z denotes the state-action pair ( x, a ) . With the notation Λ h = λ I + ∑ n i =1 φ ( z h i ) φ ( z h i ) /latticetop , we can demonstrate that if the sufficient 'coverage' in Jin et al. [24] holds: there exists a constant c † such that Λ h /followsequal I + c † n E π ∗ [ φ ( z h ) φ ( z h ) /latticetop | x 1 = x ] for all h ∈ [ H ] , our coverage coefficient is bounded: CC( λ , ̂ F , Z H n ) ≤ d/c † < ∞ . Therefore, the sufficient 'coverage' condition implies our coverage condition in the linear setting. We will discuss the connection formally in Lemma B.1.

4.2 Main Result

Before presenting the main theorem, we introduce a new general version of the well-explored dataset condition, which is the key to eliminating the uncertainty-related weights from the suboptimality bound and deriving the final result.

Assumption 4.1 (Well-Explored Dataset) . There exists a constant C ( ̂ F , µ ) > 8 n -1 such that for a ( n -1 , ‖ · ‖ ∞ ) cover of F h , denoted by ¯ F h , and for any two distinct f, f ′ ∈ ¯ F h ,

We interpret this condition with the linear model, where the condition (10) becomes: for any two distinct f, f ′ ∈ ¯ F h , min h ∈ [ H ] ( w ( f ) -w ( f ′ ) ) /latticetop ¯ Λ h ( w ( f ) -w ( f ′ ) ) ≥ C ( ̂ F , µ ) . As proved in Lemma B.2, with high probability, the above condition holds with C ( ̂ F , µ ) = Θ ( d -1 ) when the n trajectories of D are independent, and the data distribution µ h satisfies the minimum eigenvalue condition:

where ¯ c > 0 is an absolute constant. This is a widely-adopted assumption in the literature [11, 46, 63]. Note that Θ ( d -1 ) is the largest possible minimum eigenvalue since for any data distribution ˜ µ h , σ min ( E z h ∼ µ h [ φ ( z h ) φ ( z h ) /latticetop ]) ≤ d -1 by using ‖ φ ( z h ) ‖ ≤ 1 for any z h ∈ X × A .

Theorem 1. If the coverage coefficient in Definition 4.1 is finite, under Assumption 4.1, for corruption ζ = ∑ H h =1 ζ h and δ > 0 , we choose the covering parameter γ = 1 / ( n max h β h ζ h ) , λ = ln( N n ( γ )) , the weighting parameter α = H √ ln N n ( γ ) / ζ , and the confidence radius where N n ( γ ) = max h N ( γ /n, F h ) · N ( γ /n, F h +1 ) · N ( γ /n, B h +1 ( λ )) . Then, with probability at least 1 -2 δ , the sub-optimality of Algorithm 2 is bounded by

When ζ = O ( √ n ) , our algorithm achieves the same order of suboptimality as the uncorrupted case. Whenever ζ = o ( n ) , our algorithm is sample-efficient. Moreover, when specialized to the linear MDP with dimension d , where CC( λ , ̂ F , Z H n ) = O ( d/c † ) , C ( ̂ F , µ ) = Θ ( d -1 ) and ln N n ( γ ) = ˜ O ( d 2 ) , the suboptimality bound in Theorem 1 becomes ˜ O ( Hd 3 / 2 n -1 / 2 + d ζ n -1 ) . The corruption-independent term ˜ O ( Hd 3 / 2 n -1 / 2 ) matches that of PEVI [24]. The corruption-dependent term nearly matches the lower bound, as will be discussed later.

Remark 4.1. Although the theory requires a known corruption level ζ , in the experiments, we treat the uncertainty ratio α = O (1 / ζ ) as a tuning hyperparameter. The use of independent and identically distributed (i.i.d.) trajectories in our experiments renders the hyperparameter tuning process straightforward and conducive to optimizing the performance. Additionally, we can offer a choice of α = Θ (1 / √ n ) . This choice finds support in the online setting [55, 18], where this specific choice of α ensures that suboptimality remains in the order of uncorrupted error bound, even when ζ = O ( √ n ) .

Proof sketch. The detailed proof of Theorem 1 is provided in Appendix A. Here we present a brief proof sketch for the suboptimality bound, which is accomplished by three steps: (1) by Lemma A.1, if the uncorrupted Bellman backup T h f n ∈ ̂ F h for each h ∈ [ H ] , we can bound the suboptimality by the sum of the bonus ∑ H h =1 β h E π ∗ [ b h ( x h , a h ) | x 1 = x ] ; (2) by Lemma A.2, we demonstrate that an approximate uncertainty weight satisfying (6) is the key to bound the weighted Bellman error ( ∑ n i =1 (( ˆ f h -( T h f h +1 n ))( x h i , a h i )) 2 / ( σ h i ) 2 ) 1 / 2 by β h = c β ( αζ h + √ ln( HN n ( γ ) / δ )) ; and (3) combining the results in the first two steps, we can obtain the suboptimality bounded by:

where the weight for the trajectory induced by the optimal policy π ∗ is

To control the weighted coverage coefficient CC σ by its unweighted counterpart CC , which is a challenging task due to the intricate form of uncertainty-related weights, a novel technique has been employed. The main idea behind this technique is to transform CC σ into a modified version of CC multiplied by a term that can be bounded by leveraging a cover of the function class. The connection between CC and CC σ is shown in the following lemma.

Lemma 4.1. Under Assumption 4.1 and choose β h = C β √ ln N (where C β > 0 contains the logarithmic terms that are omitted) and λ given in Theorem 1, we have

Intuitively, because the weights themselves are closely related to the uncertainty, the weighted coverage coefficient CC σ where the high-uncertain samples are down-weighted can be upper-bounded by the square root of the unweighted coefficient CC . See Appendix B.3 for details.

(a)

(b)

(c)

(d)

Lower Bound. We construct a lower bound for linear MDPs with adversarial corruption ζ to show that the O ( d ζ n -1 ) corruption term of our suboptimality is optimal. The construction of the lower bound is adapted from Zhang et al. [60], where an /epsilon1 -constant fraction of the dataset is contaminated. We present the proof of Theorem 2 in Appendix B.4.

Theorem 2 (Minimax Lower Bound for Linear MDPs) . Under linear MDPs with corruption (Definition 2.2), for any fixed data-collecting distribution ν , any algorithm L : D → Π with the knowledge of ζ cannot find a better policy than O ( d ζ n -1 ) -optimal policy with probability more than 1 / 4 :

where D is the corrupted dataset initially generated from the MDP and then corrupted by an adversary, and ˆ π L is the policy generated by the algorithm L .

5 Experiments

Based on our theoretical results, we propose a practical implementation for CR-PEVI and verify its effectiveness on simulation tasks with corrupted offline data.

Practical Implementation. To make our algorithm more practical, we use neural networks to estimate the Q function (i.e., f in our theory) and the weight function σ . In linear MDPs, under a sufficiently broad function approximation class ̂ F h and sufficiently small parameter λ in Eq. (8), the bonus function can be simplified as the bootstrapped uncertainty of Q functions, which in turn can be estimated via the standard deviation of an ensemble of Q networks. We defer the detailed discussion to Appendix B.5. Following the state-of-the-art uncertainty-based offline RL algorithm Model Standard-deviation Gradients (MSG) [16], we learn a group of Q networks Q w i , i = 1 , . . . , K with independent targets and optimize a policy π θ with a lower-confidence bound (LCB) objective [16, 2]. Specifically, Q w i is learned to minimize a weighted regression objective similar to Eq.(7). The weight function σ is estimated via bootstrapped uncertainty: σ ( x, a ) = max( √ V i =1 ,...,K [ Q w i ( x, a )] , 1) , where V i =1 ,...,K [ Q w i ] is the variance between the group of Q functions. This uncertainty estimation method has also been adopted by prior works [2, 16, 54]. We refer to our practical algorithm as Uncertainty Weighted MSG (UWMSG) and defer details to Appendix D.

Experimental Setup. We assess the performance of our approach using continuous control tasks from [15] and introduce both random and adversarial attacks on either the rewards or dynamics for the offline datasets. Details about the four types of data corruption and their cumulative corruption levels are deferred to Appendix D. The ensemble size K is set to 10 for all experiments. For evaluation, we report average returns with standard deviations over 10 random seeds. More implementation details are also provided in Appendix D.

Experimental Results. We compare UWMSG with the state-of-the-art uncertainty-base offline RL methods, MSG [16], EDAC [1], and SAC-N [1] under four types of data corruption. In particular, MSGcanbeconsidered as UWMSG with a constant weighting function σ ( s, a ) = 1 . As demonstrated in Table 1 and Figure 1, our empirical results find that (1) current offline RL methods are susceptible to data corruption, e.g., MSG, EDAC, SAC-N achieve poor performance under adversarial attacks, and (2) our proposed UWMSG significantly improves performance under different data corruption scenarios, with an average improvement of 104% over MSG. More results can be found in Appendix F.

In summary, the experimental results validate the theoretical impact of data corruption for value-based offline RL algorithms. Our practical implementation algorithm demonstrates superior efficacy under different data corruption types, thereby highlighting its potential for real-world applications.

6 Conclusion

This work investigates the adversarially corrupted offline RL with general function approximation. We propose the uncertainty weight iteration and a weighted version of PEVI. Under a partial coverage condition and a well-explored dataset, our algorithm achieves a suboptimality bound, where the corruption-independent term recovers the uncorrupted bound, and the corruption-related term nearly matches the lower bound in linear models. Furthermore, our experiments demonstrate promising results showing that our practical implementation, UWMSG, significantly enhances the performance of the state-of-the-art offline RL algorithm under reward and dynamics data corruptions.

Our work suggests several potential future directions. First, it remains unsolved whether one can design robust algorithms to handle additional distribution shifts of the state-action pairs. Second, when the corruption level is unknown, how to design a theoretically robust offline RL algorithm and overcome the lack of adaptivity in the offline setting requires further research. Finally, we hope that our work sheds light on applying uncertainty weights to improve robustness in deep offline RL works and even practical applications.

References

An, G., Moon, S., Kim, J.-H., and Song, H. O. (2021). Uncertainty-based offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems , 34:74367447.

Bai, C., Wang, L., Yang, Z., Deng, Z., Garg, A., Liu, P., and Wang, Z. (2022). Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. arXiv preprint arXiv:2202.11566 .

Behzadan, V. and Munir, A. (2018). Mitigation of policy manipulation attacks on deep q-networks with parameter-space noise. In Computer Safety, Reliability, and Security: SAFECOMP 2018 Workshops, ASSURE, DECSoS, SASSUR, STRIVE, and WAISE, Västerås, Sweden, September 18, 2018, Proceedings 37 , pages 406-417. Springer.

Bogunovic, I., Losalka, A., Krause, A., and Scarlett, J. (2021). Stochastic linear bandits robust to adversarial attacks. In International Conference on Artificial Intelligence and Statistics , pages 991-999. PMLR.

Chen, L. and Luo, H. (2021). Finding the stochastic shortest path with low regret: The adversarial cost and unknown transition case. In International Conference on Machine Learning , pages 1651-1660. PMLR.

Chen, Z., Li, C. J., Yuan, A., Gu, Q., and Jordan, M. I. (2023). A general framework for sample-efficient function approximation in reinforcement learning. In International Conference on Learning Representations .

Deng, D., Chen, G., Yu, Y., Liu, F., and Heng, P.-A. (2023). Uncertainty estimation by fisher information-based evidential deep learning. arXiv preprint arXiv:2303.02045 .

Ding, Q., Hsieh, C.-J., and Sharpnack, J. (2022). Robust stochastic linear contextual bandits under adversarial attacks. In International Conference on Artificial Intelligence and Statistics , pages 7111-7123. PMLR.

Du, S., Kakade, S., Lee, J., Lovett, S., Mahajan, G., Sun, W., and Wang, R. (2021). Bilinear classes: A structural framework for provable generalization in rl. In International Conference on Machine Learning , pages 2826-2836. PMLR.

Du, Y., Watkins, O., Wang, Z., Colas, C., Darrell, T., Abbeel, P., Gupta, A., and Andreas, J. (2023). Guiding pretraining in reinforcement learning with large language models. arXiv preprint arXiv:2302.06692 .

Duan, Y., Jia, Z., and Wang, M. (2020). Minimax-optimal off-policy evaluation with linear function approximation. In International Conference on Machine Learning , pages 2701-2709. PMLR.

Eykholt, K., Evtimov, I., Fernandes, E., Li, B., Rahmati, A., Xiao, C., Prakash, A., Kohno, T., and Song, D. (2018). Robust physical-world attacks on deep learning visual classification. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1625-1634.

Foster, D. J., Gentile, C., Mohri, M., and Zimmert, J. (2020). Adapting to misspecification in contextual bandits. Advances in Neural Information Processing Systems , 33:11478-11489.

Foster, D. J., Kakade, S. M., Qian, J., and Rakhlin, A. (2021). The statistical complexity of interactive decision making. arXiv preprint arXiv:2112.13487 .

Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. (2020). D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 .

Ghasemipour, K., Gu, S. S., and Nachum, O. (2022). Why so pessimistic? estimating uncertainties for offline rl through ensembles, and why their independence matters. Advances in Neural Information Processing Systems , 35:18267-18281.

Gupta, A., Koren, T., and Talwar, K. (2019). Better algorithms for stochastic bandits with adversarial corruptions. In Conference on Learning Theory , pages 1562-1578. PMLR.

He, J., Zhou, D., Zhang, T., and Gu, Q. (2022). Nearly optimal algorithms for linear contextual bandits with adversarial corruptions. arXiv preprint arXiv:2205.06811 .

Jiang, N., Krishnamurthy, A., Agarwal, A., Langford, J., and Schapire, R. E. (2017). Contextual decision processes with low bellman rank are pac-learnable. In International Conference on Machine Learning , pages 1704-1713. PMLR.

Jin, C., Jin, T., Luo, H., Sra, S., and Yu, T. (2020a). Learning adversarial markov decision processes with bandit feedback and unknown transition. In International Conference on Machine Learning , pages 4860-4869. PMLR.

Jin, C., Liu, Q., and Miryoosefi, S. (2021a). Bellman eluder dimension: New rich classes of rl problems, and sample-efficient algorithms. Advances in Neural Information Processing Systems , 34.

Jin, C., Yang, Z., Wang, Z., and Jordan, M. I. (2020b). Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory , pages 2137-2143. PMLR.

Jin, T. and Luo, H. (2020). Simultaneously learning stochastic and adversarial episodic mdps with known transition. Advances in neural information processing systems , 33:16557-16566.

Jin, Y., Yang, Z., and Wang, Z. (2021b). Is pessimism provably efficient for offline rl? In International Conference on Machine Learning , pages 5084-5096. PMLR.

Kang, Y., Hsieh, C.-J., and Lee, T. (2023). Robust lipschitz bandits to adversarial corruptions. arXiv preprint arXiv:2305.18543 .

Kober, J., Bagnell, J. A., and Peters, J. (2013). Reinforcement learning in robotics: A survey. The International Journal of Robotics Research , 32(11):1238-1274.

Lee, C.-W., Luo, H., Wei, C.-Y., Zhang, M., and Zhang, X. (2021). Achieving near instanceoptimality and minimax-optimality in stochastic and adversarial linear bandits simultaneously. In International Conference on Machine Learning , pages 6142-6151. PMLR.

Liu, Y., Logan, B., Liu, N., Xu, Z., Tang, J., and Wang, Y. (2017). Deep reinforcement learning for dynamic treatment regimes on medical registry data. In 2017 IEEE international conference on healthcare informatics (ICHI) , pages 380-385. IEEE.

Luo, H., Wei, C.-Y., and Lee, C.-W. (2021). Policy optimization in adversarial mdps: Improved exploration via dilated bonuses. Advances in Neural Information Processing Systems , 34:2293122942.

Lykouris, T., Mirrokni, V., and Paes Leme, R. (2018). Stochastic bandits robust to adversarial corruptions. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing , pages 114-122.

Ma, Y., Zhang, X., Sun, W., and Zhu, J. (2019). Policy poisoning in batch reinforcement learning and control. Advances in Neural Information Processing Systems , 32.

Neff, G. (2016). Talking to bots: Symbiotic agency and the case of tay. International Journal of Communication .

Neu, G., György, A., Szepesvári, C., et al. (2010). The online loop-free stochastic shortest-path problem. In COLT , volume 2010, pages 231-243. Citeseer.

Osband, I., Blundell, C., Pritzel, A., and Van Roy, B. (2016). Deep exploration via bootstrapped dqn. Advances in neural information processing systems , 29.

Pan, Y., Cheng, C.-A., Saigol, K., Lee, K., Yan, X., Theodorou, E., and Boots, B. (2017). Agile autonomous driving using end-to-end deep imitation learning. arXiv preprint arXiv:1709.07174 .

Rashidinejad, P., Zhu, B., Ma, C., Jiao, J., and Russell, S. (2021). Bridging offline reinforcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems , 34:11702-11716.

Rosenberg, A. and Mansour, Y. (2019). Online convex optimization in adversarial markov decision processes. In International Conference on Machine Learning , pages 5478-5486. PMLR.

Rosenberg, A. and Mansour, Y. (2020). Stochastic shortest path with adversarially changing costs. arXiv preprint arXiv:2006.11561 .

Russo, D. and Van Roy, B. (2013). Eluder dimension and the sample complexity of optimistic exploration. Advances in Neural Information Processing Systems , 26.

Sun, H., Han, L., Yang, R., Ma, X., Guo, J., and Zhou, B. (2022a). Exploit reward shifting in value-based deep-rl: Optimistic curiosity-based exploration and conservative exploitation via linear reward shaping. Advances in Neural Information Processing Systems , 35:37719-37734.

Sun, H., van Breugel, B., Crabbe, J., Seedat, N., and van der Schaar, M. (2022b). Daux: a density-based approach for uncertainty explanations. arXiv preprint arXiv:2207.05161 .

Tarasov, D., Nikulin, A., Akimov, D., Kurenkov, V., and Kolesnikov, S. (2022). CORL: Research-oriented deep offline reinforcement learning library. In 3rd Offline RL Workshop: Offline RL as a 'Launchpad' .

Tropp, J. A. (2012). User-friendly tail bounds for sums of random matrices. Foundations of computational mathematics , 12:389-434.

Uehara, M. and Sun, W. (2021). Pessimistic model-based offline reinforcement learning under partial coverage. arXiv preprint arXiv:2107.06226 .

Wang, L., Zhang, W., He, X., and Zha, H. (2018). Supervised reinforcement learning with recurrent neural network for dynamic treatment recommendation. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining , pages 2447-2456.

Wang, R., Foster, D. P., and Kakade, S. M. (2020a). What are the statistical limits of offline rl with linear function approximation? arXiv preprint arXiv:2010.11895 .

Wang, R., Salakhutdinov, R. R., and Yang, L. (2020b). Reinforcement learning with general value function approximation: Provably efficient approach via bounded eluder dimension. Advances in Neural Information Processing Systems , 33:6123-6135.

Wei, C.-Y., Dann, C., and Zimmert, J. (2022). A model selection approach for corruption robust reinforcement learning. In International Conference on Algorithmic Learning Theory , pages 1043-1096. PMLR.

Wu, F., Li, L., Zhang, H., Kailkhura, B., Kenthapadi, K., Zhao, D., and Li, B. (2022). Copa: Certifying robust policies for offline reinforcement learning against poisoning attacks. In International Conference on Learning Representations .

Wu, T., Yang, Y., Du, S., and Wang, L. (2021). On reinforcement learning with adversarial corruption and its application to block mdp. In International Conference on Machine Learning , pages 11296-11306. PMLR.

Xie, T., Cheng, C.-A., Jiang, N., Mineiro, P., and Agarwal, A. (2021). Bellman-consistent pessimism for offline reinforcement learning. Advances in neural information processing systems , 34:6683-6694.

Xiong, W., Zhong, H., Shi, C., Shen, C., Wang, L., and Zhang, T. (2022). Nearly minimax optimal offline reinforcement learning with linear function approximation: Single-agent mdp and markov game. arXiv preprint arXiv:2205.15512 .

Yang, R., Bai, C., Ma, X., Wang, Z., Zhang, C., and Han, L. (2022). Rorl: Robust offline reinforcement learning via conservative smoothing. In Advances in Neural Information Processing Systems .

Yang, R., Yong, L., Ma, X., Hu, H., Zhang, C., and Zhang, T. (2023). What is essential for unseen goal generalization of offline goal-conditioned rl? In International Conference on Machine Learning , pages 39543-39571. PMLR.

Ye, C., Xiong, W., Gu, Q., and Zhang, T. (2022). Corruption-robust algorithms with uncertainty weighting for nonlinear contextual bandits and markov decision processes.

Yin, M. and Wang, Y.-X. (2021). Towards instance-optimal offline reinforcement learning with pessimism. Advances in neural information processing systems , 34:4065-4078.

Zanette, A., Wainwright, M. J., and Brunskill, E. (2021). Provable benefits of actor-critic methods for offline reinforcement learning. Advances in neural information processing systems , 34:13626-13640.

Zhang, H., Chen, H., Xiao, C., Li, B., Liu, M., Boning, D., and Hsieh, C.-J. (2020a). Robust deep reinforcement learning against adversarial perturbations on state observations. Advances in Neural Information Processing Systems , 33:21024-21037.

Zhang, T. (2023). Mathematical Analysis of Machine Learning Algorithms . Cambridge University Press. in press, also available as http://tongzhang-ml.org/lt-book.html .

Zhang, X., Chen, Y., Zhu, X., and Sun, W. (2022). Corruption-robust offline reinforcement learning. In International Conference on Artificial Intelligence and Statistics , pages 5757-5773. PMLR.

Zhang, X., Ma, Y., Singla, A., and Zhu, X. (2020b). Adaptive reward-poisoning attacks against reinforcement learning. In International Conference on Machine Learning , pages 11225-11234. PMLR.

Zhao, H., Zhou, D., and Gu, Q. (2021). Linear contextual bandits with adversarial corruptions. arXiv preprint arXiv:2110.12615 .

Zhong, H., Xiong, W., Tan, J., Wang, L., Zhang, T., Wang, Z., and Yang, Z. (2022a). Pessimistic minimax value iteration: Provably efficient equilibrium learning from offline datasets. In International Conference on Machine Learning , pages 27117-27142. PMLR.

Zhong, H., Xiong, W., Zheng, S., Wang, L., Wang, Z., Yang, Z., and Zhang, T. (2022b). A posterior sampling framework for interactive decision making. arXiv preprint arXiv:2211.01962 .

Zhou, D. and Gu, Q. (2022). Computationally efficient horizon-free reinforcement learning for linear mixture mdps. Advances in neural information processing systems , 35:36337-36349.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies offline RL with general function approximation in the presence of adversarial corruption. Specifically, an adversary can corrupt the rewards and transitions in the offline dataset with a fixed cumulative budget. The corruption formulation encompasses prior corruption formulations. The proposed algorithm is based on an uncertainty-weight iteration subroutine, inspired by similar techniques in prior robust online RL papers. The authors conduct a finite-sample analysis, showing a tighter result on the corruption term. The theory is accompanied by a practical implementation tested over a couple of D4RL tasks.\n\nSTRENGTHS:\n- The authors propose a corruption formulation, which admits prior formulations such as fixed fraction contamination of Zhang et al. as special cases.\n- The theoretical result improves over the most relevant prior work Zhang et al. from $\\sqrt{\\epsilon}$ dependency to $\\epsilon$ dependency of the corruption term, which matches the lower bound when specialized to the linear case.\n- Inspired by the uncertainty weighting approach in robust online RL, the authors propose a similar weight iteration method in the offline RL case. However, in the offline case, the\n- The authors implement their algorithm over a subset of continuous control tasks in D4RL.\n\nWEAKNESSES:\n- The main limitation of this work is that the proposed algorithm requires the corruption level to be known. I did not find any attempts on possible heuristics for estimating the corruption level, which limits the practicality of the proposed approach.\n- The experiments section is generally weak. The practical implementation is limted to only two continous control tasks (Walker and Half Cheetah) from D4RL that are artificially corrupted. It would be good to see and compare performance with other tasks in D4RL as well. Furthermore, the approach is compared with uncertainty-based offline RL algorithms that are not designed to be corruption robust, making the comparison unfair.\n\nQUESTIONS:\nSuggestions:\n- Perhaps comparison with an implementation of Zhang et al instead of uncertainty-based offline RL is more appropriate.\n\nLIMITATIONS:\nYes, clearly stated.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe research investigates the issue of corruption robustness in offline reinforcement learning with general function approximation, where an adversary can corrupt samples in the offline dataset. The objective is to find a policy that remains robust against such corruption and minimizes the suboptimality gap compared to the optimal policy in uncorrupted Markov decision processes. To address this, a new uncertainty weight iteration procedure inspired by the robust online RL setting is designed, enabling efficient computation on batched samples. The proposed corruption-robust algorithm achieves a suboptimality bound, considering single policy coverage and knowledge of the corruption level. The analysis reveals that the corruption-dependent error term matches the existing lower bound for corrupted linear MDPs, indicating the tightness of the analysis in terms of corruption dependence.\n\nSTRENGTHS:\nI think this work really pushes the data-corruption solving real-world RL community research efforts further by answering:\n\n> Can we design a generic algorithm for data-corrupted offline RL in the context of function approximation?\n\nThe main contribution: algorithmic introduction to uncertainty reweighting idea. Both theoretical and practical performances have been shown. This is good for publication at NeurIPS.\n\nWEAKNESSES:\nI have only one minor weakness for this work as follows:\n\nOn Assumption 2.1. The recent offline RL theoretical guarantees proposing pessimism is to avoid completeness assumption and having weaker concentrabilities, like that of [23]. I don't see what is pessimism helping in this work if not to avoid the completeness. Maybe I am missing something. I am okay with this assumption if the work is trying to address the corruption-robust problem which requires (?) completeness?\n\nQUESTIONS:\nplease see weaknesses for more.\n\nLIMITATIONS:\nLimitations is marked Yes, but I couldn't find the discussion in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a new robust offline RL algorithm called the Corruption Robust PEVI (CR-PEVI). The CR-PEVI achieves smaller suboptimality error compared to existing methods. The key idea behind CR-PEVI is to utilize the uncertainty-weighting technique when solving the least-squares problem in PEVI. The weights in the proposed method do not have closed-form solution, thus the authors proposed an iterative approach and relies on the monotone convergence theorem to prove convergence to the target weights. Based on that and several other assumptions, e.g., dataset needs to be well-explored, the paper proved an upper bound on the suboptimality of the policy learned by the CR-PEVI algorithm. When specializing to the linear function approximation with dimension d, the proposed CR-PEVI achieves O(eta*d/n) convergence rate. Experiments demonstrate the effectiveness of the CR-PEVI method.\n\nSTRENGTHS:\n(1). This paper studied robust offline RL, which is a hot and timely topic in recent years. Prior works mostly consider test-time attacks that are relatively easier to defend. However, less is understood about robustness against training-time attacks. This paper pushes the frontier in this domain and gained deeper understanding on how to design poisoning-resilient offline RL algorithms in the function approximation scenario, which is important and fascinating work.\n\n(2). The theoretical analysis is intriguing. The authors combined the uncertainty-weighting technique and the PEVI algorithm to propose a new algorithm that is robust to offline poisoning attacks. The analysis is based on several critical theoretical results established in the paper, which are complete and solid. Overall, the paper has strong theory contributions.\n\n(3). The empirical results are also convincing. On all the tasks, the CR-PEVI achieves much better performance compared to baselines.\n\nWEAKNESSES:\n(1). It seems like the definition of cumulative corruption is not commonly used by prior works. It is defined as the gap between the Bellman operators induced by the clean dataset and corrupted dataset, which is less intuitive and harder to quantize. In contrast, it makes more sense to define corruption level as the real modification on the reward/state transition over the trajectories.\n\n(2). In the experiment, the setup does not strictly conform with attacks under limited budget. The authors implemented different attacks without reporting the corruption level for every different attack and victim algorithm.  Furthermore, there is not parameter study on the corruption level eta as well. This makes me wonder if the comparison to prior works, in particular, Zhang et al, is fair enough.\n\n(3) Missing a few related references on training-time attacks in RL:\n\n[1]. Policy Poisoning in Batch Reinforcement Learning and Control\n\n[2]. Adaptive Reward-Poisoning Attacks against Reinforcement Learning\n\nQUESTIONS:\n(1). Why the paper chooses to define the cumulative corruption based on the Bellman operator instead of the modification on the dataset? Is the goal to favor theoretical analysis? How does the theory changes if the corruption level is defined differently, e.g., total change on the rewards?\n\n(2). It's important to report the corruption level induced by different attack and victim RL algorithms in the experiment section. Furthermore, I was wondering how the cumulative corruption level affects the policy learned by CR-PEVI in practical scenarios.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

comments: SUMMARY:
This work considers corrupted offline RL with general function approximation, and propose an algorithm based on PEVI and the uncertainty weighting technique. Under certain date coverage and well-explored dataset assumptions, the proposed algorithm achieves neal-optimal bound, where the corruption-independent term matches with the uncorrupted bound while the corruption-dependent term nearly matches the lower bound for linear models. Futher, experiments showcase the efficetiveness of the algorithm.

STRENGTHS:
1 This work is well-organized and well-written.

2 The studied corrupted offline RL with general function approximation is not being widely explored.

3 Adapting uncertainty-related weights technique to offline setting may be of interest.

WEAKNESSES:
1 The uncertainty-weighting technique and the PEVI are not new.

2 This work extends the corrupted online setting to corrupted offline setting

QUESTIONS:
1 Is the result comparable to the existing corrupted online RL with general function approximation? 

2 Could the authors comment more on Eqn. (6) and give intuition? It is unclear what tuning parameter $\alpha$ and $\lambda$ are designed for in this uncertainty-related weights. 

3 For the lower bound shown in Theorem 2, it would be great if the authors provide any insight.

In line 12, ''$\widehat{F}$'' -> (or <-) ''$\hat{F}$''.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07