Title: Corruption-Robust Linear Bandits: Minimax Optimality and Gap-Dependent Misspecification
Venue: NeurIPS 2024
Accepted: yes
Keywords: corruption-robust

linear bandits

misspecification

reinforcement learning
Abstract: In linear bandits, how can a learner effectively learn when facing corrupted rewards? While significant work has explored this question, a holistic understanding across different adversarial models and corruption measures is lacking, as is a full characterization of the minimax regret bounds. In this work, we compare two types of corruptions commonly considered: strong corruption, where the corruption level depends on the learner’s chosen action, and weak corruption, where the corruption level does not depend on the learner’s chosen action. We provide a unified framework to analyze these corruptions. For stochastic linear bandits, we fully characterize the gap between the minimax regret under strong and weak corruptions. We also initiate the study of corrupted adversarial linear bandits, obtaining upper and lower bounds with matching dependencies on the corruption level. Next, we reveal a connection between corruption-robust learning and learning with gap-dependent misspecification—a setting first studied by Liu et al. (2023a), where the misspecification level of an action or policy is proportional to its suboptimality. We present a general reduction that enables any corruption-robust algorithm to handle gap-dependent misspecification. This allows us to recover the results of Liu et al. (2023a) in a black-box manner and significantly generalize them to settings like linear MDPs, yielding the first results for gap-dependent misspecification in reinforcement learning. However, this general reduction does not attain the optimal rate for gap-dependent misspecification. Motivated by this, we develop a specialized algorithm that achieves optimal bounds for gap-dependent misspecification in linear bandits, thus answering an open question posed by Liu et al. (2023a).

Corruption-Robust Linear Bandits: Minimax Optimality and Gap-Dependent Misspecification

Abstract

In linear bandits, how can a learner effectively learn when facing corrupted rewards? While significant work has explored this question, a holistic understanding across different adversarial models and corruption measures is lacking, as is a full characterization of the minimax regret bounds. In this work, we compare two types of corruptions commonly considered: strong corruption , where the corruption level depends on the learner's chosen action, and weak corruption , where the corruption level does not depend on the learner's chosen action. We provide a unified framework to analyze these corruptions. For stochastic linear bandits, we fully characterize the gap between the minimax regret under strong and weak corruptions. We also initiate the study of corrupted adversarial linear bandits, obtaining upper and lower bounds with matching dependencies on the corruption level.

Next, we reveal a connection between corruption-robust learning and learning with gap-dependent misspecification -asetting first studied by Liu et al. (2023a), where the misspecification level of an action or policy is proportional to its suboptimality. We present a general reduction that enables any corruption-robust algorithm to handle gap-dependent misspecification. This allows us to recover the results of Liu et al. (2023a) in a black-box manner and significantly generalize them to settings like linear MDPs, yielding the first results for gap-dependent misspecification in reinforcement learning. However, this general reduction does not attain the optimal rate for gap-dependent misspecification. Motivated by this, we develop a specialized algorithm that achieves optimal bounds for gap-dependent misspecification in linear bandits, thus answering an open question posed by Liu et al. (2023a).

1 Introduction

The real world is rarely truly stochastic-in practice, our observations are often corrupted-and furthermore, rarely are the modeling assumption typically made in theory-that the true datagenerating process lives in our model class-met in reality. Therefore, robustly handling these deviations from idealized assumptions is crucial. These challenges are particularly pronounced in interactive decision-making settings, where deviations from idealized assumptions could lead an algorithm to take unsafe or severely suboptimal actions. In this work, we seek to address these challenges, and develop a unified understanding for robust learning in corruption-robust and misspecified settings.

We first consider the corruption-robust learning setting. Robust learning in the presence of corruptions requires designing algorithms whose guarantee have a tight scaling in the corruption level. That is, although some amount of suboptimality is inevitable if our observations are corrupted, we would hope to obtain the minimum amount of suboptimality possible at a given corruption level. While much work has been done on learning with corrupted observations, existing work has failed to yield a tight characterization of this scaling in the corruption level, even in simple settings such as linear bandits. We address this shortcoming, and develop an algorithm which achieves the optimal scaling in the corruption level, and further extend this to a novel corrupted adversarial linear bandit setting, where in addition to corrupted observations, the rewards themselves may be adversarially chosen from round to round. We obtain the first provably efficient bounds in this setting.

Model misspecification, another extensively studied problem in the literature, can be thought of as a form of corruption, where the corruption level is the amount of misspecification between the 'closest' model in the model class and the true environment. Standard discussions on misspecification usually assume that the misspecification for every action has a uniform upper bound, and the final regret guarantee scales linearly with the amount of misspecification. The work of Liu et al. (2023a) initiated the study on the gap-dependent misspecification setting, where the misspecification level for a given action scales with the suboptimality of that action. They demonstrated that the linear scaling in regret is not necessary in this case. We revisit this problem, and show a general reduction from the gap-dependent misspecified setting to the corruption setting. We utilize this reduction to show that settings previously not known to be learnable-for example, linear MDPs with policy gap-dependent misspecification-are in fact efficiently learnable with existing corruption robust algorithms.

Together, our results present a unified picture of optimally learning in the presence of observation corruption, and (certain types of) model misspecification. We summarize our contributions as follows (see Section 2 and Section 3 for formal definitions of the mentioned quantities):

In Section 4, we develop a stochastic linear bandit algorithm with ˜ O ( d √ T +min { dC, √ dC ∞ } ) regret, where d is the feature dimension, T is the number of rounds, C is the strong corruption measure, and C ∞ is the weak corruption measure. These bounds are unimprovable.

In Section 5, we initiate the study of adversarial linear bandits with corruptions. We obtain ˜ O ( d √ T + √ dC ∞ ) and ˜ O ( √ d 3 T + dC ) regret for weak and strong corruptions, respectively.

We prove a general reduction that efficiently handles gap-dependent misspecification with corruption-robust algorithms. We apply our reduction to show that linear MDPs with gapdependent misspecification are efficiently learnable (Section 6).

Finally, while the reduction in item 3 is general, it is unable to obtain the tightest possible rate for gap-dependent misspecification. We thus develop a specialized algorithm which, in the linear bandit setting, obtains the optimal rate. This resolves the open problem of Liu et al. (2023a).

In Section 2 we present our problem setting, and in Section 3, compare the corruption notions in previous and our work. More related works are discussed in Appendix A. In Section 4-Section 6, we present our main results as outlined above.

2 Problem Setting and Preliminaries

We consider the corrupted linear bandit problem. The learner interacts with the environment for T rounds. The learner is given an action set A ⊂ R d . At the beginning of round t , the environment determines a reward vector θ t ∈ R d and a corruption function ϵ t ( · ) : A → [ -1 , 1] , which are both hidden from the learner. The learner then selects an action a t ∈ A . Then a reward value r t = a ⊤ t θ t + ϵ t ( a t ) + ζ t is revealed to the learner, for some zero-mean noise ζ t ∈ [ -1 , 1] 2 . We assume that ∥ a ∥ 2 ≤ 1 , ∥ θ t ∥ 2 ≤ √ d , and a ⊤ θ t ∈ [ -1 , 1] for any a ∈ A and any t = 1 , 2 , . . . , T . We define ϵ t = max a ∈A | ϵ t ( a ) | .

In the stochastic setting, the environment is restricted to choose θ t = θ ⋆ for all t , while in the adversarial setting, θ t can arbitrarily depend on the history up to round t -1 . The regret of the learner is defined as

Note that although the non-stationarity of θ t in the adversarial setting captures a certain degree of corruption, this form of corruption is limited to a linear form a ⊤ ( θ t -θ ⋆ ) , which is not as general as ϵ t ( a ) that could be an arbitrarily function. Therefore, the corrupted linear bandit problem cannot be reduced to an adversarial linear bandit problem.

Notation. We denote [ n ] = { 1 , 2 , . . . , n } . Let ∆( A ) be the set of distribution over A . For any p ∈ ∆( A ) , define the lifted covariance matrix ̂ Cov ( p ) = E a ∼ p [ aa ⊤ a a ⊤ 1 ] ∈ R ( d +1) × ( d +1) . For A,B ∈ R d × d , define ⟨ A,B ⟩ = Tr ( AB ⊤ ) . E t [ · ] is the expectation conditioned on history up to t -1 .

G-Optimal Design. A G-optimal design over A is a distribution ρ ∈ ∆( A ) such that ∥ a ∥ 2 G -1 ≤ d for all a ∈ A , where G = ∑ a ∈A ρ ( a ) aa ⊤ . Note that such a distribution is guaranteed to exist, and can be efficiently computed (Pukelsheim, 2006; Lattimore and Szepesvári, 2020).

3 Two Equivalent Views: On Adversary Adaptivity and Corruption Measure

Previous works have studied corruption with various assumptions on the adaptivity of the adversary and different measures for the corruption level. In this work, we consider both the strong and weak guarantees, which can cover different notions of corruptions studied in previous works. We provide two different viewpoints to understand them. In the first viewpoint, the weak and strong guarantee differ by the adaptivity of the adversary, while in the second viewpoint, the two guarantees differ in the measure of corruption . Then we argue that the two viewpoints are equivalent.

Adversary Adaptivity (AA) Viewpoint. In this viewpoint, the corruption is specified only for the chosen action. That is, in each round t , the adversary only decides a single corruption level ϵ t ∈ R ≥ 0 and ensures | E [ r t ] - ⟨ a t , θ ⋆ ⟩ | ≤ ϵ t . We consider two kinds of adversary: strong adversary who decides ϵ t after seeing the chosen action a t , and weak adversary who decides ϵ t before seeing a t . The robustness of the algorithm is measured by how the regret depends on ∑ T ϵ t .

t =1

Corruption Measure (CM) Viewpoint. In this viewpoint, the corruption is individually specified for every action. That is, at each round t , the adversary decides ϵ t ( a ) for all action a ∈ A and ensures E [ r t | a t = a ] -⟨ a, θ ⋆ ⟩ = ϵ t ( a ) for all a . The adversary always decides ϵ t ( · ) before seeing a t . To evaluate the performance, we consider two different measures of the total corruption: the strong measure ∑ T t =1 | ϵ t ( a t ) | and the weak measure ∑ T t =1 max a ∈A | ϵ t ( a ) | .

We argue that the two viewpoints are equivalent in the sense that the performance guarantee of an algorithm under strong/weak adversary in the AA viewpoint are the same as those under strong/weak measure in the CM viewpoint, respectively. This is by the following observation. A strong adversary in the AA viewpoint who decides the corruption level ϵ t after seeing a t can be viewed as deciding the corruption ϵ t ( a ) for all action a before seeing a t , and set ϵ t = | ϵ t ( a t ) | after seeing a t . In other words, ϵ t ( a ) is the corruption planned (before seeing a t ) by a strong adversary assuming a t = a , and the adversary simply carries out its plan after seeing a t . It is clear that this is equivalent to the CM viewpoint with ∑ T t =1 | ϵ t ( a t ) | as the corruption measure. See Appendix B for more details. On the other hand, a weak adversary in the AA viewpoint has to decide an upper bound of the corruption level ϵ t no matter which action a t is chosen by the learner. This can be viewed as deciding the corruption ϵ t ( a ) for every action a before seeing a t with the restriction | ϵ t ( a ) | ≤ ϵ t for all a . Therefore, this is equivalent to using ∑ T t =1 max a | ϵ t ( a ) | to measure total corruption in the CM viewpoint.

In this work, we adopt the CM viewpoint as described in Section 2. With the CM viewpoint, for both strong and weak settings, the power of the adversary remains the same as the standard 'adaptive adversary' (i.e., deciding the corruption function ϵ t ( · ) based on the history up to time t -1 ), and we only need to derive regret bounds with different corruption measures. All our results can also be interpreted in the AA viewpoint, as the above argument suggests.

With this unified viewpoint, we categorize in Table 1 previous works on linear (contextual) bandits based on the corruption measure, all under the same type of adversary. According to the definitions in Table 1, C and C ∞ correspond to the strong measure and weak measure mentioned above, respectively. It is easy to see that C ≤ { C ∞ , C sq } ≤ C sq , ∞ ≤ C ms , where C ∞ and C sq are incomparable.

For stochastic linear bandits, considering the relations among different corruption measures, the Pareto frontiers of the existing upper bounds are ˜ O ( d √ T + √ dC sq ) by Foster et al. (2020) and Takemura et al. (2021), and ˜ O ( d √ T + dC ) by He et al. (2022). The lower bound frontiers are Ω( d √ T + √ dC ms ) by Lattimore et al. (2020) and Ω( d √ T + dC ) by Bogunovic et al. (2020). These results imply an ˜ O ( d √ T + dC ∞ ) upper bound and an Ω( d √ T + √ dC ∞ ) lower bound, which still have a gap. In this work, we close the gap by showing an ˜ O ( d √ T + √ dC ∞ ) upper bound.

For adversarial linear bandits, we are only aware of upper bound ˜ O ( d √ T + √ dC sq , ∞ ) by Liu et al. (2024), and not aware of any upper bounds related to C ∞ or C . In this work, we show ˜ O ( d √ T + √ dC ∞ ) and ˜ O ( √ d 3 T + dC ) upper bounds. The results are summarized in Table 2. As in most previous work, we assume that C ∞ and C (or their upper bounds) are known by the learner when developing the algorithms. The case of unknown C ∞ or C is discussed in Appendix C.

We emphasize that before our work, for both stochastic and adversarial linear bandits, it was unknown how to achieve ˜ O ( d √ T + √ dC ∞ ) regret. To see how C ∞ is different from other notions such as C ms and C sq , we observe that for stochastic linear bandits, while ˜ O ( d √ T + √ dC sq ) can be achieved via deterministic algorithms, it is not the case for ˜ O ( d √ T + √ dC ∞ ) . The reason is that for deterministic algorithms, the adversary can control C ∞ to be the same as C , for which Ω( d √ T + dC ) is unavoidable. We formalize this in Proposition 1, with the proof given in Appendix D. This precludes the possibility of many previous algorithms to actually achieve the ˜ O ( d √ T + √ dC ∞ ) upper bound, e.g., Lattimore et al. (2020), Takemura et al. (2021), Bogunovic et al. (2020, 2021), He et al. (2022).

Proposition 1. For stochastic linear bandits, there exists a deterministic algorithm achieving Reg T = ˜ O ( d √ T + √ dC sq ) , while any deterministic algorithm must suffer Reg T = Ω( d √ T + dC ∞ ) .

4 Stochastic Linear Bandits

In this section, we introduce Algorithm 1, which achieves optimal regret for both C and C ∞ .

Algorithm 1 is an elimination-based algorithm. At each epoch k , it samples actions from a fixed distribution p k ∈ ∆( A k ) , which is a G-optimal design over the active action set A k (Line 4). At the end of epoch k , only actions that are within the error threshold will be kept in the active action set of the next epoch (Eq. (1)). While previous works by Lattimore et al. (2020) and Bogunovic et al. (2021) have used a similar elimination framework to obtain ˜ O ( d √ T + √ dC ms ) and ˜ O ( d √ T + d 3 2 C ) bounds, respectively, we note that their algorithms only specify the number of times the learner should sample for each action in each epoch. This is different from our algorithm that requires the learner to exactly use the distribution p k to sample actions in every round in epoch k . As argued in Proposition 1, if their algorithms are instantiated as a deterministic algorithm, then the regret will be at least Ω( d √ T + dC ∞ ) . Thus, this subtle difference is important.

Note that to achieve the tight C ∞ (or C ) bound, Z = √ dC ∞ (or Z = dC ) has to be input to the algorithm to decide the error threshold. The guarantee of Algorithm 1 is stated in Theorem 4.1.

Theorem 4.1. With input Z = √ dC ∞ or Z = dC , Algorithm 1 ensures with probability at least 1 -δ that Reg T ≤ O ( d √ T log( T/δ ) + Z log T ) .

Algorithm 1 can also be shown to ensure that Reg T ≤ O ( √ dT log( |A| T/δ )+ Z log T ) , which could be smaller than the bound given in Theorem 4.1 when |A| is small.

5 Adversarial Linear Bandits

In this section, we consider corrupted adversarial linear bandits. Although adversarial linear bandits have been widely studied, robustness under corruption is an under-explored topic: there is no prior work obtaining regret bounds that linearly depends on either C ∞ or C .

5.1 C ∞ bound in Adversarial Linear Bandits

Our algorithm (Algorithm 2) is based on follow-the-regularized-leader (FTRL) with logdet regularizer. Similar to previous works (Foster et al., 2020; Zimmert and Lattimore, 2022; Liu et al., 2024, 2023b) that utilize logdet regularizer, the feasible set H is in R ( d +1) × ( d +1) , which is the space of the covariance matrix for distributions over the lifted action space (Line 2-Line 3). At round t , the algorithm obtains a covariance matrix H t by solving the FTRL objective (Eq. (2)). The action distribution p t is such that the induced covariance matrix is equal to H t (Eq. (3)). After sampling a t ∼ p t and obtaining the reward r t , the algorithm constructs reward vector estimator ̂ θ t (Line 8) and feeds it to FTRL. The reader may refer to Zimmert and Lattimore (2022) for more details.

In typical corruption-free adversarial linear bandits, the learner would construct an unbiased reward vector estimator. However, in the presence of corruption, the learner can no longer construct an unbiased estimator. To compensate the bias, we adopt the idea of 'adding exploration bonus' inspired

Algorithm 2: FTRL with log-determinant barrier regularizer (for adversarial C ∞ bound)

2 Let ρ ∈ ∆( A ) be a G-optimal design over A , and let ∆ γ ( A ) = { p : p = (1 -γ ) p ′ + γρ, p ′ ∈ ∆( A ) } .

3 Define feasible set H = { ̂ Cov ( p ) : p ∈ ∆ γ ( A ) } . ( ̂ Cov ( p ) is defined in Section 2)

4 Define G ( H ) = -log det( H ) and B 0 = 0 .

5 for t = 1 , 2 , . . . do

6 Solve the fixed-point problem Eq. (2)-Eq. (5).

7 Sample a t ∼ p t . Observe reward r t with E [ r t ] = a ⊤ t θ t + ϵ t ( a t ) 8 Construct reward estimator ̂ θ t = Σ -1 t a t r t .

.

𝐵𝑡-1

Σ𝑡

-1

𝐵𝑡

(a) The bonus function

(b) Illustration for B t = BONUS ( B t -1 , Σ t ) . A psd matrix with eigenvalues ( λ i ) d i =1 is represented as an ellipsoid with radius ( √ λ i ) d i =1 .

by previous work on high-probability adversarial linear bandits (Lee et al., 2020; Zimmert and Lattimore, 2022). In the regret analysis, the exploration bonus creates a negative term that cancels the bias of the loss estimator. The bonus is represented by the B t in Eq. (5).

To decide the form of B t , we first analyze the bias. With the standard construction of the reward estimator, the bias on the benchmark action u can be calculated as (with ϵ t := max a | ϵ t ( a ) | )

where Σ t is the feature covariance matrix induced by p t (defined in Eq. (4)). Below, we compare different bonus designs in previous and our work.

Bonus design in previous work. In Zimmert and Lattimore (2022), which is also based on logdetFTRL but where the goal is only to get a high-probability bound, the bonus introduces an additional regret the form -α ∥ u ∥ 2 Σ -1 t + α ∑ a p t ( a ) ∥ a ∥ 2 Σ -1 t . This can be used to cancel off the bias in Eq. (6):

where we use AM-GM. Unfortunately, with the optimal α , this only leads to an additive regret √ dT ∑ t ϵ 2 t = √ dC sq , ∞ > √ dC ∞ , which does not meet our goal.

Bonus design in our work. To obtain the tighter √ dC ∞ = √ d ∑ t ϵ t bound, our idea is to construct a positive-definite matrix B t such that B t ⪰ Σ -1 τ for all τ ∈ [ t ] , and add bonus B t -B t -1 at round t . This way, the total negative regret on u becomes -α ∥ u ∥ 2 B T and the cancellation becomes

where we use B T ⪰ Σ -1 t for all t and AM-GM. With this, it suffices to find B t satisfying our condition B t ⪰ Σ -1 τ for τ ≤ t , and bound the overhead ∑ T t =1 ⟨ Σ t , B t -B t -1 ⟩ by ˜ O ( d ) .

It turns out that there exists a way to inductively construct B t so that B t ⪰ Σ -1 τ for all τ ≤ t and ∑ T t =1 ⟨ Σ t , B t -B t -1 ⟩ ≲ log det( B T ) = ˜ O ( d ) . This is by letting B t to be a minimal matrix such that B t ⪰ B t -1 and B t ⪰ Σ -1 t . By induction, this ensures B t ⪰ Σ -1 τ for all τ ≤ t . The function B t = BONUS ( B t -1 , Σ t ) is formally defined in Figure 1a. The geometric interpretation is finding the minimal ellipsoid that contains both ellipsoids induced by B t -1 and Σ -1 t . An illustration figure is given in Figure 1b.

We adopt the fixed-point formulation in Zimmert and Lattimore (2022) (see their FTRL-FB) that includes the bonus for round t (i.e., B t ) in the FTRL objective when calculating the policy at round t (Eq. (2)). Notice that B t , in turn, depends on the policy at round t (Eq. (5), where Σ t depends on p t ), and thus this forms a fixed-point problem. In the regret analysis, this avoids the 'stability term' of the bonus to appear in the regret bound. While the fixed-point solution always exists, it may not be computationally efficient to find. For completeness, in Algorithm 4 (Appendix F), we present a version that does not require solving fixed point but has a suboptimal d √ log TC ∞ additive regret. The guarantee of Algorithm 2 is stated in Theorem 5.1, with its proof deferred to Appendix F.

Theorem 5.1. Algorithm 2 ensures with probability of 1 -δ , Reg T = ˜ O ( d √ T + √ dC ∞ ) , where ˜ O ( · ) hides log( T/δ ) factors.

5.2 C bound in Adversarial Linear Bandits

To see how to obtain a C bound, we perform the bias analysis again. Similar but slightly different from Eq. (6), with the standard loss estimator, the bias on action u 's reward is bounded by

Unlike in Eq. (6), we do not relax | ϵ t ( a t ) | to ϵ t = max a | ϵ t ( a ) | because we want the final bound to depend on C = ∑ t | ϵ t ( a t ) | . The idea to ensure that the sum of Eq. (9) over t can be related to C is to make ∥ a t ∥ Σ -1 t bounded by a constant poly( d ) , which allows us to further bound Eq. (9) by poly( d ) ∥ u ∥ Σ -1 t | ϵ t ( a t ) | . Such a property holds in standard linear bandit algorithms that operate in the continuous action space where a t is a point in the convex hull of A , and utilize a more concentrated action sampling scheme. Algorithms that are of this type include SCRiBLe (Abernethy et al., 2008) and continuous exponential weights (CEW) (Ito et al., 2020).

For SCRiBLe and CEW, the work by Lee et al. (2020) and Zimmert and Lattimore (2022) developed techniques that incorporate bonus terms to get high probability regret bounds. The bonus terms introduced by Zimmert and Lattimore (2022) is similar to that discussed in Eq. (7), which only allows us to get a C sq bound. The bonus terms introduced by Lee et al. (2020) allows us to obtain a C bound, but the overhead introduced by the bonus terms is much larger, resulting in a highly sub-optimal regret bound. Indeed, as shown in Appendix J, adopting their bonus construction results in an additional regret of d 5 2 C . With several attempts, we are only able to obtain the tight corruption dependency dC using the bonus in Section 5.1. To use that bonus, however, it is necessary to lift the problem to ( d +1) 2 -dimensional space. Unfortunately, existing SCRiBLe and CEW algorithms only operate in the original d -dimensional space, and as discussed above, we need them to ensure ∥ a t ∥ Σ -1 t ≤ poly( d ) .

In order to combine these two useful ideas (i.e., our bonus design in Section 5.1, and the concentrated sampling scheme by SCRiBLe or CEW), we end up with the algorithm that runs CEW over the lifted action space (Algorithm 3). In order to simplify the exposition, we assume without loss of generality

Algorithm 3: Continuous exponential weights (for adversarial C bound)

1 Parameters : γ = 1 /T , α = √ dT + C , β = 4log(10 dT ) , η = √ d/T .

2 for t = 1 , 2 , . . . , T do

3 Solve the fixed-point problem Eq. (10)-Eq. (13).

4

5

6

7

Let q t ∈ ∆( H ) and p t ∈ ∆( A ) be the distributions of H ∈ H and a ∈ A , respectively, generated by the following ( Z is a d × ( d +1) matrix):

Sample a t ∼ ˜ p t , and observe reward r t with E [ r t ] = a ⊤ t θ t + ϵ t ( a t ) .

Construct reward estimator ̂ θ t = ˜ Σ -1 t a t r t .

that A = conv ( A ) . The lifted action space is H = { ̂ Cov ( p ) : p ∈ ∆( A ) } ⊂ R ( d +1) × ( d +1) . The price of the lifting is that the 'regularization penalty term' in the regret analysis now grows from ˜ O ( d/η ) to ˜ O ( d 2 /η ) , which gives us the √ d 3 T sub-optimal regret.

Note that CEW requires the assumption that the feasible set is a convex body with non-zero volume, but the effective dimension of H is strictly smaller than ( d +1) 2 and thus have zero volume in R ( d +1) 2 . To correctly write the algorithm, we introduce an invertible linear transformation ϕ : R ( d +1) 2 → R m that maps an ( d +1) 2 -dimensional action set H to an m -dimensional one, where m is the effective dimension of H . In Appendix I, we formally define this ϕ . The algorithm uses ϕ to map all lifted actions and reward estimators from R ( d +1) × ( d +1) to R m .

The exponential weights runs over the space of ϕ ( H ) (see Eq. (10)). A point h ∈ ϕ ( H ) sampled from the exponential weights can be linearly mapped to an action a ∈ A according to Eq. (11). We use q ′ t to denote the exponential weight distribution in ϕ ( H ) , and use p t to denote the corresponding distribution in A . Instead of sampling a t from p t , we sample it through rejection sampling that rejects samples with ∥ a t ∥ Σ -1 t > ˜ Θ( √ d ) (Eq. (12)). This technique was developed by Ito et al. (2020), and this guarantees ∥ a t ∥ Σ -1 t ≤ ˜ O ( √ d ) -which is our goal as discussed in Eq. (9)-while keeping the clipped distribution ˜ p t close enough to the original distribution p t . This last property heavily relies on the log-concavity of the exponential weight distribution (Ito et al., 2020). The definition of the bonus term is similar to that in Algorithm 2 (Eq. (13)). The construction of the reward estimator (Line 7) and the way of lifting (Eq. (10)) are also similar to those in Algorithm 2. Again, we adopt the fixed-point formulation where the calculation of the policy at time t involves the bonus at time t , which, in turn, depends on the policy at time t . It is unlikely that this algorithm can be polynomial time. As a remedy, we provide a polynomial time algorithm (Algorithm 6) in Appendix J with a much worse regret bound of ˜ O ( d 3 √ T + d 5 / 2 C ) . The regret guarantee of Algorithm 3 is given in the following theorem.

Theorem 5.2. Algorithm 3 ensures with probability at least 1 -δ , Reg T = ˜ O ( √ d 3 T + dC ) , where ˜ O ( · ) hides polylog ( T/δ ) factors.

6 Gap-Dependent Misspecification

Intimately related to corrupted settings are misspecified settings, settings where our model class is unable to capture the true environment we are working with. For example, we might consider a stochastic linear bandit problem where the underlying reward function f ( · ) is nearly linear, i.e., there exists some θ and ϵ mis ( · ) such that | f ( a ) -a ⊤ θ | ≤ ϵ mis ( a ) for each a . Indeed, in such settings, playing on our true (nearly linear) environment is equivalent to playing on the environment with reward mean a ⊤ θ and with corruption ϵ mis ( a ) at each step. Thus, if we can solve corruption settings, it stands to reason that we can solve misspecified settings.

Here we are particularly interested in obtaining bounds on misspecified decision-making that scale precisely with action-dependent misspecification, ϵ mis ( a ) . While it is relatively straightforward to obtain bounds on learning in misspecified settings for a uniform level of misspecification ϵ ≥ max a ∈A ϵ mis ( a ) , obtaining bounds on learning with action-dependent misspecification have proved more elusive. To formalize this, we consider, in particular, the following gap-dependent notion of misspecification defined in Liu et al. (2023a).

Assumption 1 (Gap-Dependent Misspecification (Liu et al., 2023a)) . There exists some θ ∈ R d such that some ρ > 0 , denoting ∆( a ) = max a ′ f ( a ′ ) -f ( a ) , we have for any a ∈ A ,

We let M ⋆ denote the original environment with reward function f ( a ) (with Reg M ⋆ T the corresponding regret), and M 0 the environment with linear reward, a ⊤ θ , (with Reg M 0 T the corresponding regret).

Assumption 1 allows the reward to be misspecified, but the misspecification level for an action scales with how suboptimal that action is. This could correspond to real-world settings where, for example, significant attention has been given to modeling near-optimal behavior, such that it is accurately represented within our model class, but much less attention has been given to modeling suboptimal behavior. We assume access to a generic corruption-robust algorithm.

Assumption 2. We have access to a regret minimization algorithm which takes as input some C ′ and with probability at least 1 -δ has regret bounded on M 0 as

if C ′ ≥ C ≜ ∑ T t =1 ϵ mis ( a t ) , and by T otherwise, for C as defined above and for (problem-dependent) constants C 1 ( δ, T ) , C 2 ( δ, T ) which may scale at most logarithmically with T and 1 δ .

Assumption 2 is essentially the guarantee of a corruption-robust algorithm in terms of strong corruption measure (defined in Section 3). Note, in particular, that Assumption 2 only needs to obtain a sub-linear regret guarantee in the known-corruption setting, and can have linear regret in the setting where the corruption level is unknown. We then have the following result.

Theorem 6.1. Assume our environment satisfies Assumption 1 and that we have access to a corruptionrobust algorithm satisfying Assumption 2. Then as long as ρ ≤ min { 1 2 , 1 4 C 2 ( δ T , T ) -1 } , with probability at least 1 -2 δ we can achieve regret bounded as:

Theorem 6.1 states that, assuming our environment exhibits gap-dependent misspecification with tolerance ρ ≤ min { 1 2 , 1 4 C 2 ( δ T , T ) -1 } , then we can achieve regret on the true environment bounded as the leading-order term of our corruption-robust oracle, C 1 ( δ T , T ) √ T , with additional overhead of only ˜ O ( √ T ) . This reduction is almost entirely black-box: it requires knowledge of C 1 ( δ, T ) and C 2 ( δ, T ) , but does not require knowledge of ρ or any other facts about the corruption-robust algorithm.

Remark 1 (Anytime Algorithm) . The oracle of Assumption 5 must be anytime , achieving the above regret guarantee for any T not given as an input. Though many existing corruption-robust algorithms take T as input, the standard doubling trick can convert them into an anytime algorithm.

6.1 Optimal Misspecification Rate for Linear Bandits

We are particularly interested in how stringent a condition on the misspecification level-how small a value of ρ -Theorem 6.1 requires. As we have shown, Theorem 4.1 obtains the optimal misspecification level of dC . We then have the following corollary.

Corollary 6.1.1. Assume our environment is a misspecified linear bandit satisfying Assumption 1 with ρ ≤ O ( 1 d log T ) . Then instantiating Assumption 2 with the algorithm of Theorem 4.1, we can achieve regret bounded with probability 1 -δ as Reg M ⋆ T ≤ O ( d √ T log( T/δ )) .

While the regret bound of Corollary 6.1.1 achieves a scaling of ˜ O ( d √ T ) , which is tight for linear bandits (Lattimore and Szepesvári, 2020), it is unclear its requirement on ρ of ρ ≤ ˜ O ( 1 d ) is optimal. The result below shows that it is not optimal because ρ ≤ O ( 1 √ d ) suffices for ˜ O ( d √ T ) regret.

Theorem 6.2. Assume our environment is a misspecified linear bandit satisfying Assumption 1 with ρ ≤ O ( 1 √ d ) . Then there exists an algorithm that achieves, w.p. 1 -δ : Reg M ⋆ T ≤ O ( d √ T log( T/δ )) .

Theorem 6.2 relies on a specialized algorithm for the gap-dependent misspecification setting, and improves on the best-known bound for gap-dependent misspecification in linear bandits, which requires ρ ≤ ˜ O ( 1 d ) (Liu et al., 2023a). Moreover, for ρ > c T 1 √ d for some logarithmic term c T , adapting the lower-bound instance from Lattimore et al. (2020), we show that achieving sub-linear regret is not possible (Theorem K.2). These results jointly show that ρ ≈ 1 √ d is the best ρ we can hope for. This disproves the conjecture of Liu et al. (2023a) that ρ = Θ(1) is possible.

Note that the reduction in Theorem 6.1 is not able to achieve a tight ρ -while reducing from gapdependent misspecification to corruption allows for black-box usage of existing algorithms, it requires more stringent conditions on the misspecification level than specialized algorithms for this setting.

6.2 Gap-Dependent Misspecification in Reinforcement Learning

Theorem 6.1 is a corollary of a more general result, Theorem L.1, which applies to misspecified reinforcement learning, where we there assume a generalized notion of gap-dependent misspecification: for each policy π , E M ⋆ ,π [ ∑ H h =1 ϵ mis h ( s h , a h )] ≤ ρ · ( V ⋆ 0 -V π 0 ) , for V π 0 the expected reward of policy π , and ϵ mis h ( s, a ) a measure of the misspecification at step h , state s , and action a . To illustrate this general reduction, we consider the following setting, a generalization of linear MDPs (Jin et al., 2020). Assumption 3 (Gap-Dependent Misspecified Linear MDPs) . Let ϕ ( s, a ) : S × A → R d denote some feature map and µ h · S → R d some measure which satisfy ∥ ϕ ( s, a ) ∥ 2 ≤ 1 , ∀ s, a , and ∥ ∫ s | d µ h ( s ) |∥ 2 ≤ √ d . Assume that the transitions P h ( · | s, a ) on our true environment satisfy:

for some ϵ mis h ( s, a ) ≥ 0 and ∥ P -Q ∥ TV the total variation distance between P and Q . Furthermore, assume that for any policy π , we have E π [ ∑ H h =1 ϵ mis h ( s h , a h )] ≤ ρ · ( V ⋆ 0 -V π 0 ) .

We then have the following result.

Corollary 6.2.1. Assume our environment satisfies Assumption 3 with ρ ≤ ˜ O ( 1 dH ) . Then there exists an algorithm that achieves regret bounded with probability 1 -δ as Reg M ⋆ T ≤ ˜ O ( √ d 3 H 2 T ) .

To the best of our knowledge, Corollary 6.2.1 is the first result showing that it is possible to efficiently learn in linear MDPs with gap-dependent misspecification. Note that under Assumption 3, our MDP could be far from a linear MDP-we simply assume that if we play a 'good' policy, it appears as approximately linear. This result is almost immediate by instantiating our reduction with a known corruption-robust algorithm for linear MDPs (Ye et al., 2023).

7 Open Problems

It remains open how to achieve d √ T + dC regret in corrupted adversarial linear bandits. The tight C ∞ bound for corrupted linear contextual bandits, where the action set can be chosen by an adaptive adversary in every round, also remains open. The best known upper and lower bounds for this setting are ˜ O ( d √ T + dC ∞ ) by He et al. (2022) and Ω( d √ T + √ dC ∞ ) by Lattimore and Szepesvári (2020).

With the AA viewpoint in Section 3, our work first shows the separation between the achievable regret under weak adversary and strong adversary in corrupted linear bandits. An interesting future direction is to investigate similar separation in general decision making (Foster et al., 2021).

References

Abernethy, J. D., Hazan, E., and Rakhlin, A. (2008). Competing in the dark: An efficient algorithm for bandit linear optimization. In COLT , pages 263-274. Citeseer.

Auer, P., Cesa-Bianchi, N., Freund, Y., and Schapire, R. E. (2002). The nonstochastic multiarmed bandit problem. SIAM journal on computing , 32(1):48-77.

Auer, P. and Chiang, C.-K. (2016). An algorithm with nearly optimal pseudo-regret for both stochastic and adversarial bandits. In Conference on Learning Theory .

Bogunovic, I., Krause, A., and Scarlett, J. (2020). Corruption-tolerant gaussian process bandit optimization. In International Conference on Artificial Intelligence and Statistics , pages 10711081. PMLR.

Bogunovic, I., Li, Z., Krause, A., and Scarlett, J. (2022). A robust phased elimination algorithm for corruption-tolerant gaussian process bandits. Advances in Neural Information Processing Systems , 35:23951-23964.

Bogunovic, I., Losalka, A., Krause, A., and Scarlett, J. (2021). Stochastic linear bandits robust to adversarial attacks. In International Conference on Artificial Intelligence and Statistics , pages 991-999. PMLR.

Bubeck, S., Cesa-Bianchi, N., and Kakade, S. M. (2012). Towards minimax policies for online linear optimization with bandit feedback. In Conference on Learning Theory , pages 41-1. JMLR Workshop and Conference Proceedings.

Bubeck, S. and Slivkins, A. (2012). The best of both worlds: Stochastic and adversarial bandits. In Conference on Learning Theory .

Chewi, S. (2023). The entropic barrier is n-self-concordant. In Geometric Aspects of Functional Analysis: Israel Seminar (GAFA) 2020-2022 , pages 209-222. Springer.

Dann, C., Lattimore, T., and Brunskill, E. (2017). Unifying pac and regret: Uniform pac bounds for episodic reinforcement learning. Advances in Neural Information Processing Systems , 30.

Dann, C., Wei, C.-Y., and Zimmert, J. (2023). A blackbox approach to best of both worlds in bandits and beyond. In The Thirty Sixth Annual Conference on Learning Theory , pages 5503-5570. PMLR.

Du, S. S., Kakade, S. M., Wang, R., and Yang, L. F. (2019). Is a good representation sufficient for sample efficient reinforcement learning? arXiv preprint arXiv:1910.03016 .

Foster, D. J., Gentile, C., Mohri, M., and Zimmert, J. (2020). Adapting to misspecification in contextual bandits. Advances in Neural Information Processing Systems , 33:11478-11489.

Foster, D. J., Kakade, S. M., Qian, J., and Rakhlin, A. (2021). The statistical complexity of interactive decision making. arXiv preprint arXiv:2112.13487 .

Hajiesmaili, M., Talebi, M. S., Lui, J., Wong, W. S., et al. (2020). Adversarial bandits with corruptions: Regret lower bound and no-regret algorithm. Advances in Neural Information Processing Systems , 33:19943-19952.

He, J., Zhou, D., Zhang, T., and Gu, Q. (2022). Nearly optimal algorithms for linear contextual bandits with adversarial corruptions. Advances in Neural Information Processing Systems , 35:3461434625.

Ito, S. (2021). Parameter-free multi-armed bandit algorithms with hybrid data-dependent regret bounds. In Conference on Learning Theory .

Ito, S., Hirahara, S., Soma, T., and Yoshida, Y . (2020). Tight first-and second-order regret bounds for adversarial linear bandits. Advances in Neural Information Processing Systems , 33:2028-2038.

Ito, S. and Takemura, K. (2023). Best-of-three-worlds linear bandit algorithm with variance-adaptive regret bounds. In The Thirty Sixth Annual Conference on Learning Theory , pages 2653-2677. PMLR.

Ito, S. and Takemura, K. (2024). An exploration-by-optimization approach to best of both worlds in linear bandits. Advances in Neural Information Processing Systems , 36.

Jiang, N., Krishnamurthy, A., Agarwal, A., Langford, J., and Schapire, R. E. (2017). Contextual decision processes with low bellman rank are pac-learnable. In International Conference on Machine Learning , pages 1704-1713. PMLR.

Jin, C., Yang, Z., Wang, Z., and Jordan, M. I. (2020). Provably efficient reinforcement learning with linear function approximation. In Conference on learning theory , pages 2137-2143. PMLR.

Jin, T., Liu, J., Rouyer, C., Chang, W., Wei, C.-Y ., and Luo, H. (2024). No-regret online reinforcement learning with adversarial losses and transitions. Advances in Neural Information Processing Systems , 36.

Kong, F., Zhao, C., and Li, S. (2023). Best-of-three-worlds analysis for linear bandits with follow-theregularized-leader algorithm. In The Thirty Sixth Annual Conference on Learning Theory , pages 657-673. PMLR.

Lattimore, T. and Szepesvári, C. (2020). Bandit algorithms . Cambridge University Press.

Lattimore, T., Szepesvari, C., and Weisz, G. (2020). Learning with good feature representations in bandits and in rl with a generative model. In International conference on machine learning , pages 5662-5670. PMLR.

Lee, C.-W., Luo, H., Wei, C.-Y., and Zhang, M. (2020). Bias no more: high-probability datadependent regret bounds for adversarial bandits and mdps. Advances in neural information processing systems , 33:15522-15533.

Lee, C.-W., Luo, H., Wei, C.-Y., Zhang, M., and Zhang, X. (2021). Achieving near instanceoptimality and minimax-optimality in stochastic and adversarial linear bandits simultaneously. In International Conference on Machine Learning , pages 6142-6151. PMLR.

Li, Y., Lou, E. Y., and Shan, L. (2019). Stochastic linear optimization with adversarial corruption. arXiv preprint arXiv:1909.02109 .

Li, Y. and Yang, L. (2024). On the model-misspecification in reinforcement learning. In International Conference on Artificial Intelligence and Statistics , pages 2764-2772. PMLR.

Liu, C., Yin, M., and Wang, Y.-X. (2023a). No-regret linear bandits beyond realizability. arXiv preprint arXiv:2302.13252 .

Liu, H., Wei, C.-Y., and Zimmert, J. (2023b). Towards optimal regret in adversarial linear mdps with bandit feedback. arXiv preprint arXiv:2310.11550 .

Liu, H., Wei, C.-Y., and Zimmert, J. (2024). Bypassing the simulator: Near-optimal adversarial linear contextual bandits. Advances in Neural Information Processing Systems , 36.

Nesterov, Y. and Nemirovskii, A. (1994). Interior-point polynomial algorithms in convex programming . SIAM.

Neu, G. and Olkhovskaya, J. (2020). Efficient and robust algorithms for adversarial linear contextual bandits. In Conference on Learning Theory , pages 3049-3068. PMLR.

Pukelsheim, F. (2006). Optimal design of experiments . SIAM.

Seldin, Y. and Lugosi, G. (2017). An improved parametrization and analysis of the exp3++ algorithm for stochastic and adversarial bandits. In Conference on Learning Theory .

Seldin, Y. and Slivkins, A. (2014). One practical algorithm for both stochastic and adversarial bandits. In International Conference on Machine Learning .

Takemura, K., Ito, S., Hatano, D., Sumita, H., Fukunaga, T., Kakimura, N., and Kawarabayashi, K.-i. (2021). A parameter-free algorithm for misspecified linear contextual bandits. In International Conference on Artificial Intelligence and Statistics , pages 3367-3375. PMLR.

Wang, R., Salakhutdinov, R. R., and Yang, L. (2020). Reinforcement learning with general value function approximation: Provably efficient approach via bounded eluder dimension. Advances in Neural Information Processing Systems , 33:6123-6135.

Wei, C.-Y., Dann, C., and Zimmert, J. (2022). A model selection approach for corruption robust reinforcement learning. In International Conference on Algorithmic Learning Theory , pages 1043-1096. PMLR.

Wei, C.-Y. and Luo, H. (2018). More adaptive algorithms for adversarial bandits. In Conference On Learning Theory .

Ye, C., Xiong, W., Gu, Q., and Zhang, T. (2023). Corruption-robust algorithms with uncertainty weighting for nonlinear contextual bandits and markov decision processes. In International Conference on Machine Learning , pages 39834-39863. PMLR.

Zanette, A., Lazaric, A., Kochenderfer, M., and Brunskill, E. (2020). Learning near optimal policies with low inherent bellman error. In International Conference on Machine Learning , pages 1097810989. PMLR.

Zhang, W., He, J., Fan, Z., and Gu, Q. (2023). On the interplay between misspecification and sub-optimality gap in linear contextual bandits. In International Conference on Machine Learning , pages 41111-41132. PMLR.

Zimmert, J. and Lattimore, T. (2022). Return of the bias: Almost minimax optimal high probability bounds for adversarial linear bandits. In Conference on Learning Theory , pages 3285-3312. PMLR.

Zimmert, J., Luo, H., and Wei, C.-Y. (2019). Beating stochastic and adversarial semi-bandits optimally and simultaneously. In International Conference on Machine Learning .

Zimmert, J. and Seldin, Y. (2019). An optimal algorithm for stochastic and adversarial bandits. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 467-475. PMLR.

Appendices

A Related Work

Model Misspecification Theoretical works on bandits or RL often assume that the underlying world is well-specified by a particular model. Algorithms that are purely built on this assumption are vulnerable to potential misspecifications. Therefore, some works, besides proposing the main results, also discuss the case where the model is misspecified, such as Jiang et al. (2017); Jin et al. (2020); Zanette et al. (2020); Wang et al. (2020); Li and Yang (2024). These discussions, however, usually assume that the amount of misspecification has a uniform upper bound for all actions / states / policies, and the performance degradation is proportional to this uniform upper bound.

For settings like stochastic linear bandits and stochastic linear contextual bandits, it was also found that some widely used algorithm such as LinUCB cannot achieve the tightest guarantee under misspecification (Du et al., 2019). Therefore, a line of work developed better algorithms that have optimal robustness against misspecification, such as Lattimore et al. (2020); Foster et al. (2020); Takemura et al. (2021).

While most work focus on the stochastic setting, Neu and Olkhovskaya (2020) took a first step in studying misspecification in linear contextual bandits with stochastic contexts and adversarial rewards. They established near-optimal regret dependencies on the amount of misspecification.

Gap-dependent Misspecification Gap-dependent misspecification is a setting where the amount of misspecification for an action is bounded by a constant times that action's sub-optimality gap. To our knowledge, this setting is first studied by Liu et al. (2023a) for linear bandits. Another related work is Zhang et al. (2023), which assumes that the misspecification is bounded by a constant times the minimal sub-optimality gap among all actions. Although this assumption is more restrictive, they handle the more general linear contextual bandit setting, and derive instance-dependent logarithmic regret bounds.

Corruption-robust Bandits The guarantees on model misspecification is rather pessimistic in the sense that if the misspecification is time-varying, and large misspecification only appears in a few rounds, then the existing guarantees for misspeicifcation still scale with the largest misspeicification. To refine such guarantee, previous works have consider different notions of time-varying corruption, and established more fine-grained regret guarantees. These include C sq , ∞ , C sq , C ∞ , and C discussed in Section 3. Among them, C sq , ∞ and C ∞ are usually studied under the 'weak adversary' framework where the adversary decides the corruption before seeing the action chosen by the learner. On the other hand, C sq and C are usually studied under the 'strong adversary' framework where the adversary decides the corruption after seeing the action chosen the learner. In Section 3, we provide a unified view for them so that they can both be regarded as weak adversarial setting but with different corruption measure.

The algorithms of Foster et al. (2020) and Takemura et al. (2021) achieved the optimal bound with respect to C sq for stochastic linear contextual bandits (i.e., d √ T + √ dC sq ), and He et al. (2022) showed the optimal bound with respect to C (i.e., d √ T + dC ). However, it is still unclear whether the tight dependency on C ∞ is √ dC ∞ or dC ∞ . In this paper, we answer it for the context-free linear bandit setting, showing that d √ T + √ dC ∞ is achievable. However, the question remains open for linear contextual bandits.

For the adversarial setting, Liu et al. (2024) showed d 2 √ T + √ dC sq , ∞ bound for linear contextual bandits with stochastic contexts and adversarial rewards, which can be improved to d √ T + √ dC sq , ∞ when specialized to adversarial linear bandits. To our knowledge, no C ∞ or C bound has been shown for adversarial linear bandits, and our work make the first attempts on them.

√

We remark that for A -armed adversarial bandits, it is easy to see that AT + C ∞ bound is achievable simply by running standard adversarial multi-armed bandit algorithm that handles adaptive adversary (e.g., EXP3.P by Auer et al. (2002)). The work of Hajiesmaili et al. (2020) is the only one that we know to obtain C bound for adversarial bandits. They showed a √ AT + AC bound for A -armed bandits, which is tight.

Best-of-both-worlds Bounds The study of the best-of-both-world problem was initiated by Bubeck and Slivkins (2012) and extended by Seldin and Slivkins (2014); Auer and Chiang (2016); Seldin

and Lugosi (2017); Wei and Luo (2018); Zimmert and Seldin (2019); Zimmert et al. (2019); Ito (2021); Ito and Takemura (2023, 2024); Dann et al. (2023); Kong et al. (2023). The goal of this line of work is to have a single algorithm that achieves a ˜ O ( √ T ) regret when the reward is adversarial and O (log T ) when the reward is stochastic, without knowing the type of reward in advance. These results should be viewed as refinements of the standard adversarial setting but not the corruption setting considered in our work, though they also used the term 'corruption' in their work.

For example, Lee et al. (2021); Ito and Takemura (2024, 2023, 2024); Dann et al. (2023); Kong et al. (2023) studied the best-of-both-world linear bandits problem. The underlying world could be stochastic ( θ t = θ ⋆ for all t ) or adversarial ( θ t 's are arbitrary). Their algorithm achieves a bound of O ( d 2 log( T ) / ∆) in the former case, where ∆ is the reward gap between the best and the second-best arm, and ˜ O ( d √ T ) in the latter phase. They also define the corruption C ′ = ∑ t max a | a ⊤ ( θ t -θ ⋆ ) | and show that their algorithm achieves a regret of O ( d 2 log( T ) / ∆+ √ d 2 log( T ) C ′ / ∆) . Compared to our setting, their corruption is in a more limited form, but their target regret bound in the stochastic setting is tighter than ours.

B Equivalence Between AA and CM Viewpoints for Strong Corruption

We show that strong corruption in both definitions is equivalent, that is, for any adversary having strong corruption C = ∑ t | ϵ t | from AA viewpoint, there exists an adversary using the equal amount of strong corruption ∑ t | ϵ ′ t ( a t ) | from CM viewpoint, where | ϵ t | = | ϵ ′ t ( a t ) | for all t , and vice versa.

Assume that ϵ ( H t -1 , a t ) is the function used by an AA strong adversary to decide the corruption at time t , where H t -1 is the history up to time t -1 and a t is the chosen action at time t . Then we define ϵ ′ t ( a ) ≜ ϵ ( H t -1 , a ) , ∀ a for the CM viewpoint, thus | ϵ t | = | ϵ ( H t -1 , a t ) | = | ϵ ′ t ( a t ) | . Note that the function ϵ ′ t ( · ) only depends on the history up to time t -1 , so the definition of ϵ ′ t is known to adversary before observing a t . The other direction of this equivalence is achieved by setting the corruption in AA viewpoint as ϵ t = ϵ ′ t ( a t ) . Note that since a t is known to a strong adversary in AA viewpoint, ϵ t is also known.

C The Case of Unknown C ∞ or C

In the corrupted stochastic setting, Wei et al. (2022) developed a black-box reduction that can turn any algorithm achieving β 1 √ T + β 2 + β 3 C ∞ regret with the knowledge of C ∞ into an algorithm achieving log( T ) × ( β 1 √ T + β 2 + β 3 C ∞ ) regret without knowledge of C ∞ . This reduction can be directly applied to our stochastic C ∞ bound result (Theorem 4.1), which allows us to achieve almost the same regret bound without knowledge of C ∞ . The idea of Wei et al. (2022) has been extended to the adversarial setting by Jin et al. (2024) (see their Section 4). Similarly, for the adversarial setting, one can turn any algorithm achieving β 1 √ T + β 2 + β 3 C ∞ regret with known C ∞ into one achieving log( T ) × ( β 1 √ T + β 2 + β 3 C ∞ ) regret without knowing C ∞ . This can be directly applied to our adversarial C ∞ result (Theorem 5.1).

The case of unknown C is quite different. It has been proven by Bogunovic et al. (2021) that it is impossible to achieve a bound that has linear scaling in C (e.g., β 1 √ T + β 2 + β 3 C ) for all C simultaneously if C is not known by the learner. This is also mentioned in He et al. (2022) again. Hence, almost all previous work studying C bound assumes knowledge on C . If C is unknown, simply setting ¯ C = √ T as an upper bound of C yields a bound of O ( √ T + C 2 ) -if C ≤ √ T indeed holds, then ¯ C is a correct upper bound, so the regret can be bounded by O ( √ T + ¯ C ) = O ( √ T ) ; if C > √ T , then simply bound the regret by T ≤ O ( C 2 ) .

D Proof of Proposition 1

First, we argue that there exists a deterministic algorithm achieving ˜ O ( d √ T + √ dC sq ) upper bound. The algorithm of Takemura et al. (2021) is such an algorithm, although they only showed an upper bound of ˜ O ( d √ T + √ dC ms ) . To argue the stronger ˜ O ( d √ T + √ dC sq ) bound, we only need to slightly modify their analysis: In their proof of Lemma 2 (in their Page 6), the original proof bound

the per-step regret due to the misspecification as the following (the calculation below uses their original notation):

We can tighten their analysis by doing the following:

Since the regret for every step in Ψ t,s can be bounded by this value, when summing the regret over Ψ T +1 ,s , one can get a regret of order

Further summing this over s (there are logarithmically many different s ) and using that [ T ] = ⋃ s Ψ T +1 ,s and using Cauchy-Schwarz, we get a √ dT ∑ T t =1 ϵ t ( i t ) 2 = √ dC sq bound.

̸

To argue that any deterministic algorithm must suffer at least Ω( d √ T + dC ∞ ) regret, we only need to use the lower bound instance of Ω( d √ T + dC ) . At the beginning of round t , the adversary simply change the corruptions ϵ t ( a ) to be zero for all a = a t (the adversary knows what a t since the algorithm is deterministic). This makes C = C ∞ , and thus the lower bound Ω( d √ T + dC ∞ ) holds.

E Proof of Theorem 4.1

Lemma E.1. With probability at least 1 -2 δ , for all k and for all b ∈ A k ,

Proof. Let E t [ · ] be the expectation conditioned on the history up to round t -1 . We fix k and b and consider

for t ∈ I k . Notice that ∑ t ∈I k X t = b ⊤ ̂ θ k

Also, by the definition of G k , we have | X t | = ∣ ∣ b ⊤ ( m k G k ) -1 a t r t ∣ ∣ ≤ 1 m k ∥ b ∥ G -1 k ∥ a t ∥ G -1 k ≤ d m k . Thus, by Freedman's inequality, with probability at least 1 -δ |A| T , the following holds:

We bound term 1 by

and term 2 by or

Thus, for any b ∈ A k , with probability at least 1 -δ |A| T ,

Taking a union bound over k and b ∈ A k finishes the proof.

Lemma E.2. Let a ⋆ = argmax a ∈A a ⊤ θ ⋆ . Then with probability at least 1 -2 δ , a ⋆ ∈ A k for all k .

Proof. Suppose that the high-probability event in Lemma E.1 holds. For any k , if a ⋆ ∈ A k , then for any b ∈ A k ,

By the definition of A k +1 in Eq. (1), we have a ⋆ ∈ A k +1 . The lemma is then proven by an induction argument.

Proof of Theorem 4.1. We first calculate the regret in epoch k > 1 assuming that the event in Lemma E.2 holds.

Summing this over k and using that m 1 = d log( |A| T/δ ) , we get

Notice that without loss of generality we can assume d log( |A| T/δ ) ≤ T (otherwise the right-hand side is vacuous). Using this fact gives the desired bound.

From Exercise 27.6 in Lattimore and Szepesvári (2020), the ϵ -covering number of A is bounded by ( 6 d ϵ ) d . Let C ( A , ϵ ) be the ϵ -net of A , we then have ∣ ∣ C ( A , 6 d T ) ∣ ∣ ≤ T d . Thus, when |A| ≥ T d , we can use C ( A , 6 d T ) as A 1 in Algorithm 1 to conduct phase elimination. In that case, following above proof, we have

From the definition of covering number, there exists a a ⋆ c ∈ C ( A , 6 d T ) such that

We have

Thus,

F Proof of Theorem 5.1

In this section, we use the following notation:

Algorithm 2 is equivalent to the FTRL update:

Algorithm 4 is equivalent to

By the standard analysis for FTRL algorithms (e.g., Theorem 2 in Zimmert and Lattimore (2022)), the regret bounds of Eq. (14) and Eq. (15) are given by the following lemmas, respectively.

Lemma F.1. The update rule Eq. (14) (Algorithm 2) ensures for any U ∈ H ,

Lemma F.2. The update rule Eq. (15) (Algorithm 4) ensures for any U ∈ H ,

We consider an arbitrary comparator p ⋆ ∈ ∆( A ) with u ⋆ = E a ∼ p ⋆ [ a ] . Define p = (1 -γ ) p ⋆ + γρ . We have p ∈ ∆ γ ( A ) , and define u = E a ∼ p [ a ] and U = ̂ Cov ( p ) . The regret with respect to p ⋆ can be

decomposed as the following: With probability at least 1 -δ ,

By Lemma F.1, the FTRL term can further be bounded by

In the following five lemmas, we bound the five terms Bias , Deviation , Penalty , Bonus , and Stability . Lemma F.3.

Proof.

Thus,

Lemma F.4. With probability of at least 1 -δ , we have

Proof. Notice that

By the strengthened Freedman's inequality (Lemma M.3), with probability at least 1 -δ ,

Lemma F.5.

Proof. Define H 0 = E a ∼ ρ [ aa ⊤ a a ⊤ 1 ] . By the definition of the feasible set H , for any H ∈ H , H ⪰ γH 0 = d +1 T H 0 and H ⪯ ( d +1) H 0 . Thus, Penalty can be upper bounded by

Lemma F.6.

Proof. Given E a ∼ p 0 [ aa ⊤ ] ⪰ E a ∼ p 0 [ a ] E a ∼ p 0 [ a ] ⊤ = uu ⊤ , we have

Recall that B 1 = Σ -1 1 and for t ≥ 2 ,

which ensures B t ⪰ B t -1 and B t ⪰ Σ -1 t . By induction, it leads to B T ⪰ Σ -1 t for any t . This implies

Thus, ∑ T t =1 ⟨ U,D t ⟩ ≥ α max t ∥ u ∥ 2 Σ -1 t .

Next, we upper bound ∑ T t =1 ⟨ H t , D t ⟩ . First, notice that ⟨ H 1 , D 1 ⟩ = α Tr (Σ t B t ) = Tr ( I ) = αd . For t ≥ 2 ,

We also have

Thus,

Finally, we bound log det ( B T ) . Since Σ t = ∑ a p t ( a ) aa ⊤ ⪰ γ ∑ a ρ ( a ) aa ⊤ , by Theorem 3 of Bubeck et al. (2012), we have Σ t ⪰ γ d I and Σ -1 t ⪯ d γ I for all t . Thus, B 1 = Σ -1 1 ⪯ d γ I . Below, we use induction to show that B t ⪯ td γ I . Assume B t -1 ⪯ ( t -1) d γ I . Then,

By induction, we get B T ⪯ Td γ I and log det ( B T ) ≤ 2 d log( T ) by setting γ = d √ T . Overall, by Eq. (20), we have

Combining the upper bound for ∑ T t =1 ⟨ H t , D t ⟩ and the lower bound for ∑ T t =1 ⟨ U,D t ⟩ finishes the proof.

Lemma F.7. With probability at least 1 -δ

Proof. For any p , define µ ( p ) = E a ∼ p [ a ] and

By Freedman's inequality, since E t [ ∥ a t ∥ 2 Σ -1 t ] = d , and η ∥ a t ∥ 2 Σ -1 t ≤ ηd γ , with probability at least 1 -δ , we have

Proof of Theorem 5.1. Using Lemma F.3-Lemma F.7 in Eq. (17) and Eq. (18), we get Reg T

G Computationally Efficient Algorithm for Adversarial C ∞ Bound

Most proof is the same as Appendix F. Namely, we follow Eq. (17) in Appendix F together with a different decomposition

By Lemma F.2, we can further bound FTRL by

Among the terms above, Bias , Penalty , Deviation , Bonus , and Stability-1 follow the same bounds as Lemma F.3, Lemma F.5, Lemma F.6, and Lemma F.7, respectively. It remains to bound Stability-2 .

Proof. From the analysis of bias term and H t and D t are both positive semi-definite, we have

where the last inequality is due to Σ -1 t ⪰ B t -B t -1 . Since η ≤ 1 16 √ dα , by Lemma M.2, with probability of at least 1 -δ , we have

where the last step follows the similar analysis in Lemma F.6.

Proof of Theorem 5.1 (Option II). Using Lemma F.3-Lemma G.1 in Eq. (21) and Eq. (22), we get

By choosing α = max { C ∞ √ d log( T ) , √ T } and η = min { √ log( T ) 16 C ∞ , √ log( T ) T } , and γ = d √ T , we could ensure η ≤ 1 16 √ dα . This gives the final regret O ( d √ T log( T/δ ) + dC ∞ √ log T ) . The additional √ d factor comes from the additional condition for the Stability-2 term.

H Proof of Theorem 5.2

Similar to before, we define

and x t = E a ∼ p t [ a ] , ˜ x t = E a ∼ ˜ p t [ a ] . We perform the regret decomposition as the following.

The FTRL term can be further bounded as the following.

FTRL

In the following four lemmas, we bound the four terms Bias , Deviation , Bonus , Stability . Lemma H.1.

Proof.

Lemma H.2.

Proof. Notice that

By the strengthened Freedman's inequality (Lemma M.3), with probability at least 1 -δ ,

Lemma H.3.

Proof. The proof the same as in the logdet case. See the proof of Lemma F.6.

Lemma H.4.

Proof.

Since q ′ t is a log-concave distribution, so are q t and p t , which further implies that η 〈 a, ̂ θ t 〉 follows a log-concave distribution. Furthermore,

where we use Lemma J.2 in the second-last inequality . By Lemma 6 of Ito et al. (2020), we have

Proof of Theorem 5.2. Combining Eq. (23), Eq. (24), and Lemma H.1, Lemma H.2, Lemma H.3, Lemma H.4, we see that the regret is bounded by

Choosing optimal α and η leads to ˜ O ( d 3 T + dC ) .

I Dimension Reduction for Continuous Exponential Weights

First, the intrinsic dimension of X can be defined as the following:

Definition 1. The intrinsic dimension of X is defined as

A convex region X ⊂ R n can be translated and rotated so that it entirely lies in R m where m = dim( X ) and has non-zero volume in R m . We more precisely define this transformation below.

Definition 2. Let X ⊂ R n be a convex region with dim( X ) = m . We define ϕ : R n → R m as the following linear transformation:

where M ∈ R n × n is a rotation matrix (i.e., orthogonal matrix) such that for any v ∈ X - X , Mv has non-zero elements only in the first m coordinates (this is always possible by the definition of dim( X ) in Definition 1), and

extracts the first m coordinates of a given n -dimensional vector.

Lemma I.1. For any x ∈ X and any θ ∈ R n ,

where f ( ϕ, θ ) ∈ R is some quantity that only depends on ϕ and θ but not x .

Proof. Let x, x ′ ∈ X . By the definition of ϕ , we have

By the choice of M in Definition 2, M ( x -x ′ ) only has non-zero elements in the first m coordinates. Furthermore, since Z extracts the first m coordinates, we have

Thus,

meaning that the value of ⟨ x, θ ⟩ - ⟨ ϕ ( x ) , ϕ ( θ ) ⟩ is shared by all x ∈ X . Defining this value as f ( ϕ, θ ) finishes the proof.

We consider the continuous exponential weight algorithm (Algorithm 5) running on ϕ ( X ) ⊂ R m :

Algorithm 5: Exponential Weights

̸

In Algorithm 5, we require that the inverse mapping of ϕ exists. This is true because for any x, x ′ ∈ X , we have ∥ ϕ ( x ) -ϕ ( x ′ ) ∥ = ∥ ZM ( x -x ′ ) ∥ = ∥ M ( x -x ′ ) ∥ = ∥ x -x ′ ∥ , and thus ϕ cannot map x, x ′ ∈ X with x = x ′ to the same point.

Theorem I.2. Let q t ∈ ∆( X ) be the distribution such that x ∼ q t is equivalent to first drawing y ∼ p t and then taking x = ϕ -1 ( y ) . Algorithm 5 ensures for any x ∈ X ,

Proof. Note that Algorithm 5 is a standard continuous exponential weight algorithm over reward vectors ϕ ( θ t ) and in the space of ϕ ( X ) ⊂ R m . By the standard analysis (see, e.g., Ito et al. (2020); Zimmert and Lattimore (2022)), we have for any sequence λ 1 , . . . , λ T ∈ R and any y ∈ ϕ ( X ) ,

By Lemma I.1, the above implies

which further implies that for any x ∈ X ,

t =1 t =1 t =1 by the definition of q t and by letting λ t = ηf ( ϕ, θ t ) .

J Computationally Efficient Algorithm for Adversarial C Bound

In this section, we present Algorithm 6, a polynomial-time algorithm that ensures ˜ O ( d 3 √ T + d 5 2 C ) regret. The algorithm is based on the continuous exponential weight algorithm in the original feature space (Ito et al., 2020; Zimmert and Lattimore, 2022), with the bonus construction similar to Lee et al. (2020).

J.1 Preliminaries for Entropic Barrier

Entropic barrier For any convex body A , the family of exponential distribution is

For any x ∈ A , there is a unique w ( x ) such that E y ∼ p w ( x ) [ y ] = x . The entropic barrier F ( x ) is the negative entropy of p w ( x ) . Namely

We have ∇ F ( x ) = w ( x ) and ∇ 2 F ( x ) = E y ∼ p w ( x ) [ ( y -x )( y -x ) ⊤ ] . We know that F ( x ) is a d -self-concordant barrier on A .

The equivalence of mean-oriented FTRL and continuous exponential weights Consider FTRL with entropic barrier as the regularizer that solves x t for round t ∈ [ T ] following

Given that E y ∼ p w ( x t +1 ) [ y ] = x t +1 and ∇ F ( x t +1 ) = w ( x t +1 ) , playing x t +1 yields the same expected reward as playing according to distribution p w ( x t +1 ) where w ( x t +1 ) = η t ∑ t s =1 θ s . Thus, we have p w ( x t +1 ) ( x ) ∝ exp ( η t 〈 x, ∑ t s =1 θ s 〉) for x ∈ A .

J.2 Auxiliary Lemmas

Lemma J.1 (Lemma 1 of Ito et al. (2020)) . If x follows a log-concave distribution p over R d and E x ∼ p [ xx ⊤ ] ⪯ I , we have

This is equivalent to for arbitrary β > 0 .

Lemma J.2. With the choice of β ≥ 4 log(10 dT ) , we have

for any f : A → [ -1 , 1] and

Proof. The proof follows that of Lemma 4 of Ito et al. (2020), with the observation that p t is a log-concave distribution.

Lemma J.3 (Lemma 14 of Zimmert and Lattimore (2022)) . Let f be a ν -self-concordant barrier for A ⊂ R d . Then for any u, x ∈ A ,

where γ ′ = 8 3 √ 3 + 7 3 2 6 √ 3 ν ( γ ′ ∈ [1 , 4] for ν ≥ 1 ).

Minkowsky Functions. The Minkowsky function of a convex boday A with the pole at w ∈ int ( A ) is a function π w : A → R defined as

Algorithm 6: Continuous exponential weights (for adversarial C bound)

Lemma J.4 (Proposition 2.3.2 in Nesterov and Nemirovskii (1994)) . Let f be a ν -self-concordant barrier on A ⊆ R d , and u, w ∈ int ( A ) . Then

J.3 Regret Analysis

We perform regret decomposition. For regret comparator u ⋆ ∈ A , define x ⋆ = min x ∈A F ( x ) and u = (1 -1 T ) u ⋆ + 1 T x ⋆ . With probability at least 1 -δ ,

By standard FTRL analysis, we have

The individual terms Bias , Deviation , Bonus , Penalty , Stability terms are bounded in Lemma J.6, Lemma J.7, Lemma J.9, Lemma J.10, Lemma J.12.

Lemma J.5. For any t ∈ [ T ] , if a ∼ p t , then with probability of at least 1 -δ ,

Proof. Define y = Σ -1 2 t a . Then E y [ yy ⊤ ] = Σ -1 2 t E a ∼ p t [ aa ⊤ ]Σ -1 2 t = I . Since p t is a logconcave distribution, and log-concavity is preserved under liner transformation, y is also log-concave. Applying Lemma J.1 on it leads to

Setting δ = 3 d exp( -β ) , we conclude that with probability at least 1 -δ , ∥ a ∥ 2 Σ -1 t ≤ d log ( 3 d δ ) 2 .

Lemma J.6. With probability at least 1 -O ( δ ) ,

Proof. The proof is the same as that of Lemma H.1.

Proof. The proof is the same as that of Lemma H.2.

Lemma J.8.

̸

Proof. Our proof is similar to Lemma B.12 in Lee et al. (2020). Let { t 1 , · · · , t n +1 } be the rounds such that b t = 0 . Define A i = ∑ i j =1 B t j . For any i > 1 , since λ max ( B t i -A i -1 ) > 0 , there exists a vector y ∈ R d +1 such that y ⊤ B t i y > y ⊤ A i -1 y . Thus, y ⊤ A i y ≥ 2 y ⊤ A i -1 y . Let z = A 1 2 i -1 y , we have z ⊤ A -1 2 i -1 A i A -1 2 i -1 z ≥ 2 ∥ z ∥ 2 2 . This implies λ max ( A -1 2 i -1 A i A -1 2 i -1 ) ≥ 2 . Moreover, we have λ min ( A -1 2 i -1 A i A -1 2 i -1 ) ≥ 1 because

Thus, det( A i ) det( A i -1 ) = det ( A -1 2 i -1 A i A -1 2 i -1 ) ≥ 2 . By induction, we have det ( A n +1 ) ≥ 2 n det ( A 1 ) . We now give a upper bound for det( A n +1 ) det( A 1 ) . Define a = [ a 1 ] . By AM-GM inequality, we have

Notice that for any t , B t ⪰ I and Tr ( B t ) = Tr ( I ) + Tr ( ˜ Σ -1 t ) + ∥ x t ∥ 2 ˜ Σ -1 t ≤ 2( d +1) γ . Thus, we can upper bound the last expression further by

Overall, we have 2 n ≤ det( A n +1 ) det( A 1 ) ≤ (4 T/γ ) d , and thus n ≤ d log 2 (4 T/γ ) .

Lemma J.9.

Proof. Let ρ = max t ∥ u -x t ∥ ˜ Σ -1 t and t ∗ = argmax t ∥ u -x t ∥ ˜ Σ -1 t , We discuss two conditions:

where we use the definitions of B t and u in the second and the last equality.

Thus, max t ∥ u -x t ∥ ˜ Σ -1 t ≤ ∑ τ ∈I ∥ u -x τ ∥ ˜ Σ -1 τ .

(Lemma J.3 and F is d -self-concordant barrier)

Lemma J.10.

Proof. Since x ⋆ = min x ∈A F ( x ) and π x ⋆ ( u ) ≤ 1 -1 T from Eq. (25). We have from Lemma J.4

Lemma J.11 (Lemma 17 in Zimmert and Lattimore (2022)) . Let F be the entropic barrier and ∥ w ∥ ∇ 2 F ( x t ) -1 ≤ 1 16 η , then

Lemma J.12. With probability at least 1 -δ ,

Proof. Since F is a d -self-concordant barrier (Chewi, 2023), we have

By Lemma J.2, we have ˜ Σ -1 t ⪯ ( E a ∼ ˜ p t [ aa ⊤ ] ) -1 ⪯ 2Σ -1 t , and thus

Thus, ∥ ̂ θ t + b t ∥ ∇ 2 F ( x t ) -1 ≤ β √ 2 d + α √ d . If η ≤ 1 16( β √ 2 d + α √ d ) , by Lemma J.11, we have

Theorem J.13. Algorithm 6 ensures with probability at least 1 -δ , Reg T = ˜ O ( d 3 √ T + d 5 2 C ) , where ˜ O ( · ) hides polylog ( T/δ ) factors.

Proof. Putting Lemma J.6, Lemma J.7, Lemma J.9, Lemma J.10, Lemma J.12 into Eq. (26) and Eq. (27), with η ≤ 1 16( β √ 2 d + α √ d ) and γ = 1 T , we have with probability at least 1 -O ( δ ) ,

By setting α 8 = ˜ Θ( d √ T + √ dC ) , we have

Setting η = 1 160 d √ T +32 α √ d , we get

K Gap-dependent Misspecification

We consider the same setting as Liu et al. (2023a), but remove an assumption for it. Consider bandit learning with general reward function f 0 where for any action x t ∈ X ⊂ R d at round t , the learner get reward y t = f 0 ( x t ) + η t where η t s are zero mean, σ -sub-Gaussian noise. We assume there exists a linear function θ ⊤ x that could approximate f 0 ( x ) in the following manner.

Definition 3.

The algorithm in Liu et al. (2023a) only gets O ( √ T ) regret when ρ ≤ 1 d √ log T and we improve it to ρ ≤ 1 √ d by using elimination-based methods in Algorithm 7. For any design π on action set A , define

Algorithm 7: Phased Elimination for Misspecification

1 Input: Action set A 1 = A . Initialize m 1 = ⌈ 64 d log log d log ( |A| δ ) ⌉ +16 .

2 for ℓ = 1 , 2 , · · · , L do

3

Find the approximate G-optimal design

|

Supp

(

π

)

4

π

ℓ

on

A

ℓ

with

d

log log

d

+16

5 Take each action a ∈ A ℓ exactly u ( a ) times with reward y ( a ) .

6 Calculate

Update active action set

m

ℓ

+1

4

m

ℓ

Define Gap ( x ) = f ⋆ 0 -f 0 ( x ) as the suboptimal gap at point x . Definition 3 implies the true value function f 0 ( x ) = θ ⊤ x +∆( x ) where | ∆( x ) | ≤ ρ ( f ⋆ 0 -f 0 ( x )) = ρ Gap ( x ) . We further assume that | ∆( x ) | ≤ ρ Gap ( x ) which captures both standard uniform misspecification and the gap-dependent misspecification. With this assumption, our main result is summarized in Theorem K.1.

Theorem K.1. For action a , assume y ( a ) = f 0 ( a ) + η a where η a is zero-mean sub-gaussian noise and f 0 ( a ) = θ ⊤ a +∆( a ) with | ∆( a ) | ≤ ρ Gap ( a ) . If ρ ≤ 1 64 √ d , with probability of at least 1 -δ , we have

Proof. First, with probability of at least 1 -δ , for any ℓ and b ∈ A ℓ , we have

where in the last step, we use standard concentration by Equation (20.2) of Lattimore and Szepesvári (2020) and the apply union bound for all actions.

g

(

π

)

2

d

and

For the last term, we have

Thus, for any b ∈ A ℓ ,

When ℓ = 1 , since m 1 = ⌈ 256 d log log d log ( |A| δ ) ⌉ +16 , we have √ 4 d m 1 log ( |A| δ ) ≤ 1 2 4 . Moreover, by trivial bound, max a ∈A 1 Gap ( a ) ≤ 2 and a ⋆ ∈ A 1 .

We will jointly do two inductions. Assume for round ℓ , we have a ⋆ ∈ A ℓ and max a ∈A ℓ Gap ( a ) ≤ 1 2 ℓ -2 . We first show a ⋆ ∈ A ℓ +1 . Thus, for any b ∈ A ℓ , given ρ ≤ 1 64 √ d , since m ℓ = 4 ℓ -1 m 1 , we have

From the induction hypothesis, let ̂ a ℓ = arg max b ∈A ℓ ⟨ ̂ θ ℓ , b ⟩ we have

For ℓ +1 , the remaining actions a ∈ A ℓ +1 satisfy

This implies a ⋆ ∈ A ℓ +1 . Moreover, since a ⋆ ∈ A ℓ , for a ∈ A ℓ +1 , we have

Given ρ ≤ 1 64 √ d ≤ 1 64 , this implies

The above arguments show that as ℓ increases, max a ∈A ℓ Gap ( a ) will shrink by 1 2 at every step. Since for a ∈ A ℓ , Gap ( a ) ≤ 1 2 ℓ -1 = O (√ 4 d m ℓ log ( |A| δ ) )

Finally, given L = log( T ) , we have

When |A| ≥ T d , we can apply similar covering number arguments as in Appendix E, replacing A 1 with a 1 T -net of A . Combined with Theorem K.1, this yields the result in Theorem 6.2.

Using the hard instance for ϵ -misspecified linear bandits setting in Lattimore et al. (2020), we now show that ρ = ˜ Ω( 1 √ d ) for an algorithm to achieve sub-linear regret, proving the above algorithm is optimal in terms of ρ assumption.

Proof. Using Theorem F.5 in Lattimore et al. (2020), there exist a discrete time-invariant action space { a i ∈ R d } 3 T i =1 that satisfies these two conditions:

∥ a i ∥ = 1 ∀ i

̸

and let θ ∗ = √ d -1 8 log(3 T ) ϵa i ∗ for some i ∗ , and let misspecification at each round for all non-optimal arms be ϵ to make the true expected reward zero. Defining τ := max( t | i s = i ∗ ∀ s ≤ t ) , we have E [ R T ] ≥ √ d -1 8 log(3 T ) ϵ E [ τ ] . Since the observed rewards are independent of a i ∗ before time τ , and i ∗ is chosen randomly, we have E [ τ ] ≥ min { T, 3 T -1 2 } . So,

Finally, we have ρ ≥ ϵ √ d -1 8 log(3 T ) ϵ -0 = √ 8 log(3 T ) d -1 , so choosing ϵ = min( √ 8 log(3 T ) d -1 , √ d -1 8 log(3 T ) ) completes the proof showing linear regret when ρ is large enough.

L General Reduction from Corruption-Robust Algorithms to Misspecification

In this section, we extend the results of Section 6 to the reinforcement learning setting. We consider episodic MDPs, denoted by a tuple M = ( S , A , { P h } H h =1 , { r h } H h =1 , s 1 ) for S the set of states, A the set of actions, P h : S × A → △ S the transition kernel, r h : S × A → △ [0 , 1] the reward, and s 1 the starting state. We assume each episode starts in state s 1 , where the agent takes action a 1 , transitions to s 2 ∼ P 1 ( · | s 1 , a 1 ) and receives reward r 1 ∼ r 1 ( s 1 , a 1 ) . This proceeds for H steps at which point the episode terminates and the process resets. We assume that ∑ H h =1 r h ∈ [0 , 1] almost surely (note that the linear bandit setting with rewards in [-1,1] can be incorporated into this with a simple rescaling).

We let π denote a policy, π h : S → △ S , a mapping from states to actions. We denote the value of a policy π on MDP M as V M ,π 0 := E M ,π [ ∑ H h =1 r h ] . We assume access to some function class F ⊆ {S × A → R } . In the MDP setting, we define regret on MDP M as:

In the MDP setting, we consider the following notion of misspecification.

̸

Definition 4 (Misspecification) . For our environment of interest M ⋆ , there exists some environment M 0 such that, for each f h +1 ∈ F , π , and ( s, a, h ) , we have:

and

for some ϵ mis h ( s, a ) > 0 .

We make the following assumption on gap-dependent misspecification.

Assumption 4 (Gap-Dependent Misspecification) . For any policy π , we have

for some ρ ≥ [0 , 1) .

We are interested in relating the above misspecification setting to the corruption-robust setting. In the MDP setting, we allow both the reward and transitions to be corrupted. For some MDP M , define the corruption at episode t and step h as:

where

denotes the Bellman operator, and T h b denotes the corrupted Bellman operator, i.e. T h b denotes the expected reward and next state under the corrupted reward and transition distribution. We denote the total corruption level as

Note that this definition of corruption encompasses both bandits and RL with function approximation. Now assume we have access to the following oracle.

Assumption 5. We have access to a regret minimization algorithm which takes as input F and some C ′ and with probability at least 1 -δ has regret bounded on M 0 as

if C ′ ≥ C , and by HT otherwise, for C as defined above and for (problem-dependent) constants C 1 ( δ, T ) , C 2 ( δ, T ) which may scale at most logarithmically with T and 1 δ .

Before stating our main reduction from corruption-robust to gap-dependent misspecification, we require the following assumption.

Assumption 6. For any π , we have that there exists some f ∈ F such that for all ( s, a, h ) , Q M 0 ,π h ( s, a ) = f h ( s, a ) .

We then have the following result.

Theorem L.1. Assume our environment satisfies Assumption 4. Then under Assumption 5 and Assumption 6, as long as ρ C 2 ( δ T ,T ) 1 -ρ ≤ 1 / 2 , with probability at least 1 -2 δ we can achieve regret bounded as:

Proof of Theorem L.1. First, note that by Assumption 4, we can bound

where we abbreviate Reg T := Reg M ⋆ T . Furthermore, note that under Assumption 4 interacting with M ⋆ is equivalent to interacting with M 0 but where the rewards and transitions are corrupted up to level ϵ mis h ( s, a ) at ( s, a, h ) .

Relating Regret on M 0 to M ⋆ . Define the regret on M 0 as

Under Assumption 4, we have that E M ⋆ ,π ⋆ [ ∑ H h =1 ϵ mis h ( s h , a h ) ] = 0 . Lemma L.2 then implies that

Furthermore, Lemma L.2 also implies

Putting these together we can bound

where the last inequality holds by Assumption 4. Rearranging this gives

Bounding the Regret. Consider running the algorithm of Assumption 5 on M ⋆ and assume we run with parameter C ′ ← β which we will choose shortly. From the above observation, this is equivalent to running on M 0 with corruption level ϵ mis h ( s, a ) at ( s, a, h ) . Then by Assumption 5, with probability at least 1 -δ we have regret on M 0 bounded as

if β ≥ ∑ T t =1 ∑ H h =1 ϵ mis h ( s t h , a t h ) , and by HT otherwise. Furthermore, by the above argument this then immediately implies a regret bound on Reg T .

Let E 1 ,t denote the event that { β ≥ ∑ t t ′ =1 ∑ H h =1 ϵ mis h ( s t ′ h , a t ′ h ) } . Let E 2 denote the event that for all t ≤ T , we have

and note that by the above and under Assumption 5 we then have that E 2 occurs with probability at least 1 -δ . For simplicity, for the remainder of the proof we abbreviate C 1 := C 1 ( δ T , T ) and C 2 := C 2 ( δ T , T ) .

and so

Note that ϵ h ( s t h , a t h ) ∈ [0 , H ] by construction. It follows that, with probability at least 1 -δ , via Azuma-Hoeffding,

Denote this event as E 3 .

Now consider choosing

so that

On E 2 ∩ E 3 , assume that

Let t ⋆ denote the minimum time such that

and note that such a time is guaranteed to exist under (29) and since β ≥ H √ 2 T log 1 /δ + H by construction so ρ ∆( π 1 ) + H √ 2 T log 1 /δ ≤ H + H √ 2 T log 1 /δ ≤ β . Furthermore, since ∆( π ) ≤ H , we have here that ∑ t ⋆ -1 t =1 ρ ∆( π t ) > β -ρH -H √ 2 T log 1 /δ . We then have

However, since by assumption ∑ t ⋆ -1 t =1 ρ ∆( π t ) + H √ 2 T log 1 /δ ≤ β , on E 3 E 1 ,t ⋆ -1 holds so on E 2 ∩ E 3 we have that

This contradicts the above. Therefore, on E 2 ∩E 3 we must have that β ≥ ρ · Reg T + H √ 2 T log 1 /δ , so E 1 ,T holds on E 3 , and so on E 2 ∩ E 3 ,

From our setting of β we can bound this as

The result follows from some simplification.

Lemma L.2. For MDPs M ⋆ , M 0 satisfying Definition 4, under Assumption 6 we have

Proof. Let r ′ denote the reward function on M 0 , and note that under Assumption 6 we have that there exists f ∈ F such that V M 0 ,π h ( s ) = f h ( s, π h ( s )) for all π, s, h . Then Lemma E.15 of Dann et al. (2017) gives that

By Definition 4, we can bound this as

Proof of Corollary 6.2.1. First, note that under Assumption 3, we have that Assumption 4 holds for F the set of functions linear in ϕ , F = { ϕ ( s, a ) ⊤ w : w ∈ R d s.t. ϕ ( s, a ) ⊤ w ∈ [0 , H ] , ∀ s, a } , and ϵ mis h ( s, a ) of Assumption 4 set to Hϵ mis h ( s, a ) for ϵ mis h ( s, a ) of Assumption 3. To see this, let M 0 be the MDP with transitions ⟨ ϕ ( s, a ) , µ h ( · ) ⟩ , and note that this the immediately implies linear realizability on M 0 (and furthermore that Assumption 6 holds). Furthermore, since the total reward is at most H , it is easy to see that under Assumption 3, we can take ϵ mis h ( s, a ) ← Hϵ mis h ( s, a ) .

Next, note that Theorem 4.2 of Ye et al. (2023) gives an algorithm on M 0 satisfying Assumption 5 with C 1 = ˜ O ( √ H 2 d 3 ) and C 2 = ˜ O ( Hd ) (assuming that ∑ H h =1 r h ∈ [0 , 1] almost surely). We can then apply Theorem L.1 to obtain the result.

M Auxiliary Lemmas

Lemma M.1 (Lemma 16 of Zimmert and Lattimore (2022)) . Let X = [ X + xx ⊤ x x ⊤ 1 ] and Y =

Lemma M.2 (Lemma 34 of Liu et al. (2023b)) . Let G be the log-determinant barrier. For any matrix D , if √ Tr ( H t DH t D ) ≤ 1 16 η , then

Lemma M.3 (Strengthened Freedman's inequality (Theorem 9 of Zimmert and Lattimore (2022))) . Let X 1 , X 2 , . . . , X T be a martingale difference sequence with a filtration F 1 ⊆ F 2 ⊆ · · · such that E [ X t |F t ] = 0 and E [ | X t | | F t ] < ∞ almost surely. Then with probability at least 1 -δ ,

where V T = ∑ T t =1 E [ X 2 t | F t ] and U T = max { 1 , max t ∈ [ T ] | X t |} .

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes] .

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope. The claims are validated by detailed proofs.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] .

Justification: The paper discuss the limitations of the work when introducing the algorithms.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes] .

Justification: The paper provides detailed assumptions and proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA] .

Justification: This is a theoretical paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA] .

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA] .

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA] .

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes] .

Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This is a theoretical work. There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA] .

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA] .

Justification: This paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA] .

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA] .

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA] .

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper studies corruption-robust linear bandit optimization and characterizes the regret bound in terms of both weak and strong corruption measures. Under the stochastic setting, this paper proposes a phased elimination algorithm, and the regret bounds match the lower bound. Under the adversarial setting, the paper proposes two individual algorithms for the two corruption measures respectively. In addition, this paper studies gap-dependent misspecification setting through reduction, and discusses a use case for linear MDPs.

STRENGTHS:
- The regret bounds in terms of both corruption measures are provided, where the regret bound depending on $C_\infty$ is first introduced in this paper.
- The theoretical results are supported with detailed proof.
- This paper is generally well-written.

WEAKNESSES:
- The algorithms are efficient regarding regret bound, but the computational complexity is not discussed.
- A conclusion section could be added.

QUESTIONS:
What is the computational cost of the proposed algorithms?

LIMITATIONS:
Some limitations are discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors characterize the problem of learning the presence of reward corruption in the linear bandit setting. They provide matching upper and lower bounds in the corrupted stochastic setting, and initiate the study on the corrupted adversarial setting, for which they obtain optimal scaling in the corruption level.\n\nNot only that, the authors prove a general reduction that efficiently handles gap-dependent misspecification with corruption-robust algorithms. They were able to show that linear MDPs with gap-dependent misspecification are efficiently learnable. While this reduction is general, interestingly they denied the possibility to obtain the tightest rate for gap-dependent misspecification. This observation leads them to develop a specialized algorithm which, in the linear bandit setting, obtains the optimal rate. According to their argument, this resolves the open problem of Liu et al. (2023a).\n\nSTRENGTHS:\n- Interesting results\n    - Deterministic algorithm cannot avoid suboptimal regret (Proposition 1)\n    - Matching upper and lower bound on the stochastic setting, by just changing deterministic sampling to stochastic.\n    - Solving an open problem of instance-dependent misspecified setting.\n\n- Clearly state the limitations of previous works and their technical novelties.\n    - Easy to understand their contributions.\n\nWEAKNESSES:\n- (Minor) The algorithms are not seriously different from the previous works as they mentioned, but this is just a minor point - every theoretical improvement is important. \n\n- Not clear what they tried to say on page 9\n    - Why Theorem 6.2 shows that $\\rho \\leq \\frac{1}{d}$ is not optimal?\n    - Impossibility result (from line 304): so basically what authors are trying to say is, that 'their' reduction is not applicable for a tighter result, right? It is not about any reduction from corruption to misspecification. \n\n- No future works.\n\nQUESTIONS:\n- Unclear points in the weakness section.\n\n- It would be great if authors could explain why the gap-dependent misspecification assumption (Assumption 1) is necessary.\n\n### Minor\n\n- Theorem G.1 in line 296 - is it correct?\n\n- Corollary 6.2.1 in line 209 - it seems like it is the result for the MDP...\n\nLIMITATIONS:\nThe authors adequately addressed the limitations. It would be great if authors provide possible future works. \nThere is no potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

comments: SUMMARY:
This paper studied the corrupted linear bandits. The authors propose four different metrics to evaluate the total corruption in Eq. (1). Many settings are considered in this paper. For stochastic LB, the proposed algorithm achieves a regret bound of $d\sqrt{T}+\sqrt{d} C_{\infty}$. For adversarial LB, the proposed algorithm achieves a regret bound in the order of $d\sqrt{T}+\sqrt{d} C_{\infty}$ or $d^{3}\sqrt{T}+d^{5/2} C$. The authors also consider the gap-dependent misspecification, where the misspecification level of an arm $a$ can be evaluated by $\rho$ times the gap of arm $a$.

STRENGTHS:
See summary.

WEAKNESSES:
**Weaknesses and Questions:**
1. At lines 107-109, the authors claim that the strong adversary is equivalent to the CM viewpoint. This doesn't seem right. For regret, the strong adversary is harder than the CM viewpoint. Thus, it is unfair and wrong to compare He et al. (2022) in the same way.
2. At line 131, adversarial linear bandits are discussed. However, no problem definition of this problem is introduced before line 131.
3. This paper studies the fixed action set, while the previous works He et al. (2022) and Foster et al. (2020) allow the action set to be chosen by an adaptive adversary, which is much harder than this paper. Table 1 is not fair. He et al. (2022) is for the adaptive adversarial viewpoint, which is totally different from the stochastic LB. For a fixed action set, the optimal regret without $C$ should be $\sqrt{d T \log k}$, where $k$ is the number of arms.
4. Assumption 1 is not very reasonable.

QUESTIONS:
See weaknesses.

LIMITATIONS:
N.A.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-14