Linear Contextual Bandits with Adversarial Corruptions

Abstract:
We study the linear contextual bandits problem in the presence of adversarial 1 corruption, where the interaction between the player and a possibly infinite decision 2 set is contaminated by an adversary that can corrupt the reward up to a corruption 3 level C measured by the sum of the largest alteration on rewards in each round. 4 We present a variance-aware algorithm that is adaptive to the level of adversarial 5 contamination C. The key algorithmic design includes (1) a multi-level partition 6 scheme of the observed data, (2) a cascade of confidence sets that are adaptive to 7 the level of the corruption, and (3) a variance-aware confidence set construction 8 that can take advantage of low-variance reward. We further prove that the regret 9 of the proposed algorithm is Õ(Cd √∑T t=1 σ 2 t + C 2 √ dT + CR √ dT ), where 10 d is the dimension of context vectors, T is the number of rounds, R is the range 11 of noise and σ t , t = 1 . . . , T are the variances of instantaneous reward. We also 12 prove a gap-dependent regret bound for the proposed algorithm, which is instance13 dependent and thus leads to better performance on good practical instances. To the 14 best of our knowledge, this is the first variance-aware corruption robust algorithm 15 for contextual bandits. 16

We study the linear contextual bandits problem in the presence of adversarial1 corruption, where the interaction between the player and a possibly infinite decision2 set is contaminated by an adversary that can corrupt the reward up to a corruption3 level C measured by the sum of the largest alteration on rewards in each round.4 We present a variance-aware algorithm that is adaptive to the level of adversarial5 contamination C. The key algorithmic design includes (1) a multi-level partition6 scheme of the observed data, (2) a cascade of confidence sets that are adaptive to7 the level of the corruption, and (3) a variance-aware confidence set construction8 that can take advantage of low-variance reward. We further prove that the regret9
of the proposed algorithm is Õ(C2d √∑T
t=1 σ 2 t + C
2 √ dT + CR √ dT ), where10
d is the dimension of context vectors, T is the number of rounds, R is the range11 of noise and σ2t , t = 1 . . . , T are the variances of instantaneous reward. We also12 prove a gap-dependent regret bound for the proposed algorithm, which is instance-13 dependent and thus leads to better performance on good practical instances. To the14 best of our knowledge, this is the first variance-aware corruption robust algorithm15 for contextual bandits.16
1 Introduction17
Multi-armed bandits algorithms are widely applied in online advertising (Li et al., 2010), clinical18 trials (Villar et al., 2015), recommendation system (Deshpande and Montanari, 2012) and many other19 real-world tasks. In the model of multi-armed bandits, the algorithm needs to decide which action20 (or arm) to take (or pull) at each round and receive a reward for the chosen action. In the stochastic21 setting, the reward is subject to a fixed but unknown distribution for each action. In reality, however,22 these rewards can easily be “corrupted” by some malicious users. A typical example is click fraud23 (Lykouris et al., 2018), where botnets simulate the legitimate users clicking on an ad to fool the24 recommendation systems. This motivates the studies of the bandits algorithms that are robust to25 adversarial corruptions.26
For example, Lykouris et al. (2018) introduced a bandit model in which an adversary could corrupt27 the stochastic reward generated by an arm pull. They proposed an algorithm and show that the28 regret of this “middle ground” scenario degrades smoothly with the amount of corruption injected29 by the adversary. Gupta et al. (2019) proposed an alternative algorithm which gives a significant30 improvement in regret.31
While the algorithms that are robust to the corruptions have been studied in the setting of multi-armed32 bandits in a number of prior works, they are still understudied in the setting of linear contextual33 bandits. The linear contextual bandits problem can be regarded as an extension of the multi-armed34 bandit problem to linear optimization, in order to tackle an unfixed and possibly infinite set of feasible35
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.
actions. There is a large body of literature on efficient algorithms for linear contextual bandits with36 no corruptions (Abe et al., 2003; Auer, 2002; Chu et al., 2011; Dani et al., 2008; Rusmevichientong37 and Tsitsiklis, 2010; Abbasi-Yadkori et al., 2011; Li et al., 2019b), to mention a few. The significance38 of this setting lies in the fact that linear regression approaches are widely used in recommendation39 systems and advertising (Li et al., 2010; Jhalani et al., 2016; Deshpande and Montanari, 2012). Linear40 contextual bandits with adversarial corruptions is an arguably more challenging setting since most of41 the previous corruption-robust algorithms are based on the idea of action elimination (Lykouris et al.,42 2018; Gupta et al., 2019; Bogunovic et al., 2021), which is not applicable to the contextual bandits43 settings where the decision set is time varying and possibly infinite at each round. In Garcelon et al.44 (2020), it is shown that a malicious agent can force a linear contextual bandit algorithm to take any45 desired action T − o(T ) times over T rounds, while applying adversarial corruptions to rewards with46 a cumulative cost that only grow logarithmically. This poses a big challenge for designing corruption47 robust algorithms for linear contextual bandits.48
In this paper, we make a first attempt to study a linear contextual bandit model where an adversary49 can corrupt the rewards up to a corruption level C, which is defined as the the sum of biggest50 alteration the adversary made on rewards in each round. We propose a linear contextual bandits51 algorithm that is robust to reward corruption, dubbed multi-level optimism-in-the-face-of-uncertainty52 weighted learning (Multi-level OFUL). More specifically, our algorithm consists of the following53 novel techniques: (1) We design a multi-level partition scheme and adopt the idea of sub-sampling to54 do the robust estimation of the model parameters; (2) We maintain a cascade of candidate confidence55 sets corresponding to different corruption level (which is unknown) and randomly select a confidence56 set at each round to take the action; and (3) We design confidence sets that depend on the variances57 of rewards, which lead to a potentially tighter regret bound.58
Our contributions are summarized as follows:59
• We propose a variance-aware algorithm which is adaptive to the amount of adversarial corruptions60 C. To the best of our knowledge, it is the first algorithm for the setting of linear contextual bandits61 with adversarial corruptions which does not rely on the finite number of actions and other additional62 assumptions.63
• We prove that the regret of our algorithm is in Õ ( C2d √∑T t=1 σ 2 t + C 2 √ dT + CR √ dT ) , where64
d is the dimension of context vectors, T is the number of rounds, R is the range of noise and σ2t , t =65 1 . . . , T are the variances of instantaneous reward. Our regret upper bound has a multiplicative66 dependence on C2 which indicates that our algorithm achieves a sub-linear regret when the67 corruption level satisfies C = o(T 1/4).68
• We also derive a gap-dependent regret bound Õ (
1 ∆ · C 2R2d+ 1∆ · d 2C2 maxt∈[T ] σ 2 t
) for our69 proposed algorithm, which is instance-dependent and thus leads to a better performance on good70 practical instances.71
Notation. We use lower case letters to denote scalars, and use lower and upper case bold face letters72 to denote vectors and matrices respectively. We denote by [n] the set {1, . . . , n}. For a vector x ∈ Rd73 and matrix Σ ∈ Rd×d, a positive semi-definite matrix, we denote by ‖x‖2 the vector’s Euclidean74 norm and define ‖x‖Σ = √ x>Σx. For two positive sequences {an} and {bn} with n = 1, 2, . . . ,75 we write an = O(bn) if there exists an absolute constant C > 0 such that an ≤ Cbn holds for all76 n ≥ 1 and write an = Ω(bn) if there exists an absolute constant C > 0 such that an ≥ Cbn holds77 for all n ≥ 1. We use Õ(·) to further hide the polylogarithmic factors. We use 1(·) to denote the78 indicator function.79
2 Related Work80
Bandits with Adversarial Attacks: There is a large body of literature on the problems of multi-81 armed bandits with adversarial corruptions. Most research in this area aims to design algorithms that82 achieve desirable regret bound in both stochastic multi-armed bandits and adversarial bandits, known83 as “the best of both worlds” guarantees (Bubeck and Slivkins, 2012; Seldin and Slivkins, 2014; Auer84 and Chiang, 2016; Seldin and Lugosi, 2017; Zimmert and Seldin, 2019). These works mainly focus85 on achieving bounds in the worst case and the case where there is no adversary. As a result, these86
algorithms are either not robust to instances that moderate amount of corruptions occur, or suffer87 from restrictive assumptions on adversarial corruptions. Distinctive from the above line of research,88 Lykouris et al. (2018) focus on a variant of classic multi-armed bandit model in which each pull of an89 arm generates a stochastic reward that may be contaminated by an adversary before it is revealed90 to the player. In their work, the corruption level C is defined as C = ∑ t maxa |rt(a) − rtS(a)|91
where rtS(a) is the stochastic reward of arm a and r t(a) is the corrupted reward of arm a at round t.92
They develop algorithms adaptive to the unknown corruption level, which achieves an O(K1.5C √ T )93 regret bound. Gupta et al. (2019) proposed an improved algorithm that can achieve a regret bound94 with only additive dependence on C.95
On the other hand, many research efforts have also been devoted into designing adversarial attacks96 that cause standard algorithms to fail (Jun et al., 2018; Liu and Shroff, 2019; Gupta et al., 2019;97 Garcelon et al., 2020).98
Linear Bandits with Corruptions: Li et al. (2019a) studied stochastic linear bandits with adversarial99 corruptions and achieved Õ( 1∆ ·d 5/2C+ 1∆2 ·d 6) regret bound where d is the dimension of the context100 vectors, ∆ is the gap between the rewards of the best and the second best action in the decision101 set D. The distinction between Li et al. (2019a) and our work is that Li et al. (2019a) considers a102 fixed decision set D throughout all T rounds, while we consider contextual bandits with changing103 decision set observed before each round. Bogunovic et al. (2021) also studied linear bandits with104 adversarial corruptions and considered the setting under the assumption that context vectors undergo105 small random perturbations, which is previously introduced by Kannan et al. (2018). Aside from106 the additional assumption, another major distinction in Bogunovic et al. (2021) is that the number107 of actions k is finite and the regret bound depends on k in the contextual setting with unknown108 corruption level C. Recently, Lee et al. (2021) considered corrupted linear bandits with a finite and109 fixed decision set and achieve an instance-independent regret of Õ(d √ T + C). Though both their110 work and the work by Li et al. (2019a) focus on corrupted linear stochastic bandits, Lee et al. (2021)111 have a slightly different definition of regret and adopt a strong assumption on corruptions that in112 each round t, the corruptions on rewards are linear in the actions. Neu and Olkhovskaya (2020)113 studied linear contextual bandits with a finite decision set (i.e., K actions) and an adversary. Unlike114 our model, they assume that the adversary can add an arbitrary noise to the loss under a limited115 amount and prove an Õ((Kd) 1 3T 2 3 ) + · √ dT regret bound for their proposed algorithm. Kapoor116 et al. (2019) considered the corrupted linear contextual bandits setting under a strong assumption on117 corruptions that for any prefix, at most an η fraction of the rounds are corrupted.118
3 Preliminaries119
In this paper, we study linear contextual bandits with adversarial corruptions. We will introduce our120 model and some basic concepts in this section.121
Corrupted linear contextual bandits. We consider the the linear contextual bandits model studied122 in Abbasi-Yadkori et al. (2011) under the same corruption studied by Lykouris et al. (2018). In detail,123 distinctive from the linear contextual bandits Abbasi-Yadkori et al. (2011), the interaction between124 the agent and the environment is now contaminated by an adversary. The protocol between the agent125 and the adversary at each round t ∈ [T ] can be described as follows:126
1. At the beginning of round t, the environment generates an arbitrary decision set Dt ⊆ Rd where127 each element represents a feasible action that can be selected by the agent.128
2. The environment generates stochastic reward function r′t(a) = 〈a,µ∗〉+ t(a) together with an129 upper bound on the standard variance of t(a), i.e., σt(a) for all a ∈ Dt.130
3. The adversary observes Dt, r′t(a), σt(a) for all a ∈ Dt and decides a corrupted reward function131 rt defined over Dt.132
4. The agent observes Dt and selects at ∈ Dt.133 5. The adversary observes at and then returns rt(at) and σt(at).134
6. The agent observes rt(at), σt(at).135
Let Ft be the σ-algebra generated by D1:t,a1:t−1, 1:t−1, r1:t−1 and σ1:t−1.136
At step 2, µ∗ is a hidden vector unknown to the agent which can be observed by the adversary at the137 beginning. We assume that for all t ≥ 1 and all a ∈ Dt, ‖a‖2 ≤ A, |〈a,µ∗〉| ≤ 1 and ‖µ∗‖2 ≤ B138 almost surely. t(a) can be any form of random noise as long as it satisfies139
∀t ≥ 1,∀a ∈ Dt, | t(a)| ≤ R, E[ t(a)|Ft] = 0, E[ 2t (a)|Ft] ≤ σ2t (a). (3.1) This assumption on t is a variant of that in Zhou et al. (2020): We now require the noise to be140 generated for all a ∈ Dt in advance before the adversary decides the corrupted reward function. Our141 assumption on noises is more general than those in (Li et al., 2019a; Bogunovic et al., 2021; Kapoor142 et al., 2019) where they are assumed to be 1-sub-Gaussian or Gaussian.143
At step 3, we assume that the adversary has observed all the previous information and thus may144 predict which policy the agent will take at the current round. However, since the agent can take a145 randomized policy, the adversary may not know exactly which action the agent will take.146
Corruption level. We define corruption level147
C = 1
R+ 1 T∑ t=1 sup a∈Dt |r′t(a)− rt(a)|. (3.2)
to indicate the level of adversarial contamination. We say a model is C-corrupted if the corruption148 level is no larger than C.149
Our definition of corruption level is equivalent to the counterpart in Lykouris et al. (2018) and Gupta150 et al. (2019) where they define C = ∑T t=1 maxa |r′t(a) − rt(a)| in our notation of rewards. We151 introduce a factor of 1R+1 since the noise is of range R in our model, while they assume all the152 rewards are in range [0, 1].153
Regret. Since the actions selected by the agent may not be deterministic, we define the regret for this154 model as follows:155
Regret(T ) = T∑ t=1 〈a∗t ,µ∗〉 − E [ T∑ t=1 〈at,µ∗〉 ] . (3.3)
Our definition follows from the definition in Gupta et al. (2019) where the standard metric in stochastic156 multi-armed bandit models of pseudo-regret is adopted. But note that we need to take the expectation157 on ∑T t=1 r ′ t(at) (the second term in (3.3)), since a randomized policy is applied in each round.158
Gap. Let ∆t be the gap between the rewards of the best and the second best action in the decision set159 Dt as defined in Dani et al. (2008) which can be formally written as160
∆t = min a∈Dt,a6∈A∗t
(〈a∗t ,µ∗〉 − 〈a,µ∗〉) . (3.4)
where A∗t = argmaxa∈Dt〈a,µ ∗〉 and a∗t is an arbitrary element in A∗t . Let ∆ denotes the smallest161 gap mint∈[T ] ∆t.162
4 The Proposed Algorithm163
In this section, we propose a variance-aware algorithm, Multi-level OFUL, in Algorithm 1, to tackle164 the corrupted linear contextual bandits problem. At the core of our algorithm is an action partition165 scheme to group historical selected actions and use them to select the future actions in different166 groups with different probabilities. Such a scheme is introduced to deal with the unknown corruption167 level. For simplicity, we denote rt(at), σt(at) in Section 3 by rt, σt in our algorithm.168
Main difficulty in our setting. We begin with the main difficulty that prevents us from applying169 existing algorithms to our setting. Consider a simpler setting where the agent knows the corruption170 level C in prior, and we have σt = R for all t. Then we can apply OFUL (Abbasi-Yadkori et al.,171 2011) to solve our problem. In detail, in each round we estimate µ∗ by µt, which is the minimizer of172 the following ridge regression problem:173
µt = argmin µ∈Rd λ‖µ‖22 + t−1∑ i=1 [〈µ,ai〉 − ri]2. (4.1)
Algorithm 1 Multi-level OFUL 1: Set the largest level of confidence sets: `max ← dlog2 2T e. 2: For ` ∈ [`max], set Σ1,` ← λI,µ1,` ← 0, c1,` ← 0. 3: Set Σ1 ← λI,µ1 ← 0, c1 ← 0. 4: for t = 1, · · · , T do 5: Observe Dt. 6: for ` = 1, · · · , `max do 7: Set βt,` and γt,` as defined in (4.5) and (4.6). 8: C′t,` ← {µ|‖µ− µt‖Σt ≤ βt,`} ∩ { µ|‖µ− µt,`‖Σt,` ≤ γt,` } .
9: Ct,` ← { C′t,`, C′t,` 6= ∅ Ct,`+1, otherwise .
10: end for 11: Set f(t) = { ` with probability 2−` 1 < ` ≤ `max 1 otherwise .
12: Select at ← argmaxa∈Dt maxµ∈Ct,f(t)〈µ,a〉 and observe rt, σt. 13: Set σt = max{(R+ 1)/ √ d, σt}. 14: Σt+1 ← Σt + ata>t /σ2t , ct+1 ← ct + rtat/σ2t ,µt+1 ← Σ−1t+1ct+1. 15: for ` 6= f(t) do 16: Σt+1,` ← Σt,`, ct+1,` ← ct,`,µt+1,` ← µt,`. 17: end for 18: Σt+1,f(t) ← Σt,f(t) + ata>t /σ2t , ct+1,f(t) ← ct,f(t) + rtat/σ2t . 19: µt+1,f(t) ← Σ−1t+1,f(t)ct+1,f(t). 20: end for
By slightly modifying the self-normalized martingale concentration inequality proposed in Abbasi-174 Yadkori et al. (2011), we can conclude that µ∗ belongs to the ellipsoid ‖µ− µt‖Σ−1t ≤ βt with high175 probability, where βt = Õ(R √ d + C √ d). Such a confidence bound leads to a final regret which176 has a polynomial dependence on C. However, such a simple approach have two limitations. First,177 the agent does not know C apriori in our setting, thus it is impossible to set βt to be dependent on178 C. Second, vanilla ridge regression estimator does not consider different variances σt in each round,179 thus it only gives a very conservative estimation.180
Action partition scheme. To address the unknown C issue, besides the original estimator µt which181 uses all previous data, Algorithm 1 maintains several additional learners to learn µ∗ at different182 accuracy level simultaneously, and it randomly selects one of the learners with different probabilities183 at each round. Such a “parallel learning” idea is inspired by Lykouris et al. (2018). In detail,184 we partition the observed data into `max levels indexed by [`max] and maintain `max sub-sampled185 estimators µt,1, · · · ,µt,`max . According to line 11, the observed data in round t goes into level ` with186 probability 2−` if 1 < ` ≤ `max and it goes to level 1 with probability 1− ∑`max `=2 2
−` = 1/2+2−`max .187 The intuition is that if 2` ≥ C, then the corruption level experienced by level `188
Corruptiont,` = t∑ i=1 1(f(i) = `) R+ 1 · sup a∈Di |ri(a)− r′i(a)| (4.2)
can be bounded by some quantity that is independent of C. That says, the individual learners whose189 level is greater than logC can learn µ∗ successfully, even with the corruption. For the learners whose190 level is less than logC, we can also control the error by controlling the probability for the agent to191 select them.192
Weighted regression estimator. After introducing the partition scheme, we still need to deal193 with the varying variance (heteroscedastic) case. Similar to (Kirschner and Krause, 2018; Zhou194 et al., 2020), we proposed the following weighted ridge regression estimator, which incorporates the195 variance information of the rewards into estimation:196
µt = argmin µ∈Rd λ‖µ‖22 + t−1∑ i=1 [〈µ,ai〉 − ri]2/σ2i . (4.3)
Here σt is defined as the upper bound of the true variance σt in line 13. The closed-form solution to197 (4.3) is calculated at each round in line 14. The use of σt, as we will show later, makes our estimator198 more efficient in the heteroscedastic case. Meanwhile, we also apply our weighted regression199 estimator to each individual learner, and their estimator µt,` can be written as follows:200
µt,` = argmin µ∈Rd λ‖µ‖22 + t−1∑ i=1 1(f(i) = `) · [〈µ,ai〉 − ri]2/σ2i . (4.4)
The closed-form solution to (4.4) is calculated at each round in lines 15–20.201
Final Multi-Level confidence sets. With the estimators µt, µt,1, · · · ,µt,`max at the beginning of202 round t, we define a cascade of candidate confidence sets as in lines 6–10, where203
βt,` = 8
√ d log (R+ 1)2λ+ tA2
(R+ 1)2λ log(4t2/δ) + 4
√ d log(4t2/δ) + 2` √ d+ √ λB, (4.5)
γt,` = 8
√ d log (R+ 1)2λ+ tA2
(R+ 1)2λ log(8t2T/δ) + 4
√ d log(8t2T/δ) + C` √ d+ √ λB, (4.6)
with C` = log(2`2/δ) + 3. For simplicity, we define204 `∗ = max{2, dlog2 Ce} (4.7) as an important threshold in our later proof for regret bound analysis. Later we will prove that Ct,`205 contains µ∗ for all ` ≥ `∗, t ≥ 1 with high probability.206 Note that each candidate confidence set can be written as the intersection of two ellipsoids. The207 intuition behind our construction of candidate confidence sets is that we hope that Ct,` is robust208 enough to handle the 2`-corrupted case, i.e., µ∗ ∈ Ct,` with high probability. To achieve this, the first209 ellipsoid makes use of the global information and the “radius” βt,` need to contain a factor of 2` to210 tolerate a corruption level of 2`, and the second ellipsoid makes use of the observed data in level `211 since this level only contain a few times of corruptions in 2`-corrupted case.212
Action selection. With the candidate confidence sets, we use line 11 to randomly decide one213 confidence set and select an action based on the optimism-in-the-face-of-uncertainty (OFU) principle214 in line 12. Then we update the estimators for the next round t+ 1.215 Remark 4.1. Our algorithm shares a similar strategy for partitioning the observed data with the216 algorithm in Lykouris et al. (2018) but note that there is a major difference in that: Lykouris et al.217 (2018) regard the partition scheme as a “layer structure”, i.e., their algorithm further uses different218 estimators in layers of parallel learners and do action elimination layer by layer in each round. In219 contrast, the sub-sampled estimators in our algorithm are used independently, i.e., the selected action220 only relies on one of the partitions. As a result, Algorithm 1 does not need to do action elimination,221 thus is capable of handling the cases where the number of actions is huge or even infinite.222
5 Main Results223
In this section we present our main theorem, which establishes the regret bound for Multi-level224 OFUL.225 Theorem 5.1. Set λ = 1/B2. Suppose that C = Ω(1), R = Ω(1), for all t ≥ 1 and all a ∈ Dt, 〈a,µ∗〉 ∈ [−1, 1]. Then with probability at least 1 − 3δ, the regret of Algorithm 1 is bounded as follows:
Regret(T ) = Õ C2d √√√√ T∑
t=1
σ2t + C 2 √ dT + CR √ dT  . Remark 5.2. When σt, R = Ω(1), the regret bound in Theorem 5.1 matches the regret bound of226 OFUL proposed in Zhou et al. (2020) when the corruption level C is a constant.227
Remark 5.3. Compared with the Õ(d √ T + C) result in Lee et al. (2021), our result has a multi-228 plicative quadratic dependence on C, which seems to be worse. However, we want to emphasize that229 we focus on a more challenging contextual bandits setting where the decision sets Dt at each round230 are not identical, which is different from that in Lee et al. (2021). Therefore, our result and that in231 Lee et al. (2021) are not directly comparable.232
Remark 5.4. Note that this instance-independent regret upper bound also holds in a stronger model233 than the one described in Section 3, where the adversary can even decide the decision set Dt at each234 round t since our regret bound can hold without any assumption on the decision sets.235 Corollary 5.5. Under the same conditions as in Theorem 5.1, if σt given by the environment are all R, the regret of Algorithm 1 is bounded by:
Regret(T ) = Õ ( C2dR √ T ) .
We also provide a gap-dependent regret bound.236 Theorem 5.6. Suppose that C = Ω(1), R = Ω(1), for all t ≥ 1 and all a ∈ Dt, 〈a,µ∗〉 ∈ [−1, 1]. Then with probability at least 1− 3δ, the regret of Algorithm 1 is bounded as follows:
Regret(T ) = Õ
( 1
∆ · C2R2d+ 1 ∆ · d2C2 max t∈[T ] σ2t
) .
Remark 5.7. Theorem 5.6 automatically suggests an Õ(R2d2C2/∆) regret bound, by the fact237 σt = O(R). Compared with previous result Õ(d5/2C/∆ + d6/∆2) (Lee et al., 2021), our result238 has a better dependence on the dimension d but a worse dependence on the corruption level C. As239 Remark 5.3 suggests, we focus on a more challenging contextual bandits setting, and the worse240 dependence on C might be due to this.241
6 Proof Outline242
First we have the following lemma which is a corruption-tolerant variant of Bernstein inequality for243 self-normalized vector-valued martingales introduced in Zhou et al. (2020).244 Lemma 6.1 (Bernstein inequality for vector-valued martingales with corruptions). Let {Gt}∞t=1 be a filtration, {xt, ηt}t≥1 a stochastic process so that xt ∈ Rd is Gt-measurable and ηt ∈ R is Gt+1-measurable. Fix R,L, σ, λ > 0, µ∗ ∈ Rd. For t ≥ 1 let ystocht = 〈µ∗,xt〉 + ηt and suppose that ηt,xt also satisfy
|ηt| ≤ R,E[ηt|Gt] = 0,E[η2t |Gt] ≤ σ2, ‖xt‖2 ≤ L.
Suppose {yt} is a sequence such that ∑t i=1 |yi − ystochi | = C(t) for all t ≥ 1. Then, for any245 0 < δ < 1, with probability at least 1− δ we have ∀t > 0,246
‖µt − µ∗‖Zt ≤ βt + C(t) + √ λ‖µ∗‖2,
where for t ≥ 1, µt = Z−1t bt, Zt = λI + ∑t i=1 xix > i , bt = ∑t i=1 yixi, and
βt = 8σ
√ d log dλ+ tL2
dλ log(4t2/δ) + 4R log(4t2/δ).
Next, we have that with high probability, all the level ` ≥ `∗ only influenced by limited amount of247 corruptions as mentioned in Section 4.248 Lemma 6.2. Let Corruptiont,` be defined in (4.2). Then we have with probability at least 1− δ, for all ` ≥ `∗, t ≥ 1:
Corruptiont,` ≤ C` = log(2`2/δ) + 3.
We denote by Esub the event that the above inequality holds.249
We define the following event to further show that our candidate confidence sets with ` ≥ `∗ are250 “robust” enough, i.e. Ct,` contains µ∗ with high probability.251 Definition 6.3. Let `∗ be defined in (4.7). We introduce the event E1 as follows.252
E1 := { ∀` ≥ `∗ and t ≥ 1, ‖µ∗ − µt‖Σt ≤ βt,` and ‖µ∗ − µt,`‖Σt,` ≤ γt,` } . (6.1)
where βt,`, γt,` are defined in (4.5) and (4.6).253
Next lemma suggests that the event E1 happens with high probability.254 Lemma 6.4. Let E1 be defined in (6.1). For any 0 < δ < 1/3, we have P(E1) ≥ 1− 3δ.255
For simplicity, we define at,` = argmaxa∈Dt maxµ∈Ct,`〈µ,a〉 for each level `. at can be seen as an256 action vector randomly chosen from at,`, ` ∈ [`max]. Next two lemmas suggest that under event E1,257 at each round, the gap between the optimal reward and the selected reward can be upper bounded by258 some bonus terms related to at,`.259 Lemma 6.5. Suppose E1 occurs. If f(t) ≤ `∗, we have 〈a∗t − at,µ∗〉 ≤ 2βt,`∗‖at‖Σ−1t +260 2βt,`∗‖at,`∗‖Σ−1t .261 Lemma 6.6. On event E1, if f(t) = ` > `∗, we have 〈a∗t − at,µ∗〉 ≤ 2γt,`‖at‖Σ−1t,` .262
Now we provide the proof sketch of Theorem 5.1.263
Proof sketch of Theorem 5.1 . Suppose E1 occurs. The main idea to bound the regret is to decompose264 the total rounds [T ] into two non-overlapping parts, based on which individual learner is selected at265 that round. In detail, we have266
Regret(T ) = E [ T∑ t=1 (〈a∗t ,µ∗〉 − 〈at,µ∗〉) ]
= E [ T∑ t=1 1(f(t) ≤ `∗) (〈a∗t ,µ∗〉 − 〈at,µ∗〉) ] ︸ ︷︷ ︸
I1
+ `max∑ `=`∗+1 E [ T∑ t=1 1(f(t) = `) (〈a∗t ,µ∗〉 − 〈at,µ∗〉) ] ︸ ︷︷ ︸
I2(`)
. (6.2)
Here I1 represents the regret where the the "low-level" learner is selected, where the corruption level267 is beyond the learner level.For this case, by Lemma 6.5, we can directly show that268
I1 ≤ E [ T∑ t=1 1(f(t) ≤ `∗) min { 2, 2βt,`∗‖at,`∗‖Σ−1t + 2βt,`∗‖at‖Σ−1t }] . (6.3)
We further bound (6.3). Let Ft be the σ-algebra generated by as, rs, σs, f(s) for s ≤ t − 1.269 Then by the property of our partition scheme (note that P(f(t) = `∗) = 2−`∗), we can show that270 E [ 1(f(t) ≤ `∗)‖at,`∗‖Σ−1t |Ft ] ≤ 2`∗E [ ‖at‖Σ−1t |Ft ] . Therefore, we can further bound I1 by271
I1 ≤ 4 · 2` ∗ E [ T∑ t=1 min { 2, βT,`∗‖at‖Σ−1t }]
︸ ︷︷ ︸ I3
. (6.4)
To further bound I3, we split [T ] into 2 parts, I1 = {t ∈ [T ]|‖at/σt‖Σ−1t > 1}, I2 = {t ∈272 [T ]|‖at/σt‖Σ−1t ≤ 1} to bound I3. The intuition here is that the cardinality of I1 is bounded, and273 the sum of terms with t ∈ I2 can be bounded using Cauchy-Schwarz inequality.274
∑ t∈I1 min { 2, βT,`∗‖at‖Σ−1t } ≤ 2|I1| ≤ 2 T∑ t=1 min { 1, ‖at/σt‖2Σ−1t } ≤ 4d log (R+ 1) 2λ+ TA2 (R+ 1)2λ ,
(6.5) where the first inequality holds since min { 2, βT,`∗‖at‖Σ−1t } ≤ 2, the second inequality follows275 from the definition of I1, the third inequality holds by Lemma C.2.276
∑ t∈I2 min { 2, βT,`∗‖at‖Σ−1t } ≤ βT,`∗ √∑ t∈I2 σ2t · √∑ t∈I2 min { 1, ‖at/σt‖2Σ−1t }
≤ βT,`∗ √√√√(R+ 1)2T/d+ T∑ t=1 σ2t · √ 2d log (R+ 1)2λ+ TA2 (R+ 1)2λ ,
(6.6)
where the first inequality follows from Cauchy-Schwarz inequality, the second inequality follows277 from the definition of σt and Lemma C.2.278
Substituting (6.5) and (6.6) into (6.3), we have279
I1 = Õ
C2d √√√√ T∑
t=1
σ2t + C 2 √ dT + CR √ dT  . (6.7) Now it remains to bound I2(`). By Lemma 6.6, we have280
I2(`) ≤ 2E [ T∑ t=1 1(f(t) = `) min { 1, γt,`‖at,`‖Σ−1t,` }]
︸ ︷︷ ︸ I4
= Õ R√Td+ d √√√√ T∑
t=1
σ2t  , (6.8) where the second equality can be proved by analysis similar to that of (6.5) and (6.6). Finally,281 substituting (6.7) and (6.8) into (6.2) ends our proof.282
283
7 Conclusion and Future Work284
In this paper, we have considered the linear contextual bandits problem in the presence of adversarial285 corruptions. We propose a Multi-level OFUL algorithm, which is provably robust to the adversarial286 attacks. We prove a gap-independent regret bound of Õ ( C2d √∑T t=1 σ 2 t + C 2 √ dT + CR √ dT ) 287
together with a gap-dependent bound of Õ (
1 ∆ · C 2R2d+ 1∆ · d 2C2 maxt∈[T ] σ 2 t
) .288
We leave it as an open question that whether the multiplicative dependence on C2 in the regret upper289 bounds can be removed without making additional assumptions in our setting.290
References291 ABBASI-YADKORI, Y., PÁL, D. and SZEPESVÁRI, C. (2011). Improved algorithms for linear292 stochastic bandits. In NIPS, vol. 11.293
ABE, N., BIERMANN, A. W. and LONG, P. M. (2003). Reinforcement learning with immediate294 rewards and linear hypotheses. Algorithmica 37 263–293.295
AUER, P. (2002). Using confidence bounds for exploitation-exploration trade-offs. Journal of296 Machine Learning Research 3 397–422.297
AUER, P. and CHIANG, C.-K. (2016). An algorithm with nearly optimal pseudo-regret for both298 stochastic and adversarial bandits. In Conference on Learning Theory. PMLR.299
BOGUNOVIC, I., LOSALKA, A., KRAUSE, A. and SCARLETT, J. (2021). Stochastic linear bandits300 robust to adversarial attacks. In International Conference on Artificial Intelligence and Statistics.301 PMLR.302
BUBECK, S. and SLIVKINS, A. (2012). The best of both worlds: Stochastic and adversarial bandits.303 In Conference on Learning Theory. JMLR Workshop and Conference Proceedings.304
CHU, W., LI, L., REYZIN, L. and SCHAPIRE, R. (2011). Contextual bandits with linear payoff305 functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and306 Statistics. JMLR Workshop and Conference Proceedings.307
DANI, V., HAYES, T. P. and KAKADE, S. (2008). Stochastic linear optimization under bandit308 feedback. In COLT.309
DESHPANDE, Y. and MONTANARI, A. (2012). Linear bandits in high dimension and recommendation310 systems. In 2012 50th Annual Allerton Conference on Communication, Control, and Computing311 (Allerton). IEEE.312
GARCELON, E., ROZIERE, B., MEUNIER, L., TARBOURIECH, J., TEYTAUD, O., LAZARIC, A.313 and PIROTTA, M. (2020). Adversarial attacks on linear contextual bandits. arXiv preprint314 arXiv:2002.03839 .315
GUPTA, A., KOREN, T. and TALWAR, K. (2019). Better algorithms for stochastic bandits with316 adversarial corruptions. In Conference on Learning Theory. PMLR.317
JHALANI, T., KANT, V. and DWIVEDI, P. (2016). A linear regression approach to multi-criteria318 recommender system. In International Conference on Data Mining and Big Data. Springer.319
JUN, K.-S., LI, L., MA, Y. and ZHU, X. J. (2018). Adversarial attacks on stochastic bandits. In320 NeurIPS.321
KANNAN, S., MORGENSTERN, J. H., ROTH, A., WAGGONER, B. and WU, Z. (2018). A smoothed322 analysis of the greedy algorithm for the linear contextual bandit problem. In NeurIPS.323
KAPOOR, S., PATEL, K. K. and KAR, P. (2019). Corruption-tolerant bandit learning. Machine324 Learning 108 687–715.325
KIRSCHNER, J. and KRAUSE, A. (2018). Information directed sampling and bandits with het-326 eroscedastic noise. In Conference On Learning Theory. PMLR.327
LEE, C.-W., LUO, H., WEI, C.-Y., ZHANG, M. and ZHANG, X. (2021). Achieving near instance-328 optimality and minimax-optimality in stochastic and adversarial linear bandits simultaneously.329 arXiv preprint arXiv:2102.05858 .330
LI, L., CHU, W., LANGFORD, J. and SCHAPIRE, R. E. (2010). A contextual-bandit approach to331 personalized news article recommendation. In Proceedings of the 19th international conference on332 World wide web.333
LI, Y., LOU, E. Y. and SHAN, L. (2019a). Stochastic linear optimization with adversarial corruption.334 arXiv preprint arXiv:1909.02109 .335
LI, Y., WANG, Y. and ZHOU, Y. (2019b). Nearly minimax-optimal regret for linearly parameterized336 bandits. In Conference on Learning Theory. PMLR.337
LIU, F. and SHROFF, N. (2019). Data poisoning attacks on stochastic bandits. In International338 Conference on Machine Learning. PMLR.339
LYKOURIS, T., MIRROKNI, V. and PAES LEME, R. (2018). Stochastic bandits robust to adversarial340 corruptions. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing.341
NEU, G. and OLKHOVSKAYA, J. (2020). Efficient and robust algorithms for adversarial linear342 contextual bandits. In Conference on Learning Theory. PMLR.343
RUSMEVICHIENTONG, P. and TSITSIKLIS, J. N. (2010). Linearly parameterized bandits. Mathemat-344 ics of Operations Research 35 395–411.345
SELDIN, Y. and LUGOSI, G. (2017). An improved parametrization and analysis of the exp3++346 algorithm for stochastic and adversarial bandits. In Conference on Learning Theory. PMLR.347
SELDIN, Y. and SLIVKINS, A. (2014). One practical algorithm for both stochastic and adversarial348 bandits. In International Conference on Machine Learning. PMLR.349
VILLAR, S. S., BOWDEN, J. and WASON, J. (2015). Multi-armed bandit models for the optimal350 design of clinical trials: benefits and challenges. Statistical science: a review journal of the351 Institute of Mathematical Statistics 30 199.352
ZHOU, D., GU, Q. and SZEPESVARI, C. (2020). Nearly minimax optimal reinforcement learning for353 linear mixture markov decision processes. arXiv preprint arXiv:2012.08507 .354
ZIMMERT, J. and SELDIN, Y. (2019). An optimal algorithm for stochastic and adversarial bandits.355 In The 22nd International Conference on Artificial Intelligence and Statistics. PMLR.356
Checklist357
1. For all authors...358 (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s359 contributions and scope? [Yes]360 (b) Did you describe the limitations of your work? [Yes]361 (c) Did you discuss any potential negative societal impacts of your work? [N/A] Our work362 studies the regret bounds for contextual linear bandits with corruption. That is a pure363 theoretical problem, thus it does not have any negative social impact.364
(d) Have you read the ethics review guidelines and ensured that your paper conforms to365 them? [Yes]366
2. If you are including theoretical results...367 (a) Did you state the full set of assumptions of all theoretical results? [Yes]368 (b) Did you include complete proofs of all theoretical results? [Yes]369
3. If you ran experiments...370 (a) Did you include the code, data, and instructions needed to reproduce the main experi-371 mental results (either in the supplemental material or as a URL)? [N/A]372 (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they373 were chosen)? [N/A]374 (c) Did you report error bars (e.g., with respect to the random seed after running experi-375 ments multiple times)? [N/A]376 (d) Did you include the total amount of compute and the type of resources used (e.g., type377 of GPUs, internal cluster, or cloud provider)? [N/A]378 4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...379
(a) If your work uses existing assets, did you cite the creators? [N/A]380 (b) Did you mention the license of the assets? [N/A]381 (c) Did you include any new assets either in the supplemental material or as a URL? [N/A]382
383
(d) Did you discuss whether and how consent was obtained from people whose data you’re384 using/curating? [N/A]385
(e) Did you discuss whether the data you are using/curating contains personally identifiable386 information or offensive content? [N/A]387
5. If you used crowdsourcing or conducted research with human subjects...388 (a) Did you include the full text of instructions given to participants and screenshots, if389 applicable? [N/A]390 (b) Did you describe any potential participant risks, with links to Institutional Review391 Board (IRB) approvals, if applicable? [N/A]392 (c) Did you include the estimated hourly wage paid to participants and the total amount393 spent on participant compensation? [N/A]394

Review summaries:
The paper studies the problem of linear contextual bandits where the rewards of the actions are not always stochastic but rather they are corrupted in C of the T rounds. This model of adversarial corruptions is well understood in the multi-armed bandit setting. The paper combines the OFUL algorithm that works for linear contextual bandits without corruptions with a multi-level algorithm that is robust to corruptions for the multi-armed bandit setting. Unlike multi-armed bandits, in linear contextual bandits, the action space is not finite and therefore techniques that are based on eliminating actions after selecting them enough times are not as direct. Moreover, the variance of the estimator can be changing over time which is dealt with via the use of a weighted ridge regression estimator. The resulting guarantee of Theorem 5.1 matches the regret bound of OFUL when C is constant and adapts to the corruption level C. 

This paper studies the linear contextual bandit problem with adversarial noises in the reward function. The authors assume that the decision set can change over time; that reward can be corrupted up to a corruption level (similar to previous work); and that an upper-bound on the true variance of the reward is revealed after each instance. An algorithm, which is based on the linear bandit algorithm OFUL with two notable modifications, is proposed for the problem. First, it considers multiple levels of possible corruption noise to deal with unknown value of C and second it considers a weighted version of the ridge regression estimator to deal with varying variances (this modification was already present in previous work). The authors prove a regret upper-bound for their algorithm and compare it to the literature. They also provide some experiments in the supplementary material.

The authors study  a linear contextual bandit problem in the presence of adversarial corruptions that are bounded by an unknown C.  Specifically, they propose an algorithm that uses a parallel-learning idea from Lykouris et. al, 2019 using multi-level partition of the observed data.  To deal with varying variance in the observations, they use a ridge regression estimator similar to the earlier work of Kirschner and Krause, 2018; and Zhou et al., 2020.   Finally, they show their algorithm produces a regret that is quadratic in C and the dimensionality of the context vectors. 

This paper proposes an algorithm that achieves the first corruption-robust result for contextual linear stochastic bandits with infinite arms using the idea of multi-level sampling rates. The regret bound has multiplicative dependence on C^2 and is variance-aware. The algorithm recovers the optimal regret in terms of T and gap when no corruptions are presented.