Title: Robust Lipschitz Bandits to Adversarial Corruptions
Venue: NeurIPS 2023
Accepted: yes
Keywords: bandits
Abstract: Lipschitz bandit is a variant of stochastic bandits that deals with a continuous arm set defined on a metric space, where the reward function is subject to a Lipschitz constraint. In this paper, we introduce a new problem of Lipschitz bandits in the presence of adversarial corruptions where an adaptive adversary corrupts the stochastic rewards up to a total budget $C$. The budget is measured by the sum of corruption levels across the time horizon $T$. We consider both weak and strong adversaries, where the weak adversary is unaware of the current action before the attack, while the strong one can observe it. Our work presents the first line of robust Lipschitz bandit algorithms that can achieve sub-linear regret under both types of adversary, even when the total budget of corruption $C$ is unrevealed to the agent. We provide a lower bound under each type of adversary, and show that our algorithm is optimal under the strong case. Finally, we conduct experiments to illustrate the effectiveness of our algorithms against two classic kinds of attacks.

Robust Lipschitz Bandits to Adversarial Corruptions

Abstract

Lipschitz bandit is a variant of stochastic bandits that deals with a continuous arm set defined on a metric space, where the reward function is subject to a Lipschitz constraint. In this paper, we introduce a new problem of Lipschitz bandits in the presence of adversarial corruptions where an adaptive adversary corrupts the stochastic rewards up to a total budget C . The budget is measured by the sum of corruption levels across the time horizon T . We consider both weak and strong adversaries, where the weak adversary is unaware of the current action before the attack, while the strong one can observe it. Our work presents the first line of robust Lipschitz bandit algorithms that can achieve sub-linear regret under both types of adversary, even when the total budget of corruption C is unrevealed to the agent. We provide a lower bound under each type of adversary, and show that our algorithm is optimal under the strong case. Finally, we conduct experiments to illustrate the effectiveness of our algorithms against two classic kinds of attacks.

1 Introduction

Multi-armed Bandit (MAB) [3] is a fundamental and powerful framework in sequential decisionmaking problems. Given the potential existence of malicious users in real-world scenarios [10], a recent line of works considers the stochastic bandit problem under adversarial corruptions: an agent adaptively updates its policy to choose an arm from the arm set, and an adversary may contaminate the reward generated from the stochastic bandit before the agent could observe it. To robustify bandit learning algorithms under adversarial corruptions, several algorithms have been developed in the setting of traditional MAB [16, 18, 29] and contextual linear bandits [6, 12, 17, 27, 39]. These works consider either the weak adversary [29], which has access to all past data but not the current action before choosing its attack, or the strong adversary [6], which is also aware of the current action for contamination. Details of these two adversaries will be elaborated in Section 3. In practice, bandits under adversarial corruptions can be used in many real-world problems such as pay-per-click advertising with click fraud and recommendation systems with fake reviews [29], and it has been empirically validated that stochastic MABs are vulnerable to slight corruption [12, 15, 18].

Although there has been extensive research on the adversarial robustness of stochastic bandits, most existing works consider problems with a discrete arm set, such as the traditional MAB and contextual linear bandit. In this paper, we investigate robust bandit algorithms against adversarial corruptions in the Lipschitz bandit setting, where a continuously infinite arm set lie in a known metric space

with covering dimension d and the expected reward function is an unknown Lipschitz function. Lipschitz bandit can be used to efficiently model many real-world tasks such as dynamic pricing, auction bidding [35] and hyperparameter tuning [19]. The stochastic Lipschitz bandit has been well understood after a large body of literature [8, 23, 30], and state-of-the-art algorithms could achieve a cumulative regret bound of order ˜ O ( T dz +1 dz +2 ) 1 in time T . However, to the best of our knowledge, the stochastic Lipschitz bandit problem with adversarial corruptions has never been explored, and we believe it is challenging since most of the existing robust MAB algorithms utilized the idea of elimination, which is much more difficult under a continuously infinite arm pool. Furthermore, the complex structure of different metric spaces also poses challenges for defending against adversarial attacks [36] in theory. Therefore, it remains intriguing to design computationally efficient Lipschitz bandits that are robust to adversarial corruptions under both weak and strong adversaries.

We develop efficient robust algorithms whose regret bounds degrade sub-linearly in terms of the corruption budget C . Our contributions can be summarized as follows: (1) Under the weak adversary, we extend the idea in [29] and propose an efficient algorithm named Robust Multi-layer Elimination Lipschitz bandit algorithm (RMEL) that is agnostic to C and attains ˜ O ( C 1 dz +2 T dz +1 dz +2 ) regret bound. This bound matches the minimax regret bound of Lipschitz bandits [8, 23] in the absence of corruptions up to logarithmic terms. This algorithm consists of multiple parallel sub-layers with different tolerance against the budget C , where each layer adaptively discretizes the action space and eliminates some less promising regions based on its corruption tolerance level in each crafted epoch. Interactions between layers assure the promptness of the elimination process. (2) Under the strong adversary, we first show that when the budget C is given, a simple modification on the classic Zooming algorithm [23] would lead to a robust method, namely, Robust Zooming algorithm, which could obtain a regret bound of order ˜ O ( T dz +1 dz +2 + C 1 dz +1 T dz dz +1 ) . We then provide a lower bound to prove the extra O ( C 1 dz +1 T dz dz +1 ) regret is unavoidable. Further, inspired by the Bandit-over-Bandit (BoB) model selection idea [11, 13, 31], we design a two-layer framework adapting to the unknown C where a master algorithm in the top layer dynamically tunes the corruption budget for the Robust Zooming algorithm. Three types of master algorithms are discussed and compared in both theory and practice. Table 1 outlines our algorithms as well as their regret bounds under different scenarios.

2 Related Work

Stochastic and Adversarial Bandit Extensive studies have been conducted on MAB and its variations, including linear bandit [1], matrix bandit [20], etc. The majority of literature can be categorized into two types of models [24]: stochastic bandit, in which rewards for each arm are independently sampled from a fixed distribution, and adversarial bandit, where rewards are maliciously generated at all time. However, adversarial bandit differs from our problem setting in the sense that rewards are arbitrarily chosen without any budget or distribution constraint. Another line of work aims to obtain 'the best of both worlds' guarantee simultaneously [7]. However, neither of these models is reliable in practice [9], since the former one is too ideal, while the latter one remains very pessimistic, assuming a fully unconstrained setting. Therefore, it is more natural to consider the scenario that lies "in between" the two extremes: the stochastic bandit under adversarial corruptions.

Lipschitz Bandit Most existing works on the stochastic Lipschitz bandit [2] follow two key ideas. One is to uniformly discretize the action space into a mesh in the initial phase so that any MAB algorithm could be implemented [21, 30]. The other is to adaptively discretize the action space by placing more probes in more promising regions, and then UCB [8, 23, 28], TS [19] or elimination [14] method could be utilized to deal with the exploration-exploitation tradeoff. The adversarial Lipschitz bandit was recently introduced and solved in [32], where the expected reward Lipschitz function is arbitrarily chosen at each round. However, as mentioned in the previous paragraph, this fully adversarial setting is quite different from ours. And their algorithm relies on several unspecified hyperparameters and hence is computationally formidable in practice. In addition, some interesting variations of Lipschitz bandits have also been carefully examined, such as contextual Lipschitz bandits [33], full-feedback Lipschitz bandits [22], and taxonomy bandits [34].

Robust Bandit to Adversarial Corruptions Adversarial attacks were studied in the setting of MAB [18] and linear bandits [15]. And we will use two classic attacks for experiments in Section 6. To defend against attacks from weak adversaries, [29] proposed the first MAB algorithm robust to corruptions with a regret C times worse than regret in the stochastic setting. An improved algorithm whose regret only contains an additive term on C was then proposed in [16]. [27] subsequently studied the linear bandits with adversarial corruptions and achieved instance-dependent regret bounds. [25] also studied the corrupted linear bandit problem while assuming the attacks on reward are linear in action. Recently, a robust VOFUL algorithm achieving regret bound only logarithmically dependent on T was proposed in [37]. Another line of work on the robust bandit problem focuses on a more challenging setting with strong adversaries who could observe current actions before attacking rewards. [6] considered the corrupted linear bandit when small random perturbations are applied to context vectors, and [12, 17, 39] extended the OFUL algorithm [1] and achieved improved regret bounds. [38] further considers general non-linear contextual bandits and also MDPs with strong adversarial corruptions. However, the study of Lipschitz bandits under attacks remains an unaddressed open area.

3 Preliminaries

We will introduce the setting of Lipschitz bandits with adversarial corruptions in this section. The Lipschitz bandit is defined on a triplet ( X , D, µ ) , where X is the arm set space equipped with some metric D , and µ : X → R is an unknown Lipschitz reward function on the metric space ( X , D ) with Lipschitz constant 1 . W.l.o.g. we assume X is compact with its diameter no more than 1 . Under the stochastic setting, at each round t ∈ [ T ] := { 1 , 2 , . . . , T } , stochastic rewards are sampled for each arm x ∈ X from some unknown distribution P x independently, and then the agent pulls an arm x t and receives the corresponding stochastic reward ˜ y t such that,

where η t is i.i.d. zero-mean random error with sub-Gaussian parameter σ conditional on the filtration F t = { x t , x t -1 , η t -1 , . . . , x 1 , η 1 } . W.l.o.g we assume σ = 1 for simplicity in the rest of our analysis. At each round t ∈ [ T ] , the weak adversary observes the payoff function µ ( · ) , the realizations of P x for each arm x ∈ X and choices of the agent { x i } t -1 i =1 in previous rounds, and injects an attack c t ( x t ) into the reward before the agent pulls x t . The agent then receives a corrupted reward y t = ˜ y t + c t ( x t ) . The strong adversary would be omniscient and have complete information about the problem F t . In addition to the knowledge that a weak adversary possesses, it would also be aware of the current action x t while contaminating the data, and subsequently decide upon the corrupted reward y t = ˜ y t + c t ( x t ) . Some literature in corrupted bandits [12, 15] also consider attacking on the contexts or arms, i.e. the adversary modifies the true arm x t in a small region, while in our problem setting it is obvious that attacking contexts is only a sub-case of attacking rewards due to the Lipschitzness of µ ( · ) , and hence studying the adversarial attacks on rewards alone is sufficient under the Lipschitz bandit setting.

The total corruption budget C of the adversary is defined as C = ∑ T t =1 max x ∈X | c t ( x ) | , which is the sum of maximum perturbation from the adversary at each round across the horizon T . Note the strong adversary may only corrupt the rewards of pulled arms and hence we could equivalently write C = ∑ T t =1 | c t ( x t ) | in that case as [6, 17]. Define the optimal arm x ∗ = arg max x ∈X µ ( x ) and the loss of arm x as ∆( x ) = µ ( x ∗ ) -µ ( x ) , x ∈ X . W.l.o.g. we assume C ≤ T and each instance of attack | c t ( x ) | ≤ 1 , ∀ t ∈ [ T ] , x ∈ X as in other robust bandit literature [16, 29] since

Algorithm 1 Robust Zooming Algorithm

the adversary could already make any arm x ∈ X optimal given that ∆( x ) ≤ 1 . (We can assume | c t ( x ) | ≤ u, ∀ t ∈ [ T ] , x ∈ X for any positive constant u .) Similar to the stochastic case [21], the goal of the agent is to minimize the cumulative regret defined as:

An important pair of concepts in Lipschitz bandits defined on ( X , D, µ ) are the covering dimension d and the zooming dimension d z . Let B ( x, r ) denotes a closed ball centered at x with radius r in X , i.e. B ( x, r ) = { x ′ ∈ X : D ( x, x ′ ) ≤ r } , the r -covering number N c ( r ) of metric space ( X , D ) is defined as the minimal number of balls with radius of no more than r required to cover X . On the contrary, the r -zooming number N z ( r ) introduced in [23] not only depends on the metric space ( X , D ) but also the payoff function µ ( · ) . It describes the minimal number of balls of radius not more than r/ 16 required to cover the r -optimal region defined as { x ∈ X : ∆( x ) ≤ r } [8, 14] 2 . Next, we define the covering dimension d (zooming dimension d z ) as the smallest q ≥ 0 such that for every r ∈ (0 , 1] the r -covering number N c ( r ) ( r -zooming number N z ( r ) ) can be upper bounded by αr -q for some multiplier α > 0 that is free of r :

It is clear that 0 ≤ d z ≤ d since the r -optimal region is a subset of X . On the other hand, d z could be much smaller than d in some benign cases. For example, if the payoff function µ ( · ) defined on the metric space ( R k , ∥·∥ 2 ) , k ∈ N is C 2 -smooth and strongly concave in a neighborhood of the optimal arm x ∗ , then it could be easily verified that d z = k/ 2 whereas d = k . However, d z is never revealed to the agent as it relies on the underlying function µ ( · ) , and hence designing an algorithm whose regret bound depends on d z without knowledge of d z would be considerably difficult.

4 Warm-up: Robust Lipschitz Bandit with Known Budgets

To defend against attacks on Lipschitz bandits, we first consider a simpler case where the agent is aware of the corruption budget C . We demonstrate that a slight modification of the classic Zooming algorithm [23] can result in a robust Lipschitz bandit algorithm even under the strong adversary, called the Robust Zooming algorithm, which achieves a regret bound of order ˜ O ( T dz +1 dz +2 + C 1 dz +1 T dz dz +1 ) .

We first introduce some notations of the algorithm: denote J as the active arm set. For each active arm x ∈ J , let n ( x ) be the number of times arm x has been pulled, f ( x ) be the corresponding average sample reward, and r ( x ) be the confidence radius controlling the deviation of the sample average

f ( x ) from its expectation µ ( x ) . We also define B ( x, r ( x )) as the confidence ball of an active arm x . In essence, the Zooming algorithm works by focusing on regions that have the potential for higher rewards and allocating fewer probes to less promising regions. The algorithm consists of two phases: in the activation phase, a new arm gets activated if it is not covered by the confidence balls of all active arms. This allows the algorithm to quickly zoom into the regions where arms are frequently pulled due to their encouraging rewards. In the selection phase, the algorithm chooses an arm with the largest value of f ( v ) + 2 r ( v ) among J based on the UCB methodology.

Our key idea is to enlarge the confidence radius of active arms to account for the known corruption budget C . Specifically, we could set the value of r ( x ) as:

where the first term accounts for deviation in stochastic rewards and the second term is used to defend the corruptions from the adversary. The robust algorithm is shown in Algorithm 1. In addition to the two phases presented above, our algorithm also conducts a removal procedure at the beginning of each round for better efficiency. This step adaptively removes regions that are likely to yield low rewards with high confidence. Theorem 4.1 provides a regret bound for Algorithm 1.

Theorem 4.1. Given the total corruption budget that is at most C , with probability at least 1 -δ , the overall regret of Robust Zooming Algorithm (Algorithm 1) can be bounded as:

Furthermore, the following Theorem 4.2 implies that our regret bound attains the lower bound and hence is unimprovable. The detailed proof is given in Appendix A.6.1.

Theorem 4.2. Under the strong adversary with a corruption budget of C , for any zooming dimension d z ∈ Z + , there exists an instance for which any algorithm (even one that is aware of C ) must suffer a regret of order Ω( C 1 dz +1 T dz dz +1 ) with probability at least 0 . 5 .

In addition to the lower bound provided in Theorem 4.2, we further propose another lower bound for the strong adversary particularly in the case that C is unknown in the following Theorem 4.3:

Theorem 4.3. For any algorithm, when there is no corruption, we denote R 0 T as the upper bound of cumulative regret in T rounds under our problem setting described in Section 3, i.e. Regret T ≤ R 0 T with high probability, and it holds that R 0 T = o ( T ) . Then under the strong adversary and unknown attacking budget C , there exists a problem instance on which this algorithm will incur linear regret Ω( T ) with probability at least 0 . 5 , if C = Ω( R 0 T / 4 d z ) = Ω( R 0 T ) .

However, there are also some weaknesses to our Algorithm 1. The first weakness is that the algorithm is too conservative and pessimistic in practice since the second term of r ( x ) would dominate under a large given value of C . We could set the second term of r ( x ) as min { 1 , C/n ( x ) } to address this issue and the analysis of Theorem 4.1 will still hold as shown in Appendix A.1 Remark A.5. The second weakness is that it still incurs the same regret bound shown in Theorem 4.1 even if there are actually no corruptions applied. To overcome these problems and to further adapt to the unknown corruption budget C , we propose two types of robust algorithms in the following Section 5.

5 Robust Lipschitz Bandit with Unknown Budgets

In practice, a decent upper bound of the corruption budget C would never be revealed to the agent, and hence it is important to develop robust Lipschitz bandit algorithms agnostic to the amount of C . In this section, we first propose an efficient adaptive-elimination-based approach to deal with the weak adversary, and then we adapt our Algorithm 1 by using several advances of model selection in bandits to defend against the attacks from the strong adversary.

5.1 Algorithm for Weak Adversaries

The weak adversary is unaware of the agent's current action before contaminating the stochastic rewards. We introduce an efficient algorithm called Robust Multi-layer Elimination Lipschitz bandit algorithm (RMEL) that is summarized in Algorithm 2. Four core steps are introduced as follows.

Algorithm 2 Robust Multi-layer Elimination Lipschitz Bandit Algorithm (RMEL)

Input: Arm metric space ( X , D ) , time horizon T , probability rate δ , base parameter B . v = ln(4 T/δ ) B l -1 , m = 1 , n = 0 , A = 1 / 2

̸

Initialization: Tolerance level l l l l -covering of X , f l,A = n l,A = 0 for all A ∈ A l , l ∈ [ l ∗ ] where l ∗ := min { l ∈ N : ln (4 T/δ ) B l -1 ≥ T } . 1: for t = 1 to T do 2: Sample layer l ∈ [ l ∗ ] with probability 1 /v l , with the remaining probability sampling l = 1 . Find the minimum layer index l t ≥ l such that A l t = ∅ . ▷ Layer sampling 3: Choose A t = arg min A ∈A l t n l t ,A , break ties arbitrary. 4: Randomly pull an arm x t ∈ A t , and observe the payoff y t . 5: Set n l t = n l t +1 , n l t ,A t = n l t ,A t +1 , and f l t ,A t = ( f l t ,A t ( n l t ,A t -1) + y t ) /n l t ,A t . 6: if n l t = 6ln(4 T/δ ) · 4 m l ×|A l t | then 7: Obtain f l t , ∗ = max A ∈A l t f l t ,A . 8: For each A ∈ A l t , if f l t , ∗ -f l t ,A > 4 / 2 m l t , then we eliminate A from A l t and all active regions A ′ from A l ′ in the case that A ′ ⊆ A,A ′ ∈ A l ′ , l ′ < l . ▷ Removal 9: Find 1 / 2 m l +1 -covering of each remaining A ∈ A l t in the same way as A was partitioned in other layers. Then reload the active region set A l t as the collection of these coverings. 10: Set n l t = 0 , m l t = m l t +1 . And renew n l t ,A = f l t ,A = 0 , ∀ A ∈ A l t . ▷ Refresh 11: end if 12: end for

Multi-layer Parallel Running: Inspired by the multi-layer idea used in robust MABs [29], our algorithm consists of multiple sub-layers running in parallel, each with a different tolerance level against corruptions. As shown in Algorithm 2, there are l ∗ layers and the tolerance level of each layer, denoted as v l , increases geometrically with a ratio of B (a hyperparameter). At each round, a layer l is sampled with probability 1 /v l , meaning that layers that are more resilient to attacks are less likely to be chosen and thus may make slower progress. This sampling scheme helps mitigate adversarial perturbations across layers by limiting the amount of corruptions distributed to layers whose tolerance levels exceed the unknown budget C to at most O (ln( T/δ )) . For the other low-tolerance layers which may suffer from high volume of attacks, we use the techniques introduced below to rectify them in the guidance of the elimination procedure on robust layers. While we build on the multi-layer idea introduced in [29], our work introduces significant refinements and novelty by extending this approach to continuous and infinitely large arm sets, as demonstrated below.

Active Region Mechanism: For each layer ℓ , our algorithm proceeds in epochs: we initialize the epoch index m l = 1 and construct a 1 / 2 m l -covering of X as the active region set A l . In addition, we denote n l as the number of times that layer l has been chosen, and for each active region A ∈ A l we define n l,A , f l,A as the number of times A has been chosen as well as its corresponding average empirical reward respectively. Assume layer l t is selected at time t , then only one active region (denoted as A t ) in A l t would be played where we arbitrarily pull an arm x t ∈ A t and collect the stochastic payoff y t . For any layer l , if each active region in A l is played for 6 ln (4 T/δ ) · 4 m times (i.e. line 6 of Algorithm 2), it will progress to the next epoch after an elimination process that is described below. All components mentioned above that are associated with the layer l will subsequently be refreshed (i.e. line 10 of Algorithm 2).

Although the elimination idea has been used for various bandit algorithms [35], applying it to our problem setting poses significant challenges due to three sources of error: (1) uncertainty in stochastic rewards; (2) unknown corruptions assigned to each layer; (3) approximation bias between the pulled arm and its neighbors in the same active region. We encapsulate our carefully designed elimination procedure that is specified in line 8 of Algorithm 2 from two aspects:

Within-layer Region Elimination and Discretization: For any layer l ∈ [ l ∗ ] , the within-layer elimination occurs at the end of each epoch as stated above. We obtain the average empirical reward f l,A for all A ∈ A l and then discard regions with unpromising payoffs compared with the optimal one with the maximum estimated reward (i.e. f l, ∗ defined in line 7 of Algorithm 2). We further 'zoom in' on the remaining regions of the layer l that yield satisfactory rewards: we divide them into 1 / 2 m l +1 -covering and then reload A l as the collection of these new partitions for the next epoch (line 9 of Algorithm 2) for the layer l . In consequence, only regions with nearly optimal rewards would remain and be adaptively discretized in the long run.

Cross-layer Region Elimination: While layers are running in parallel, it is essential to facilitate communication among them to prevent less reliable layers from getting trapped in suboptimal regions. In our Algorithm 2, if an active region A ∈ A l is eliminated based on the aforementioned rule, then A will also be discarded in all layers l ′ ≤ l . This is because the lower layers are faster whereas more vulnerable and less resilient to malicious attacks, and hence they should learn from the upper trustworthy layers whose tolerance levels surpass C by imitating their elimination decisions.

Note that the elimination step (line 8 of Algorithm 2) could be executed either after each epoch or after each round. The former, which is used in the current Algorithm 2, is computationally simpler as it does not require the entire elimination procedure to be repeated each time. On the other hand, the latter version is more precise as it can identify and eliminate sub-optimal regions more quickly. We defer the pseudocode and more details of the latter version in Appendix A.5 due to the space limit. Another tradeoff lies in the selection of the hyperparameter B , which controls the ratio of tolerance levels between adjacent layers. With a larger value of B , only fewer layers are required, and hence more samples could be assigned to each layer for better efficiency. But the cumulative regret bound would deteriorate since it's associated with B sub-linearly. The cumulative regret bound is presented in the following Theorem 5.1, with its detailed proof in Appendix A.2.

Theorem 5.1. If the underlying corruption budget is C , then with probability at least 1 -δ , the overall regret of our RMEL algorithm (Algorithm 2) could be bounded as:

Proof sketch: It is highly non-trivial to deduce the regret bound depending on C and d z in Theorem 5.1 without the knowledge of these two values. We first bound the cumulative regret occurred in the robust layers whose tolerance levels v l are no less than the unknown C by showing the estimated mean f l,A is close to the underlying ground truth µ ( x ) , x ∈ A for all time t and active regions A simultaneously with high probability, i.e. we define the set Φ as follows and prove P (Φ) ≥ 1 -3 δ/ 4 .

This concentration result can guarantee that regions with unfavorable rewards will be adaptively eliminated. And we could show that the extra cumulative regret from the other vulnerable layers be controlled by the regret occurred in the robust layers. A detailed proof is presented in Appendix A.2.

Note that if no corruption is actually applied (i.e. C = 0 ), our RMEL algorithm could attain a regret bound of order ˜ O ( T dz +1 dz +2 ) which coincides with the lower bound of stochastic Lipschitz bandits up to logarithmic terms. We further prove a regret lower bound of order Ω( C ) under the weak adversary in Theorem 5.2 with its detailed proof in Appendix A.6.2. Therefore, a compelling open problem is to narrow the regret gap by proposing an algorithm whose regret bound depends on C in another additive term free of T under the weak adversary, like [16] for MABs and [17] for linear bandits.

Theorem 5.2. Under the weak adversary with corruption budget C , for any zooming dimension d z , there exists an instance such that any algorithm (even is aware of C ) must suffer from the regret of order Ω( C ) with probability at least 0 . 5 .

5.2 Algorithm for Strong Adversaries

In Section 4, we propose the Robust Zooming algorithm to handle the strong adversary given the knowledge of budget C and prove that it achieves the optimal regret bound. However, compared with the known budget C case, defending against strong adversaries naturally becomes more challenging when the agent is unaware of the budget C . Motivated by the literature on model selection in bandits, we extend our Robust Zooming algorithm by combining it with different master algorithms to learn and adapt to the unknown C on the fly. We consider two approaches along this line: the first approach uses the master algorithms EXP3.P and CORRAL with the smoothing transformation [31] to deal with unknown C , which leads to a promising regret bound but a high computational cost. We then equip Robust Zooming algorithm with the efficient bandit-over-bandit (BoB) idea [11] to adapt to the unknown C , leading to a more efficient algorithm with a slightly worse regret bound.

Model Selection: When an upper bound on C is known, we propose the Robust Zooming algorithm with regret bound ˜ O ( T dz +1 dz +2 + C 1 dz +1 T dz dz +1 ) against strong adversaries in Section 4. Therefore, it is natural to consider a decent master algorithm that selects between ⌈ log 2 ( T ) ⌉ base algorithms where the i -th base algorithm is the Robust Zooming algorithm with corruptions at most 2 i . As C ≤ T , there must exist a base algorithm that is at most 2 C -corrupted. Here we choose the stochastic EXP3.P and CORRAL with smooth transformation proposed in [31] as the master algorithm due to the following two reasons with respect to theoretical analysis: (1). our action set A is fixed and the expected payoff is a function of the chosen arm, which satisfies the restrictive assumptions of this master algorithm (Section 2, [31]); (2). the analysis in [31] still works even the regret bounds of base algorithms contain unknown values, and note the regret bound of our Zooming Robust algorithm depends on the unknown C . Based on Theorem 3.2 in [31], the expected cumulative regret of our Robust Zooming algorithm with these two types of master algorithms could be bounded as follows:

Theorem 5.3. When the corruption budget C is unknown, by using our Algorithm 1 with { 2 i } ⌈ log 2 ( T ) ⌉ i =1 corruptions as base algorithms and the EXP3.P and CORRAL with smooth transformation [31] as the master algorithm, the expected regret could be upper bounded by

We can observe that the regret bounds given in Theorem 5.3 are consistent with the lower bounds presented in Theorem 4.3. And CORRAL is better under small corruption budgets C (i.e. C = ˜ O ( T d +1 d +3 ) ) whereas EXP3.P is superior otherwise. Note that the order of regret relies on d instead of d z since the unknown d z couldn't be used as a parameter in practice, and both regret bounds are worse than the lower bound given in Theorem 4.2 for the strong adversary. Another drawback of the above method is that a two-step smoothing procedure is required at each round, which is computationally expensive. Therefore, for better practical efficiency, we propose a simple BoB-based method as follows:

BoB Robust Zooming: The BoB idea [11] is a special case of model selection in bandits and aims to adjust some unspecified hyperparameters dynamically in batches. Here we use ⌈ log 2 ( T ) ⌉ Robust Zooming algorithms with different corruption levels shown above as base algorithms in the bottom layer and the classic EXP3.P [4] as the top layer. Our method, named BoB Robust Zooming, divides T into H batches of the same length, and in one batch keeps using the same base algorithm that is selected from the top layer at the beginning of this batch. When a batch ends, we refresh the base algorithm and use the normalized accumulated rewards of this batch to update the top layer EXP3.P since the EXP3.P algorithm [4] requires the magnitude of rewards should at most be 1 in default. Specifically, we normalize the cumulative reward at the end of each batch by dividing it with (2 H + √ 2 H log(12 T/Hδ )) due to the fact that the magnitude of the cumulative reward at each batch would at most be this value with high probability as shown in Lemma A.14 in Appendix A.4. Note that this method is highly efficient since a single update of the EXP3.P algorithm only requires O (1) time complexity, and hence the additional computation from updating EXP3.P is only O ( H ) . Due to space limit, we defer Algorithm 4 to Appendix A.5, and the regret bound is given as follows:

Theorem 5.4. When the corruption budget C is unknown, with probability at least 1 -δ , the regret of our BoB Robust Zooming algorithm with H = T ( d +2) / ( d +4) could be bounded as:

Although we could deduce the more challenging high-probability regret bound for this algorithm, its order is strictly worse than those given in Theorem 5.3. In summary, the BoB Robust Zooming algorithm is more efficient and easier to use in practice, while yielding worse regret bound in theory. However, due to its practical applicability, we will implement this BoB Robust Zooming algorithm in the experiments. It is also noteworthy that we can attain a better regret bound with Algorithm 2 under the weak adversary as shown in Theorem 5.1, which aligns with our expectation since the strong adversary considered here is more malicious and difficult to defend against.

0

10000

20000

30000

40000

50000

Iterations

0

500

1000

1500

2000

Cumulative Regret

Triangle mean function under Oracle attack

Zooming

RMEL

0

10000

20000

30000

40000

50000

Iterations

0

500

1000

1500

2000

2500

3000

Cumulative Regret

Sine mean function under Oracle attack

Zooming

RMEL

0

10000 20000 30000 40000 50000 60000

Iterations

0

2000

4000

6000

8000

Cumulative Regret

Two dim mean function under Oracle attack

Zooming

RMEL

0

10000

20000

30000

40000

50000

Iterations

0

500

1000

1500

2000

2500

Cumulative Regret

Triangle mean function under Garcelon attack

Zooming

RMEL

0

10000

20000

30000

40000

50000

Iterations

0

250

500

750

1000

1250

1500

1750

2000

Cumulative Regret

Sine mean function under Garcelon attack

Zooming

RMEL

0

10000 20000 30000 40000 50000 60000

Iterations

0

1000

2000

3000

4000

5000

6000

7000

8000

Cumulative Regret

Two dim mean function under Garcelon attack

Zooming

RMEL

6 Experiments

In this section, we show by simulations that our proposed RMEL and BoB Robust Zooming algorithm outperform the classic Zooming algorithm in the presence of adversarial corruptions. To firmly validate the robustness of our proposed methods, we use three types of models and two sorts of attacks with different corruption levels. We first consider the metric space ([0 , 1] , | · | ) with two expected reward functions that behave differently around their maximum: (1). µ ( x ) = 0 . 9 -0 . 95 | x -1 / 3 | (triangle) and (2). µ ( x ) = 2 / (3 π ) · sin (3 πx/ 2) (sine). We then utilize a more complicated metric space ([0 , 1] 2 , ∥·∥ ∞ ) with the expected reward function (3). µ ( x ) = 1 -0 . 8 ∥ x -(0 . 75 , 0 . 75) ∥ 2 -0 . 4 ∥ x -(0 , 1) ∥ 2 (two dim). We set the time horizon T = 50 , 000 (60 , 000) for the metric space with d = 1(2) and the false probability rate δ = 0 . 01 . The random noise at each round is sampled IID from N (0 , 0 . 01) . Average cumulative regrets over 20 repetitions are reported in Figure 1.

Since adversarial attacks designed for stochastic Lipschitz bandits have never been studied, we extend two types of classic attacks, named Oracle [18] for the MAB and Garcelon [15] for the linear bandit, to our setting. The details of these two attacks are summarized as follows:

Oracle: This attack [18] was proposed for the traditional MAB, and it pushes the rewards of 'good arms' to the very bottom. Specifically, we call an arm is benign if the distance between it and the optimal arm is no larger than 0 . 2 . And we inject this attack by pushing the expected reward of any benign arm below that of the worst arm with an additional margin of 0 . 1 with probability 0 . 5 .

Garcelon: We modify this type of attack studied in [15] for linear bandit framework, which replaces expected rewards of arms outside some targeted region with IID Gaussian noise. For d = 1 , since the optimal arm is set to be 1 / 3 for both triangle and sine payoff functions, we set the targeted arm interval as [0 . 5 , 1] . For d = 2 , since the optimal arm is close to (0 . 75 , 0 . 75) , we set the targeted region as [0 , 0 . 5] 2 . Here we contaminate the stochastic reward if the pulled arm is not inside the target region by modifying it into a random Gaussian noise N (0 , 0 . 01) with probability 0 . 5 .

We consider the strong adversary in experiments as both types of attack are injected only if the pulled arms lie in some specific regions. Note although we originally propose RMEL algorithm for the weak adversary in theory, empirically we find it works exceptionally well (Figure 1) across all settings here. We also conduct simulations based on the weak adversary and defer their settings and results to Appendix A.7 due to the limited space. The first Oracle attack is considered to be more malicious in the sense that it specifically focuses on the arms with good rewards, while the second Garcelon attack could corrupt rewards generated from broader regions, which may contain some 'bad arms' as well.

Since there is no existing robust Lipschitz bandit algorithm, we use the classic Zooming algorithm [23] as the baseline. As shown in Figure 1, we consider three levels of quantities of corruptions applied on each case to show how attacks progressively disturb different methods. Specifically, we set C = 0 for the non-corrupted case, C = 3 , 000 for the moderate-corrupted case and C = 4 , 500 for the strong-corrupted case. Due to space limit, we defer detailed settings of algorithms to Appendix A.7.

From the plots in Figure 1, we observe that our proposed algorithms consistently outperform the Zooming algorithm and achieve sub-linear cumulative regrets under both types of attacks, whereas the Zooming algorithm becomes incompetent and suffers from linear regrets even under a moderate volume of corruption. This fact also implies that the two types of adversarial corruptions used here are severely detrimental to the performance of stochastic Lipschitz bandit algorithms. And it is evident our proposed RMEL yields the most robust results under various scenarios with different volumes of attacks. It is also worth noting that the Zooming algorithm attains promising regrets under a purely stochastic setting, while it experiences a huge increase in regrets after the corruptions emerge. This phenomenon aligns with our expectation and highlights the fact that our proposed algorithms balance the tradeoff between accuracy and robustness in a much smoother fashion.

7 Conclusion

In this work we introduce a new problem of Lipschitz bandits in the presence of adversarial corruptions, and we originally provide efficient algorithms against both weak adversaries and strong adversaries when agnostic to the total corruption budget C . The robustness and efficiency of our proposed algorithms is then validated under comprehensive experiments.

Limitations: For both the weak and strong adversary, our work leaves a regret gap between the lower bound deduced in Theorem 5.2 (Theorem 4.2) and the upper bound in Theorem 5.1 (Theorem 5.3) when agnostic to C . Closing or narrowing this regret gap seems highly non-trivial since the regret gap still exists under the simpler MAB setting [16], and we will leave it as a future work.

Acknowledgments and Disclosure of Funding

We appreciate the insightful comments from the anonymous reviewers and area chair. This work was partially supported by the National Science Foundation under grants CCF-1934568, DMS-1916125, DMS-2113605, DMS-2210388, IIS-2008173 and IIS2048280.

References

Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24, 2011.

Rajeev Agrawal. The continuum-armed bandit problem. SIAM journal on control and optimization , 33(6):1926-1951, 1995.

Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning , 47:235-256, 2002.

Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic multiarmed bandit problem. SIAM journal on computing , 32(1):48-77, 2002.

Alina Beygelzimer, John Langford, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandit algorithms with supervised learning guarantees. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pages 19-26. JMLR Workshop and Conference Proceedings, 2011.

Ilija Bogunovic, Arpan Losalka, Andreas Krause, and Jonathan Scarlett. Stochastic linear bandits robust to adversarial attacks. In International Conference on Artificial Intelligence and Statistics , pages 991-999. PMLR, 2021.

Sébastien Bubeck and Aleksandrs Slivkins. The best of both worlds: Stochastic and adversarial bandits. In Conference on Learning Theory , pages 42-1. JMLR Workshop and Conference Proceedings, 2012.

Sébastien Bubeck, Gilles Stoltz, Csaba Szepesvári, and Rémi Munos. Online optimization in x-armed bandits. Advances in Neural Information Processing Systems , 21, 2008.

Yang Cao, Zheng Wen, Branislav Kveton, and Yao Xie. Nearly optimal adaptive procedure with change detection for piecewise-stationary bandit. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 418-427. PMLR, 2019.

Pin-Yu Chen and Cho-Jui Hsieh. Adversarial robustness for machine learning . Academic Press, San Diego, CA, August 2022.

Wang Chi Cheung, David Simchi-Levi, and Ruihao Zhu. Learning to optimize under nonstationarity. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1079-1087. PMLR, 2019.

Qin Ding, Cho-Jui Hsieh, and James Sharpnack. Robust stochastic linear contextual bandits under adversarial attacks. In International Conference on Artificial Intelligence and Statistics , pages 7111-7123. PMLR, 2022.

Qin Ding, Yue Kang, Yi-Wei Liu, Thomas Chun Man Lee, Cho-Jui Hsieh, and James Sharpnack. Syndicated bandits: A framework for auto tuning hyper-parameters in contextual bandit algorithms. Advances in Neural Information Processing Systems , 35:1170-1181, 2022.

Yasong Feng, Zengfeng Huang, and Tianyu Wang. Lipschitz bandits with batched feedback. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Evrard Garcelon, Baptiste Roziere, Laurent Meunier, Jean Tarbouriech, Olivier Teytaud, Alessandro Lazaric, and Matteo Pirotta. Adversarial attacks on linear contextual bandits. Advances in Neural Information Processing Systems , 33:14362-14373, 2020.

Anupam Gupta, Tomer Koren, and Kunal Talwar. Better algorithms for stochastic bandits with adversarial corruptions. In Conference on Learning Theory , pages 1562-1578. PMLR, 2019.

Jiafan He, Dongruo Zhou, Tong Zhang, and Quanquan Gu. Nearly optimal algorithms for linear contextual bandits with adversarial corruptions. arXiv preprint arXiv:2205.06811 , 2022.

Kwang-Sung Jun, Lihong Li, Yuzhe Ma, and Jerry Zhu. Adversarial attacks on stochastic bandits. Advances in Neural Information Processing Systems , 31, 2018.

Yue Kang, Cho-Jui Hsieh, and Thomas Lee. Online continuous hyperparameter optimization for contextual bandits. arXiv preprint arXiv:2302.09440 , 2023.

Yue Kang, Cho-Jui Hsieh, and Thomas Chun Man Lee. Efficient frameworks for generalized lowrank matrix bandit problems. Advances in Neural Information Processing Systems , 35:1997119983, 2022.

Robert Kleinberg. Nearly tight bounds for the continuum-armed bandit problem. Advances in Neural Information Processing Systems , 17, 2004.

Robert Kleinberg and Aleksandrs Slivkins. Sharp dichotomies for regret minimization in metric spaces. In Proceedings of the twenty-first annual ACM-SIAM symposium on Discrete Algorithms , pages 827-846. SIAM, 2010.

Robert Kleinberg, Aleksandrs Slivkins, and Eli Upfal. Bandits and experts in metric spaces. Journal of the ACM (JACM) , 66(4):1-77, 2019.

Tor Lattimore and Csaba Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Chung-Wei Lee, Haipeng Luo, Chen-Yu Wei, Mengxiao Zhang, and Xiaojin Zhang. Achieving near instance-optimality and minimax-optimality in stochastic and adversarial linear bandits simultaneously. In International Conference on Machine Learning , pages 6142-6151. PMLR, 2021.

Wenjie Li, Haoze Li, Jean Honorio, and Qifan Song. Pyxab - a python library for X -armed bandit and online blackbox optimization algorithms, 2023.

Yingkai Li, Edmund Y Lou, and Liren Shan. Stochastic linear optimization with adversarial corruption. arXiv preprint arXiv:1909.02109 , 2019.

Shiyin Lu, Guanghui Wang, Yao Hu, and Lijun Zhang. Optimal algorithms for lipschitz bandits with heavy-tailed rewards. In International Conference on Machine Learning , pages 4154-4163. PMLR, 2019.

Thodoris Lykouris, Vahab Mirrokni, and Renato Paes Leme. Stochastic bandits robust to adversarial corruptions. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing , pages 114-122, 2018.

Stefan Magureanu, Richard Combes, and Alexandre Proutiere. Lipschitz bandits: Regret lower bound and optimal algorithms. In Conference on Learning Theory , pages 975-999. PMLR, 2014.

Aldo Pacchiano, My Phan, Yasin Abbasi Yadkori, Anup Rao, Julian Zimmert, Tor Lattimore, and Csaba Szepesvari. Model selection in contextual stochastic bandit problems. Advances in Neural Information Processing Systems , 33:10328-10337, 2020.

Chara Podimata and Alex Slivkins. Adaptive discretization for adversarial lipschitz bandits. In Conference on Learning Theory , pages 3788-3805. PMLR, 2021.

Aleksandrs Slivkins. Contextual bandits with similarity information. In Proceedings of the 24th annual Conference On Learning Theory , pages 679-702. JMLR Workshop and Conference Proceedings, 2011.

Aleksandrs Slivkins. Multi-armed bandits on implicit metric spaces. Advances in Neural Information Processing Systems , 24, 2011.

Aleksandrs Slivkins et al. Introduction to multi-armed bandits. Foundations and Trends® in Machine Learning , 12(1-2):1-286, 2019.

Yitzchak Solomon, Alexander Wagner, and Paul Bendich. A fast and robust method for global topological functional optimization. In International Conference on Artificial Intelligence and Statistics , pages 109-117. PMLR, 2021.

Chen-Yu Wei, Christoph Dann, and Julian Zimmert. A model selection approach for corruption robust reinforcement learning. In International Conference on Algorithmic Learning Theory , pages 1043-1096. PMLR, 2022.

Chenlu Ye, Wei Xiong, Quanquan Gu, and Tong Zhang. Corruption-robust algorithms with uncertainty weighting for nonlinear contextual bandits and markov decision processes. In International Conference on Machine Learning , pages 39834-39863. PMLR, 2023.

Heyang Zhao, Dongruo Zhou, and Quanquan Gu. Linear contextual bandits with adversarial corruptions. arXiv preprint arXiv:2110.12615 , 2021.

Peer reviews:
comments: SUMMARY:
This work studies Lipschitz bandits robust to adversarial corruption, where both strong and weak adversaries are considered. Robust Zooming along with its cumulative regret upper bound is proposed for strong adversary with known corruption budget. For unknown corruption budget, this work proposes RMEL for weak adversary and a two-layer framework BoB for strong adversary. The lower bounds for both adversaries are also proposed, showing Robust Zooming is theoretically optimal for strong adversary. Experimental results are provided to show the practical performance of RMEL and BoB.

STRENGTHS:
- This is the first work that studies robust Lipschitz bandit.
- This work considers multiple settings in terms of adversary type and whether the corruption budget is known.
- Both algorithmic upper bounds and lower bounds are provided.

WEAKNESSES:
- For the case of unknown budget, there is a gap between the upper and lower bounds for both weak and strong adversaries.
- In the experiment section, although RMEL and BoB outperform the baseline, their cumulative regrets do not converge (have large slope) in several settings.

QUESTIONS:
- This work assumed Lipschitz constant one and $\Delta(x)\le 1$. I wonder how the cumulative regret would scale when we have a higher  Lipschitz constant and a higher function magnitude.
- In figure 1, even the 'no corruption' trials have positive simple regret near T. Does it imply that all the algorithms fail to find the optimal arm?
- I'm more interested in the practical performance of Robust Zooming, as its cumulative regret upper bound is already theoretically optimal.

LIMITATIONS:
The limitations have been discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe article focuses on the problem of Lipschitz bandits in the presence of adversarial corruptions, where an adaptive adversary corrupts the stochastic rewards up to a total budget $C$. Both weak and strong adversaries are considered, where the weak adversary is unaware of the current action before the attack, while the strong one can observe it. This article presents the first line of robust Lipschitz bandit algorithms that can achieve sub-linear regret under both types of adversary, even when the total budget of corruption $C$ is unrevealed to the agent.\n\nSTRENGTHS:\nThis work studies Lipschitz bandits in the presence of adversarial corruptions, which are naturally motivated by a broad range of applications with corrupted data. The innovation of the proposed algorithm is that they combined a couple of existing techiniques to design the algorithm,  which help a lot to improve the performance. For example, the authors adopted a removal procedure with multi layers, which could adaptively remove regions at different rates such that the algorithm can torelate corruptions in different levels. This techinique was first proposed in [29] to improve the efficiency of the algorithm for multi-armed bandit in the presense of corruption. Enough intuitions are provided through the presentation of the algorthms and I can easily follow the ideas.\n\nThe authors show the advantages of the proposals over past algroithms both theoretically and numerically. The analysis and proofs look correct and intuitive to me.\n\nWEAKNESSES:\n1. In additon to the overall budge constraint $C$, the authors require $c_t(x)$ to be bounded by 1. It is stange to put it in Section 3 as an assumption, since the authors claim it is can be easily addressed. Moreover, $c_t(x) > 1$ does not mean the adversary always has enough power to make a suboptimal arm optimal, so I feel it is not straightforward to ignore the case $c_t(x) > 1$.\n\n2. The poposed algorithms (at least some important components) rely on existing techiniques, that somehow reduces the novelty of this work.\n\nQUESTIONS:\nNone.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers a model of Lipschitz bandits with adversarial corruptions. Two types of adversaries are analysed hand-in-hand: weak adversary which may not have knowledge of the current action of the learner before injecting its corruption into the reward that is actually observed by the learner, and strong adversary which has knowledge of the current action of the learner and is hence more challenging to defend against. Irrespective of its nature, the adversary may only inject a total corruption of $C$ over $T$ rounds of sampling. The learner may or may not have prior knowledge of the value of $C$, and is tasked with the goal of minimizing the cumulative regret over the horizon $T$.\n\nThe paper analyzes the following scenarios.\n1. The learner has prior knowledge of $C$ (Section 4).\n2. The learner has no prior knowledge of $C$, and must deal with a weak adversary (Section 5.1).\n3. The learner has no prior knowledge of $C$, and must deal with a strong adversary (Section 5.2).\n\nFor each of the above scenarios, the paper presents a lower bound on the cumulative regret and one or more algorithms for cumulative regret minimization. For scenario 1, the paper proposes a robust zooming algorithm, inspired by the UCB algorithm, albeit for a continuum of arms. For scenario 2, a robust multi-layer elimination algorithm is proposed. For scenario 3, three different algorithms are proposed. The paper derives upper bounds on the cumulative regret for each of the algorithms under every scenario. A noteworthy aspect of the lower and the upper bounds are their dependence on the corruption budget $C$, regardless of whether $C$ is known or unknown.\n\nSTRENGTHS:\n1. The paper extends the analysis of prior works on adversarial corruptions in stochastic multi-armed bandits to handle a continuum of arms; most prior works in bandits deal with only finitely many arms. \n2. The authors' presentation of the key results of the paper in Table 1, as a quick, one-stop summary of all the important results of the paper, is commendable. A mere cursory glance of Table 1 suffices for an expert reader to recall the important contributions of the paper.\n3. The authors' effort towards extracting the key ideas from several prior works (notably [23], [29], [31], and [11]) and applying these ideas to solve the problem at hand (with a continuum of arms) is truly commendable.\n4. The dependence of the lower and the upper bounds on the corruption budget $C$, the zooming dimension $d_z$, and the covering dimension $d$ is noteworthy.\n\nWEAKNESSES:\nThe section on Preliminaries (Section 3) needs to be supplemented with additional details and rewritten to correct some glaring errors. Below are some points I have mentioned based on my detailed reading of the paper. \n\n1. In the definitions of the covering dimension and the zooming dimension, $\\exists c>0$ should be replaced with $\\exists \\alpha>0$. \n2. For any $r \\in (0,1]$, the \"$r$-covering\" of a compact set must be defined formally, noting that it appears frequently in Algorithm 2 (where phrases such as \"$1/2$-covering\", \"$1/2^{m_l+1}$-covering\" are used).\n3. It is better to denote the covering dimension by $d_c$ instead of $d$; the latter could be referred to as the Euclidean dimension, following the lines of [23]. This avoids strange (yet correct) phrases such as \"$d=d$\" appearing on line 136, which as per the suggested notations would read \"$d_c=d$\".\n4. It is not quite clear why $d_z \\leq d$. The definition of the covering dimension $d$ entails using balls of radius at most $r$ for the covering, whereas that of the zooming dimension $d_z$ entails using balls of radius at most $r/16$ for the covering. The authors must clarify why $d_z\\leq d$, as this is crucial for comparing the upper bound of Theorem 5.3 with the lower bound of Theorem 4.2. \n5. In continuation to the previous point, the paper [23] alludes to the zooming dimension as \"the smallest $q$ such that for every $r>0$, only $O(r^{-q})$ sets of radius $r/16$ are required to cover the set $\\lbrace x \\in \\mathcal{X}: r\\leq \\Delta(x) \\leq 2r \\rbrace$ (this is rephrased in terms of the notations used in the current paper); see, for instance, the last paragraph in [23, pp. 30:5]. In contrast, the authors define the zooming dimension as \"the smallest $q$ such that for every $r>0$, only $O(r^{-q})$ sets of radius $r/16$ are required to cover the \"$r$-optimal set\" $\\lbrace x \\in \\mathcal{X}: \\Delta(x) \\leq r \\rbrace$\", which differs from the one in [23]. Can the authors elaborate on why they chose to define the zooming dimension differently from [23], and what are the implications of this difference on their results?\n6. I am not very convinced about the nature of adversarial corruptions used for the simulations. I cannot readily see if \"Oracle\" and \"Garcelon\" attacks respect the adversarial attack model $\\max_{x \\in \\mathcal{X}} |c_t(x)| \\leq 1$ for all $x,t$ that forms the heart of the paper. In fact, in Garcelon, using a Gaussian random variable as the corrupted reward has the implication that the corruption is unbounded, thereby not adhering to the proposed adversarial attack model. In my opinion, it is quite difficult to judge in a fair manner the performance of the proposed algorithms on adversarial attack models that differ from the underlying model proposed in Section 3.\n7. Continuing on the above point, why do the authors not consider in their simulations an explicit corruption function that satisfies the requirement $\\max_{x \\in \\mathcal{X}} |c_t(x)| \\leq 1$ for all $x,t$? For instance, say $c_t(x)=\\sin(2\\pi xt/T)$, $x \\in [0,1]$, $t \\in \\{1, \\ldots, T\\}$. Is incorporating such functions into the simulations challenging? If so, what exactly are the challenges? Simulating an attack model that matches exactly with the description of Section 3 would significantly enhance the purpose of simulations, lead to a fair evaluation of the performances of the proposed algorithms, and demonstrate the strength/utility of the proposed algorithms.\n\nQUESTIONS:\n1. The authors provide separate algorithms for the case when the adversary is weak and when the adversary is strong. While I appreciate the effort put into analysing both of these scenarios separately in the paper, I have the following question: in practice, is the algorithm (more specifically, the learner) aware of the nature of the adversary beforehand? Are there practical scenarios where the learner has prior knowledge about whether the adversary is weak/strong? I would imagine that it is very unlikely that such prior knowledge may actually be available in practice, in which case it would be reasonable for the learner to simply design an algorithm that works against the strong adversary. Can the authors comment about this (if possible also include a sentence or two on this point in the paper)?\n2. The paper [23] alludes to the zooming dimension as \"the smallest $q$ such that for every $r>0$, only $O(r^{-q})$ sets of radius $r/16$ are required to cover the set $\\lbrace x \\in \\mathcal{X}: r\\leq \\Delta(x) \\leq 2r \\rbrace$ (this is rephrased in terms of the notations used in the current paper); see, for instance, the last paragraph in [23, pp. 30:5]. In contrast, the authors define the zooming dimension as \"the smallest $q$ such that for every $r>0$, only $O(r^{-q})$ sets of radius $r/16$ are required to cover the \"$r$-optimal set\" $\\lbrace x \\in \\mathcal{X}: \\Delta(x) \\leq r \\rbrace$\", which differs from the one in [23].\nCan the authors elaborate on why they chose to define the zooming dimension differently from [23], and what are the implications of this difference on their results?\n3. This is more of an observation than a question. The upper bound on cumulative regret derived in Theorem 5.1 relies on lower bounding the probability of the event $\\Phi$ as defined in the proof sketch. In the definition of $\\Phi$, the layers whose tolerance values exceed the unknown corruption budget $C$ are considered. To me, this seems like an \"artificial\" way of introducing $C$ into $\\Phi$ in order to derive an upper bound that depends on $C$. However, this also suggests that without bringing $C$ into the definition of $\\Phi$, an upper bound on cumulative regret that does not depend on $C$ may be derived. Such an upper bound may be relevant to problems where $C$ is potentially infinite (i.e., the adversary may corrupt the rewards in an unbounded fashion), thereby broadening the scope of the current work (when $C=+\\infty$, the upper bounds in the paper become degenerate). However, I envisage that the analysis of the case $C=+\\infty$ could be hard. Perhaps, this could be a direction of future work.\n\nLIMITATIONS:\nSee points 6,7 under \"weaknesses\".", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper studies Lipschitz bandits with adversarial corruptions, where the reward of the pulled arm can be maliciously corrupted by an adversary. The authors consider both weak adversary and strong adversary and present robust algorithms for each setting. Under the weak adversary setting, the paper proposes an elimination-based algorithm and derives a regret bound with nearly optimal dependence on the time horizon. For strong adversary, the paper shows that a simple modification of the Zooming algorithm can attain sub-linear regret when the corruption budget is known in advance and applies several model selection methods to handle unknown corruption budget. The paper also provide numerical results to demonstrate the advantage of the proposed algorithm in corrupted scenarios.

STRENGTHS:
The paper is the first to study Lipschitz bandits with adversarial corruptions and proposes several algorithms with both sub-linear regret bounds and promising empirical performance.

WEAKNESSES:
1. For weak adversary, the regret bound of the proposed algorithm has a multiplicative dependence on the corruption budget, which is unsatisfactory and an additive dependence is desired.

2. For strong adversary, the proposed algorithm is mainly a combination of existing methods with little novelty. The derived regret bounds are also weak as they depend on the covering dimension rather than the zooming dimension.

QUESTIONS:
In the experiment with two dim mean function under oracle attack, why the performance of RMEL is better than Zooming?

LIMITATIONS:
The authors have discussed the limitations of the proposed algorithms.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
The paper studied robust Lipschitz bandits under two types of corruptions - a weak adversary who perturbs the reward function before observing the selected action; and a strong adversary who perturbs the instantiated reward of the selected action. When the attack budget is known, the paper proposed to use enlarged confidence budget, and the regret upper bound of the corresponding algorithms attains the lower bound derived in the paper. When the attack budget is unknown, the paper developed two robust algorithms targeting the weak and strong adversary separately. For the weak adversary, both upper and lower regret bound are derived. There is a gap between these two bounds. For the strong adversary, the paper derived upper bound of the regret. Experiments on synthetic datasets demonstrate the effectiveness of the proposed robust algorithms.

STRENGTHS:
(1). The topic of studying robustness in bandits is a timely topic. This extends prior works on robust bandit to the Lipschitz bandit scenario, which pushes the frontier in this domain.

(2). The work is relatively complete - two types of adversaries are considered, and the attack budget are defined accordingly; two threat models are considered, including known and unknown attack budget. Besides that, both upper and lower regret bounds are derived, which shows that the robust algorithm designed in this paper is close to optimal.

WEAKNESSES:
I don't see major weaknesses with the paper. A minor weakness is that for the strong adversary and the unknown attack budget setting, the paper is missing a regret lower bound.

QUESTIONS:
Can the authors provide some discussions on whether it's possible to derive regret lower bound for the strong adversary and unknown attack budget setting?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-06