SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yue Kang
Department of Statistics  
University of California, Davis  
Davis, CA 95616  
yuekang@ucdavis.edu
Google and Department of Computer Science, UCLA
Los Angeles, CA  
chohsieh@cs.ucla.edu
Thomas C. M. Lee
Department of Statistics  
University of California, Davis  
Davis, CA 95616  
tcmlee@ucdavis.edu
Lipschitz bandit is a variant of stochastic bandits that deals with a continuous arm set defined on a metric space, where the reward function is subject to a Lipschitz constraint. In this paper, we introduce a new problem of Lipschitz bandits in the presence of adversarial corruptions where an adaptive adversary corrupts the stochastic rewards up to a total budget  $C$ . The budget is measured by the sum of corruption levels across the time horizon  $T$ . We consider both weak and strong adversaries, where the weak adversary is unaware of the current action before the attack, while the strong one can observe it. Our work presents the first line of robust Lipschitz bandit algorithms that can achieve sub-linear regret under both types of adversary, even when the total budget of corruption  $C$  is unrevealed to the agent. We provide a lower bound under each type of adversary, and show that our algorithm is optimal under the strong case. Finally, we conduct experiments to illustrate the effectiveness of our algorithms against two classic kinds of attacks.
Multi-armed Bandit (MAB) [3] is a fundamental and powerful framework in sequential decision-making problems. Given the potential existence of malicious users in real-world scenarios [10], a recent line of works considers the stochastic bandit problem under adversarial corruptions: an agent adaptively updates its policy to choose an arm from the arm set, and an adversary may contaminate the reward generated from the stochastic bandit before the agent could observe it. To robustify bandit learning algorithms under adversarial corruptions, several algorithms have been developed in the setting of traditional MAB [16, 18, 29] and contextual linear bandits [6, 12, 17, 27, 39]. These works consider either the weak adversary [29], which has access to all past data but not the current action before choosing its attack, or the strong adversary [6], which is also aware of the current action for contamination. Details of these two adversaries will be elaborated in Section 3. In practice, bandits under adversarial corruptions can be used in many real-world problems such as pay-per-click advertising with click fraud and recommendation systems with fake reviews [29], and it has been empirically validated that stochastic MABs are vulnerable to slight corruption [12, 15, 18].
Although there has been extensive research on the adversarial robustness of stochastic bandits, most existing works consider problems with a discrete arm set, such as the traditional MAB and contextual linear bandit. In this paper, we investigate robust bandit algorithms against adversarial corruptions in the Lipschitz bandit setting, where a continuously infinite arm set lie in a known metric space
Table 1: Comparisons of regret bounds for our proposed robust Lipschitz bandit algorithms.
ALGORITHM	REGRET BOUND	FORMAT	C	ADVERSARY
Robust Zooming	\(\tilde{O}\left(T^{\frac{dz+1}{dz+2}}+C^{\frac{1}{dz+1}}T^{\frac{dz}{dz+1}}\right)\)	HIGH. PROB.	KNOWN	STRONG
RMEL	\(\tilde{O}\left((C^{\frac{1}{dz+2}}+1)T^{\frac{dz+1}{dz+2}}\right)\)	HIGH. PROB.	UNKNOWN	WEAK
EXP3.P	\(\tilde{O}\left((C^{\frac{1}{d+2}}+1)T^{\frac{d+2}{d+3}}\right)\)	EXPECTED	UNKNOWN	STRONG
CORRAL	\(\tilde{O}\left((C^{\frac{1}{d+1}}+1)T^{\frac{d+1}{d+2}}\right)\)	EXPECTED	UNKNOWN	STRONG
BoB Robust Zooming	\(\tilde{O}\left(T^{\frac{d+3}{d+4}}+C^{\frac{1}{d+1}}T^{\frac{d+2}{d+3}}\right)\)	HIGH. PROB.	UNKNOWN	STRONG
with covering dimension  $d$  and the expected reward function is an unknown Lipschitz function. Lipschitz bandit can be used to efficiently model many real-world tasks such as dynamic pricing, auction bidding [35] and hyperparameter tuning [19]. The stochastic Lipschitz bandit has been well understood after a large body of literature [8, 23, 30], and state-of-the-art algorithms could achieve a cumulative regret bound of order  $\tilde{O}(T^{\frac{dz + 1}{dz + 2}})^1$  in time  $T$ . However, to the best of our knowledge, the stochastic Lipschitz bandit problem with adversarial corruptions has never been explored, and we believe it is challenging since most of the existing robust MAB algorithms utilized the idea of elimination, which is much more difficult under a continuously infinite arm pool. Furthermore, the complex structure of different metric spaces also poses challenges for defending against adversarial attacks [36] in theory. Therefore, it remains intriguing to design computationally efficient Lipschitz bandits that are robust to adversarial corruptions under both weak and strong adversaries.
We develop efficient robust algorithms whose regret bounds degrade sub-linearly in terms of the corruption budget  $C$ . Our contributions can be summarized as follows: (1) Under the weak adversary, we extend the idea in [29] and propose an efficient algorithm named Robust Multi-layer Elimination Lipschitz bandit algorithm (RMEL) that is agnostic to  $C$  and attains  $\tilde{O}\left(C\frac{1}{dz + 2} T^{\frac{dz + 1}{dz + 2}}\right)$  regret bound. This bound matches the minimax regret bound of Lipschitz bandits [8, 23] in the absence of corruptions up to logarithmic terms. This algorithm consists of multiple parallel sub-layers with different tolerance against the budget  $C$ , where each layer adaptively discretizes the action space and eliminates some less promising regions based on its corruption tolerance level in each crafted epoch. Interactions between layers assure the promptness of the elimination process. (2) Under the strong adversary, we first show that when the budget  $C$  is given, a simple modification on the classic Zooming algorithm [23] would lead to a robust method, namely, Robust Zooming algorithm, which could obtain a regret bound of order  $\tilde{O}\left(T^{\frac{dz + 1}{dz + 2}} + C^{\frac{1}{dz + 1}}T^{\frac{dz}{dz + 1}}\right)$ . We then provide a lower bound to prove the extra  $O(C^{\frac{1}{dz + 1}}T^{\frac{dz}{dz + 1}})$  regret is unavoidable. Further, inspired by the Bandit-over-Bandit (BoB) model selection idea [11, 13, 31], we design a two-layer framework adapting to the unknown  $C$  where a master algorithm in the top layer dynamically tunes the corruption budget for the Robust Zooming algorithm. Three types of master algorithms are discussed and compared in both theory and practice. Table 1 outlines our algorithms as well as their regret bounds under different scenarios.
Stochastic and Adversarial Bandit Extensive studies have been conducted on MAB and its variations, including linear bandit [1], matrix bandit [20], etc. The majority of literature can be categorized into two types of models [24]: stochastic bandit, in which rewards for each arm are independently sampled from a fixed distribution, and adversarial bandit, where rewards are maliciously generated at all time. However, adversarial bandit differs from our problem setting in the sense that rewards are arbitrarily chosen without any budget or distribution constraint. Another line of work aims to obtain "the best of both worlds" guarantee simultaneously [7]. However, neither of these models is reliable in practice [9], since the former one is too ideal, while the latter one remains very pessimistic, assuming a fully unconstrained setting. Therefore, it is more natural to consider the scenario that lies "in between" the two extremes: the stochastic bandit under adversarial corruptions.
$^1\tilde{O}$  ignores the polylogarithmic factors.  $d_z$  is the zooming dimension defined in Section 3.
Lipschitz Bandit Most existing works on the stochastic Lipschitz bandit [2] follow two key ideas. One is to uniformly discretize the action space into a mesh in the initial phase so that any MAB algorithm could be implemented [21, 30]. The other is to adaptively discretize the action space by placing more probes in more promising regions, and then UCB [8, 23, 28], TS [19] or elimination [14] method could be utilized to deal with the exploration-exploitation tradeoff. The adversarial Lipschitz bandit was recently introduced and solved in [32], where the expected reward Lipschitz function is arbitrarily chosen at each round. However, as mentioned in the previous paragraph, this fully adversarial setting is quite different from ours. And their algorithm relies on several unspecified hyperparameters and hence is computationally formidable in practice. In addition, some interesting variations of Lipschitz bandits have also been carefully examined, such as contextual Lipschitz bandits [33], full-feedback Lipschitz bandits [22], and taxonomy bandits [34].
Robust Bandit to Adversarial Corruptions Adversarial attacks were studied in the setting of MAB [18] and linear bandits [15]. And we will use two classic attacks for experiments in Section 6. To defend against attacks from weak adversaries, [29] proposed the first MAB algorithm robust to corruptions with a regret  $C$  times worse than regret in the stochastic setting. An improved algorithm whose regret only contains an additive term on  $C$  was then proposed in [16]. [27] subsequently studied the linear bandits with adversarial corruptions and achieved instance-dependent regret bounds. [25] also studied the corrupted linear bandit problem while assuming the attacks on reward are linear in action. Recently, a robust VOFUL algorithm achieving regret bound only logarithmically dependent on  $T$  was proposed in [37]. Another line of work on the robust bandit problem focuses on a more challenging setting with strong adversaries who could observe current actions before attacking rewards. [6] considered the corrupted linear bandit when small random perturbations are applied to context vectors, and [12, 17, 39] extended the OFUL algorithm [1] and achieved improved regret bounds. [38] further considers general non-linear contextual bandits and also MDPs with strong adversarial corruptions. However, the study of Lipschitz bandits under attacks remains an unaddressed open area.
We will introduce the setting of Lipschitz bandits with adversarial corruptions in this section. The Lipschitz bandit is defined on a triplet  $(\mathcal{X},D,\mu)$ , where  $\mathcal{X}$  is the arm set space equipped with some metric  $D$ , and  $\mu : \mathcal{X} \to \mathbb{R}$  is an unknown Lipschitz reward function on the metric space  $(\mathcal{X},D)$  with Lipschitz constant 1. W.l.o.g. we assume  $\mathcal{X}$  is compact with its diameter no more than 1. Under the stochastic setting, at each round  $t \in [T] := \{1,2,\dots,T\}$ , stochastic rewards are sampled for each arm  $x \in \mathcal{X}$  from some unknown distribution  $\mathbb{P}_x$  independently, and then the agent pulls an arm  $x_t$  and receives the corresponding stochastic reward  $\tilde{y}_t$  such that,
$$
\tilde {y} _ {t} = \mu \left(x _ {t}\right) + \eta_ {t}, \tag {1}
$$
where  $\eta_t$  is i.i.d. zero-mean random error with sub-Gaussian parameter  $\sigma$  conditional on the filtration  $\mathcal{F}_t = \{x_t, x_{t-1}, \eta_{t-1}, \ldots, x_1, \eta_1\}$ . W.l.o.g we assume  $\sigma = 1$  for simplicity in the rest of our analysis. At each round  $t \in [T]$ , the weak adversary observes the payoff function  $\mu(\cdot)$ , the realizations of  $\mathbb{P}_x$  for each arm  $x \in \mathcal{X}$  and choices of the agent  $\{x_i\}_{i=1}^{t-1}$  in previous rounds, and injects an attack  $c_t(x_t)$  into the reward before the agent pulls  $x_t$ . The agent then receives a corrupted reward  $y_t = \tilde{y}_t + c_t(x_t)$ . The strong adversary would be omniscient and have complete information about the problem  $\mathcal{F}_t$ . In addition to the knowledge that a weak adversary possesses, it would also be aware of the current action  $x_t$  while contaminating the data, and subsequently decide upon the corrupted reward  $y_t = \tilde{y}_t + c_t(x_t)$ . Some literature in corrupted bandits [12, 15] also consider attacking on the contexts or arms, i.e. the adversary modifies the true arm  $x_t$  in a small region, while in our problem setting it is obvious that attacking contexts is only a sub-case of attacking rewards due to the Lipschitzness of  $\mu(\cdot)$ , and hence studying the adversarial attacks on rewards alone is sufficient under the Lipschitz bandit setting.
The total corruption budget  $C$  of the adversary is defined as  $C = \sum_{t=1}^{T} \max_{x \in \mathcal{X}} |c_t(x)|$ , which is the sum of maximum perturbation from the adversary at each round across the horizon  $T$ . Note the strong adversary may only corrupt the rewards of pulled arms and hence we could equivalently write  $C = \sum_{t=1}^{T} |c_t(x_t)|$  in that case as [6, 17]. Define the optimal arm  $x_* = \arg \max_{x \in \mathcal{X}} \mu(x)$  and the loss of arm  $x$  as  $\Delta(x) = \mu(x_*) - \mu(x)$ ,  $x \in \mathcal{X}$ . W.l.o.g. we assume  $C \leq T$  and each instance of attack  $|c_t(x)| \leq 1, \forall t \in [T]$ ,  $x \in \mathcal{X}$  as in other robust bandit literature [16, 29] since
Algorithm 1 Robust Zooming Algorithm
Input: Arm metric space  $(\mathcal{X},D)$  , time horizon  $T$  , probability rate  $\delta$    
Initialization: Active arm set  $J = \{\}$  , active space  $\mathcal{X}_{act} = \mathcal{X}$    
1: for  $t = 1$  to  $T$  do   
2: if  $f(v) - f(u)\geq r(v) + 2r(u)$  for some pair of active arms  $u,v\in J$  . then   
3: Set  $J = J\backslash \{u\}$  and  $\mathcal{X}_{act} = \mathcal{X}_{act}\backslash \mathcal{B}(u,r(u))$  . Removal   
4: end if   
5: if  $\mathcal{X}_{act}\not\subseteq\cup_{v\in J}\mathcal{B}(v,r(v))$  then   
6: Activate and pull some arm  $x\not\in\cup_{v\in J}\mathcal{B}(v,r(v))$  in  $\mathcal{X}_{act}$  such that  $x_{t} = x$ $J = J\cup \{x\}$  and set the components  $n(x) = 0$ $f(x) = 0$  Activation   
7: else   
8: Pull  $x_{t} = \arg \max_{v\in J}I(v) = f(v) + 2r(v)$  , and break ties arbitrarily. Selection   
9: end if   
10: Observe the payoff  $y_{t}$  . And update components associated with  $x_{t}$  in the Robust Zooming Algorithm:  $n(x_{t}) = n(x_{t}) + 1$ $f(x_{t}) = (f(x_{t})(n(x_{t}) - 1) + y_{t}) / n(x_{t})$    
11: end for
the adversary could already make any arm  $x \in \mathcal{X}$  optimal given that  $\Delta(x) \leq 1$ . (We can assume  $|c_t(x)| \leq u, \forall t \in [T]$ ,  $x \in \mathcal{X}$  for any positive constant  $u$ .) Similar to the stochastic case [21], the goal of the agent is to minimize the cumulative regret defined as:
$$
\operatorname {R e g r e t} _ {T} = T \mu \left(x _ {*}\right) - \sum_ {t = 1} ^ {T} \mu \left(x _ {t}\right). \tag {2}
$$
An important pair of concepts in Lipschitz bandits defined on  $(\mathcal{X},D,\mu)$  are the covering dimension  $d$  and the zooming dimension  $d_z$ . Let  $\mathcal{B}(x,r)$  denotes a closed ball centered at  $x$  with radius  $r$  in  $\mathcal{X}$ , i.e.  $\mathcal{B}(x,r) = \{x' \in \mathcal{X} : D(x,x') \leq r\}$ , the  $r$ -covering number  $N_c(r)$  of metric space  $(\mathcal{X},D)$  is defined as the minimal number of balls with radius of no more than  $r$  required to cover  $\mathcal{X}$ . On the contrary, the  $r$ -zooming number  $N_z(r)$  introduced in [23] not only depends on the metric space  $(\mathcal{X},D)$  but also the payoff function  $\mu(\cdot)$ . It describes the minimal number of balls of radius not more than  $r/16$  required to cover the  $r$ -optimal region defined as  $\{x \in \mathcal{X} : \Delta(x) \leq r\}$  [8, 14] $^2$ . Next, we define the covering dimension  $d$  (zooming dimension  $d_z$ ) as the smallest  $q \geq 0$  such that for every  $r \in (0,1]$  the  $r$ -covering number  $N_c(r)$  ( $r$ -zooming number  $N_z(r)$ ) can be upper bounded by  $\alpha r^{-q}$  for some multiplier  $\alpha > 0$  that is free of  $r$ :
$$
d = \min  \{q \geq 0: \exists \alpha > 0, N _ {c} (r) \leq \alpha r ^ {- q}, \forall r \in (0, 1 ] \},
$$
$$
d _ {z} = \min  \{q \geq 0: \exists \alpha > 0, N _ {z} (r) \leq \alpha r ^ {- q}, \forall r \in (0, 1 ] \}.
$$
It is clear that  $0 \leq d_{z} \leq d$  since the  $r$ -optimal region is a subset of  $\mathcal{X}$ . On the other hand,  $d_{z}$  could be much smaller than  $d$  in some benign cases. For example, if the payoff function  $\mu(\cdot)$  defined on the metric space  $(\mathbb{R}^{k}, \| \cdot \|_{2})$ ,  $k \in \mathbb{N}$  is  $C^2$ -smooth and strongly concave in a neighborhood of the optimal arm  $x_{*}$ , then it could be easily verified that  $d_{z} = k/2$  whereas  $d = k$ . However,  $d_{z}$  is never revealed to the agent as it relies on the underlying function  $\mu(\cdot)$ , and hence designing an algorithm whose regret bound depends on  $d_{z}$  without knowledge of  $d_{z}$  would be considerably difficult.
4 Warm-up: Robust Lipschitz Bandit with Known Budgets
To defend against attacks on Lipschitz bandits, we first consider a simpler case where the agent is aware of the corruption budget  $C$ . We demonstrate that a slight modification of the classic Zooming algorithm [23] can result in a robust Lipschitz bandit algorithm even under the strong adversary, called the Robust Zooming algorithm, which achieves a regret bound of order  $\tilde{O}(T^{\frac{d_z + 1}{d_z + 2}} + C^{\frac{1}{d_z + 1}} T^{\frac{d_z}{d_z + 1}})$ .
We first introduce some notations of the algorithm: denote  $J$  as the active arm set. For each active arm  $x \in J$ , let  $n(x)$  be the number of times arm  $x$  has been pulled,  $f(x)$  be the corresponding average sample reward, and  $r(x)$  be the confidence radius controlling the deviation of the sample average
2We actually use the near-optimality dimension introduced in [8], where the authors imply the equivalence between this definition and the original zooming dimension proposed in [23].
$f(x)$  from its expectation  $\mu (x)$ . We also define  $\mathcal{B}(x,r(x))$  as the confidence ball of an active arm  $x$ . In essence, the Zooming algorithm works by focusing on regions that have the potential for higher rewards and allocating fewer probes to less promising regions. The algorithm consists of two phases: in the activation phase, a new arm gets activated if it is not covered by the confidence balls of all active arms. This allows the algorithm to quickly zoom into the regions where arms are frequently pulled due to their encouraging rewards. In the selection phase, the algorithm chooses an arm with the largest value of  $f(v) + 2r(v)$  among  $J$  based on the UCB methodology.
Our key idea is to enlarge the confidence radius of active arms to account for the known corruption budget  $C$ . Specifically, we could set the value of  $r(x)$  as:
$$
r (x) = \sqrt {\frac {4 \ln {(T)} + 2 \ln {(2 / \delta)}}{n (x)}} + \frac {C}{n (x)},
$$
where the first term accounts for deviation in stochastic rewards and the second term is used to defend the corruptions from the adversary. The robust algorithm is shown in Algorithm 1. In addition to the two phases presented above, our algorithm also conducts a removal procedure at the beginning of each round for better efficiency. This step adaptively removes regions that are likely to yield low rewards with high confidence. Theorem 4.1 provides a regret bound for Algorithm 1.
Theorem 4.1. Given the total corruption budget that is at most  $C$ , with probability at least  $1 - \delta$ , the overall regret of Robust Zooming Algorithm (Algorithm 1) can be bounded as:
$$
R e g r e t _ {T} = O \left(\ln (T) ^ {\frac {1}{d _ {z} + 2}} T ^ {\frac {d _ {z} + 1}{d _ {z} + 2}} + C ^ {\frac {1}{d _ {z} + 1}} T ^ {\frac {d _ {z}}{d _ {z} + 1}}\right) = \tilde {O} \left(T ^ {\frac {d _ {z} + 1}{d _ {z} + 2}} + C ^ {\frac {1}{d _ {z} + 1}} T ^ {\frac {d _ {z}}{d _ {z} + 1}}\right).
$$
Furthermore, the following Theorem 4.2 implies that our regret bound attains the lower bound and hence is unimprovable. The detailed proof is given in Appendix A.6.1.
Theorem 4.2. Under the strong adversary with a corruption budget of  $C$ , for any zooming dimension  $d_z \in \mathbb{Z}^+$ , there exists an instance for which any algorithm (even one that is aware of  $C$ ) must suffer a regret of order  $\Omega(C^{\frac{1}{d_z + 1}} T^{\frac{d_z}{d_z + 1}})$  with probability at least 0.5.
In addition to the lower bound provided in Theorem 4.2, we further propose another lower bound for the strong adversary particularly in the case that  $C$  is unknown in the following Theorem 4.3:
Theorem 4.3. For any algorithm, when there is no corruption, we denote  $R_T^0$  as the upper bound of cumulative regret in  $T$  rounds under our problem setting described in Section 3, i.e.  $\text{Regret}_T \leq R_T^0$  with high probability, and it holds that  $R_T^0 = o(T)$ . Then under the strong adversary and unknown attacking budget  $C$ , there exists a problem instance on which this algorithm will incur linear regret  $\Omega(T)$  with probability at least 0.5, if  $C = \Omega(R_T^0 / 4^{d_z}) = \Omega(R_T^0)$ .
However, there are also some weaknesses to our Algorithm 1. The first weakness is that the algorithm is too conservative and pessimistic in practice since the second term of  $r(x)$  would dominate under a large given value of  $C$ . We could set the second term of  $r(x)$  as  $\min \{1, C / n(x)\}$  to address this issue and the analysis of Theorem 4.1 will still hold as shown in Appendix A.1 Remark A.5. The second weakness is that it still incurs the same regret bound shown in Theorem 4.1 even if there are actually no corruptions applied. To overcome these problems and to further adapt to the unknown corruption budget  $C$ , we propose two types of robust algorithms in the following Section 5.
5 Robust Lipschitz Bandit with Unknown Budgets
In practice, a decent upper bound of the corruption budget  $C$  would never be revealed to the agent, and hence it is important to develop robust Lipschitz bandit algorithms agnostic to the amount of  $C$ . In this section, we first propose an efficient adaptive-elimination-based approach to deal with the weak adversary, and then we adapt our Algorithm 1 by using several advances of model selection in bandits to defend against the attacks from the strong adversary.
5.1 Algorithm for Weak Adversaries
The weak adversary is unaware of the agent's current action before contaminating the stochastic rewards. We introduce an efficient algorithm called Robust Multi-layer Elimination Lipschitz bandit algorithm (RMEL) that is summarized in Algorithm 2. Four core steps are introduced as follows.