SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 2 Robust Multi-layer Elimination Lipschitz Bandit Algorithm (RMEL)
Input: Arm metric space  $(\mathcal{X},D)$  , time horizon  $T$  , probability rate  $\delta$  , base parameter  $B$    
Initialization: Tolerance level  $v_{l} = \ln (4T / \delta)B^{l - 1},m_{l} = 1,n_{l} = 0,\mathcal{A}_{l} = 1 / 2$  -covering of   
 $\mathcal{X}$ $f_{l,A} = n_{l,A} = 0$  for all  $A\in \mathcal{A}_l,l\in [l^* ]$  where  $l^{*}\coloneqq \min \{l\in \mathbb{N}:\ln (4T / \delta)B^{l - 1}\geq T\}$    
1: for  $t = 1$  to  $T$  do   
2: Sample layer  $l\in [l^{*}]$  with probability  $1 / v_{l}$  , with the remaining probability sampling  $l = 1$  Find the minimum layer index  $l_{t}\geq l$  such that  $\mathcal{A}_{l_t}\neq \emptyset$  ▷Layer sampling   
3: Choose  $A_{t} = \arg \min_{A\in \mathcal{A}_{l_{t}}}n_{l_{t},A}$  , break ties arbitrary.   
4: Randomly pull an arm  $x_{t}\in A_{t}$  , and observe the payoff  $y_{t}$    
5: Set  $n_{l_t} = n_{l_t} + 1$ $n_{l_t,A_t} = n_{l_t,A_t} + 1$  , and  $f_{l,t,A_t} = (f_{l_t,A_t}(n_{l_t,A_t} - 1) + y_t) / n_{l_t,A_t}$    
6: if  $n_{l_t} = 6\ln (4T / \delta)\cdot 4^{m_l}\times |\mathcal{A}_{l_t}|$  then   
7: Obtain  $f_{l,t,*} = \max_{A\in \mathcal{A}_{l_t}}f_{l,t,A}$    
8: For each  $A\in \mathcal{A}_{l_t}$  , if  $f_{l,t,*} - f_{l,t,A} > 4 / 2^{m_l t}$  , then we eliminate  $A$  from  $\mathcal{A}_{l_t}$  and all active regions  $A^\prime$  from  $\mathcal{A}_{l^{\prime}}$  in the case that  $A^{\prime}\subseteq A$ $A^{\prime}\in \mathcal{A}_{l^{\prime}}$ $l^{\prime} <   l$  . ▷Removal   
9: Find  $1 / 2^{m_l + 1}$  -covering of each remaining  $A\in \mathcal{A}_{l_t}$  in the same way as  $A$  was partitioned in other layers. Then reload the active region set  $\mathcal{A}_{l_t}$  as the collection of these coverings.   
10: Set  $n_{l_t} = 0$ $m_{l_t} = m_{l_t} + 1$  . And renew  $n_{l_t,A} = f_{l_t,A} = 0,\forall A\in \mathcal{A}_{l_t}$  ▷Refresh   
11: end if   
12: end for
Multi-layer Parallel Running: Inspired by the multi-layer idea used in robust MABs [29], our algorithm consists of multiple sub-layers running in parallel, each with a different tolerance level against corruptions. As shown in Algorithm 2, there are  $l^*$  layers and the tolerance level of each layer, denoted as  $v_l$ , increases geometrically with a ratio of  $B$  (a hyperparameter). At each round, a layer  $l$  is sampled with probability  $1 / v_l$ , meaning that layers that are more resilient to attacks are less likely to be chosen and thus may make slower progress. This sampling scheme helps mitigate adversarial perturbations across layers by limiting the amount of corruptions distributed to layers whose tolerance levels exceed the unknown budget  $C$  to at most  $O(\ln(T / \delta))$ . For the other low-tolerance layers which may suffer from high volume of attacks, we use the techniques introduced below to rectify them in the guidance of the elimination procedure on robust layers. While we build on the multi-layer idea introduced in [29], our work introduces significant refinements and novelty by extending this approach to continuous and infinitely large arm sets, as demonstrated below.
Active Region Mechanism: For each layer  $\ell$ , our algorithm proceeds in epochs: we initialize the epoch index  $m_{l} = 1$  and construct a  $1/2^{m_{l}}$ -covering of  $\mathcal{X}$  as the active region set  $\mathcal{A}_{l}$ . In addition, we denote  $n_{l}$  as the number of times that layer  $l$  has been chosen, and for each active region  $A \in \mathcal{A}_{l}$  we define  $n_{l,A}, f_{l,A}$  as the number of times  $A$  has been chosen as well as its corresponding average empirical reward respectively. Assume layer  $l_{t}$  is selected at time  $t$ , then only one active region (denoted as  $A_{t}$ ) in  $\mathcal{A}_{l_{t}}$  would be played where we arbitrarily pull an arm  $x_{t} \in A_{t}$  and collect the stochastic payoff  $y_{t}$ . For any layer  $l$ , if each active region in  $\mathcal{A}_{l}$  is played for  $6\ln(4T/\delta) \cdot 4^{m}$  times (i.e. line 6 of Algorithm 2), it will progress to the next epoch after an elimination process that is described below. All components mentioned above that are associated with the layer  $l$  will subsequently be refreshed (i.e. line 10 of Algorithm 2).
Although the elimination idea has been used for various bandit algorithms [35], applying it to our problem setting poses significant challenges due to three sources of error: (1) uncertainty in stochastic rewards; (2) unknown corruptions assigned to each layer; (3) approximation bias between the pulled arm and its neighbors in the same active region. We encapsulate our carefully designed elimination procedure that is specified in line 8 of Algorithm 2 from two aspects:
Within-layer Region Elimination and Discretization: For any layer  $l \in [l^*]$ , the within-layer elimination occurs at the end of each epoch as stated above. We obtain the average empirical reward  $f_{l,A}$  for all  $A \in \mathcal{A}_l$  and then discard regions with unpromising payoffs compared with the optimal one with the maximum estimated reward (i.e.  $f_{l,*}$  defined in line 7 of Algorithm 2). We further "zoom in" on the remaining regions of the layer  $l$  that yield satisfactory rewards: we divide them into  $1/2^{m_l + 1}$ -covering and then reload  $A_l$  as the collection of these new partitions for the next epoch (line 9 of Algorithm 2) for the layer  $l$ . In consequence, only regions with nearly optimal rewards would remain and be adaptively discretized in the long run.
Cross-layer Region Elimination: While layers are running in parallel, it is essential to facilitate communication among them to prevent less reliable layers from getting trapped in suboptimal regions. In our Algorithm 2, if an active region  $A \in \mathcal{A}_l$  is eliminated based on the aforementioned rule, then  $A$  will also be discarded in all layers  $l' \leq l$ . This is because the lower layers are faster whereas more vulnerable and less resilient to malicious attacks, and hence they should learn from the upper trustworthy layers whose tolerance levels surpass  $C$  by imitating their elimination decisions.
Note that the elimination step (line 8 of Algorithm 2) could be executed either after each epoch or after each round. The former, which is used in the current Algorithm 2, is computationally simpler as it does not require the entire elimination procedure to be repeated each time. On the other hand, the latter version is more precise as it can identify and eliminate sub-optimal regions more quickly. We defer the pseudocode and more details of the latter version in Appendix A.5 due to the space limit. Another tradeoff lies in the selection of the hyperparameter  $B$ , which controls the ratio of tolerance levels between adjacent layers. With a larger value of  $B$ , only fewer layers are required, and hence more samples could be assigned to each layer for better efficiency. But the cumulative regret bound would deteriorate since it's associated with  $B$  sub-linearly. The cumulative regret bound is presented in the following Theorem 5.1, with its detailed proof in Appendix A.2.
Theorem 5.1. If the underlying corruption budget is  $C$ , then with probability at least  $1 - \delta$ , the overall regret of our RMEL algorithm (Algorithm 2) could be bounded as:
$$
R e g r e t _ {T} = \tilde {O} \left(\left((B C) ^ {\frac {1}{d _ {z} + 2}} + 1\right) T ^ {\frac {d _ {z} + 1}{d _ {z} + 2}}\right) = \tilde {O} \left(\left(C ^ {\frac {1}{d _ {z} + 2}} + 1\right) T ^ {\frac {d _ {z} + 1}{d _ {z} + 2}}\right).
$$
Proof sketch: It is highly non-trivial to deduce the regret bound depending on  $C$  and  $d_z$  in Theorem 5.1 without the knowledge of these two values. We first bound the cumulative regret occurred in the robust layers whose tolerance levels  $v_{l}$  are no less than the unknown  $C$  by showing the estimated mean  $f_{l,A}$  is close to the underlying ground truth  $\mu (x), x\in A$  for all time  $t$  and active regions  $A$  simultaneously with high probability, i.e. we define the set  $\Phi$  as follows and prove  $P(\Phi)\geq 1 - 3\delta /4$ .
$$
\begin{array}{l} \Phi = \left\{\left| f _ {l, A} - \mu (x) \right| \leq \frac {1}{2 ^ {m _ {l}}} + \sqrt {\frac {4 \ln (T) + 2 \ln (4 / \delta)}{n _ {l , A , t}}} + \frac {\ln (T) + \ln (4 / \delta)}{n _ {l , A , t}}: \right. \\ \left. \forall x \in A, \forall A \in \mathcal {A} _ {l}, \forall l \text {s . t .} v _ {l} \geq C, \forall t \in [ T ] \right\}. \\ \end{array}
$$
This concentration result can guarantee that regions with unfavorable rewards will be adaptively eliminated. And we could show that the extra cumulative regret from the other vulnerable layers be controlled by the regret occurred in the robust layers. A detailed proof is presented in Appendix A.2.
Note that if no corruption is actually applied (i.e.  $C = 0$ ), our RMEL algorithm could attain a regret bound of order  $\tilde{O}\left(T^{\frac{d_z + 1}{d_z + 2}}\right)$  which coincides with the lower bound of stochastic Lipschitz bandits up to logarithmic terms. We further prove a regret lower bound of order  $\Omega(C)$  under the weak adversary in Theorem 5.2 with its detailed proof in Appendix A.6.2. Therefore, a compelling open problem is to narrow the regret gap by proposing an algorithm whose regret bound depends on  $C$  in another additive term free of  $T$  under the weak adversary, like [16] for MABs and [17] for linear bandits.
Theorem 5.2. Under the weak adversary with corruption budget  $C$ , for any zooming dimension  $d_z$  there exists an instance such that any algorithm (even is aware of  $C$ ) must suffer from the regret of order  $\Omega(C)$  with probability at least 0.5.
5.2 Algorithm for Strong Adversaries
In Section 4, we propose the Robust Zooming algorithm to handle the strong adversary given the knowledge of budget  $C$  and prove that it achieves the optimal regret bound. However, compared with the known budget  $C$  case, defending against strong adversaries naturally becomes more challenging when the agent is unaware of the budget  $C$ . Motivated by the literature on model selection in bandits, we extend our Robust Zooming algorithm by combining it with different master algorithms to learn and adapt to the unknown  $C$  on the fly. We consider two approaches along this line: the first approach uses the master algorithms EXP3.P and CORRAL with the smoothing transformation [31] to deal with unknown  $C$ , which leads to a promising regret bound but a high computational cost. We then equip Robust Zooming algorithm with the efficient bandit-over-bandit (BoB) idea [11] to adapt to the unknown  $C$ , leading to a more efficient algorithm with a slightly worse regret bound.
Model Selection: When an upper bound on  $C$  is known, we propose the Robust Zooming algorithm with regret bound  $\tilde{O}\left(T^{\frac{d_z + 1}{d_z + 2}} + C^{\frac{1}{d_z + 1}}T^{\frac{d_z}{d_z + 1}}\right)$  against strong adversaries in Section 4. Therefore, it is natural to consider a decent master algorithm that selects between  $\lceil \log_2(T) \rceil$  base algorithms where the  $i$ -th base algorithm is the Robust Zooming algorithm with corruptions at most  $2^i$ . As  $C \leq T$ , there must exist a base algorithm that is at most  $2C$ -corrupted. Here we choose the stochastic EXP3.P and CORRAL with smooth transformation proposed in [31] as the master algorithm due to the following two reasons with respect to theoretical analysis: (1). our action set  $\mathcal{A}$  is fixed and the expected payoff is a function of the chosen arm, which satisfies the restrictive assumptions of this master algorithm (Section 2, [31]); (2). the analysis in [31] still works even the regret bounds of base algorithms contain unknown values, and note the regret bound of our Zooming Robust algorithm depends on the unknown  $C$ . Based on Theorem 3.2 in [31], the expected cumulative regret of our Robust Zooming algorithm with these two types of master algorithms could be bounded as follows:
Theorem 5.3. When the corruption budget  $C$  is unknown, by using our Algorithm 1 with  $\{2^i\}_{i=1}^{\lceil \log_2(T) \rceil}$  corruptions as base algorithms and the EXP3.P and CORRAL with smooth transformation [31] as the master algorithm, the expected regret could be upper bounded by
$$
\mathbb {E} (R e g r e t _ {T}) = \left\{ \begin{array}{l l} \tilde {O} \left((C ^ {\frac {1}{d + 2}} + 1) T ^ {\frac {d + 2}{d + 3}}\right) & E X P 3. P, \\ \tilde {O} \left((C ^ {\frac {1}{d + 1}} + 1) T ^ {\frac {d + 1}{d + 2}}\right) & C O R R A L. \end{array} \right.
$$
We can observe that the regret bounds given in Theorem 5.3 are consistent with the lower bounds presented in Theorem 4.3. And CORRAL is better under small corruption budgets  $C$  (i.e.  $C = \tilde{O}(T^{\frac{d + 1}{d + 3}})$ ) whereas EXP3.P is superior otherwise. Note that the order of regret relies on  $d$  instead of  $d_z$  since the unknown  $d_z$  couldn't be used as a parameter in practice, and both regret bounds are worse than the lower bound given in Theorem 4.2 for the strong adversary. Another drawback of the above method is that a two-step smoothing procedure is required at each round, which is computationally expensive. Therefore, for better practical efficiency, we propose a simple BoB-based method as follows:
BoB Robust Zooming: The BoB idea [11] is a special case of model selection in bandits and aims to adjust some unspecified hyperparameters dynamically in batches. Here we use  $\lceil \log_2(T) \rceil$  Robust Zooming algorithms with different corruption levels shown above as base algorithms in the bottom layer and the classic EXP3.P [4] as the top layer. Our method, named BoB Robust Zooming, divides  $T$  into  $H$  batches of the same length, and in one batch keeps using the same base algorithm that is selected from the top layer at the beginning of this batch. When a batch ends, we refresh the base algorithm and use the normalized accumulated rewards of this batch to update the top layer EXP3.P since the EXP3.P algorithm [4] requires the magnitude of rewards should at most be 1 in default. Specifically, we normalize the cumulative reward at the end of each batch by dividing it with  $(2H + \sqrt{2H\log(12T / H\delta)})$  due to the fact that the magnitude of the cumulative reward at each batch would at most be this value with high probability as shown in Lemma A.14 in Appendix A.4. Note that this method is highly efficient since a single update of the EXP3.P algorithm only requires  $O(1)$  time complexity, and hence the additional computation from updating EXP3.P is only  $O(H)$ . Due to space limit, we defer Algorithm 4 to Appendix A.5, and the regret bound is given as follows:
Theorem 5.4. When the corruption budget  $C$  is unknown, with probability at least  $1 - \delta$ , the regret of our BoB Robust Zooming algorithm with  $H = T^{(d + 2) / (d + 4)}$  could be bounded as:
$$
R e g r e t _ {T} = \tilde {O} \left(T ^ {\frac {d + 3}{d + 4}} + C ^ {\frac {1}{d + 1}} T ^ {\frac {d + 2}{d + 3}}\right).
$$
Although we could deduce the more challenging high-probability regret bound for this algorithm, its order is strictly worse than those given in Theorem 5.3. In summary, the BoB Robust Zooming algorithm is more efficient and easier to use in practice, while yielding worse regret bound in theory. However, due to its practical applicability, we will implement this BoB Robust Zooming algorithm in the experiments. It is also noteworthy that we can attain a better regret bound with Algorithm 2 under the weak adversary as shown in Theorem 5.1, which aligns with our expectation since the strong adversary considered here is more malicious and difficult to defend against.
Figure 1: Plots of regrets of Zooming algorithm (blue), RMEL (green) and BoB Robust Zooming algorithm (red) under different settings with three levels of corruptions: (1) dotted line: no corruption; (2) dashed line: moderate corruptions; (3) solid line: strong corruptions. Numerical values of final cumulative regrets in our experiments are also displayed in Table 3 in Appendix A.7.
In this section, we show by simulations that our proposed RMEL and BoB Robust Zooming algorithm outperform the classic Zooming algorithm in the presence of adversarial corruptions. To firmly validate the robustness of our proposed methods, we use three types of models and two sorts of attacks with different corruption levels. We first consider the metric space  $[0,1],\lvert\rangle$  with two expected reward functions that behave differently around their maximum: (1).  $\mu (x) = 0.9 - 0.95|x - 1 / 3|$  (triangle) and (2).  $\mu (x) = 2 / (3\pi)\cdot \sin (3\pi x / 2)$  (sine). We then utilize a more complicated metric space  $([0,1]^2,\| \cdot \|_{\infty})$  with the expected reward function (3).  $\mu (x) = 1 - 0.8\| x - (0.75,0.75)\| _2-$ $0.4\| x - (0,1)\| _2$  (two dim). We set the time horizon  $T = 50,000$  (60,000) for the metric space with  $d = 1$  (2) and the false probability rate  $\delta = 0.01$ . The random noise at each round is sampled IID from  $N(0,0.01)$ . Average cumulative regrets over 20 repetitions are reported in Figure 1.
Since adversarial attacks designed for stochastic Lipschitz bandits have never been studied, we extend two types of classic attacks, named Oracle [18] for the MAB and Garcelon [15] for the linear bandit, to our setting. The details of these two attacks are summarized as follows:
- Oracle: This attack [18] was proposed for the traditional MAB, and it pushes the rewards of "good arms" to the very bottom. Specifically, we call an arm is benign if the distance between it and the optimal arm is no larger than 0.2. And we inject this attack by pushing the expected reward of any benign arm below that of the worst arm with an additional margin of 0.1 with probability 0.5.
- **Garcelona:** We modify this type of attack studied in [15] for linear bandit framework, which replaces expected rewards of arms outside some targeted region with IID Gaussian noise. For  $d = 1$ , since the optimal arm is set to be  $1/3$  for both triangle and sine payoff functions, we set the targeted arm interval as  $[0.5, 1]$ . For  $d = 2$ , since the optimal arm is close to  $(0.75, 0.75)$ , we set the targeted region as  $[0, 0.5]^2$ . Here we contaminate the stochastic reward if the pulled arm is not inside the target region by modifying it into a random Gaussian noise  $N(0, 0.01)$  with probability 0.5.
We consider the strong adversary in experiments as both types of attack are injected only if the pulled arms lie in some specific regions. Note although we originally propose RMEL algorithm for the weak adversary in theory, empirically we find it works exceptionally well (Figure 1) across all settings here. We also conduct simulations based on the weak adversary and defer their settings and results to Appendix A.7 due to the limited space. The first Oracle attack is considered to be more malicious in the sense that it specifically focuses on the arms with good rewards, while the second Garcelon attack could corrupt rewards generated from broader regions, which may contain some "bad arms" as well.
Since there is no existing robust Lipschitz bandit algorithm, we use the classic Zooming algorithm [23] as the baseline. As shown in Figure 1, we consider three levels of quantities of corruptions applied on each case to show how attacks progressively disturb different methods. Specifically, we set  $C = 0$  for the non-corrupted case,  $C = 3,000$  for the moderate-corrupted case and  $C = 4,500$  for the strong-corrupted case. Due to space limit, we defer detailed settings of algorithms to Appendix A.7.
From the plots in Figure 1, we observe that our proposed algorithms consistently outperform the Zooming algorithm and achieve sub-linear cumulative regrets under both types of attacks, whereas the Zooming algorithm becomes incompetent and suffers from linear regrets even under a moderate volume of corruption. This fact also implies that the two types of adversarial corruptions used here are severely detrimental to the performance of stochastic Lipschitz bandit algorithms. And it is evident our proposed RMEL yields the most robust results under various scenarios with different volumes of attacks. It is also worth noting that the Zooming algorithm attains promising regrets under a purely stochastic setting, while it experiences a huge increase in regrets after the corruptions emerge. This phenomenon aligns with our expectation and highlights the fact that our proposed algorithms balance the tradeoff between accuracy and robustness in a much smoother fashion.
In this work we introduce a new problem of Lipschitz bandits in the presence of adversarial corruptions, and we originally provide efficient algorithms against both weak adversaries and strong adversaries when agnostic to the total corruption budget  $\bar{C}$ . The robustness and efficiency of our proposed algorithms is then validated under comprehensive experiments.
Limitations: For both the weak and strong adversary, our work leaves a regret gap between the lower bound deduced in Theorem 5.2 (Theorem 4.2) and the upper bound in Theorem 5.1 (Theorem 5.3) when agnostic to  $C$ . Closing or narrowing this regret gap seems highly non-trivial since the regret gap still exists under the simpler MAB setting [16], and we will leave it as a future work.
We appreciate the insightful comments from the anonymous reviewers and area chair. This work was partially supported by the National Science Foundation under grants CCF-1934568, DMS-1916125, DMS-2113605, DMS-2210388, IIS-2008173 and IIS2048280.
[1] Yasin Abbasi-Yadkori, David Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems, 24, 2011.
[2] Rajeev Agrawal. The continuum-armed bandit problem. SIAM journal on control and optimization, 33(6):1926-1951, 1995.
[3] Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47:235-256, 2002.
[4] Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic multiarmed bandit problem. SIAM journal on computing, 32(1):48-77, 2002.
[5] Alina Beygelzimer, John Langford, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandit algorithms with supervised learning guarantees. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 19-26. JMLR Workshop and Conference Proceedings, 2011.
[6] Ilija Bogunovic, Arpan Losalka, Andreas Krause, and Jonathan Scarlett. Stochastic linear bandits robust to adversarial attacks. In International Conference on Artificial Intelligence and Statistics, pages 991–999. PMLR, 2021.
[7] Sébastien Bubeck and Aleksandris Slivkins. The best of both worlds: Stochastic and adversarial bandits. In Conference on Learning Theory, pages 42-1. JMLR Workshop and Conference Proceedings, 2012.