SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\tilde{N}_a(t) \coloneqq \sum_{s=1}^t \sum_{i=1}^K \kappa_i I\{a_{i,s} = a\}$  is the position bias-corrected counter,  $\hat{\mu}_a(t)$  is the empirical mean of item  $a$ , and  $\epsilon$  is a parameter of the algorithm. The algorithm then chooses the corresponding first  $K$  items with the highest UCB indices as the recommendation list.
We propose our attack algorithm against PBM-UCB in Algorithm 2. It works by first randomly taking  $K - 1$  items out and making them a set with the target item  $L$ , denoted as  $\pmb{a}^{*} = \{a_{1}^{*},\dots ,a_{K - 1}^{*},L\}$ . Then, based on the conservative estimation idea from Algorithm 1, it maintains a timestamp  $h(l,a)$  for each item  $l$  with respect to each  $a\in \pmb{a}^*$ . The intuition is that, to ensure a similar inequality as Eq. (4) for all rounds, we need to make conservative estimation on the lower bounds of  $\mu_{a}$  for all  $a\in \pmb{a}^{*}$ . This is handled by Algorithm 3, which maintains the timestamps  $h_{l,a}(t)$  for the input item  $l$  and outputs the appropriate attack value  $\alpha$  on  $l$  that can always ensure the required inequality. The value of parameters  $\gamma_t(l,a)$  and  $\tilde{\gamma}_t(l,a)$  in Algorithm 3 are computed as
$$
\gamma_ {t} (l, a) = \left[ N _ {l} (t) \hat {\mu} _ {l} ^ {0} (t) - \sum_ {s \in \tau_ {l} (t - 1)} \alpha_ {l} (s) - N _ {l} (t) \left[ \underline {{\mu}} _ {a} (t) - \Delta_ {0} \right] _ {+} \right] _ {+}, \tag {6}
$$
$$
\tilde {\gamma} _ {t} (l, a) = \left[ N _ {l} (t) \hat {\mu} _ {l} ^ {0} (t) - \sum_ {s \in \tau_ {l} (t - 1)} \alpha_ {l} (s) - N _ {l} (t) \left[ \underline {{\mu}} _ {a} \left(h _ {l, a} (t - 1)\right) - \Delta_ {0} \right] _ {+} \right] _ {+}. \tag {7}
$$
Algorithm 4 Attack against the CascadeUCB algorithm
1: Initialization: Randomly select  $K - 1$  items with  $L$  to build  $\pmb{a}^{*}$  ..  $h_{l,a}(0) = 1\forall l\in [L]\forall a\in \pmb{a}^{*}$    
2: for  $t = 1,2,3,\ldots$  do   
3: Observe  $\pmb {a}_t,\pmb {r}_t^0,\mathbf{C}_t$  ; set  $\alpha_{t} = (\alpha_{1,t},\dots ,\alpha_{K,t}) = (0,\dots ,0)$    
4: if  $\mathbf{C}_t\leq K$  and  $a_{\mathbf{C}_t,t}\notin \pmb{a}^*$  then   
5:  $\alpha_{\mathbf{C}_{t},t} = \mathrm{CAL\_ALPHA}(a_{\mathbf{C}_{t},t},r_{\mathbf{C}_{t},t}^{0},\pmb{a}^{*})$    
6: if  $\alpha_{\mathbf{C}_{t},t} = 1$  and  $\exists i > C_t$  s.t.  $a_{i,t}\in \pmb{a}^{*}$  then   
7:  $\alpha_{i,t} = -1$    
8: end if   
9: end if   
10: Return  $\boldsymbol {r}_t = \boldsymbol {r}_t^0 -\boldsymbol {\alpha}_t$  ; update  $h_{l,a}(t) = h_{l,a}(t - 1)$  for all  $l\neq a_{\mathbf{C}_{t},t},a\in \pmb{a}^{*}$    
11: end for
Notice that Algorithm 2 could also work when there are more than one but less than  $K + 1$  target arms (the goal of the attacker becomes misguiding the player to recommend all target arms very often with sublinear cost). The only modification required is to put all of these target arms into  $\mathbf{a}^*$ .
Analysis. The following theorem shows the attack against PBM-UCB is successful.
Theorem 2. Suppose  $T \geq L$  and  $\delta \leq 1/2$ . With probability at least  $1 - \delta$ , Algorithm 2 misguides the PBM-UCB algorithm to recommend the target item  $L$  at least  $T - (L - K)\left(\frac{1 + \epsilon}{2\kappa_K^2\Delta_0^2}\log T\right)$  rounds, using a cumulative attack cost at most
$$
C (T) \leq \left(\frac {1 + \epsilon}{2 \kappa_ {K} ^ {2} \Delta_ {0} ^ {2}} \log T\right) \sum_ {a <   L} \left(\Delta_ {a} + \Delta_ {0} + 4 \beta \left(\frac {1 + \epsilon}{2 \kappa_ {K} ^ {2} \Delta_ {0} ^ {2}} \log h _ {a, L} (T)\right)\right).
$$
When  $T$  goes to infinity, we have
$$
\lim _ {T \to \infty} \frac {C (T)}{\log T} \leq O \left(\sum_ {a <   K} \frac {(1 + \epsilon) (\Delta_ {a} + \Delta_ {0})}{\kappa_ {K} ^ {2} \Delta_ {0} ^ {2}}\right).
$$
Proof sketch. Whenever  $a_{i,t} \notin \mathbf{a}^*$  is chosen by the player, there must exist  $a \in \mathbf{a}^*$  such that  $\bar{\mu}_{a_i,t}(t) \geq \bar{\mu}_a(t)$ . The output  $\alpha_{i,t}$  of Algorithm 3 would ensure  $\hat{\mu}_{a_i,t}(t) \leq \hat{\mu}_a(h_{a_i,t,a}(t - 1)) - 2\beta (N_a(h_{a_i,t,a}(t - 1))) - \Delta_0$ , owing to the conservative estimations in  $\gamma_t(a_{i,t},a)$  and  $\tilde{\gamma}_t(a_{i,t},a)$ . Combining these two inequalities, we can get  $B_{a_i,t}(t) - B_a(t) \geq \Delta_0$ . With a careful calculation on this inequality and the involved bias-corrected counters, we have  $N_{a_i,t}(t) \leq \left(\frac{1 + \epsilon}{2\kappa_K^2\Delta_0^2}\log t\right)$ . This result holds for any  $a_{i,t} \notin \mathbf{a}^*$ . Thus, we immediately get the bound of  $N_L(t)$ . The remaining proof for the cost bound will be similar to that of Theorem 1.
Compared with Theorem 1, the asymptotic cost of Algorithm 2 has an additional dependency on  $1 / \kappa_K^2$ , which suggests it may require more cost to achieve a successful attack in the PBM model, though the cost dependency on  $T$  is still logarithmic.
4.2 Cascade Click Model
Threat Model. We introduce the threat model for OLTR with cascade click feedback. In each round  $t$ , the player chooses a list of  $K$  items,  $\mathbf{a}_t = (a_{1,t},\dots ,a_{K,t})$  to recommend. The environment generates the pre-attack click feedback  $\boldsymbol{r}_t^0 = (r_{1,t}^0,\dots ,r_{K,t}^0)\in \{0,1\} ^K,||\boldsymbol {r}_t^0 ||\leq 1$ . Let  $\mathbf{C}_t$  denote the position of the clicked item, i.e.,  $r_{\mathbf{C}_t,t}^0 = 1$  ( $\mathbf{C}_t = \infty$  if no item was clicked). The attacker observes  $\mathbf{a}_t,r_t^0,\mathbf{C}_t$ , and decides the post-attack click feedback  $\boldsymbol{r}_t\in \{0,1\} ^K,||\boldsymbol {r}_t||_1\leq 1$ . The player only receives  $\mathbf{r}_t$  as the feedback and uses it to decide the next list to recommend,  $\mathbf{a}_{t + 1}$ , for round  $t + 1$ . The goal of the attacker in this setting is the same as that in the PBM model.
Attack against CascadeUCB. We propose an attack algorithm against CascadeUCB in Algorithm 4. Similar to the attack against PBM-UCB, it first randomly generates a set of items  $\mathbf{a}^{*} = \{a_{1}^{*},\dots ,a_{K - 1}^{*},L\}$ . It also follows the idea of conservative estimation: when the clicked item  $a_{\mathbf{C}_t,t}$  does not belong to  $\mathbf{a}^*$ , it calls Algorithm 3 to maintain  $h_{a_{\mathbf{C}_t,t},a}$  for all  $a\in \mathbf{a}^*$  and compute the attack value  $\alpha_{\mathbf{C}_t,t}$  based on the conservative estimations. If the output  $\alpha_{\mathbf{C}_t,t} = 1$ , which means
Algorithm 5 Attack against arbitrary algorithm
1: Initialization: Randomly select  $K - 1$  items with target  $L$  to build  $\pmb{a}^{*}$   
2: for  $t = 1,2,3,\ldots$  do  
3: Observe  $\pmb{a}_t, \pmb{r}_t^0$   
4: for  $i \in [K]$  do  
5:  $\alpha_{i,t} = 0$   
6: if  $a_{i,t} \notin \pmb{a}^{*}$  and  $r_{i,t}^0 = 1$  then  
7:  $p_{i,t} = \max_{a \in \pmb{a}^{*}} \frac{\left[\hat{\mu}_{a_i,t}^0(t) + \beta(N_{a_i,t}(t)) - \hat{\mu}_a^0(t) + \beta(N_a(t))\right]_+}{\hat{\mu}_{a_i,t}^0(t) + \beta(N_{a_i,t}(t))}$   
8: With prob.  $p_{i,t}$ , set  $\alpha_{i,t} = 1$   
9: end if  
10: end for  
11: Return  $\pmb{r}_t = \pmb{r}_t^0 - \pmb{\alpha}_t$   
12: end for
the algorithm sets the clicked position to be zero, the player will keep checking the positions after  $\mathbf{C}_t$ . Since the pre-attack feedback of all items after position  $\mathbf{C}_t$  is zero, we need to find the first item  $a_{i,t} \in \pmb{a}^*$  after position  $\mathbf{C}_t$  and set  $\alpha_{i,t} = -1$  ( $r_{i,t} = 1$ ). The empirical means of the items between position  $\mathbf{C}_t$  and  $i$  that are not in  $\pmb{a}^*$  decreases, and the empirical mean of  $a_{i,t} \in \pmb{a}^*$  increases. Thus, it does not affect the success of the attack while still making the post-attack feedback  $\pmb{r}_t$  valid.
Analysis. First, note that there is a mismatch between recommendation and observation in the cascade model: for all  $i$  such that  $\mathbf{C}_t < i \leq K$ ,  $a_{i,t}$  is recommended but not observed, i.e., there is no new click sample for item  $a_{i,t}$  for estimation. We can still follow a similar proof of Theorem 2 to get the upper bound of the number of observations (samples) for  $a_{i,t} \notin \mathbf{a}^*$ , but it is less than the number of recommendations and thus cannot ensure the success of the attack. To tackle this problem, we use a new regret-based analysis on the expected number of recommendations. We can also prove that the post-attack feedback of Algorithm 4 is always valid. Define  $p^* := \prod_{i=1}^{K-1} \mu_i$ . We give the following theorem of the successful attack against CascadeUCB.
Theorem 3. Suppose  $T \geq L$  and  $\delta \leq 1/2$ . With probability at least  $1 - \delta$ , Algorithm 2 misguides the CascadeUCB algorithm to choose the target arm at least  $T - (L - K)\left(\frac{12}{p^{*}\Delta_{0}^{2}}\log T\right)$  rounds in expectation. Its cumulative attack cost is at most
$$
C (T) \leq \left(1 + \frac {3}{\Delta_ {0} ^ {2}} \log T\right) \sum_ {a <   K} \left(\Delta_ {a} + \Delta_ {0} + 4 \beta \left(1 + \frac {3}{\Delta_ {0} ^ {2}} \log h _ {a, L} (T)\right)\right).
$$
Proof sketch. As mentioned above, we use a new regret-based analysis. The intuition is that the regret caused by any suboptimal item is the product of its reward gap with respect to the optimal list and its expected number of recommendations. If we know the regret upper bound and the reward gap lower bound, we can derive the upper bound of the expected number of recommendations. To do so, we first show that the post-attack problem can be viewed as a problem with a known reward gap lower bound. This can be obtained by  $\hat{\mu}_{a_{i,t}(t)} \leq \hat{\mu}_L(h_{L_{i,t},a}(t - 1)) - 2\beta (N_a(h_{a_{i,t},L}(t - 1))) - \Delta_0$  for any  $a_{i,t} \notin \mathbf{a}^*$ , which indicates the post-attack expected reward gap between  $a_{i,t}$  and  $L$  is always larger than  $\Delta_0$ . Then, the lower bound of the reward gap of  $a_{i,t}$  with respect to the optimal list is  $p^*\Delta_0$ . Based on the regret upper bound  $12\log T / \Delta_0$  given in [5], we can get the upper bound of the expected number of recommendations  $\frac{12}{p^*\Delta_0^2}\log T$ . For the remaining cost analysis, since the attack cost only depends on the observed items, we can still follow the proof of Theorem 2.
4.3 General Attacks on OLTR with General Click Model
We have provided attack strategies against UCB-based OLTR algorithms under two specific click models. A natural follow-up question is whether there exists an attack strategy that can attack any
OLTR algorithm under the general click model. To answer this question in what follows, we design an attack strategy that can misguide any OLTR algorithm without knowing the underlying algorithm. However, it may pay more cost (still sublinear) than the others since it cannot take advantage of the details of the algorithm to make fine-tuned adjustments.
Threat Model. We consider the threat model for OLTR with general click feedback. In each round  $t$ , the player chooses a list of  $K$  items,  $\pmb{a}_t = (a_{1,t},\dots ,a_{K,t})$  to recommend. The environment generates the pre-attack click feedback  $\pmb{r}_t^0 = (r_{1,t}^0,\dots ,r_{K,t}^0)\in \{0,1\} ^K$ ,  $\pmb{r}_t^0\in \mathcal{R}_c$ , where  $\mathcal{R}_c$  is the feasible feedback space of click model  $c$ . The attacker observes  $\pmb{a}_t,\pmb{r}_t^0$ , and decides the post-attack click feedback  $\pmb{r}_t\in \{0,1\} ^K$ ,  $\pmb{r}_t\in \mathcal{R}_c$ . The attack should be aware of  $\mathcal{R}_c$ ; otherwise, ensuring valid post-attack feedback is impossible. The player only receives  $\pmb{r}_t$  as the feedback and uses it to decide the next list to recommend,  $\pmb{a}_{t + 1}$ . The goal of the attacker is the same as that in the PBM model.
General Attack against Arbitrary OLTR Algorithm. We propose an attack algorithm against arbitrary OLTR algorithms in Algorithm 5. Similar to the attack against PBM-UCB, it first randomly generates a set of items  $\pmb{a}^{*} = \{a_{1}^{*},\dots ,a_{K - 1}^{*},L\}$ . In each round, for each clicked item  $a_{i,t}\notin \pmb{a}^{*}$ , the algorithm calculates an attack probability  $p_{i,t}$  and uses that to decide whether its feedback needs to be changed to unclicked  $(r_{i,t} = 0)$ . We prove that  $p_{i,t}$  is actually an estimated upper bound of  $\Delta_i / \mu_i$ , thus by such probabilistic feedback perturbations, the algorithm makes all items outside  $\pmb{a}^{*}$  be worse than the items inside  $\pmb{a}^{*}$ , which guarantees the success of the attack. We consider a general assumption for the OLTR algorithm to be attacked.
Assumption 1. The OLTR algorithm chooses suboptimal items no more than  $R(T) = o(T)$  times for  $T$  rounds,
Notice that both PBM-UCB and CascadeUCB satisfy the assumption with  $R(T) = O(\log T)$ . We give the following theorem to show the success of our general attack algorithm.
Theorem 4. Suppose  $T \geq L$  and  $\delta \leq 1/2$ . With probability at least  $1 - \delta$ , Algorithm 5 misguides arbitrary OLTR algorithm that satisfies Assumption 1 to choose the target item at least  $T - R(T)$  rounds in expectation. Its cumulative attack cost is at most
$$
C (T) \leq O \left(\sum_ {a <   L} (\Delta_ {a} + 4 \beta (1)) R (T)\right).
$$
Compared with the results in the PBM and cascade models, the  $\beta()$  term in the cumulative cost of Algorithm 5 is  $\beta(1)$ , which can be much larger than the others. Also, the asymptotic costs of Algorithm 2 and Algorithm 4 is independent of  $\beta$ , while the asymptotic cost of Algorithm 5 depends on  $\beta$ , showing that Algorithm 5 is more costly than attack strategies specific to OLTR algorithms.
We conduct experiments using both synthetic and real data (MovieLens 20M dataset [26]). Due to space limitations, we report only the results of OLTR using the position-based model. We use  $\epsilon = 0.1$  for the PBM-UCB algorithm. For the synthetic data, we take  $L = 16$ ,  $K = 8$ ,  $T = 100$ , 000;  $\{\mu_i\}_{i=1}^L$  are sampled from uniform distribution  $U(0,1)$  for Figure 1a, and from  $U(0,x)$  for Figure 1b. For the real data, we take  $L = 100$ ,  $K = 10$ ,  $T = 100$ , 000;  $\{\mu_i\}_{i=1}^L$  are extracted according to [2].
Using synthetic data, we first study how the algorithm's performance is influenced by the algorithmic and problem parameters. As shown in Figure 1a, the cost decreases with an increase in  $\Delta_0$ , aligning with our observations in Theorem 2. Figure 1b shows that the cost increases as  $x$  increases, which suggests that our algorithm pays more costs when  $\Delta_a$  is large. We then compare our algorithm with two baselines: Trivial<sub>all</sub> attacks all arms except the target arm as long as the attack is valid; Trivial<sub set</sub> first randomly takes  $K - 1$  arms out to generate a set  $a^*$  and then attacks all arms outside the set as long as the attack is valid. Figure 2 shows that Trivial<sub>all</sub> algorithm cannot successfully attack PBM-UCB even with linear costs. Our algorithm and Trivial<sub set</sub> have similar performance on the chosen ratio of the target arm as shown in Figures 2a and 2c, which is the chosen time of the target arm divided by the current round. However, our algorithm pays  $50\%$  and  $40\%$  less cost than Trivial<sub set</sub> in Figures 2b and 2d, respectively, which validates the necessity of our attack design.
(a) Cost as  $\Delta_0$  varies
(b) Cost as  $x$  varies
(a) Chosen ratio (syn.)
(b) Cost (syn.)
(c) Chosen ratio (real)
Figure 2: Companion with baseline algorithms.
Figure 1: Impact of algorithmic and problem parameters on PBM-UCB.
(d) Cost (real)
6 Concluding Remarks
This paper presents the first study on adversarial attacks against OLTR with click feedback. We propose attack strategies that can successfully mislead various OLTR algorithms across multiple click models. One limitation of our work is that it cannot be applied to feedback models without clicks, such as the top- $K$  feedback [13]. Although we provide theoretical results for all these strategies, as discussed in Section 3.3, the finite-time cost results might be further improved by a more fine-grained analysis on  $h_a(t)$ . Additionally, there is no known lower bound for the cumulative attack cost, even for the stochastic bandit setting in the literature, making it unclear whether our attack strategies are (asymptotically) order-optimal. This study opens up several future directions. One is to design attack strategies for other OLTR algorithms and feedback models. Lastly, our study on the vulnerability of existing OLTR algorithms inspires the design of robust OLTR algorithms against adversarial attacks.
Shuai Li is supported by National Key Research and Development Program of China (2022ZD0114804) and National Natural Science Foundation of China (62376154, 62006151, 62076161). M. Hajiesmaili's work is supported by CAREER-2045641, CNS-2102963, CNS-2106299, and CPS-2136199. Wierman is supported by NSF grants CNS-2146814, CPS-2136197, CNS-2106403, NGSDI-2105648.
[1] Masour Zoghi, Tomas Tunys, Mohammad Ghavamzadeh, Branislav Kveton, Csaba Szepesvari, and Zheng Wen. Online learning to rank in stochastic click models. In International conference on machine learning, pages 4199-4208. PMLR, 2017.
[2] Shuai Li, Tor Lattimore, and Csaba Szepesvári. Online learning to rank with features. In International Conference on Machine Learning, pages 3856-3865. PMLR, 2019.
[3] Tor Lattimore, Branislav Kveton, Shuai Li, and Csaba Szepesvari. Toprank: A practical algorithm for online stochastic ranking. Advances in Neural Information Processing Systems, 31, 2018.
[4] Paul Lagrée, Claire Vernade, and Olivier Cappe. Multiple-play bandits in the position-based model. Advances in Neural Information Processing Systems, 29, 2016.
[23] Evrard Garcelon, Baptiste Roziere, Laurent Meunier, Jean Tarbouriech, Olivier Teytaud, Alessandro Lazaric, and Matteo Pirotta. Adversarial attacks on linear contextual bandits. Advances in Neural Information Processing Systems, 33:14362-14373, 2020.
[24] Yuzhe Ma and Zhijin Zhou. Adversarial attacks on adversarial bandits. International Conference on Learning Representations, 2023.
[25] Huazheng Wang, Haifeng Xu, and Hongning Wang. When are linear stochastic bandits attackable? In International Conference on Machine Learning, pages 23254-23273. PMLR, 2022.
[26] Movielens 20m dataset. https://grouplens.org/datasets/movielens/20m/. [Online].
A Proofs
A.1 Proof of Lemma 1
Proof. When  $\gamma_t \leq r_t^0$ , it is easy to check  $\alpha_t = \lceil \gamma_t \rceil \leq r_t^0$ , thus  $r_t \in \{0,1\}$ .
When  $\gamma_t > r_t^0$ , if  $\tilde{\gamma}_t \leq r_t^0$ , we will have  $\alpha_t = \lceil \tilde{\gamma}_t \rceil \leq r_t^0$  and  $r_t \in \{0,1\}$ . Since  $\tilde{\gamma}_t$  depends on  $a_t$ , our goal is to show  $\tilde{\gamma}_t \leq r_t^0$  for any  $a_t$ . Consider any arm  $a \neq K$ . We denote  $t_{a,j}$  as the  $j$ -th time that the UCB algorithm played  $a$ . Since the UCB algorithm would play each arm one time in the beginning, we have
$$
\tilde {\gamma} _ {t _ {a, 1}} \leq \left[ N _ {a} \left(t _ {a, 1}\right) \hat {\mu} _ {a} ^ {0} \left(t _ {a, 1}\right) \right] _ {+} = \left[ r _ {t _ {a, 1}} ^ {0} \right] _ {+} \leq r _ {t _ {a, 1}} ^ {0}. \tag {8}
$$
Next, we want to show that if  $\tilde{\gamma}_{t_{a,j}} \leq r_{t_{a,j}}^{0}$ ,  $\tilde{\gamma}_{t_{a,j+1}} \leq r_{t_{a,j+1}}^{0}$ . Since  $\tilde{\gamma}_t$  also depends on  $h_{a_t}(t - 1)$  we consider two cases:
1)  $\gamma_{t_{a,j}} \leq r_{t_{a,j}}^{0}$ . In this case, we have  $h_a(t_{a,j+1} - 1) = t_{a,j}$ . We can bound  $\tilde{\gamma}_{t_{a,j+1}}$  using  $\gamma_{t_{a,j}}$ :
$$
\tilde {\gamma} _ {t _ {a, j + 1}} \leq \gamma_ {t _ {a, j}} + r _ {t _ {a, j + 1}} ^ {0} - \alpha_ {t _ {a, j}} - \left[ \underline {{\mu}} _ {L} \left(t _ {a, j}\right) - \Delta_ {0} \right] _ {+} \leq r _ {t _ {a, j + 1}} ^ {0}, \tag {9}
$$
where the second inequality is due to  $\gamma_{t_{a,j}}\leq \alpha_{t_{a,j}}$
2)  $\gamma_{t_{a,j}} > r_{t_{a,j}}^{0}$ . In this case, we have  $h_a(t_{a,j+1} - 1) = h_a(t_{a,j} - 1)$ . We can bound  $\tilde{\gamma}_{t_{a,j+1}}$  using
$$
\tilde {\gamma} _ {t _ {a, j + 1}} \leq \tilde {\gamma} _ {t _ {a, j}} + r _ {t _ {a, j + 1}} ^ {0} - \alpha_ {t _ {a, j}} - \left[ \underline {{\mu}} _ {L} \left(h _ {a} \left(t _ {a, j} - 1\right)\right) - \Delta_ {0} \right] _ {+} \leq r _ {t _ {a, j + 1}} ^ {0}, \tag {10}
$$
where the second inequality is due to  $\tilde{\gamma}_{t_{a,j}}\leq r_{t_{a,j}}^{0}$  and  $\tilde{\gamma}_{t_{a,j}}\leq \alpha_{t_{a,j}}$
Since  $\tilde{\gamma}_{t_{a,1}} \leq r_{t_{a,1}}^{0}$ , by induction, we have  $\tilde{\gamma}_{t_{a,j}} \leq r_{t_{a,j}}^{0}$  for any  $a, j$ , which concludes the proof.
A.2 Proof of Theorem 1
Proof. We use the following two lemmas to prove Theorem 1. The proof is similar to those of Lemma 5 and Lemma 6 in [9], while as discussed in Section 3, the algorithm in [9] relies on Eq. (1) that may not hold due to the binary feedback, and our proof is mainly based on Eq. (4) that can be ensured by the conservative estimations. Define event  $E \coloneqq \{\forall i, \forall t > L : |\hat{\mu}_i^0(t) - \mu_i| < \beta(N_i(t))\}$ . With Hoeffding's inequality, it is easy to prove for  $\delta \in (0,1)$ ,  $\mathbb{P}(E) > 1 - \delta$ .
Lemma 2. Assume event  $E$  holds and  $\delta \leq 1/2$ . For any  $a \neq L$  and any  $t \geq L$ , we have
$$
N _ {a} (t) \leq \min  \left\{N _ {L} (t), 1 + \frac {3}{\Delta_ {0} ^ {2}} \log t \right\} \tag {11}
$$
Proof. Fix some  $t \geq L$ . We consider  $a_{t} = a \neq L$  and denote  $t' = \max \{s < t : a_{s} = a\}$  as the previous round that  $a$  was pulled. Since arm  $a$  should be pulled at least once by the UCB algorithm, we have  $t' \geq 1$ , and the attack in round  $t'$  would ensure
$$
\hat {\mu} _ {a} \left(t ^ {\prime}\right) \leq \underline {{\mu}} _ {L} \left(h _ {a} \left(t ^ {\prime}\right)\right) - \Delta_ {0}. \tag {12}
$$
Since arm  $a$  was pulled in round  $t$ , we know its UCB value must be greater than that of target arm  $L$ :
$$
\hat {\mu} _ {a} (t - 1) + \frac {3}{2} \sqrt {\frac {\log (t)}{N _ {a} (t - 1)}} \geq \hat {\mu} _ {L} (t - 1) + \frac {3}{2} \sqrt {\frac {\log (t)}{N _ {L} (t - 1)}}. \tag {13}
$$
Considering that  $t'$  and  $t$  are two consecutive rounds when arm  $a$  was pulled, we have  $\hat{\mu}_a(t - 1) = \hat{\mu}_a(t')$  and  $N_a(t - 1) = N_a(t')$ . Rearranging Eq. (13), we get
$$
\frac {3}{2} \sqrt {\frac {\log (t)}{N _ {a} \left(t ^ {\prime}\right)}} - \frac {3}{2} \sqrt {\frac {\log (t)}{N _ {L} (t - 1)}} \geq \hat {\mu} _ {L} (t - 1) - \hat {\mu} _ {a} \left(t ^ {\prime}\right) \geq \hat {\mu} _ {L} (t - 1) - \left(\underline {{\mu}} _ {L} \left(h _ {a} \left(t ^ {\prime}\right)\right) - \Delta_ {0}\right) \geq \Delta_ {0}, \tag {14}
$$
where the second inequality comes from Equation (12). Since  $\Delta_0 > 0$ , we have
$$
N _ {a} (t) = N _ {a} \left(t ^ {\prime}\right) + 1 \leq N _ {L} (t - 1) = N _ {L} (t). \tag {15}
$$
Also, since  $\frac{3}{2}\sqrt{\frac{\log(t)}{N_L(t - 1)}} >0$ , we have  $\frac{3}{2}\sqrt{\frac{\log(t)}{N_a(t')}} >\Delta_0$ , which implies
$$
N _ {a} (t) = N _ {a} \left(t ^ {\prime}\right) + 1 \leq 1 + \frac {3}{\Delta_ {0} {} ^ {2}} \log (t). \tag {16}
$$
Lemma 3. Assume event  $E$  holds and  $\delta \leq 1 / 2$ .
1) For any  $t \geq L$ , the cumulative attack cost to any fixed arm  $a \neq L$  can be bounded as:
$$
\sum_ {s \in \tau_ {a} (t)} \alpha_ {s} \leq N _ {a} (t) \left(\Delta_ {a} + \Delta_ {0} + 3 \beta \left(N _ {L} \left(h _ {a} (t)\right) + \beta \left(N _ {a} (t)\right)\right) + 1 \right. \tag {17}
$$
2) When  $t$  goes to infinity, we have
$$
\lim  _ {t \rightarrow \infty} \frac {\sum_ {s \in \tau_ {a} (t)} \alpha_ {s}}{\log t} \leq \frac {3}{\Delta_ {0} ^ {2}} \left(\Delta_ {a} + \Delta_ {0}\right) \tag {18}
$$
Proof. Fix any arm  $a \neq L$ . By the definitions of  $\gamma_t$  and  $\widetilde{\gamma}_t$ , it follows that: