SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jinhang Zuo $^{1,2}$  Zhiyao Zhang $^{3}$  Zhiyong Wang $^{4}$  Shuai Li $^{5*}$  Mohammad Hajiesmaili $^{1}$  Adam Wierman $^{2}$
<sup>1</sup>University of Massachusetts Amherst <sup>2</sup>California Institute of Technology  
<sup>3</sup>Southeast University <sup>4</sup>The Chinese University of Hong Kong <sup>5</sup>Shanghai Jiao Tong University {jhzuo, Hajiesmaili}@cs.umass.edu muirheadzhang@gmail.com zywang21@cse.cuhk.edu.hk shuai18@sjtu.edu.cn adamw@caltech.edu
Online learning to rank (OLTR) is a sequential decision-making problem where a learning agent selects an ordered list of items and receives feedback through user clicks. Although potential attacks against OLTR algorithms may cause serious losses in real-world applications, there is limited knowledge about adversarial attacks on OLTR. This paper studies attack strategies against multiple variants of OLTR. Our first result provides an attack strategy against the UCB algorithm on classical stochastic bandits with binary feedback, which solves the key issues caused by bounded and discrete feedback that previous works cannot handle. Building on this result, we design attack algorithms against UCB-based OLTR algorithms in position-based and cascade models. Finally, we propose a general attack strategy against any algorithm under the general click model. Each attack algorithm manipulates the learning agent into choosing the target attack item  $T - o(T)$  times, incurring a cumulative cost of  $o(T)$ . Experiments on synthetic and real data further validate the effectiveness of our proposed attack algorithms.
Online learning to rank (OLTR) has been extensively studied [1, 2, 3, 4, 5] as a sequential decision-making problem where, in each round, a learning agent presents a list of items to users and receives implicit feedback from user interactions. One of the most common forms of feedback considered in literature is in the form of user clicks [1, 4, 5]. OLTR with such click feedback can lead to major improvements over traditional supervised learning to rank methods [6, 7, 8]. However, there is a security concern that user-generated click feedback might be generated by malicious users with the goal of manipulating the learning agent. Understanding the vulnerability of OLTR under adversarial attacks plays an essential role in developing effective defense mechanisms for trustworthy OLTR.
There has been a surge of interest in adversarial attacks on multi-armed bandits [9, 10, 11]. For example, [9] shows that, for stochastic bandits, it is possible to manipulate the bandit algorithm into pulling a target arm very often with sublinear cumulative cost. Though it generally follows the bandit formulation, it differs from stochastic bandits in the action space and feedback model. More specifically, OLTR chooses a list of  $K$  out of  $L$  arms, instead of just one arm, to play in each round; the realized rewards of the chosen arms are usually censored by a click model, e.g., position-based [4] or cascade model [5]. Thus, it is nontrivial to design efficient adversarial attacks on the censored feedback of the chosen arms for OLTR.
Moreover, previous works [9, 10] can only handle unbounded and continuous reward feedback, while the (binary) click feedback of OLTR is bounded and discrete. Such binary feedback brings new
Table 1: Summary of the settings and proposed attack strategies†
Setting	Attack against	NL(T)	lim C(T)/ log T
L-armed bandits	UCB [12]	T-O((L-1)(1/Δ02log T))	O(∑a<L Δa+Δ0/Δ02)
Position-based model	PBM-UCB [4]	T-O((L-K)(1+ε/K2Δ02log T))	O(∑a<L (1+ε)(Δa+Δ0)/K2Δ02)
Cascade model	CascadeUCB [5]	T-O((L-K)(1/p*Δ02log T))	O(∑a<L Δa+Δ0/Δ02)
General model	Arbitrary	T-O(log T)	O(∑a<L (Δa+4β(1))))
$\dagger \Delta_0$  : parameter of the attack algorithm;  $\Delta_{a}$  : mean gap between arm  $a$  and  $L$  ;  $\beta$  : a decreasing function in Section 3.2
challenges to the attack design against OLTR algorithms. Since the post-attack feedback must also be binary, this scenario introduces a new problem of deciding whether to attack when the output attack value from previous algorithms is between 0 and 1: a simple rounding up might be costly, while skipping the attack may lead to undesired non-target arm pulls. Furthermore, the attack value computed by previous attack algorithms can be larger than 1, which is higher than the maximum attack value in the click model. In other words, in the bounded and discrete model, it is impossible to always find a feasible attack value to ensure the required conditions for their theoretical guarantees.
Contributions. In this paper, we propose the first study of adversarial attacks on OLTR with click feedback, aiming to overcome the new issues raised by the OLTR click model as well as the binary feedback. Table 1 summarizes our proposed attack algorithms with their theoretical guarantees. Since the click feedback itself complicates the attack design, we first consider adversarial attacks on stochastic  $L$ -armed bandits with Bernoulli rewards where the feedback from the chosen arm is always binary. We propose an attack algorithm that can mislead the well-known UCB algorithm [12] to pull the target arm  $L$  for  $N_{L}(T)$  times in  $T$  rounds, with a cumulative cost  $C(T)$  in the order of  $\log T$  asymptotically. Based on this approach, we study the two most popular click models of OLTR, position-based model [4] and cascade model [5], and propose attack strategies against UCB-type OLTR algorithms. Their cumulative costs depend on  $\kappa_{K}$  and  $p^{*}$ , which are instance-dependent parameters of the position-based and cascade models, respectively. Lastly, we introduce the threat model for OLTR with a general click model and design an attack algorithm that can misguide arbitrary OLTR algorithms using click feedback. Our technical contributions are summarized as follows.
1. We propose the new idea of conservative estimation of the target arm for attacking UCB in stochastic bandits with binary feedback, which resolves the issues caused by the bounded and discrete requirements of the post-attack feedback. This approach is also the backbone of other attack strategies in more complicated OLTR settings.
2. The PBM-UCB algorithm uses position bias-corrected counters rather than simple click counters to compute UCB indices; we provide the theoretical analysis of our attack algorithm against PBM-UCB by carefully treating these unusual counters.
3. The partial feedback of OLTR with the cascade click model brings a new challenge to the attack design. We provide a new regret-based analysis of our attack algorithm against CascadeUCB without suffering from the partial feedback issue.
4. We devise a general attack strategy based on a new probabilistic attack design. It can successfully attack arbitrary OLTR algorithms without knowing the details of the algorithm.
We also conduct experiments on both synthetic and real-world data to evaluate our proposed attack strategies. Experimental results show that they can effectively attack the corresponding OLTR algorithms with less cost compared to other baselines. Due to space constraints, proofs and empirical results are included in the appendix.
Related Work. Online learning to rank with different feedback models has attracted much attention in recent years. Though there are other types of feedback models such as top-  $K$  feedback [13], click feedback has been widely used in literature. [4, 14, 15] consider the position-based model (PBM), where each position of the item list has an examination probability known or unknown by the learning agent. The cascade model in [5, 16, 17, 18, 19, 20] considers the setting where the user would check the recommended items sequentially and stop at the first clicked one; all items after the clicked item will not be examined. The dependent click model (DCM) is a generalization of the cascade model where the user may click on multiple items [21, 22]. There are also general click models [1, 2, 3]
that can cover some of the previous click models. In this paper, we mainly focus on attack design on PBM and cascade models; both of them adopt bounded and discrete click feedback thus require a new design other than previous works like [9] that can only handle unbounded and continuous feedback.
Adversarial attacks on different types of multi-armed bandit problems have been studied recently [9, 10, 23, 24, 25]. [9] proposes the first study of adversarial attacks on the classical stochastic bandit problem. It designs effective attack strategies against the  $\epsilon$ -Greedy and UCB algorithms. [10] extends it to a more general setting, where the algorithm of the learning agent can be unknown. [23] studies adversarial attacks on linear contextual bandits where the adversarial modifications can be added to either rewards or contexts. To the best of our knowledge, we are the first to study adversarial attacks on OLTR where, in addition to the challenges raised by click feedback, the combinatorial action space and the censored feedback of OLTR make it non-trivial to design efficient attack strategies.
In this section, we briefly discuss the three problem settings we consider in this paper.
Stochastic  $L$ -armed bandit. We consider an arm set  $[L] = \{1,2,\dots ,L\}$ , with  $\mu_{i}$  as the expected reward of arm  $i$ . Without loss of generality, we assume  $\mu_{1} \geq \mu_{2} \geq \dots \geq \mu_{L}$ . In round  $t$ , the player chooses an arm  $a_{t} \in [L]$  to play and receives a reward  $r_t^0$  as feedback. In the click feedback setting, the realized reward  $r_t^0 \in \{0,1\}$  of arm  $a_{t}$  is sampled from a Bernoulli distribution with expectation  $\mu_{a_t}$ . The player aims to find an optimal policy to maximize the long-term cumulative reward.
OLTR with position-based model [4]. This setting also considers the item (arm) set  $[L]$  where  $\mu_{i}$  represents the click probability of item  $i$ . However, in OLTR with the position-based model (PBM), in each round  $t$ , the player chooses an ordered list of  $K$  items,  $\pmb{a}_{t} = (a_{1,t},\dots ,a_{K,t})$ , with known examination probability  $\kappa_{k}$  for the  $k$ -th position in the list (assuming  $\kappa_{1}\geq \dots \geq \kappa_{K}$ ). The player then observes the click feedback of the chosen list from the user, denoted as  $\boldsymbol{r}_t^0 = (r_{1,t}^0,\dots ,r_{K,t}^0)\in$ $\{0,1\} ^K$ , where  $r_{i,t}^{0}$  is sampled from a Bernoulli distribution with expectation  $\kappa_{i}\mu_{a_{i,t}}$ . The reward obtained by the player is the sum of the clicks, i.e.,  $\sum_{k = 1}^{K}r_{k,t}^{0}$ . The goal of the player is to find an optimal policy that can maximize the long-term cumulative user clicks.
OLTR with cascade model [5]. Here, we consider the same item set  $[L]$  as that in the PBM model. For OLTR with cascade model, in each round  $t$ , the player chooses an ordered list of  $K$  items,  $\mathbf{a}_t = (a_{1,t},\dots ,a_{K,t})$ , for the user. The user then checks the list from  $a_{1,t}$  to  $a_{K,t}$ , with probability  $\mu_{a_{k,t}}$  to click the  $k$ -th item. She immediately stops at the first clicked item, and returns the click result back to the player. We denote the position of the first clicked item as  $\mathbf{C}_t$  ( $\mathbf{C}_t = \infty$  if no item was clicked). The click feedback of the player is  $\boldsymbol{r}_t^0 = (r_{1,t}^0,\dots ,r_{K,t}^0)\in \{0,1\}^K$ , where only  $r_{\mathbf{C}_t,t}^0 = 1$  and  $r_{k,t}^{0} = 0$  for all  $k\neq \mathbf{C}_t$ . The reward obtained by the player is again the sum of user clicks  $\sum_{k = 1}^{K}r_{k,t}^{0}$ , but in the cascade model, it is at most 1. The goal of the player is also to find an optimal policy that can maximize the long-term cumulative user clicks.
3 Attacks on Stochastic L-armed Bandits with Binary Feedback
As mentioned in the introduction, one main challenge of attacking OLTR algorithms comes from the binary click feedback: such binary feedback limits the possible actions of the attacker since they need to ensure the post-attack reward feedback is still valid (binary). This is a common issue for all OLTR with click feedback. Hence, in this section, we focus on adversarial attacks on the  $L$ -armed bandit problem with binary feedback. We propose an attack algorithm against the UCB algorithm, which is the backbone of the attack strategies for more complicated OLTR settings.
3.1 Threat Model
We first introduce the threat model for the  $L$ -armed bandit problem with binary feedback. In each round  $t$ , the player chooses an arm  $a_{t} \in [L]$  to play. The environment generates the pre-attack reward feedback  $r_{t}^{0} \in \{0,1\}$  based on a Bernoulli distribution with mean  $\mu_{a_t}$ . The attacker then observes  $a_{t}, r_{t}^{0}$ , and decides the post-attack feedback  $r_{t} \in \{0,1\}$ . The player only receives  $r_{t}$  as the feedback and uses that to decide the next arm to pull,  $a_{t+1}$ , for round  $t + 1$ . Without loss of generality, we
Algorithm 1 Attacks against the UCB algorithm on stochastic bandits with binary feedback
1: Initialization:  $h_a(0) = 1$  for all  $a \in [L]$   
2: for  $t = 1, 2, 3, \ldots$  do  
3: Observe  $a_t, r_t^0$   
4: if  $a_t \neq L$  then  
5: Calculate  $\gamma_t, \tilde{\gamma}_t$  according to Eqs. (2) and (3)  
6: if  $\gamma_t \leq r_t^0$  then  
7:  $\alpha_t = \lceil \gamma_t \rceil$ ,  $h_{a_t}(t) = t$   
8: else  
9:  $\alpha_t = \lceil \tilde{\gamma}_t \rceil$ ,  $h_{a_t}(t) = h_{a_t}(t - 1)$   
10: end if  
11: end if  
12: Return  $r_t = r_t^0 - \alpha_t$ ; update  $h_a(t) = h_a(t - 1)$  for all  $a \neq a_t$   
13: end for
assume arm  $L$  is a sub-optimal target arm. The attacker's goal is to misguide the player to pull the target arm  $L$  very often while using small attack costs. Let  $N_{i}(t)$  denote the number of pulls of arm  $i$  up to round  $t$ . We say the attack is successful after  $T$  rounds if  $N_{L}(T) = T - o(T)$  in expectation or with high probability, while the cumulative attack cost  $C(T) = \sum_{t=1}^{T} |r_{t} - r_{t}^{0}| = o(T)$ .
3.2 Attack Algorithm against UCB
For a better illustration of the attack strategy, we first define the following auxiliary notations. Let  $\tau_{i}(t) := \{s : s \leq t, a_{s} = i\}$  be the set of rounds up to  $t$  when arm  $i$  is played. We denote the pre-attack average reward of arm  $i$  up to round  $t$  by  $\hat{\mu}_{i}^{0}(t) := N_{i}(t)^{-1} \sum_{s \in \tau_{i}(t)} r_{s}^{0}$ . Lastly, let  $\hat{\mu}_{i}(t) := N_{i}(t)^{-1} \sum_{s \in \tau_{i}(t)} r_{s}$  be the post-attack average reward of arm  $i$  up to round  $t$ .
As in [9], we consider attacks against the  $(\alpha, \psi)$ -UCB algorithm from [12], where  $\alpha = 4.5$  and  $\psi : \lambda \mapsto \lambda^2 / 8$  since Bernoulli random variables are  $1/4$ -sub-Gaussian. The original attack algorithm in [9] calculates an attack value  $\alpha_t$  for round  $t$  such that
$$
\hat {\mu} _ {a _ {t}} (t) \leq \hat {\mu} _ {L} (t - 1) - 2 \beta \left(N _ {L} (t - 1)\right) - \Delta_ {0}, \tag {1}
$$
where  $\beta(N) := \sqrt{\frac{1}{2N} \log \frac{\pi^2 LN^2}{3\delta}}$ ;  $\Delta_0 > 0$  and  $\delta > 0$  are the parameters of the attack algorithm. The attacker then gives the post-attack reward  $r_t = r_t^0 - \alpha_t$  back to the player. The main idea of the attack algorithm in [9] is to compute the attack value so that the post-attack empirical estimates of the non-target arms are always less than those of the target arm.
However, this attack design only works when the reward space is unbounded and continuous, while in the threat model with binary feedback, the Bernoulli rewards are discrete and bounded. There are two issues raised by the discrete and bounded requirements. First, the calculated attack value  $\alpha_{t}$  is a real number, which may not make the post-attack reward feedback  $r_t$  to be valid (binary). Second, in order to ensure that Eq. (1) is true, the calculated  $\alpha_{t}$  might be larger than 1, which is higher than the maximum attack value in our threat model. In other words, it is impossible to let Eq. (1) hold for all rounds, while such a guarantee was essential for the theoretical analysis in [9]. To overcome these issues, we propose a new attack algorithm against UCB on  $L$ -armed bandits with binary feedback. It is described in Algorithm 1. The algorithm maintains timestamp  $h_a(t)$  for each arm  $a$ . In round  $t$ , if the arm pulled by the player,  $a_{t}$ , is not the target arm  $L$ , it first checks the condition  $\gamma_t \leq r_t^0$  (Line 6), with  $\gamma_{t}$  computed as
$$
\gamma_ {t} = \left[ N _ {a _ {t}} (t) \hat {\mu} _ {a _ {t}} ^ {0} (t) - \sum_ {s \in \tau_ {a _ {t}} (t - 1)} \alpha_ {s} - N _ {a _ {t}} (t) \left[ \underline {{\mu}} _ {L} (t) - \Delta_ {0} \right] _ {+} \right] _ {+}, \tag {2}
$$
where  $[z]_{+} = \max (0,z)$  and  $\underline{\mu}_L(t)\coloneqq \hat{\mu}_L(t) - 2\beta (N_L(t))$ . In fact, condition  $\gamma_t\leq r_t^0$  is equivalent to checking whether there exists a feasible  $\alpha_{t}$  to ensure Eq. (1) holds: if  $\gamma_t\leq r_t^0$ , with  $\alpha_{t}$  set to be  $\lceil \gamma_t\rceil$  (Line 7), Eq. (1) will hold in round  $t$ . Using a similar analysis as in [9], we can derive an upper bound of  $N_{a_t}(t)$  and prove the success of the attack up to round  $t$ . The algorithm also updates the timestamp  $h_{a_t}(t) = t$  for  $a_{t}$  (line 7). If  $\gamma_t > r_t^0$ , it indicates that there is no feasible  $\alpha_{t}$  that can
ensure Eq. (1). Instead, the algorithm sets  $\alpha_{t} = \lceil \tilde{\gamma}_{t}\rceil$  with  $\tilde{\gamma}_t$  computed as
$$
\tilde {\gamma} _ {t} = \left[ N _ {a _ {t}} (t) \hat {\mu} _ {a _ {t}} ^ {0} (t) - \sum_ {s \in \tau_ {a _ {t}} (t - 1)} \alpha_ {s} - N _ {a _ {t}} (t) \left[ \underline {{\mu}} _ {L} \left(h _ {a _ {t}} (t - 1)\right) - \Delta_ {0} \right] _ {+} \right] _ {+}, \tag {3}
$$
where  $h_{a_t}(t - 1)$  records the last round that Eq. (1) was satisfied. We can prove such an  $\alpha_t$  is always feasible  $(\alpha_t \leq r_t^0)$  and it can ensure that
$$
\hat {\mu} _ {a _ {t}} (t) \leq \hat {\mu} _ {L} \left(h _ {a _ {t}} (t - 1)\right) - 2 \beta \left(N _ {L} \left(h _ {a _ {t}} (t - 1)\right)\right) - \Delta_ {0}. \tag {4}
$$
This new inequality always holds for all rounds, which helps guarantee the success of the attack.
Remark. Compared with Eq. (1), Eq. (4) uses a more conservative lower bound of  $\mu_L, \underline{\mu}_L(h_{a_t}(t - 1))$ , instead of  $\underline{\mu}_L(t)$ , on the right-hand side of the inequality. We refer to this as conservative estimation for the target arm  $L$  with respect to arm  $a_t$ , where the estimated lower bound of  $L$  will only be updated when there exists feasible  $\alpha_t$  to ensure Eq. (1). Intuitively, it makes the attack algorithm more conservative and less likely to use a small attack value that may result in an invalid large attack value in later rounds. We use an inductive proof to show that there always exists  $\alpha_t$  such that Eq. (4) holds while keeping  $r_t$  valid (binary). Therefore, conservative estimation addresses the problems posed by binary feedback. This forms the foundation for the attack algorithms tailored for OLTR with click feedback.
3.3 Analysis
We first show that our attack algorithm always returns valid binary feedback to the player.
Lemma 1. The post-attack feedback of Algorithm 1 is always valid, i.e.,  $r_t \in \{0,1\}$  for any  $t$ .
The proof of Lemma 1, which uses an inductive analysis on  $\gamma_t, \tilde{\gamma}_t$ , can be found in the appendix.
Define  $\Delta_{a} \coloneqq \mu_{a} - \mu_{L}$ . We give the following theorem to show the successful attack of Algorithm 1.
Theorem 1. Suppose  $T \geq L$  and  $\delta \leq 1/2$ . With probability at least  $1 - \delta$ , Algorithm 1 misguides the UCB algorithm to choose the target arm  $L$  at least  $T - (L - 1)\left(1 + \frac{3}{\Delta_0^2}\log T\right)$  rounds, using a cumulative attack cost at most
$$
C (T) \leq \left(1 + \frac {3}{\Delta_ {0} ^ {2}} \log T\right) \sum_ {a <   L} \left(\Delta_ {a} + \Delta_ {0} + 4 \beta \left(1 + \frac {3}{\Delta_ {0} ^ {2}} \log h _ {a} (T)\right)\right).
$$
As  $T$  goes to infinity, we have
$$
\lim  _ {T \rightarrow \infty} \frac {C (T)}{\log T} \leq O \left(\sum_ {a <   K} \frac {\Delta_ {a} + \Delta_ {0}}{\Delta_ {0} ^ {2}}\right).
$$
Compared with Theorem 2 in [9], the  $\beta$  term in our cost bound depends on  $\log h_a(T)$  instead of  $\log T$ . Since  $\beta$  is a decreasing function and  $h_a(T) \leq T$ , our finite-time cost bound can be larger than that in [9]. However, our asymptotic analysis of the cost suggests that when  $T$  is large, such difference becomes negligible. Notice that the attack algorithm in [9] does not have any theoretical guarantee in the binary feedback setting, so this comparison is only meant to show the additional cost potentially caused by the conservative estimation.
4 Attacks on Online Learning to Rank
We now move to developing effective attack algorithms for more complicated OLTR settings. Since attacking OLTR also faces the challenges caused by binary click feedback, the attack algorithms in this section rely on our attack design for stochastic bandits with binary feedback in Section 3.
4.1 Position-Based Click Model
Threat Model. We introduce the threat model for online stochastic ranking with position-based click feedback. In each round  $t$ , the player chooses a list of  $K$  item,  $\pmb{a}_t = (a_{1,t},\dots ,a_{K,t})$  to recommend. The environment generates the pre-attack click feedback  $r_t^0 = (r_{1,t}^0,\dots ,r_{K,t}^0)\in \{0,1\} ^K$  where  $r_{i,t}^{0}$
Algorithm 2 Attack against the PBM-UCB algorithm
1: Initialization: Randomly select  $K - 1$  items with  $L$  to build  $\pmb{a}^{*}$  ..  $h_{l,a}(0) = 1\forall l\in [L]\forall a\in \pmb{a}^{*}$    
2: for  $t = 1,2,3,\ldots$  do   
3: Observe  $\pmb {a}_t,\pmb {r}_t^0$  , set  $\alpha_{t} = (\alpha_{1,t},\dots ,\alpha_{K,t}) = (0,\dots ,0)$    
4: for  $i\in [K]$  do   
5: if  $a_{i,t}\notin a^{*}$  then   
6:  $\alpha_{i,t} = \mathrm{CAL\_ALPHA}(a_{i,t},r_{i,t}^{0},\pmb{a}^{*})$    
7: else   
8:  $\alpha_{i,t} = 0$    
9: end if   
10: end for   
11: Return  $\pmb {r}_t = \pmb {r}_t^0 -\pmb {\alpha}_t$  ; update  $h_{l,a}(t) = h_{l,a}(t - 1)$  for all  $l\notin \pmb{a}_t,a\in \pmb{a}^*$    
12: end for
Algorithm 3 CAL_ALPHA
1: Input: item  $l$ , click feedback  $r^0$ , item set  $\pmb{a}^{*}$   
2: for  $a \in \pmb{a}^{*}$  do  
3: Calculate  $\gamma_{t}(l,a), \tilde{\gamma}_{t}(l,a)$  according to Eqs. (6) and (7)  
4: end for  
5:  $\gamma_{\max} = \max_{a \in \pmb{a}^{*}} \gamma_{t}(l,a)$ ,  $\tilde{\gamma}_{\max} = \max_{a \in \pmb{a}^{*}} \tilde{\gamma}_{t}(l,a)$   
6: if  $\gamma_{\max} \leq r^0$  then  
7:  $\alpha = [\gamma_{\max}], h_{l,a}(t) = t$  for all  $a \in \pmb{a}^{*}$   
8: else  
9:  $\alpha = [\tilde{\gamma}_{\max}], h_{l,a}(t) = h_{l,a}(t - 1)$  for all  $a \in \pmb{a}^{*}$   
10: end if  
11: Return  $\alpha$
is sampled from a Bernoulli distribution with mean  $\kappa_{i}\mu_{a_{i,t}}$ . The attacker then observes  $a_{t}$  and  $\boldsymbol{r}_t^0$  and decides the post-attack click feedback  $\boldsymbol{r}_t \in \{0,1\}^K$ . The player only receives  $\boldsymbol{r}_t$  as the feedback and uses it to decide the next list to recommend,  $\boldsymbol{a}_{t + 1}$ , for round  $t + 1$ . Without loss of generality, we assume item  $L$  is a sub-optimal target item. Similar to Section 3.1, we say the attack is successful after  $T$  rounds if the number of target item recommendations is  $N_L(T) = T - o(T)$  in expectation or with high probability, while the cumulative attack cost  $C(T) = \sum_{t = 1}^{T}||\boldsymbol{r}_t - \boldsymbol{r}_t^0||_1 = o(T)$ .
Attack against PBM-UCB. We consider the PBM-UCB algorithm in [4] as the online ranking algorithm of the player, which computes the UCB index of each item  $a$  as
$$
\bar {\mu} _ {a} (t) = \hat {\mu} _ {a} (t - 1) + B _ {a} (t) = \hat {\mu} _ {a} (t - 1) + \sqrt {\frac {N _ {a} (t - 1) (1 + \epsilon) \log t}{2 \tilde {N} _ {a} (t - 1) ^ {2}}}, \tag {5}
$$