SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \sum_ {s \in \tau_ {a} (t)} \alpha_ {s} \leq \sum_ {s \in \tau_ {a} (t - 1)} \alpha_ {s} + 1 (19) \\ \leq N _ {a} (t) \hat {\mu} _ {a} ^ {0} (t) - N _ {a} (t) \left[ \underline {{\mu}} _ {L} \left(h _ {a} (t - 1)\right) - \Delta_ {0} \right] + 1 (20) \\ \leq N _ {a} (t) \left[ \Delta_ {a} + 3 \beta \left(N _ {L} \left(h _ {a} (t)\right)\right) + \beta \left(N _ {a} (t)\right) \right] + 1, (21) \\ \end{array}
$$
where the last inequality is due to the decrease of  $\beta$ .
For the asymptotic result, combining Lemma 2 with Equation (17), we have
$$
\begin{array}{l} \lim  _ {t \rightarrow \infty} \frac {\sum_ {s \in \tau_ {a} (t)} \alpha_ {s}}{\log t} \leq \lim  _ {t \rightarrow \infty} \frac {N _ {a} (t)}{\log (t)} \left[ \Delta_ {a} + 3 \beta \left(N _ {L} \left(h _ {a} (t)\right)\right) + \beta \left(N _ {a} (t)\right)\right] + \frac {1}{\log (t)} (22) \\ \leq \lim  _ {t \rightarrow \infty} \left(\frac {1}{\log (t)} + \frac {3}{\Delta_ {0} {} ^ {2}}\right)\left[ \Delta_ {a} + 3 \beta \left(N _ {L} \left(h _ {a} (t)\right)\right) + \beta \left(N _ {a} (t)\right)\right] + \frac {1}{\log (t)}. (23) \\ \end{array}
$$
It is easy to check that  $\lim_{t\to \infty}\beta (t) = 0$ . Hence, to get the asymptotic cost bound in the lemma, we need to prove  $\lim_{t\to \infty}\beta (N_L(h_a(t))) = 0$ . We find  $\lim_{t\to \infty}h_a(t) = t$  is a sufficient condition for it; in other words,  $h_a(t)$  should be always updated when  $t$  goes to infinity. To obtain this, we consider two cases of  $\gamma_t$ .
1) If  $\underline{\mu}_L(t) - \Delta_0 \geq 0$ , we have
$$
\begin{array}{l} \lim  _ {t \rightarrow \infty} \gamma_ {t} = \lim  _ {t \rightarrow \infty} (N _ {a} (t) \hat {\mu} _ {a} ^ {0} (t) - \sum_ {s \in \tau_ {a} (t - 1)} \alpha_ {s} - N _ {a} (t) (\hat {\mu} _ {L} (t) - 2 \beta (N _ {L} (t)) - \Delta_ {0})) \\ = \lim  _ {t \rightarrow \infty} \left(N _ {a} (t) \hat {\mu} _ {a} ^ {0} (t) - N _ {a} (t ^ {\prime}) (\hat {\mu} _ {L} (h _ {a} (t ^ {\prime})) - 2 \beta (N _ {L} (h _ {a} (t ^ {\prime}))) - \Delta_ {0}\right) + r _ {t} ^ {0} - \alpha_ {t ^ {\prime}} \\ + N _ {a} \left(t ^ {\prime}\right) \left(\hat {\mu} _ {L} \left(h _ {a} \left(t ^ {\prime}\right)\right) - 2 \beta \left(N _ {L} \left(h _ {a} \left(t ^ {\prime}\right)\right)\right) - \Delta_ {0}\right) - N _ {a} (t) \left(\hat {\mu} _ {L} (t) - 2 \beta \left(N _ {L} (t)\right) - \Delta_ {0}\right)) \\ \leq \lim  _ {t \rightarrow \infty} \left(r _ {t} ^ {0} - \alpha_ {t ^ {\prime}} + N _ {a} (t ^ {\prime}) (\hat {\mu} _ {L} (h _ {a} (t ^ {\prime})) - 2 \beta (N _ {L} (h _ {a} (t ^ {\prime}))) - \Delta_ {0}\right) - N _ {a} (t) (\hat {\mu} _ {L} (t) - 2 \beta (N _ {L} (t)) - \Delta_ {0})) \\ \leq r _ {t} ^ {0} + \lim  _ {t \rightarrow \infty} \left(N _ {a} \left(t ^ {\prime}\right)\left(\hat {\mu} _ {L} \left(h _ {a} \left(t ^ {\prime}\right)\right) - \hat {\mu} _ {L} (t) - 2 \beta \left(N _ {L} \left(h _ {a} \left(t ^ {\prime}\right)\right)\right)\right) - \left(\hat {\mu} _ {L} (t) - \Delta_ {0}\right)\right) \\ \leq r _ {t} ^ {0}. \\ \end{array}
$$
The first inequality is due to the attack at round  $t'$ . The last inequality is due to the confidence radius based on  $\beta(N_L(h_a(t'))')$  and  $\beta$  is decreasing.
2) If  $\underline{\mu}_L(t) - \Delta_0 < 0$ , we have  $\hat{\mu}_L(t) < 2\beta(N_L(t)) + \Delta_0$  and  $\hat{\mu}_L(h_a(t')) \leq \hat{\mu}_L(t) + 2\beta(N_L(h_a(t')))$ .
$$
\begin{array}{l} \lim  _ {t \rightarrow \infty} \gamma_ {t} = r _ {t} ^ {0} + \lim  _ {t \rightarrow \infty} \left(N _ {a} (t ^ {\prime}) \hat {\mu} _ {a} ^ {0} (t ^ {\prime}) - \sum_ {s \in \tau_ {a} (t - 1)} \alpha_ {s}\right) \\ = r _ {t} ^ {0} + \lim  _ {t \rightarrow \infty} N _ {a} \left(t ^ {\prime}\right) \hat {\mu} _ {a} \left(t ^ {\prime}\right) \\ \leq r _ {t} ^ {0} + \lim _ {t \rightarrow \infty} N _ {a} (t ^ {\prime}) (\hat {\mu} _ {L} (h _ {a} (t ^ {\prime})) - 2 \beta (N _ {L} (h _ {a} (t ^ {\prime}))) - \Delta_ {0}) \\ \leq r _ {t} ^ {0} + \lim  _ {t \rightarrow \infty} N _ {a} (t ^ {\prime}) 2 \beta \left(N _ {L} (t)\right) \\ = r _ {t} ^ {0}. \\ \end{array}
$$
Since in both cases,  $\gamma_t \leq r_t^0$ , we have  $h_a(t) = t$  when  $t$  goes to infinity.
With Lemma 2 and Lemma 3, the proof is completed by summing the corresponding upper bounds over all non-target arms  $a < L$ ,
A.3 Proof of Theorem 2
Proof. As for Algorithm 1, we first need to prove the post-attack feedback of Algorithm 2 is always valid. It is equivalent to showing that the output  $\alpha$  of Algorithm 3 is a valid attack value on the input pre-attack feedback  $r^0$ , i.e.,  $\alpha \leq r^0$ . Similar to the proof of Lemma 1, we consider two cases: when  $\gamma_{\max} \leq r^0$ ,  $\alpha = \lceil \gamma_{\max} \rceil \leq r^0$ ; when  $\gamma_{\max} < r^0$ , we can use the same inductive proof of Lemma 1 to show that  $\alpha = \lceil \tilde{\gamma}_{\max} \rceil \leq r^0$ . Thus, the post-attack feedback  $r^0 - \alpha$  is always valid.
Fix some  $t \geq L$  such that  $a_{i,t} \notin \mathbf{a}^*$ . We denote  $t' = \max \{s < t : a_{i,t} \in \mathbf{a}_s\}$  as the previous round that  $a_{i,t}$  was chosen. With the conservative estimations in  $\gamma_t(a_{i,t}, a)$  and  $\tilde{\gamma}_t(a_{i,t}, a)$ , the output  $\alpha_{i,t}$  of Algorithm 3 could ensure
$$
\hat {\mu} _ {a _ {i, t}} (t) \leq \hat {\mu} _ {a} \left(h _ {a _ {i, t}, a} (t - 1)\right) - 2 \beta \left(N _ {a} \left(h _ {a _ {i, t}, a} (t - 1)\right)\right) - \Delta_ {0}, \tag {24}
$$
for any  $a \in \mathbf{a}^*$ . Since  $a_{i,t}$  is chosen by the PBM-UCB algorithm, there must exist  $a \in \mathbf{a}^*$  with its UCB value less than that of  $a_{i,t}$ :
$$
\hat {\mu} _ {a} (t - 1) + B _ {a} (t) \leq \hat {\mu} _ {a _ {i, t}} (t - 1) + B _ {a _ {i, t}} (t). \tag {25}
$$
Since  $t'$  and  $t$  are two consecutive rounds when item  $a_{i,t}$  was chosen, we have  $\hat{\mu}_{a_{i,t}}(t - 1) = \hat{\mu}_{a_{i,t}}(t')$ ,  $N_{a_{i,t}}(t - 1) = N_{a_{i,t}}(t')$ . Rearranging Eq. (25), we have
$$
B _ {a _ {i, t}} (t) - B _ {a} (t) \geq \hat {\mu} _ {a} (t - 1) - \hat {\mu} _ {a _ {i, t}} \left(t ^ {\prime}\right) \geq \Delta_ {0}, \tag {26}
$$
where the last inequality is due to Eq. (24). Since  $B_{a_{i,t}}(t) \geq \Delta_0$ , with the definition of  $B_{a_{i,t}}(t)$  in Eq. (5), we have
$$
N _ {a _ {i, t}} (t) \leq \frac {1 + \epsilon}{2 \kappa_ {K} ^ {2} \Delta_ {0} ^ {2}} \log t, \tag {27}
$$
for any  $a_{i,t} \notin \mathbf{a}^*$ . Thus, for the target item  $L$ ,
$$
N _ {L} (T) \geq T - \sum_ {l \notin \mathbf {a} ^ {*}} N _ {l} (T) \geq T - (L - K) \left(\frac {1 + \epsilon}{2 \kappa_ {K} ^ {2} \Delta_ {0} ^ {2}} \log T\right), \tag {28}
$$
which guarantees the chosen time of the target item.
For the cumulative attack cost analysis, we consider any arm  $a_{i,t} \notin \mathbf{a}^*$ . Since Eq. (24) holds for any  $a \in \mathbf{a}^*$ , we take  $a = L$  and get
$$
\begin{array}{l} \sum_ {s \in \tau_ {a _ {i, t}} (t)} \alpha_ {a _ {i, t}, s} \leq N _ {a _ {i, t}} (t) \left[ \Delta_ {a _ {i, t}} + \Delta_ {0} + 3 \beta \left(N _ {L} \left(h _ {a _ {i, t}, L} (t)\right)\right) + \beta \left(N _ {a _ {i, t}} (t)\right) \right] (29) \\ \leq N _ {a _ {i, t}} (t) \left[ \Delta_ {a _ {i, t}} + \Delta_ {0} + 4 \beta \left(N _ {a _ {i, t}} \left(h _ {a _ {i, t}, L} (t)\right)\right) \right], (30) \\ \end{array}
$$
where the second line is due to  $N_{a_{i,t}}(t) \leq N_L(t)$ ,  $h_{a_{i,t},L}(t) \leq N_{a_{i,t}}(t)$ , and  $\beta$  is decreasing. Based on this inequality, we can follow the same steps in the proof of Lemma 3 to derive the upper bound of the cumulative attack cost for Algorithm 2.
A.4 Proof of Theorem 3
Proof. Since Algorithm 4 also calls Algorithm 3, we can use the same proof in Appendix A.3 to show its post-attack feedback is always valid. Also, the output  $\alpha_{i,t}$  of Algorithm 3 would still ensure Eq. (24) for any  $a_{i,t} \notin \mathbf{a}^*, a \in \mathbf{a}^*$ . Based on this, we can consider the post-attack problem as a cascading bandit problem with known expected reward gaps, where the expected reward of any item  $l \notin \mathbf{a}^*$  is less than that of the target item  $L$  by at least  $\Delta_0$ . Since  $L$  is the worst item in  $\mathbf{a}^*$ , with the regret analysis of CascadeUCB in [5], we know the regret caused by any list containing  $l \notin \mathbf{a}^*$  is bounded by
$$
R _ {l} (T) \leq \frac {1 2}{\Delta_ {0}} \log T. \tag {31}
$$
We also know the lower bound of the one-round instantaneous regret caused by any list containing  $l$  is  $p^* \Delta_0$ , where  $p^* := \prod_{i=1}^{K-1} \mu_i$ . We can then write the lower bound of  $R_l(T)$  as
$$
R _ {l} (T) \geq p ^ {*} \Delta_ {0} \cdot \mathbb {E} [ N _ {l} (T) ]. \tag {32}
$$
Combining Eq. (31) and Eq. (32), we obtain the upper bound of the expected number of recommendations of item  $l$ :
$$
\mathbb {E} \left[ N _ {l} (T) \right] \leq \frac {1 2}{p ^ {*} \Delta_ {0} ^ {2}} \log T. \tag {33}
$$
Thus, for the target item  $L$ ,
$$
\mathbb {E} \left[ N _ {L} (T) \right] \geq T - \sum_ {l \notin \boldsymbol {a} ^ {*}} \mathbb {E} \left[ N _ {l} (T) \right] \geq T - (L - K) \left(\frac {1 2}{p ^ {*} \Delta_ {0} ^ {2}} \log T\right). \tag {34}
$$
For the remaining cost analysis, since the attack cost only depends on the observed items, we can still follow the proof of Theorem 2: we need to change the confidence radius  $B_{a}(t)$  in Eq. (5) to the typical form  $\frac{3}{2}\sqrt{\frac{\log t}{N_a(t - 1)}}$ , then derive the corresponding cost upper bound.
A.5 Proof of Theorem 4
Proof. Since Algorithm 5 only attacks when  $r_{i,t}^{0} = 1$ , it is easy to verify its post-attack feedback is valid. The key step for proving its successful attack is to show that  $p_{i,t}$  is an upper bound of  $\Delta_{a_{i,t}} / \mu_{a_{i,t}}$ : if it is true, by the probabilistic attack with probability  $p_{i,t}$ , Algorithm 5 makes a post-attack bandit problem where the expected reward of any item  $l \notin \pmb{a}^*$  is worse than that of target item  $L$ , which guarantees the number of target arm pulls is  $T - R(T)$ . Actually, we have
$$
\frac {\Delta_ {a _ {i , t}}}{\mu_ {a _ {i , t}}} = \frac {\mu_ {a _ {i , t}} - \mu_ {L}}{\mu_ {a _ {i , t}}} = 1 - \frac {\mu_ {L}}{\mu_ {a _ {i , t}}}, \tag {35}
$$
which increases as  $\mu_{a_{i,t}}$  increases. Replacing  $\mu_{a_{i,t}}$  with  $\hat{\mu}_{a_{i,t}}^{0}(t) + \beta (N_{a_{i,t}}(t))$ ,
$$
\frac {\Delta_ {a _ {i , t}}}{\mu_ {a _ {i , t}}} \leq \frac {\hat {\mu} _ {a _ {i , t}} ^ {0} (t) + \beta \left(N _ {a _ {i , t}} (t)\right) - \mu_ {L}}{\hat {\mu} _ {a _ {i , t}} ^ {0} (t) + \beta \left(N _ {a _ {i , t}} (t)\right)} \leq \frac {\left[ \hat {\mu} _ {a _ {i , t}} ^ {0} (t) + \beta \left(N _ {a _ {i , t}} (t)\right) - \hat {\mu} _ {L} ^ {0} (t) + \beta \left(N _ {L} (t)\right) \right] _ {+}}{\hat {\mu} _ {a _ {i , t}} ^ {0} (t) + \beta \left(N _ {a _ {i , t}} (t)\right)}, \tag {36}
$$
where  $\hat{\mu}_{a_{i,t}}^{0}(t) + \beta (N_{a_{i,t}}(t))\geq \mu_{a_{i,t}}$  and  $\hat{\mu}_L^0 (t) + \beta (N_L(t))\geq \mu_L$  under event  $E$ . Since the last term is the definition of  $p_{i,t}$ , we have proved it is a high-probability upper bound of  $\Delta_{a_i,t} / \mu_{a_i,t}$ . For the cost analysis, we have
$$
\begin{array}{l} C (T) \leq \sum_ {i, t} \left[ \hat {\mu} _ {a _ {i, t}} ^ {0} (t) + \beta \left(N _ {a _ {i, t}} (t)\right) - \hat {\mu} _ {L} ^ {0} (t) + \beta \left(N _ {L} (t)\right) \right] _ {+} \mathbb {I} \left\{a _ {i, t} \notin \boldsymbol {a} ^ {*} \right\} (37) \\ \leq \sum_ {i, t} \left[ \mu_ {a _ {i, t}} + 2 \beta \left(N _ {a _ {i, t}} (t)\right) - \mu_ {L} + 2 \beta \left(N _ {L} (t)\right) \right] _ {+} \mathbb {I} \left\{a _ {i, t} \notin \boldsymbol {a} ^ {*} \right\} (38) \\ \leq O \left(\sum_ {a <   L} \left(\Delta_ {a} + 4 \beta (1)\right) R (T)\right), (39) \\ \end{array}
$$
B.1 Synthetic Data
We first consider adversarial attacks on 2-armed stochastic bandits with binary feedback. The rewards of arms 1 and 2 follow Bernoulli distributions with means  $\mu_{1}$  and  $\mu_{2}$  ( $\mu_{1} > \mu_{2}$ ). We designate arm 2 as the target arm. We take  $T = 10,000$ ,  $\delta = 0.1$ ,  $\Delta_0 = 0.1$ . We set  $\mu_{1} = 0.85$  and sample the value of  $\mu_{2}$  from a uniform distribution  $U(0.05,0.15)$ . We compare Algorithm 1 with a modified algorithm from [9], which attacks every non-target arm whenever the calculated  $\alpha_{t} > 0$  and the pre-attack feedback is one. Note that this modified algorithm lacks a theoretical guarantee. We show the costs and target arm chosen times relative to a trivial baseline, which sets the post-attack feedback of all non-target arms to zero as long as the pre-attack feedback is one. Figure 3 demonstrates that our algorithm is more efficient than the modified one, incurring fewer costs for a greater number of target arm selections.
We also conduct experiments under the cascade model with synthetic data. We take  $L = 16$ ,  $K = 8$ ,  $T = 100,000$ , and  $\{\mu_i\}_{i=1}^L$  are sampled from a uniform distribution  $U(0,1)$ . We compare Algorithm 4 with the same baselines introduced in Section 5. Figure 4 shows that Trivial<sub>all</sub> algorithm suffers an extremely high cumulative cost, while it cannot misguide the agent to recommend the target arm very often. On the other hand, our algorithm and Trivial<sub>set</sub> algorithm can successfully attack CascadeUCB and perform similarly on the chosen ratio of the target arm. However, our algorithm pays about  $30\%$  less cost than Trivial<sub>set</sub>.
B.2 Real Data
As discussed in Section 5, we have shown results under the position-based model with real data. We now show the experimental results with real data (MovieLens 20M dataset) under the cascade model. We take  $L = 100$ ,  $K = 10$ ,  $T = 100$ , 000;  $\{\mu_i\}_{i=1}^L$  are extracted according to [2]. Similarly, we compare the chosen ratio and cumulative cost of our algorithm with Trivial all and Trivial set. Figure 5 shows that Trivial all cannot successfully attack CascadeUCB as the chosen ratio of the target arm is very low and it suffers a linear cost. Algorithm 4 and Trivial set have similar performance on the chosen ratio of the target arm. However, our algorithm dramatically decreases the cost by more than 50%, which indicates that it is more effective than Trivial set.
(a) Relative Cost
(b) Relative chosen times
Figure 3: Attacks against UCB on 2-armed bandits with binary feedback.
(a) Cost
Figure 4: Attacks against CascadeUCB with synthetic data.
(b) Chosen times
(a) Cost
Figure 5: Attacks against CascadeUCB with real data.