SlowGuess/NeurIPS_2023_OCR huggingface dataset:
zywang21@cse.cuhk.edu.hk
xjzzjl@sjtu.edu.cn
worktongyu@gmail.com
In real-world online web systems, multiple users usually arrive sequentially into the system. For applications like click fraud and fake reviews, some users can maliciously perform corrupted (disrupted) behaviors to trick the system. Therefore, it is crucial to design efficient online learning algorithms to robustly learn from potentially corrupted user behaviors and accurately identify the corrupted users in an online manner. Existing works propose bandit algorithms robust to adversarial corruption. However, these algorithms are designed for a single user, and cannot leverage the implicit social relations among multiple users for more efficient learning. Moreover, none of them consider how to detect corrupted users online in the multiple-user scenario. In this paper, we present an important online learning problem named LOCUD to learn and utilize unknown user relations from disrupted behaviors to speed up learning, and identify the corrupted users in an online setting. To robustly learn and utilize the unknown relations among potentially corrupted users, we propose a novel bandit algorithm RCLUB-WCU. To detect the corrupted users, we devise a novel online detection algorithm OCCUD based on RCLUB-WCU's inferred user relations. We prove a regret upper bound for RCLUB-WCU, which asymptotically matches the lower bound with respect to  $T$  up to logarithmic factors, and matches the state-of-the-art results in degenerate cases. We also give a theoretical guarantee for the detection accuracy of OCCUD. With extensive experiments, our methods achieve superior performance over previous bandit algorithms and high corrupted user detection accuracy.
In real-world online recommender systems, data from many users arrive in a streaming fashion [4, 15, 2, 7, 35, 27, 26]. There may exist some corrupted (malicious) users, whose behaviors (e.g., click, rating) can be adversarially corrupted (disrupted) over time to fool the system [29, 30, 12, 10, 9]. These corrupted behaviors could disrupt the user preference estimations of the algorithm. As a result, the system would easily be misled and make sub-optimal recommendations [14, 23, 7, 41],
which would hurt the user experience. Therefore, it is essential to design efficient online learning algorithms to robustly learn from potentially disrupted behaviors and detect corrupted users in an online manner.
There exist some works on bandits with adversarial corruption [29, 9, 22, 5, 12, 16]. However, they have the following limitations. First, existing algorithms are initially designed for robust online preference learning of a single user. In real-world scenarios with multiple users, they cannot robustly infer and utilize the implicit user relations for more efficient learning. Second, none of them consider how to identify corrupted users online in the multiple-user scenario. Though there also exist some works on corrupted user detection [34, 6, 39, 28, 13], they all focus on detection with known user information in an offline setting, thus can not be applied to do online detection from bandit feedback.
To address these limitations, we propose a novel bandit problem "Learning and Online Corrupted Users Detection from bandit feedback" (LOCUD). To model and utilize the relations among users, we assume there is an unknown clustering structure over users, where users with similar preferences lie in the same cluster [8, 19, 21]. The agent can infer the clustering structure to leverage the information of similar users for better recommendations. Among these users, there exists a small fraction of corrupted users. They can occasionally perform corrupted behaviors to fool the agent [12, 29, 30, 9] while mimicking the behaviors of normal users most of the time to make themselves hard to discover. The agent not only needs to learn the unknown user preferences and relations robustly from potentially disrupted feedback, balance the exploration-exploitation trade-off to maximize the cumulative reward, but also needs to detect the corrupted users online from bandit feedback.
The LOCUD problem is very challenging. First, the corrupted behaviors would cause inaccurate user preference estimations, which could lead to erroneous user relation inference and sub-optimal recommendations. Second, it is nontrivial to detect corrupted users online since their behaviors are dynamic over time (sometimes regular while sometimes corrupted), whereas, in the offline setting, corrupted users' information can be fully represented by static embeddings and the existing approaches [18, 32] can typically do binary classifications offline, which are not adaptive over time.
We propose a novel learning framework composed of two algorithms to address these challenges.
RCLUB-WCU. To robustly estimate user preferences, learn the unknown relations from potentially corrupted behaviors, and perform high-quality recommendations, we propose a novel bandit algorithm "Robust CLUstering of Bandits With Corrupted Users" (RCLUB-WCU), which maintains a dynamic graph over users to represent the learned clustering structure, where users linked by edges are inferred to be in the same cluster. RCLUB-WCU adaptively deletes edges and recommends arms based on aggregated interactive information in clusters. We do the following to ensure robust clustering structure learning. (i) To relieve the estimation inaccuracy caused by disrupted behaviors, we use weighted ridge regressions for robust user preference estimations. Specifically, we use the inverse of the confidence radius to weigh each sample. If the confidence radius associated with user  $i_t$  and arm  $a_t$  is large at  $t$ , the learner is quite uncertain about the estimation of  $i_t$ 's preference on  $a_t$ , indicating the sample at  $t$  is likely to be corrupted. Therefore, we use the inverse of the confidence radius to assign minor importance to the possibly disrupted samples when doing estimations. (ii) We design a robust edge deletion rule to divide the clusters by considering the potential effect of corruptions, which, together with (i), can ensure that after some interactions, users in the same connected component of the graph are in the same underlying cluster with high probability.
OCCUD. To detect corrupted users online, based on the learned clustering structure of RCLUB-WCU, we devise a novel algorithm named "Online Cluster-based Corrupted User Detection" (OC-CUD). At each round, we compare each user's non-robustly estimated preference vector (by ridge regression) and the robust estimation (by weighted regression) of the user's inferred cluster. If the gap exceeds a carefully-designed threshold, we detect this user as corrupted. The intuitions are as follows. With misleading behaviors, the non-robust preference estimations of corrupted users would be far from ground truths. On the other hand, with the accurate clustering of RCLUB-WCU, the robust estimations of users' inferred clusters should be close to ground truths. Therefore, for corrupted users, their non-robust estimates should be far from the robust estimates of their inferred clusters.
We summarize our contributions as follows.
- We present a novel online learning problem LOCUD, where the agent needs to (i) robustly learn and leverage the unknown user relations to improve online recommendation qualities under the disruption of corrupted user behaviors; (ii) detect the corrupted users online from bandit feedback.
- We propose a novel online learning framework composed of two algorithms, RCLUB-WCU and OCCUD, to tackle the challenging LOCUD problem. RCLUB-WCU robustly learns and utilizes the unknown social relations among potentially corrupted users to efficiently minimize regret. Based on RCLUB-WCU's inferred user relations, OCCUD accurately detects corrupted users online.
- We prove a regret upper bound for RCLUB-WCU, which matches the lower bound asymptotically in  $T$  up to logarithmic factors and matches the state-of-the-art results in several degenerate cases. We also give a theoretical performance guarantee for the online detection algorithm OCCUD.
- Experiments on both synthetic and real-world data clearly show the advantages of our methods.
Our work is related to bandits with adversarial corruption and bandits leveraging user relations.
The work [29] first studies stochastic bandits with adversarial corruption, where the rewards are corrupted with the sum of corruption magnitudes in all rounds constrained by the corruption level  $C$ . They propose a robust elimination-based algorithm. The paper [9] proposes an improved algorithm with a tighter regret bound. The paper [22] first studies stochastic linear bandits with adversarial corruptions. To tackle the contextual linear bandit setting where the arm set changes over time, the work [5] proposes a variant of the OFUL [1] that achieves a sub-linear regret. A recent work [12] proposes the CW-OFUL algorithm that achieves a nearly optimal regret bound. All these works focus on designing robust bandit algorithms for a single user; none consider how to robustly learn and leverage the implicit relations among potentially corrupted users for more efficient learning. Moreover, none of them consider how to online detect corrupted users in the multiple-user case.
Some works study how to leverage user relations to accelerate the bandit learning process in the multiple-user case. The work [38] utilizes a known user adjacency graph to share context and payoffs among neighbors. To adaptively learn and utilize unknown user relations, the paper [8] proposes the clustering of bandits (CB) problem where there is an unknown user clustering structure to be learned by the agent. The work [20] uses collaborative effects on items to guide the clustering of users. The paper [19] studies the CB problem in the cascading bandit setting. The work [21] considers the setting where users in the same cluster share both the same preference and the same arrival rate. The paper [25] studies the federated CB problem, considering privacy and communication issues. All these works only consider utilizing the relations among normal users; none of them consider how to robustly learn the user relations from potentially disrupted behaviors, thus would easily be misled by corrupted users. Also, none of them consider how to detect corrupted users from bandit feedback.
To the best of our knowledge, this is the first work to study the problem to (i) learn the unknown user relations and preferences from potentially corrupted feedback, and leverage the learned relations to speed up learning; (ii) adaptively detect the corrupted users online from bandit feedback.
This section formulates the problem of "Learning and Online Corrupted Users Detection from bandit feedback" (LOCUD) (illustrated in Fig.1). We denote  $\| \pmb{x} \|_M = \sqrt{\pmb{x}^\top \pmb{M}\pmb{x}}$ ,  $[m] = \{1, \dots, m\}$ , number of elements in set  $\mathcal{A}$  as  $|\mathcal{A}|$ .
In LOCUD, there are  $u$  users, which we denote by set  $\mathcal{U} = \{1,2,\dots ,u\}$ . Some of them are corrupted users, denoted by set  $\tilde{\mathcal{U}}\subseteq \mathcal{U}$ . These corrupted users, on the one hand, try to mimic normal users to make themselves hard to detect; on the other hand, they can occasionally perform corrupted behaviors to fool the agent into making sub-optimal decisions. Each user  $i\in \mathcal{U}$ , no matter a normal one or corrupted one, is associated with a (possibly mimicked for corrupted users) preference feature vector  $\pmb {\theta}_i\in \mathbb{R}^d$  that is unknown and  $\| \pmb {\theta}_i\| _2\leq 1$ . There is an underlying clustering structure among all the users representing the similarity of their preferences, but it is unknown to the agent and needs to be learned via interactions. Specifically, the set of users  $\mathcal{U}$  can be partitioned into  $m(m\ll u)$  clusters,  $V_{1},V_{2},\ldots V_{m}$ , where  $\cup_{j\in [m]}V_j = \mathcal{U}$ , and  $V_{j}\cap V_{j^{\prime}} = \emptyset$ , for  $j\neq j^{\prime}$ . Users in the same cluster have the same preference feature vector, while users in different clusters have different preference vectors. We use  $\pmb{\theta}^{j}$  to denote the common preference vector shared by users in the  $j$ -th cluster  $V_{j}$ , and use  $j(i)$  to denote the index of cluster user  $i$  belongs to (i.e.,  $i\in V_{j(i)}$ ). For
any two users  $k, i \in \mathcal{U}$ , if  $k \in V_{j(i)}$ , then  $\pmb{\theta}_k = \pmb{\theta}^{j(i)} = \pmb{\theta}_i$ ; otherwise  $\pmb{\theta}_k \neq \pmb{\theta}_i$ . We assume the arm set  $\mathcal{A} \subseteq \mathbb{R}^d$  is finite. Each arm  $a \in \mathcal{A}$  is associated with a feature vector  $\pmb{x}_a \in \mathbb{R}^d$  with  $\| \pmb{x}_a \|_2 \leq 1$ .
The learning process of the agent is as follows. At each round  $t \in [T]$ , a user  $i_t \in \mathcal{U}$  comes to be served, and the learning agent receives a set of arms  $\mathcal{A}_t \subseteq \mathcal{A}$  to choose from. The agent infers the cluster  $V_t$  that user  $i_t$  belongs to based on the interaction history, and recommends an arm  $a_t \in \mathcal{A}_t$  according to the aggregated information gathered in the cluster  $V_t$ . After receiving the recommended arm  $a_t$ , a normal user  $i_t$  will give a random reward with expectation  $\pmb{x}_{a_t}^\top \pmb{\theta}_{i_t}$  to the agent.
To model the behaviors of corrupted users, following [29, 9, 5, 12], we assume that they can occasionally corrupt the rewards to mislead the agent into recommending sub-optimal arms. Specifically, at each round  $t$ , if the current served user is a corrupted user (i.e.,  $i_t \in \tilde{\mathcal{U}}$ ), the user can corrupt the reward by  $c_t$ . In summary, we model the reward received by the agent at round  $t$  as
$t_3$
$t_4$
Figure 1: Illustration of LOCUD. The unknown user relations are represented by dotted circles, e.g., user 3, 7 have similar preferences and thus can be in the same user segment (i.e., cluster). Users 6 and 8 are corrupted users with dynamic behaviors over time (e.g., for user 8, the behaviors are normal at  $t_1$  and  $t_3$  (blue), but are adversarially corrupted at  $t_2$  and  $t_4$  (red)[29, 12]), making them hard to be detected online. The agent needs to learn user relations to utilize information among similar users to speed up learning, and detect corrupted users 6, 8 online from bandit feedback.
$$
r _ {t} = \boldsymbol {x} _ {a _ {t}} ^ {\top} \boldsymbol {\theta} _ {i _ {t}} + \eta_ {t} + c _ {t},
$$
where  $c_{t} = 0$  if  $i_{t}$  is a normal user, (i.e.,  $i_{t} \notin \mathcal{U}$ ), and  $\eta_{t}$  is 1-sub-Gaussian random noise.
As the number of corrupted users is usually small, and they only corrupt the rewards occasionally with small magnitudes to make themselves hard to detect, we assume the sum of corruption magnitudes in all rounds is upper bounded by the corruption level  $C$ , i.e.,  $\sum_{t=1}^{T} |c_t| \leq C$  [29, 9, 5, 12].
We assume the clusters, users, and items satisfy the following assumptions. Note that all these assumptions basically follow the settings from classical works on clustering of bandits [8, 19, 25, 36].
Assumption 1 (Gap between different clusters). The gap between any two preference vectors for different clusters is at least an unknown positive constant  $\gamma$
Assumption 2 (Uniform arrival of users). At each round  $t$ , a user  $i_t$  comes uniformly at random from  $\mathcal{U}$  with probability  $1 / u$ , independent of the past rounds.
Assumption 3 (Item regularity). At each round  $t$ , the feature vector  $\mathbf{x}_a$  of each arm  $a \in \mathcal{A}_t$  is drawn independently from a fixed unknown distribution  $\rho$  over  $\{\pmb{x} \in \mathbb{R}^d : \| \pmb{x} \|_2 \leq 1\}$ , where  $\mathbb{E}_{\pmb{x} \sim \rho}[\pmb{x}\pmb{x}^\top]$ 's minimal eigenvalue  $\lambda_x > 0$ . At  $\forall t$ , for any fixed unit vector  $z \in \mathbb{R}^d$ ,  $(\pmb{\theta}^\top z)^2$  has sub-Gaussian tail with variance no greater than  $\sigma^2$ .
Let  $a_{t}^{*}\in \arg \max_{a\in A_{t}}\pmb{x}_{a}^{\top}\pmb{\theta}_{i_{t}}$  denote an optimal arm with the highest expected reward at round  $t$ . One objective of the learning agent is to minimize the expected cumulative regret
$$
R (T) = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left(\boldsymbol {x} _ {a _ {t} ^ {*}} ^ {\top} \boldsymbol {\theta} _ {i _ {t}} - \boldsymbol {x} _ {a _ {t}} ^ {\top} \boldsymbol {\theta} _ {i _ {t}}\right) \right]. \tag {1}
$$
Another objective is to detect corrupted users online accurately. Specifically, at round  $t$ , the agent will give a set of users  $\tilde{\mathcal{U}}_t$  as the detected corrupted users, and we want  $\tilde{\mathcal{U}}_t$  to be as close to the ground-truth set of corrupted users  $\tilde{\mathcal{U}}$  as possible.
4 Algorithms
This section introduces our algorithms RCLUB-WCU (Algo.1) and OCCUD (Algo.2). RCLUB-WCU robustly learns the unknown user clustering structure and preferences from corrupted feed-
back, and leverages the cluster-based information to accelerate learning. Based on the clustering structure learned by RCLUB-WCU, OCCUD can accurately detect corrupted users online.
Algorithm 1 RCLUB-WCU
1: Input: Regularization parameter  $\lambda$ , confidence radius parameter  $\beta$ , threshold parameter  $\alpha$ , edge deletion parameter  $\alpha_{1}$ ,  $f(T) = \sqrt{(1 + \ln(1 + T)) / (1 + T)}$ .
2: Initialization:  $M_{i,0} = \mathbf{0}_{d\times d},\pmb {b}_{i,0} = \mathbf{0}_{d\times 1}$ $\tilde{M}_{i,0} = \mathbf{0}_{d\times d},\tilde{\pmb{b}}_{i,0} = \mathbf{0}_{d\times 1},T_{i,0} = 0,\forall i\in \mathcal{U};$  A complete graph  $G_{0} = (\mathcal{U},E_{0})$  over  $\mathcal{U}$
3: for all  $t = 1,2,\ldots ,T$  do
4:Receive the index of the current served user  $i_t\in \mathcal{U}$  , get the feasible arm set at this round  $\mathcal{A}_t$
5: Determine the connected components  $V_{t}$  in the current maintained graph  $G_{t - 1} = (\mathcal{U},E_{t - 1})$  such that  $i_t\in V_t$ .
6: Calculate the robustly estimated statistics for the cluster  $V_{t}$ :  $M_{V_{t},t - 1} = \lambda I + \sum_{i\in V_{t}}M_{i,t - 1},b_{V_{t},t - 1} = \sum_{i\in V_{t}}b_{i,t - 1},\hat{\pmb{\theta}}_{V_{t},t - 1} = M_{V_{t},t - 1}^{-1}\pmb{b}_{V_{t},t - 1}$
7: Select an arm  $a_{t}$  with largest UCB index in Eq.(3) and receive the corresponding reward  $r_{t}$ ;
8: Update the statistics for robust estimation of user  $i_t$ :
$$
\boldsymbol {M} _ {i _ {t}, t} = \boldsymbol {M} _ {i _ {t}, t - 1} + w _ {i _ {t}, t - 1} \boldsymbol {x} _ {a _ {t}} \boldsymbol {x} _ {a _ {t}} ^ {\top}, \boldsymbol {b} _ {i _ {t}, t} = \boldsymbol {b} _ {i _ {t}, t - 1} + w _ {i _ {t}, t - 1} r _ {t} \boldsymbol {x} _ {a _ {t}}, T _ {i _ {t}, t} = T _ {i _ {t}, t - 1} + 1,
$$
$$
\boldsymbol {M} _ {i _ {t}, t} ^ {\prime} = \lambda \boldsymbol {I} + \boldsymbol {M} _ {i _ {t}, t}, \hat {\boldsymbol {\theta}} _ {i _ {t}, t} = \boldsymbol {M} _ {i _ {t}, t} ^ {\prime - 1} \boldsymbol {b} _ {i _ {t}, t}, w _ {i _ {t}, t} = \min  \left\{1, \alpha / \| \boldsymbol {x} _ {a _ {t}} \| _ {\boldsymbol {M} _ {i _ {t}, t} ^ {\prime - 1}} \right\};
$$
9: Keep robust estimation statistics of other users unchanged:
$$
\boldsymbol {M} _ {\ell , t} = \boldsymbol {M} _ {\ell , t - 1}, \boldsymbol {b} _ {\ell , t} = \boldsymbol {b} _ {\ell , t - 1}, T _ {\ell , t} = T _ {\ell , t - 1}, \boldsymbol {\theta} _ {\ell , t} = \boldsymbol {\theta} _ {\ell , t - 1}, \text {f o r a l l} \ell \in \mathcal {U}, \ell \neq i;
$$
10: Delete the edge  $(i_t, \ell) \in E_{t-1}$ , if
$$
\left\| \hat {\boldsymbol {\theta}} _ {i _ {t}, t} - \hat {\boldsymbol {\theta}} _ {\ell , t} \right\| _ {2} \geq \alpha_ {1} \left(f \left(T _ {i _ {t}, t}\right) + f \left(T _ {\ell , t}\right) + \alpha C\right),
$$
11: Use the OCCUD Algorithm (Algo.2) to detect the corrupted users.
12: end for
4.1 RCLUB-WCU
The corrupted behaviors may cause inaccurate preference estimations, leading to erroneous relation inference and sub-optimal decisions. In this case, how to learn and utilize unknown user relations to accelerate learning becomes non-trivial. Motivated by this, we design RCLUB-WCU as follows.
Assign the inferred cluster  $V_{t}$  for user  $i_{t}$ . RCLUB-WCU maintains a dynamic undirected graph  $G_{t} = (\mathcal{U}, E_{t})$  over users, which is initialized to be a complete graph (Algo.1 Line 2). Users with similar learned preferences will be connected with edges in  $E_{t}$ . The connected components in the graph represent the inferred clusters by the algorithm. At round  $t$ , user  $i_{t}$  comes to be served with a feasible arm set  $\mathcal{A}_{t}$  for the agent to choose from (Line 4). In Line 5, RCLUB-WCU detects the connected component  $V_{t}$  in the graph containing user  $i_{t}$  to be the current inferred cluster for  $i_{t}$ .
Robust preference estimation of cluster  $V_{t}$ . After determining the cluster  $V_{t}$ , RCLUB-WCU estimates the common preferences for users in  $V_{t}$  using the historical feedback of all users in  $V_{t}$  and recommends an arm accordingly. The corrupted behaviors could cause inaccurate preference estimates, which can easily mislead the agent. To address this, inspired by [40, 12], we use weighted ridge regression to make corruption-robust estimations. Specifically, RCLUB-WCU robustly estimates the common preference vector of cluster  $V_{t}$  by solving the following weighted ridge regression
$$
\hat {\boldsymbol {\theta}} _ {V _ {t}, t - 1} = \underset {\boldsymbol {\theta} \in \mathbb {R} ^ {d}} {\arg \min } \sum_ {\substack {s \in [ t - 1 ] \\ i _ {s} \in V _ {t}}} w _ {i _ {s}, s} \left(r _ {s} - \boldsymbol {x} _ {a _ {s}} ^ {\top} \boldsymbol {\theta}\right) ^ {2} + \lambda \| \boldsymbol {\theta} \| _ {2} ^ {2}, \tag{2}
$$
where  $\lambda > 0$  is a regularization coefficient. Its closed-form solution is  $\hat{\theta}_{V_t,t-1} = M_{V_t,t-1}^{-1}\pmb{b}_{V_t,t-1}$ , where  $M_{V_t,t-1} = \lambda I + \sum_{\substack{s \in [t-1] \\ i_s \in V_t}} w_{i_s,s} \pmb{x}_{a_s} \pmb{x}_{a_s}^\top$ ,  $b_{V_t,t-1} = \sum_{\substack{s \in [t-1] \\ i_s \in V_t}} w_{i_s,s} r_{a_s} \pmb{x}_{a_s}$ .
We set the weight of sample for user  $i_s$  in  $V_t$  at round  $s$  as  $w_{i_s,s} = \min \{1, \alpha / \| \pmb{x}_{a_s} \|_{M_{i_s,s}^{\prime - 1}}\}$ , where  $\alpha$  is a coefficient to be determined later. The intuitions of designing these weights are as follows. The term  $\| \pmb{x}_{a_s} \|_{M_{i_s,s}^{\prime - 1}}$  is the confidence radius of arm  $a_s$  for user  $i_s$  at  $s$ , reflecting how confident the algorithm is about the estimation of  $i_s$ 's preference on  $a_s$  at  $s$ . If  $\| \pmb{x}_{a_s} \|_{M_{i_s,s}^{\prime - 1}}$  is large, it means the agent is uncertain of user  $i_s$ 's preference on  $a_s$ , indicating this sample is probably corrupted.
Therefore, we use the inverse of confidence radius to assign a small weight to this round's sample if it is potentially corrupted. In this way, uncertain information for users in cluster  $V_{t}$  is assigned with less importance when estimating the  $V_{t}$ 's preference vector, which could help relieve the estimation inaccuracy caused by corruption. For technical details, please refer to Section 5.1 and Appendix.
Recommend  $a_{t}$  with estimated preference of cluster  $V_{t}$ . Based on the corruption-robust preference estimation  $\hat{\theta}_{V_{t},t - 1}$  of cluster  $V_{t}$ , in Line 7, the agent recommends an arm using the upper confidence bound (UCB) strategy to balance exploration and exploitation
$$
a _ {t} = \operatorname {a r g m a x} _ {a \in \mathcal {A} _ {t}} \boldsymbol {x} _ {a} ^ {\top} \hat {\boldsymbol {\theta}} _ {V _ {t}, t - 1} + \beta \| \boldsymbol {x} _ {a} \| _ {\boldsymbol {M} _ {V _ {t}, t - 1} ^ {- 1}} \triangleq \hat {R} _ {a, t} + C _ {a, t}, \tag {3}
$$
where  $\beta = \sqrt{\lambda} +\sqrt{2\log(\frac{1}{\delta}) + d\log(1 + \frac{T}{\lambda d})} +\alpha C$  is the confidence radius parameter,  $\hat{R}_{a,t}$  denotes the estimated reward of arm  $a$  at  $t$ ,  $C_{a,t}$  denotes the confidence radius of arm  $a$  at  $t$ . The design of  $C_{a,t}$  theoretically relies on Lemma 2 that will be given in Section 5.