SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhiyong Wang
The Chinese University of Hong Kong zywang21@cse.cuhk.edu.hk
Jize Xie
Shanghai Jiao Tong University xjzzjl@sjtu.edu.cn
Xutong Liu
The Chinese University of Hong Kong liuxt@cse.cuhk.edu.hk
Shanghai Jiao Tong University shuaili8@sjtu.edu.cn
The Chinese University of Hong Kong cslui@cse.cuhk.edu.hk
The contextual linear bandit is an important online learning problem where given arm features, a learning agent selects an arm at each round to maximize the cumulative rewards in the long run. A line of works, called the clustering of bandits (CB), utilize the collaborative effect over user preferences and have shown significant improvements over classic linear bandit algorithms. However, existing CB algorithms require well-specified linear user models and can fail when this critical assumption does not hold. Whether robust CB algorithms can be designed for more practical scenarios with misspecified user models remains an open problem. In this paper, we are the first to present the important problem of clustering of bandits with misspecified user models (CBMUM), where the expected rewards in user models can be perturbed away from perfect linear models. We devise two robust CB algorithms, RCLUMB and RSCLUMB (representing the learned clustering structure with dynamic graph and sets, respectively), that can accommodate the inaccurate user preference estimations and erroneous clustering caused by model misspecifications. We prove regret upper bounds of  $O(\epsilon_{*}T\sqrt{md\log T} + d\sqrt{mT}\log T)$  for our algorithms under milder assumptions than previous CB works (notably, we move past a restrictive technical assumption on the distribution of the arms), which match the lower bound asymptotically in  $T$  up to logarithmic factors, and also match the state-of-the-art results in several degenerate cases. The techniques in proving the regret caused by misclustering users are quite general and may be of independent interest. Experiments on both synthetic and real-world data show our outperformance over previous algorithms.
Stochastic multi-armed bandit (MAB) [2, 4, 22] is an online sequential decision-making problem, where the learning agent selects an action and receives a corresponding reward at each round, so as to maximize the cumulative reward in the long run. MAB algorithms have been widely applied in
recommendation systems and computer networks to handle the exploration and exploitation trade-off [20, 30, 38, 5].
To deal with large-scale applications, the contextual linear bandits [24, 9, 1, 29, 21] have been studied, where the expected reward of each arm is assumed to be perfectly linear in their features. Leveraging the contextual side information about the user and arms, linear bandits can provide more personalized recommendations [16]. Classical linear bandit approaches, however, ignore the often useful tool of collaborative filtering. To utilize the relationships among users, the problem of clustering of bandits (CB) has been proposed [12]. Specifically, CB algorithms adaptively partition users into clusters and utilize the collaborative effect of users to enhance learning performance.
Although existing CB algorithms have shown great success in improving recommendation qualities, there exist two major limitations. First, all previous works on CB [12, 25, 27, 39] assume that for each user, the expected rewards follow a perfectly linear model with respect to the user preference vector and arms' feature vectors. In many real-world scenarios, due to feature noises or uncertainty [15], the reward may not necessarily conform to a perfectly linear function, or even deviates a lot from linearity [14]. Second, previous CB works assume that for users within the same cluster, their preferences are exactly the same. Due to the heterogeneity in users' personalities and interests, similar users may not have identical preferences, invalidating this strong assumption.
To address these issues, we propose a novel problem of clustering of bandits with misspecified user models (CBMUM). In CBMUM, the expected reward model of each user does not follow a perfectly linear function but with possible additive deviations. We assume users in the same underlying cluster share a common preference vector, meaning they have the same linear part in reward models, but the deviation parts are allowed to be different, better reflecting the varieties of user personalities.
The relaxation of perfect linearity and the reward homogeneity within the same cluster bring many challenges to the CBMUM problem. In CBMUM, we not only need to handle the uncertainty from the unknown user preference vectors, but also have to tackle the additional uncertainty from model misspecifications. Due to such uncertainties, it becomes highly challenging to design a robust algorithm that can cluster the users appropriately and utilize the clustered information judiciously. On the one hand, the algorithm needs to be more tolerant in the face of misspecifications so that more similar users can be clustered together to utilize the collaborative effect. On the other hand, it has to be more selective to rule out the possibility of misclustering users with large preference gaps.
This paper makes the following four contributions.
New Model Formulation. We are the first to formulate the clustering of bandits with misspecified user models (CBMUM) problem, which is more practical by removing the perfect linearity assumption in previous CB works.
Novel Algorithm Designs. We design two novel algorithms, RCLUMB and RSCLUMB, which robustly learn the clustering structure and utilize this collaborative information for faster user preference elicitation. Specifically, RCLUMB keeps updating a dynamic graph over all users, where users connected directly by edges are supposed to be in the same cluster. RCLUMB adaptively removes edges and recommends items based on historical interactions. RSCLUMB represents the clustering structure with sets, which are dynamically merged and split during the learning process. Due to the page limit, we only illustrate the RCLUMB algorithm in the main paper. We leave the exposition, illustration, and regret analysis of the RSCLUMB algorithm in Appendix K.
To overcome the challenges brought by model misspecifications, we do the following key steps in the RCLUMB algorithm. (i) To ensure that with high probability, similar users will not be partitioned apart, we design a more tolerant edge deletion rule by taking model misspecifications into consideration. (ii) Due to inaccurate user preference estimations caused by model misspecifications, trivially following previous CB works [12, 25, 28] to directly use connected components in the maintained graph as clusters would miscluster users with big preference gaps, causing a large regret. To be discriminative in cluster assignments, we filter users directly linked with the current user in the graph to form the cluster used in this round. With these careful designs of (i) and (ii), we can guarantee that with high probability, information of all similar users can be leveraged, and only users with close enough preferences might be misclustered, which will only mildly impair the
learning accuracy. Additionally: (iii) we design an enlarged confidence radius to incorporate both the exploration bonus and the additional uncertainty from misspecifications when recommending arms. The design of RSCLUMB follows similar ideas, which we leave in the Appendix K due to page limit.
Theoretical Analysis with Milder Assumptions. We prove regret upper bounds for our algorithms of  $O(\epsilon_{*}T\sqrt{md\log T} + d\sqrt{mT}\log T)$  in CBMUM under much milder and practical assumptions (in arm generation distribution) than previous CB works, which match the state-of-the-art results in degenerate cases. Our proof is quite different from the typical proof flow of previous CB works (details in Appendix C). One key challenge is to bound the regret caused by misclustering users with close but not the same preference vectors and use the inaccurate cluster-based information to recommend arms. To handle the challenge, we prove a key lemma (Lemma 5.7) to bound this part of regret. We defer its details in Section 5 and Appendix G. The techniques and results for bounding this part are quite general and may be of independent interest. We also give a regret lower bound of  $\Omega (\epsilon_{*}T\sqrt{d})$  for CBMUM, showing that our upper bounds are asymptotically tight with respect to  $T$  up to logarithmic factors. We leave proving a tighter lower bound for CBMUM as an open problem.
Good Experimental Performance. Extensive experiments on both synthetic and real-world data show the advantages of our proposed algorithms over the existing algorithms.
Our work is closely related to two lines of research: online clustering of bandits (CB) and misspecified linear bandits (MLB). More discussions on related works can be found in Appendix A.
The paper [12] first formulates the CB problem and proposes a graph-based algorithm. The work [26] further considers leveraging the collaborative effects on items to guide the clustering of users. The work [25] considers the CB problem in the cascading bandits setting with random prefix feedback. The paper [27] also considers users with different arrival frequencies. A recent work [28] proposes the setting of clustering of federated bandits, considering both privacy protection and communication requirements. However, all these works assume that the reward model for each user follows a perfectly linear model, which is unrealistic in many real-world applications. To the best of our knowledge, this paper is the first work to consider user model misspecifications in the CB problem.
The work [14] first proposes the misspecified linear bandits (MLB) problem, shows the vulnerability of linear bandit algorithms under deviations, and designs an algorithm RLB that is only robust to non-sparse deviations. The work [23] proposes two algorithms to handle general deviations, which are modifications of the phased elimination algorithm [22] and LinUCB [1]. Some recent works [31, 11] use model selection methods to deal with unknown exact maximum model misspecification level. Note that the work [11] has an additional assumption on the access to an online regression oracle, and the paper [31] still needs to know an upper bound of the unknown exact maximum model deviation level. None of them consider the CB setting with multiple users, thus differing from ours.
We are the first to initialize the study of the important CBMUM problem, and propose a general framework for dealing with model misspecifications in CB problems. Our study is based on fundamental models on CB [12, 27] and MLB [23], the algorithm design ideas and theoretical analysis are pretty general. We leave incorporating the model selection methods [31, 11] into our framework to address the unknown exact maximum model misspecification level as an interesting future work.
This section formulates the problem of "clustering of bandits with misspecified user models" (CB-MUM). We use boldface lowercase and boldface CAPITALIZED letters for vectors and matrices. We use  $|\mathcal{A}|$  to denote the number of elements in  $\mathcal{A}$ ,  $[m]$  to denote  $\{1, \ldots, m\}$ , and  $\|x\|_M = \sqrt{x^\top M x}$  to denote the matrix norm of vector  $x$  regarding the positive semi-definite (PSD) matrix  $M$ .
In CBMUM, there are  $u$  users denoted by  $\mathcal{U} = \{1,2,\dots,u\}$ . Each user  $i\in \mathcal{U}$  is associated with an unknown preference vector  $\pmb{\theta}_i\in \mathbb{R}^d$ , with  $\| \pmb{\theta}_i\| _2\leq 1$ . We assume there is an unknown underlying clustering structure over users representing the similarity of their behaviors. Specifically,  $\mathcal{U}$  can be partitioned into a small number  $m$  (i.e.,  $m\ll u$ ) clusters,  $V_{1},V_{2},\ldots V_{m}$ , where  $\cup_{j\in [m]}V_j = \mathcal{U}$ , and  $V_{j}\cap V_{j^{\prime}} = \emptyset$ , for  $j\neq j^{\prime}$ . We call these clusters ground-truth clusters and use  $\mathcal{V} = \{V_1,V_2,\dots ,V_m\}$  to denote the set of these clusters. Users in the same ground-truth cluster share the same preference vector, while users from different ground-truth clusters have different preference vectors. Let  $\pmb{\theta}^{j}$
denote the common preference vector for  $V_{j}$  and  $j(i) \in [m]$  denote the index of the ground-truth cluster that user  $i$  belongs to. For any  $\ell \in \mathcal{U}$ , if  $\ell \in V_{j(i)}$ , then  $\pmb{\theta}_{\ell} = \pmb{\theta}_{i} = \pmb{\theta}^{j(i)}$ .
At each round  $t \in [T]$ , a user  $i_t \in \mathcal{U}$  comes to be served. The learning agent receives a finite arm set  $\mathcal{A}_t \subseteq \mathcal{A}$  to choose from (with  $|\mathcal{A}_t| \leq C, \forall t$ ), where each arm  $a \in \mathcal{A}$  is associated with a feature vector  $\boldsymbol{x}_a \in \mathbb{R}^d$ , and  $\| \boldsymbol{x}_a \|_2 \leq 1$ . The agent assigns an appropriate cluster  $\overline{V}_t$  for user  $i_t$  and recommends an item  $a_t \in \mathcal{A}_t$  based on the aggregated historical information gathered from cluster  $\overline{V}_t$ . After receiving the recommended item  $a_t$ , user  $i_t$  gives a random reward  $r_t \in [0,1]$  to the agent. To better model real-world scenarios, we assume that the reward  $r_t$  follows a misspecified linear function of the item feature vector  $\boldsymbol{x}_{a_t}$  and the unknown user preference vector  $\boldsymbol{\theta}_{i_t}$ . Formally,
$$
r _ {t} = \boldsymbol {x} _ {a _ {t}} ^ {\top} \boldsymbol {\theta} _ {i t} + \boldsymbol {\epsilon} _ {a _ {t}} ^ {i _ {t}, t} + \eta_ {t}, \tag {1}
$$
where  $\epsilon^{i_t,t} = [\epsilon_1^{i_t,t},\epsilon_2^{i_t,t},\dots,\epsilon_{|\mathcal{A}_t|}^{i_t,t}]^\top \in \mathbb{R}^{|A_t|}$  denotes the unknown deviation in the expected rewards of arms in  $\mathcal{A}_t$  from linearity for user  $i_t$  at  $t$ , and  $\eta_t$  is the 1-sub-Gaussian noise. We allow the deviation vectors for users in the same ground-truth cluster to be different.
We assume the clusters, users, items, and model misspecifications satisfy the following assumptions.
Assumption 3.1 (Gap between different clusters). The gap between any two preference vectors for different ground-truth clusters is at least an unknown positive constant  $\gamma$
$$
\left\| \boldsymbol {\theta} ^ {j} - \boldsymbol {\theta} ^ {j ^ {\prime}} \right\| _ {2} \geq \gamma > 0, \forall j, j ^ {\prime} \in [ m ], j \neq j ^ {\prime}.
$$
Assumption 3.2 (Uniform arrival of users). At each round  $t$ , a user  $i_t$  comes uniformly at random from  $\mathcal{U}$  with probability  $1 / u$ , independent of the past rounds.
Assumption 3.3 (Item regularity). At each time step  $t$ , the feature vector  $\pmb{x}_a$  of each arm  $a \in \mathcal{A}_t$  is drawn independently from a fixed but unknown distribution  $\rho$  over  $\{\pmb{x} \in \mathbb{R}^d : \| \pmb{x} \|_2 \leq 1\}$ , where  $\mathbb{E}_{\pmb{x} \sim \rho}[\pmb{x}\pmb{x}^\top]$  is full rank with minimal eigenvalue  $\lambda_x > 0$ . Additionally, at any time  $t$ , for any fixed unit vector  $\pmb{\theta} \in \mathbb{R}^d$ ,  $(\pmb{\theta}^\top \pmb{x})^2$  has sub-Gaussian tail with variance upper bounded by  $\sigma^2$ .
Assumption 3.4 (Bounded misspecification level). We assume that there is a pre-specified maximum misspecification level parameter  $\epsilon_{*}$  such that  $\| \pmb{\epsilon}^{i,t}\|_{\infty}\leq \epsilon_{*},\forall i\in \mathcal{U},t\in [T]$
Remark 1. All these assumptions basically follow previous works on CB [12, 13, 25, 3, 28] and MLB [23]. Note that Assumption 3.3 is less stringent and more practical than previous CB works which also put restrictions on the variance upper bound  $\sigma^2$ . For Assumption 3.2, our results can easily generalize to the case where the user arrival follows any distributions with minimum arrival probability greater than  $p_{min}$ . For Assumption 3.4, note that  $\epsilon_*$  can be an upper bound on the maximum misspecification level, not the exact maximum itself. In real-world applications, the deviations are usually small [14], and we can set a relatively big  $\epsilon_*$  as an upper bound. For more discussions please refer to Appendix B
Let  $a_{t}^{*}\in \arg \max_{a\in \mathcal{A}_{t}}\pmb{x}_{a}^{\top}\pmb{\theta}_{i_{t}} + \pmb{\epsilon}_{a}^{i_{t},t}$  denote an optimal arm which gives the highest expected reward at  $t$ . The goal of the agent is to minimize the expected cumulative regret
$$
R (T) = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left(\boldsymbol {x} _ {a _ {t} ^ {*}} ^ {\top} \boldsymbol {\theta} _ {i _ {t}} + \boldsymbol {\epsilon} _ {a _ {t} ^ {*}} ^ {i _ {t}, t} - \boldsymbol {x} _ {a _ {t}} ^ {\top} \boldsymbol {\theta} _ {i _ {t}} - \boldsymbol {\epsilon} _ {a _ {t}} ^ {i _ {t}, t}\right) \right]. \tag {2}
$$
This section introduces our algorithm called "Robust CLUstering of Misspecified Bandits" (RCLUMB) (Algo.1). RCLUMB is a graph-based algorithm. The ideas and techniques of RCLUMB can be easily generalized to set-based algorithms. To illustrate this generalizability, we also design a set-based algorithm RSLUMB. We leave the exposition and analysis of RSLUMB in Appendix K.
For ease of interpretation, we define the coefficient
$$
\zeta \triangleq 2 \epsilon_ {*} \sqrt {\frac {2}{\tilde {\lambda} _ {x}}}, \tag {3}
$$
where  $\tilde{\lambda}_x\triangleq \int_0^{\lambda_x}(1 - e^{-\frac{(\lambda_x - x)^2}{2\sigma^2}})^C dx$ .  $\zeta$  is theoretically the minimum gap between two users' preference vectors that an algorithm can distinguish with high probability, as supported by Eq.(50) in the proof of Lemma H.1 in Appendix H. Note that the algorithm does not require knowledge of  $\zeta$ . We also make the following definition for illustration.
Algorithm 1 Robust Clustering of Misspecified Bandits Algorithm (RCLUMB)
1: Input: Deletion parameter  $\alpha_{1},\alpha_{2} > 0,f(T) = \sqrt{\frac{1 + \ln(1 + T)}{1 + T}},\lambda ,\beta ,\epsilon_{*} > 0.$
2: Initialization:  $M_{i,0} = 0_{d\times d},\pmb{b}_{i,0} = 0_{d\times 1},T_{i,0} = 0,\forall i\in \mathcal{U}$ ; a complete Graph  $G_0 = (\mathcal{U},E_0)$  over  $\mathcal{U}$ .
3: for all  $t = 1,2,\ldots ,T$  do
4: Receive the index of the current user  $i_t \in \mathcal{U}$ , and the current feasible arm set  $\mathcal{A}_t$ ;
5: Filter user  $i_t$  and users  $i \in \mathcal{U}$  that are directly connected with user  $i_t$  via edge  $(i, i_t) \in E_{t-1}$ , to form the cluster  $\overline{V}_t$ ;
6: Compute the estimated statistics for cluster  $\overline{V}_t$
$$
\overline {{M}} _ {\bar {V} _ {t}, t - 1} = \lambda I + \sum_ {i \in \bar {V} _ {t}} M _ {i, t - 1}, \bar {b} _ {\bar {V} _ {t}, t - 1} = \sum_ {i \in \bar {V} _ {t}} b _ {i, t - 1}, \hat {\theta} _ {\bar {V} _ {t}, t - 1} = \overline {{M}} _ {\bar {V} _ {t}, t - 1} ^ {- 1} \bar {b} _ {\bar {V} _ {t}, t - 1};
$$
7: Recommend an arm  $a_{t}$  with the largest UCB index (Eq.(5)), and receive the reward  $r_t \in [0,1]$ ;
8: Update the statistics for user  $i_t$ $M_{i_t,t} = M_{i_t,t-1} + \boldsymbol{x}_{a_t} \boldsymbol{x}_{a_t}^\top$ ,  $\boldsymbol{b}_{i_t,t} = \boldsymbol{b}_{i_t,t-1} + r_t \boldsymbol{x}_{a_t}$ ,  $T_{i_t,t} = T_{i_t,t-1} + 1$ ,  $\hat{\boldsymbol{\theta}}_{i_t,t} = (\lambda I + M_{i_t,t})^{-1} \boldsymbol{b}_{i_t,t}$ .
9: Keep the statistics of other users unchanged  $M_{\ell ,t} = M_{\ell ,t - 1},b_{\ell ,t} = b_{\ell ,t - 1},T_{\ell ,t} = T_{\ell ,t - 1},\hat{\pmb{\theta}}_{\ell ,t} = \hat{\pmb{\theta}}_{\ell ,t - 1}$  for all  $\ell \in \mathcal{U},\ell \neq i_t$
10: Delete the edge  $(i_t,\ell)\in E_{t - 1}$  , if
$$
\left\| \hat {\boldsymbol {\theta}} _ {i _ {t}, t} - \hat {\boldsymbol {\theta}} _ {\ell , t} \right\| _ {2} \geq \alpha_ {1} \left(f \left(T _ {i _ {t}, t}\right) + f \left(T _ {\ell , t}\right)\right) + \alpha_ {2} \epsilon_ {*},
$$
and get an updated graph  $G_{t} = (\mathcal{U},E_{t})$
Definition 4.1 ( $\zeta$ -close users and  $\zeta$ -good clusters). Two users  $i, i' \in \mathcal{U}$  are  $\zeta$ -close if  $\|\pmb{\theta}_i - \pmb{\theta}_{i'}\|_2 \leq \zeta$ . Cluster  $\overline{V}$  is a  $\zeta$ -good cluster at time  $t$ , if  $\forall i \in \overline{V}$ , user  $i$  and the coming user  $i_t$  are  $\zeta$ -close.
We also say that two ground-truth clusters are “ $\zeta$ -close” if their preference vectors' gap is less than  $\zeta$ . Now we introduce the process and intuitions of RCLUMB (Algo.1). The algorithm maintains an undirected user graph  $G_{t} = (\mathcal{U}, E_{t})$ , where users are connected with edges if they are inferred to be in the same cluster. We denote the connected component in  $G_{t-1}$  containing user  $i_{t}$  at round  $t$  as  $\tilde{V}_{t}$ .
Cluster Detection.  $G_{0}$  is initialized to be a complete graph, and will be updated adaptively based on the interactive information. At round  $t$ , user  $i_{t} \in \mathcal{U}$  comes to be served with a feasible arm set  $\mathcal{A}_{t}$  (Line 4). Due to model misspecifications, it is impossible to cluster users with exactly the same preference vector  $\theta$ , but similar users whose preference vectors are within the distance of  $\zeta$ . According to the proof of Lemma H.1, after a sufficient time, with high probability, any pair of users directly connected by an edge in  $E_{t-1}$  are  $\zeta$ -close. However, if we trivially follow previous CB works [12, 25, 28] to directly use the connected component  $\tilde{V}_{t}$  as the inferred cluster for user  $i_{t}$  at round  $t$ , it will cause a large regret. The reason is that in the worst case, the preference vector  $\theta$  of the user in  $\tilde{V}_{t}$  who is  $h$ -hop away from user  $i_{t}$  could deviate by  $h\zeta$  from  $\theta_{i_t}$ , where  $h$  can be as large as  $|\tilde{V}_t|$ . Based on this reasoning, our key point is to select the cluster  $\overline{V}_t$  as the users at most 1-hop away from  $i_{t}$  in the graph. In other words, after some interactions,  $\overline{V}_t$  forms a  $\zeta$ -good cluster with high probability; thus, RCLUMB can avoid using misleading information from dissimilar users for recommendations.
Cluster-based Recommendation. After finding the appropriate cluster  $\overline{V}_t$  for  $i_t$ , the agent estimates the common user preference vector based on the historical information associated with cluster  $\overline{V}_t$  by
$$
\hat {\boldsymbol {\theta}} _ {\overline {{V}} _ {t}, t - 1} = \underset {\boldsymbol {\theta} \in \mathbb {R} ^ {d}} {\arg \min } \sum_ {s \in [ t - 1 ] \atop i _ {s} \in \overline {{V}} _ {t}} \left(r _ {s} - \boldsymbol {x} _ {a _ {s}} ^ {\top} \boldsymbol {\theta}\right) ^ {2} + \lambda \| \boldsymbol {\theta} \| _ {2} ^ {2}, \tag {4}
$$
where  $\lambda > 0$  is a regularization coefficient. Its closed-form solution is  $\hat{\pmb{\theta}}_{\overline{V}_t,t - 1} = \overline{M}_{\overline{V}_t,t - 1}^{-1}\overline{b}_{\overline{V}_t,t - 1}$ , where  $\overline{M}_{\overline{V}_t,t - 1} = \lambda I + \sum_{\substack{s\in [t - 1]\\ i_s\in \overline{V}_t}}\boldsymbol{x}_{a_s}\boldsymbol{x}_{a_s}^\top$ ,  $\overline{b}_{\overline{V}_t,t - 1} = \sum_{\substack{s\in [t - 1]\\ i_s\in \overline{V}_t}}r_{a_s}\boldsymbol{x}_{a_s}$ .
Based on this estimation, in Line 7, the agent recommends an arm using the UCB strategy
$$
a _ {t} = \operatorname {a r g m a x} _ {a \in \mathcal {A} _ {t}} \min  \{1, \underbrace {\boldsymbol {x} _ {a} ^ {\top} \hat {\boldsymbol {\theta}} _ {\overline {{V}} _ {t} , t - 1}} _ {\hat {R} _ {a, t}} + \underbrace {\beta \| \boldsymbol {x} _ {a} \| _ {\overline {{M}} _ {V _ {t} , t - 1} ^ {- 1}} + \epsilon_ {*} \sum_ {\substack {s \in [ t - 1 ] \\ i _ {s} \in \overline {{V}} _ {t}}} \left| \boldsymbol {x} _ {a} ^ {\top} \overline {{M}} _ {V _ {t} , t - 1} ^ {- 1} \boldsymbol {x} _ {a _ {s}} \right|} _ {C _ {a, t}} \}, \tag{5}
$$
where  $\beta = \sqrt{\lambda} +\sqrt{2\log(\frac{1}{\delta}) + d\log(1 + \frac{T}{\lambda d})}$ ,  $\hat{R}_{a,t}$  denotes the estimated reward of arm  $a$  at  $t$ ,  $C_{a,t}$  denotes the confidence radius of arm  $a$  at round  $t$ .