Title: Online Clustering of Bandits with Misspecified User Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: online clustering of bandits
Abstract: The contextual linear bandit is an important online learning problem where given arm features, a learning agent selects an arm at each round to maximize the cumulative rewards in the long run. A line of works, called the clustering of bandits (CB), utilize the collaborative effect over user preferences and have shown significant improvements over classic linear bandit algorithms. However, existing CB algorithms require well-specified linear user models and can fail when this critical assumption does not hold. Whether robust CB algorithms can be designed for more practical scenarios with misspecified user models remains an open problem. In this paper, we are the first to present the important problem of clustering of bandits with misspecified user models (CBMUM), where the expected rewards in user models can be perturbed away from perfect linear models. We devise two robust CB algorithms, RCLUMB and RSCLUMB (representing the learned clustering structure with dynamic graph and sets, respectively), that can accommodate the inaccurate user preference estimations and erroneous clustering caused by model misspecifications. We prove regret upper bounds of $O(\epsilon_*T\sqrt{md\log T}  + d\sqrt{mT}\log T)$ for our algorithms under milder assumptions than previous CB works, which match the lower bound asymptotically in $T$ up to logarithmic factors, and also match the state-of-the-art results in several degenerate cases. Our regret analysis is novel and different from the typical proof flow of previous CB works. The techniques in proving the regret caused by misclustering users are quite general and may be of independent interest. Experiments on both synthetic and real-world data show our outperformance over previous algorithms.

Online Clustering of Bandits with Misspecified User Models

Abstract

The contextual linear bandit is an important online learning problem where given arm features, a learning agent selects an arm at each round to maximize the cumulative rewards in the long run. A line of works, called the clustering of bandits (CB), utilize the collaborative effect over user preferences and have shown significant improvements over classic linear bandit algorithms. However, existing CB algorithms require well-specified linear user models and can fail when this critical assumption does not hold. Whether robust CB algorithms can be designed for more practical scenarios with misspecified user models remains an open problem. In this paper, we are the first to present the important problem of clustering of bandits with misspecified user models (CBMUM), where the expected rewards in user models can be perturbed away from perfect linear models. We devise two robust CB algorithms, RCLUMB and RSCLUMB (representing the learned clustering structure with dynamic graph and sets, respectively), that can accommodate the inaccurate user preference estimations and erroneous clustering caused by model misspecifications. We prove regret upper bounds of O ( ϵ ∗ T √ md log T + d √ mT log T ) for our algorithms under milder assumptions than previous CB works (notably, we move past a restrictive technical assumption on the distribution of the arms), which match the lower bound asymptotically in T up to logarithmic factors, and also match the state-of-the-art results in several degenerate cases. The techniques in proving the regret caused by misclustering users are quite general and may be of independent interest. Experiments on both synthetic and real-world data show our outperformance over previous algorithms.

1 Introduction

Stochastic multi-armed bandit (MAB) [2, 4, 22] is an online sequential decision-making problem, where the learning agent selects an action and receives a corresponding reward at each round, so as to maximize the cumulative reward in the long run. MAB algorithms have been widely applied in

Jize Xie

Shuai Li ∗

recommendation systems and computer networks to handle the exploration and exploitation trade-off [20, 30, 38, 5].

To deal with large-scale applications, the contextual linear bandits [24, 9, 1, 29, 21] have been studied, where the expected reward of each arm is assumed to be perfectly linear in their features. Leveraging the contextual side information about the user and arms, linear bandits can provide more personalized recommendations [16]. Classical linear bandit approaches, however, ignore the often useful tool of collaborative filtering. To utilize the relationships among users, the problem of clustering of bandits (CB) has been proposed [12]. Specifically, CB algorithms adaptively partition users into clusters and utilize the collaborative effect of users to enhance learning performance.

Although existing CB algorithms have shown great success in improving recommendation qualities, there exist two major limitations. First, all previous works on CB [12, 25, 27, 39] assume that for each user, the expected rewards follow a perfectly linear model with respect to the user preference vector and arms' feature vectors. In many real-world scenarios, due to feature noises or uncertainty [15], the reward may not necessarily conform to a perfectly linear function, or even deviates a lot from linearity [14]. Second, previous CB works assume that for users within the same cluster, their preferences are exactly the same. Due to the heterogeneity in users' personalities and interests, similar users may not have identical preferences, invalidating this strong assumption.

To address these issues, we propose a novel problem of clustering of bandits with misspecified user models (CBMUM). In CBMUM, the expected reward model of each user does not follow a perfectly linear function but with possible additive deviations. We assume users in the same underlying cluster share a common preference vector, meaning they have the same linear part in reward models, but the deviation parts are allowed to be different, better reflecting the varieties of user personalities.

The relaxation of perfect linearity and the reward homogeneity within the same cluster bring many challenges to the CBMUM problem. In CBMUM, we not only need to handle the uncertainty from the unknown user preference vectors, but also have to tackle the additional uncertainty from model misspecifications. Due to such uncertainties, it becomes highly challenging to design a robust algorithm that can cluster the users appropriately and utilize the clustered information judiciously. On the one hand, the algorithm needs to be more tolerant in the face of misspecifications so that more similar users can be clustered together to utilize the collaborative effect. On the other hand, it has to be more selective to rule out the possibility of misclustering users with large preference gaps.

1.1 Our Contributions

This paper makes the following four contributions.

New Model Formulation. We are the first to formulate the clustering of bandits with misspecified user models (CBMUM) problem, which is more practical by removing the perfect linearity assumption in previous CB works.

Novel Algorithm Designs. We design two novel algorithms, RCLUMB and RSCLUMB, which robustly learn the clustering structure and utilize this collaborative information for faster user preference elicitation. Specifically, RCLUMB keeps updating a dynamic graph over all users, where users connected directly by edges are supposed to be in the same cluster. RCLUMB adaptively removes edges and recommends items based on historical interactions. RSCLUMB represents the clustering structure with sets, which are dynamicly merged and split during the learning process. Due to the page limit, we only illustrate the RCLUMB algorithm in the main paper. We leave the exposition, illustration, and regret analysis of the RSCLUMB algorithm in Appendix K.

To overcome the challenges brought by model misspecifications, we do the following key steps in the RCLUMB algorithm. (i) To ensure that with high probability, similar users will not be partitioned apart, we design a more tolerant edge deletion rule by taking model misspecifications into consideration. (ii) Due to inaccurate user preference estimations caused by model misspecifications, trivially following previous CB works [12, 25, 28] to directly use connected components in the maintained graph as clusters would miscluster users with big preference gaps, causing a large regret. To be discriminative in cluster assignments, we filter users directly linked with the current user in the graph to form the cluster used in this round. With these careful designs of (i) and (ii), we can guarantee that with high probability, information of all similar users can be leveraged, and only users with close enough preferences might be misclustered , which will only mildly impair the

learning accuracy. Additionally: (iii) we design an enlarged confidence radius to incorporate both the exploration bonus and the additional uncertainty from misspecifications when recommending arms. The design of RSCLUMB follows similar ideas, which we leave in the Appendix K due to page limit.

Theoretical Analysis with Milder Assumptions . We prove regret upper bounds for our algorithms of O ( ϵ ∗ T √ md log T + d √ mT log T ) in CBMUM under much milder and practical assumptions (in arm generation distribution) than previous CB works, which match the state-of-the-art results in degenerate cases. Our proof is quite different from the typical proof flow of previous CB works (details in Appendix C). One key challenge is to bound the regret caused by misclustering users with close but not the same preference vectors and use the inaccurate cluster-based information to recommend arms. To handle the challenge, we prove a key lemma (Lemma 5.7) to bound this part of regret. We defer its details in Section 5 and Appendix G. The techniques and results for bounding this part are quite general and may be of independent interest. We also give a regret lower bound of Ω( ϵ ∗ T √ d ) for CBMUM, showing that our upper bounds are asymptotically tight with respect to T up to logarithmic factors. We leave proving a tighter lower bound for CBMUM as an open problem.

Good Experimental Performance. Extensive experiments on both synthetic and real-world data show the advantages of our proposed algorithms over the existing algorithms.

2 Related Work

Our work is closely related to two lines of research: online clustering of bandits (CB) and misspecified linear bandits (MLB). More discussions on related works can be found in Appendix A.

The paper [12] first formulates the CB problem and proposes a graph-based algorithm. The work [26] further considers leveraging the collaborative effects on items to guide the clustering of users. The work [25] considers the CB problem in the cascading bandits setting with random prefix feedback. The paper [27] also considers users with different arrival frequencies. A recent work [28] proposes the setting of clustering of federated bandits, considering both privacy protection and communication requirements. However, all these works assume that the reward model for each user follows a perfectly linear model, which is unrealistic in many real-world applications. To the best of our knowledge, this paper is the first work to consider user model misspecifications in the CB problem.

The work [14] first proposes the misspecified linear bandits (MLB) problem, shows the vulnerability of linear bandit algorithms under deviations, and designs an algorithm RLB that is only robust to non-sparse deviations. The work [23] proposes two algorithms to handle general deviations, which are modifications of the phased elimination algorithm [22] and LinUCB [1]. Some recent works [31, 11] use model selection methods to deal with unknown exact maximum model misspecification level. Note that the work [11] has an additional assumption on the access to an online regression oracle, and the paper [31] still needs to know an upper bound of the unknown exact maximum model deviation level. None of them consider the CB setting with multiple users, thus differing from ours.

We are the first to initialize the study of the important CBMUM problem, and propose a general framework for dealing with model misspecifications in CB problems. Our study is based on fundamental models on CB [12, 27] and MLB [23], the algorithm design ideas and theoretical analysis are pretty general. We leave incorporating the model selection methods [31, 11] into our framework to address the unknown exact maximum model misspecification level as an interesting future work.

3 Problem Setup

This section formulates the problem of 'clustering of bandits with misspecified user models" (CBMUM).Weuseboldface lowercase and boldface CAPITALIZED letters for vectors and matrices. We use |A| to denote the number of elements in A , [ m ] to denote { 1 , . . . , m } , and ∥ x ∥ M = √ x ⊤ Mx to denote the matrix norm of vector x regarding the positive semi-definite (PSD) matrix M .

̸

In CBMUM, there are u users denoted by U = { 1 , 2 , . . . , u } . Each user i ∈ U is associated with an unknown preference vector θ i ∈ R d , with ∥ θ i ∥ 2 ≤ 1 . We assume there is an unknown underlying clustering structure over users representing the similarity of their behaviors. Specifically, U can be partitioned into a small number m (i.e., m ≪ u ) clusters, V 1 , V 2 , . . . V m , where ∪ j ∈ [ m ] V j = U , and V j ∩ V j ′ = ∅ , for j = j ′ . We call these clusters ground-truth clusters and use V = { V 1 , V 2 , . . . , V m } to denote the set of these clusters. Users in the same ground-truth cluster share the same preference vector, while users from different ground-truth clusters have different preference vectors. Let θ j

denote the common preference vector for V j and j ( i ) ∈ [ m ] denote the index of the ground-truth cluster that user i belongs to. For any ℓ ∈ U , if ℓ ∈ V j ( i ) , then θ ℓ = θ i = θ j ( i ) .

At each round t ∈ [ T ] , a user i t ∈ U comes to be served. The learning agent receives a finite arm set A t ⊆ A to choose from (with |A t | ≤ C, ∀ t ), where each arm a ∈ A is associated with a feature vector x a ∈ R d , and ∥ x a ∥ 2 ≤ 1 . The agent assigns an appropriate cluster V t for user i t and recommends an item a t ∈ A t based on the aggregated historical information gathered from cluster V t . After receiving the recommended item a t , user i t gives a random reward r t ∈ [0 , 1] to the agent. To better model real-world scenarios, we assume that the reward r t follows a misspecified linear function of the item feature vector x a t and the unknown user preference vector θ i t . Formally,

where ϵ i t ,t = [ ϵ i t ,t 1 , ϵ i t ,t 2 , . . . , ϵ i t ,t |A t | ] ⊤ ∈ R |A t | denotes the unknown deviation in the expected rewards of arms in A t from linearity for user i t at t , and η t is the 1-sub-Gaussian noise. We allow the deviation vectors for users in the same ground-truth cluster to be different.

We assume the clusters, users, items, and model misspecifications satisfy the following assumptions.

Assumption 3.1 (Gap between different clusters) . The gap between any two preference vectors for different ground-truth clusters is at least an unknown positive constant γ

̸

Assumption 3.2 (Uniform arrival of users) . At each round t , a user i t comes uniformly at random from U with probability 1 /u , independent of the past rounds.

Assumption 3.3 (Item regularity) . At each time step t , the feature vector x a of each arm a ∈ A t is drawn independently from a fixed but unknown distribution ρ over { x ∈ R d : ∥ x ∥ 2 ≤ 1 } , where E x ∼ ρ [ xx ⊤ ] is full rank with minimal eigenvalue λ x > 0 . Additionally, at any time t , for any fixed unit vector θ ∈ R d , ( θ ⊤ x ) 2 has sub-Gaussian tail with variance upper bounded by σ 2 .

Assumption 3.4 (Bounded misspecification level) . We assume that there is a pre-specified maximum misspecification level parameter ϵ ∗ such that ∥ ∥ ϵ i,t ∥ ∥ ∞ ≤ ϵ ∗ , ∀ i ∈ U , t ∈ [ T ] .

Remark 1. All these assumptions basically follow previous works on CB [12, 13, 25, 3, 28] and MLB [23]. Note that Assumption 3.3 is less stringent and more practical than previous CB works which also put restrictions on the variance upper bound σ 2 . For Assumption 3.2, our results can easily generalize to the case where the user arrival follows any distributions with minimum arrival probability greater than p min . For Assumption 3.4, note that ϵ ∗ can be an upper bound on the maximum misspecification level, not the exact maximum itself. In real-world applications, the deviations are usually small [14], and we can set a relatively big ϵ ∗ as an upper bound. For more discussions please refer to Appendix B

Let a ∗ t ∈ arg max a ∈A t x ⊤ a θ i t + ϵ i t ,t a denote an optimal arm which gives the highest expected reward at t . The goal of the agent is to minimize the expected cumulative regret

4 Algorithm

This section introduces our algorithm called 'Robust CLUstering of Misspecified Bandits" (RCLUMB) (Algo.1). RCLUMB is a graph-based algorithm. The ideas and techniques of RCLUMB can be easily generalized to set-based algorithms. To illustrate this generalizability, we also design a set-based algorithm RSCLUMB. We leave the exposition and analysis of RSCLUMB in Appendix K.

For ease of interpretation, we define the coefficient

where ˜ λ x ≜ ∫ λ x 0 (1 -e -( λx -x ) 2 2 σ 2 ) C dx . ζ is theoretically the minimum gap between two users' preference vectors that an algorithm can distinguish with high probability, as supported by Eq.(50) in the proof of Lemma H.1 in Appendix H. Note that the algorithm does not require knowledge of ζ . We also make the following definition for illustration.

Algorithm 1 Robust Clustering of Misspecified Bandits Algorithm (RCLUMB)

1: Input: Deletion parameter α 1 , α 2 > 0 , f ( T ) = √ 1+ln(1+ T ) 1+ T , λ, β, ϵ ∗ > 0 .

2: Initialization: M i, 0 = 0 d × d , b i, 0 = 0 d × 1 , T i, 0 = 0 , ∀ i ∈ U ; a complete Graph G 0 = ( U , E 0 ) over U .

3: for all t = 1 , 2 , . . . , T do

4: Receive the index of the current user i t ∈ U , and the current feasible arm set A t ;

5: Filter user i t and users i ∈ U that are directly connected with user i t via edge ( i, i t ) ∈ E t -1 , to form the cluster V t ;

6: Compute the estimated statistics for cluster V t

7: Recommend an arm a t with the largest UCB index (Eq.(5)), and receive the reward r t ∈ [0 , 1] ;

8: Update the statistics for user i t M i t ,t = M i t ,t -1 + x a t x ⊤ a t , b i t ,t = b i t ,t -1 + r t x a t , T i t ,t = T i t ,t -1 +1 , ˆ θ i t ,t = ( λ I + M i t ,t ) -1 b i t ,t ;

̸

9: Keep the statistics of other users unchanged M ℓ,t = M ℓ,t -1 , b ℓ,t = b ℓ,t -1 , T ℓ,t = T ℓ,t -1 , ˆ θ ℓ,t = ˆ θ ℓ,t -1 , for all ℓ ∈ U , ℓ = i t ;

10: Delete the edge ( i t , ℓ ) ∈ E t -1 , if

and get an updated graph G t = ( U , E t ) ;

Definition 4.1 ( ζ -close users and ζ -good clusters) . Two users i, i ′ ∈ U are ζ -close if ∥ θ i -θ i ′ ∥ 2 ≤ ζ . Cluster V is a ζ -good cluster at time t , if ∀ i ∈ V , user i and the coming user i t are ζ -close.

We also say that two ground-truth clusters are ' ζ -close" if their preference vectors' gap is less than ζ . Now we introduce the process and intuitions of RCLUMB (Algo.1). The algorithm maintains an undirected user graph G t = ( U , E t ) , where users are connected with edges if they are inferred to be in the same cluster. We denote the connected component in G t -1 containing user i t at round t as ˜ V t .

Cluster Detection. G 0 is initialized to be a complete graph, and will be updated adaptively based on the interactive information. At round t , user i t ∈ U comes to be served with a feasible arm set A t (Line 4). Due to model misspecifications, it is impossible to cluster users with exactly the same preference vector θ , but similar users whose preference vectors are within the distance of ζ . According to the proof of Lemma H.1, after a sufficient time, with high probability, any pair of users directly connected by an edge in E t -1 are ζ -close. However, if we trivially follow previous CB works [12, 25, 28] to directly use the connected component ˜ V t as the inferred cluster for user i t at round t , it will cause a large regret. The reason is that in the worst case, the preference vector θ of the user in ˜ V t who is h -hop away from user i t could deviate by hζ from θ i t , where h can be as large as | ˜ V t | . Based on this reasoning, our key point is to select the cluster V t as the users at most 1-hop away from i t in the graph. In other words, after some interactions, V t forms a ζ -good cluster with high probability; thus, RCLUMB can avoid using misleading information from dissimilar users for recommendations.

Cluster-based Recommendation. After finding the appropriate cluster V t for i t , the agent estimates the common user preference vector based on the historical information associated with cluster V t by

where λ > 0 is a regularization coefficient. Its closed-form solution is ˆ θ V t ,t -1 = M -1 V t ,t -1 b V t ,t -1 , where M V t ,t -1 = λ I + ∑ s ∈ [ t -1] i s ∈ V t x a s x ⊤ a s , b V t ,t -1 = ∑ s ∈ [ t -1] i s ∈ V t r a s x a s .

Based on this estimation, in Line 7, the agent recommends an arm using the UCB strategy

where β = √ λ + √ 2 log( 1 δ ) + d log(1 + T λd ) , ˆ R a,t denotes the estimated reward of arm a at t , C a,t denotes the confidence radius of arm a at round t .

Due to deviations from linearity, the estimation ˆ R a,t computed by a linear function is no longer accurate. To handle the estimation uncertainty of model misspecifications, we design an enlarged confidence radius C a,t . The first term of C a,t in Eq.(5) captures the uncertainty of online learning for the linear part, and the second term related to ϵ ∗ reflects the additional uncertainty from deviations from linearity. The design of C a,t theoretically relies on Lemma 5.6 which will be given in Section 5.

Update User Statistics. Based the feedback r t , in Line 8 and 9, the agent updates the statistics for user i t . Specifically, the agent estimates the preference vector θ i t by

with solution ˆ θ i t ,t = ( λ I + M i t ,t ) -1 b i t ,t , where M i t ,t = ∑ s ∈ [ t ] i s = i t x a s x ⊤ a s , b i t ,t = ∑ s ∈ [ t ] i s = i t r a s x a s .

Update the Graph G t . Finally, in Line 10, the agent verifies whether the similarities between user i t and other users are still true based on the updated estimation ˆ θ i t ,t . For every user ℓ ∈ U connected with user i t via edge ( i t , ℓ ) ∈ E t -1 , if the gap between her estimated preference vector ˆ θ ℓ,t and ˆ θ i t ,t is larger than a threshold supported by Lemma H.1, the agent will delete the edge ( i t , ℓ ) to split them apart. The threshold in Line 10 is carefully designed, taking both estimation uncertainty in a linear model and deviations from linearity into consideration. As shown in the proof of Lemma H.1 (in Appendix H), using this threshold, with high probability, edges between users in the same ground-truth clusters will not be deleted, and edges between users that are not ζ -close will always be deleted. Together with the filtering step in Line 5, with high probability, the algorithm will leverage all the collaborative information of similar users and avoid misusing the information of dissimilar users. The updated graph G t will be used in the next round.

5 Theoretical Analysis

In this section, we theoretically analyze the performance of the RCLUMB algorithm by giving an upper bound of the expected regret defined in Eq.(2). Due to the space limitation, we only show the main result (Theorem 5.3), key lemmas, and a sketched proof for Theorem 5.3. Detailed proofs, other technical lemmas, and the regret analysis of the RSLUMB algorithm can be found in the Appendix.

To state our main result, we first give two definitions as follows. The first definition is about the minimum separable gap constant γ 1 of a CBMUM problem instance.

Definition 5.1 (Minimum separable gap γ 1 ) . The minimum separable gap constant γ 1 of a CBMUM problem instance is the minimum gap over the gaps among users that are greater than ζ (Eq. (3))

Remark 2. In CBMUM, the role of γ 1 -ζ is similar to that of γ (given in Assumption 3.1) in the previous CB problem with perfectly linear models, quantifying the hardness of performing clustering on the problem instance. Intuitively, users are easier to cluster if γ 1 is larger, and the deduction of ζ shows the additional difficulty due to model diviations. If there are no misspecifications, i.e.,

ζ = 2 ϵ ∗ √ 2 λ x = 0 , then γ 1 = γ , recovering the minimum separable gap between clusters in the classic CB problem [12, 25] without model misspecifications.

The second definition is about the number of 'hard-to-cluster users" ˜ u .

Definition 5.2 (Number of 'hard-to-cluster users" ˜ u ) . The number of 'hard-to-cluster users" ˜ u is the number of users in the ground-truth clusters which are ζ -close to some other ground-truth cluster s

̸

where I {·} denotes the indicator function of the argument, | V j | denotes the number of users in V j .

Remark 3. ˜ u captures the number of users who belong to different ground-truth clusters but their gaps are less than ζ . These users may be merged into one cluster by mistake and cause certain regret.

The following theorem gives an upper bound on the expected regret achieved by RCLUMB.

Theorem 5.3 (Main result on regret bound) . Suppose that the assumptions in Section 3 are satisfied. Then the expected regret of the RCLUMB algorithm for T rounds satisfies

where γ 1 is defined in Definition 5.1, and ˜ u is defined in Definition 5.2).

Discussion and Comparison. The bound in Eq.(7) has four terms. The first term is the time needed to gather enough information to assign appropriate clusters for users. The second term is the regret caused by misclustering ζ -close but not precisely similar users together, which is unavoidable with model misspecifications. The third term is from the preference estimation errors caused by model deviations. The last term is the usual term in CB with perfectly linear models [12, 25, 27].

Let us discuss how the parameters affect this regret bound.

If γ 1 -ζ is large, the gaps between clusters that are not ' ζ -close" are much greater than the minimum gap ζ for the algorithm to distinguish, the first term in Eq.(7) will be small as it is easy to identify their dissimilarities. The role of γ 1 -ζ in CBMUM is similar to that of γ in the previous CB.

If ˜ u is small, indicating that few ground-truth clusters are ' ζ -close", RCLUMB will hardly miscluster different ground-truth clusters together thus the second term in Eq.(7) will be small.

If the deviation level ϵ ∗ is small, the user models are close to linearity and the misspecifications will not affect the estimations much, then both the second and third term in Eq.(7) will be small.

The following theorem gives a regret lower bound of the CBMUM problem.

Theorem 5.4 (Regret lower bound for CBMUM) . There exists a problem instance for the CBMUM problem such that for any algorithm R ( T ) ≥ Ω( ϵ ∗ T √ d ) .

The proof can be found in Appendix F. The upper bounds in Theorem 5.3 asymptotically match this lower bound with respect to T up to logarithmic factors (and a constant factor of √ m where m is typically small in real-applications), showing the tightness of our theoretical results. Additionally, we conjecture the gap for the m factor is due to the strong assumption that cluster structures are known to prove this lower bound, and whether there exists a tighter lower bound is left for future work.

We then compare our results with two degenerate cases. First, when m = 1 (indicating ˜ u = 0 ), our setting degenerates to the MLB problem where all users share the same preference vector. In this case, our regret bound is O ( ϵ ∗ T √ d log T + d √ T log T ) , exactly matching the current best bound of MLB [23]. Second, when ϵ ∗ = 0 , our setting reduces to the CB problem with perfectly linear user models and our bounds become O ( d √ mT log T ) , also perfectly match the existing best bound of the CB problem [25, 27]. The above discussions and comparisons show the tightness of our regret bounds. Additionally, we also provide detailed discussions on why trivially combining existing works on CB and MLB would not get any non-vacuous regret upper bound in Appendix D.

We define the following 'good partition" for ease of interpretation.

Definition 5.5 (Good partition) . RCLUMB does a 'good partition" at t , if the cluster V t assigned to i t is a ζ -good cluster, and it contains all the users in the same ground-truth cluster as i t , i.e.,

Note that when the algorithm does a 'good partition" at t , V t will contain all the users in the same ground-truth cluster as i t and may only contain some other ζ -close users with respect to i t , which means the gathered information associated with V t can be used to infer user i t 's preference with high accuracy. Also, it is obvious that under a 'good partition", if V t ∈ V , then V t = V j ( i t ) by definition.

Next, we give a sketched proof for Theorem 5.3.

Proof. [Sketch for Theorem 5.3] The proof mainly contains two parts. First, we prove there is a sufficient time T 0 for RCLUMB to get a 'good partition" with high probability. Second, we prove the regret upper bound for RCLUMB after maintaining a 'good partition". The most challenging part is to bound the regret caused by misclustering ζ -close users after getting a 'good partition".

1. Sufficient time to maintain a 'good partition". With the item regularity (Assumption 3.3), we can prove after some T 0 (defined in Lemma H.1 in Appendix H), RCLUMB will always have a

'good partition". Specifically, after t ≥ O ( u ( d ˜ λ x ( γ 1 -ζ ) 2 + 1 ˜ λ 2 x ) log T ) , for any user i ∈ U , the gap between the estimated ˆ θ i,t and the ground-truth θ j ( i ) is less than γ 1 4 with high probability. With this, we can get: for any two users i and ℓ , if their gap is greater than ζ , it will trigger the deletion of the edge ( i, ℓ ) (Line 10 of Algo.1) with high probability; on the other hand, when the deletion condition of the edge ( i, ℓ ) is satisfied, then ∥ ∥ ∥ θ j ( i ) -θ j ( ℓ ) ∥ ∥ ∥ 2 > 0 , which means user i and ℓ belong to different ground-truth clusters by Assumption 3.1 with high probability. Therefore, we can get that with high probability, all those users in the same ground-truth cluster as i t will be directly connected with i t , and users directly connected with i t must be ζ -close to i t . By filtering users directly linked with i t as the cluster V t (Algo.1 Line 5) and the definition of 'good partition", we can ensure that RCLUMB will keep a 'good partition" afterward with high probability.

Bounding the regret after getting a 'good partition". After T 0 , with the 'good partition", we can prove the following lemma that gives a bound of the difference between ˆ θ V t ,t -1 and ground-truth θ i t in direction of action vector x a , and supports the design of the confidence radius C a,t in Eq.(5).

Lemma 5.6. With probability at least 1 -5 δ for some δ ∈ (0 , 1 5 ) , ∀ t ≥ T 0

To prove this lemma, we consider the following two situations.

(i) Assigning a perfect cluster for i t . In this case, V t ∈ V , meaning the cluster assigned for user i t is the same as her ground-truth cluster , i.e., V t = V j ( i t ) . Therefore, we have that ∀ ℓ ∈ V t , θ ℓ = θ i t . With careful analysis, we can bound ∣ ∣ ∣ x ⊤ a ( θ i t -ˆ θ V t ,t -1 ) ∣ ∣ ∣ by C a,t (defined in Eq.(5)).

̸

(ii) Bounding the term of misclustering i t 's ζ -close users. In this case, V t / ∈ V , meaning the algorithm misclusters user i t , i.e., V t = V j ( i t ) . Thus, we do not have ∀ ℓ ∈ V t , θ ℓ = θ i t anymore, but we have all the users in V t are ζ -close to i t (by 'good partition"), i.e., ∥ θ i s -θ i t ∥ 2 ≤ ζ , ∀ ℓ ∈ V t . Then an additional term can be caused by using the information of i t 's ζ -close users in V t lying in different ground-truth clusters from i t to estimate θ i t . It is highly challenging to bound this part.

We will get an extra term ∣ ∣ ∣ ∣ x ⊤ a M -1 V t ,t -1 ∑ s ∈ [ t -1] i s ∈ V t x a s x ⊤ a s ( θ i s -θ i t ) ∣ ∣ ∣ ∣ when bounding the regret in this case, where ∥ θ ℓ -θ i t ∥ 2 ≤ ζ , ∀ ℓ ∈ V t . It is an easy-to-be-made mistake to directly drag ∥ θ i s -θ i t ∥ 2 out to bound it by ∥ ∥ ∥ ∥ x ⊤ a M -1 V t ,t -1 ∑ s ∈ [ t -1] i s ∈ V t x a s x ⊤ a s ∥ ∥ ∥ ∥ 2 × ζ . With subtle analysis, we propose the following lemma to bound the above term.

Lemma 5.7 (Bound of error caused by misclustering ) . ∀ t ≥ T 0 , if the current partition by RCLUMB is a 'good partition", and V t / ∈ V , then for all x a ∈ R d , ∥ x a ∥ 2 ≤ 1 , with probability at least 1 -δ :

This lemma is quite general. Please see Appendix G for details about its proof.

The expected occurrences of { V t / ∈ V} is bounded by ˜ u u T with Assumption 3.2, Definition 5.2 and 5.5. The result follows by bounding the expected sum of the bounds for the instantaneous regret using Lemma 5.6 with delicate analysis due to the time-varying clustering structure kept by RCLUMB.

6 Experiments

This section compares RCLUMB and RSCLUMB with CLUB [12], SCLUB [27], LinUCB with a single estimated vector for all users, LinUCB-Ind with separate estimated vectors for each user, and two modifications of LinUCB in [23] which we name as RLinUCB and RLinUCB-Ind. We use averaged reward as the evaluation metric, where the average is taken over ten independent trials.















1e6



































(a) Synthetic















1e6





























(b) Yelp Case 1















1e6





























(c) Yelp Case 2















1e6

















































1e6





























6.1 Synthetic Experiments

We consider a setting with u = 1 , 000 users, m = 10 clusters and T = 10 6 rounds. The preference and feature vectors are in d = 50 dimension with each entry drawn from a standard Gaussian distribution, and are normalized to vectors with ∥ . ∥ 2 = 1 [27]. We fix an arm set with |A| = 1000 items, at each round t , 20 items are randomly selected to form a set A t for the user to choose from. We construct a matrix ϵ ∈ R 1 , 000 × 1 , 000 in which each element ϵ ( i, j ) is drawn uniformly from the range ( -0 . 2 , 0 . 2) to represent the deviation. At t , for user i t and the item a t , ϵ ( i t , a t ) will be added to the feedback as the deviation, which corresponds to the ϵ i t ,t a t defined in Eq.(1).

The result is provided in Figure 1(a), showing that our algorithms have clear advantages: RCLUMB improves over CLUB by 21.9%, LinUCB by 194.8%, LinUCB-Ind by 20.1%, SCLUB by 12.0%, RLinUCB by 185.2% and RLinUCB-Ind by 10.6%. The performance difference between RCLUMB and RSCLUMB is very small as expected. RLinUCB performs better than LinUCB; RLinUCB-Ind performs better than LinUCB-Ind and CLUB, showing that the modification of the recommendation policy is effective. The set-based RSCLUMB and SCLUB can separate clusters quicker and have advantages in the early period, but eventually RCLUMB catches up with RSCLUMB, and SCLUB is surpassed by RLinUCB-Ind because it does not consider misspecifications. RCLUMB and RSCLUMB perform better than RLinUCB-Ind, which shows the advantage of the clustering. So it can be concluded that both the modification for misspecification and the clustering structure are critical to improving the algorithm's performance. We also have done some ablation experiments on different scales of ϵ ∗ in Appendix P , and we can notice that under different ϵ ∗ , our algorithms always outperform the baselines, and some baselines will perform worse as ϵ ∗ increases.

6.2 Experiments on Real-world Datasets

We conduct experiments on the Yelp data and the 20 m MovieLens data [17]. For both data, we have two cases due to the different methods for generating feedback. For case 1, we extract 1,000 items with most ratings and 1,000 users who rate most; then we construct a binary matrix H 1 , 000 × 1 , 000 based on the user rating [40, 42]: if the user rating is greater than 3, the feedback is 1; otherwise, the feedback is 0. Then we use this binary matrix to generate the preference and feature vectors by singular-value decomposition (SVD) [27, 25, 40]. Similar to the synthetic experiment, we construct a matrix ϵ ∈ R 1 , 000 × 1 , 000 in which each element is drawn uniformly from the range ( -0 . 2 , 0 . 2) . For case 2, we extract 1,100 users who rate most and 1000 items with most ratings. We construct a binary feedback matrix H 1 , 100 × 1 , 000 based on the same rule as case 1. Then we select the first 100 rows H 100 × 1 , 000 1 to generate the feature vectors by SVD. The remaining 1,000 rows F 1 , 000 × 1 , 000

is used as the feedback matrix, meaning user i receives F ( i, j ) as feedback while choosing item j . In both cases, at time t , we randomly select 20 items for the algorithms to choose from. In case 1, the feedback is computed by the preference and feature vector with misspecification, in case 2, the feedback is from the feedback matrix.

The results on Yelp are shown in Fig 1(b) and Fig 1(c). In case 1, RCLUMB improves CLUB by 45.1%, SCLUB by 53.4%, LinUCB-One by 170.1% , LinUCB-Ind by 46.2%, RLinUCB by 171.0% and RLinUCB-Ind by 21.5%. In case 2, RCLUMB improves over CLUB by 13.9%, SCLUB by 5.1%, LinUCB-One by 135.6% , LinUCB-Ind by 10.1%, RLinUCB by 138.6% and RLinUCB by 8.5%. It is notable that our modeling assumption 3.4 is violated in case 2 since the misspecification range is unknown. We set ϵ ∗ = 0 . 2 following our synthetic dataset and it can still perform better than other algorithms. When the misspecification level is known as in case 1, our algorithms' improvement is significantly enlarged, e.g., RCLUMB improves over SCLUB from 5.1% to 53.4%.

The results on Movielens are shown in Fig 1(d) and 1(e). In case 1, RCLUMB improves CLUB by 58.8%, SCLUB by 92.1%, LinUCB-One by 107.7%, LinUCB-Ind by 61.5 %, RLinUCB by 109.5%, and RLinUCB-Ind by 21.3%. In case 2, RCLUMB improves over CLUB by 5.5%, SCLUB by 2.9%, LinUCB-One by 28.5%, LinUCB-Ind by 6.1%, RLinUCB by 29.3% and RLinUCB-Ind by 5.8%. The results are consistent with the Yelp data, confirming our superior performance.

7 Conclusion

We present a new problem of clustering of bandits with misspecified user models (CBMUM), where the agent has to adaptively assign appropriate clusters for users under model misspecifications. We propose two robust CB algorithms, RCLUMB and RSCLUMB. Under milder assumptions than previous CB works, we prove the regret bounds of our algorithms, which match the lower bound asymptotically in T up to logarithmic factors, and match the state-of-the-art results in several degenerate cases. It is challenging to bound the regret caused by misclustering users with close but not the same preference vectors and use inaccurate cluster-based information to select arms. Our analysis to bound this part of the regret is quite general and may be of independent interest. Experiments on synthetic and real-world data demonstrate the advantage of our algorithms. We would like to state some interesting future works: (1) Prove a tighter regret lower bound for CBMUM, (2) Incorporate recent model selection methods into our fundamental framework to design robust algorithms for CBMUMwith unknown exact maximum model misspecification level, and (3) Consider the setting with misspecifications in the underlying user clustering structure rather than user models.

References

Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24, 2011.

Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning , 47(2):235-256, 2002.

Yikun Ban and Jingrui He. Local clustering in contextual multi-armed bandits. In Proceedings of the Web Conference 2021 , pages 2335-2346, 2021.

Sébastien Bubeck, Nicolo Cesa-Bianchi, et al. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends® in Machine Learning , 5(1):1-122, 2012.

Kechao Cai, Xutong Liu, Yu-Zhen Janice Chen, and John CS Lui. An online learning approach to network application optimization with guarantee. In IEEE INFOCOM 2018-IEEE Conference on Computer Communications , pages 2006-2014. IEEE, 2018.

Leonardo Cella and Massimiliano Pontil. Multi-task and meta-learning with sparse linear bandits. In Uncertainty in Artificial Intelligence , pages 1692-1702. PMLR, 2021.

Leonardo Cella, Alessandro Lazaric, and Massimiliano Pontil. Meta-learning with stochastic linear bandits. In International Conference on Machine Learning , pages 1360-1370. PMLR, 2020.

Leonardo Cella, Karim Lounici, Grégoire Pacreau, and Massimiliano Pontil. Multi-task representation learning with stochastic linear bandits. In International Conference on Artificial Intelligence and Statistics , pages 4822-4847. PMLR, 2023.

Wei Chu, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pages 208-214. JMLR Workshop and Conference Proceedings, 2011.

Aniket Anand Deshmukh, Urun Dogan, and Clay Scott. Multi-task learning for contextual bandits. Advances in neural information processing systems , 30, 2017.

Dylan J Foster, Claudio Gentile, Mehryar Mohri, and Julian Zimmert. Adapting to misspecification in contextual bandits. Advances in Neural Information Processing Systems , 33: 11478-11489, 2020.

Claudio Gentile, Shuai Li, and Giovanni Zappella. Online clustering of bandits. In International Conference on Machine Learning , pages 757-765. PMLR, 2014.

Claudio Gentile, Shuai Li, Purushottam Kar, Alexandros Karatzoglou, Giovanni Zappella, and Evans Etrue. On context-dependent clustering of bandits. In International Conference on machine learning , pages 1253-1262. PMLR, 2017.

Avishek Ghosh, Sayak Ray Chowdhury, and Aditya Gopalan. Misspecified linear bandits. In Thirty-First AAAI Conference on Artificial Intelligence , 2017.

Jens Hainmueller and Chad Hazlett. Kernel regularized least squares: Reducing misspecification bias with a flexible and interpretable machine learning approach. Political Analysis , 22(2): 143-168, 2014.

Negar Hariri, Bamshad Mobasher, and Robin Burke. Context adaptation in interactive recommender systems. In Proceedings of the 8th ACM Conference on Recommender Systems , pages 41-48, 2014.

F Maxwell Harper and Joseph A Konstan. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis) , 5(4):1-19, 2015.

Joey Hong, Branislav Kveton, Manzil Zaheer, and Mohammad Ghavamzadeh. Hierarchical bayesian bandits. In International Conference on Artificial Intelligence and Statistics , pages 7724-7741. PMLR, 2022.

Ruiquan Huang, Weiqiang Wu, Jing Yang, and Cong Shen. Federated linear contextual bandits. Advances in neural information processing systems , 34:27057-27068, 2021.

Pushmeet Kohli, Mahyar Salek, and Greg Stoddard. A fast bandit algorithm for recommendation to users with heterogenous tastes. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 27, pages 1135-1141, 2013.

Fang Kong, Canzhe Zhao, and Shuai Li. Best-of-three-worlds analysis for linear bandits with follow-the-regularized-leader algorithm. arXiv preprint arXiv:2303.06825 , 2023.

Tor Lattimore and Csaba Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Tor Lattimore, Csaba Szepesvari, and Gellert Weisz. Learning with good feature representations in bandits and in rl with a generative model. In International Conference on Machine Learning , pages 5662-5670. PMLR, 2020.

Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web , pages 661-670, 2010.

Shuai Li and Shengyu Zhang. Online clustering of contextual cascading bandits. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Shuai Li, Alexandros Karatzoglou, and Claudio Gentile. Collaborative filtering bandits. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval , pages 539-548, 2016.

Shuai Li, Wei Chen, Shuai Li, and Kwong-Sak Leung. Improved algorithm on online clustering of bandits. In Proceedings of the 28th International Joint Conference on Artificial Intelligence , IJCAI'19, page 2923-2929. AAAI Press, 2019. ISBN 9780999241141.

Xutong Liu, Haoru Zhao, Tong Yu, Shuai Li, and John Lui. Federated online clustering of bandits. In The 38th Conference on Uncertainty in Artificial Intelligence , 2022.

Xutong Liu, Jinhang Zuo, Siwei Wang, John CS Lui, Mohammad Hajiesmaili, Adam Wierman, and Wei Chen. Contextual combinatorial bandits with probabilistically triggered arms. In International Conference on Machine Learning , pages 22559-22593. PMLR, 2023.

Xutong Liu, Jinhang Zuo, Hong Xie, Carlee Joe-Wong, and John CS Lui. Variance-adaptive algorithm for probabilistic maximum coverage bandits with general feedback. In IEEE INFOCOM 2023-IEEE Conference on Computer Communications , pages 1-10. IEEE, 2023.

Aldo Pacchiano, My Phan, Yasin Abbasi Yadkori, Anup Rao, Julian Zimmert, Tor Lattimore, and Csaba Szepesvari. Model selection in contextual stochastic bandit problems. Advances in Neural Information Processing Systems , 33:10328-10337, 2020.

Chengshuai Shi and Cong Shen. Federated multi-armed bandits. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 9603-9611, 2021.

Marta Soare, Ouais Alsharif, Alessandro Lazaric, and Joelle Pineau. Multi-task linear bandits. In NIPS2014 workshop on transfer and multi-task learning: theory meets practice , 2014.

Runzhe Wan, Lin Ge, and Rui Song. Metadata-based multi-task bandits with bayesian hierarchical models. Advances in Neural Information Processing Systems , 34:29655-29668, 2021.

Runzhe Wan, Lin Ge, and Rui Song. Towards scalable and robust structured bandits: A metalearning framework. In International Conference on Artificial Intelligence and Statistics , pages 1144-1173. PMLR, 2023.

Zhi Wang, Chicheng Zhang, Manish Kumar Singh, Laurel Riek, and Kamalika Chaudhuri. Multitask bandit learning through heterogeneous feedback aggregation. In International Conference on Artificial Intelligence and Statistics , pages 1531-1539. PMLR, 2021.

Zhi Wang, Chicheng Zhang, and Kamalika Chaudhuri. Thompson sampling for robust transfer in multi-task bandits. arXiv preprint arXiv:2206.08556 , 2022.

Zhiyong Wang, Xutong Liu, Shuai Li, and John CS Lui. Efficient explorative key-term selection strategies for conversational contextual bandits. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 10288-10295, 2023.

Zhiyong Wang, Jize Xie, Tong Yu, Shuai Li, and John Lui. Online corrupted user detection and regret minimization. arXiv preprint arXiv:2310.04768 , 2023.

Junda Wu, Canzhe Zhao, Tong Yu, Jingyang Li, and Shuai Li. Clustering of conversational bandits for user preference learning and elicitation. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management , pages 2129-2139, 2021.

Qingyun Wu, Huazheng Wang, Quanquan Gu, and Hongning Wang. Contextual bandits in a collaborative environment. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval , pages 529-538, 2016.

Shi Zong, Hao Ni, Kenny Sung, Nan Rosemary Ke, Zheng Wen, and Branislav Kveton. Cascading bandits for large-scale recommendation problems. arXiv preprint arXiv:1603.05359 , 2016.

Appendix

A More Discussions on Related Work

In this section, we will give more comparisions and discussions on some previous works that are related to our work to some extent.

There are some other works on bandits leveraging user (or task) relations, which have some relations with the clustering of bandits (CB) works to some extent, but are in different lines of research from CB, and are quite different from our work. First, besides CB, the work [41] also leverages user relations. Specifically, it utilizes a known user adjacency graph to share context and payoffs among neighbors, whereas in CB, the user relations are unknown and need to be learnt, thus the setting differs a lot from CB. Second, there are lines of works on multi-task learning [6, 10, 33, 8, 37, 36], meta-learning [35, 18, 7] and federated learning [32, 19], where multiple different tasks are solved jointly and share information. Note that all of these works do not assume an underlying unknown user clustering structure which needs to be inferred by the agent to speed up learning. For works on multi-task learning [6, 10, 33, 8, 37, 36], they assume the tasks are related but no user clustering structures, and to the best of our knowledge, none of them consider model misspefications, thus differing a lot from ours. For some recent works on meta-learning [35, 18, 34], they propose general Bayesian hierarchical models to share knowledge across tasks, and design Thompson-Samplingbased algorithms to optimize the Bayes regret, which are quite different from the line of CB works, and differ a lot from ours. And additionally, as supported by the discussions in the works [7, 36], multi-task learning and meta-learning are different lines of research from CB. For the works on federated learning [32, 19], they consider the privacy and communication costs among multiple servers, whose setting is also very different from the previous CB works and our work.

Remark. Again, we emphasize that the goal of this work is to initialize the study of the important CBMUM problem, and propose general design ideas for dealing with model misspecifications in CB problems. Therefore, our study is based on fundamental models on CB [12, 27] and MLB [23], and the algorithm design ideas and theoretical analysis are pretty general. We leave incorporating the more recent model selection methods [31, 11] into our framework to address the unknown exact maximum model misspecification level as an interesting future work. It would also be interesting to consider incorporating our methods and ideas of tackling model misspecifications into the studies of multi-task learning, meta learning and federated learning.

B More Discussions on Assumptions

All the assumptions (Assumptions 3.1,3.2,3.3,3.4)in this work are natural and basically follow (or less strigent than) previous works on CB and MLB [12, 25, 27, 28, 23].

B.1 Less Strigent Assumption on on the Generating Distribution of Arm Vectors

We also make some contributions to relax a widely-used but stringent assumption on the generating distribution of arm vectors. Specifically, our Assumption 3.3 on item regularity relaxes the previous one used in previous CB works [12, 25, 27, 28] by removing the condition that the variance should be upper bounded by λ 2 8 log(4 |A t | ) . For technical details on this, please refer to the theoretical analysis and discussions in Appendix J.

B.2 Discussions on Assumption 3.4 about Bounded Misspecification Level

This assumption follows [23]. Note that this ϵ ∗ can be an upper bound on the maximum misspecification level, not the exact maximum itself. In real-world applications, the deviations are usually small [14], and we can set a relatively big ϵ ∗ (e.g., 0.2) to be the upper bound. Our experimental results support this claim. As shown in our experimental results on real-data case 2, even when ϵ ∗ is unknown, our algorithms still perform well by setting ϵ ∗ = 0 . 2 . Some recent studies [31, 11] use model selection methods to theoretically deal with unknown exact maximum misspecification level in the single-user case, which is not the emphasis of this work. Additionally, the work [11] assumes that the learning agent has access to a regression oracle. And for the work [31], though their regret bound

is dependent on the exact maximum misspecification level that needs not to be known by the agent, an upper bound of the exact maximum misspecification level is still needed. We leave incorporating their methods to deal with unknown exact maximum misspecification level as an interesting future work.

B.3 Discussions on Assumption 3.2 about the Theoretical Results under General User Arrival Distributions

The uniform arrival in Assumption 3.2 follows previous CB works [12, 25, 28], it only affects the T 0 term, which is the time after which the algorithm maintains a 'good partition' and is of O ( u log T ) . For an arbitrary arrival distribution, T 0 becomes O (1 /p min log T ) , where p min is the minimal arrival probability of a user. And since it is a lower-order term (of O (log T ) ), it will not affect the main order of our regret upper bound which is of O ( ϵ ∗ T √ md log T + d √ mT log T ) . The work [27] studies arbitrary arrivals and aims to remove the 1 /p min factor in this term, but their setting is different. They make an additional assumption that users in the same cluster not only have the same preference vector, but also the same arrival probability, which is different from our setting and other classic CB works [12, 25, 28] where we only assume users in the same cluster share the same preference vector.

C Highlight of the Theoretical Analysis

Our proof flow and methodologies are novel in clustering of bandits (CB), which are expected to inspire future works on model misspecifications and CB. The main challenge of the regret analysis in CBMUM is that due to the estimation inaccuracy caused by misspecifications, it is impossible to cluster all users exactly correctly, and it is highly non-trivial to bound the regret caused by 'misclustering" ζ -close users .

To the best of our knowledge, the common proof flow of previous CB works (e.g., [12, 25, 28]) can be summarized in two steps: The first is to prove a sufficient time T ′ 0 after which the algorithms can cluster all users exactly correctly with high probability. Note that the inferred clustering structure remains static after T ′ 0 , making the analysis easy. Second, after the correct static clustering , the regret can be trivially bounded by bounding m (number of underlying clusters) independent linear bandit algorithms, resulting in a O ( d √ mT log T ) regret.

The above common proof flow is straightforward in CB with perfectly linear models, but it would fail to get a non-vacuous regret bound for CBMUM. In CBMUM, it is impossible to learn an exactly correct static clustering structure with model misspecifications. In particular, we prove that we can only expect the algorithm to cluster ζ -close users together rather than cluster all users exactly correctly. Therefore, the previous flow can not be applied to the more challenging CBMUM problem.

We do the following to address the challenges in obtaining a tight regret bound for CBMUM. With the carefully-designed novel key components of RCLUMB, we can prove a sufficient time T 0 after which RCLUMB can get a 'good partition" (Definition 5.5) with high probability, which means the cluster V t assigned to i t contains all users in the same ground-truth cluster as i t , and possibly some other i t 's ζ -close users. Intuitively, after T 0 , the algorithm can leverage all the information from the users' ground-truth clusters but may misuse some information from other ζ -close users with preference gaps up to ζ , causing a regret of 'misclustering" ζ -close users . It is highly non-trivial to bound this part of regret, and the proof methods would be beneficial for future studies in CB in challenging cases when it is impossible to cluster all users exactly correctly. For details, please refer to the discussions '(ii) Bounding the term of misclustering it's ζ -close users" in Section 5, the key Lemma 5.7 (Bound of error caused by misclustering), its proof and tightness discussion in Appendix G. Also, a more subtle analysis is needed to handle the time-varying inferred clustering structure since the 'good partition" may change over time, whereas in the previous CB works, the clustering structure remains static after T ′ 0 . For theoretical details on this, please refer to Appendix E.

D Discussions on why Trivially Combining Existing CB and MLB Works Could Not Achieve a Non-vacuous Regret Upper Bound

We consider discussing regret upper bounds for CB without considering misspecifications for three cases: (1) neither the clustering process nor the decision process considers misspecifications (previous

CBalgorithms); (2) the decision process does not consider misspecifications; (3) the clustering process does not consider misspecifications.

For cases (1) and (2), the decision process could contribute to the leading regret. We consider the case where there are m underlying clusters, with each cluster's arrival being T/m , and the agent knows the underlying clustering structure. For this case, there exist some instances where the regret upper bound R ( T ) is strictly larger than ϵ ∗ T √ m log T asymptotically in T . Formally, in the discussion of 'Failure of unmodified algorithm" in Appendix E in [23], they give an example to show that in the single-user case, the regret R 1 ( T ) of the classic linear bandit algorithms without considering misspecifications will have:

lim

R

1

(

T

m

T

→

+

∞

ϵ

∗

T

)

log

T

clusters, even if we know the underlying clustering structure and keep m independent linear bandit algorithms with T i for the cluster i ∈ [ m ] to leverage the common information of clusters, the best we can get is R 2 ( T ) = ∑ i ∈ [ m ] R 1 ( T i ) . By the above results, if the decision process does not consider misspecifications, we have lim T → + ∞ R 2 ( T ) ϵ ∗ T √ m log T = lim T → + ∞ mR 1 ( T/m ) ϵ ∗ T √ m log T = + ∞ . Recall that the √ √

we have lim T → + ∞ ϵ ∗ T √ m log T < + ∞ ), which gives a proof that that the regret upper bound of our proposed algorithms is asymptotically much better than CB algorithms in cases (1)(2).

regret upper bound R ( T ) of our proposed algorithms is of O ( ϵ ∗ T md log T + d mT log T ) (thus, R ( T )

For case (3), if the clustering process does not use the more tolerant deletion rule in Line 10 of Algo.1, the gap between users linked by edges would possibly exceed ζ ( ζ = 2 ϵ ∗ √ 2 ˜ λ x ) even after T 0 , which √

will result in a regret upper bound no better than O ( ϵ ∗ u dT ) . As the number of users u is usually huge in practice, this result is vacuous. The reasons for getting the above claim are as follows. Even if the clustering process further uses our deletion rule considering misspecifications, and the users linked by edges are within ζ distance, failing to extract 1 -hop users (Line 5 in Algo.1) would cause the leading O ( ϵ ∗ u √ dT ) regret term, as in the worst case, the preference vector θ of the user in ˜ V t who is h -hop away from user i t could deviate by hζ from θ i t , where h can be as large as u , and it would make the second term in Eq.(8) a O ( ϵ ∗ u √ dT ) term. If we completely do not consider the misspecifications in the clustering process, the above user gap between users linked by edges would possibly exceed ζ , which will cause a regret upper bound worse than O ( ϵ ∗ u √ dT ) .

E Proof of Theorem 5.3

We first prove the result in the case when γ 1 defined in Definition 5.1 is not infinity, i.e., 4 ϵ ∗ √ 2 ˜ λ x < γ 1 < ∞ . The proof of the special case when γ 1 = ∞ will directly follow the proof of this case. For the instantaneous regret R t at round t , with probability at least 1 -5 δ for some δ ∈ (0 , 1 5 ) , at ∀ t ≥ T 0 :

where the last inequality holds by the UCB arm selection strategy in Eq.(5), the concentration bound given in Lemma 5.6, and the fact that ∥ ∥ ϵ i,t ∥ ∥ ∞ ≤ ϵ ∗ , ∀ i ∈ U , ∀ t .

We define the following events. Let

E 0 = { R t ≤ 2 C a t ,t + 2 ϵ ∗ √ 2 d ˜ λ 3 2 x I { V t / ∈ V} +2 ϵ ∗ , for all { t : t ≥ T 0 , and the algorithm maintains a 'good partition" at t }} ,

E 1 = { the algorithm maintains a 'good partition" for all t ≥ T 0 } ,

E

=

E

0

∩ E

1

.

√

∞

= +

.

In our problem with multiple users and

m

underlying

P ( E 0 ) ≥ 1 -2 δ . According to Lemma H.1, P ( E 1 ) ≥ 1 -3 δ . Thus, P ( E ) ≥ 1 -5 δ for some δ ∈ (0 , 1 5 ) . Take δ = 1 T , we can get that

where E denotes the complementary event of E , I {E} R ( T ) denotes R ( T ) under event E , I {E} R ( T ) denotes R ( T ) under event E , and we use R ( T ) ≤ T to bound R ( T ) under event E .

Then it remains to bound I {E} R ( T ) :

where Eq.(12) follows from Eq.(10). Eq.(13) holds since under Assumption 3.2 about user arrival uniformness and by Definition 5.5 of 'good partition", P ( I {E , V t / ∈ V} ) ≤ ˜ u u , ∀ t ≥ T 0 , where ˜ u is defined in Definition 5.2.

Then we need to bound

T

where we use m t to denote the number of connected components partitioned by the algorithm at t , ˜ V ′ t,k , k ∈ [ m t ] to denote the connected components partitioned by the algorithm at t , V ′ t,k ⊆ ˜ V ′ t,k to denote the subset extracted to be the cluster V t for i t from ˜ V ′ t,k conditioned on i t ∈ ˜ V ′ t,k , and T V j ,T to denote the number of times that the served users lie in the ground-truth cluster V j up to time T , i.e., T V j ,T = ∑ t ∈ [ T ] I { i t ∈ V j } .

The reasons for having Eq.(15) are as follows. Under event E , the algorithm will always have a 'good partition" after T 0 . By Definition 5.5 and the proof process of Lemma H.1 about the edge deletion conditions, we can get m t ≤ m and if i t ∈ ˜ V ′ t,k , i t ∈ V j , then V j ⊆ V ′ t,k since V ′ t,k contains V j and possibly other ground-truth clusters V n , n ∈ [ m ] , whose preference vectors are ζ -close to θ j . Therefore, by the definition of the regularized Gramian matrix, we can get M V ′ t,k ,t -1 ⪰ M V j ,t -1 , ∀ t ≥ T 0 +1 . Thus by the above reasoning, ∑ m t k =1 I { i t ∈ ˜ V ′ t,k } ∥ x a t ∥ M -1 V ′ t,k ,t -1 ≤ ∑ m j =1 I { i t ∈ V j } ∥ x a t ∥ M -1 V j ,t -1 , ∀ t ≥ T 0 +1 . Eq.(16) holds by the Cauchy-Schwarz inequality; Eq.(17) follows by the following technical Lemma J.2. Eq.(18) is from the Cauchy-Schwarz inequality and the fact that ∑ m j =1 T V j ,T = T .

We then bound the last term in Eq.(14):

where Eq.(19), Eq.(21) and Eq.(23) hold because of the Cauchy-Schwarz inequality, Eq.(20) holds since M V ′ t,k ,t -1 ⪰ ∑ s ∈ [ t -1] i s ∈ V ′ t,k x a s x ⊤ a s , Eq.(22) is because T V ′ t,k ,t -1 ≤ T , Eq. (24) follows from the same reasoning as Eq.(15), and Eq.(25) comes from the following technical Lemma J.2.

Finally, plugging Eq.(18) and Eq.(25) into Eq.(14), take expectation and plug it into Eq.(13), we can get:

where

is given in the following Lemma H.1 in Appendix H.

F Proof and Discussions of Theorem 5.4

In the work [23], they give a lower bound for misspecified linear bandits with a single user. The lower bound of R ( T ) is given by: R 3 ( T ) ≥ ϵ ∗ T √ d . Therefore, suppose our problem with multiple users and m underlying clusters where the arrival times are T i for each cluster, then for any algorithms, even if they know the underlying clustering structure and keep m independent linear bandit algorithms to leverage the common information of clusters, the best they can get is R ( T ) = ∑ i ∈ [ m ] R 3 ( T i ) ≥ ϵ ∗ ∑ i ∈ [ m ] T i √ d = ϵ ∗ T √ d , which gives a lower bound of O ( ϵ ∗ T √ d ) for the CBMUM problem. Recall that the regret upper bound of our algorithms is of O ( ϵ ∗ T √ md log T + d √ mT log T ) , asymptotically matching this lower bound with respect to T up to logarithmic factors and with respect to m up to O ( √ m ) factors, showing the tightness of our theoretical results (where m are typically very small for real applications).

We conjecture that the gap for the m factor is due to the strong assumption that cluster structures are known to prove our lower bound, and whether there exists a tighter lower bound will be left for future work.

G Proof of the key Lemma 5.7

In Lemma 5.7, we want to bound the term ∣ ∣ ∣ ∣ x ⊤ a M -1 V t ,t -1 ∑ s ∈ [ t -1] i s ∈ V t x a s x ⊤ a s ( θ i s -θ i t ) ∣ ∣ ∣ ∣ . By the definition of 'good partition", we have ∥ θ i s -θ i t ∥ 2 ≤ ζ , ∀ i s ∈ V t . It is an easy-to-be-made mistake to directly drag ∥ θ i s -θ i t ∥ 2 out to upper bound it by ∥ ∥ ∥ ∥ x ⊤ a M -1 V t ,t -1 ∑ s ∈ [ t -1] i s ∈ V t x a s x ⊤ a s ∥ ∥ ∥ ∥ 2 × ζ and then proceed. We need more careful analysis.

We first prove the following general lemma.

Lemma G.1. For vectors x 1 , x 2 , . . . , x k ∈ R d , ∥ x i ∥ 2 ≤ 1 , ∀ i ∈ [ k ] , and vectors θ 1 , θ 2 , . . . , θ k ∈ R d , ∥ θ i ∥ 2 ≤ C, ∀ i ∈ [ k ] , where C > 0 is a constant, we have:

Proof. Let X ∈ R d × k be a matrix such that it has x i s as its columns, i.e., X = [ x 1 , . . . , x k ] =

Let y ∈ R k × 1 be a vector that has x ⊤ i θ i s as its elements, i.e., y = [ x ⊤ 1 θ 1 , . . . , x ⊤ k θ k ] ⊤ . Then we have:

where Eq. (27) follows by the matrix operator norm inequality, Eq. (28) follows by the Cauchy-Schwarz inequality, Eq. (29) follows by ∥ X ∥ F ≤ √ d ∥ X ∥ 2 , Eq. (30) follows from ∥ X ∥ 2 2 = ∥ ∥ ∥ XX ⊤ ∥ ∥ ∥ 2 .

The above result is tight. We can show that the lower bound of ∥ ∥ ∥ ∑ k i =1 x i x ⊤ i θ i ∥ ∥ ∥ 2 under the conditions in the lemma is exactly C √ d ∥ ∥ ∥ ∑ k i =1 x i x ⊤ i ∥ ∥ ∥ 2 . Specifically, let k = 2 , C = 1 , d = 2 , x 1 = [0 , 1] ⊤ , x 2 = [1 , 0] ⊤ , θ 1 = [1 , 0] ⊤ , θ 2 = [0 , 1] ⊤ , then we have ∥ ∥ ∥ ∑ 2 i =1 x i x ⊤ i θ i ∥ ∥ ∥ 2 = ∥ ∥ [1 , 1] ⊤ ∥ ∥ 2 = √ 2 , and C √ d ∥ ∥ ∥ ∑ 2 i =1 x i x ⊤ i ∥ ∥ ∥ 2 = 1 × √ 2 × ∥ ∥ ∥ ∥ [ 1 0 0 1 ]∥ ∥ ∥ ∥ 2 = √ 2 . Therefore, we have that the upper bound given in Lemma G.1 matches the lower bound.

We are now ready to prove the key Lemma 5.7 with the above Lemma G.1.

At any t ≥ T 0 , if the current partition is a 'good partition", and V t / ∈ V , then for all x a ∈ R d , ∥ x a ∥ 2 ≤ 1 , with probability at least 1 -δ :

where Eq.(32) follows by the Cauchy-Schwarz inequality, Eq.(33) follows from the inequality of matrix's operator norm, Eq.(34) follows from the fact that in a 'good partition", ∥ θ i t -θ l ∥ 2 ≤ 2 ϵ ∗ √ 2 ˜ λ x , ∀ l ∈ V t and Lemma G.1, Eq.(35) follows by Eq.(47) with probability ≥ 1 -δ .

H Lemma H.1 of the sufficient time T 0 and its proof

The following lemma gives a sufficient time T 0 for the algorithm to get a 'good partition".

Lemma H.1. With the carefully designed edge deletion rule, after

rounds, with probability at least 1 -3 δ for some δ ∈ (0 , 1 3 ) , RCLUMB can always get a 'good partition".

Below is the detailed proof of Lemma H.1.

Proof. We first prove the following result: With probability at least 1 -δ for some δ ∈ (0 , 1) , at any t ∈ [ T ] :

where β ( T i,t , δ u ) ≜ √ λ + √ 2 log( u δ ) + d log(1 + T i,t λd ) .

where we denote ˜ M i,t = M i,t + λ I , and Eq.(37) holds by definition. Therefore,

We then bound the three terms in Eq.(38) one by one. For the first term:

where we use the Cauchy-Schwarz inequality, the inequality for the operator norm of matrices, and the fact that λ min ( ˜ M i,t ) ≥ λ .

For the second term in Eq.(38):

where we denote S d -1 = { x ∈ R d : ∥ x ∥ 2 = 1 } , Eq.(40) follows from Holder's inequality, Eq.(41) follows by the Cauchy-Schwarz inequality, Eq.(42) holds because ˜ M i,t ⪰ ∑ s ∈ [ t ] i s = i x a s x ⊤ a s , Eq.(43) follows from the Courant-Fischer theorem.

For the last term in Eq.(38)

where Eq.(44) follows by the Cauchy-Schwarz inequality and the inequality for the operator norm of matrices, and Eq.(45) follows by the Courant-Fischer theorem.

Following Theorem 1 in [1], with probability at least 1 -δ for some δ ∈ (0 , 1) , for any i ∈ U , we have:

where det ( M ) denotes the determinant of matrix M , Eq.(46) is because det ( ˜ M i,t ) ( trace ( λ I + ∑ s ∈ [ t ] i s = i x as x ⊤ as ) d ) d ≤ ( λd + T i,t d ) d , and det ( λ I ) = λ d .

Plugging Eq.(46) into Eq. (45), then plugging Eq. (39), Eq.(43) and Eq.(45) into Eq.(38), we can get that Eq.(73) holds with probability ≥ 1 -δ .

Then, with the item regularity assumption stated in Assumption 3.3, the technical Lemma J.1, together with Lemma 7 in [25], with probability at least 1 -δ , for a particular user i , at any t such that T i,t ≥ 16 ˜ λ 2 x log( 8 d ˜ λ 2 x δ ) , we have:

Based on the above reasoning, we have: if T i,t ≥ 16 ˜ λ 2 x log( 8 d ˜ λ 2 x δ ) , then with probability ≥ 1 -2 δ , we have:

for any i ∈ U .

Let

which is equivalent to

where γ 1 is given in Definition 5.1.

Assume λ ≤ 2 log( u δ ) + d log(1 + T i,t λd ) , which is typically held, then a sufficient condition for Eq. (50) is:

To satisfy the condition in Eq.(51), it is sufficient to show

and

From Eq.(52), we can get:

Following Lemma 9 in [25], we can get the following sufficient condition for Eq.(53):

Assume u δ ≥ 4 λ ˜ λ x ( γ 1 4 -ϵ ∗ √ 1 2 ˜ λx ) 2 , which is typically held, we can get that

is a sufficient condition for Eq.(49). Together with the condition that T i,t ≥ 16 ˜ λ 2 x log( 8 d ˜ λ 2 x δ ) , we can get that if

then with probability ≥ 1 -2 δ :

By Lemma 8 in [25], and Assumption 3.2 of user arrival uniformness, we have that for all

with probability at least 1 -δ , condition in Eq.(57) is satisfied.

Therefore we have that for all t ≥ T 0 , with probability ≥ 1 -3 δ :

Next, we show that with Eq.(59), we can get that the RCLUMB keeps a 'good partition". First, if we delete the edge ( i, l ) , then user i and user j belong to different ground-truth clusters , i.e., ∥ θ i -θ l ∥ 2 > 0 . This is because by the deletion rule of the algorithm, the concentration bound,

From the above reasoning, we can get that at round t , any user within V t is ζ -close to i t , and all the users belonging to V j ( i ) are contained in V t , which means the algorithm has done a 'good partition" at t by Definition 5.5.

I Proof of Lemma 5.6

We prove the result in two situations: when V t ∈ V and when V t / ∈ V .

(1) Situation 1: for any t ≥ T 0 and V t ∈ V , which means that the current user i t is clustered completely correctly, i.e., V t = V j ( i t ) , therefore θ l = θ i t , ∀ l ∈ V t , then we have:

Therefore we have

Next, we bound the three terms in Eq.(60). For the first term:

where we use the inequality of matrix norm, the Cauchy-Schwarz inequality, ∥ θ i t ∥ 2 ≤ 1 , and the fact that λ max ( M -1 V t ,t -1 ) = 1 λ min ( M V t ,t -1 ) ≤ 1 λ .

For the second term in Eq.(60):

where in the second inequality we use the Holder's inequality. For the last term, with probability at least 1 -δ :

where the second inequality follows by Theorem 1 in [1], Eq.(64) is because det ( M V t ,t -1 ) ≤ ( trace ( λ I + ∑ s ∈ [ t ] i s ∈ V t x as x ⊤ as ) d ) d ≤ ( λd + T V t ,t d ) d ≤ ( λd + T d ) d , and det ( λ I ) = λ d .

Plugging Eq.(61), Eq.(62) and Eq.(64) into Eq.(60), we can prove Lemma 5.6 in situation 1, i.e., for any t ≥ T 0 and V t ∈ V , with probability at least 1 -δ :

̸

(2) Situation 2: for any t ≥ T 0 and V t / ∈ V , which means that the current user is misclustered by the algorithm, i.e., V t = V j ( i t ) , but with Lemma H.1, with probability at least 1 -3 δ , the current

Thus, with Lemma 5.7 and with the previous reasoning, with probability at least 1 -5 δ , we have:

Therefore, combining situation 1 and situation 2, the result of Lemma 5.6 then follows.

J Technical Lemmas and Their Proofs

We first prove the following technical lemma which is used to prove Lemma H.1.

Lemma J.1. Under Assumption 3.3, at any time t , for any fixed unit vector θ ∈ R d

Proof. The proof of this lemma mainly follows the proof of Claim 1 in [12], but with more careful analysis, since their assumption is more stringent than ours.

Denote the feasible arms at round t by A t = { x t, 1 , x t, 2 , . . . , x t, |A t | } . Consider the corresponding i.i.d. random variables θ i = ( θ ⊤ x t,i ) 2 -E t [( θ ⊤ x t,i ) 2 | |A t | ] , i = 1 , 2 , . . . , |A t | . By Assumption 3.3, θ i s are sub-Gaussian random variables with variance bounded by σ 2 . Therefore, we have that

for any α > 0 and any i ∈ [ |A t | ] :

where P t ( · ) is the shorthand for the conditional probability P ( ·| ( i 1 , A 1 , r 1 ) , . . . , ( i t -1 , A t -1 , r t -1 ) , i t ) .

We also have that E t [( θ ⊤ x t,i ) 2 | |A t | = E t [ θ ⊤ x t,i x ⊤ t,i θ | |A t | ] ≥ λ min ( E x ∼ ρ [ xx ⊤ ]) ≥ λ x by Assumption 3.3. With the above inequalities, we can get

where C is the upper bound of |A t | .

Therefore, we have

Finally, we prove the following lemma which is used in the proof of Theorem 5.3.

Proof.

K Algorithms of RSCLUMB

This section introduces the Robust Set-based Clustering of Misspecified Bandits Algorithm (RSCLUMB). Unlike RCLUMB, which maintains a graph-based clustering structure, RSCLUMB maintains a set-based clustering structure. Besides, RCLUMB only splits clusters during the learning process, while RSCLUMB allows both split and merge operations. A brief illustration is that the agent will split a user out of its current set(cluster) if it finds an inconsistency between the user and its set, and if there are two clusters whose estimated preferences are close enough, the agent will merge them. A detailed discussion of the connection between the graph structure and the set structure can be found in [27].

Now we introduce the details of RSCLUMB. The algorithm first initializes a single set S 1 containing all users and updates it during the learning process. The whole learning process consists of phases (Algo. 2 Line 3), where the s -th phase contains 2 s rounds. At the beginning of each phase, the agent marks all users as "unchecked", and if a user comes later, it will be marked as "checked". If all users in a cluster are checked, then this cluster will be marked as "checked" meaning it is an accurate cluster in the current phase. With this mechanism, every phase can maintain an accuracy level, and the agent can put the accurate clusters aside and focus on exploring the inaccurate ones. For each cluster V j , the algorithm maintains two estimated vectors ˆ θ V j and ˜ θ V j , where the ˆ θ V j is similar to the ˆ θ V j in RCLUMB and is used for the recommendation, while the ˜ θ V j is the average of all the estimated user preference vectors in this cluster and is used for the split and merge operations.

At time t in phase s , the user i τ comes with the item set D τ , where τ represents the index of total time steps. Then the algorithm determines the cluster and makes a cluster-based recommendation. This process is similar to RCLUMB. After updating the information (Algo. 2 Line12), the agent checks if a split or a merge is possible (Algo. 2 Line13-17).

By our assumption, users in the same cluster have the same vectors. So a cluster can be regarded as a good cluster only when all the estimated user vectors are close to the estimated cluster vector. We call a user is consistent with the cluster if their estimated vectors are close enough. If a user is inconsistent with its current cluster, the agent will split it out. Two clusters are consistent when their estimated vectors are close, and the agent will merge them.

RSCLUMBmaintains two sets of estimated cluster vectors: (i) cluster-level estimation with integrated user information, which is for recommendations (Line 12 and Line 10 in Algo.2); (ii) the average of estimated user vectors, which is used for robust clustering (Line 3 in Algo.3 and Line 2 in Algo.4). The previous set-based CB work [27] only uses (i) for both recommendations and clustering, which would lead to erroneous clustering under misspecifications, and cannot get any non-vacuous regret bound in CBMUM.

L Main Theorem and Lemmas of RSCLUMB

Theorem L.1 (main result on regret bound for RSCLUMB) . With the same assumptions in Theorem 5.3, the expected regret of the RSCLUMB algorithm for T rounds satisfies:

Lemma L.2. For RSCLUMB, we use T 1 to represent the corresponding T 0 of RCLUMB. Then :

Algorithm 2 Robust Set-based Clustering of Misspecified Bandits Algorithm (RSCLUMB)

1: Input: Deletion parameter α 1 , α 2 > 0 , f ( T ) = √ 1+ln(1+ T ) 1+ T , λ, β, ϵ ∗ > 0 .

2: Initialization:

M i, 0 = 0 d × d , b i, 0 = 0 d × 1 , T i, 0 = 0 , ∀ i ∈ U ;

Initialize the set of cluster indexes by J = { 1 } and the single cluster S 1 by M 1 = 0 d × d , b 1 = 0 d × 1 , T 1 = 0 , C 1 = U , j ( i ) = 1 , ∀ i .

3: for all s = 1 , 2 , . . . do

4: Mark every user unchecked for each cluster.

5: For each cluster V j , compute ˜ T V j = T V j , ˆ θ V j = ( λ I + M V j ) -1 b V j , ˜ θ V j = ∑ i ∈ V j ˆ θ i [ V j ]

6: for all t = 1 , 2 , . . . , T do

7: Compute τ = 2 s -2 + t

8: Receive the user i τ and the decision set D τ

9: Determine the cluster index j = j ( i τ )

10: Recommend item a τ with the largest UCB index as shown in Eq. (5)

11: Received the feedback r τ .

12: Update the information:

13: if i τ is unchecked then

14: Run Split

15: Mark user i τ has been checked

16: Run Merge

Algorithm 3 Split

Define F ( T ) = √ 1+ln(1+ T ) 1+ T

3: Split user i τ from cluster V j and form a new cluster V ′ j of user i τ

Algorithm 4 Merge

1: for any two checked clusters V j 1 , V j 2 satisfying

3: Set j ( i ) = j 1 , ∀ i ∈ j 2 , delete V j 2

M Proof of Lemma L.3

where the last inequality holds due to the fact ∥ x a ∥ ≤ 1 and the condition of "split" and "merge". For | x T a ( ˆ θ V t ,t -θ V t ) | :

Thus, with the same method in Lemma 5.7 but replace ζ = 4 ϵ ∗ √ 1 2 ˜ λ x with ζ 1 = 6 ϵ ∗ √ 1 2 ˜ λ x , and with the previous reasoning, with probability at least 1 -5 δ , we have:

do

2: Merge them:

The lemma can be concluded.

N Proof of Lemma L.2

With the analysis in the proof of Lemma H.1, with probability at least 1 -δ :

and the estimated error of the current cluster ∥ ∥ ∥ ˜ θ j ( i ) -θ j ( i ) ∥ ∥ ∥ also satisfies this inequality. For set-based clustering structure, to ensure for each user there is only one ζ -close cluster, we let:

By assuming λ < 2 log( u δ ) + d log(1 + T i,t λd ) , we can simplify it to

which can be proved by 2 log( u δ ) 2 ˜ λ x T i,t ≤ 1 8 ( γ 1 6 -ϵ ∗ √ 1 2 ˜ λ x ) 2 and d log(1+ T i,t λd ) 2 ˜ λ x T i,t ≤ 1 8 ( γ 1 6 -ϵ ∗ √ 1 2 ˜ λ x ) 2 . It's obvious that the former one can be satisfied by T i,t ≥ 8 log( u/δ ) ˜ λ x ( γ 1 6 -ϵ ∗ √ 1 / 2 ˜ λ x ) 2 . As for the latter one, by

[25] Lemma 9, we can get T i,t ≥ 8 d log( 16 ˜ λxλ ( γ 1 6 -ϵ ∗ √ 1 / 2 ˜ λx ) 2 4 ˜ λ x ( γ 1 6 -ϵ ∗ √ 1 / 2 ˜ λ x ) 2 . By assuming u δ ≥ 16 4 ˜ λ x λ ( γ 1 6 -ϵ ∗ √ 2 / 4 ˜ λ x ) 2 , the lemma is proved.

O Proof of Theorem L.1

After 2 T 1 rounds,in each phase, at most u times split operations will happen, we use u log( T ) to bound the regret generated in these rounds. Then in the remained rounds the cluster num will be no more than m .

For the instantaneous regret R t at round t , with probability at least 1 -2 δ for some δ ∈ (0 , 1 2 ) :

where the last inequality holds due to the UCB arm selection strategy, the concentration bound given in LemmaL.3 and the fact that ∥ ∥ ϵ i,t ∥ ∥ ∞ ≤ ϵ ∗ .

Define such events. Let:

From previous analysis, we can know that P ( E 2 ) ≥ 1 -3 δ and P ( E 3 ) ≥ 1 -2 δ , thus P ( E ′ ≥ 1 -5 δ ) . By taking δ = 1 T , we can get:

Now we need to bound 2 ∑ T 2 T 1 C a t . We already know that after 2 T 1 rounds, in each phase k after the first u rounds,there will be at most m clusters Consider phase k , for simplicity, ignore the fist u rounds. For the first term in C :

a t

For all phases:

Similarly, for the second term in C a t :

λd Thus the total regret can be bounded by:

0.05

0.1

0.2

0.3

0.4



0

50000

100000

150000

200000

250000



RSCLUMB

RCLUMB

RLinUCB-Ind

SCLUB

CLUB

(a) Known Misspecification Level

0.05

0.1

0.2

0.3

0.4



0

50000

100000

150000

200000

250000



RSCLUMB

RCLUMB

RLinUCB-Ind

(b) Unknown Misspecification Level

P More Experiments

For ablation study, we test our algorithms' performance under different scales of deviation. We test RCLUMB and RSCLUMB when ϵ ∗ = 0 . 05 , 0 . 1 , 0 . 2 , 0 . 3 and 0 . 4 in both misspecification level known and unknown cases. In the known case, we set ϵ ∗ according to the real misspecification level, and we compare our algorithms' performance to the baselines except LinUCB and CW-OFUL which perform worst; in the unknown case, we keep ϵ ∗ = 0 . 2 , and we compare our algorithms to RLinUCB-Ind as only it has the pre-spicified parameter ϵ ∗ among the baselines. The results are shown in Fig.2. We plot each algorithm's final cumulative regret under different misspecification levels. All the algorithms' performance get worse when the deviation gets larger, and our two algorithms always perform better than the baselines. Besides, the regrets in the unknown case are only slightly larger than the known case. These results can match our theoretical results and again show our algorithms' effectiveness, as well as verify that our algorithm can handle the unknown misspecification level.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes the clustering bandit problem with misspecified reward functions. It proposes the RCLUMB and RSCLUMB algorithms and their upper bounds match the state-of-the-art ones in some degenerate cases. Numerical results are provided to evaluate the performances of proposed algorithms.\n\n\n\n==================\n\nThe score is updated to 6 after rebuttal.\n\nSTRENGTHS:\n1. It is good to study the 'combination' of clustering bandits and misspecified bandits, as they are both realistic topics.\n2. The paper is well organized, and contributions are mostly well described. For instance, Lemma 5.7 is highlighted as a key result to bound the regret due to misclustering users.\n3. The intuition of algorithm design of RCLUMB is in great details and easy to understand. Especially, in Section 4, the steps Cluster Detection, Cluster-based Recommendation, Update User Statistic and Update User Statistics are well explained.\n4. Section 5 presents a clear description of the analytical analysis of RCLUMB. In detail, the 'Discussion and Comparison' and proof sckech for Theorem 5.3 are easy to follow.\n5. The proposed RCLUMB algorithm is evaluated by numerical experiments in both synthetic and real data sets.\n\nWEAKNESSES:\nOverall, I believe this work is challenging and it provides a set of meaningful results and I still here is some possible venues for improvement:\n1. It is stated that the CB algorithms which were designed for bandits without misspecification will cause a large regret. I believe it is true with high probability and some discussion is provided in Remark 1, but it would be appreciated if an upper bound on the regret of them in bandits with misspecification can be presented in the paper.\n2. A lower bound may help us to learn whether the proposed algorithms are efficient, but is not provided in the work.\n\nQUESTIONS:\nPlease refer to the *Weakness* section for questions/suggestions.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the contextual linear bandit setup with model misspecification in the framework of clustering bandits. This investigation revolves around a particular model misspecification framework, aiming to address the challenge posed by two independent sources of error: model misspecification and the bandit process itself. To tackle this challenge, the paper introduces two algorithms including RCLUMB and RSCLUMB. The former algorithm considers the feature similarity of users as clusters on graphs, while the latter treats it as a set. The authors establish regret upper bounds and present a corresponding lower bound to showcase the theoretical guarantees offered by both the proposed algorithms. Lastly, the paper provides empirical evaluations of the proposed algorithms by employing both synthetic and real-world data. These evaluations serve to validate and assess the effectiveness of the algorithms in practical scenarios.\n\nSTRENGTHS:\nThe paper presents a distinctive approach by integrating two previously examined bandit setups, resulting in novel contributions in terms of problem formulation, proposed algorithm, and theoretical insights. The manuscript exhibits exemplary writing, effectively conveying the central idea. While the individual concepts of CB and MLB have been extensively studied, the combination of them as provided in the paper introduces a more practical setup, thereby amplifying the significance of the central problem. The authors of the paper explicitly state the assumptions, which either draw from previously established work or represent a milder version of the same, providing apt discussion concerning the parameters defined in the Theorem, ensuring a comprehensive understanding of their implications. To support the theoretical claims, the manuscript furnishes ample empirical evidence, underscoring the effectiveness of the proposed algorithms.\n\nWEAKNESSES:\nConcerns are highlighted with the following points : \n1) Comparison between the two proposed algorithms: The main paper does mention two algorithms, but the paper seems to be lacking any discussion surrounding any theoretical comparison between the two. Can they be applied to the same problem? Scenarios to consider for highlighting the pros and cons of using one over another? Sample toy problem to support the claims? Any differences in the meta performance by using set nature over graph nature?\n2) Justification of algorithmic choices : \n i) While 1-hop clusters would provide a more close network, they will also lead to lower number of data points. How is the comparison taken into account? It is possible that the increase in data by considering a 2-hop network could be better aligned with the current user. How does the paper handle this possibility?\n ii) Line 72 -- we design a more tolerant edge design algorithm. More tolerant in comparison to?\n iii) In order to ensure the theorem statements, do the model misspecification parameter $\\epsilon^*$, noise variance $\\sigma^2$, and magnitude of the feature vector need to be in some relation? How is the paper quantifying the relationship between the three?\n3) Limitation and Failure: Would really appreciate a discussion on the limitation and failure cases of what the algorithm and problem setup is lacking and future directions of research with literature reference.\n\nQUESTIONS:\n1) Traditional notions of robustness typically involve outliers in data. In the case of the two algorithms mentioned in the paper, which include the term ``Robust'' in the titles, what is robustness tackling here? It is important to clarify what specific aspect of robustness they aim to address in this context.\n2) Is there a specific rationale for starting with a complete graph and employing successive elimination rather than an empty graph and successive addition? Intuitively, I would assume the latter approach would yield superior rewards in the initial few rounds, as it would be more focused on the data provided by the sampled arm itself. \n3) While the paper engages in a discussion regarding the selection of parameters within the range of (-0.2, 0.2), it is worth exploring other system parameters that influenced this decision-making process. Were there any additional system parameters that contributed to determining the range of parameters chosen?\n\nLIMITATIONS:\nThe paper demonstrates commendable discussions concerning the limitations of the theorem statements and assumptions. However, it would be highly beneficial to have similar discussions regarding the problem setup in relation to the existing landscape of works in the bandit domain.\nFurther, by examining the gaps or areas that have not been sufficiently addressed, and highlighting how the proposed problem formulation fills those gaps, the authors can contextualize their work and reinforce its significance. Additionally, discussing the differences or similarities between the proposed problem setup and the previous approach can contribute to a more comprehensive evaluation of the paper's novelty and the implications of its findings.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers a new bandit problem setting, clustering of bandits with misspecified user models. Here, the misspecified user model means the expected reward does not follow a perfect linear model. Authors give a regret upper bound of $O(\\epsilon_*T\\sqrt{d\\log{T}}+d\\sqrt{T}\\log{T})$, which is basically a combination of cost from regular CB and the cost from bounded deviation, which is not surprising. And then authors also provide a \"desperate\" and linear lower bound due to the bounded deviation.\n\nSTRENGTHS:\nThe only strength I can find is the lower bound, which conveys the message that this problem do not have any value as it will result in a linear lower bound. If I miss anything, please notify me.\n\nWEAKNESSES:\n1. The presentation is poor. i) the notation system is chaotic, ii) lack of description when introducing the algorithm.\n2. Lack of novelty in problem setting and result. At least for me, this paper considers an A (CB) + B (MLB) problem, so as the result (upper bound).\n3. The proof is, as well, a simple combination of CB and MLB.\n\nQUESTIONS:\n1. Could you provide some recent literature review about CB? The most related work [25] you mentioned was published 4 years ago, and there is a huge gap.\n2. In line 123, for the matrix norm, the matrix $M$ should be PD. Otherwise, you should not call it a norm. See ref: https://en.wikipedia.org/wiki/Norm_(mathematics).\n3. Could you please improve your notations, for example, in line 131 and 132,  $j(i)$, $i$, and $\\ell$? Please check the consistency. \n4. In line 133, since it is a contextual bandit problem, can it be extend to infinite arm set?\n5. Line 146, what if the user do not come uniformly? For example, for social network, some users are heavy user, whereas some are not.\n6. Line 152, I found the assumption of the bounded misspecification is useless, because the most worse case of this misspecification is a uniform distribution. Consequently, it is sub-Gaussian, and then can be absorbed into $\\eta$. Please provide explanation.\n\nLIMITATIONS:\nSee weakness before.", "TITLE": "Official Review", "RECOMMENDATION": 1, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-08"}

comments: SUMMARY:
This paper studies multi-agent linear bandits with cluster-structure and mis-specification. At each time, an user is chosen independently and an arm from a finite set of arms is to be recommended for this user. Subsequently, the decision maker observes a noisy and mis-specified reward. The users are assumed to have an underlying latent cluster structure with well separated clusters. 

The goal of the decision maker is to minimize the total regret of all the decisions made over time. The main result of the paper is an algorithm in which the decision maker has a running estimate of the clustering of users which is used to make recommendations. The regret bound scales sub-linearly in the number of agents (which indicates the learning algorithm is able to exploit the underlying cluster structure) and optimally in the mis-specification bound.

STRENGTHS:
1) The paper combines in a non-trivial manner, works on mis-specified linear bandits and clustered multi-agent bandits.

2) Technically, the paper moves past a restrictive technical assumption on the distribution of the arms first adopted by Gentile at.al. and followed by all subsequent works. This is a good contribution just in the study of clustered multi-agent bandits.

WEAKNESSES:
The only weakness is that the mis-specification upper bound \epsilon^* is both 
1. needed for the algorithm as an input, and 
2. The regret scales as this upper bound \epsilon^* as opposed to the true unknown mis-specification upper bound of the problem.

QUESTIONS:
Comment on the weakness in terms of knowledge of \epsilon^* can help elucidate why the assumption is needed or is non-trivial to overcome.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the clustering of bandits with misspecified user models problem. Clustering of bandits combines collaborative filtering with standard contextual bandits in order to cluster users into groups which share a common reward $\\theta$. This paper modifies the setting by allowing users to have deviation vectors $ \\epsilon^{i_t, t}$ so that the reward is $r_t = x_{a_t}^T\\theta_{i_t} + \\epsilon_{a_t}^{i_t, t} + \\eta_t$. They then propose two algorithms for this setting, prove their performance, and do experiments to show the improved results.\n\nSTRENGTHS:\n1) Model misspecification is a problem in online clustering bandits \n2) Experimental comparison is quite favorable towards the developed algorithms\n3) The developed algorithms have proven regret guarantees\n\nWEAKNESSES:\n1) ~~RSCLUMB is not well motivated~~\n2) Writing is a bit tricky to follow at times. For example, lines 309-310 are unclear to me are we moving the $\\|\\theta_{i_s} - \\theta_{i_t}\\|_2$ out and bounding it by the term on line 310? Or are we using the term from Lemma 5.7? Is the term from 5.7 substantially better?\n3) ~~The experiments are missing the baseline of learning a reward $\\theta$ for each user separately~~\n\nQUESTIONS:\n1) Does $\\epsilon_{a}^{i_t, t}$ vary over time? The $t$ superscript and line 141 make it seem like it does, but it's not clear to me how the algorithm could learn a user's deviation vector which is unknown and different at each timestep. If it varies over time, how does  $\\epsilon_{a}^{i_t, t}$ differ from the noise $\\eta_t$?\n2) Why do we talk about RCLUMB and RSCLUMB?\n3) How do your algorithms compare to the baseline of just learning a reward $\\theta$ for each user separately?\n\nLIMITATIONS:\nThe authors have not discussed the limitations, though they are similar to those for other bandit papers", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the problem of clustering of bandits with misspecified user models (CBMUM), where the expected rewards in user models can deviate from perfect linear models. The paper presents two robust CB algorithms, RCLUMB and RSCLUMB, that aim to deal with inaccurate user preference estimations and erroneous clustering caused by model misspecifications. The regret upper bounds of the algorithms are provided under introduced assumptions, and experiments show their outperformance over previous linear bandit algorithms.\n\nSTRENGTHS:\n- The paper studied a practical and important problem of clustering of bandits with misspecified user models, which is a more realistic scenario than assuming perfect linearity.\n- The theoretical analysis provides regret upper bounds for the algorithms under introduced assumptions.\n\nWEAKNESSES:\n- The paper's theoretical results are significantly undermined by the inclusion of impractical assumptions. The analysis relies on the existence of a pre-specified maximum misspecification level, denoted as $\\epsilon^*$, which is used as a hyper-parameter. However, in real-world scenarios, such prior knowledge is generally unavailable. Furthermore, the paper fails to offer any insights on how to determine this parameter and how it affects the model. As a result, the practical relevance and applicability of the findings are considerably weakened.\n- The motivation behind the proposed algorithm is insufficient. The paper suggests clustering bandits with misspecified user models as a solution to handle situations where the reward function does not necessarily adhere to linearity. However, it fails to discuss the extensive research conducted on non-linear bandits in recent years, as mentioned in [1,2]. Moreover, the existing work on clustering in the non-linear setting, as discussed in [3], significantly undermines the perceived contribution and novelty of this paper.\n- The experimental evaluation is limited. Only 1000 items and users are selected to perform the experiments, which is far from practical. The paper also fails to compare the proposed clustering algorithm with other recommender system algorithms [4]. Furthermore, the paper lacks an analysis of the learned clustering strategy, which could offer valuable insights into the practical pattern of clustering. Including such comparisons and analyses would enhance the overall quality and applicability of the experimental evaluation.\n\n[1] Neural Contextual Bandits with UCB-based Exploration. ICML 2020\n\n[2] Neural Thompson Sampling. ICLR 2021\n\n[3] Neural Collaborative Filtering Bandits via Meta-Learning.\n\n[4] Neural Collaborative Filtering. WWW 2017\n\n\n/****************/\n\nI couldn't include the authors in the comments below, so I've commented here. Thanks for the rebuttal. After carefully reading the points and others’ reviews, I‘m still not satisfied with the motivation, experimental evaluation, and the empirical selected misspecification level. For the motivation, the main benefit of the proposed bandit among neural bandits to handle non-linearity is efficiency. However, the efficiency of the proposed algorithm among other linear or non-linear bandits is not verified in the paper. The scalability of the used datasets is also quite small (maximum 10000 users only), which is difficult to support the claim of efficient. For the experimental evaluation, despite the dataset size, the choice to only rank a subset of items of each user is far from practical. The discussion with the recommender system literature is also insufficient as the online recommendation with the same setting are ignored. Furthermore, the significance of deriving the bound involving the term O(T) does not appear substantial. My rating will remain unchanged.\n\nQUESTIONS:\nPlease refer to the weaknesses I mentioned above.\n\nLIMITATIONS:\nI'm not aware of any potential negative societal impact in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-11"}