Title: Blocked Collaborative Bandits: Online Collaborative Filtering with Per-Item Budget Constraints
Venue: NeurIPS 2023
Accepted: yes
Keywords: Blocked Bandits

Collaborative Filtering

Clustering
Abstract: We consider the problem of \emph{blocked} collaborative bandits where there are multiple users, each with an associated multi-armed bandit problem. These users are grouped into \emph{latent} clusters such that the mean reward vectors of users within the same cluster are identical. Our goal is to design algorithms that maximize the cumulative reward accrued by all the users over time, under the \emph{constraint} that no arm of a user is pulled more than $\mathsf{B}$ times. 
This problem has been originally considered by \cite{Bresler:2014}, and designing regret-optimal algorithms for it has since remained an open problem.
In this work, we propose an algorithm called B-LATTICE (Blocked Latent bAndiTs via maTrIx ComplEtion) that collaborates across users, while simultaneously satisfying the budget constraints, to maximize their cumulative rewards. Theoretically, under certain reasonable assumptions on the latent structure, with 
$\mathsf{M}$ users, $\mathsf{N}$ arms, $\mathsf{T}$ rounds per user, and $\mathsf{C}=O(1)$ latent clusters, B-LATTICE achieves a per-user regret of  $\widetilde{O}(\sqrt{\mathsf{T}(1 + \mathsf{N}\mathsf{M}^{-1})})$ under a budget constraint of $\mathsf{B}=\Theta(\log 
\mathsf{T})$. These are the first sub-linear regret bounds for this problem, and match the minimax regret bounds when $\mathsf{B}=\mathsf{T}$. Empirically, we demonstrate that our algorithm has superior performance over baselines even when $\mathsf{B}=1$. B-LATTICE is a phased algorithm where in each phase it clusters users into groups and collaborates across users within a group to quickly learn their reward models.

Blocked Collaborative Bandits: Online Collaborative Filtering with Per-Item Budget Constraints

Abstract

We consider the problem of blocked collaborative bandits where there are multiple users, each with an associated multi-armed bandit problem. These users are grouped into latent clusters such that the mean reward vectors of users within the same cluster are identical. Our goal is to design algorithms that maximize the cumulative reward accrued by all the users over time, under the constraint that no arm of a user is pulled more than B times. This problem has been originally considered by [4], and designing regret-optimal algorithms for it has since remained an open problem. In this work, we propose an algorithm called B-LATTICE (Blocked Latent bAndiTs via maTrIx ComplEtion) that collaborates across users, while simultaneously satisfying the budget constraints, to maximize their cumulative rewards. Theoretically, under certain reasonable assumptions on the latent structure, with M users, N arms, T rounds per user, and C = O (1) latent clusters, B-LATTICE achieves a per-user regret of ˜ O ( √ T (1 + NM -1 ) under a budget constraint of B = Θ(log T ) . These are the first sub-linear regret bounds for this problem, and match the minimax regret bounds when B = T . Empirically, we demonstrate that our algorithm has superior performance over baselines even when B = 1 . B-LATTICE runs in phases where in each phase it clusters users into groups and collaborates across users within a group to quickly learn their reward models.

1 Introduction

Modern recommendation systems cater to millions of users and items [1] on a daily basis, typically in an online fashion. A critical feature of such systems is to quickly learn tastes of individual users from sequential actions and feedback, and suggest personalized products for each user. Furthermore, in practice, an item that has already been consumed by a user is recommended very few times to the same user (or not recommended at all). This is because, in applications such as movie/book recommendations, a typical user will find little interest in consuming the same item multiple times.

This problem - that we rename as Blocked Collaborative Bandits - was abstracted out by [4] with a modeling assumption that users have a latent clustering structure. That is, each user can belong to an unknown cluster and the expected reward for an item/movie is same for all users in a cluster. Formally, consider M users, N items and T rounds with M , N glyph[greatermuch] T ( M , N ≈ 10 6 in recommendation systems such as YouTube) and in each round, every user is recommended some item (potentially different for each user). On consuming the item, a noisy reward is assigned. As mentioned above,

Karthikeyan Shanmugam

Table 1: Comparison of various approaches for blocked collaborative bandits. All the regret bounds stated here are worst-case bounds and assume C = O (1) . Moreover, the regret is averaged across users. The worst-case pseudo-regret in [5] is linear when the items have rewards close to 0 . 5 . In [2], the authors' proposed a greedy algorithm whose worst-case has T 2 / 3 dependence.

it is assumed that the users can be clustered in C clusters, where C glyph[lessmuch] M , N , and users in the same cluster have identical reward distributions over items. Furthermore, any item can be recommended to a particular user at most B times, after which the item is blocked for the user.

[4] considered this problem for B = 1 in the setting where a user in cluster c on being recommended item j provides a like ( +1 ) with probability p cj and a dislike ( -1 ) with probability 1 -p cj . The authors studied a notion of pseudo-regret corresponding to minimizing the number of un-likeable items for a user, i.e., items for which the probability of user giving a like ( +1 ) is less than 1 / 2 . To this end, the authors proposed the Collaborative-Greedy algorithm, an glyph[epsilon1] -greedy style algorithm that performs random exploration in each round with certain probability. During exploitation it provides recommendations based on a neighborhood of similar users. However, maximizing the number of likeable items is limiting as it does not prioritize items with large rewards and completely disregards the item ordering. Despite these theoretical limitations, variants of the collaborative algorithm designed in [4] have found applications in predicting Bitcoin price, [29], information retrieval [12] among others.

Recent works have studied this problem under a more practical notion of regret which involves maximizing the cumulative rewards accrued over time [2, 5]. However, these works assume a cluster structure among both users and items. This assumption entails that there are many copies of the highest rewarding item for any user; this voids the blocking constraint and makes the problem significantly easier. Moreover, the algorithms designed in [2] are greedy (they perform exploration first to cluster users, items, and then perform exploitation) and achieve significantly sub-optimal regret. These bounds, translated to the original blocked bandit problem where there are no item clusters, have sub-optimal dependence on the number of items N . The algorithms designed in [5] assumed a noiseless binary feedback model which allowed for constant regret. However, these algorithms are not easily extendable to the more general noisy reward setting we consider in this work.

Another line of work has studied this problem when B = T (i.e., no budget constraints on arms) [24, 11, 25]. While some of these works have designed regret optimal algorithms [25], the B = T budget constraint is too lenient in many modern recommendation systems. To summarize, existing works have either attempted to solve the much harder setting of B = 1 by imposing additional restrictions on the problem or the simpler B = T setting which is not very relevant in practice.

This Work. In this work, we make progress on this problem by considering the intermediate setting of B = Θ(log T ) . We do not impose any cluster structure among items (as in [4]), and consider a general notion of regret which involves maximizing the cumulative rewards under the budget constraints. We propose B-LATTICE (Alg. 1), a phased algorithm which carefully balances exploration and exploitation. Our algorithm also performs on-the-fly clustering of users and collaborates across users within a cluster to quickly learn their preferences, while simultaneously satisfying the budget constraints. The key contribution of our work is that under certain standard incoherence assumptions (also used recently in [25]) and a budget constraint of B = Θ(log T ) , we show that B-LATTICE achieves a per-user regret of ˜ O ( √ T (1 ∨ NM -1 ) ) 1 . Empirically, our algorithm can even handle a budget constraint of B = 1 and perform better than baselines (Alg. 6). However, bounding its regret

in this setting is much more challenging which we aim to address in a future work. That being said, under an additional cluster structure on the items, we show that our theoretical guarantees hold even for B = 1 (Appendix F).

√

We also provide a non-trivial regret lower bound of ˜ Ω( NM -1 ∨ TM -1 / 2 ) . Our proof involves a novel reduction to multi-hypothesis testing and relies on Fano's inequality for approximate recovery [28] (see Appendix E). Our techniques could be of independent interest even outside the context of this work. Our lower bound is tight in two regimes - when the number of rounds T is very small or very large. However, we conjecture that our upper bounds are actually tight since they recover the same rates as in the B = T setting [25]; tightening the lower bound is left as a future work. Finally, we verify our theoretical results by simulations on synthetic data (see Appendix C). Here, we compare a more practical version of B-LATTICE (Algorithm 6) with Collaborative-Greedy [4]. We demonstrate that B-LATTICE not only has good regret but also other practically desirable properties such as a small cold-start period, and repeated high quality recommendations.

Techniques. Our algorithm is built on recent works that have studied this problem without the budget constraint [17, 25]. [17] developed a regret optimal algorithm called LATTICE that runs in phases. At any phase, the algorithm maintains a grouping of users which it refines over time (these groups are nothing but our current estimate of user clusters). Within each group, the algorithm relies on collaboration to quickly estimate the reward matrix. To be precise, the algorithm performs random exploration followed by low-rank matrix completion [8] to get a good estimate of the user-item reward matrix. Next, the algorithm uses the estimated reward matrix to eliminate sub-optimal arms for each user. Finally, it refines the user clusters by placing users with similar reward structure in the same group.

Extending the above algorithmic recipe to our setting poses a couple of challenges. First, observe that the oracle optimal strategy in the budget constrained setting is to recommend each of the top T / B items B times; we refer to these top times as golden items in the sequel. To compete against such a policy, our algorithm needs to quickly identify the golden items for each user. The LATTICE algorithm described above doesn't perform this, as it aims to only identify the top item for every user. So, one of the key novelties in our algorithm is to design a test to quickly identify the golden items and recommend them to the users. The second challenge arises from the usage of low-rank matrix completion oracles in LATTICE. To be precise, LATTICE requires more accurate estimates of the user-item reward matrix in the later phases of the algorithm. To this end, it repeatedly recommends an item to a user to reduce the noise in its estimates. However, this is infeasible in our setting due to the budget constraints. So, the second novelty in our algorithm is to avoid repeated recommendations and design an alternate exploration strategy that adheres to the budget constraints.

Other Related Work:

Item-Item Collaborative Filtering (CF). Acomplementary theoretical line of work proposes to exploit a clustering structure across the items instead of users [7, 23, 27, 22]. Under a similar blocking constraint, the authors have provided a sub-linear regret guarantee based on a certain measure of complexity called the doubling dimension. Since then, there have been several works in the literature that have attempted to exploit a cluster structure on both users and items [6, 2].

Variants of User-User CF. [15] has looked into the problem of non-stationary user-user collaborative filtering where the preferences of users change over time. [13] studies a variant where the user only provides positive ratings i.e when the user has liked an item. [9] and [3] studies probabilistic models for user user CF in an online and offline model respectively.

Cluster Structure across users. In several problems related to multi-armed bandits, cluster structure across users have been explored. In particular, in [11, 20, 10, 21, 26], the authors have imposed a cluster-structure across users while each item has a d -dimensional context chosen uniformly at random at each round. The preferences of each user is a linear function of the context vector and the cluster id. Under these settings, the authors prove a strong regret bound. However, note that in our setting, the item contexts are hidden; hence, these techniques are not applicable to our setting.

2 Problem Setting and Background

Notation. We write [ m ] to denote the set { 1 , 2 , . . . , m } . For a vector v ∈ R m , v i denotes the i th element; for any set U ⊆ [ m ] , let v U denote the vector v restricted to the indices in U . Similarly,

for A ∈ R m × n , A ij , A i denotes the ( i, j ) -th element and the i th row of A respectively. For any set U ⊆ [ m ] , V ⊆ [ n ] , A U , V denotes A restricted to the rows in U and columns in V . Let ‖ A ‖ ∞ denote absolute value of the largest entry in matrix A . N (0 , σ 2 ) denotes the Gaussian distribution with 0 mean and variance σ 2 . We write E X to denote the expectation of a random variable X . || U || 2 , ∞ corresponds to the maximum euclidean norm of a row of the matrix U . More precisely, for a matrix U ∈ R M × r , the norm || U || 2 , ∞ = max i ∈ [ M ] || U i || 2 . Thus, if || U || 2 , ∞ is small as in Lemma 1, then all rows of U have a small glyph[lscript] 2 norm.

Problem Setting. Consider an online recommendation system with M users, N items and T rounds. Let P ∈ R M × N be the reward matrix which is unknown. Here, we assume the set of M users can be partitioned into C disjoint but unknown clusters C (1) , C (2) , . . . , C ( C ) such that any two users u, v ∈ [ M ] belonging to the same cluster have identical reward vectors i.e. P u = P v . In each round t ∈ [ T ] , every user is recommended an item (can be different for each user) by the system. In turn, the system receives feedback in the form of reward from each user. Let, R ( t ) uρ u ( t ) be the observed reward for recommending item ρ u ( t ) ∈ [ N ] to user u at round t such that:

where E ( t ) uρ u ( t ) denotes the unbiased additive noise. 2 We assume that elements of { E ( t ) uρ u ( t ) } u ∈ [ M ] t ∈ [ T ] are i.i.d. zero mean sub-gaussian random variables with variance proxy σ 2 i.e. for all u, t ∈ [ M ] × [ T ] , we have E E ( t ) uρ u ( t ) = 0 and for all s ∈ R , we have E exp( s E ( t ) uρ u ( t ) ) ≤ exp( σ 2 s 2 / 2) . As in practical recommendation systems, we impose an additional constraint that the same item cannot be recommended more than B times to a user, for some small B = Θ(log T ) . For simplicity of presentation, we assume T is a multiple of B . However, we would like to note that our results hold for general T . Without loss of generality, we assume N ≥ T / B , since otherwise the budget constraints cannot be satisfied.

Our goal is to design a method that maximizes cumulative reward. Let π u : [ N ] → [ N ] denote the function that sorts the rewards of user u ∈ [ M ] in descending order, i.e., for any i < j , P uπ u ( i ) ≥ P uπ u ( j ) . The oracle optimal strategy for this problem is to recommend { π u ( s ) } s =1 ... T / B , the top T / B items with the highest reward, B times each. This leads us to the following notion of regret

where the expectation is over the randomness in the policy and the rewards, and ρ u ( t ) is any policy that satisfies the budget constraints.

Importance of collaboration. Suppose we treat each user independently and try to minimize their regret. In this case, when B is as small as O (log T ) , we will incur a regret that is almost linear in T . This is because we will not have enough data for any item to know if it has a high or a low reward. This shows the importance of collaboration across users to achieve optimal regret. The latent cluster structure in our problem allows for collaboration and sharing information about items across users.

For ease of exposition of our ideas, we introduce a couple of definitions.

Definition 1. A subset of users S ⊆ [ M ] is called 'nice" if S ≡ ⋃ j ∈A C ( j ) for some A ⊆ [ C ] . In other words, S can be represented as the union of some subset of clusters.

Definition 2. For any user u ∈ [ M ] , we call the set of items { π u ( t ) } T / B t =1 (i.e., the set of best T / B items) to be the golden items for user u .

2.1 Low-Rank Matrix Completion

Additional Notation. For an estimate ̂ P of reward matrix P , we will use ˜ π u : [ N ] → [ N ] to denote the items ordered according to their estimated reward for the user u i.e., ̂ P u ˜ π u ( i ) ≥ ̂ P u ˜ π u ( j ) when i < j . For any user u ∈ [ M ] and any subset of items A ⊆ [ N ] , we will use π u | A : [ |A| ] → [ N ] to denote the permutation π u restricted to items in A ⊆ [ N ] i.e. for any user u ∈ [ M ] and any i, j ∈ [ |A| ] satisfying i < j , we will have P uπ u ( i ) |A ≥ P uπ u ( j ) |A . Here, π u ( i ) | A corresponds to the i th item among items in A sorted in descending order of expected reward for user u .

An important workhorse of our algorithm is low-rank matrix completion, which is a well studied problem in the literature [16, 18, 8, 17]. Given a small set of entries that are randomly sampled from a matrix, these algorithms infer the missing values of the matrix. More precisely, consider a low-rank matrix Q ∈ R M × N . Let Ω ⊆ [ M ] × [ N ] be a random set of indices obtained by picking each index in [ M ] × [ N ] independently with probability p > 0 . Let { Z ij } ( i,j ) ∈ Ω be the corresponding noisy entries (sub-gaussian random variables with variance proxy σ 2 > 0 ) that we observe which satisfy E Z ij = Q ij ∀ ( i, j ) ∈ Ω . In this work we rely on nuclear norm minimization to estimate Q (see Algorithm 4). Recent works have provided tight element-wise recovery guarantees for this algorithm [8, 17]. We state these guarantees in the following Lemma.

Lemma 1 (Lemma 2 in [17]) . Consider matrix Q ∈ R M × N with rank r and SVD decomposition ¯ UΣ ¯ V T satisfying ‖ ¯ U ‖ 2 , ∞ ≤ √ µr/ M , ‖ ¯ V ‖ 2 , ∞ ≤ √ µr/ N and condition number κ = O (1) . Let d 1 = max( M , N ) , d 2 = min( M , N ) , noise variance σ 2 > 0 , and let sampling probability p be such that p = ˜ Ω ( µ 2 d 2 max ( 1 , σ 2 µ || P || 2 ∞ )) (for some constant c > 0 ). Suppose we sample a subset of indices Ω ⊆ [ M ] × [ N ] such that each tuple of indices ( i, j ) ∈ [ M ] × [ N ] is included in Ω independently with probability p . Let { Z ij } ( i,j ) ∈ Ω be the noisy observations corresponding to all indices in Ω . Then Algorithm 4, when run with inputs ( M , N , σ 2 , r, Ω , { Z ij } ( i,j ) ∈ Ω ), returns an estimate ˜ Q of Q which satisfies the following error bounds with probability at least 1 -O ( δ + d -12 2 )

Lemma 1 provides guarantees on the estimate ˜ Q of Q given the set of noisy observations corresponding to the entries in Ω . Equation 3 says that the quality of the estimate becomes better with the increase in sampling probability p that determines the number of entries in Ω . Note the detailed Algorithm ESTIMATE (Alg. 4) designed in [17, 8] is provided in Appendix A.

Remark 1 (Warm-up (Greedy Algorithm)) . Using Lemma 1, it is easy to design a greedy algorithm (Alg. 5 in Appendix B) for B = 1 ; for a fixed number of rounds m , we recommend random unblocked items to every user. Since the reward matrix P is a low-rank matrix, we use Algorithm 4 to estimate the entire matrix. Subsequently, we recommend the best estimated unblocked items for each user for the remaining rounds. We can prove that the regret suffered by such an algorithm is ˜ O ( T 2 / 3 (1 ∨ ( N / M )) 1 / 3 ) . The dependence on number of rounds T is sub-optimal, but Alg. 5 does not require the knowledge of any gaps corresponding to the reward matrix P . See Appendix B for a detailed proof. We would like to note that the result can be generalized easily to B = Θ(log T ) .

3 Main Results

Let X ∈ R C × N be the sub-matrix of the expected reward matrix P comprising C distinct rows of P corresponding to each of the C true clusters. Also, let τ := max i,j ∈ [ r ] |C ( i ) | / |C ( j ) | denote the ratio of the maximum and minimum cluster size. To obtain regret bounds, we make the following assumptions on the matrix X :

Assumption 1 (Assumptions on X ) . Let X = UΣV T be the SVD of X . Also, let X satisfy the following: 1) Condition number: X is full-rank and has non zero singular values λ 1 > · · · > λ C with condition number λ 1 /λ C = O (1) , 2) µ -incoherence: || V || 2 , ∞ ≤ √ µ C / N , 3) Subset Strong Convexity: For some α satisfying α log M = Ω(1) , γ = ˜ O (1) for all subset of indices S ⊆ [ M ] , |S| ≥ γ C , the minimum non zero singular value of V S must be at least √ α |S| / M .

Remark 2 (Discussion on Assumption 1) . We need Assumption 1 only to bound the incoherence factors and condition numbers of sub-matrices whose rows correspond to a nice subset of users and the columns comprise of at least T 1 / 3 golden items for each user in that nice subset. This implies that information is well spread-out across the entries instead of being concentrated in a few. Thus, we invoke Lemma 1 for those sub-matrices and utilize existing guarantees for low rank matrix completion. Our theorem statement can hold under significantly weaker assumptions but for clarity, we used the theoretically clean and simple to state Assumption 1. Note that the assumption is not required to run our algorithm or any offline matrix completion oracle - the only purpose of the assumption is to invoke existing theoretical guarantees of offline low rank matrix completion algorithms with glyph[lscript] ∞ error guarantees.

Assumption 2. We will assume that µ, σ, τ, || P || ∞ , C are positive constants and do not scale with the number of users M , items N or the number of rounds T .

Assumption 2 is only for simplicity of exposition/analysis and is standard in the literature. As in [25], we can easily generalize our guarantees if any of the parameters in Assumption 2 scale with In that case, the regret has additional polynomial factors of µ, σ, τ, || P || ∞ , C - moreover these quantities (or a loose upper bound) are assumed to be known to the algorithm. Now, we present our main theorem:

Theorem 1. Consider the blocked collaborative bandits problem with M users, N items, C clusters, T rounds with blocking constraint B = Θ(log T ) 3 such that at every round t ∈ [ T ] , we observe reward R ( t ) as defined in eq. (1) with noise variance proxy σ 2 > 0 . Let P ∈ R M × N be the expected reward matrix and X ∈ R C × M be the sub-matrix of P with distinct rows. Suppose Assumption 1 is satisfied by X and Assumption 2 is true. Then Alg. B-LATTICE initialized with phase index glyph[lscript] = 1 , M (1) = [[ M ]] , N (1) = [[ N ]] , C , T , t 0 = 1 , t exploit = 0 , ˜ P = 0 and glyph[epsilon1] 1 = c (log M ) -1 for some constant c > 0 guarantees the regret Reg ( T ) defined under the blocking constraint (eq. 2) to be:

Note that the regret guarantee in Theorem 1 is small as long as N ≈ M (but M , N can be extremely large). The dependence of the regret on number of rounds scales as √ T . This result also matches the √ T dependence on the number of rounds without the blocking constraint [17, 25]. Furthermore, Theorem 1 also highlights the importance of collaboration across users - compared with standard multi-armed bandits, the effective number of arms is now the number of items per user.

Remark 3. Note that in our model, we have assumed that users in the same latent cluster have identical true rewards across all items. Although we have made this assumption for simplicity as in other works in the literature [25, 2, 3, 5], the assumption of each user having exactly 1 of C different mean vectors can be relaxed significantly. In fact, in [25], a similar cluster relaxation was done in the following way for a known value of ν > 0 - there are C clusters such that 1) users in the same cluster have same best item and mean reward vectors that differ entry-wise by at most ν 2) any 2 users in different clusters have mean reward vectors that differ entry-wise by at least 20 ν . Our analysis can be extended to the relaxed cluster setting with an addition cost in regret in terms of ν .

We now move on to a rigorous lower bound in this setting.

Theorem 2. Consider the blocked collaborative bandits problem with M users, N items, C = 1 cluster, reward matrix P ∈ [0 , 1] M × N , T rounds with blocking constraint B such that at every round t ∈ [ T ] , we observe reward R ( t ) as defined in eq. (1) with noise random variables { E ( t ) uρ u ( t ) } u ∈ [ M ] t ∈ [ T ]

generated i.i.d according to a zero mean Gaussian with variance 1 . In that case, any algorithm must suffer a regret of at least

Note that the main difficulty in proving tight lower bounds in the blocked collaborative bandits setting is that, due to the blocking constraint of B , a single item having different rewards can only cause a

difference in regret of at most B in two separate instances constructed in standard measure change arguments [19]. However, we prove two regret lower bounds on this problem, out of which the latter is the technically more interesting one.

The first term in the lower bound in Thm. 2 follows from a simple reduction from standard multiarmed bandits. Intuitively, if we have T / B known identical copies of each item, then the blocking constraint is void - but with C = 1 , this is (almost) equivalent to a standard multi-armed bandit problem with MT rounds, NBT -1 distinct arms (up to normalization with M ). The lower bound follows from invoking standard results in MAB literature. For B = T , we recover the matching regret lower bound in [25] without the blocking constraint. Furthermore, for B = Θ(log T ) , the first term is tight up to log factors when number of rounds is small for example when T = O (1) . (Appendix E)

The second term in the lower bound is quite non-trivial. Note that the second term is tight up to log factors when the number of rounds T = Θ( N ) is very large (close to the number of items) and B = O (log T ) is small. We obtain this bound via reduction of the regret problem to a multiple hypothesis testing problem with exponentially many instances and applying Fano's inequality [28] (commonly used in proving statistical lower bounds in parameter estimation) for approximate recovery. Our approach might be of independent interest in other online problems as well. (Appendix E)

Remark 4. We leave the problem of extending our guarantees for B = 1 as future work. However we make progress by assuming a cluster structure over items as well. More precisely, suppose both items and users can be grouped into a constant number of disjoint clusters as such that (user,item) pairs in same cluster have same expected reward. Then, we can sidestep the statistical dependency issues in analysis of Alg. 1 for B = 1 and provide similar guarantees as in Thm. 1 (see Appendix F).

4 B-LATTICE Algorithm

B-LATTICE is a recursive algorithm and runs in O (log T ) phases of exponentially increasing length. B-LATTICE assume the knowledge of the following quantities - users M , items N , clusters C , rounds T , blocking constraint B , maximum true reward || P || ∞ , incoherence factor µ , cluster size ratio τ , noise variance σ 2 > 0 and other hyper-parameters in Assumption 1. At any phase, the algorithm maintains a partitioning of users into crude clusters, and a set of active items for each such crude cluster. Let M ( glyph[lscript] ) be the partitioning of users in the glyph[lscript] th phase, and N ( glyph[lscript] ) be the list containing the set of active items for each group of users in M ( glyph[lscript] ) . In the first phase, we place all users into a single group and keep all items active for every user; that is, M (1) = [[ M ]] , N (1) = [[ N ]] . There are three key components in each phase: (a) exploration, (b) exploitation, and (c) user clustering. In what follows, we explain these components in detail. For simplicity of exposition, suppose Assumption 2 is true namely the parameters µ, σ, τ, || P || ∞ , C are constants.

Exploration (Alg. 2). The goal of the Explore sub-routine is to gather enough data to obtain a better estimate of the reward matrix. To be precise, let M ( glyph[lscript],i ) be the users in the i th group at phase glyph[lscript] , and N ( glyph[lscript],i ) be the set of active items for this group. Let P M ( glyph[lscript],i ) , N ( glyph[lscript],i ) be the sub-matrix of P corresponding to rows M ( glyph[lscript],i ) and columns N ( glyph[lscript],i ) (recall, this sub-matrix has rank at most C ). Our goal is to get an estimate ˜ P M ( glyph[lscript],i ) , N ( glyph[lscript],i ) of this matrix that is entry-wise O (2 -glyph[lscript] ) close to the true matrix. To do this, for each user in M ( glyph[lscript],i ) , we randomly recommend items from N ( glyph[lscript],i ) with probability p = O ( 2 2 glyph[lscript] log d 1 d 2 ) (Line 6). Here, d 1 = max( |M ( glyph[lscript],i ) | , |N ( glyph[lscript],i ) | ) , d 2 = min( |M ( glyph[lscript],i ) | , |N ( glyph[lscript],i ) | ) . We then rely on low-rank matrix completion algorithm (Algorithm 4) to estimate the low rank sub-matrix. By relying on Lemma 1, we show that our estimate is entry-wise O (2 -glyph[lscript] ) accurate. This also shows that our algorithm gets more accurate estimate of P M ( glyph[lscript],i ) , N ( glyph[lscript],i ) as we go to the later phases.

User Clustering (lines 7-8 of Alg. 1). After the exploration phase, we refine the user partition to make it more fine-grained. An important property we always want to ensure in our algorithm is that M ( glyph[lscript],i ) , the i th group/crude cluster in phase glyph[lscript] , is a nice subset for all ( glyph[lscript], i ) (see Definition 1 for a definition of nice subset). To this end, we build a user-similarity graph whose nodes are users in M ( glyph[lscript],i ) , and draw an edge between two users if they have similar rewards for all the arms in N ( glyph[lscript],i ) (Line 8 in Alg. 1). Next, we partition M ( glyph[lscript],i ) based on the connected components of the graph. That is, we group all the users in a connected component into a single cluster. In our analysis, we show that each connected component of the graph is a nice subset .

Algorithm 1 B-LATTICE (Blocked Latent bAndiTs via maTrIx ComplEtion )

Exploitation (Alg. 3). The main goal in the Exploit sub-routine of our algorithm is to identify common golden items for a group of users jointly. Consider a group of users M with active items N at the beginning of exploit sub-routine invoked in the glyph[lscript] th phase of Algorithm 1. We perform the following test in Line 1 of the Exploit sub-routine: for every user u ∈ M , we check if ˜ P u ˜ π u (1) |N -˜ P u ˜ π u ( |N| ) |N ≥ c ′ 2 -glyph[lscript] for some constant c ′ > 0 - that is if the estimated highest rewarding and lowest rewarding items of the user u have a significant gap. For all the users that satisfy the above property, we take a union of the items close to the estimated highest rewarding item for each of them (Line 3). We can show that these identified items are actually golden items for all users in the nice subset M . Hence, we immediately recommend these identified golden items to every user in the nice subset B times. In case a golden item is blocked for a user, we recommend an unblocked active item (Line 5). Subsequently we remove the golden items from the active set of items (Line 8) and prune it. We go on repeating the process with the pruned set of items until we can find no user that satisfy the above gap property between highest and lowest estimated rewarding items in the active set.

To summarize, in each phase of Algorithm 1, we perform the exploration, exploitation and user clustering steps described above. As the algorithm proceeds, we get more fine grained clustering of users, and more accurate estimates of the rewards of active items. Using this information, the algorithm tries to identify golden items and recommends the identified golden items to users.

Implementation Details: The actual implementation of the algorithm described above is a bit more involved due to the fact that every user has to be recommended an item at every time step (see

Algorithm 2 Explore (Explore Component of a phase)

problem setting in Section 2). To see this, consider the following scenario. Suppose, we identified item j as a golden item for users in the nice subset M (crude cluster) in the Exploit sub-routine invoked in some phase of Alg. 1. Moreover, suppose there are two users u 1 , u 2 in the cluster for whom the item has been recommended n 1 , n 2 times respectively, for some n 1 < n 2 . So, for the final n 2 -n 1 iterations during which the algorithm recommends item j to u 1 , it has to recommend some other item to u 2 . In our algorithm, we randomly recommend an item from the remaining active set of items for u 2 during those n 2 -n 1 rounds. We store the rewards from these recommendations and use them in the exploration component of future phases. A similar phenomenon also occurs in the Explore sub-routine where we sometimes need to recommend items outside the sub-sampled set of entries in Ω (Line 7 and 10 in Alg. 2) To this end, we introduce matrices K , L ∈ R M × N which perform the necessary book-keeping for us.

K tracks number of times an item has been recommended to a user and the corresponding observation has been already used in computing an estimate of some reward sub-matrix (Line 2 of Alg. 2). Since these estimates are used to cluster users and eliminate items (Lines 7-10 in Alg. 1), these observations are not reused in subsequent phases to avoid statistical dependencies. L tracks the number of times an item has been recommended to a user and the corresponding observation has not been used in computing an estimate of reward sub-matrix so far. These observations can still be used once in Line 2 of Alg. 2. Observe that K ij + L ij is the total times user i has been recommended item j . In practice, eliminating observations is unnecessary and we can reuse observations whenever required. Hence Alg. 1 can work even when B = 1 (see Alg. 6 for a simplified and practical version).

Handling very few active items (Line 5 in Alg. 1). Recall, in the exploration step of phase glyph[lscript] , we randomly recommend active items with probability p = O ( 2 2 glyph[lscript] log d 1 d 2 ) . If p > 1 , then we simply recommend all the remaining unblocked items for each user until the end. In our analysis, we can

Algorithm 3 Exploit (Exploit Component of a phase)

show that this happens only if the size of surviving items is too small, and when the number of remaining rounds is very small. Hence the regret is small if we follow this approach (Lemma 5). Similarly, when remaining rounds become smaller than T 1 / 3 , we follow the same approach.

Running Time: Computationally speaking, the main bottleneck of our algorithm is the matrix completion function Estimate invoked in Line 13 of the Explore Component (Algorithm 2). All the remaining steps have lower order run-times. Note that the Estimate function is invoked at most O ( C log T ) times since there are can be at most C disjoint nice subsets of users at a time and the number of phases is log T . Moreover, note that the Estimate function (Algorithm 4) solves a convex objective in Line 3 - this has a time complexity of O ( M 2 N + N 2 M ) which is slightly limiting because of the quadratic dependence. However, a number of highly efficient techniques have been proposed for optimizing the aforementioned objective even when M , N are in the order of millions (see [14]).

Proof Sketch of Theorem 1 We condition on the high probability event that the low rank matrix completion estimation step is always successful (Lemma 17). Note that for a fixed user, the items chosen for recommendation in the exploit component of any phase are golden items and costs zero regret if they are unblocked and recommended. Even if it is blocked, we show a swapping argument to a similar effect. That is, with an appropriate permutation of the recommended items, we can ignore the regret incurred from golden-items altogether. Moreover, in the explore component of the glyph[lscript] th phase, we can bound the sub-optimality gap of the surviving active items by some pre-determined glyph[epsilon1] glyph[lscript] . We prove that this holds even with the (chosen) permutation of the recommended items (Lemma 18). We choose glyph[epsilon1] glyph[lscript] to be exponentially decreasing in glyph[lscript] and the number of rounds in the explore component of phase glyph[lscript] is roughly 1 /glyph[epsilon1] 2 glyph[lscript] . Putting these together, we obtain the regret guarantee in Theorem 1.

5 Conclusion and Future Work

We study the problem of Collaborative Bandits in the setting where each item can be recommended to a user a small number of times. Under some standard assumptions and a blocking constraint of Θ(log T ) , we show a phased algorithm B-LATTICE with regret guarantees that match the tight results with no blocking constraint [25]. To the best of our knowledge, this is the first regret guarantee for such a general problem with no assumption of item clusters. We also provide novel regret lower bounds that match the upper bound in several regimes. Relaxing the assumptions, extending guarantees to a blocking constraint of B = 1 and tightening the gap between the regret upper and lower bounds are very interesting directions for future work.

References

Charu C Aggarwal et al. Recommender systems , volume 1. Springer, 2016.

Kaito Ariu, Narae Ryu, Se-Young Yun, and Alexandre Proutière. Regret in online recommendation systems. Advances in Neural Information Processing Systems , 33:21141-21150, 2020.

Kishor Barman and Onkar Dabeer. Analysis of a collaborative filter based on popularity amongst neighbors. IEEE Transactions on Information Theory , 58(12):7110-7134, 2012.

Guy Bresler, George H. Chen, and Devavrat Shah. A latent source model for online collaborative filtering. In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2 , NIPS'14, pages 3347-3355, 2014.

Guy Bresler and Mina Karzand. Regret bounds and regimes of optimality for user-user and item-item collaborative filtering. In 2018 Information Theory and Applications Workshop (ITA) , pages 1-37. IEEE, 2018.

Guy Bresler and Mina Karzand. Regret bounds and regimes of optimality for user-user and item-item collaborative filtering. IEEE Transactions on Information Theory , 67(6):4197-4222, 2021.

Guy Bresler, Devavrat Shah, and Luis Filipe Voloch. Collaborative filtering with low regret. In Proceedings of the 2016 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Science , SIGMETRICS '16, pages 207-220, New York, NY, USA, 2016. ACM.

Yuxin Chen, Yuejie Chi, Jianqing Fan, Cong Ma, and Yuling Yan. Noisy matrix completion: Understanding statistical guarantees for convex relaxation via nonconvex optimization. arXiv preprint arXiv:1902.07698 , 2019.

Onkar Dabeer. Adaptive collaborating filtering: The low noise regime. In 2013 IEEE International Symposium on Information Theory , pages 1197-1201. IEEE, 2013.

Claudio Gentile, Shuai Li, Purushottam Kar, Alexandros Karatzoglou, Giovanni Zappella, and Evans Etrue. On context-dependent clustering of bandits. In International Conference on Machine Learning , pages 1253-1262. PMLR, 2017.

Claudio Gentile, Shuai Li, and Giovanni Zappella. Online clustering of bandits. In International Conference on Machine Learning , pages 757-765. PMLR, 2014.

Dorota Glowacka et al. Bandit algorithms in information retrieval. Foundations and Trends R © in Information Retrieval , 13(4):299-424, 2019.

Reinhard Heckel and Kannan Ramchandran. The sample complexity of online one-class collaborative filtering. In International Conference on Machine Learning , pages 1452-1460. PMLR, 2017.

Cho-Jui Hsieh and Peder Olsen. Nuclear norm minimization via active subspace selection. In International Conference on Machine Learning , pages 575-583. PMLR, 2014.

Wasim Huleihel, Soumyabrata Pal, and Ofer Shayevitz. Learning user preferences in nonstationary environments. In International Conference on Artificial Intelligence and Statistics , pages 1432-1440. PMLR, 2021.

Prateek Jain, Praneeth Netrapalli, and Sujay Sanghavi. Low-rank matrix completion using alternating minimization. In Proceedings of the forty-fifth annual ACM symposium on Theory of computing , pages 665-674, 2013.

Prateek Jain and Soumyabrata Pal. Online low rank matrix completion. arXiv preprint arXiv:2209.03997 , 2022.

Vladimir Koltchinskii, Karim Lounici, and Alexandre B Tsybakov. Nuclear-norm penalization and optimal rates for noisy low-rank matrix completion. 2011.

Tor Lattimore and Csaba Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Shuai Li, Wei Chen, and Kwong-Sak Leung. Improved algorithm on online clustering of bandits. arXiv preprint arXiv:1902.09162 , 2019.

Shuai Li, Alexandros Karatzoglou, and Claudio Gentile. Collaborative filtering bandits. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval , pages 539-548, 2016.

Greg Linden, Brent Smith, and Jeremy York. Amazon. com recommendations: Item-to-item collaborative filtering. IEEE Internet computing , 7(1):76-80, 2003.

Gregory D Linden, Jennifer A Jacobi, and Eric A Benson. Collaborative recommendations using item-to-item similarity mappings, July 24 2001. US Patent 6,266,649.

Odalric-Ambrym Maillard and Shie Mannor. Latent bandits. In International Conference on Machine Learning , pages 136-144. PMLR, 2014.

Soumyabrata Pal, Arun Sai Suggala, Karthikeyan Shanmugam, and Prateek Jain. Optimal algorithms for latent bandits with cluster structure. arXiv preprint arXiv:2301.07040 , 2023.

Yunzhe Qi, Tianxin Wei, Jingrui He, et al. Neural collaborative filtering bandits via meta learning. arXiv preprint arXiv:2201.13395 , 2022.

Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl. Item-based collaborative filtering recommendation algorithms. In Proceedings of the 10th international conference on World Wide Web , pages 285-295, 2001.

Jonathan Scarlett and Volkan Cevher. An introductory guide to fano's inequality with applications in statistical estimation. arXiv preprint arXiv:1901.00555 , 2019.

Devavrat Shah and Kang Zhang. Bayesian regression and bitcoin. In 2014 52nd annual Allerton conference on communication, control, and computing (Allerton) , pages 409-414. IEEE, 2014.

Limitations: The main contributions of our works are theoretical. From a theoretical point of view, the limitations of our paper are discussed in Sections 3 and 5. In particular, we believe that tightening the gap between the upper and lower bounds in regret will require novel and non-trivial algorithmic ideas - we leave this as an important direction of future work.

Broader Impact: Due to the theoretical nature of the work, we do not foresee any adverse societal impact.

A Low rank matrix completion

Below, we describe the Estimate sub-routine to estimate a M × N low rank matrix Q of rank r given a partially observed set of noisy entries { Z ij } ( i,j ) ∈ Ω corresponding to a subset Ω ⊆ [ M ] × [ N ] . Here E Z ij = Q ij for all ( i, j ) ∈ Ω and furthermore, { Z ij } ( i,j ) ∈ Ω are independent sub-gaussian random variables with variance proxy at most σ 2 .

Algorithm 4 Estimate (Low-rank matrix completion ) [17]

Require: Matrix dimensions ( M , N ) , noise variance σ 2 , rank r , subset of indices that are observed Ω ⊆ [ M ] × [ N ] and noisy observations { Z ij } ( i,j ) ∈ Ω .

1: Partition the rectangular matrix into square matrices. Without loss of generality, assume M ≤ N . For each i ∈ [ N ] , randomly set ζ i to be a value in the set [ glyph[ceilingleft] N / M glyph[ceilingright] ] uniformly at random. Partition indices in [ N ] into [ N ] (1) , [ N ] (2) , . . . , [ N ] ( k ) where k = glyph[ceilingleft] N / M glyph[ceilingright] and [ N ] ( q ) = { i ∈ [ N ] | ζ i = q } for each q ∈ [ k ] . Set Ω ( q ) ← Ω ∩ ([ M ] × [ N ] ( q ) ) for all q ∈ [ k ] . { If M ≥ N , we partition the indices in [ M ] .}

2: for q ∈ [ k ] do

3: Solve the following convex program with λ = C λ σ √ | Ω | / max( M , N ) , for some constant C λ > 0

where ‖ ˜ Q ( q ) ‖ glyph[star] denotes nuclear norm of matrix ˜ Q ( q ) and γ u ( j ) is index of j in set [ N ] ( q ) .

4: end for

5: Return ˜ Q ∈ R M × N such that ˜ Q [ M ] , [ N ] ( q ) = ˜ Q ( q ) for all q ∈ [ k ] .

B Explore-Then-Commit (ETC)

We first present a greedy algorithm in the blocked setting with B = 1 (no repetition) that uses the Explore-Then-Commit (ETC) framework. Such an algorithm has two disjoint phases exploration and exploitation . We will first jointly explore the set of items (without repeating same item for any user) for all users for a certain number of rounds and compute an estimate ˜ P of the reward matrix P . Subsequently, in the exploitation phase, for each user, we recommend the best estimated distinct items (that have not been recommended in the exploration phase to that user) inferred from the estimated reward matrix ˜ P . Note that if we explore too less, then our estimate will be poor and hence we will suffer large regret once we commit in the exploitation phase. On the other hand, if we explore too much, then the exploration cost will be high. Our goal is to balance both the exploration length and the exploitation cost under the blocked setting. Thus, we obtain the following result:

Theorem 3. Consider the GBB setting with M users, C = O (1) clusters, N items, T recommendation rounds and blocking constraint B = 1 . Set d 2 = min( M , N ) . Let R ( t ) uρ u ( t ) be the reward in each round, defined as in (1) . Suppose d 2 = Ω( µr log( rd 2 )) . Let P ∈ R M × N be the expected reward matrix that satisfies the conditions stated in Lemma 1 , and let σ 2 be the noise variance in rewards. Then, Algorithm 5, applied to the online rankr matrix completion problem under the blocked setting guarantees the regret defined as in eq. 2 to be:

In order to understand the result, note that the second term in the regret bound stems from the fact that in our algorithmic framework, the low rank matrix completion module needs a certain number of

Algorithm 5 ETC (Explore-Then-Commit Algorithm with Blocking constraint B = 1 )

Require: users M , items N , rounds T , noise σ 2 , rank r of P .

1: Set p = ˜ O ( ( N ‖ P ‖ ∞ ) -2 / 3 ( T σr √ d 2 √ µ 3 ) 2 / 3 ∨ µ 2 d 2 ) . Set d 2 = min( M , N ) and λ = Cσ √ d 2 p for constant C .

2: For each tuple of indices ( i, j ) ∈ [ M ] × [ N ] , independently set δ ij = 1 with probability p and δ ij = 0 with probability 1 -p .

3: Denote Ω = { ( i, j ) ∈ [ M ] × [ N ] | δ ij = 1 } and m = max i ∈ [ M ] | | j ∈ [ N ] | ( i, j ) ∈ Ω | to be the maximum number of index tuples in a particular row. For all ( i, j ) ∈ Ω , set Mask ij = 0 .

4: for rounds t = 1 , 2 , . . . , m do

5: For each user u ∈ U , recommend an item ρ u ( t ) in { j ∈ [ N ] | ( u, j ) ∈ Ω , Mask uj = 0 } and set Mask uρ u ( t ) = 1 . If not possible then recommend any item ρ u ( t ) in [ N ] s.t. ( u, ρ u ( t )) glyph[negationslash]∈ Ω and has not been recommended yet to user u . Observe R ( t ) uρ u ( t ) .

6: end for

7: Compute the estimate ˜ P ∈ R M × N as output of ˜ P ← Estimate ([ M ] , [ N ] , σ 2 , r, Ω , { R ( t ) uρ u ( t ) ) } t ∈ [ m ] ).

8: for each of remaining rounds do

9: Set j ′ u = argmax j ∈ [ N ] ˜ P u ˜ π u ( j ) for each user u , s.t. ˜ π u ( j ′ u ) has not been recommended before to u .

10: For each user u ∈ [ M ] , recommend the item ˜ π u ( j ′ u ) to user u .

11: end for

observed indices and therefore a certain number of exploration rounds (note that p ≥ Cµ 2 d -1 2 log 3 d 2 for some constant C > 0 in Lemma 1). Similarly, the third term stems from the failure of the estimation module; again, the term T -2 can be replaced by T -c for any constant c > 0 . The first term in the regret bound captures the dependence on the number of rounds T - the scaling of T 2 / 3 is sub-optimal and our subsequent goal is to improve this dependence to the rate of √ T .

Proof of Theorem 3. Suppose we explore for a period of S rounds such that the exploration period succeeds with a probability of 1 -ν . Conditioned on the event that the exploration period succeeds, we obtain an estimate ˜ P of the reward matrix P satisfying ‖ P -˜ P ‖ ∞ ≤ ρ . Recall π u : [ N ] → [ N ] to be the permutation on [ N ] such that for any i, j ∈ [ N ]; i < j , we have P uπ u ( i ) ≥ P uπ u ( j ) . Similarly, denote ˜ π u : [ N ] → [ N ] such that for for any i, j ∈ [ N ]; i < j , we have ˜ P u ˜ π u ( i ) ≥ ˜ P u ˜ π u ( j ) . Now consider any index i ∈ [ T ] for which we will analyze P u ˜ π u ( i ) -P uπ u ( i ) which is the error if we choose the i th item according to the estimated matrix ˜ P (instead of P ). There are several cases that we need to consider. First, suppose ˜ π u ( i ) = π u ( j ) where j ≤ i . In that case, we have P u ˜ π u ( i ) -P uπ u ( i ) ≥ 0 . Now, consider the other case where j > i implying that the element in the j th position in the permutation π u has shifted to the left in ˜ π u . In order for this to happen, there must exist an element i 1 ≤ i ≤ i 2 for which ˜ π u ( i 2 ) = π u ( i 1 ) implying that an element i 1 in the permutation π u has shifted to the right in ˜ π u . Therefore,

where we used the fact that ‖ P -˜ P ‖ ∞ ≤ ρ , ˜ P u ˜ π u ( i ) -˜ P u ˜ π u ( i 2 ) ≥ 0 (since i ≤ i 2 ), P u ˜ π u ( i 2 ) -P uπ u ( i ) = P uπ u ( i 1 ) -P uπ u ( i ) ≥ 0 (since i 1 ≤ i ). Therefore, at each step of the exploitation stage, for each user u , we recommend one of the top T -S items (as inferred from ˜ P ) with the highest reward that have not been recommended until that round to the user u ; in each such step, we will suffer a regret of at most 2 ρ if we compare with the item at the same index in π u . As before, conditioned on the event that the exploration fails (and in the exploration stage as well), the regret at each step can be

bounded from above by 2 ‖ P ‖ ∞ . In that case, we have

We use Lemma 1 to set S = O ( N p + √ N p log M δ -1 ) such that ρ = O ( σr √ d 2 √ µ 3 log d 2 p ) and ν = 1 -δ -O ( d -3 2 ) . We have

For simplicity, we ignore the logarithmic and lower order terms and attempt to minimize N p ‖ P ‖ ∞ + T ( σr √ d 2 √ µ 3 log d 2 p ) ( Simplified Expression ) by choosing p = ( N ‖ P ‖ ∞ ) -2 / 3 ( T σr √ d 2 √ µ 3 log d 1 ) 2 / 3 , δ = T -4 . If p ≥ Cµ 2 d -1 2 log 3 d 2 , then notice that N p ≥ 1 and therefore N p + √ N p log M δ -1 = O ( N p √ log M δ -1 ) . Subsequently, we have

There exists an edge case when the value of p that minimizes the simplified expression satisfies p ≤ Cµ 2 d -1 2 log 3 d 2 . Then we can substitute p = Cµ 2 d -1 2 log 3 d 2 . In that case, the second term in the simplified expression will still be bounded as before. On the other hand the first term in the simplified expression will now be bounded by O ( N µ 2 d 2 log 3 d 2 log 2 ( MNT ) || P || ∞ ) . Hence, our regret will be bounded by

C Experiments

We conduct detailed synthetic experiments in order to validate the theoretical guarantees/properties of our algorithm. For this purpose we use a simplified version of B-LATTICE described in Alg. 6 (all experiments have been performed on a Google Colab instance with 12GB RAM) :

In PB-LATTICE (Alg. 6), we do not use the exploit component for simplicity. Intuitively, if the user is recommended a golden item during the explore component, it is a good event in any case. Furthermore, in Step 14, we use k -means to cluster the users. Since we obtain a significantly large embedding vector for each user in Step 13, k -means is quite practical.

We run PB-LATTICE on several synthetic datasets. There are two main baselines for us to consider 1) Greedy Algorithm namely Alg. 5 2) In the setting where the user u , on being recommended item j provides a like ( +1 ) with probablity P uj and a dislike ( -1 ) with probability 1 -P uj , we also compare with Collaborative-Greedy in [4]. However, recall that [4], even in the restricted setting, only provides theoretical guarantees on the number of likeable items (items with probability of liking > 0 . 5 ) recommended during the course of T rounds.

We generate three synthetic datasets to validate our algorithms. For each of them, we take M = 150 users, N = 150 items and T = 60 rounds (hence the total items recommended will be 9000 ). The reward matrix P = UV T ( U ∈ R M × C and V ∈ R N × C ) is generated as in [25] - in the i th row of U , the ( i % C ) th entry is set to be 1 while the other entries are 0 ; the entries of V are sampled in the following way for the three datasets.

200

B-LATTICE

Reward

150

Greedy(m=30)

Greedy (m=10)

Cumulative

100

50

0

0

10

20

30

40

50

60

Rounds

12

B-LATTICE

Greedy(m=10)

Reward

10

Greedy(m=30)

8

Roundwise

6

4

2

0

0

10

20

30

40

50

60

Rounds

(a)

V

has entries distributed according to observe

R

(

t

)

uρ

u

(

t

)

N

(0

,

25)

=

N

(

P

uρ

u

(

t

)

)

,

0

.

25)

250

B-LATTICE

Reward

200

Greedy(m=10)

Greedy(m=30)

Cumulative

150

100

50

0

0

10

20

30

40

50

60

Rounds

(c)

V

has entries distributed according to observe

5.0

B-LATTICE

Reward

4.5

Greedy(m=10)

Greedy(m=30)

4.0

Roundwise

3.5

3.0

2.5

2.0

0

10

20

30

40

50

60

Rounds

R

(

t

)

uρ

u

(

t

)

=

N

(

P

uρ

u

(

t

)

)

,

0

.

25)

50

B-LATTICE

Reward

40

Greedy(m=10)

Greedy(m=30)

Collaborative-Greedy

30

Cumulative

20

10

0

0

10

20

30

40

50

60

Rounds

1.0

0.9

Reward

0.8

0.7

RoundwiseP

B-LATTICE

0.6

Collaborative-Greedy

0.5

0.4

0.3

0.2

0

10

20

30

40

50

09

Rounds

U

(0

,

5)

Algorithm 6 PB-LATTICE (Practical Blocked Latent bAndiTs via maTrIx ComplEtion with blocking constraint B = 1 )

Require: Phase index glyph[lscript] , phase length m glyph[lscript] , List of disjoint nice subsets of users M ( glyph[lscript] ) , list of corresponding subsets of active items N ( glyph[lscript] ) , clusters C , rounds T , noise σ 2 > 0 , round index t 0 , gap factor ν glyph[lscript] .

1: Set λ = 10 σ √ ( m glyph[lscript] / M ) .

2: for rounds t 0 +1 , t 0 +2 , . . . , min( t 0 + m, T ) do

3: for i th nice subset of users M ( glyph[lscript],i ) ∈ M ( glyph[lscript] ) with active items N ( glyph[lscript],i ) ( i th set in list N ( glyph[lscript] ) ) do 4: Initialize Ω ( i ) = Φ

5: for user u in M ( glyph[lscript],i ) do

6: Choose a random item j ∈ N ( glyph[lscript],i ) for recommendation to user u . If j is not blocked, recommend j to user u and observe Z uj to be the feedback from user u for item j . If j is blocked, then recommend any j ′ ∈ N ( glyph[lscript],i ) that is unblocked for user u ∈ M ( glyph[lscript],i ) . Observe Z uj ′ to be the feedback from user u for item j ′ .

7: Update Ω ( i ) = Ω ( i ) ∪ { ( u, j ) }

8: end for

9: end for

10: end for

11: for i th nice subset of users M ( glyph[lscript],i ) ∈ M ( glyph[lscript] ) with active items N ( glyph[lscript],i ) ( i th set in list N ( glyph[lscript] ) ) do

12: Initialize M ( glyph[lscript] +1) = [] and N ( glyph[lscript] +1) = [] .

13: Compute T ∈ R M × N by solving the convex program

14: Solve k -means for users in M ( glyph[lscript],i ) using the vector embedding formed by the rows in T M ( glyph[lscript],i ) , N ( glyph[lscript],i ) . Choose best k ≤ C by using ELBOW method. Denote the cluster of users by {M ( glyph[lscript],i,j ) } j .

15: for each cluster of users M ( glyph[lscript],i,j ) do

16: Compute set of active arms N ( glyph[lscript],i,j ) as { s ∈ N ( glyph[lscript],i ) | T us ≥ T u ˜ π u ( T ) -ν glyph[lscript] for some u ∈ M ( glyph[lscript],i,j ) } . Here ˜ π u is the permutation of the surviving items in N ( glyph[lscript],i ) in descending order of estimated reward for user u .

17: Append M ( glyph[lscript],i,j ) to M ( glyph[lscript] +1) and N ( glyph[lscript],i,j ) to N ( glyph[lscript] +1) .

18: end for

19: Compute m glyph[lscript] +1 , ν glyph[lscript] +1 as a function of glyph[lscript] . Set t = min( t 0 + m, T ) .

20: If t < T , invoke Algorithm PB-LATTICE (phase glyph[lscript] +1 , phase length m glyph[lscript] +1 , list of users M ( glyph[lscript] +1) , list of items N ( glyph[lscript] +1) , clusters C , rounds T , noise σ 2 , round index t 0 + m , gap factor ν glyph[lscript] +1 ). 21: end for

( D1 :) Each entry of V is sampled from a gaussian distribution N (0 , 25) with mean zero and variance 25 . User u on being recommended item j provides a random feedback distributed according to N ( P uj , 0 . 25) .

( D2 :) Each entry of V is sampled from a uniform distribution U (0 , 5) with range in [0 , 5] . User u on being recommended item j provides a random feedback distributed according to N ( P uj , 0 . 25) .

( D3 :) Each entry of V is sampled from a uniform distribution [0 . 05 , 0 . 95] with equal probability. User u on being recommended item j provides a like (+1) with probability P uj and a dislike ( -1) with probability 1 -P uj .

For the PB-LATTICE algorithm (Alg. 6), we choose the hyper-parameters on the phase lengths and gap parameters as m glyph[lscript] = 10 + 2 glyph[lscript] and ν glyph[lscript] = || P || ∞ / (8 · 2 glyph[lscript] ) . Since PB-LATTICE is a recursive algorithm, we initialize PB-LATTICE with phase index 1 , phase length 12 , list of users [[ M ]] , list of items [[ N ]] , clusters C , rounds T , noise σ 2 , round index 1 , gap factor || P || ∞ / 16 . For the greedy algorithm (Alg. 5), we experiment with two exploration periods ( m = 10 and m = 30 ). Finally, for the Collaborative-Greedy algorithm in [4], we choose θ = 0 . 5 and α = 0 . 5 .

Results and Insights: The cumulative reward until round t (defined as M -1 ∑ t j =1 ∑ M u =1 P uρ u ( j ) ) and the round wise reward at round t (defined as M -1 ∑ M u =1 P uρ u ( t ) ) is plotted for all the three datasets D1, D2, D3. in Figures 1a, 1b, 1c, 1d, 1e and 1f. Apart from obtaining good empirical rewards, PB-LATTICE has several practically relevant properties which are demonstrated through our experiments:

Note from Figures 1b and 1d that the greedy algorithm has a large cold-start period for good exploitation properties. If the exploration is too small, then the estimation becomes poor. On the other hand, PB-LATTICE improves in phases - therefore, it has a small cold-start period i.e. it starts recommending relevant items very quickly and also has good estimation guarantees

Collaborative-greedy in [4] proposes exploration rounds throughout the entire course of T rounds. This is often impractical as users will demand good recommendations throughout. This is demonstrated in Fig. 1f where the red curve (round-wise reward of CollaborativeGreedy) has several dips but the round-wise regret of PB-LATTICE stays high.

D Detailed Proof of Theorem 1

Next, we characterize some properties namely the condition number and incoherence of sub-matrices of P restricted to a nice subset of users. The following Lemmas 2 and 3 can be found in [25]

Lemma 2. Suppose Assumption 1 is true. Consider a sub-matrix P sub of P having non-zero singular values λ ′ 1 > · · · > λ ′ C ′ (for C ′ ≤ C ). Then, if the rows of P sub correspond to a nice subset of users, we have λ ′ 1 λ ′ C ′ ≤ λ 1 λ C √ τ .

Lemma 3. Suppose Assumption 1 is true. Consider a sub-matrix P sub ∈ R B ′ × A ′ (with SVD decomposition P sub = ˜ U ˜ Σ ˜ V ) of P whose rows correspond to a nice subset of users. Then, ∣ ∣ ∣ ∣ ∣ ∣ ˜ U ∣ ∣ ∣ ∣ ∣ ∣ 2 , ∞ ≤ √ C τ N ′ and ∣ ∣ ∣ ∣ ∣ ∣ ˜ V ∣ ∣ ∣ ∣ ∣ ∣ 2 , ∞ ≤ √ µ C α M ′ .

Lemmas 2 and 3 allow us to apply low rank matrix completion (Lemma 1) to relevant sub-matrices of the reward matrix P .

D.1 Main Analysis

Blocked LATTICE is run in phases consisting of exploit component and explore component indexed by glyph[lscript] = 1 , 2 , . . . . Note that the exploit component of a phase is followed by the explore component. However, for the first phase, the exploit component has a length of zero rounds. Importantly note that any phase (say glyph[lscript] ) for distinct nice subsets of users M ( glyph[lscript],i ) , M ( glyph[lscript],j ) will be run asynchronously as reward observations corresponding to one subset is not used to determine the policy for any other subset of users. At the beginning of the explore component of each phase glyph[lscript] , we have the following set of desirable properties:

(A) We will run the explore component of phase glyph[lscript] asynchronously and separately for a list of disjoint nice subsets of users M ( glyph[lscript] ) ≡ {M ( glyph[lscript],i ) , . . . , M ( glyph[lscript],a glyph[lscript] ) } and respective sets of arms N ( glyph[lscript] ) ≡ {N ( glyph[lscript], 1) , . . . , N ( glyph[lscript],a glyph[lscript] ) } where a glyph[lscript] ≤ C such that ∪ i ∈ [ a glyph[lscript] ] M ( glyph[lscript],i ) ⊆ [ M ] , M ( glyph[lscript],i ) ∩ M ( glyph[lscript],j ) = ∅ for any i, j ∈ [ a glyph[lscript] ] (i.e. the groups of users are nice subsets of [ M ] with no overlap) and N ( glyph[lscript],i ) ⊆ [ N ] (i.e. the active set of items N ( glyph[lscript],i ) for users in M ( glyph[lscript],i ) are a subset of [ N ] but the active sets can overlap). The sets { ( M ( glyph[lscript],i ) , N ( glyph[lscript],i ) } i ∈ [ a glyph[lscript] ] remain unchanged during the explore component of phase glyph[lscript] . As mentioned before, the round at which the explore component of the phase glyph[lscript] starts is different for each subset of nice users M ( glyph[lscript],i ) .

(B) At any round t ∈ [ T ] in a particular phase glyph[lscript] , let us denote by O ( glyph[lscript],t ) M ( glyph[lscript],i ) to be the set of items chosen for recommendation (see Step 5 in Alg. 3) in exploit components so far (from round t = 1 ) for users in M ( glyph[lscript],i ) . In other words, the items in O ( glyph[lscript],t ) M ( glyph[lscript],i ) have already been recommended to all users in M ( glyph[lscript],i ) up to glyph[lscript] th phase. In that case, we also maintain that

the best T / B items (golden items) not chosen for recommendation so far in the exploit phases for users in M ( glyph[lscript],i ) are contained in the set of surviving items i.e. if Z = [ N ] \ O ( glyph[lscript],t ) M ( glyph[lscript],i ) , then it must happen that

Note that eq. 8 implies that for every user u ∈ M ( glyph[lscript],i ) , there are sufficient golden items in N ( glyph[lscript],i ) to be recommended for the remaining rounds . To see this, note that there are T / B golden items at the beginning and no golden items which are unblocked are eliminated. Hence the number of remaining rounds at any point must be smaller than the number of possible allowed recommendations of golden items belonging to the surviving set of items.

(C) Furthermore, for all i ∈ [ a glyph[lscript] ] , the set N ( glyph[lscript],i ) must also satisfy the following:

where glyph[epsilon1] glyph[lscript] is a fixed exponentially decreasing sequence in glyph[lscript] (in particular, we choose glyph[epsilon1] 1 = || P || ∞ and glyph[epsilon1] glyph[lscript] = C ′ 2 -glyph[lscript] min ( ‖ P ‖ ∞ , σ √ µ log N ) for glyph[lscript] > 1 for some constant C ′ > 0 ).

Next, at the beginning of the exploit component of phase glyph[lscript] +1 , we will have the following set of desirable properties:

(a) At every round t in the exploit component of phase glyph[lscript] +1 , we maintain a list of disjoint nice subsets of users M ( glyph[lscript] +1) ≡ {M ( glyph[lscript] +1 , 1) , . . . , M ( glyph[lscript] +1 ,a glyph[lscript] +1 ) } (where ∪ i ∈ [ a glyph[lscript] +1 ] M ( glyph[lscript] +1 ,i ) ⊆ [ M ] ) and corresponding sets of items N ( glyph[lscript] +1 ,t ) ≡ {N ( glyph[lscript] +1 ,t, 1) , . . . , N ( glyph[lscript] +1 ,t,a glyph[lscript] +1 ) } where a glyph[lscript] +1 ≤ C and ∪ i ∈ [ a glyph[lscript] +1 ] N ( glyph[lscript] +1 ,t,i ) ⊆ [ N ] . Note that in the exploit component, we also use the round index in the superscript for item subsets as they can change during the exploit component (unlike the explore component).

(b) We ensure that for any user u ∈ M ( glyph[lscript] +1 ,i ) , the set of items chosen for recommendation in the exploit component of phase glyph[lscript] +1 belongs to the set of best T items i.e. { π u ( t ) } T t =1 .

Since LATTICE is random, we will say that our algorithm is ( glyph[epsilon1] glyph[lscript] , glyph[lscript] ) -good if at the beginning of the explore component of the glyph[lscript] th phase the algorithm can maintain a list of users and items satisfying properties A-C. Let us also define the event E ( glyph[lscript] ) 2 to be true if properties (A-C) are satisfied at the beginning of the explore component of phase glyph[lscript] by the phased elimination algorithm. We can show that if our algorithm is ( glyph[epsilon1] glyph[lscript] , glyph[lscript] ) -good then the low rank matrix completion step (denoted by the event E ( glyph[lscript] ) 3 ) in the explore component of phase glyph[lscript] is successful (i.e. the event E ( glyph[lscript] ) 3 is true) with high probability.

Conditioned on the aforementioned two events E ( glyph[lscript] ) 2 , E ( glyph[lscript] ) 3 , with probability 1 , during all rounds t of the exploit component in phase glyph[lscript] +1 , properties a -b are satisfied and the event E ( glyph[lscript] +1) 2 is going to be true. We are going to prove inductively that our algorithm is ( glyph[epsilon1] glyph[lscript] , glyph[lscript] ) -good for all phases indexed by glyph[lscript] for our choice of { glyph[epsilon1] glyph[lscript] } glyph[lscript] with high probability.

Base Case: For glyph[lscript] = 1 (the first phase), the number of rounds in the exploit component is zero and we start with the explore component. We initialize M (1 , 1) = [ N ] , N (1 , 1) = [ M ] and therefore, we have

Clearly, [ M ] is a nice subset of users and finally for every user u ∈ [ M ] , the best T / B items ( golden items ) { π u ( t ) } T / B t =1 belong to the entire set of items. Thus for glyph[lscript] = 1 , conditions A-C are satisfied at the beginning of the explore component and therefore the event E (1) 2 is true. Hence, our initialization makes the algorithm ( || P || ∞ , 1) -good.

Inductive Argument: Suppose, at the beginning of the phase glyph[lscript] , we condition on the events ⋂ glyph[lscript] j =1 E ( j ) 2 that Algorithm is ( glyph[epsilon1] j , j ) -good for all j ≤ glyph[lscript] . This means that conditions (A-C) are satisfied at the beginning of the explore component of all phases up to and including that of glyph[lscript] for each

reward sub-matrix (indexed by i ∈ [ a glyph[lscript] ] ) corresponding to the users in M ( glyph[lscript],i ) and items in N ( glyph[lscript],i ) . Next, our goal is to run low rank matrix completion in order to estimate each of the sub-matrices corresponding to { ( M ( glyph[lscript],i ) , N ( glyph[lscript],i ) ) } i ∈ [ a glyph[lscript] ] .

Exploration Strategy: Consider a particular nice subset of users M ( glyph[lscript],i ) and corresponding active items N ( glyph[lscript],i ) (s.t. |N ( glyph[lscript],i ) | ≥ T 1 / 3 ) at the beginning of the explore component of phase glyph[lscript] for M ( glyph[lscript],i ) . For each user u ∈ M ( glyph[lscript],i ) , we are going to sample each item j ∈ N ( glyph[lscript],i ) with probability p (to be determined based on the desired error guarantee). Suppose the set of indices sampled in the explore component of phase glyph[lscript] is denoted by Ω ( glyph[lscript] ) ⊆ M ( glyph[lscript],i ) ×N ( glyph[lscript],i ) . Now, for each user u ∈ M ( glyph[lscript],i ) we recommend all the unblocked items in the set A ( glyph[lscript] ) u ≡ { j ∈ N ( glyph[lscript],i ) | ( u, j ) | Ω ( glyph[lscript] ) } and obtain the corresponding noisy reward (note that for blocked items in the aforementioned set, we have already obtained the corresponding noisy rewards).

For simplicity, we intend to complete the explore component at the same round for all users in M ( glyph[lscript],i ) . If, for two users u, v ∈ M ( glyph[lscript],i ) , it happens that recommendation of all items in A ( glyph[lscript] ) u is complete for user u but recommendation of all items in A ( glyph[lscript] ) v is incomplete for user v , then for the remaining rounds we recommend to user u arbitrary unblocked items from N ( glyph[lscript],i ) . Note that this is always possible since the set N ( glyph[lscript],i ) has sufficiently many unblocked items allowing recommendations in the remaining rounds at beginning of explore component of phase glyph[lscript] . We start with the following lemma to characterize the round complexity of estimating the sub-matrix P M ( glyph[lscript],i ) , N ( glyph[lscript],i ) up to entry-wise error ∆ glyph[lscript] +1 with high probability using the noisy observations corresponding to the subset of indices Ω ( glyph[lscript] ) :

Lemma 4. Consider a particular subset of nice users M ( glyph[lscript],i ) and corresponding active items N ( glyph[lscript],i ) (such that min ( M / ( τ C ) , ∣ ∣ N ( glyph[lscript],i ) ∣ ∣ ) ≥ T 1 / 3 ) at the beginning of the explore component of phase glyph[lscript] . Suppose d 1 = max( |M ( glyph[lscript],i ) | , |N ( glyph[lscript],i ) | ) and d 2 = min( |M ( glyph[lscript],i ) | , |N ( glyph[lscript],i ) | ) . Let us fix ∆ glyph[lscript] +1 = Ω( σ √ µ 3 log d 1 / √ d 2 ) and condition on the event E ( glyph[lscript] ) 2 . Suppose Assumptions 1 and 2 are satisfied. In that case, in explore component of phase glyph[lscript] , by choosing 1 ≥ p = c ( σ 2 ˜ µ 3 log d 1 ∆ 2 glyph[lscript] +1 d 2 ) (for some constant c > 0 ) and using

rounds under the blocked constraint, we can compute an estimate ˜ P ( glyph[lscript] ) ∈ R M × N such that with probability 1 -O ( T -3 ) , we have

where σ 2 is the noise variance, µ is the incoherence of reward matrix P and ˜ µ is the incoherence factor of reward sub-matrix P M ( glyph[lscript],i ) , N ( glyph[lscript],i ) .

Proof of Lemma 4. We are going to use Lemma 1 in order to compute an estimate ˜ P ( glyph[lscript] ) M ( glyph[lscript],i ) , N ( glyph[lscript],i ) of the sub-matrix P M ( glyph[lscript],i ) , N ( glyph[lscript],i ) satisfying ∣ ∣ ∣ ∣ ∣ ∣ ˜ P ( glyph[lscript] ) M ( glyph[lscript],i ) , N ( glyph[lscript],i ) -P M ( glyph[lscript],i ) , N ( glyph[lscript],i ) ∣ ∣ ∣ ∣ ∣ ∣ ∞ ≤ ∆ glyph[lscript] +1 . Since M ( glyph[lscript],i ) is a nice subset of users, the cardinality of ∣ ∣ M ( glyph[lscript],i ) ∣ ∣ must be larger than M / ( τ C ) . Recall that τ is the ratio of the maximum cluster size and the minimum cluster size; M / C being the average cluster size implies that the minimum cluster size is bounded from above by M / ( τ C ) . From Lemma 1, we know that by using m glyph[lscript] = O ( p ∣ ∣ N ( glyph[lscript],i ) ∣ ∣ + √∣ ∣ N ( glyph[lscript],i ) ∣ ∣ p log( ∣ ∣ M ( glyph[lscript],i ) ∣ ∣ δ -1 ) ) rounds (see Lemma 1) restricted to users in M ( glyph[lscript],i ) such that with probability at least 1 -( δ + d -12 2 ) (see Remark 5),

where d 1 = max( |M ( glyph[lscript],i ) | , |N ( glyph[lscript],i ) | ) , d 2 = min( |M ( glyph[lscript],i ) | , |N ( glyph[lscript],i ) | ) and ˜ µ is the incoherence factor of the matrix P M ( glyph[lscript],i ) , N ( glyph[lscript],i ) . In order for the right hand side to be less than ∆ glyph[lscript] +1 , we can set

p = c ( σ 2 ˜ µ 3 log d 1 ∆ 2 glyph[lscript] +1 d 2 ) for some appropriate constant c > 0 . Since the event E ( glyph[lscript] ) 2 is true, we must have that ∣ ∣ M ( glyph[lscript],i ) ∣ ∣ ≥ M / ( τ C ); hence d 2 ≥ min ( M / ( τ C ) , ∣ ∣ N ( glyph[lscript],i ) ∣ ∣ ) . Therefore, we must have that

where we substitute δ -1 = 1 / poly ( T ) and furthermore, the condition of the Lemma statement implies that d -12 2 = O ( T -3 ) . Hence, we complete the proof of the lemma.

Remark 5 (Remark 1 in [8]) . The error probability can be reduced from O ( δ + d -3 2 ) to O ( δ + d -c 2 ) for any constant c > 0 with only constant factor changes in the round complexity m and the estimation guarantees ‖ P -˜ P ‖ ∞ . We will use c = 12 in rest of the paper.

Note that although ˜ µ , the incoherence factor of the sub-matrix P M ( glyph[lscript],i ) , N ( glyph[lscript],i ) is unknown, from Lemma 3, we know that ˜ µ is bounded from above by O ( µ ) (recall that C, α, τ = O (1) ).

Exploration Strategy continued: In particular, we choose ∆ glyph[lscript] +1 = glyph[epsilon1] glyph[lscript] / 176 C at the beginning of the explore component of phase glyph[lscript] for the set of users M ( glyph[lscript],i ) with active items N ( glyph[lscript],i ) . One edge case scenario is when for a particular set of nice users M ( glyph[lscript],i ) , at the beginning of the explore component of phase glyph[lscript] , we have ∣ ∣ N ( glyph[lscript],i ) ∣ ∣ ≤ T 1 / 3 . In this case, this set of nice users M ( glyph[lscript],i ) do not progress to the next phase and in the explore component, we simply recommend arbitrary items in N ( glyph[lscript],i ) to users in M ( glyph[lscript],i ) for the remaining rounds. Another interesting edge case scenario is when d 2 = min( ∣ ∣ M ( glyph[lscript],i ) ∣ ∣ , ∣ ∣ N ( glyph[lscript],i ) ∣ ∣ , ) is so small that the requisite error guarantee ∆ glyph[lscript] +1 (eq. 10) cannot be achieved even by setting p = 1 i.e. we recommend all the items in the active set. Recall that by our induction assumption, N ( glyph[lscript],i ) is sufficiently large so that it is possible to recommend unblocked items for the number of remaining rounds (say T -t glyph[lscript] with t glyph[lscript] being the round at which the explore component of phase glyph[lscript] starts). In that case, the set of users M ( glyph[lscript],i ) do not progress to the subsequent phase ; we simply recommend arbitrary unblocked items in the set N ( glyph[lscript],i ) to the users in M ( glyph[lscript],i ) for the remaining rounds. In both the above edge case scenarios, the explore component of phase glyph[lscript] for users in M ( glyph[lscript],i ) would last for the remaining rounds from where it starts. We can now show the following lemma:

Lemma 5. Consider a particular subset of nice users M ( glyph[lscript],i ) and corresponding surviving items N ( glyph[lscript],i ) at the beginning of the explore component of phase glyph[lscript] . Suppose d 1 = max( |M ( glyph[lscript],i ) | , |N ( glyph[lscript],i ) | ) and d 2 = min( |M ( glyph[lscript],i ) | , |N ( glyph[lscript],i ) | ) . Now ∆ glyph[lscript] +1 = glyph[epsilon1] glyph[lscript] / 176 C is such that there does not exist any p ∈ [0 , 1] for which RHS in eq. 3 can be ∆ glyph[lscript] +1 for estimation of matrix P restricted to the rows in M ( glyph[lscript],i ) and columns in N ( glyph[lscript],i ) . In that case, we must have for all users u ∈ M ( glyph[lscript],i ) ,

where t glyph[lscript] is the round at which the explore component of phase glyph[lscript] starts.

Proof. Note that by our induction hypothesis, we must have B ∣ ∣ N ( glyph[lscript],i ) ∣ ∣ ≥ T -t glyph[lscript] because the set of surviving items must contain sufficient unblocked items (for recommendation in the remaining rounds) for every user u ∈ M ( glyph[lscript],i ) . Hence, by setting p = 1 in eq. 3, with a certain number of rounds, we can obtain an estimate Q of P satisfying

Hence, this implies that our choice of ∆ glyph[lscript] +1 satisfies ∆ glyph[lscript] +1 = O ( σ √ µ 3 log d 1 √ d 2 ) implying that glyph[epsilon1] glyph[lscript] = O ( σ C √ µ 3 log d 1 √ d 2 ) . Now, there are two possibilities: 1) either d 2 = ∣ ∣ M ( glyph[lscript],i ) ∣ ∣ implying that M / ( τ C ) ≤

∣ ∣ M ( glyph[lscript],i ) ∣ ∣ ≤ ∣ ∣ N ( glyph[lscript],i ) ∣ ∣ . In that case, we have that glyph[epsilon1] glyph[lscript] = O ( σ C 1 . 5 √ µ 3 log d 1 √ M τ ) . Now, because of our induction assumption, we will have

Therefore, for any user u ∈ M ( glyph[lscript],i ) , if we recommend arbitrary unblocked items in N ( glyph[lscript],i ) for the remaining rounds then we can bound the following quantity

2) The second possibility is the following: d 2 = ∣ ∣ N ( glyph[lscript],i ) ∣ ∣ ≥ B -1 ( T -t glyph[lscript] ) . In that case, from our induction assumption, we have that ( B = Θ(log T ) )

and therefore

Consider M ′ ( glyph[lscript] ) ⊆ M ( glyph[lscript] ) to be the family of nice subsets of users which do not fall into the edge case scenarios i.e. 1) there exists 0 ≤ p ≤ 1 for which the theoretical bound in RHS in eq. 3 can be smaller than ∆ glyph[lscript] +1 2) we have ∣ ∣ N ( glyph[lscript],i ) ∣ ∣ ≥ T 1 / 3 . More precisely M ′ ( glyph[lscript] ) corresponds to the set

Suppose M / ( τ C ) = Ω( T 1 / 3 ) . As mentioned before, the event E ( glyph[lscript] ) 3 is true if the algorithm has successfully computed an estimate ˜ P ( glyph[lscript] ) ∈ R M × N such that for all M ( glyph[lscript],i ) ∈ M ′ ( glyph[lscript] )

implying that for each of the distinct nice subsets M ( glyph[lscript],i ) ∈ M ′ ( glyph[lscript] ) , after the explore component of phase glyph[lscript] , the algorithm finds a good entry-wise estimate of the sub-matrix P M ( glyph[lscript],i ) , N ( glyph[lscript],i ) . In the following part of the analysis, we will repeatedly condition on the events E ( glyph[lscript] ) 2 (conditions A-C are satisfied at the beginning of the explore component of phase glyph[lscript] ) and the event E ( glyph[lscript] ) 3 (eq. 10 is true for all nice subsets M ( glyph[lscript],i ) ∈ M ′ ( glyph[lscript] ) in the explore component of phase glyph[lscript] ). Note that the event E ( glyph[lscript] ) 2 is true due to the induction hypothesis and conditioned on the event E ( glyph[lscript] ) 2 , the event E ( glyph[lscript] ) 3 is true with probability at least 1 -O ( T -3 C ) (after taking a union bound over C clusters).

Fix any M ( glyph[lscript],i ) ⊆ M ′ ( glyph[lscript] ) and condition on the events E ( glyph[lscript] ) 2 , E ( glyph[lscript] ) 3 . For each user u ∈ M ( glyph[lscript],i ) , once the algorithm has computed the estimate ˜ P ( glyph[lscript] ) M ( glyph[lscript],i ) , N ( glyph[lscript],i ) in eq. 11, let us denote a set of good items for the user u by

where t is the round index at the end of the explore component of phase glyph[lscript] and |O ( glyph[lscript],t ) M ( glyph[lscript],i ) | is the number of rounds in exploit components of phases that the user u ∈ M ( glyph[lscript],i ) has encountered so far. Note from our algorithm, that users in the same nice subset M ( glyph[lscript],i ) have encountered exactly the same number of rounds in exploit components of phases until the end of phase glyph[lscript] . Recall that users in M ( glyph[lscript],i ) have been recommended the same set of items until blocked (unless blocked already for some user in which case the item in question has already been recommended B times) in the exploit components of phases until the end of phase glyph[lscript] (this set of items chosen for recommendation in the exploit components up to end of phase glyph[lscript] is denoted by O ( glyph[lscript],t ) M ( glyph[lscript],i ) ). If we condition on the event E ( glyph[lscript] ) 3 , then we can show the following statement to be true (in the following lemma, we remove the superscript t in O ( glyph[lscript],t ) M ( glyph[lscript],i ) for simplicity - the round index t corresponds to the end of phase glyph[lscript] for users in M ( glyph[lscript],i ) ):

Lemma 6. Condition on the events E ( glyph[lscript] ) 2 , E ( glyph[lscript] ) 3 being true. Consider a nice subset of users M ( glyph[lscript],i ) ∈ M ′ ( glyph[lscript] ) and their corresponding set of active items N ( glyph[lscript],i ) for which guarantees in eq. 11 holds. Let O ( glyph[lscript] ) M ( glyph[lscript],i ) be the set of items that have been chosen for recommendation to users in M ( glyph[lscript],i ) in exploit components of the first glyph[lscript] phases. Denote Z = [ N ] \ O ( glyph[lscript] ) M ( glyph[lscript],i ) . In that case, for every user u ∈ M ( glyph[lscript],i ) , the items π u ( s ) | Z for all s ∈ [ TB -1 -∣ ∣ ∣ O ( glyph[lscript] ) M ( glyph[lscript],i ) ∣ ∣ ∣ ] must belong to the set T ( glyph[lscript] ) u .

Proof. Let us fix a user u ∈ M ( glyph[lscript],i ) with active set of arms N ( glyph[lscript],i ) . Let us also fix an item a ≡ π u ( t ) | Z for t ∈ [ TB -1 -∣ ∣ ∣ O ( glyph[lscript] ) M ( glyph[lscript],i ) ∣ ∣ ∣ ] . Recall that ˜ π | Z is the permutation of the items sorted in descending order according to their estimated reward in ˜ P ( glyph[lscript] ) M ( glyph[lscript],i ) , N ( glyph[lscript],i ) . Now there are two possibilities regarding the position of the arm a in the permutation ˜ π | Z - 1) π u ( t ) | Z ≡ ˜ π u ( t 1 ) | Z for some t 1 ≤ t which implies that in the permutation ˜ π | Z , the position of the arm a is t 1 . In that case, the arm a survives in the set T ( glyph[lscript] ) u by definition (see eq. 12) 2) Now, suppose that π u ( t ) | Z ≡ ˜ π u ( t 1 ) | Z for some t 1 ≤ t which implies that in the permutation ˜ π | Z , the arm a has been shifted to the right (from π | Z ). In that case, there must exist another item b ≡ π u ( t 2 ) | Z for t 2 > t such that π u ( t 2 ) | Z ≡ ˜ π u ( t 3 ) | Z for t 3 ≤ t . Now, we will have

where we used the following facts a) conditioned on the event E ( glyph[lscript] ) 3 , we have ˜ P ( glyph[lscript] ) us -P us ≤ ∆ glyph[lscript] +1 for all u ∈ M ( glyph[lscript],i ) , s ∈ N ( glyph[lscript],i ) b) P ub -P ua ≤ 0 by definition. Hence, this implies that a ∈ T ( glyph[lscript] ) u .

Corollary 1. Condition on the events E ( glyph[lscript] ) 2 , E ( glyph[lscript] ) 3 being true. Consider a nice subset of users M ( glyph[lscript],i ) ∈ M ′ ( glyph[lscript] ) and their corresponding set of active items N ( glyph[lscript],i ) for which guarantees in eq. 11 holds. Suppose t glyph[lscript] is the final round of the explore component of phase glyph[lscript] for users in M ( glyph[lscript],i ) . In that case, for user u , the set T ( glyph[lscript] ) u contains sufficient items that are unblocked and can be recommended for the remaining T -t glyph[lscript] rounds.

Proof. From Lemma 11, we showed that T ( glyph[lscript] ) u comprises the set of items π u ( s ) | Z for all s ∈ [ TB -1 -∣ ∣ ∣ O ( glyph[lscript] ) M ( glyph[lscript],i ) ∣ ∣ ∣ ] where Z = [ N ] \ O ( glyph[lscript] ) M ( glyph[lscript],i ) . Now, suppose the k th item in the set H u ≡ { π u ( t ′ ) | Z} T -∣ ∣ ∣ O ( glyph[lscript] ) M ( glyph[lscript],i ) ∣ ∣ ∣ t ′ =1 has been recommended to user u b k < B times in previous phases. In that case, the number of allowed recommendations of items in T ( glyph[lscript] ) u is at least TB -1 -B ∣ ∣ ∣ O ( glyph[lscript] ) M ( glyph[lscript],i ) ∣ ∣ ∣ -∑ k ∈H u b k which is more than the remaining rounds that is at most TB -1 -B ∣ ∣ ∣ O ( glyph[lscript] ) M ( glyph[lscript],i ) ∣ ∣ ∣ -∑ k ∈H u b k ( B ∣ ∣ ∣ O ( glyph[lscript] ) M ( glyph[lscript],i ) ∣ ∣ ∣ rounds have already been used up when the golden items were identified and recommended).

Consider a nice subset of users M ( glyph[lscript],i ) ∈ M ′ ( glyph[lscript] ) and its corresponding active set of items N ( glyph[lscript],i ) for which eq. 11 holds true. At the end of the explore component of phase glyph[lscript] , based on the estimate ˜ P ( glyph[lscript] ) M ( glyph[lscript],i ) , N ( glyph[lscript],i ) , we construct a graph G ( glyph[lscript],i ) whose nodes are given by the users in M ( glyph[lscript],i ) . Now, we

draw an edge between two users u, v ∈ M ( glyph[lscript],i ) if ∣ ∣ ∣ ˜ P ( glyph[lscript] ) ux -˜ P ( glyph[lscript] ) vx ∣ ∣ ∣ ≤ 2∆ glyph[lscript] +1 for all considered items x ∈ N ( glyph[lscript],i ) .

Lemma 7. Condition on the events E ( glyph[lscript] ) 2 , E ( glyph[lscript] ) 3 being true. Consider a nice subset of users M ( glyph[lscript],i ) ∈ M ′ ( glyph[lscript] ) and their corresponding set of active items N ( glyph[lscript],i ) for which guarantees in eq. 11 holds. Consider the graph G ( glyph[lscript],i ) formed by the users in M ( glyph[lscript],i ) such that an edge exists between two users u, v ∈ M ( glyph[lscript],i ) if ∣ ∣ ∣ ˜ P ( glyph[lscript] ) ux -˜ P ( glyph[lscript] ) vx ∣ ∣ ∣ ≤ 2∆ glyph[lscript] +1 for all considered items x ∈ N ( glyph[lscript],i ) . Nodes in G ( glyph[lscript],i ) corresponding to users in the same cluster form a clique. Also, users in each connected component of the graph G ( glyph[lscript],i ) form a nice subset of users.

Proof. For any two users u, v ∈ M ( glyph[lscript],i ) belonging to the same cluster, consider an arm x ∈ N ( glyph[lscript],i ) . We must have

Now, consider two users u, v ∈ M ( glyph[lscript],i ) that belongs to different clusters P , Q respectively. Note that since the event E ( glyph[lscript] ) is true, M ( glyph[lscript],i ) is a union of clusters comprising P , Q . Furthermore, we have already established that nodes in G ( glyph[lscript],i ) (users in M ( glyph[lscript],i ) ) restricted to the same cluster form a clique. There every connected component of the graph G ( glyph[lscript],i ) can be represented as a union of a subset of clusters.

Lemma 8. Condition on the events E ( glyph[lscript] ) 2 , E ( glyph[lscript] ) 3 being true. Consider a nice subset of users M ( glyph[lscript],i ) ∈ M ′ ( glyph[lscript] ) and their corresponding set of active items N ( glyph[lscript],i ) for which guarantees in eq. 11 holds. In that case, for any subset Y ⊆ N ( glyph[lscript],i ) and any s ∈ [ |Y| ] , for every user u ∈ M ( glyph[lscript],i ) , we must have

Proof. Since, we condition on the event E ( glyph[lscript] ) 3 , we must have computed an estimate ˜ P ( glyph[lscript] ) , an estimate of P restricted to users in M ( glyph[lscript],i ) and items in N ( glyph[lscript],i ) such that

We can decompose the term being studied in the following manner:

Let us bound the quantity ˜ P ( glyph[lscript] ) u ˜ π u ( s ) |Y -˜ P ( glyph[lscript] ) uπ u ( s ) |Y . In order to analyze this quantity, we will consider a few cases. In the first case, suppose that π u ( s ) | Y = ˜ π u ( t 1 ) | Y for t 1 ≥ s . In that case, we will have that ˜ P ( glyph[lscript] ) u ˜ π u ( s ) |Y -˜ P ( glyph[lscript] ) uπ u ( s ) |Y ≥ 0 . In the second case, suppose π u ( s ) | Y = ˜ π u ( t 2 ) | Y for t 2 < s and π u ( t 3 ) | Y = ˜ π u ( s ) | Y for t 3 < s . In that case, we have

where we used the fact P uπ u ( t 3 ) |Y -P uπ u ( s ) |Y ≥ 0 . In the final case, we assume that π u ( s ) | Y = ˜ π u ( t 2 ) | Y for s > t 2 and π u ( t 3 ) | Y = ˜ π u ( s ) | Y for t 3 > s . This means that both the items π u ( s ) | Y , π u ( t 3 ) | Y have been shifted to the left in the permutation ˜ π u | Y . Hence,

Hence there must exist an element π u ( t 4 ) | Y such that t 4 < s and ˜ π u ( t 5 ) | Y = π u ( t 4 ) | Y for t 5 > s . In that case, we must have

Therefore, in this case, we get that ˜ P ( glyph[lscript] ) u ˜ π u ( s ) -˜ P ( glyph[lscript] ) uπ u ( s ) ≥ -4∆ glyph[lscript] +1 . Again, we will have that

By combining the above arguments, we have that

By a similar set of arguments involving triangle inequalities, we will also have

This completes the proof of the lemma.

We now show the following lemma characterizing the union of good items for a connected component of the graph G ( glyph[lscript],i ) . Recall that T -∣ ∣ ∣ O ( glyph[lscript] ) M ( glyph[lscript],i ) ∣ ∣ ∣ counts the number of rounds excluding the ones used up in exploit component so far up to the glyph[lscript] th phase.

Lemma 9. Condition on the events E ( glyph[lscript] ) 2 , E ( glyph[lscript] ) 3 being true. Consider a nice subset of users M ( glyph[lscript],i ) ∈ M ′ ( glyph[lscript] ) and their corresponding set of active items N ( glyph[lscript],i ) for which guarantees in eq. 11 holds. Consider a subset of users G ∈ M ( glyph[lscript],i ) forming a connected component. Fix any set Y = ⋃ u ∈G T ( glyph[lscript] ) g \ J for some J such that for every user u ∈ G , we have ˜ π u ( s ′ ) | N ( glyph[lscript],i ) ≡ ˜ π u ( s ) | Y for s ′ = TB -1 -∣ ∣ ∣ O ( glyph[lscript] ) M ( glyph[lscript],i ) ∣ ∣ ∣ and some common index s . In that case, we must have

Proof. Let us fix a user v ∈ G . We have max x,y ∈Y ˜ P vx -˜ P vy = ˜ P v ˜ π v (1) |Y -min y ∈Y ˜ P vy . Now, there are two possibilities for any y ∈ Y : first, suppose that y ∈ T ( glyph[lscript] ) v . In that case, we have

Recall that the set T ( glyph[lscript] ) v was constructed as

Since y ∈ T ( glyph[lscript] ) v , we can bound ˜ P v ˜ π v ( s ′ ) |T ( glyph[lscript] ) v -˜ P vy ≤ 2∆ glyph[lscript] +1 . Also, from the construction of T ( glyph[lscript] ) v , { ˜ π v ( r ) | N ( glyph[lscript],i ) } s r =1 is present in the set T ( glyph[lscript] ) v . Also, since T ( glyph[lscript] ) u is just a subset of N ( glyph[lscript],i ) , the positions of the items { ˜ π v ( r ) | N ( glyph[lscript],i ) } s r =1 do not change in the permutation corresponding to the items in T ( glyph[lscript] ) v sorted by estimated reward for v in decreasing order i.e. ˜ π v ( r ) | N ( glyph[lscript],i ) = ˜ π v ( r ) | T ( glyph[lscript] ) v for any r ∈ [ s ] . Hence ˜ P v ˜ π v ( s ) |Y -˜ P v ˜ π v ( s ′ ) |T ( glyph[lscript] ) v = 0 . Hence, by combining the above, we have

Next, consider the case when y glyph[negationslash]∈ T ( glyph[lscript] ) v . Hence, there must exist an user u such that y ∈ T ( glyph[lscript] ) u \ T ( glyph[lscript] ) v and u is connected to v via a path of length L ≤ 2 C -1 (since there are C clusters and users in the same cluster form a clique as proved in Lemma 7). In that case, we have the following decomposition:

Now, let us consider the terms pairwise. Due to our construction of the graph G ( glyph[lscript],i ) , for two users u, v connected via a path of length L , we must have ∣ ∣ ∣ ˜ P ( glyph[lscript] ) ux -˜ P ( glyph[lscript] ) vx ∣ ∣ ∣ ≤ 2 L ∆ glyph[lscript] +1 . Note that

where we used that ˜ P ( glyph[lscript] ) u ˜ π v (1) |Y -˜ P ( glyph[lscript] ) u ˜ π u (1) |Y ≤ 0 . Next, we have ˜ P v ˜ π u (1) |Y -˜ P v ˜ π u ( s ) |Y ≤ ˜ P u ˜ π u (1) |Y -˜ P u ˜ π u ( s ) |Y +4 L ∆ glyph[lscript] +1 . Furthermore, again we have ˜ π u ( r ) | N ( glyph[lscript],i ) = ˜ π u ( r ) | T ( glyph[lscript] ) u for any r ∈ [ s ] and therefore ˜ P u ˜ π u ( s ) |Y -˜ P u ˜ π u ( s ′ ) |T ( glyph[lscript] ) v = 0 . Finally, ˜ P u ˜ π u ( s ′ ) |T ( glyph[lscript] ) u -˜ P uy + ˜ P uy -˜ P vy ≤ 4 L ∆ glyph[lscript] +1 (since ˜ P u ˜ π u ( s ′ ) |T ( glyph[lscript] ) u -˜ P uy ≤ 2∆ glyph[lscript] +1 ). Hence by combining, we have that

Hence, we complete the proof of the lemma (by substituting L ≤ 2 C ).

Lemma 10. Condition on the events E ( glyph[lscript] ) 2 , E ( glyph[lscript] ) 3 being true. Consider a nice subset of users M ( glyph[lscript],i ) ∈ M ′ ( glyph[lscript] ) and their corresponding set of active items N ( glyph[lscript],i ) for which guarantees in eq. 11 holds. Consider two users u, v ∈ M ( glyph[lscript],i ) having an edge i.e. ∣ ∣ ∣ ˜ P ( glyph[lscript] ) ux -˜ P ( glyph[lscript] ) vx ∣ ∣ ∣ ≤ 2∆ glyph[lscript] +1 for all x ∈ N ( glyph[lscript],i ) . In that case, for any subset Y ⊆ N ( glyph[lscript],i ) and any s ∈ [ |Y| ] , for every user u ∈ M ( glyph[lscript],i ) , we must have

Proof. We can decompose the term being studied in the following manner:

Let us bound the quantity ˜ P ( glyph[lscript] ) v ˜ π v ( s ) |Y -˜ P ( glyph[lscript] ) v ˜ π u ( s ) |Y . In order to analyze this quantity, we will consider a few cases. In the first case, suppose that ˜ π u ( s ) | Y = ˜ π v ( t 1 ) | Y for t 1 ≥ s . In that case, we will have that ˜ P ( glyph[lscript] ) v ˜ π v ( s ) |Y -˜ P ( glyph[lscript] ) v ˜ π u ( s ) |Y ≥ 0 . In the second case, suppose ˜ π u ( s ) | Y = ˜ π v ( t 2 ) | Y for t 2 < s and ˜ π u ( t 3 ) | Y = ˜ π v ( s ) | Y for t 3 < s . In that case, we have

where we used the fact ˜ P ( glyph[lscript] ) u ˜ π v ( s ) |Y -˜ P ( glyph[lscript] ) u ˜ π u ( s ) |Y = ˜ P ( glyph[lscript] ) u ˜ π u ( t 3 ) |Y -˜ P ( glyph[lscript] ) u ˜ π u ( s ) |Y ≥ 0 . In the final case, we assume that ˜ π u ( s ) | Y = ˜ π v ( t 2 ) | Y for s > t 2 and ˜ π u ( t 3 ) | Y = ˜ π v ( s ) | Y for t 3 > s . This means that both the items ˜ π u ( s ) | Y , ˜ π u ( t 3 ) | Y have been shifted to the left in the permutation ˜ π v | Y . Hence,

Hence there must exist an element ˜ π u ( t 4 ) | Y such that t 4 < s and ˜ π v ( t 5 ) | Y = ˜ π u ( t 4 ) | Y for t 5 > s . In that case, we must have

Therefore, in this case, we get that ˜ P ( glyph[lscript] ) v ˜ π v ( s ) -˜ P ( glyph[lscript] ) vπ u ( s ) ≥ -8∆ glyph[lscript] +1 . Hence we get

Also, we will have that

By combining the above arguments, we have that

By a similar set of arguments involving triangle inequalities, we will also have

This completes the proof of the lemma.

Now, for any fixed subset Y ⊆ N ( glyph[lscript],i ) , let us define the set R ( glyph[lscript] ) u | Y for user u below:

Hence, R ( glyph[lscript] ) u | Y corresponds to the set of items for user u that is close to the item with the highest estimated reward for user u restricted to the set Y at the end of the explore component of phase glyph[lscript] .

Lemma 11. Condition on the events E ( glyph[lscript] ) 2 , E ( glyph[lscript] ) 3 being true. Consider a nice subset of users M ( glyph[lscript],i ) ∈ M ′ ( glyph[lscript] ) and their corresponding set of active items N ( glyph[lscript],i ) for which guarantees in eq. 11 holds. Fix any subset Y ⊆ N ( glyph[lscript],i ) . In that case, for every user u ∈ M ( glyph[lscript],i ) , the item with the highest reward π u (1) | Y in the set Y must belong to the set R ( glyph[lscript] ) u | Y . Moreover, max s,s ′ ∈R ( glyph[lscript] ) u |Y | P us -P us ′ | ≤ 4∆ glyph[lscript] +1 .

Proof. Let us fix a user u ∈ M ( glyph[lscript],i ) with active set of arms N ( glyph[lscript],i ) . Now, we will have

which implies that π u (1) | Y ∈ R ( glyph[lscript] ) u | Y . Here we used the fact that ˜ P ( glyph[lscript] ) u ˜ π u (1) |Y -P u ˜ π u (1) |Y ≤ ∆ glyph[lscript] +1 , P u |Y -˜ P ( glyph[lscript] ) uπ u (1) |Y ≤ ∆ glyph[lscript] +1 and P u ˜ π u (1) |Y -P uπ u (1) |Y ≤ 0 . Next, notice that for any s, s ′ ∈ R ( glyph[lscript] ) u | Y

Lemma 12. Condition on the events E ( glyph[lscript] ) 2 , E ( glyph[lscript] ) 3 being true. Consider a nice subset of users M ( glyph[lscript],i ) ∈ M ′ ( glyph[lscript] ) and their corresponding set of active items N ( glyph[lscript],i ) for which guarantees in eq. 11 holds. Fix any subset Y ⊆ N ( glyph[lscript],i ) . Consider two users u, v ∈ M ( glyph[lscript],i ) having an edge in the graph G ( glyph[lscript],i ) . Conditioned on the events E ( glyph[lscript] ) 2 , E ( glyph[lscript] ) 3 , we must have

Proof. From the construction of G ( glyph[lscript],i ) , we know that users u, v ∈ M ( glyph[lscript],i ) have an edge if ∣ ∣ ∣ ˜ P ( glyph[lscript] ) ux -˜ P ( glyph[lscript] ) vx ∣ ∣ ∣ ≤ 2∆ glyph[lscript] +1 (and therefore | P ux -P vx | ≤ ∣ ∣ ∣ ˜ P ( glyph[lscript] ) ux -P ux ∣ ∣ ∣ + ∣ ∣ ∣ P vx -˜ P ( glyph[lscript] ) vx ∣ ∣ ∣ + ∣ ∣ ∣ ˜ P ( glyph[lscript] ) ux -˜ P ( glyph[lscript] ) vx ∣ ∣ ∣ ≤ 4∆ glyph[lscript] +1 ) for all x ∈ N ( glyph[lscript],i ) . For simplicity of notation, let us denote R u to

be the set R ( glyph[lscript] ) u | Y . Suppose P uπ u (1) |Y = a and P vπ v (1) |Y = b . Consider any pair of items x ∈ R u , y ∈ R v respectively. Therefore, for x ∈ R u this must mean that P ux ≥ a -4∆ glyph[lscript] +1 (note that π u (1) | Y ∈ R u ). For y ∈ R v , we must similarly have P vy ≥ b -4∆ glyph[lscript] +1 . Since u, v are connected by an edge, we must have P vx ≥ a -8∆ glyph[lscript] +1 and P uy ≥ b -8∆ glyph[lscript] +1 . Now, we have P vπ v (1) |Y ≥ P vx implying that b ≥ a -8∆ glyph[lscript] +1 ; hence

Asimilar analysis for v shows that P vy -P vx ≤ 16∆ glyph[lscript] +1 . This completes the proof of the lemma.

Lemma 13. Condition on the events E ( glyph[lscript] ) 2 , E ( glyph[lscript] ) 3 being true. Consider a nice subset of users M ( glyph[lscript],i ) ∈ M ′ ( glyph[lscript] ) and their corresponding set of active items N ( glyph[lscript],i ) for which guarantees in eq. 11 holds. Fix any subset Y ⊆ N ( glyph[lscript],i ) . Consider two users u, v ∈ M ( glyph[lscript],i ) having a path of length L in the graph G ( glyph[lscript],i ) . Conditioned on the events E ( glyph[lscript] ) 2 , E ( glyph[lscript] ) 3 , we must have

Proof. Recall from the construction of G ( glyph[lscript],i ) that conditioned on E ( glyph[lscript] ) 3 users u, v ∈ M ( glyph[lscript],i ) have an edge if ∣ ∣ ∣ ˜ P ( glyph[lscript] ) ux -˜ P ( glyph[lscript] ) vx ∣ ∣ ∣ ≤ 2∆ glyph[lscript] +1 (and therefore | P ux -P vx | ≤ 4∆ glyph[lscript] +1 ) for all x ∈ N ( glyph[lscript],i ) . Again, for simplicity of notation, let us denote R u to be the set R ( glyph[lscript] ) u | Y . Suppose P uπ u (1) |Y = a and P vπ v (1) |Y = b . Consider any pair of items x ∈ R u , y ∈ R v respectively. Therefore, for x ∈ R u this must mean that P ux ≥ a -4∆ glyph[lscript] +1 . For y ∈ R v , we must similarly have P vy ≥ b -4∆ glyph[lscript] +1 . Since u, v are connected by an path of length L (say a 1 , a 2 , . . . , a ( L -1) ), we must have P a 1 x ≥ a -8∆ glyph[lscript] +1 , P a 2 x ≥ a -12∆ glyph[lscript] +1 and finally P vx ≥ a -4( L + 1)∆ glyph[lscript] +1 . By a similar analysis P uy ≥ b -4( L +1)∆ glyph[lscript] +1 . Now, we have P vπ v (1) |Y ≥ P vx implying that b ≥ a -4( L +1)∆ glyph[lscript] +1 ; hence

Again, a similar analysis for v shows that P vy -P vx ≤ 8( L +1)∆ glyph[lscript] +1 . This completes the proof of the lemma.

Corollary 2. Condition on the events E ( glyph[lscript] ) 2 , E ( glyph[lscript] ) 3 being true. Consider a nice subset of users M ( glyph[lscript],i ) ∈ M ′ ( glyph[lscript] ) and their corresponding set of active items N ( glyph[lscript],i ) for which guarantees in eq. 11 holds. Fix any subset Y ⊆ N ( glyph[lscript],i ) . Consider two users u, v ∈ M ( glyph[lscript],i ) having a path in the graph G ( glyph[lscript],i ) . Conditioned on the events E ( glyph[lscript] ) 2 , E ( glyph[lscript] ) 3 , we must have

Proof. The proof follows from the fact that any two users u, v ∈ M ( glyph[lscript],i ) connected via a path must have a shortest path of length at most 2C -1 conditioned on the events E glyph[lscript] 2 , E ( glyph[lscript] ) 3 . This is because, from Lemma 13, we know that users in the same cluster form a clique and since there are at most C clusters, the shortest path must be of length at most 2 C -1 .

Hence, for a particular set of users M ( glyph[lscript],i ) ∈ M ′ ( glyph[lscript] ) , consider the j th connected component of G ( glyph[lscript],i ) comprising of users M ( glyph[lscript],i,j ) . If we consider the union of items S ≡ ∪ u ∈M ( glyph[lscript],i,j ) R ( glyph[lscript] ) u | Y for any subset Y | N ( glyph[lscript],i ) , then from Corollary 2, we must have that for any user u ∈ M ( glyph[lscript],i,j ) ,

This follows from the fact that every element s ∈ S must exist in R ( glyph[lscript] ) v | Y for some v ∈ M ( glyph[lscript],i,j ) ; moreover, v is connected to u and therefore the shortest path joining them must be of length at most 2 C -1 . Finally we use Corollary 2 to conclude equation 15.

Lemma 14. Condition on the events E ( glyph[lscript] ) 2 , E ( glyph[lscript] ) 3 being true. Consider a nice subset of users M ( glyph[lscript],i ) ∈ M ′ ( glyph[lscript] ) and their corresponding set of active items N ( glyph[lscript],i ) for which guarantees in eq. 11 holds. Consider the j th connected component of the graph G ( glyph[lscript],i ) comprising of users M ( glyph[lscript],i,j ) . Let N ( glyph[lscript],i,j ) ≡ ∪ u ∈M ( glyph[lscript],i,j ) T ( glyph[lscript] ) u denote the union of good items for users in M ( glyph[lscript],i,j ) . In that case,

For every user u ∈ M ( glyph[lscript],i,j ) , the items π u ( s ) | [ N ] \O ( glyph[lscript] ) M ( glyph[lscript],i ) for s ∈ [ TB -1 -∣ ∣ ∣ O ( glyph[lscript] ) M ( glyph[lscript],i ) ∣ ∣ ∣ ] must belong to the set N ( glyph[lscript],i,j ) i.e. the golden items that have not been chosen for recommendation to users in M ( glyph[lscript],i,j ) in exploit components of previous phases must belong to the surviving set of items N ( glyph[lscript],i,j ) .

For any subset Y ⊆ N ( glyph[lscript],i,j ) and any s ≤ |Y| , we must have the following for any A > 0 :

Proof. The proof of the first part follows directly from Lemma 11 where we showed that for a particular user u ∈ M ( glyph[lscript],i ) , the items P uπ u ( s ) | [ N ] \O ( glyph[lscript] ) M ( glyph[lscript],i ) for s ∈ [ TB -1 -∣ ∣ ∣ O ( glyph[lscript] ) M ( glyph[lscript],i ) ∣ ∣ ∣ ] must belong to the set T ( glyph[lscript] ) u (and the fact that N ( glyph[lscript],i,j ) ⊇ T ( glyph[lscript] ) u ).

We move on to the proof of the second part of the lemma. Recall that any two users u, v ∈ M ( glyph[lscript],i ) have an edge in the graph G ( glyph[lscript],i ) if ∣ ∣ ∣ ˜ P ( glyph[lscript] ) ux -˜ P ( glyph[lscript] ) vx ∣ ∣ ∣ ≤ 2∆ glyph[lscript] +1 for all x ∈ N ( glyph[lscript],i ) . In that case, we have

We are going to prove the Lemma statement by induction on length of the shortest path joining the users u, v . The base case (when the path length is 1 i.e. u, v are joined by an edge) is proved above. Suppose the statement is true when length of the shortest path is L -1 . In that case, we have the following set of inequalities (suppose w is the neighbor of v and the length of the shortest path joining u, w is L -1 )

The lemma statement follows from the fact that the length of the shortest path between users u, v in the same connected component is at most 2 C -1 . This completes the proof of the lemma.

Partition of M ( glyph[lscript],i ) in phase glyph[lscript] + 1 into nice subsets of users and their corresponding active subsets of items: Consider a nice set of users M ( glyph[lscript],i ) at the end of the explore component of phase glyph[lscript] i.e. the start of the subsequent phase glyph[lscript] +1 for the users in M ( glyph[lscript],i ) . At this point, conditioned on events E ( glyph[lscript] ) 2 , E ( glyph[lscript] ) 3 , our goal is to further partition the users in M ( glyph[lscript],i ) into more nuanced nice subsets. Suppose the number of rounds is at least T 1 / 3 / B = ˜ O ( T 1 / 3 ) implying that the active set of items N ( glyph[lscript],i ) is of size at least ˜ Ω( T 1 / 3 ) (induction assumption property B). Suppose, we index the connected components of the graph G ( glyph[lscript],i ) formed by the users in M ( glyph[lscript],i ) ∈ M ′ ( glyph[lscript] ) . Now, for each index j , the set of users corresponding to the j th connected component of the graph {G ( glyph[lscript],i ) } forms the j th nice subset of users (Lemma 7) stemming from users in M ( glyph[lscript],i ) ∈ M ′ ( glyph[lscript] ) - let us denote this set of users by M ( glyph[lscript] +1 ,z ) for some index s > 0 . For this set of users M ( glyph[lscript] +1 ,z ) , at the start of the exploit component of phase glyph[lscript] +1 (round t ), we define the active set of items N ( glyph[lscript] +1 ,t,z ) to be ⋃ u ∈M ( glyph[lscript] +1 ,z ) T ( glyph[lscript] ) u . Hence { ( M ( glyph[lscript] +1 ,z ) , N ( glyph[lscript] +1 ,t,z ) } z forms the family of nice sets of users (that progress to the ( glyph[lscript] +1) th phase) and their corresponding active set of items at the beginning ( exploit component) of phase glyph[lscript] +1 for users stemming from M ′ ( glyph[lscript] ) . Next, we discuss our recommendation strategy for M ( glyph[lscript] +1 ,z ) (a nice subset of users) in the exploit component of phase glyph[lscript] +1 .

Strategy in exploit component of phase glyph[lscript] +1 : Note that in the exploit component of phase glyph[lscript] +1 for users in M ( glyph[lscript],i,j ) ≡ M ( glyph[lscript] +1 ,z ) (new notation indicating that M ( glyph[lscript],i,j ) ≡ M ( glyph[lscript] +1 ,z ) is a nice subset of users at phase glyph[lscript] + 1 ) for some indices i, j and z , we follow a recursive approach to identify and recommend items in { π u ( t ) } TB -1 t =1 for all users u ∈ M ( glyph[lscript] +1 ,z ) . At the beginning of the exploit component of phase glyph[lscript] + 1 (say round t ), we will also initialize O ( glyph[lscript] +1 ,t ) M ( glyph[lscript] +1 ,z ) to be the set of items O ( glyph[lscript] ) M ( glyph[lscript],i ) (i.e. the golden items that have been chosen for recommendation in the exploit components of previous phases ( 1 -glyph[lscript] ) to users in M ( glyph[lscript],i ) and recommended sufficiently enough number of times to be blocked). At start of the exploit component of phase glyph[lscript] +1 (round t ), recall that the active set of items is given by N ( glyph[lscript] +1 ,t,z ) ≡ ⋃ u ∈M ( glyph[lscript] +1 ,z ) T ( glyph[lscript] ) u . We reiterate here that t corresponds to the index of the starting round in the exploit component of phase glyph[lscript] +1 for users in the nice subset M ( glyph[lscript] +1 ,z ) .

Let us denote s = TB -1 -∣ ∣ ∣ O ( glyph[lscript] +1 ,t ) M ( glyph[lscript] +1 ,z ) ∣ ∣ ∣ and Y = N ( glyph[lscript] +1 ,t,z ) . First of all, note that from Lemma 14, for all users u ∈ M ( glyph[lscript] +1 ,z ) the items π u ( r ) | [ N ] \ O ( glyph[lscript] +1 ,t ) M ( glyph[lscript] +1 ,z ) for r ∈ [ s ] must belong to the set Y i.e. the best s items among those that are not in the set O ( glyph[lscript] +1 ,t ) M ( glyph[lscript] +1 ,z ) must survive in Y (Lemma 14). Now, we look at two possibilities:

(Possibility A): For all users u ∈ M ( glyph[lscript] +1 ,z ) , we have that ˜ P u ˜ π u (1) |Y -˜ P u ˜ π u ( s ) |Y ≤ 64 C ∆ glyph[lscript] +1 In that case, we stop the exploit component of phase glyph[lscript] + 1 and move on to the explore component of phase glyph[lscript] + 1 for users in M ( glyph[lscript] +1 ,z ) and active items Y . Conditioned on events E ( glyph[lscript] ) 2 , E ( glyph[lscript] ) 3 , in Lemma 15, we show that the above condition implies for every user u ∈ M ( glyph[lscript] +1 ,z ) , we must have

Furthermore, for Z = [ N ] \ O ( glyph[lscript] +1) M ( glyph[lscript] +1 ,z ) , it must happen that Y ⊇ { π u ( s ) | Z} T / B -∣ ∣ ∣ O ( glyph[lscript] +1) M ( glyph[lscript] +1 ,z ) ∣ ∣ ∣ s =1 for every user u ∈ M ( glyph[lscript] +1 ,z ) . In other words, for each user u ∈ M ( glyph[lscript] +1 ,z ) , the set Y must contain all the top TB -1 golden items ( { π u ( r ) } TB -1 r =1 ) that were not recommended in the exploit components so far.

(Possibility B): For some user u ∈ M ( glyph[lscript] +1 ,z ) , we have that ˜ P u ˜ π u (1) |Y -˜ P u ˜ π u ( s ) |Y ≥ 64 C ∆ glyph[lscript] +1 . In that case, from Lemma 14, we know that for every user v ∈ M ( glyph[lscript] +1 ,z ) , we must have P vπ v (1) |Y -P vπ v ( s ) |Y ≥ 56 C ∆ glyph[lscript] +1 . In that case, if we consider the set of items S ≡ ∪ u ∈M ( glyph[lscript] +1 ,z ) R ( glyph[lscript] ) u | Y , then from Lemma 13 (or see eq. 15), we must have that for every user u ∈ M ( glyph[lscript] +1 ,z ) ,

For every user u ∈ M ( glyph[lscript] +1 ,z ) , recall that in Lemma 11, we showed that π u (1) | Y ∈ R ( glyph[lscript] ) u | Y and in Lemma 14, we showed that π u (1) | Y = π u (1) | [ N \ ∣ ∣ ∣ O ( glyph[lscript] +1 ,t ) M ( glyph[lscript] +1 ,z ) ∣ ∣ ∣ ] . Hence, π u (1) | Y = π u (1) | [ N \ ∣ ∣ ∣ O ( glyph[lscript] +1 ,t ) M ( glyph[lscript] +1 ,z ) ∣ ∣ ∣ ] belongs to the set S for every user u ∈ M ( glyph[lscript] +1 ,z ) .

Hence we must have that the items S is a subset of { π u ( r ) | [ N ] \ O ( glyph[lscript] +1 ,t ) M ( glyph[lscript] +1 ,z ) } s r =1 . Suppose we index the items in S . For each of the subsequent B |S| rounds (indexed by b ∈ [ B S ] ), for every user u ∈ M ( glyph[lscript] +1 ,z ) , we go to the glyph[ceilingleft] ( b/ B ) glyph[ceilingright] th item in S and recommend it to user u if unblocked. On the other hand, if the glyph[ceilingleft] ( b/ B ) glyph[ceilingright] th item in S is blocked (or becomes blocked) for the user u ∈ M ( glyph[lscript] +1 ,z ) , then we simply recommend any unblocked item in N ( glyph[lscript] +1 ,t,z ) . This is always possible because we will prove via induction (see Lemma 15 and in particular eq. 23) that at every round in the exploit component of phase glyph[lscript] +1 , the number of unblocked items for any user u ∈ M ( glyph[lscript] +1 ,z ) in the set N ( glyph[lscript] +1 ,t,z ) (where t is the previous decision round for M ( glyph[lscript] +1 ,z ) on whether possibility A or B is true) is always larger than the number

N

of remaining rounds. We make the following updates:

i.e we update the set O ( glyph[lscript] +1 ,t ) M ( glyph[lscript] +1 ,z ) by taking union with the set of |S| identified items in { π u ( t ) } T t =1 for all users u ∈ M ( glyph[lscript] +1 ,z ) . After these |S| rounds, for the set of users M ( glyph[lscript] +1 ,z ) , the set of active items N ( glyph[lscript] +1 ,t,z ) is pruned by removing the items in S and the time index is increased from t to t + |S| .

At this point, we repeat the same process again for users in M ( glyph[lscript] +1 ,z ) with the pruned set of active items N ( glyph[lscript] +1 ,t,s ) i.e. we check for possibility A or possibility B . If we encounter possibility B , then we again find the set of items S ≡ ∪ u ∈M ( glyph[lscript] +1 ,z ) R ( glyph[lscript] ) u | Y and recommended it to all users in M in |S| B steps as outlined above. We do this step recursively until we encounter Step A for the users in M ( glyph[lscript] +1 ,z ) and at that point we exit the exploit component of phase glyph[lscript] +1 and enter the explore component of phase glyph[lscript] +1 .

As before, at the beginning of the explore component of phase glyph[lscript] + 1 for the nice subset of users M ( glyph[lscript] +1 ,z ) , let us denote the set of active items by N ( glyph[lscript] +1 ,z ) and the set of items considered for recommendation in the exploit phases including the ( glyph[lscript] +1) th one by O ( glyph[lscript] +1) M ( glyph[lscript] +1 ,z ) (i.e. we remove the t in the superscript for simplicity). Therefore, at the end of the explore component of phase glyph[lscript] +1 for the nice subset of users M ( glyph[lscript] +1 ,z ) , the set of active items N ( glyph[lscript] +1 ,z ) satisfy the following:

Lemma 15. Consider a nice subset of users M ( glyph[lscript] +1 ,z ) and their corresponding set of active items N ( glyph[lscript] +1 ,z ) at the end of the exploit stage of phase glyph[lscript] + 1 i.e. for all users u ∈ M ( glyph[lscript] +1 ,z ) , we have ˜ P u ˜ π u (1) |N ( glyph[lscript] +1 ,z ) -˜ P u ˜ π u ( s ) |N ( glyph[lscript] +1 ,z ) ≤ 64 C ∆ glyph[lscript] +1 for s = TB -1 -∣ ∣ ∣ O ( glyph[lscript] +1) M ( glyph[lscript] +1 ,z ) ∣ ∣ ∣ . Suppose M ( glyph[lscript] +1 ,z ) is comprised of the users in a connected component of the graph G ( glyph[lscript],i ) which in turn is formed by the users in M ( glyph[lscript],i ) for which guarantees in eq. 11 holds true i.e. we condition on the events E ( glyph[lscript] ) 2 , E ( glyph[lscript] ) 3 being true. In that case, we must have that

for all users u ∈ M ( glyph[lscript] +1 ,z ) ,

i.e. the best and worst items in the set N ( glyph[lscript] +1 ,z ) for any user u ∈ M ( glyph[lscript] +1 ,z ) has close rewards.

Denote Z = [ N ] \ O ( glyph[lscript] +1) M ( glyph[lscript] +1 ,z ) to be set of items not chosen for recommendation to users in M ( glyph[lscript] +1 ,z ) in the exploit component of phases until (and including) phase glyph[lscript] +1 . Then it must happen that

Proof. Suppose at the end of the explore component of phase glyph[lscript] , M ( glyph[lscript] +1 ,z ) ⊆ M ( glyph[lscript],i ) . We will prove a more general statement. Consider the rounds t 1 , t 2 , . . . at which we check for possibility A or possibility B (this includes the starting and ending rounds of the exploit component of phase glyph[lscript] +1 ). At any such round t r , for all users u ∈ M ( glyph[lscript] +1 ,z ) , we must have that ˜ π u ( TB -1 -∣ ∣ ∣ O ( glyph[lscript],t r ) M ( glyph[lscript],i ) ∣ ∣ ∣ ) | N ( glyph[lscript],i ) ∈ ( glyph[lscript] +1 ,t r ,z )

and

for s = T -∣ ∣ ∣ O ( glyph[lscript] +1 ,t r ) M ( glyph[lscript] +1 ,z ) ∣ ∣ ∣ We will prove the statement above via induction on the recursions performed in the exploit component of the phase glyph[lscript] +1 . For the base case, we consider the round t 1 which corresponds to the beginning of the exploit component of phase glyph[lscript] + 1 . At this round, recall that N ( glyph[lscript] +1 ,t 1 ,z ) = ∪ u ∈M ( glyph[lscript] +1 ,z ) T ( glyph[lscript] ) u . Of course, with s = T -∣ ∣ ∣ O ( glyph[lscript] ) M ( glyph[lscript],i ) ∣ ∣ ∣ , for every user u ∈ M ( glyph[lscript] +1 ,z ) , we must have { ˜ π u ( r ) | N ( glyph[lscript],i ) } s r =1 ∈ T ( glyph[lscript] ) u (Lemma 11) and since N ( glyph[lscript] +1 ,z ) is a union of the sets T ( glyph[lscript] ) u , ˜ π u ( r ) | N ( glyph[lscript],i ) = ˜ π u ( r ) | N ( glyph[lscript] +1 ,z ) for all r ∈ [ s ] . Hence by invoking Lemma 9 (in the statement of Lemma 9, we have J = φ i.e. Y = N ( glyph[lscript] +1 ,z ) and ˜ π u ( s ) | N ( glyph[lscript],i ) = ˜ π u ( s ) | N ( glyph[lscript] +1 ,z ) for s = T -∣ ∣ ∣ O ( glyph[lscript],t r ) M ( glyph[lscript],i ) ∣ ∣ ∣ ), we obtain the statement of the Lemma for round t 1 .

Suppose the induction statement is true for round t a and the second possibility i.e. possibility B became true. In that case, we do not exit the recursion and our goal is to show that the lemma statement is true at the next decision round t a +1 . The induction hypothesis implies that for every user u ∈ M ( glyph[lscript] +1 ,z ) , we have that ˜ π u ( TB -1 -∣ ∣ ∣ O ( glyph[lscript] ) M ( glyph[lscript],i ) ∣ ∣ ∣ ) survives in the set of items N ( glyph[lscript] +1 ,t a ,z ) and furthermore, we have ˜ π u ( TB -1 -∣ ∣ ∣ O ( glyph[lscript] ) M ( glyph[lscript],i ) ∣ ∣ ∣ ) | N ( glyph[lscript],i ) = ˜ π u ( s ) | N ( glyph[lscript] +1 ,t a ,z ) for s = TB -1 -∣ ∣ ∣ O ( glyph[lscript] +1 ,t a ) M ( glyph[lscript] +1 ,z ) ∣ ∣ ∣ (note that s is common for all users in M ( glyph[lscript] +1 ,z ) ). The induction hypothesis also implies that

Again, at the decision round t a , since the possibility B was true, for one of the users u ∈ M ( glyph[lscript] +1 ,z ) , we must have for s = T -∣ ∣ ∣ O ( glyph[lscript] +1 ,t a +1 ) M ( glyph[lscript] +1 ,z ) ∣ ∣ ∣ ,

implying that for every user v ∈ M ( glyph[lscript] +1 ,z ) , we have

The above equation further implies that (Lemma 8), for every user v ∈ M ( glyph[lscript] +1 ,z ) , we will have

Hence, as mentioned before, if we consider the set of items S ≡ ∪ u ∈M ( glyph[lscript] +1 ,z ) R ( glyph[lscript] ) u | N ( glyph[lscript] +1 ,t a ,z ) , then from Lemma 13 (or see eq. 15), we must have that for every user u ∈ M ( glyph[lscript] +1 ,z ) , π u (1) | N ( glyph[lscript] +1 ,t a ,z ) ∈ S and

Hence, if we remove the set S to update N ( glyph[lscript] +1 ,t a ,z ) i.e. N ( glyph[lscript] +1 ,t a +1 ,z ) ←N ( glyph[lscript] +1 ,t a ,z ) \ S , for every user u ∈ M ( glyph[lscript] +1 ,z ) , we must have

This is because for each user u , only elements which have larger rewards than ˜ π u ( TB -1 -∣ ∣ ∣ O ( glyph[lscript] +1 ,t a ) M ( glyph[lscript] +1 ,z ) ∣ ∣ ∣ ) | N ( glyph[lscript] +1 ,t a ,z ) are removed which makes the aforementioned item survive in N ( glyph[lscript] +1 ,t a +1 ,z ) and also moves its position up (recall that O ( glyph[lscript] +1 ,t a +1 ) M ( glyph[lscript] +1 ,z ) ← O ( glyph[lscript] +1 ,t a ) M ( glyph[lscript] +1 ,z ) ∪ S ) in the list of surviving items sorted in decreasing order by expected reward. Hence, we can apply Lemma

9 to conclude the first part of the induction proof. In order to show the final statement, with s = TB -1 -∣ ∣ ∣ O ( glyph[lscript] +1 ,t r ) M ( glyph[lscript] +1 ,z ) ∣ ∣ ∣ , we can simply substitute that

when possibility A became true at a decision round t r and we exit the exploit component to enter the explore component of phase glyph[lscript] +1 for users in M ( glyph[lscript] +1 ,z ) .

Moreover, we will also have that

u

∈M

(

glyph[lscript]

+1

,z

)

This first implication is due to our induction hypothesis (see eq.25) which implies that the best s items in the smaller set N ( glyph[lscript] +1 ,t a ,z ) is same as the best s = TB -1 -∣ ∣ ∣ O ( glyph[lscript] +1 ,t a ) M ( glyph[lscript] +1 ,z ) ∣ ∣ ∣ items in the larger set [ N ] \ O ( glyph[lscript] +1 ,t a ) M ( glyph[lscript] +1 ,z ) . Hence, it is evident that the set S must also be a subset of the best TB -1 items ( golden items ) for user u namely { π u ( t ) } t ∈ [ TB -1 ] . Since the above facts are true for all users u ∈ M ( glyph[lscript] +1 ,z ) , we can also conclude that the new pruned set of items N ( glyph[lscript] +1 ,t a +1 ,z ) at the next decision round t a +1 is a superset of the best TB -1 -∣ ∣ ∣ O ( glyph[lscript] +1 ,t a +1 ) M ( glyph[lscript] +1 ,z ) ∣ ∣ ∣ items for every user u . This completes the second part of the induction proof.

Lemma 16. Conditioned on the events E ( glyph[lscript] ) 2 , E ( glyph[lscript] ) 3 , with choice of ∆ glyph[lscript] +1 = glyph[epsilon1] glyph[lscript] +1 / 88 C , conditions A-C will be satisfied at the beginning of the explore component of phase glyph[lscript] for the different nice subsets {M ( glyph[lscript] +1 ,z ) } z and their corresponding set of active arms {N ( glyph[lscript] +1 ,z ) } z implying that the event E ( glyph[lscript] +1) 2 will be true.

Proof. 1. Proof of condition A: Due to our induction hypothesis, condition A is true for the explore component of phase glyph[lscript] . Hence, the explore component of phase glyph[lscript] is implemented separately and asynchronously for each disjoint nice subset of users M ( glyph[lscript],i ) ∈ M ′ ( glyph[lscript] ) ⊆ M ( glyph[lscript] ) (recall that M ′ ( glyph[lscript] ) corresponds to the nice subsets of users which do not fall into the edge case scenarios). Let us fix one such nice subset of users M ( glyph[lscript],i ) ∈ M ′ ( glyph[lscript] ) . After the successful low rank matrix completion step (event E ( glyph[lscript] ) 3 is true), we find the connected components of a graph G ( glyph[lscript],i ) which in turn correspond to nice subsets of users as well (see Lemma 7). Since the above facts are true for all nice subsets of users in M ′ ( glyph[lscript] ) , the nice subsets of users that progress to the ( glyph[lscript] +1) th phase are disjoint. Since these set of users are not modified during the exploit component of phase glyph[lscript] +1 , condition A is true at the beginning of the explore component of phase glyph[lscript] +1 .

Proof of conditions B and C: Again, let us fix a subset of nice users M ( glyph[lscript] +1 ,z ) that has progressed to phase glyph[lscript] +1 and was in turn a part of the nice subset of users M ( glyph[lscript],i ) in phase glyph[lscript] . In other words, the set of users M ( glyph[lscript] +1 ,z ) corresponds to a connected component of the graph G ( glyph[lscript],i ) . From Lemma 15, we can conclude that conditions B and C are true at the beginning of the explore component of phase glyph[lscript] +1 for users in M ( glyph[lscript] +1 ,z ) with choice of ∆ glyph[lscript] +1 = glyph[epsilon1] glyph[lscript] +1 / 88 C where glyph[epsilon1] glyph[lscript] +1 was pre-determined. Therefore, the conditions B and C hold for all nice subsets of users {M ( glyph[lscript] +1 ,z ) } z that have progressed to phase glyph[lscript] +1 .

Hence, conditioned on the events E ( glyph[lscript] ) 2 , E ( glyph[lscript] ) 3 , with choice of ∆ glyph[lscript] +1 = glyph[epsilon1] glyph[lscript] +1 / 88 C , the algorithm will be ( glyph[epsilon1] glyph[lscript] +1 , glyph[lscript] +1) -good and the event E ( glyph[lscript] +1) 2 will be true.

D.2 Analyzing the regret guarantee

Lemma 17. Consider a fixed decreasing sequence { glyph[epsilon1] glyph[lscript] } glyph[lscript] ≥ 1 where glyph[epsilon1] 1 = || P || ∞ and glyph[epsilon1] glyph[lscript] = C ′ 2 -glyph[lscript] min ( ‖ P ‖ ∞ , σ √ µ log N ) for glyph[lscript] > 1 for some constant C ′ > 0 . Let us denote the event E = ⋂ glyph[lscript] E ( glyph[lscript] ) 2 ⋂ glyph[lscript] E ( glyph[lscript] ) 3 to imply that our algorithm is ( glyph[epsilon1] glyph[lscript] , glyph[lscript] ) -good at all phases indexed by glyph[lscript] and the explore components of all phases are successful with the length of the explore component of phase glyph[lscript] being

The above statement implies that for any nice subset of users M ( glyph[lscript],i ) that has progressed to the glyph[lscript] th phase with active items N ( glyph[lscript],i ) at the beginning of the explore components, with m glyph[lscript] rounds, we can compute an estimate ˜ P M ( glyph[lscript],i ) , N ( glyph[lscript],i ) of P M ( glyph[lscript],i ) , N ( glyph[lscript],i ) satisfying

In that case, the event E is true with probability at least 1 -CT -2 .

Proof. Notice that

where we used the following facts 1) Pr( E (1) c 2 ) = 0 and Pr( E ( glyph[lscript] ) c 2 | E ( glyph[lscript] -1) 2 , E ( glyph[lscript] -1) 3 ) = 0 for all glyph[lscript] (Lemma 16) 2) Pr( E ( glyph[lscript] ) c 3 | E ( glyph[lscript] ) 2 ) ≤ CT -2 implied from Lemma 4 with additional union bounds over the number of phases (at most the number of rounds T ) and the number of disjoint nice subsets of users that have progressed in each phase (at most the number of clusters C ). An important fact to keep in mind is that the above analysis is possible since the observations used to compute estimates are never repeated in Alg. 2 and we are able to avoid complex dependencies.

Now, we are ready to prove our main regret bound. Suppose we condition on the event E as defined in Lemma 17. Conditioned on the event E , let us denote by ρ u to be some sequence of items recommended to the user u ∈ [ M ] by our algorithm. The probability of this sequence of items being recommended is Pr( ∩ u ∈ [ M ] ρ u | E ) .

D.2.1 Swapping argument

Let us fix a particular user u ∈ [ M ] and a sequence of recommended items ρ u such that ρ u ( t ) ∈ [ N ] is the item recommended to user u at round t . For sake of analysis, we will construct a permutation θ u : [ T ] → { ρ u ( t ) } t ∈ [ T ] of the items { ρ u ( t ) } t ∈ [ T ] with sequential modifications ( θ u is initialized with ρ u ). For any phase indexed by glyph[lscript] , consider the exploit and explore components for a nice subset of users M ( glyph[lscript],i ) . The sequence of items recommended to user u during the explore component remain unchanged i.e. for any phase glyph[lscript] , θ u ( t ) = ρ u ( t ) for all rounds t ∈ [ t ] such that t corresponds to the explore component of phase glyph[lscript] for the user u .

Now, for the exploit component in phase glyph[lscript] , recall that O ( glyph[lscript] ) M ( glyph[lscript],i ) \ O ( glyph[lscript] -1) M ( glyph[lscript],i ) is the set of items chosen for recommendation particularly in the exploit component of phase glyph[lscript] (we remove the super-script t since we refer to the end of the exploit components in phase glyph[lscript] and phase glyph[lscript] -1 respectively). Suppose at round t in the explore component of phase glyph[lscript] , the b th item in the set O ( glyph[lscript] ) M ( glyph[lscript],i ) \ O ( glyph[lscript] -1) M ( glyph[lscript],i ) was chosen

to be recommended to all users in M ( glyph[lscript],i ) but was found to be blocked for user u ∈ M ( glyph[lscript],i ) before it could be recommended B times. Let us also denote N ( glyph[lscript],t a ,i ) to be set of active items for users in M ( glyph[lscript],i ) at round t (i.e. t a was the previous decision round where it was decided whether possibility A or possibility B was true). Since the b th item in the set O ( glyph[lscript] ) M ( glyph[lscript],i ) \ O ( glyph[lscript] -1) M ( glyph[lscript],i ) was blocked for user u , instead we recommend any unblocked item for user u from the current set of active items N ( glyph[lscript],t a ,i ) . This is always possible; we showed in Lemma 15 (see eq. 23) that the active set of items always contain sufficient unblocked items for possible recommendations for remaining rounds for any user in the corresponding nice subset of users during the exploit component. Now there are two possibilities:

( b th item in the set O ( glyph[lscript] ) M ( glyph[lscript],i ) \O ( glyph[lscript] -1) M ( glyph[lscript],i ) was recommended in round t ′ in the explore component of previous phase glyph[lscript] ′ ) We consider the active set of items N ( glyph[lscript] ′ ,h ) (such that u ∈ M ( glyph[lscript] ′ ,h ) in phase glyph[lscript] ′ ) where glyph[lscript] ′ < glyph[lscript] is the phase index (and t ′ is the round index) when the b th item in O ( glyph[lscript] ) M ( glyph[lscript],i ) \ O ( glyph[lscript] -1) M ( glyph[lscript],i ) (say a ) was recommended to the user u during the explore component. Let us denote the item that we have recommended as replacement to user u at round t by a ′ . Note that since a ′ ∈ N ( glyph[lscript],t a ,i ) , it must happen that a ′ ∈ N ( glyph[lscript] ′ ,h ) since N ( glyph[lscript],t a ,i ) ⊆ N ( glyph[lscript] ′ ,h ) . In that case, we have

We swap the above items so that in the modified sequence, we have

The goal of the swapping operation at round t in the exploit component is to modify the sequence of items such that 1) the item a chosen for recommendation at round t is assigned to round t in the exploit component 2) the item a actually recommended in round t ′ < t and phase glyph[lscript] ′ < glyph[lscript] is replaced by another item a ′ (actually recommended at round t > t ′ in phase glyph[lscript] > glyph[lscript] ′ ) such that both a, a ′ belongs to the same set of active items N ( glyph[lscript] ′ ,h ) ( u ∈ M ( glyph[lscript] ′ ,h ) for some h ). We will also say that the item a chosen for recommendation is replaced by a swapping operation of length 1 . This is because the chosen element for recommendation a was recommended in the explore component of a previous phase. A more precise definition of the length of a swapping operation is provided below.

( b th item in the set O ( glyph[lscript] ) M ( glyph[lscript],i ) \O ( glyph[lscript] -1) M ( glyph[lscript],i ) was recommended in the exploit component of previous phase): This is the more difficult case. As before, let us denote the b th item in the set O ( glyph[lscript] ) M ( glyph[lscript],i ) \ O ( glyph[lscript] -1) M ( glyph[lscript],i ) by a . Of course, the item a could not have been chosen for recommendation in the exploit component of a previous phase (i.e. a cannot belong in the set O ( glyph[lscript] ′ ) M ( glyph[lscript] ′ ,j ) \ O ( glyph[lscript] ′ -1) M ( glyph[lscript] ′ ,j ) for any glyph[lscript] ′ < glyph[lscript] with u ∈ M ( glyph[lscript] ′ ,j ) in phase glyph[lscript] ′ .) In that case, the item a was recommended in the exploit component of phase glyph[lscript] as part of a swapping operation . With this intuition in mind, let us define length of a swapping operation precisely:

Definition 3 (Length of swapping operation) . For a user u , suppose a 1 ∈ O ( glyph[lscript] ) M ( glyph[lscript],i ) \O ( glyph[lscript] -1) M ( glyph[lscript],i ) is an item chosen for recommendation in the exploit component of phase glyph[lscript] but found to be blocked. In that case, we will say that a 1 is replaced by a swapping operation of length L +1 if there exists L +1 items a 1 , a 2 , . . . , a L , a L + 1 , L + 1 phases p 1 > p 2 > · · · > p L > p L +1 and respective rounds t 1 , t 2 , . . . , t L , t L +1 such that 1) a 1 is chosen for recommendation in exploit component of phase p 1 at round t 1 2) for each 2 ≤ i < L , a i -1 has been recommended in the exploit component of phase p i at round t i when the intended item chosen for recommendation was a i 3) a L has been recommended in the explore component of phase p L +1 at round t L +1 3) a L +1 is the item recommended to user u in place of a 1 in phase p 1 at round t 1 .

As before, a L +1 belongs to current set of active items N ( glyph[lscript],t a ,i ) at round t 1 for users in M ( glyph[lscript],i ) (recall that t a is the decision round just prior to t 1 ). In the above definition, note that L must be finite since there are a finite number of components and the first phase only has an explore component (recall that exploit component in the first phase has zero rounds).

For a user u ∈ M ( glyph[lscript],i ) , suppose a 1 ∈ O ( glyph[lscript] ) M ( glyph[lscript],i ) \ O ( glyph[lscript] -1) M ( glyph[lscript],i ) is an item chosen for recommendation in the exploit component of phase glyph[lscript] but found to be blocked. Moreover suppose a 1 must be replaced by a swapping operation of length L (see Definition 3 for notations). In that case, we make the following modifications to the permutation θ u :

Lemma 18. Condition on the event E . Consider the modified sequence of distinct items { θ u ( t ) } t ∈ [ T ] for a certain fixed user u . For any phase glyph[lscript] to which the user u has progressed as part of the nice subset M ( glyph[lscript],i ) , consider the exploit component starting from round index t exploit , start ,glyph[lscript] to t exploit , end ,glyph[lscript] . In that case, the set of elements { θ u ( t ) | t ∈ [ t exploit , start ,glyph[lscript] , t exploit , end ,glyph[lscript] ] } is equivalent to the set of elements O ( glyph[lscript] ) M ( glyph[lscript],i ) \O ( glyph[lscript] -1) M ( glyph[lscript],i ) repeated B times i.e. the golden items chosen for recommendation in the exploit component of phase glyph[lscript] to users in M ( glyph[lscript],i ) . Similarly, consider the explore component starting from round index t explore , start ,glyph[lscript] to t explore , end ,glyph[lscript] . In that case, for any t ∈ [ t explore , start ,glyph[lscript] , t explore , end ,glyph[lscript] ] , it must happen that θ u ( t ) ∈ N ( glyph[lscript],i ) i.e. the set of active items for M ( glyph[lscript],i ) during the explore component of phase glyph[lscript] .

Proof. We start with the following claim.

Claim 1. For a user u ∈ M ( glyph[lscript],i ) , suppose a 1 ∈ O ( glyph[lscript] ) M ( glyph[lscript],i ) \ O ( glyph[lscript] -1) M ( glyph[lscript],i ) is an item chosen for recommendation in the exploit component of phase glyph[lscript] at round t 1 but found to be blocked for user u . Moreover suppose a 1 has been replaced by a swapping operation of length L +1 (see Definition 3 for notations). In that case, we already have θ u ( t i ) = a i for all 2 ≤ i ≤ L . We only need to modify the sequence by making the following two changes: 1) θ u ( t 1 ) = a 1 2) θ u ( t L +1 ) = a L +1 . Note that in the true sequence, a L +1 has been recommended in the round t 1 replacing the intended item a 1 .

Proof. We will prove the claim via induction. Notation-wise, suppose for any glyph[lscript] ′ ≤ glyph[lscript] , the user u belongs to the set M ( glyph[lscript] ′ ,i ) and the set of active items at round t 1 for users in M ( glyph[lscript],i ) is denote by N (for simplicity, we remove the superscripts). The base case for L = 0 is true by construction - here the item a 1 is recommended in the explore component of phase p 2 at round t 2 . We find an unblocked item a 2 in the set N ⊆ N ( glyph[lscript],i ) (note that items are never added to the active set across phases and only pruned), recommend it at round t 1 according to our algorithm. For analysis, we modify

Now, suppose our claim is true for some L = l. Now, for L = l +1 , note that a 1 was recommended in the exploit component of phase p 2 at round t 2 ; this implies that a 1 must have been used to replace another item a 2 via a swapping operation of length l . By our induction hypothesis, we must have θ u ( t i ) = a i for all 2 ≤ i ≤ L (as a matter of fact, we will have θ u ( t L +1 ) = a 1 ). Therefore, only the pair of modifications θ u ( t 1 ) = a 1 and θ u ( t L +1 ) = a L +1 suffice to bring the desired changes in θ u .

We can also conclude from Claim 1 that 1) at any round t in the exploit component of some phase for user u , if the chosen item to be recommended is found to be blocked, then that chosen item is brought to the t th position in the sequence θ u 2) Once the chosen item is brought to its correct position in θ u , it will not be modified/moved in any future round. 3) All chosen items for recommendation to user u corresponding to exploit components are moved to their correct position (i.e. the intended round for their recommendation) in the sequence θ u . Next we make the following claim:

Claim 2. Consider the setting in Claim 1. It must happen that all the items a 1 , a 2 , . . . , a L +1 must belong to the set N ( p L +1 ,i ) - the set of active items in the phase p L +1 for users in the nice subset M ( p L +1 ,i ) to which u belongs and a L was recommended in its explore component.

Proof. Again, note that items are never added to the active set across phases and only pruned. If an item a 1 is replaced by a L +1 via a swapping operation of length L + 1 , it implies that a 1 was used to replace item a 2 (via a swapping operation of length L ), a 2 was used to replace a 3 (via a swapping operation of length L -1 ) and so on. The final golden item for user u in this sequence a L was recommended in phase p L +1 in the explore component and belonged to the set of active items N ( p L +1 ,i ) . Hence, this implies that all the subsequent surviving items a 1 , a 2 , a 3 , . . . , a L -1 , a L +1 must have belonged to the set of active items N ( p L +1 ,i ) as well.

From Claim 2, we can conclude that for any round t in the explore component of some phase glyph[lscript] for user u , if ρ u ( t ) is replaced in the sequence θ u at round t , then ρ u ( t ) , θ u ( t ) belong to the same set of active items N ( glyph[lscript],i ) . With both these arguments, we complete the proof of the lemma.

D.2.2 Final Regret analysis

Let us condition on the event E . Consider any user u ∈ [ M ] - recall that { ρ u ( t ) } t ∈ [ T ] is the random variable denoting the sequence of T items recommended to the user u , ρ u ( t ) is a realization of { ρ u ( t ) } t ∈ [ T ] conditioned on the event E . Furthermore, θ u is the modified sequence - a permutation of the T items { ρ u ( t ) } recommended for user u . For simplicity of notation, we will assume that u ∈ M ( glyph[lscript],i ) for every phase glyph[lscript] . Hence the active set of items during the explore component of phase glyph[lscript] for the nice subset of users M ( glyph[lscript],i ) is denoted by N ( glyph[lscript],i ) . Also, in line with our previous usage of notations, let us denote O ( glyph[lscript] ) M ( glyph[lscript],i ) \ O ( glyph[lscript] -1) M ( glyph[lscript],i ) to be the set of items chosen for recommendation particularly in the exploit component of phase glyph[lscript] . Moreover, O denotes the entire set of items chosen for recommendation in all the exploit components to user u . In other words, if glyph[lscript] u is the final phase to which user u has progressed then O u = O ( glyph[lscript] u ) M ( glyph[lscript]u,i ) . We denote the regret for the user u by

where the expectation is over the randomness in the algorithm and the noise in the observations. Note that with this notation, we have Reg ( T ) | E = M -1 ∑ u ∈ [ M ] Reg u ( T ) | E . We now have the following set of inequalities for regret of user u :

Let us denote the set of rounds in the exploit component of glyph[lscript] th phase by exploit ( glyph[lscript], M ( glyph[lscript],i ) ) and the explore component of glyph[lscript] th phase by explore ( glyph[lscript], M ( glyph[lscript],i ) ) . Therefore we can further decompose the regret for user u as follows:

Next, our arguments are conditioned on the event E and any sequence ρ u ( t ) having a non-zero probability of appearing conditioned on event E . Recall in Lemma 15, we proved by induction that O u ⊆ { π u ( t ) } T t =1 . Moreover, in Lemma 18, we proved that items chosen for recommendation in the exploit components for user u are in their correct positions (i.e. the round when they were intended to be recommended but might have been found to be blocked) in the sequence θ u . Consider a permutation of the best T items for user u σ u : [ T ] →{ π u ( t ) } T t =1 such that

where in a round in any exploit component, the permutation σ maps the item chosen for recommendation (which we know to be among the best T items) for user u to that round. For any round belonging to the explore component, the permutation σ arbitrarily maps the remaining items among the best T items (namely the set { π u ( t ′ ) } t ′ ∈ [ T ] \O u ). Notice that ∑ t ∈ [ T ] P uπ u ( t ) = ∑ t ∈ [ T ] P uσ u ( t ) . Therefore, we can further decompose the regret as

glyph[negationslash]

For the explore component in the phase indexed by glyph[lscript] , we have proved in Lemma 14 that the active set of items N ( glyph[lscript],i ) is a superset of { π u ( t ′ ) } t ′ ∈ [ T ] \ O u . Therefore, we can bound (using Lemma 4 and condition C stated at beginning of sec. D.1) for any glyph[lscript] = glyph[lscript] u ( glyph[lscript] u denotes index of the final phase that user u was part of)

where ˜ µ is the incoherence factor of the sub-matrix P M ( glyph[lscript],i ) , N ( glyph[lscript],i ) . Next, using the facts that ∆ glyph[lscript] +1 = glyph[epsilon1] glyph[lscript] +1 / 88 C , 2 glyph[epsilon1] glyph[lscript] = glyph[epsilon1] glyph[lscript] +1 , C , τ = O (1) , we get that (after hiding log factors for simplicity)

In the above statement, from Lemmas 2 and 3, we also used the fact that ˜ µ = O ( µ ) and the condition number of the sub-matrix P M ( glyph[lscript],i ) , N ( glyph[lscript],i ) is O (1) . Finally, for the explore component of the final phase glyph[lscript] u , from Lemma 11, we will also have (in the final phase, one of the edge case scenarios might appear)

Hence, we can put together everything to conclude that

where J is the number of phases with glyph[epsilon1] glyph[lscript] ≥ Φ . By choosing Φ = √ σ 2 µ 3 T max ( 1 , N M ) , we can bound

where we used the fact that N glyph[greatermuch] T and therefore the last term in the previous equation is a lower order term compared to the first two ones. Next, recall that we choose glyph[epsilon1] glyph[lscript] = C ′ 2 -glyph[lscript] min ( ‖ P ‖ ∞ , σ √ µ log N ) (so that the condition on σ > 0 in Lemma 1 is automatically satisfied for all glyph[lscript] ) for some constant C ′ > 0 , the maximum number of phases glyph[lscript] for which glyph[epsilon1] glyph[lscript] > Φ can be bounded from above by J = O ( log ( 1 Φ min ( ‖ P ‖ ∞ , σ √ µ log N ))) . Therefore, we can hide J inside ˜ O and obtain

Therefore, we must have that

Finally, we use the fact that

From Lemma 17, we know that Pr( E c ) ≤ CT -2 , Reg ( T ) | E c ≤ T || P || ∞ therefore,

E Proof of Lower Bound (Theorem 2)

Consider our problem setting with M users, N items and T rounds, blocking constraint B , noise variance proxy σ 2 = 1 and all expected rewards in [0 , 1] . Here we consider C = 1 i.e. all users belong to the same cluster. Let us denote a particular policy chosen by the recommendation system as π that belongs to the class of polices Π . Moreover, let us also denote by E the set of possible environments corresponding to the expected reward matrices that has all rows to be same (satisfies cluster structure for C = 1 ). In that case, the minimax regret is given by

E.1 Lower Bound via reduction

We can ease the problem by assuming that at each round t = 1 , 2 , . . . , T , the users come in a sequential fashion - the j th user is recommended an item based on all previous history of observations including the feedback obtained from recommending items from users 1 , 2 , . . . , j -1 at round t . Furthermore, we also assume that the N items can be partitioned into NB / T known groups where each group has identical items - hence we have a simple multi-armed bandit problem (MAB) with NB / T arms and MT rounds with a single user. A lower bound on this simplified MAB problem will imply a lower bound on our setting i.e by appropriately normalizing, we have

where we simply used the standard regret lower bound in multi-armed bandits with K arms and T rounds which is Ω( √ KT ) [19].

E.2 Lower Bound via application of Fano's inequality

As before, we can ease the original problem by assuming that at each round t = 1 , 2 , . . . , T , the users come in a sequential fashion - the j th user is recommended an item based on all previous history of observations including the feedback obtained from recommending items from users 1 , 2 , . . . , j -1 at round t . Clearly, a lower bound on the simplified problem will imply a lower bound on our setting i.e by appropriately normalizing, we have

Furthermore, ignoring the normalizing factor by M , for C = 1 , the simplified problem is equivalent to a standard Multi-armed bandit (MAB) problem with a single agent with MT rounds and an additional hard constraint that each item can be pulled at most MB times. We will construct ( N TB -1 ) environments in the following way: let T ≡ {S ⊆ [ N ] | |S| = TB -1 } be the set of all subsets of [ N ] of size TB -1 . Now for each subset S ∈ T , we construct an environment by assuming that the agent on pulling any arm in the set S observes a random reward distributed according to N (∆ , 1) and on pulling any arm outside the set S observes a random reward distributed according to N (0 , 1) . This corresponds to the the reward matrix P (in our original problem) having an entry ∆ in the TB -1 columns indexed in S and 0 in the remaining columns. Let E S , P S , E S denote the expectation, probability measure and the environment if S ∈ T is the set of chosen columns for constructing the environment.

Next we assume that the set S is chosen uniformly at random from T . Hence we must have

Fix any policy π ∈ Π . Condition on the set S being selected from T . Let R ( S ) be the number of times arms indexed in the set S are pulled in the MT rounds. In that case we must have

Now, consider the estimation problem of which set S was selected from T . Let ̂ X be an estimator that takes as input the observations in the MT rounds and returns a set ̂ S in the following way: it

finds ̂ S as the set of T arms that have been pulled the most number of times jointly and returns ̂ S if R ( ̂ S ) ≥ 3 MT / 4 and the null set ∅ otherwise. We consider the estimator ̂ X to make an error if it returns a set ̂ S such that ̂ S ∩S ≤ T / 4 B . If the estimator ̂ X makes an error, note that R ( ̂ S ) ≥ 3 MT / 4 implies that R ( ̂ S \ S ) ≥ MT / 2 (since each arm can be pulled at most M times) - hence, it implies that R ( S ) ≤ MT / 2 . Therefore, if we denote Error as the error event, then we must have

Therefore, our goal is to bound the quantity E S∼T P S ( Error ) . At this point we have reduced our problem to a multiple hypothesis testing problem. Therefore, in order to lower bound the probability of the event Error , we use Fano's inequality for approximate recovery:

Lemma (Fano's inequality with approximate recovery [28]) . For any random variables V, ̂ V on alphabets V , ̂ V , consider an error when d ( V, ̂ V ) ≥ t for some t > 0 and distance function d : V × ̂ V → R . In that case, if we denote the error event by Error , we must have

where G max = max ̂ v ∈ ̂ V ∑ v ∈V 1[ d ( v, ˆ v ) ≤ t ] and I ( V ; ̂ V ) is the mutual information between the random variables V and ̂ V .

In the special case when the random variable V is uniform, then we can upper bound the mutual information by I ( V ; ̂ V ) ≤ max v, ̂ v ∈V KL ( P ̂ V | V = v || P ̂ V | V = v ′ ) ≤ max v, ̂ v ∈V KL ( P v || P v ′ ) where the second inequality follows from Data-processing inequality ( P v = P ( · | v ) corresponds to the probability of the observations given V = v ).

Next, we apply it to our setting to prove an estimation error lower bound for our designed estimator ̂ X . In our setting, G max corresponds to the maximum possible number of sets in T that have intersection of size more than T / 4 B with some fixed set S ∈ T . Clearly we have

Furthermore, in our setting, we also have that

- this follows from the fact that the distributions P S , P S ′ are most separated in KL-Divergence if S ∩ S ′ = ∅ and by using the fact that each arm can be pulled at most MB times. Therefore, we must have (provided N = c T for some large enough constant c > 0 , and T is large enough) for some constant c ′ > 0

Therefore, substituting ∆ = c ′ √ log( N / T ) B √ M , we have that for some constant c ′′ ≥ 0

and therefore

Algorithm 7 BBUIC (Blocked Latent Bandits with User and Item Clusters)

Require: Phase index glyph[lscript] , List of disjoint nice subsets of users M ( glyph[lscript] ) , list of corresponding subsets of active items N ( glyph[lscript] ) , clusters C , rounds T , noise σ 2 > 0 , round index t 0 , exploit rounds t exploit , estimate ˜ P of P , incoherence µ , entry-wise error guarantee glyph[epsilon1] glyph[lscript] of ˜ P restricted to all users in M ( glyph[lscript] ) and all items in N ( glyph[lscript] ) , count matrix K ∈ N M × N .

1: for i th nice subset of users M ( glyph[lscript],i ) ∈ M ( glyph[lscript] ) with active items N ( glyph[lscript],i ) ( i th set in list N ( glyph[lscript] ) ) do

2: Set t = t 0 . Set glyph[epsilon1] glyph[lscript] +1 = glyph[epsilon1] glyph[lscript] / 2 , ∆ glyph[lscript] = glyph[epsilon1] glyph[lscript] / 88 C and ∆ glyph[lscript] +1 = glyph[epsilon1] glyph[lscript] +1 / 88 C .

3: Run exploit component for users in M ( glyph[lscript],i ) with active items N ( glyph[lscript],i ) . Obtain updated active set of items, round index and exploit rounds N ( glyph[lscript],i ) , t, t exploit ← Exploit _ Item _ Cluster ( M ( glyph[lscript],i ) , N ( glyph[lscript],i ) , t, t exploit , ˜ P M ( glyph[lscript],i ) , N ( glyph[lscript],i ) , ∆ glyph[lscript] ) .

4: Set d 1 = max( |M ( glyph[lscript],i ) | , |N ( glyph[lscript],i ) | ) , d 2 = min( |M ( glyph[lscript],i ) | , |N ( glyph[lscript],i ) | ) and p = c ( σ 2 µ 3 log d 1 ∆ 2 glyph[lscript] +1 d 2 ) for some appropriate fixed constant c > 0 .

5: if |N ( glyph[lscript],i ) | ≥ T 1 / 3 and p < 1 then

6: Run explore component for users in M ( glyph[lscript],i ) with active items N ( glyph[lscript],i ) . Obtain updated estimate and round index ˜ P , t ← Explore _ Item _ Cluster ( M ( glyph[lscript],i ) , N ( glyph[lscript],i ) , t, p ) such that ∣ ∣ ∣ ∣ ∣ ∣ ˜ P M ( glyph[lscript],i ) , N ( glyph[lscript],i ) -P M ( glyph[lscript],i ) , N ( glyph[lscript],i ) ∣ ∣ ∣ ∣ ∣ ∣ ∞ ≤ ∆ glyph[lscript] +1 w.h.p.

7: For every user u ∈ M ( glyph[lscript],i ) , compute T ( glyph[lscript] ) u ≡ { j ∈ N ( glyph[lscript],i ) | ˜ P uπ u ( T -t exploit ) -˜ P uj ≤ 2∆ glyph[lscript] +1 } .

8: Construct graph G ( glyph[lscript],i ) whose nodes are users in M ( glyph[lscript],i ) and an edge exists between two users u, v ∈ M ( glyph[lscript],i ) if ∣ ∣ ∣ ˜ P ( glyph[lscript] ) ux -˜ P ( glyph[lscript] ) vx ∣ ∣ ∣ ≤ 2∆ glyph[lscript] +1 for all arms x ∈ N ( glyph[lscript],i ) .

9: Intitialize lists M ( glyph[lscript] +1) i = [] and N ( glyph[lscript] +1) i = [] .

10: For each connected component M ( glyph[lscript],i,j ) ( ∪ j M ( glyph[lscript],i,j ) ≡ M ( glyph[lscript],i ) ), compute N ( glyph[lscript],i,j ) ≡ ∪ u ∈M ( glyph[lscript],i,j ) T ( glyph[lscript] ) u .

11: For each connected component M ( glyph[lscript],i,j ) , construct graph G ( glyph[lscript],i,j ) item whose nodes are items in N ( glyph[lscript],i ) and an edge exists between two items u, v ∈ N ( glyph[lscript],i ) if ∣ ∣ ∣ ˜ P ( glyph[lscript] ) xu -˜ P ( glyph[lscript] ) xv ∣ ∣ ∣ ≤ 16 C ∆ glyph[lscript] +1 for all users x ∈ M ( glyph[lscript],i,j ) . Update N ( glyph[lscript],i,j ) to be the set of items N ( glyph[lscript],i,j ) ≡ { x ∈ N ( glyph[lscript],i ) | x is connected with some node in N ( glyph[lscript],i,j ) } .

12: Invoke B-LATTICE ( glyph[lscript] +1 , M ( glyph[lscript] +1) i , N ( glyph[lscript] +1) i , C , T .σ 2 , t, t exploit , ˜ P , glyph[epsilon1] glyph[lscript] +1 , K , G ( glyph[lscript],i,j ) item ).

13: else

14: For each user u ∈ M ( glyph[lscript],i ) , recommend T -t unblocked items in N ( glyph[lscript],i ) until end of rounds.

15: end if

16: end for

F Blocked Bandits having User and Item Clusters with blocking constraint B = 1

Recall that in this setting, the N items can be grouped into C ′ disjoint clusters D (1) , D (2) , . . . , D ( C ′ ) that are unknown. The expected reward for user u (belonging to cluster a ∈ [ C ] ) on being recommended item j (belonging to cluster b ∈ [ C ′ ] ) is P ij = Q ab where Q ∈ R C × C ′ is the small core reward matrix (unknown). In this setting, we provide our theoretical guarantees with B = 1 i.e. any item can be recommended to a user only once under the blocking constraint.

Recall that the main reason our theoretical analysis required B = Θ(log T ) for Thm. 1 setting is that if the observations that were used to compute estimates of some reward sub-matrix in a certain phase with certain guarantees that hold with some probability, then conditioning on such estimates with the said guarantees make the aforementioned observations dependent in analysis of future phases. In the BBIC setting, we show that in each phase, the possibility of the nice subsets of users and their corresponding active items is actually bounded and small. On the other hand, the possibilities were exponentially large in the number of items in the GBB setting. Therefore, we can use a for all argument here implying that for a set of already used observations, we can show that for all possible nice subsets of users and their active items, they can be used again to compute acceptable estimates. Such an analysis allows us to provide theoretical guarantees even with B = 1 for the BBIC setting.

Algorithm 8 Exploit_Item_Cluster (Exploit Component of a phase)

Require: Phase index glyph[lscript] , nice subset of users M , active items N , round index t 0 , estimate ˜ P of P and error guarantee ∆ glyph[lscript] such that ∣ ∣ ∣ ∣ ∣ ∣ ˜ P M , N -P M , N ∣ ∣ ∣ ∣ ∣ ∣ ∞ ≤ 88 C ∆ glyph[lscript] with high probability, similarity graph G item over the set of items N .

1: while there exists u ∈ M ( glyph[lscript],i ) such that ˜ P u ˜ π u (1) |N -˜ P u ˜ π u ( T -t exploit ) |N ≥ 64 C ∆ glyph[lscript] } do

2: Compute R u = { j ∈ N | ˜ P uj ≥ ˜ P u ˜ π u (1) |N -2∆ glyph[lscript] +1 } for every user u ∈ M . Compute S = ∪ u ∈M R u .

3: Update S ← { v ∈ G item | v is connected with some node in S} .

4: for rounds t = t 0 +1 , t 0 +2 , . . . , t 0 + |S| B do

5: for each user u ∈ M do

6: Denote by x the ( t -t 0 ) th item in S . If K ux == 0 ( x is unblocked), then recommend x to user u and update K ux ← 1 . If K ux == 1 ( x is blocked), recommend any unblocked item y in N (i.e K uy = 0 ) for the user u and update K uy ← 1

7: end for

8: end for

9: Update N ← N \ S . Update t 0 ← t 0 + |S| .

10: end while

11: Return N , t 0 .

Algorithm 9 Explore_Item_Cluster (Explore Component of a phase)

Require: Phase index glyph[lscript] , nice subset of users M , active items N , round index t 0 , sampling probability p .

1: For each tuple of indices ( i, j ) ∈ M × N , independently set δ ij = 1 with probability p and δ ij = 0 with probability 1 -p .

2: Denote Ω = { ( i, j ) ∈ M×N | δ ij = 1 } and m = max i ∈M | | j ∈ N | ( i, j ) ∈ Ω | to be the maximum number of index tuples in a particular row. Initialize observations corresponding to indices in Ω to be A = φ .

3: for rounds t = t 0 +1 , t 0 +2 , . . . , t 0 + m do

4: for each user u ∈ M do

5: Find an item z in { j ∈ N | ( u, j ) ∈ Ω , δ uj = 1 } . If K uz == 0 ( z is unblocked), set ρ u ( t ) = z and recommend z to user u . Observe R ( t ) uρ u ( t ) and update A = A∪{ R ( t ) uρ u ( t ) } , K uz ← 1 .

6: If K uz == 1 ( z is blocked), recommend any unblocked item ρ u ( t ) in N s.t. ( u, ρ u ( t )) glyph[negationslash]∈ Ω . Update K uρ u ( t ) ← 1 . Set A = A∪{ R ( t ′ ) uρ u ( t ′ ) } where t ′ < t is the round when ρ u ( t ′ ) = z was recommended to user u .

7: end for

8: end for

9: Compute the estimate ˜ P = Estimate ( M , N , σ 2 , C , Ω , A ) and return ˜ P , t 0 + m .

Therefore in Algorithm 7 for the BBIC setting, we only have a single counter matrix K ∈ { 0 , 1 } M × N which is binary. The matrix K is initialized to be a zero matrix. Whenever an item j is recommended to user i , we set K ij = 1 . If, in a future phase, we need to recommend item j to user i again, due to the blocking constraint, we simply reuse the observation (see Step 6 in Alg. 9).

F.1 Algorithm and Discussion

We start with a definition for nice subsets of items analogous to the nice subset of users (see Definition 1).

Definition 4. A subset of items S ⊆ [ N ] will be called 'nice" if S ≡ ⋃ j ∈A D ( j ) for some A ⊆ [ C ′ ] . In other words, S can be represented as the union of some subset of clusters of items.

As in the analysis of GBB setting, we will have the desirable properties A-C that should be satisfied with high probability at the beginning of the explore component of phase glyph[lscript] . Recall that we defined the event E ( glyph[lscript] ) 2 to be true if properties (A-C) are satisfied at the beginning of the explore component

of phase glyph[lscript] by the phased elimination algorithm. Here, we stipulate a further property D that the corresponding surviving set of items for each nice subset of users M ( glyph[lscript],i ) ∈ M ( glyph[lscript] ) is also a nice subset of items .

Furthermore, we defined the event E ( glyph[lscript] ) 3 when eq. 10 is true for all nice subsets M ( glyph[lscript],i ) ∈ M ′ ( glyph[lscript] ) in the explore component of phase glyph[lscript] . Algorithm 7 is a similar recursive algorithm as Algorithm 1 and the only modification is the addition of Step 11. As in Alg. 1, we instantiate Alg. 7 with phase index 1 , list of nice subsets of users having a single element comprising all users i.e. M (1) = [[ M ]] and corresponding list of active items N (1) = [[ N ]] , clusters C , rounds T , blocking constraint B , noise σ 2 , round index 1 , exploit rounds 0 , estimate ˜ P to be 0 M × N , incoherence µ and glyph[epsilon1] 1 = O ( ‖ P ‖ ∞ , σ √ µ log M ) . Here, we will have a single count matrix K which is binary and is initialized to be an all zero matrix. We are going to show recursively that conditioned on E ( glyph[lscript] ) 2 , E ( glyph[lscript] ) 3 , properties A-D will be satisfied at the beginning of phase glyph[lscript] +1 .

Base Case: For glyph[lscript] = 1 (the first phase), the number of rounds in the exploit component is zero and we start with the explore component. We initialize M (1 , 1) = [ N ] , N (1 , 1) = [ M ] and therefore, we have

Clearly, [ M ] is a nice subset of users, [ N ] is a nice subset of items and finally for every user u ∈ [ M ] , the best T / B items ( golden items ) { π u ( t ) } T / B t =1 belong to the entire set of items. Thus for glyph[lscript] = 1 , conditions A-D are satisfied at the beginning of the explore component and therefore the event E (1) 2 is true. Furthermore, from Lemma 4, eq. 10 is true for the first phase with probability 1 -o ( T -12 ) implying that the event E (1) 3 is true with high probability.

Inductive Argument: Suppose, at the beginning of the phase glyph[lscript] , we condition on the events ⋂ glyph[lscript] j =1 E ( j ) 2 ⋂ glyph[lscript] j =1 E ( j ) 2 that Algorithm is ( glyph[epsilon1] j , j ) -good for all j ≤ glyph[lscript] . This means that conditions (A-D) are satisfied at the beginning of the explore component of all phases up to and including that of glyph[lscript] for each reward sub-matrix (indexed by i ∈ [ a glyph[lscript] ] ) corresponding to the users in M ( glyph[lscript],i ) and items in N ( glyph[lscript],i ) . Furthermore, we also condition on the event eq. 10 is true for all nice subsets of users M ( glyph[lscript],i ) ∈ M ( glyph[lscript] ) and their corresponding set of items N ( glyph[lscript],i ) - implying that the event E ( glyph[lscript] ) 3 is true. Recall that for a user v , the set T ( glyph[lscript] ) v was constructed as

which implies that every item in T ( glyph[lscript] ) v is close to one of the golden items in N ( glyph[lscript],i ) . At the end of Step 10 in Alg. 7, we can still show that Lemma 9 holds for each set of users M ( glyph[lscript],i,j ) as they form a connected component. Notice that in Step 11 in Lemma 9, for each set of users M ( glyph[lscript],i,j ) , we construct a graph G ( glyph[lscript],i,j ) item whose nodes correspond to the items in N ( glyph[lscript],i ) and an edge exists between two items u, v ∈ N ( glyph[lscript],i ) if ∣ ∣ ∣ ˜ P ( glyph[lscript] ) xu -˜ P ( glyph[lscript] ) xv ∣ ∣ ∣ ≤ 16 C ∆ glyph[lscript] +1 for all users x ∈ M ( glyph[lscript],i,j ) . Analogous to the proof of Lemma 7, we can conclude here as well that items in the same item cluster form a clique. Therefore, any connected component in the graph G ( glyph[lscript],i,j ) item must correspond to a nice subset of items since condition D is true and N ( glyph[lscript],i ) is already a nice subset of items. Hence the set of modified items N ( glyph[lscript],i,j ) constructed at the end of Step 11 in Alg. 7 is a nice subset of items. Furthermore, we can also prove the corresponding version of Lemma 9:

Lemma 19. Condition on the events E ( glyph[lscript] ) 2 , E ( glyph[lscript] ) 3 being true. Consider a nice subset of users M ( glyph[lscript],i,j ) and their corresponding set of active items N ( glyph[lscript],i,j ) for which guarantees in eq. 11 holds. Fix any set Y = N ( glyph[lscript],i,j ) \ J for some J such that for every user u ∈ G , we have ˜ π u ( s ′ ) | N ( glyph[lscript],i,j ) ≡ ˜ π u ( s ) | Y for s ′ = TB -1 -∣ ∣ ∣ O ( glyph[lscript] ) M ( glyph[lscript],i ) ∣ ∣ ∣ and some common index s . In that case, we must have

Proof. Note that with the analysis of Lemma 9, the conclusion was true for the constructed N ( glyph[lscript],i,j ) at the end of Step 10 in Alg. 7. However, in the modified set of items N ( glyph[lscript],i,j ) , we are only adding

items in N ( glyph[lscript],i ) that either belong to the same cluster (in which case there is no added gap) or if we are adding items belonging to a different cluster, then the added gap on the RHS can be at most C ′ ∆ glyph[lscript] +1 . The above statement follows from a similar argument as in Lemma 13 where we showed that gap in expected reward between two users in M ( glyph[lscript],i,j ) (or rather two users connected by a path in G ( glyph[lscript],i ) ) for the same item is at most O ( C ∆ glyph[lscript] +1 ) .

Hence, we have shown that each subset of users M ( glyph[lscript],i,j ) is a nice subset of users and furthermore, the corresponding subset of items constructed at the end of Step 11 is nice subset of items . Next, we move on to the exploit component of phase glyph[lscript] +1 where we use a similar trick (See Step 3 in Alg. 8) to ensure that at the end, we are left with a nice subset of items . As before, we can show that the constructed set of items S is itself a nice subset of items . We can again prove the following modified version of Corollary 2

Corollary 3. Condition on the events E ( glyph[lscript] ) 2 , E ( glyph[lscript] ) 3 being true. Consider a nice subset of users M ( glyph[lscript],i ) ∈ M ′ ( glyph[lscript] ) and their corresponding set of active items N ( glyph[lscript],i ) for which guarantees in eq. 11 holds. Fix any subset Y ⊆ N ( glyph[lscript],i ) . Consider two users u, v ∈ M ( glyph[lscript],i ) having a path in the graph G ( glyph[lscript],i ) . Conditioned on the events E ( glyph[lscript] ) 2 , E ( glyph[lscript] ) 3 , we must have

Proof. The proof again follows from the fact that adding items belonging to a different cluster but connected via a path to the original items in R ( glyph[lscript] ) u ∪R ( glyph[lscript] ) V can only add a term of at most 16 C ′ ∆ glyph[lscript] +1 in the RHS.

Therefore, at the beginning of the explore component of phase glyph[lscript] +1 for a particular nice subset of users M ( glyph[lscript] +1 ,z ) , the corresponding set of items N ( glyph[lscript] +1 ,z ) must be a nice subset of items as well. Most importantly, what this implies is that for the low rank matrix completion step in the explore component of phase glyph[lscript] +1 , we can re-use observations from previous phases and provide theoretical guarantees as well. This is because, in eq. 10 in Lemma 4, we can take a union bound over all possible nice subsets of users and all possible nice subsets of items . Since the number of clusters C , C ′ = O (1) , the total possibilities is O (1) as well (although exponential in the number of clusters). Hence, the complex dependencies of the previously made observations used to compute prior estimates resulted by the conditioning on surviving items and users is no longer a problem - we have simply made a for all argument. The rest of the analysis follows as in the GBB setting and we can arrive at a similar result as in Theorem 1 but with B = 1 when the set of items can be clustered into C ′ = O (1) disjoint clusters.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the online recommendation for a hard-clustered user. \n\nProblem setup summary:\nThis problem involves $M$ users and $N$ items. the reward matrix $P \\in R^{M \\times N}$ is unknown. There are $T$ rounds in total, and each user receives a recommendation $\\rho_u(t)$ at each round. Each user belongs to one of (unknown) clusters indexed by $\\{1,2,\\dots,C\\}$. Each user can consume at most $B = O(\\log T)$ same item. \n\nThe oracle-optimal strategy is to recommend top-($T/B$) items for $B$ (max possible) times, and the regret is the loss of recommending items except for these top arms. \n\nThis is multitask learning in the sense that the results are better than solving $N$-armed bandit for each of $M$ users (in that case, regret would be linear to $M$).\n\nSection 2.1 introduces a bound on the accuracy of low-rank matrix completion (Lemma 1, [16]). Based on that it describes the performance of explore-then-commit with $B=1$. Section 3 introduced the B-Lattice algorithm, its regret bound (Thm 1), and regret lower bound (Thm 2)\n\nThe paper proposes B-LATTICE algorithm for online recommendation. The algorithms is phased and at each phase $l$ it obtains accuracy of $2^{-l}$. This paper is well-written. Although the algorithm described here is not truly practical in the sense it can be applied to real recommendation (see weaknesses). The contributions of the paper are enough good. My main concerns are about the solidness of the results.\n\n#####\nI think the paper is above the acceptance threshold even though the exposition can have some space for improvement, so I keep current rating (WA).\n\nSTRENGTHS:\n* A new scheme on the factorized recommendation with bandit settings (Tbl 1) \n* parametric rate of regret\n* Non-trivial (non-tight) regret lower bound using Fano's inequality.\n* Simulation results in appendix\n\nWEAKNESSES:\n* Somewhat artificial budget constraint (top T/B items, where B is consistent over all items)\n* Many parts of the algorithms are in the appendix (mainly, Alg 4,5)\n* Alg depends on B, C that are hard-to-know before (in particular B).\n* Alg is very involved.\n* (minor) exposition on the order of elements can be improved. \n  * For example, Lemma 1 informally describes incoherency before the main results, the main results part (Thm 1) do not include B-LATTICE detail even though it is bound on this algorithm.\n\nQUESTIONS:\n* Is Lemma 1 derived from Lemma 2 of [16] (or from Lemma 5 of [16])?\n\n* Question on the applicability of Lemma1: Lemma 1 requires the sampling of $\\Omega$ to be iid. I assume B-Lattice adopts the same bound, but can it be usable for such a complex sampling scheme? \n\n* Questions on the bounds:\n  * Can the authors clarify the incoherency condition (L176: $||\\bar{U}||_{2,\\infty}$ - this norm undefined?), and subset strong smoothness? \n  * What is the dependence of Thm1 in $C$ (# of clusters)?\n  * What $B = \\Theta(\\log T)$ exactly states? In particular, can i be $C \\log T$ where $C>0$ is independent of $T,N,M$, and # of clusters? \n\n* (minor) Are the technology used in this paper (bound on low-rank matrix completion and graphs) entirely new in this paper? How the basic technology can be compared with related work (e.g., Table 1)?\n\n* (minor) Is there any way to simplify the algorithm? For example, can explore (Alg 2) be randomly recommending an available item, and exploit be randomly recommending an item weighted on the likelihood to be good?\n\nLIMITATIONS:\nNot applied.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors tackle a collaborative-filtering type multi-armed bandit problem. There are many users, each facing a multi-armed bandit problem. However, there are only $C$ base multi-armed bandit instances, where each user comes from one of these instances. The authors provide a novel algorithm which only pulls each arm at most $B$ times per user, and provide information theoretic lower bounds for this new setting. The authors also provide an improved, practical version of this method.\n\nSTRENGTHS:\n1. The proposed algorithm is reasonably easy to follow, and makes intuitive sense. The authors explain the process clearly.\n2. The authors provide a new lower bound, using a different Fano's inequality based method. The approach and proof of the upper bound make sense, but I have not checked the details thoroughly.\n3. It would be good to emphasize that $N,M\\gg T$ is actually a very relevant setting; $T$ is number of interactions per user, each user will only interact with the system a moderate number of times, but the overall number of users can be very large.\n\nWEAKNESSES:\n1. The assumptions regarding $C$ are very strong. a) that all users come from exactly 1 of $C$ instances (no within group variability), b) that $C$ is exactly known, and c) that $\\tau$ is constant (it is unclear how the bound depends on $\\tau$). Additional text regarding why these assumptions are reasonable, the brittleness of the algorithm to these differences, or on how these assumptions can be loosened, would strengthen the paper.\n2. Empirical improvement: it does not appear that the algorithm has much better performance than the greedy baselines, unless I am misreading Figure 1? It could improve the paper to add simulations for larger $T$ to show how the asymptotic behavior compares (over Greedy). Additionally, comparing against [24] in the case where $B=T$ would be interesting. Finally, to highlight the improvement of collaboration, it could help to include a baseline of running e.g. UCB independently for each user, showing that this would yields dramatically worse performance.\n3. Clarity: the work is overall quite clear and well-written, but there are some points (discussed below) that could help improve presentation.\n\nMinor:\n1. Lower bound as stated is not technically correct. Classical bandit lower bounds hold when the noise is truly Gaussian (or Bernoulli), and cannot be invoked for general sub-Gaussian random variables, as the variance proxy is an upper bound (e.g. a Gaussian with variance .01 is 1-sub-Gaussian).\n2. Eqs after Line 1252 and 1259 have typing errors; maybe missing a $\\geq$?\n\nTypos:\nLine 28: unknown cluster and *the* expected reward\n53: suboptimal dependence on *the* number\n207: \"columns comprise of at\"\n249: \"small for e.g.\"\nFig 1 caption: period missing before last sentence, space after PB-Lattice.\nRedundancy in setting $\\Delta_{\\ell+1}$ in line 2 of Alg 1\nLine 274,278: clarify that goal is to get an entrywise $O(2^{-\\ell})$ estimate.\nB-Lattice doesn't take in B as input, and this parameter doesn't show up anywhere in Alg 1. Alg 2 also doesn't take $B$ as input.\nUsing $\\mathbf{C}$ as a counts matrix and $C$ as the number of clusters is confusing notation.\n\n\nAlgorithmic clarification:\n1. It appears as though $\\mathcal{M}^{(\\ell)}$ is an *estimation* of disjoint nice subsets of users, whereas in Algorithm 1's input it is stated that they are truly nice subsets of users.\n2. Line 287: each connected component is a nice subset *with high probability*? Several results need to be clarified that they're conditional on the good event in Lemma 1 (that estimation worked).\n3. The algorithms defined do not appear to be \"stand-alone\", i.e. they have side effects involving the matrices C and D (which appear to be global variables)? A clearer description of what each algorithms requires as input, and what variables it modifies, would be helpful.\n\nQUESTIONS:\nSee weaknesses. My primary question is regarding the assumption of knowing $C$, and that each user has exactly 1 of $C$ different mean vectors.\n\nLIMITATIONS:\nDiscussed, modulo $C$ above.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper considers the problem of blocked collaborative bandits with multiple users that each belong to a cluster with associated bandit instance. The goal is to minimize the regret of each user while satisfying the constraint that each arm can be played at most B times for each user. The paper proposes an algorithm with sublinear regret upper bound.

STRENGTHS:
- The paper proposes a novel algorithm that uses matrix completion and clustering techniques.
- Although not very surprising, a regret bound of O(sqrt{TN/M}) is proved when B=O(log T).
- The paper also proves a lower bound on the regret.

WEAKNESSES:
- The regret upper bound holds only for B=O(log T). As the papers main contribution is providing regret bounds with the existence of the budget constraint B, I believe the authors should provide the dependency of the regret bound on B.
- Assumption 1 is strong. It requires the information to not be concentrated in some clusters which in my opinion is an interesting case to deal with. Moreover, the choice of assumptions and constants is unnatural; for example why is |S|=T^{1/3} in Assumption 1.3?
- The regret lower bound does not match the achieved upper bound.

QUESTIONS:
- It seems that the regret lower bound in Theorem 2 is linear in T while the upper bound is sublinear?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
This paper studies the blocked collaborative bandits where each user represent a single bandit problem. The latent groups are assummed to exist among users in which all the users in a group share the same expected reward. But the constrant is placed on the number of serving time for each user to solve the challenge of lacking user-item interactions. Then, the authors introduce a complicated algorithms including the exploitation, exploration and clustering components. The theoretical analysis is provided and achieved a tigher regret bound for this setting.

STRENGTHS:
This paper provide two regret bound. The first one recover the results and follows the regret analysis is online clustering of bandits. The second authors provide a lower bound, which consider the case when T is large.

WEAKNESSES:
The introduced algorithm is very complicated and I am worried about the efficiency of this algorithm. Authors didn't report the time cost of this algorithm and conduct ablation study for the exploration component. Because this paper tries to solve a very practical problem, the online recommendation, it only has very limited empirical evaluations, which run on the systhetic datasets. The main manuscript doesn't inlcude the empirical evaluation.

QUESTIONS:
In this problem setting, it seems to not have the assumption regarding the reward gap between two clusters. The reward gap assumption is commen in the problem of online clustering of bandits. How does it influence the analysis?

LIMITATIONS:
More empirical evaluations are needed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies blocked collaborative bandits. Multiple users, clustered in a fixed number of clusters according to the rewards obtained from them are provided recommendations in each discrete time instance till T timesteps. The paper proposes algorithms that minimize the cumulative regret with the constraint that no user is recommended the same arm (item ) more than B times. The authors propose an algorithm called B-LATTICE  that collaborates across users while satisfying budget constraints, and minimising regret. The authors show theoretical guarantees of the proposed algorithm and also validate their results empirically with synthetic data.\n\nSTRENGTHS:\nThe paper introduces an important setting with practical application. The main strength of the paper is its sublinear regret guarantee. The authors give the first sublinear regret algorithm in the proposed setting. Furthermore, the analysis and proposed algorithms are involved and nontrivial; the authors do an excellent job of providing an intuitive understanding of their proposed algorithms.\n\nWEAKNESSES:\nThe paper does a poor job of explaining the setting and its connection to Bandits. Following are my questions and (some) suggestions to the authors. \n\n1. A crisp summary of how the problem is MAB problem is missing from the introduction/setting section. For instance, the following points are not clear in the first reading of the first 2 sections; what constitutes an arm (items?), who pulls arms (a recommendation engine or users), it is not clear that rewards are not only arm-specific but also user dependent (which in my opinion most important difference from classic MAB setting).\n\n2. Can authors shed some light on what is known apriori to the algorithm/recommendation engine? For instance, the authors use the fact that the number of true clusters  $C$ is known in Algorithm B-LATTICE, however, I don't understand why T is needed as an input. In particular. can the regret guarantee of the proposed algorithm be converted into an anytime guarantee (for any value of large enough T)? In contrast, in the Greedy algorithm (Remark 1) authors explicitly use T. \n\n3. I don't see why the authors normalize the cumulative regret by $M$ (Equation 2). Is it crucial to consider per-user regret guarantee? Howe do the results fare without this normalization?\n\nQUESTIONS:\nsee weaknesses.\n\nLIMITATIONS:\nnone", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}