SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Soumyabrata Pal
soumyabratapal13@gmail.com
Arun Sai Suggala
arunss@google.com
Karthikeyan Shanmugam
karthikeyanvs@google.com
Prateek Jain
prajain@google.com
We consider the problem of blocked collaborative bandits where there are multiple users, each with an associated multi-armed bandit problem. These users are grouped into latent clusters such that the mean reward vectors of users within the same cluster are identical. Our goal is to design algorithms that maximize the cumulative reward accrued by all the users over time, under the constraint that no arm of a user is pulled more than  $\mathsf{B}$  times. This problem has been originally considered by [4], and designing regret-optimal algorithms for it has since remained an open problem. In this work, we propose an algorithm called B-LATTICE (Blocked Latent bAndiTs via maTrIx ComplEtion) that collaborates across users, while simultaneously satisfying the budget constraints, to maximize their cumulative rewards. Theoretically, under certain reasonable assumptions on the latent structure, with  $\mathsf{M}$  users,  $\mathsf{N}$  arms,  $\mathsf{T}$  rounds per user, and  $\mathsf{C} = O(1)$  latent clusters, B-LATTICE achieves a per-user regret of  $\tilde{O}(\sqrt{\mathsf{T}(1 + \mathsf{NM}^{-1})}$  under a budget constraint of  $\mathsf{B} = \Theta(\log \mathsf{T})$ . These are the first sub-linear regret bounds for this problem, and match the minimax regret bounds when  $\mathsf{B} = \mathsf{T}$ . Empirically, we demonstrate that our algorithm has superior performance over baselines even when  $\mathsf{B} = 1$ . B-LATTICE runs in phases where in each phase it clusters users into groups and collaborates across users within a group to quickly learn their reward models.
Modern recommendation systems cater to millions of users and items [1] on a daily basis, typically in an online fashion. A critical feature of such systems is to quickly learn tastes of individual users from sequential actions and feedback, and suggest personalized products for each user. Furthermore, in practice, an item that has already been consumed by a user is recommended very few times to the same user (or not recommended at all). This is because, in applications such as movie/book recommendations, a typical user will find little interest in consuming the same item multiple times.
This problem – that we rename as *Blocked Collaborative Bandits* – was abstracted out by [4] with a modeling assumption that users have a latent clustering structure. That is, each user can belong to an unknown cluster and the expected reward for an item/movie is same for all users in a cluster. Formally, consider M users, N items and T rounds with M, N ≫ T (M, N ≈ 10^6 in recommendation systems such as YouTube) and in each round, every user is recommended some item (potentially different for each user). On consuming the item, a noisy reward is assigned. As mentioned above,
Paper	Setting	Metric	Guarantees (Worst-Case)
Bresler et al., 2014 [4]	B = 1, user clusters	pseudo regret (maximize likeable items)	O(T)
Bresler et al., 2018 [5]	B = 1, user, item clusters; noiseless rewards	regret	O(1)
Ariu et al., 2020 [2]	B = 1, user, item clusters	regret	O(T2/3(1 + NM-1)1/3) sub-optimal in N, T
Pal et al., 2023 [25]	B = T, user clusters	regret	O(√T(1 + NM-1) minimax optimal in N, M, T
This Work	B = Θ(log T), user clusters	regret	O(√T(1 + NM-1)
Table 1: Comparison of various approaches for blocked collaborative bandits. All the regret bounds stated here are worst-case bounds and assume  $C = O(1)$ . Moreover, the regret is averaged across users. The worst-case pseudo-regret in [5] is linear when the items have rewards close to 0.5. In [2], the authors' proposed a greedy algorithm whose worst-case has  $T^{2/3}$  dependence.
it is assumed that the users can be clustered in C clusters, where  $\mathsf{C} \ll \mathsf{M}, \mathsf{N}$ , and users in the same cluster have identical reward distributions over items. Furthermore, any item can be recommended to a particular user at most B times, after which the item is blocked for the user.
[4] considered this problem for  $\mathsf{B} = 1$  in the setting where a user in cluster  $c$  on being recommended item  $j$  provides a like  $(+1)$  with probability  $p_{cj}$  and a dislike  $(-1)$  with probability  $1 - p_{cj}$ . The authors studied a notion of pseudo-regret corresponding to minimizing the number of un-likeable items for a user, i.e., items for which the probability of user giving a like  $(+1)$  is less than  $1/2$ . To this end, the authors proposed the Collaborative-Greedy algorithm, an  $\epsilon$ -greedy style algorithm that performs random exploration in each round with certain probability. During exploitation it provides recommendations based on a neighborhood of similar users. However, maximizing the number of likeable items is limiting as it does not prioritize items with large rewards and completely disregards the item ordering. Despite these theoretical limitations, variants of the collaborative algorithm designed in [4] have found applications in predicting Bitcoin price, [29], information retrieval [12] among others.
Recent works have studied this problem under a more practical notion of regret which involves maximizing the cumulative rewards accrued over time [2, 5]. However, these works assume a cluster structure among both users and items. This assumption entails that there are many copies of the highest rewarding item for any user; this voids the blocking constraint and makes the problem significantly easier. Moreover, the algorithms designed in [2] are greedy (they perform exploration first to cluster users, items, and then perform exploitation) and achieve significantly sub-optimal regret. These bounds, translated to the original blocked bandit problem where there are no item clusters, have sub-optimal dependence on the number of items N. The algorithms designed in [5] assumed a noiseless binary feedback model which allowed for constant regret. However, these algorithms are not easily extendable to the more general noisy reward setting we consider in this work.
Another line of work has studied this problem when  $\mathsf{B} = \mathsf{T}$  (i.e., no budget constraints on arms) [24, 11, 25]. While some of these works have designed regret optimal algorithms [25], the  $\mathsf{B} = \mathsf{T}$  budget constraint is too lenient in many modern recommendation systems. To summarize, existing works have either attempted to solve the much harder setting of  $\mathsf{B} = 1$  by imposing additional restrictions on the problem or the simpler  $\mathsf{B} = \mathsf{T}$  setting which is not very relevant in practice.
This Work. In this work, we make progress on this problem by considering the intermediate setting of  $\mathsf{B} = \Theta (\log \mathsf{T})$ . We do not impose any cluster structure among items (as in [4]), and consider a general notion of regret which involves maximizing the cumulative rewards under the budget constraints. We propose B-LATTICE(Alg. 1), a phased algorithm which carefully balances exploration and exploitation. Our algorithm also performs on-the-fly clustering of users and collaborates across users within a cluster to quickly learn their preferences, while simultaneously satisfying the budget constraints. The key contribution of our work is that under certain standard incoherence assumptions (also used recently in [25]) and a budget constraint of  $\mathsf{B} = \Theta (\log \mathsf{T})$ , we show that B-LATTICE achieves a per-user regret of  $\widetilde{O} (\sqrt{\mathsf{T}(1\bigvee\mathsf{NM}^{-1})})^{1}$ . Empirically, our algorithm can even handle a budget constraint of  $\mathsf{B} = 1$  and perform better than baselines (Alg. 6). However, bounding its regret
$\widetilde{O} (\cdot),\widetilde{\Omega} (\cdot)$  hides logarithmic factors in M, N, T
in this setting is much more challenging which we aim to address in a future work. That being said, under an additional cluster structure on the items, we show that our theoretical guarantees hold even for  $\mathsf{B} = 1$  (Appendix F).
We also provide a non-trivial regret lower bound of  $\widetilde{\Omega} (\sqrt{\mathsf{NM}^{-1}}\bigvee \mathsf{T}\mathsf{M}^{-1 / 2})$ . Our proof involves a novel reduction to multi-hypothesis testing and relies on Fano's inequality for approximate recovery [28] (see Appendix E). Our techniques could be of independent interest even outside the context of this work. Our lower bound is tight in two regimes - when the number of rounds  $\mathsf{T}$  is very small or very large. However, we conjecture that our upper bounds are actually tight since they recover the same rates as in the  $\mathsf{B} = \mathsf{T}$  setting [25]; tightening the lower bound is left as a future work. Finally, we verify our theoretical results by simulations on synthetic data (see Appendix C). Here, we compare a more practical version of B-LATTICE(Algorithm 6) with Collaborative-Greedy [4]. We demonstrate that B-LATTICE not only has good regret but also other practically desirable properties such as a small cold-start period, and repeated high quality recommendations.
Techniques. Our algorithm is built on recent works that have studied this problem without the budget constraint [17, 25]. [17] developed a regret optimal algorithm called LATTICE that runs in phases. At any phase, the algorithm maintains a grouping of users which it refines over time (these groups are nothing but our current estimate of user clusters). Within each group, the algorithm relies on collaboration to quickly estimate the reward matrix. To be precise, the algorithm performs random exploration followed by low-rank matrix completion [8] to get a good estimate of the user-item reward matrix. Next, the algorithm uses the estimated reward matrix to eliminate sub-optimal arms for each user. Finally, it refines the user clusters by placing users with similar reward structure in the same group.
Extending the above algorithmic recipe to our setting poses a couple of challenges. First, observe that the oracle optimal strategy in the budget constrained setting is to recommend each of the top T/B items B times; we refer to these top times as golden items in the sequel. To compete against such a policy, our algorithm needs to quickly identify the golden items for each user. The LATTICE algorithm described above doesn't perform this, as it aims to only identify the top item for every user. So, one of the key novelties in our algorithm is to design a test to quickly identify the golden items and recommend them to the users. The second challenge arises from the usage of low-rank matrix completion oracles in LATTICE. To be precise, LATTICE requires more accurate estimates of the user-item reward matrix in the later phases of the algorithm. To this end, it repeatedly recommends an item to a user to reduce the noise in its estimates. However, this is infeasible in our setting due to the budget constraints. So, the second novelty in our algorithm is to avoid repeated recommendations and design an alternate exploration strategy that adheres to the budget constraints.
Other Related Work:
Item-Item Collaborative Filtering (CF). A complementary theoretical line of work proposes to exploit a clustering structure across the items instead of users [7, 23, 27, 22]. Under a similar blocking constraint, the authors have provided a sub-linear regret guarantee based on a certain measure of complexity called the doubling dimension. Since then, there have been several works in the literature that have attempted to exploit a cluster structure on both users and items [6, 2].
Variants of User-User CF. [15] has looked into the problem of non-stationary user-user collaborative filtering where the preferences of users change over time. [13] studies a variant where the user only provides positive ratings i.e when the user has liked an item. [9] and [3] studies probabilistic models for user user CF in an online and offline model respectively.
Cluster Structure across users. In several problems related to multi-armed bandits, cluster structure across users have been explored. In particular, in [11, 20, 10, 21, 26], the authors have imposed a cluster-structure across users while each item has a  $d$ -dimensional context chosen uniformly at random at each round. The preferences of each user is a linear function of the context vector and the cluster id. Under these settings, the authors prove a strong regret bound. However, note that in our setting, the item contexts are hidden; hence, these techniques are not applicable to our setting.
2 Problem Setting and Background
Notation. We write  $[m]$  to denote the set  $\{1,2,\ldots,m\}$ . For a vector  $\mathbf{v} \in \mathbb{R}^m$ ,  $\mathbf{v}_i$  denotes the  $i^{\text{th}}$  element; for any set  $\mathcal{U} \subseteq [m]$ , let  $\mathbf{v}_{\mathcal{U}}$  denote the vector  $\mathbf{v}$  restricted to the indices in  $\mathcal{U}$ . Similarly,
for  $\mathbf{A} \in \mathbb{R}^{m \times n}$ ,  $\mathbf{A}_{ij}$ ,  $\mathbf{A}_i$  denotes the  $(i,j)$ -th element and the  $i^{\mathrm{th}}$  row of  $\mathbf{A}$  respectively. For any set  $\mathcal{U} \subseteq [m], \mathcal{V} \subseteq [n]$ ,  $\mathbf{A}_{\mathcal{U},\mathcal{V}}$  denotes  $\mathbf{A}$  restricted to the rows in  $\mathcal{U}$  and columns in  $\mathcal{V}$ . Let  $\| \mathbf{A} \|_{\infty}$  denote absolute value of the largest entry in matrix  $\mathbf{A}$ .  $\mathcal{N}(0,\sigma^2)$  denotes the Gaussian distribution with 0 mean and variance  $\sigma^2$ . We write  $\mathbb{E}X$  to denote the expectation of a random variable  $X$ .  $||\mathbf{U}||_{2,\infty}$  corresponds to the maximum euclidean norm of a row of the matrix  $\mathbf{U}$ . More precisely, for a matrix  $\mathbf{U} \in \mathbb{R}^{M \times r}$ , the norm  $||\mathbf{U}||_{2,\infty} = \max_{i \in [M]} ||\mathbf{U}_i||_2$ . Thus, if  $||\mathbf{U}||_{2,\infty}$  is small as in Lemma 1, then all rows of  $\mathbf{U}$  have a small  $\ell_2$  norm.
Problem Setting. Consider an online recommendation system with  $\mathsf{M}$  users,  $\mathsf{N}$  items and  $\mathsf{T}$  rounds. Let  $\mathbf{P} \in \mathbb{R}^{M \times N}$  be the reward matrix which is unknown. Here, we assume the set of  $\mathsf{M}$  users can be partitioned into  $C$  disjoint but unknown clusters  $\mathcal{C}^{(1)}, \mathcal{C}^{(2)}, \ldots, \mathcal{C}^{(C)}$  such that any two users  $u, v \in [M]$  belonging to the same cluster have identical reward vectors i.e.  $\mathbf{P}_u = \mathbf{P}_v$ . In each round  $t \in [T]$ , every user is recommended an item (can be different for each user) by the system. In turn, the system receives feedback in the form of reward from each user. Let,  $\mathbf{R}_{u\rho_u(t)}^{(t)}$  be the observed reward for recommending item  $\rho_u(t) \in [N]$  to user  $u$  at round  $t$  such that:
$$
\mathbf {R} _ {u \rho_ {u} (t)} ^ {(t)} = \mathbf {P} _ {u \rho_ {u} (t)} + \mathbf {E} _ {u \rho_ {u} (t)} ^ {(t)} \tag {1}
$$
where  $\mathbf{E}_{u\rho_u(t)}^{(t)}$  denotes the unbiased additive noise. We assume that elements of  $\{\mathbf{E}_{u\rho_u(t)}^{(t)}\}_{\substack{u\in [\mathsf{M}]\\ t\in [\mathsf{T}]}}$  are i.i.d. zero mean sub-gaussian random variables with variance proxy  $\sigma^2$  i.e. for all  $u,t\in [\mathsf{M}]\times [\mathsf{T}]$  we have  $\mathbb{E}\mathbf{E}_{u\rho_u(t)}^{(t)} = 0$  and for all  $s\in \mathbb{R}$ , we have  $\mathbb{E}\exp (s\mathbf{E}_{u\rho_u(t)}^{(t)})\leq \exp (\sigma^2 s^2 /2)$ . As in practical recommendation systems, we impose an additional constraint that the same item cannot be recommended more than B times to a user, for some small  $\mathsf{B} = \Theta (\log \mathsf{T})$ . For simplicity of presentation, we assume T is a multiple of B. However, we would like to note that our results hold for general T. Without loss of generality, we assume N  $\geq$  T/B, since otherwise the budget constraints cannot be satisfied.
Our goal is to design a method that maximizes cumulative reward. Let  $\pi_u: [\mathsf{N}] \to [\mathsf{N}]$  denote the function that sorts the rewards of user  $u \in [\mathsf{M}]$  in descending order, i.e., for any  $i < j$ ,  $\mathbf{P}_{u\pi_u(i)} \geq \mathbf{P}_{u\pi_u(j)}$ . The oracle optimal strategy for this problem is to recommend  $\{\pi_u(s)\}_{s=1\dots T/B}$ , the top  $T/B$  items with the highest reward,  $B$  times each. This leads us to the following notion of regret
$$
\operatorname {R e g} (T) \triangleq \sum_ {s \in [ T / B ], u \in [ M ]} \frac {\mathbf {B P} _ {u \pi_ {u} (s)}}{M} - \mathbb {E} \sum_ {t \in [ T ], u \in [ M ]} \frac {\mathbf {P} _ {u \rho_ {u} (t)}}{M}, \tag {2}
$$
where the expectation is over the randomness in the policy and the rewards, and  $\rho_u(t)$  is any policy that satisfies the budget constraints.
Importance of collaboration. Suppose we treat each user independently and try to minimize their regret. In this case, when B is as small as  $O(\log T)$ , we will incur a regret that is almost linear in  $T$ . This is because we will not have enough data for any item to know if it has a high or a low reward. This shows the importance of collaboration across users to achieve optimal regret. The latent cluster structure in our problem allows for collaboration and sharing information about items across users.
For ease of exposition of our ideas, we introduce a couple of definitions.
Definition 1. A subset of users  $S \subseteq [\mathsf{M}]$  is called "nice" if  $S \equiv \bigcup_{j \in \mathcal{A}} \mathcal{C}^{(j)}$  for some  $\mathcal{A} \subseteq [\mathsf{C}]$ . In other words,  $S$  can be represented as the union of some subset of clusters.
Definition 2. For any user  $u \in [\mathsf{M}]$ , we call the set of items  $\{\pi_u(t)\}_{t=1}^{\mathsf{T}/\mathsf{B}}$  (i.e., the set of best  $\mathsf{T} / \mathsf{B}$  items) to be the golden items for user  $u$ .
2This corresponds to the following analogue of multi-agent multi-armed bandits (MAB) - we have M users each involved in a separate MAB problem with the same set of N arms (corresponding to the N items) and T rounds. The mean reward of each arm is different for each user captured by the  $\mathsf{M}\times \mathsf{N}$  reward matrix  $\mathbf{P}$ . In each round  $t$ , every agent  $u\in [\mathsf{M}]$  simultaneously pulls an unblocked arm  $\rho_u(t)$  of their choice based on the feedback history of all users (including  $u$ ) from previous rounds. Subsequently the noisy feedback for all M users and their corresponding arm pulls at round  $t$  is revealed to everyone.
2.1 Low-Rank Matrix Completion
Additional Notation. For an estimate  $\widehat{\mathbf{P}}$  of reward matrix  $\mathbf{P}$ , we will use  $\widetilde{\pi}_u:[\mathbb{N}]\to [\mathbb{N}]$  to denote the items ordered according to their estimated reward for the user  $u$  i.e.,  $\widehat{\mathbf{P}}_{u\widetilde{\pi}_u(i)}\geq \widehat{\mathbf{P}}_{u\widetilde{\pi}_u(j)}$  when  $i < j$ . For any user  $u\in [\mathsf{M}]$  and any subset of items  $\mathcal{A}\subseteq [\mathsf{N}]$ , we will use  $\pi_u\mid \mathcal{A}:[|\mathcal{A}|]\to [\mathsf{N}]$  to denote the permutation  $\pi_u$  restricted to items in  $\mathcal{A}\subseteq [\mathsf{N}]$  i.e. for any user  $u\in [\mathsf{M}]$  and any  $i,j\in [|\mathcal{A}|]$  satisfying  $i < j$ , we will have  $\mathbf{P}_{u\pi_u(i)|\mathcal{A}}\geq \mathbf{P}_{u\pi_u(j)|\mathcal{A}}$ . Here,  $\pi_u(i)\mid \mathcal{A}$  corresponds to the  $i^{\mathrm{th}}$  item among items in  $\mathcal{A}$  sorted in descending order of expected reward for user  $u$ .
An important workhorse of our algorithm is low-rank matrix completion, which is a well studied problem in the literature [16, 18, 8, 17]. Given a small set of entries that are randomly sampled from a matrix, these algorithms infer the missing values of the matrix. More precisely, consider a low-rank matrix  $\mathbf{Q} \in \mathbb{R}^{M \times N}$ . Let  $\Omega \subseteq [M] \times [N]$  be a random set of indices obtained by picking each index in  $[M] \times [N]$  independently with probability  $p > 0$ . Let  $\{\mathbf{Z}_{ij}\}_{(i,j) \in \Omega}$  be the corresponding noisy entries (sub-gaussian random variables with variance proxy  $\sigma^2 > 0$ ) that we observe which satisfy  $\mathbb{E}\mathbf{Z}_{ij} = \mathbf{Q}_{ij} \forall (i,j) \in \Omega$ . In this work we rely on nuclear norm minimization to estimate  $\mathbf{Q}$  (see Algorithm 4). Recent works have provided tight element-wise recovery guarantees for this algorithm [8, 17]. We state these guarantees in the following Lemma.
Lemma 1 (Lemma 2 in [17]). Consider matrix  $\mathbf{Q} \in \mathbb{R}^{M \times N}$  with rank  $r$  and SVD decomposition  $\bar{\mathbf{U}}\Sigma \bar{\mathbf{V}}^{\top}$  satisfying  $\|\bar{\mathbf{U}}\|_{2,\infty} \leq \sqrt{\mu r / M}$ ,  $\|\bar{\mathbf{V}}\|_{2,\infty} \leq \sqrt{\mu r / N}$  and condition number  $\kappa = O(1)$ . Let  $d_1 = \max(M, N)$ ,  $d_2 = \min(M, N)$ , noise variance  $\sigma^2 > 0$ , and let sampling probability  $p$  be such that  $p = \widetilde{\Omega}\left(\frac{\mu^2}{d_2} \max\left(1, \frac{\sigma^2\mu}{||\mathbf{P}||_\infty^2}\right)\right)$  (for some constant  $c > 0$ ). Suppose we sample a subset of indices  $\Omega \subseteq [M] \times [N]$  such that each tuple of indices  $(i,j) \in [M] \times [N]$  is included in  $\Omega$  independently with probability  $p$ . Let  $\{\mathbf{Z}_{ij}\}_{(i,j) \in \Omega}$  be the noisy observations corresponding to all indices in  $\Omega$ . Then Algorithm 4, when run with inputs  $(M, N, \sigma^2, r, \Omega, \{\mathbf{Z}_{ij}\}_{(i,j) \in \Omega})$ , returns an estimate  $\widetilde{\mathbf{Q}}$  of  $\mathbf{Q}$  which satisfies the following error bounds with probability at least  $1 - O(\delta + d_2^{-12})$
$$
\left\| \mathbf {Q} - \widetilde {\mathbf {Q}} \right\| _ {\infty} = O \left(\frac {\sigma \sqrt {\mu^ {3} \log d _ {1}}}{\sqrt {p d _ {2}}}\right). \tag {3}
$$
Lemma 1 provides guarantees on the estimate  $\widetilde{\mathbf{Q}}$  of  $\mathbf{Q}$  given the set of noisy observations corresponding to the entries in  $\Omega$ . Equation 3 says that the quality of the estimate becomes better with the increase in sampling probability  $p$  that determines the number of entries in  $\Omega$ . Note the detailed Algorithm ESTIMATE (Alg. 4) designed in [17, 8] is provided in Appendix A.
Remark 1 (Warm-up (Greedy Algorithm)). Using Lemma 1, it is easy to design a greedy algorithm (Alg. 5 in Appendix B) for  $\mathsf{B} = 1$ ; for a fixed number of rounds  $m$ , we recommend random unblocked items to every user. Since the reward matrix  $\mathbf{P}$  is a low-rank matrix, we use Algorithm 4 to estimate the entire matrix. Subsequently, we recommend the best estimated unblocked items for each user for the remaining rounds. We can prove that the regret suffered by such an algorithm is  $\tilde{O}(\mathsf{T}^{2/3}(1 \vee (\mathsf{N} / \mathsf{M}))^{1/3})$ . The dependence on number of rounds  $\mathsf{T}$  is sub-optimal, but Alg. 5 does not require the knowledge of any gaps corresponding to the reward matrix  $\mathbf{P}$ . See Appendix B for a detailed proof. We would like to note that the result can be generalized easily to  $\mathsf{B} = \Theta(\log \mathsf{T})$ .
Let  $\mathbf{X} \in \mathbb{R}^{C \times N}$  be the sub-matrix of the expected reward matrix  $\mathbf{P}$  comprising  $C$  distinct rows of  $\mathbf{P}$  corresponding to each of the  $C$  true clusters. Also, let  $\tau := \max_{i,j \in [r]} |\mathcal{C}^{(i)}| / |\mathcal{C}^{(j)}|$  denote the ratio of the maximum and minimum cluster size. To obtain regret bounds, we make the following assumptions on the matrix  $\mathbf{X}$ :