SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. We will prove the claim via induction. Notation-wise, suppose for any  $\ell' \leq \ell$ , the user  $u$  belongs to the set  $\mathcal{M}^{(\ell', i)}$  and the set of active items at round  $t_1$  for users in  $\mathcal{M}^{(\ell, i)}$  is denote by  $\mathcal{N}$  (for simplicity, we remove the superscripts). The base case for  $\mathsf{L} = 0$  is true by construction - here the item  $a_1$  is recommended in the explore component of phase  $p_2$  at round  $t_2$ . We find an unblocked item  $a_2$  in the set  $\mathcal{N} \subseteq \mathcal{N}^{(\ell, i)}$  (note that items are never added to the active set across phases and only pruned), recommend it at round  $t_1$  according to our algorithm. For analysis, we modify
$$
\theta_ {u} (t _ {1}) = a _ {1} \text {a n d} \theta_ {u} (t _ {2}) = a _ {2}
$$
Now, suppose our claim is true for some  $\mathsf{L} = l$ . Now, for  $\mathsf{L} = l + 1$ , note that  $a_1$  was recommended in the exploit component of phase  $p_2$  at round  $t_2$ ; this implies that  $a_1$  must have been used to replace another item  $a_2$  via a swapping operation of length  $l$ . By our induction hypothesis, we must have  $\theta_u(t_i) = a_i$  for all  $2 \leq i \leq \mathsf{L}$  (as a matter of fact, we will have  $\theta_u(t_{\mathsf{L} + 1}) = a_1$ ). Therefore, only the pair of modifications  $\theta_u(t_1) = a_1$  and  $\theta_u(t_{\mathsf{L} + 1}) = a_{\mathsf{L} + 1}$  suffice to bring the desired changes in  $\theta_u$ .
We can also conclude from Claim 1 that 1) at any round  $t$  in the exploit component of some phase for user  $u$ , if the chosen item to be recommended is found to be blocked, then that chosen item is brought to the  $t^{\text{th}}$  position in the sequence  $\theta_u$  2) Once the chosen item is brought to its correct position in  $\theta_u$ , it will not be modified/moved in any future round. 3) All chosen items for recommendation to user  $u$  corresponding to exploit components are moved to their correct position (i.e. the intended round for their recommendation) in the sequence  $\theta_u$ . Next we make the following claim:
Claim 2. Consider the setting in Claim 1. It must happen that all the items  $a_1, a_2, \ldots, a_{\mathsf{L} + 1}$  must belong to the set  $\mathcal{N}^{(p_{\mathsf{L} + 1},i)}$  - the set of active items in the phase  $p_{\mathsf{L} + 1}$  for users in the nice subset  $\mathcal{M}^{(p_{\mathsf{L} + 1},i)}$  to which  $u$  belongs and  $a_{\mathsf{L}}$  was recommended in its explore component.
Proof. Again, note that items are never added to the active set across phases and only pruned. If an item  $a_1$  is replaced by  $a_{\mathsf{L} + 1}$  via a swapping operation of length  $\mathsf{L} + 1$ , it implies that  $a_1$  was used to replace item  $a_2$  (via a swapping operation of length  $\mathsf{L}$ ),  $a_2$  was used to replace  $a_3$  (via a swapping operation of length  $\mathsf{L} - 1$ ) and so on. The final golden item for user  $u$  in this sequence  $a_{\mathsf{L}}$  was recommended in phase  $p_{\mathsf{L} + 1}$  in the explore component and belonged to the set of active items  $\mathcal{N}^{(p_{\mathsf{L} + 1},i)}$ . Hence, this implies that all the subsequent surviving items  $a_1, a_2, a_3, \ldots, a_{\mathsf{L} - 1}, a_{\mathsf{L} + 1}$  must have belonged to the set of active items  $\mathcal{N}^{(p_{\mathsf{L} + 1},i)}$  as well.
From Claim 2, we can conclude that for any round  $t$  in the explore component of some phase  $\ell$  for user  $u$ , if  $\rho_u(t)$  is replaced in the sequence  $\theta_u$  at round  $t$ , then  $\rho_u(t), \theta_u(t)$  belong to the same set of active items  $\mathcal{N}^{(\ell, i)}$ . With both these arguments, we complete the proof of the lemma.
D.2.2 Final Regret analysis
Let us condition on the event  $\mathcal{E}$ . Consider any user  $u\in [\mathsf{M}]$  - recall that  $\{\rho_u(t)\}_{t\in [\mathsf{T}]}$  is the random variable denoting the sequence of  $\mathsf{T}$  items recommended to the user  $u$ ,  $\rho_{u}(t)$  is a realization of  $\{\rho_u(t)\}_{t\in [\mathsf{T}]}$  conditioned on the event  $\mathcal{E}$ . Furthermore,  $\theta_{u}$  is the modified sequence - a permutation of the  $\mathsf{T}$  items  $\{\rho_u(t)\}$  recommended for user  $u$ . For simplicity of notation, we will assume that  $u\in \mathcal{M}^{(\ell ,i)}$  for every phase  $\ell$ . Hence the active set of items during the explore component of phase  $\ell$  for the nice subset of users  $\mathcal{M}^{(\ell ,i)}$  is denoted by  $\mathcal{N}^{(\ell ,i)}$ . Also, in line with our previous usage of notations, let us denote  $\mathcal{O}_{\mathcal{M}^{(\ell ,i)}}^{(\ell)}\setminus \mathcal{O}_{\mathcal{M}^{(\ell ,i)}}^{(\ell -1)}$  to be the set of items chosen for recommendation particularly in the exploit component of phase  $\ell$ . Moreover,  $\mathcal{O}$  denotes the entire set of items chosen for recommendation in all the exploit components to user  $u$ . In other words, if  $\ell_{u}$  is the final phase to which user  $u$  has progressed then  $\mathcal{O}_u = \mathcal{O}_{\mathcal{M}^{(\ell_u,i)}}^{(\ell_u)}$ . We denote the regret for the user  $u$  by
$$
\operatorname {R e g} _ {u} (T) \mid \mathcal {E} \triangleq = \mathbb {E} \left[ \left(\sum_ {t \in [ T ]} \mathbf {P} _ {u \pi_ {u} (t)} - \mathbf {P} _ {u \rho_ {u} (t)}\right) \mid \mathcal {E} \right]
$$
where the expectation is over the randomness in the algorithm and the noise in the observations. Note that with this notation, we have  $\operatorname{Reg}(\mathsf{T})\mid \mathcal{E} = \mathsf{M}^{-1}\sum_{u\in [\mathsf{M}]}\operatorname{Reg}_u(\mathsf{T})\mid \mathcal{E}$ . We now have the following set of inequalities for regret of user  $u$ :
$$
\begin{array}{l} \operatorname {R e g} _ {u} (T) \mid \mathcal {E} \triangleq \mathbb {E} \left[ \left(\sum_ {t \in [ T ]} \mathbf {P} _ {u \pi_ {u} (t)} - \mathbf {P} _ {u \rho_ {u} (t)}\right) \mid \mathcal {E} \right] = \mathbb {E} \left[ \left(\sum_ {t \in [ T ]} \mathbf {P} _ {u \pi_ {u} (t)} - \mathbf {P} _ {u \rho_ {u} (t)}\right) \mid \mathcal {E} \right] \\ = \sum_ {\rho_ {u} (t)} \Pr (\rho_ {u} (t) = \rho_ {u} (t) \mid \mathcal {E}) \Big (\sum_ {t \in [ T ]} \mathbf {P} _ {u \pi_ {u} (t)} - \mathbf {P} _ {u \theta_ {u} (t)} \Big) \\ = \sum_ {\rho_ {u} (t)} \Pr (\rho_ {u} (t) = \rho_ {u} (t) \mid \mathcal {E}) \Big (\sum_ {t \in [ \mathrm {T} ]} \mathbf {P} _ {u \pi_ {u} (t)} - \mathbf {P} _ {u \widetilde {\theta} _ {u} (t)} \Big). \\ \end{array}
$$
Let us denote the set of rounds in the exploit component of  $\ell^{\mathrm{th}}$  phase by  $\mathsf{exploit}(\ell, \mathcal{M}^{(\ell,i)})$  and the explore component of  $\ell^{\mathrm{th}}$  phase by  $\mathsf{explore}(\ell, \mathcal{M}^{(\ell,i)})$ . Therefore we can further decompose the regret for user  $u$  as follows:
$$
\begin{array}{l} \operatorname {R e g} _ {u} (T) \mid \mathcal {E} = \sum_ {\rho_ {u} (t)} \Pr (\rho_ {u} (t) = \rho_ {u} (t) \mid \mathcal {E}) \Big (\sum_ {\ell \in [ \ell_ {u} ]} \sum_ {t \in \text {e x p l o i t} (\ell , \mathcal {M} ^ {(\ell , i)})} \left(\mathbf {P} _ {u \pi_ {u} (t)} - \mathbf {P} _ {u \tilde {\theta} _ {u} (t)}\right) \\ + \sum_ {\ell \in [ \ell_ {u} ]} \sum_ {t \in \text {e x p l o r e} (\ell , \mathcal {M} ^ {(\ell , i)})} \left(\mathbf {P} _ {u \pi_ {u} (t)} - \mathbf {P} _ {u \widetilde {\theta} _ {u} (t)}\right)). \\ \end{array}
$$
Next, our arguments are conditioned on the event  $\mathcal{E}$  and any sequence  $\rho_u(t)$  having a non-zero probability of appearing conditioned on event  $\mathcal{E}$ . Recall in Lemma 15, we proved by induction that  $\mathcal{O}_u \subseteq \{\pi_u(t)\}_{t=1}^T$ . Moreover, in Lemma 18, we proved that items chosen for recommendation in the exploit components for user  $u$  are in their correct positions (i.e. the round when they were intended to be recommended but might have been found to be blocked) in the sequence  $\theta_u$ . Consider a permutation of the best  $T$  items for user  $u$ $\sigma_u:[T] \to \{\pi_u(t)\}_{t=1}^T$  such that
$$
\sigma_ {u} (t) = \theta_ {u} (t) \text {f o r a l l} t \in \cup_ {\ell} \exp \operatorname {l o p i t} (\ell , \mathcal {M} ^ {(\ell , i)})
$$
$$
\left\{\sigma_ {u} (t) \right\} _ {t \in \cup_ {\ell} \text {e x p l o r e} (\ell , \mathcal {M} ^ {(\ell , i)})} \equiv \left\{\pi_ {u} \left(t ^ {\prime}\right) \right\} _ {t ^ {\prime} \in [ T ]} \setminus \mathcal {O} _ {u}
$$
where in a round in any exploit component, the permutation  $\sigma$  maps the item chosen for recommendation (which we know to be among the best  $\mathsf{T}$  items) for user  $u$  to that round. For any round belonging to the explore component, the permutation  $\sigma$  arbitrarily maps the remaining items among the best  $\mathsf{T}$  items (namely the set  $\{\pi_u(t')\}_{t' \in [\mathsf{T}] \setminus \mathcal{O}_u}$ ). Notice that  $\sum_{t \in [\mathsf{T}]} \mathbf{P}_{u\pi_u(t)} = \sum_{t \in [\mathsf{T}]} \mathbf{P}_{u\sigma_u(t)}$ . Therefore, we can further decompose the regret as
$$
\begin{array}{l} \operatorname {R e g} _ {u} (T) \mid \mathcal {E} = \sum_ {\rho_ {u} (t)} \Pr (\rho_ {u} (t) = \rho_ {u} (t) \mid \mathcal {E}) \Big (\sum_ {\ell \in [ \ell_ {u} ]} \sum_ {t \in \text {e x p l o i t} (\ell , \mathcal {M} ^ {(\ell , i)})} \left(\mathbf {P} _ {u \sigma_ {u} (t)} - \mathbf {P} _ {u \widetilde {\theta} _ {u} (t)}\right) \\ + \sum_ {\ell \in [ \ell_ {u} ]} \sum_ {t \in \text {e x p l o r e} (\ell , \mathcal {M} ^ {(\ell , i)})} \left(\mathbf {P} _ {u \sigma_ {u} (t)} - \mathbf {P} _ {u \widetilde {\theta} _ {u} (t)}\right)) \\ = \sum_ {\rho_ {u} (t)} \Pr (\rho_ {u} (t) = \rho_ {u} (t) \mid \mathcal {E}) \Big (\sum_ {\ell \in [ \ell_ {u} ]} \sum_ {t \in \text {e x p l o r e} (\ell , \mathcal {M} ^ {(\ell , i)})} \left(\mathbf {P} _ {u \sigma_ {u} (t)} - \mathbf {P} _ {u \widetilde {\theta} _ {u} (t)}\right) \Big) \\ \end{array}
$$
For the explore component in the phase indexed by  $\ell$ , we have proved in Lemma 14 that the active set of items  $\mathcal{N}^{(\ell, i)}$  is a superset of  $\{\pi_u(t')\}_{t' \in [\mathsf{T}]} \setminus \mathcal{O}_u$ . Therefore, we can bound (using Lemma 4 and condition C stated at beginning of sec. D.1) for any  $\ell \neq \ell_u$  ( $\ell_u$  denotes index of the final phase that user  $u$  was part of)
$$
\sum_ {t \in \mathrm {e x p l o r e} (\ell , \mathcal {M} ^ {(\ell , i)})} \left(\mathbf {P} _ {u \sigma_ {u} (t)} - \mathbf {P} _ {u \widetilde {\theta} _ {u} (t)}\right) \leq m _ {\ell} \cdot \epsilon_ {\ell} = O \Big (\frac {\sigma^ {2} \widetilde {\mu} ^ {3} \log (\mathsf {M} \bigvee \mathsf {N})}{\Delta_ {\ell + 1} ^ {2}} \max \left(1, \frac {\mathsf {N} \tau}{\mathsf {M}}\right) \log \mathsf {T}) \Big) \Big) \cdot \epsilon_ {\ell}.
$$
where  $\widetilde{\mu}$  is the incoherence factor of the sub-matrix  $\mathbf{P}_{\mathcal{M}^{(\ell,i)},\mathcal{N}^{(\ell,i)}}$ . Next, using the facts that  $\Delta_{\ell+1} = \epsilon_{\ell+1}/88\mathbb{C}$ ,  $2\epsilon_{\ell} = \epsilon_{\ell+1}$ ,  $\mathbb{C}, \tau = O(1)$ , we get that (after hiding log factors for simplicity)
$$
\sum_ {t \in \mathrm {e x p l o r e} (\ell , \mathcal {M} ^ {(\ell , i)})} \left(\mathbf {P} _ {u \sigma_ {u} (t)} - \mathbf {P} _ {u \widetilde {\theta} _ {u} (t)}\right) = \widetilde {O} \Big (\frac {\sigma^ {2} \widetilde {\mu} ^ {3}}{\epsilon_ {\ell}} \max  \Big (1, \frac {\mathsf {N}}{\mathsf {M}} \Big) \Big) = \widetilde {O} \Big (\frac {\sigma^ {2} \mu^ {3}}{\epsilon_ {\ell}} \max  \Big (1, \frac {\mathsf {N}}{\mathsf {M}} \Big) \Big).
$$
In the above statement, from Lemmas 2 and 3, we also used the fact that  $\widetilde{\mu} = O(\mu)$  and the condition number of the sub-matrix  $\mathbf{P}_{\mathcal{M}^{(\ell,i)},\mathcal{N}^{(\ell,i)}}$  is  $O(1)$ . Finally, for the explore component of the final phase  $\ell_u$ , from Lemma 11, we will also have (in the final phase, one of the edge case scenarios might appear)
$$
\sum_ {t \in \text {e x p l o r e} (\ell , \mathcal {M} ^ {(\ell , i)})} \left(\mathbf {P} _ {u \sigma_ {u} (t)} - \mathbf {P} _ {u \widetilde {\theta} _ {u} (t)}\right) = \widetilde {O} \Big (\sigma \mu^ {3 / 2} \max  \left(\sqrt {\mathsf {T}}, \sqrt {\frac {\mathsf {T} ^ {2}}{\mathsf {M}}}\right) \Big) + \widetilde {O} \Big (\frac {\sigma^ {2} \mu^ {3}}{\epsilon_ {\ell_ {u}}} \max  \left(1, \frac {\mathsf {N}}{\mathsf {M}}\right) \Big)
$$
Hence, we can put together everything to conclude that
$$
\begin{array}{l} \sum_ {\ell \in [ \ell_ {u} ]} \sum_ {t \in \text {e x p l o r e} (\ell , \mathcal {M} ^ {(\ell , i)})} \left(\mathbf {P} _ {u \sigma_ {u} (t)} - \mathbf {P} _ {u \widetilde {\theta} _ {u} (t)}\right) \\ = \sum_ {\ell} \widetilde {O} \left(\frac {\sigma^ {2} \mu^ {3}}{\epsilon_ {\ell}} \max  \left(1, \frac {\mathsf {N}}{\mathsf {M}}\right)\right) + \widetilde {O} \left(\sigma \mu^ {3 / 2} \max  \left(\sqrt {\mathsf {T}}, \sqrt {\frac {\mathsf {T} ^ {2}}{\mathsf {M}}}\right)\right) \\ \leq \sum_ {\ell : \epsilon_ {\ell} \leq \Phi} m _ {\ell} \Phi + \sum_ {\ell : \epsilon_ {\ell} \geq \Phi} \widetilde {O} \Big (\frac {\sigma^ {2} \mu^ {3}}{\Phi} \max  \Big (1, \frac {\mathsf {N}}{\mathsf {M}} \Big) \Big) + \widetilde {O} \Big (\sigma \mu^ {3 / 2} \max  \Big (\sqrt {\mathsf {T}}, \sqrt {\frac {\mathsf {T} ^ {2}}{\mathsf {M}}} \Big) \Big) \\ \leq \mathsf {T} \Phi + \mathsf {J} \cdot \widetilde {O} \left(\frac {\sigma^ {2} \mu^ {3}}{\Phi} \max  \left(1, \frac {\mathsf {N}}{\mathsf {M}}\right)\right) + \widetilde {O} \left(\sigma \mu^ {3 / 2} \max  \left(\sqrt {\mathsf {T}}, \sqrt {\frac {\mathsf {T} ^ {2}}{\mathsf {M}}}\right)\right) \\ \end{array}
$$
where  $\mathsf{J}$  is the number of phases with  $\epsilon_{\ell} \geq \Phi$ . By choosing  $\Phi = \sqrt{\frac{\sigma^2\mu^3}{T}\max\left(1,\frac{N}{M}\right)}$ , we can bound
$$
\sum_ {\ell \in [ \ell_ {u} ]} \sum_ {t \in \text {e x p l o r e} (\ell , \mathcal {M} ^ {(\ell , i)})} \left(\mathbf {P} _ {u \sigma_ {u} (t)} - \mathbf {P} _ {u \widetilde {\theta} _ {u} (t)}\right) = \widetilde {\mathcal {O}} \left(\mathrm {J} \sigma \mu^ {3 / 2} \sqrt {\mathrm {T} \max  \left(1 , \frac {\mathrm {N}}{\mathrm {M}}\right)}\right)
$$
where we used the fact that  $\mathsf{N} \gg \mathsf{T}$  and therefore the last term in the previous equation is a lower order term compared to the first two ones. Next, recall that we choose  $\epsilon_{\ell} = C^{\prime}2^{-\ell}\min \left(\| \mathbf{P}\|_{\infty},\frac{\sigma\sqrt{\mu}}{\log\mathsf{N}}\right)$  (so that the condition on  $\sigma >0$  in Lemma 1 is automatically satisfied for all  $\ell$ ) for some constant  $C^\prime >0$ , the maximum number of phases  $\ell$  for which  $\epsilon_{\ell} > \Phi$  can be bounded from above by  $\mathsf{J} = O\Big(\log \Big(\frac{1}{\Phi}\min \Big(\| \mathbf{P}\|_{\infty},\frac{\sigma\sqrt{\mu}}{\log\mathsf{N}}\Big)\Big)\Big)$ . Therefore, we can hide  $\mathsf{J}$  inside  $\widetilde{O}$  and obtain
$$
\sum_ {\ell \in [ \ell_ {u} ]} \sum_ {t \in \mathsf {e x p l o r e} (\ell , \mathcal {M} ^ {(\ell , i)})} \left(\mathbf {P} _ {u \sigma_ {u} (t)} - \mathbf {P} _ {u \widetilde {\theta} _ {u} (t)}\right) = \widetilde {O} \Big (\sigma \mu^ {3 / 2} \sqrt {\mathsf {T} \max \left(1 , \frac {\mathsf {N}}{\mathsf {M}}\right)} \Big).
$$
Therefore, we must have that
$$
\operatorname {R e g} _ {u} (T) \mid \mathcal {E} = \widetilde {O} \left(\sigma \mu^ {3 / 2} \sqrt {T \max  \left(1 , \frac {N}{M}\right)}\right) \Rightarrow \operatorname {R e g} (T) \mid \mathcal {E} = \widetilde {O} \left(\sigma \mu^ {3 / 2} \sqrt {T \max  \left(1 , \frac {N}{M}\right)}\right).
$$
Finally, we use the fact that
$$
\operatorname {R e g} (T) \leq \operatorname {R e g} (T) \mid \mathcal {E} + \Pr (\mathcal {E} ^ {c}) (\operatorname {R e g} (T) \mid \mathcal {E} ^ {c}).
$$
From Lemma 17, we know that  $\operatorname{Pr}(\mathcal{E}^c) \leq \mathsf{CT}^{-2}$ ,  $\operatorname{Reg}(\mathsf{T}) \mid \mathcal{E}^c \leq \mathsf{T}||\mathbf{P}||_{\infty}$  therefore,
$$
\operatorname {R e g} (T) \leq \operatorname {R e g} (T) \mid \mathcal {E} + \Pr (\mathcal {E} ^ {c}) (\operatorname {R e g} (T) \mid \mathcal {E} ^ {c}) = \widetilde {O} \left(\sigma \mu^ {3 / 2} \sqrt {\operatorname {T} \max  \left(1 , \frac {\mathrm {N}}{\mathrm {M}}\right)} + T ^ {- 1} | | \mathbf {P} | | _ {\infty}\right)
$$
which completes the proof of our main result.
E Proof of Lower Bound (Theorem 2)
Consider our problem setting with  $M$  users,  $N$  items and  $T$  rounds, blocking constraint  $B$ , noise variance proxy  $\sigma^2 = 1$  and all expected rewards in  $[0,1]$ . Here we consider  $C = 1$  i.e. all users belong to the same cluster. Let us denote a particular policy chosen by the recommendation system as  $\pi$  that belongs to the class of polices  $\Pi$ . Moreover, let us also denote by  $\mathcal{E}$  the set of possible environments corresponding to the expected reward matrices that has all rows to be same (satisfies cluster structure for  $C = 1$ ). In that case, the minimax regret is given by
$$
\inf_{\pi \in \Pi}\sup_{\nu \in \mathcal{E}}\operatorname {Reg}(\mathsf{T};\mathcal{E})
$$
E.1 Lower Bound via reduction
We can ease the problem by assuming that at each round  $t = 1,2,\ldots ,\mathsf{T}$ , the users come in a sequential fashion - the  $j^{\mathrm{th}}$  user is recommended an item based on all previous history of observations including the feedback obtained from recommending items from users  $1,2,\dots ,j - 1$  at round  $t$ . Furthermore, we also assume that the  $\mathsf{N}$  items can be partitioned into NB/T known groups where each group has identical items - hence we have a simple multi-armed bandit problem (MAB) with NB/T arms and MT rounds with a single user. A lower bound on this simplified MAB problem will imply a lower bound on our setting i.e by appropriately normalizing, we have
$$
\inf  _ {\pi \in \Pi} \sup  _ {\nu \in \mathcal {E}} \operatorname {R e g} (T; \mathcal {E}) \geq \frac {1}{M} \inf  _ {\pi \in \Pi} \sup  _ {\nu \in \mathcal {E}} \operatorname {R e g} _ {M A B} (M T; \mathcal {E}) = \Omega \left(\frac {1}{M} \sqrt {\frac {N B}{T} \cdot M T}\right) = \Omega \left(\sqrt {\frac {N B}{M}}\right)
$$
where we simply used the standard regret lower bound in multi-armed bandits with  $K$  arms and  $T$  rounds which is  $\Omega(\sqrt{K\overline{T}})$  [19].
E.2 Lower Bound via application of Fano's inequality
As before, we can ease the original problem by assuming that at each round  $t = 1,2,\ldots ,\mathsf{T}$ , the users come in a sequential fashion - the  $j^{\mathrm{th}}$  user is recommended an item based on all previous history of observations including the feedback obtained from recommending items from users  $1,2,\dots ,j - 1$  at round  $t$ . Clearly, a lower bound on the simplified problem will imply a lower bound on our setting i.e. by appropriately normalizing, we have
$$
\inf_{\pi \in \Pi}\sup_{\nu \in \mathcal{E}}\operatorname {Reg}(\mathsf{T};\mathcal{E})\geq \frac{1}{\mathsf{M}}\inf_{\pi \in \Pi}\sup_{\nu \in \mathcal{E}}\operatorname{Reg}_{\mathsf{MAB}}(\mathsf{MT};\mathcal{E}).
$$
Furthermore, ignoring the normalizing factor by  $\mathsf{M}$ , for  $C = 1$ , the simplified problem is equivalent to a standard Multi-armed bandit (MAB) problem with a single agent with MT rounds and an additional hard constraint that each item can be pulled at most MB times. We will construct  $\binom{\mathsf{N}}{\mathsf{T}\mathsf{B}^{-1}}$  environments in the following way: let  $\mathcal{T} \equiv \{S \subseteq [\mathsf{N}] \mid |\mathsf{S}| = \mathsf{T}\mathsf{B}^{-1}\}$  be the set of all subsets of  $[\mathsf{N}]$  of size  $\mathsf{T}\mathsf{B}^{-1}$ . Now for each subset  $S \in \mathcal{T}$ , we construct an environment by assuming that the agent on pulling any arm in the set  $S$  observes a random reward distributed according to  $\mathcal{N}(\Delta, 1)$  and on pulling any arm outside the set  $S$  observes a random reward distributed according to  $\mathcal{N}(0, 1)$ . This corresponds to the reward matrix  $\mathbf{P}$  (in our original problem) having an entry  $\Delta$  in the  $\mathsf{T}\mathsf{B}^{-1}$  columns indexed in  $S$  and 0 in the remaining columns. Let  $\mathbb{E}_S, \mathbb{P}_S, \mathcal{E}_S$  denote the expectation, probability measure and the environment if  $S \in \mathcal{T}$  is the set of chosen columns for constructing the environment.
Next we assume that the set  $S$  is chosen uniformly at random from  $\mathcal{T}$ . Hence we must have
$$
\inf_{\pi \in \Pi}\sup_{\nu \in \mathcal{E}}\operatorname{Reg}_{\mathrm{MAB}}(\mathrm{MT};\mathcal{E})\geq \inf_{\pi \in \Pi}\mathbb{E}_{\mathcal{S}\sim \mathcal{T}}\operatorname{Reg}_{\mathrm{MAB}}(\mathrm{T};\mathcal{E}_{\mathcal{S}})
$$
Fix any policy  $\pi \in \Pi$ . Condition on the set  $S$  being selected from  $\mathcal{T}$ . Let  $\mathsf{R}(\mathcal{S})$  be the number of times arms indexed in the set  $S$  are pulled in the MT rounds. In that case we must have
$$
\operatorname {R e g} _ {\mathrm {M A B}} (\mathsf {T}; \mathcal {E} _ {\mathcal {S}}) \geq \mathbb {P} _ {\mathcal {S}} (\mathsf {R} (\mathcal {S}) \leq \frac {3 \mathrm {M T}}{4}) \frac {\mathrm {M T} \Delta}{4}.
$$
Now, consider the estimation problem of which set  $S$  was selected from  $\mathcal{T}$ . Let  $\widehat{X}$  be an estimator that takes as input the observations in the MT rounds and returns a set  $\widehat{S}$  in the following way: it
finds  $\widehat{S}$  as the set of  $\mathsf{T}$  arms that have been pulled the most number of times jointly and returns  $\widehat{S}$  if  $\mathsf{R}(\widehat{S})\geq 3\mathsf{MT} / 4$  and the null set  $\varnothing$  otherwise. We consider the estimator  $\widehat{X}$  to make an error if it returns a set  $\widehat{S}$  such that  $\widehat{S}\cap S\leq T / 4\mathsf{B}$ . If the estimator  $\widehat{X}$  makes an error, note that  $\mathsf{R}(\widehat{S})\geq 3\mathsf{MT} / 4$  implies that  $\mathsf{R}(\widehat{S}\setminus S)\geq \mathsf{MT} / 2$  (since each arm can be pulled at most M times) - hence, it implies that  $\mathsf{R}(S)\leq \mathsf{MT} / 2$ . Therefore, if we denote Error as the error event, then we must have
$$
\begin{array}{l} \operatorname {R e g} _ {\mathrm {M A B}} (\mathsf {T}; \mathcal {E} _ {\mathcal {S}}) \geq \mathbb {P} _ {\mathcal {S}} (\mathsf {R} (\mathcal {S}) \leq \frac {3 \mathsf {M T}}{4}) \frac {\mathsf {M T} \Delta}{4} \geq \mathbb {P} _ {\mathcal {S}} (\text {E r r o r}) \frac {\mathsf {M T} \Delta}{4} \\ \Rightarrow \inf  _ {\pi \in \Pi} \sup  _ {\nu \in \mathcal {E}} \operatorname {R e g} _ {\mathrm {M A B}} (\mathrm {M T}; \mathcal {E}) \geq \mathbb {P} (\text {E r r o r}) \frac {\mathrm {M T} \Delta}{4} = \mathbb {E} _ {\mathcal {S} \sim \mathcal {T}} \mathbb {P} _ {\mathcal {S}} (\text {E r r o r}) \frac {\mathrm {M T} \Delta}{4} \\ \end{array}
$$
Therefore, our goal is to bound the quantity  $\mathbb{E}_{\mathcal{S}\sim \mathcal{T}}\mathbb{P}_S(\text{Error})$ . At this point we have reduced our problem to a multiple hypothesis testing problem. Therefore, in order to lower bound the probability of the event Error, we use Fano's inequality for approximate recovery:
Lemma (Fano's inequality with approximate recovery [28]). For any random variables  $V, \widehat{V}$  on alphabets  $\mathcal{V}, \widehat{\mathcal{V}}$ , consider an error when  $d(V, \widehat{V}) \geq t$  for some  $t > 0$  and distance function  $d: \mathcal{V} \times \widehat{\mathcal{V}} \to \mathbb{R}$ . In that case, if we denote the error event by  $\text{Error}$ , we must have
$$
\mathbb {P} (\text {E r r o r}) \geq 1 - \frac {I (V ; \widehat {V}) + \log 2}{\log \frac {| \mathcal {V} |}{G _ {\max }}}
$$
where  $\mathsf{G}_{\max} = \max_{\hat{v}\in \hat{\mathcal{V}}}\sum_{v\in \mathcal{V}}1[d(v,\hat{v})\leq t]$  and  $I(V;\widehat{V})$  is the mutual information between the random variables  $V$  and  $\widehat{V}$ .
In the special case when the random variable  $V$  is uniform, then we can upper bound the mutual information by  $I(V;\widehat{V})\leq \max_{v,\widehat{v}\in \mathcal{V}}\mathsf{KL}(P_{\widehat{V} |V = v}||P_{\widehat{V} |V = v'})\leq \max_{v,\widehat{v}\in \mathcal{V}}\mathsf{KL}(P_v||P_{v'})$  where the second inequality follows from Data-processing inequality ( $P_{v} = P(\cdot \mid v)$  corresponds to the probability of the observations given  $V = v$ ).