SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Suppose the induction statement is true for round  $t_a$  and the second possibility i.e. possibility  $B$  became true. In that case, we do not exit the recursion and our goal is to show that the lemma statement is true at the next decision round  $t_{a+1}$ . The induction hypothesis implies that for every user  $u \in \mathcal{M}^{(\ell+1,z)}$ , we have that  $\widetilde{\pi}_u(\mathsf{T}\mathsf{B}^{-1} - \left|\mathcal{O}_{\mathcal{M}^{(\ell,i)}}^{(\ell)}\right|)$  survives in the set of items  $\mathcal{N}^{(\ell+1,t_a,z)}$  and furthermore, we have  $\widetilde{\pi}_u(\mathsf{T}\mathsf{B}^{-1} - \left|\mathcal{O}_{\mathcal{M}^{(\ell,i)}}^{(\ell)}\right|) \mid \mathcal{N}^{(\ell,i)} = \widetilde{\pi}_u(s) \mid \mathcal{N}^{(\ell+1,t_a,z)}$  for  $s = \mathsf{T}\mathsf{B}^{-1} - \left|\mathcal{O}_{\mathcal{M}^{(\ell+1,z)}}^{(\ell+1,t_a)}\right|$  (note that  $s$  is common for all users in  $\mathcal{M}^{(\ell+1,z)}$ ). The induction hypothesis also implies that
$$
\mathcal {N} ^ {(\ell + 1, t _ {r}, z)} \supseteq \bigcup_ {u \in \mathcal {M} ^ {(\ell + 1, z)}} \left\{\pi_ {u} \left(t ^ {\prime}\right) \mid \mathcal {Z} \right\} _ {t ^ {\prime} = 1} ^ {s} \text {w h e r e} \mathcal {Z} \equiv [ N ] \backslash \mathcal {O} _ {\mathcal {M} ^ {(\ell + 1, z)}} ^ {(\ell + 1, t _ {a})} \text {a n d} s = T - \left| \mathcal {O} _ {\mathcal {M} ^ {(\ell + 1, z)}} ^ {(\ell + 1, t _ {a})} \right| \tag {25}
$$
$$
\mathcal {O} _ {\mathcal {M} ^ {(\ell + 1, z)}} ^ {(\ell + 1, t _ {r})} \subseteq \left\{\pi_ {u} (t) \right\} _ {t = 1} ^ {\mathrm {T B} ^ {- 1}} \tag {26}
$$
Again, at the decision round  $t_a$ , since the possibility  $B$  was true, for one of the users  $u \in \mathcal{M}^{(\ell + 1, z)}$ , we must have for  $s = T - \left| \mathcal{O}_{\mathcal{M}^{(\ell + 1, z)}}^{(\ell + 1, t_{a + 1})} \right|$ ,
$$
\widetilde {\mathbf {P}} _ {u \widetilde {\pi} _ {u} (1) | \mathcal {N} ^ {(\ell + 1, t _ {a}, z)}} - \widetilde {\mathbf {P}} _ {u \widetilde {\pi} _ {u} (s) | \mathcal {N} ^ {(\ell + 1, t _ {a}, z)}} \geq 6 4 \mathsf {C} \Delta_ {\ell + 1}
$$
implying that for every user  $v\in \mathcal{M}^{(\ell +1,z)}$  , we have
$$
\widetilde {\mathbf {P}} _ {v \widetilde {\pi} _ {v} (1) | \mathcal {N} ^ {(\ell + 1, t _ {a}, z)}} - \widetilde {\mathbf {P}} _ {v \widetilde {\pi} _ {v} (s) | \mathcal {N} ^ {(\ell + 1, t _ {a}, z)}} \geq 5 6 \mathsf {C} \Delta_ {\ell + 1}.
$$
The above equation further implies that (Lemma 8), for every user  $v \in \mathcal{M}^{(\ell + 1, z)}$ , we will have
$$
\mathbf {P} _ {v \pi_ {v} (1) | \mathcal {N} ^ {(\ell + 1, t _ {a}, z)}} - \mathbf {P} _ {v \pi_ {v} (s) | \mathcal {N} ^ {(\ell + 1, t _ {a}, z)}} \geq 5 0 \mathsf {C} \Delta_ {\ell + 1}.
$$
Hence, as mentioned before, if we consider the set of items  $\mathcal{S} \equiv \cup_{u \in \mathcal{M}^{(\ell + 1, z)}} \mathcal{R}_u^{(\ell)} \mid \mathcal{N}^{(\ell + 1, t_a, z)}$ , then from Lemma 13 (or see eq. 15), we must have that for every user  $u \in \mathcal{M}^{(\ell + 1, z)}$ ,  $\pi_u(1) \mid \mathcal{N}^{(\ell + 1, t_a, z)} \in \mathcal{S}$  and
$$
\max  _ {y \in S} \left| \mathbf {P} _ {u \pi_ {u} (1) \mid \mathcal {N} ^ {(\ell + 1, t _ {a}, z)}} - \mathbf {P} _ {u y} \right| \leq 1 6 C \Delta_ {\ell + 1}. \tag {27}
$$
Hence, if we remove the set  $\mathcal{S}$  to update  $\mathcal{N}^{(\ell + 1, t_a, z)}$  i.e.  $\mathcal{N}^{(\ell + 1, t_{a+1}, z)} \gets \mathcal{N}^{(\ell + 1, t_a, z)} \setminus \mathcal{S}$ , for every user  $u \in \mathcal{M}^{(\ell + 1, z)}$ , we must have
$$
\begin{array}{l} \widetilde {\pi} _ {u} \left(\mathsf {T B} ^ {- 1} - \left| \mathcal {O} _ {\mathcal {M} (\ell , i)} ^ {(\ell)} \right|\right) \mid \mathcal {N} ^ {(\ell , i)} = \widetilde {\pi} _ {u} \left(\mathsf {T B} ^ {- 1} - \left| \mathcal {O} _ {\mathcal {M} ^ {(\ell + 1, z)}} ^ {(\ell + 1, t _ {a})} \right|\right) \mid \mathcal {N} ^ {(\ell + 1, t _ {a}, z)} \\ = \widetilde {\pi} _ {u} (\mathsf {T B} ^ {- 1} - \left| \mathcal {O} _ {\mathcal {M} ^ {(\ell + 1, z)}} ^ {(\ell + 1, t _ {a + 1})} \right|) \mid \mathcal {N} ^ {(\ell + 1, t _ {a + 1}, z)}. \\ \end{array}
$$
This is because for each user  $u$ , only elements which have larger rewards than  $\widetilde{\pi}_u(\mathsf{TB}^{-1} - \left|\mathcal{O}_{\mathcal{M}(\ell +1,z)}^{(\ell +1,t_a)}\right|)$  or  $\mathcal{N}^{(\ell +1,t_a,z)}$  are removed which makes the aforementioned item survive in  $\mathcal{N}^{(\ell +1,t_{a + 1},z)}$  and also moves its position up (recall that  $\mathcal{O}_{\mathcal{M}(\ell +1,z)}^{(\ell +1,t_{a + 1})}\gets \mathcal{O}_{\mathcal{M}(\ell +1,z)}^{(\ell +1,t_a)}\cup \mathcal{S})$  in the list of surviving items sorted in decreasing order by expected reward. Hence, we can apply Lemma
9 to conclude the first part of the induction proof. In order to show the final statement, with  $s = \mathsf{T}\mathsf{B}^{-1} - \left|\mathcal{O}_{\mathcal{M}(\ell +1,z)}^{(\ell +1,t_r)}\right|$ , we can simply substitute that
$$
\max _ {v \in \mathcal {M} (\ell + 1, z)} \left(\widetilde {\mathbf {P}} _ {u \widetilde {\pi} _ {u} (1) | \mathcal {N} ^ {(\ell + 1, t _ {r}, z)}} - \widetilde {\mathbf {P}} _ {u \widetilde {\pi} _ {u} (s) | \mathcal {N} ^ {(\ell + 1, t _ {r}, z)}}\right) \leq 6 4 \mathsf {C} \Delta_ {\ell + 1}
$$
when possibility  $A$  became true at a decision round  $t_r$  and we exit the exploit component to enter the explore component of phase  $\ell + 1$  for users in  $\mathcal{M}^{(\ell + 1, z)}$ .
Moreover, we will also have that
$$
\begin{array}{l} \mathcal {S} \subseteq \left\{\pi_ {u} (r) \mid \mathcal {N} ^ {(\ell + 1, t _ {a}, z)} \right\} _ {r = 1} ^ {s} \text {w h e r e} s = \mathrm {T B} ^ {- 1} - \left| \mathcal {O} _ {\mathcal {M} ^ {(\ell + 1, z)}} ^ {(\ell + 1, t _ {a})} \right| (28) \\ \Rightarrow \mathcal {S} \subseteq \left\{\pi_ {u} (r) \mid \mathcal {Z} \right\} _ {r = 1} ^ {s} \text {w h e r e} \mathcal {Z} \equiv [ N ] \backslash \mathcal {O} _ {\mathcal {M} (\ell + 1, z)} ^ {(\ell + 1, t _ {a})} \text {a n d} s = T - \left| \mathcal {O} _ {\mathcal {M} (\ell + 1, z)} ^ {(\ell + 1, t _ {a})} \right| (29) \\ \Rightarrow \mathcal {S} \subseteq \left\{\pi_ {u} (t) \right\} _ {t = 1} ^ {\mathrm {T B} ^ {- 1}} (30) \\ \Rightarrow \mathcal {N} ^ {(\ell + 1, t _ {a + 1}, z)} \supseteq \bigcup_ {u \in \mathcal {M} ^ {(\ell + 1, z)}} \left\{\pi_ {u} \left(t ^ {\prime}\right) \mid \mathcal {Z} \right\} _ {t ^ {\prime} = 1} ^ {\mathrm {T B} ^ {- 1} - s} (31) \\ \end{array}
$$
$$
\text {w h e r e} \mathcal {Z} \equiv [ N ] \backslash \mathcal {O} _ {\mathcal {M} ^ {(\ell + 1, z)}} ^ {(\ell + 1, t _ {a + 1})} \text {a n d} s = \left| \mathcal {O} _ {\mathcal {M} ^ {(\ell + 1, z)}} ^ {(\ell + 1, t _ {a + 1})} \right| \tag {32}
$$
This first implication is due to our induction hypothesis (see eq.25) which implies that the best  $s$  items in the smaller set  $\mathcal{N}^{(\ell +1,t_a,z)}$  is same as the best  $s = \mathsf{T}\mathsf{B}^{-1} - \left|\mathcal{O}_{\mathcal{M}^{(\ell +1,z)}}^{(\ell +1,t_a)}\right|$  items in the larger set [N]  $\backslash \mathcal{O}_{\mathcal{M}^{(\ell +1,z)}}^{(\ell +1,t_a)}$ . Hence, it is evident that the set  $\mathcal{S}$  must also be a subset of the best  $\mathsf{T}\mathsf{B}^{-1}$  items (golden items) for user  $u$  namely  $\{\pi_u(t)\}_{t\in [\mathsf{T}\mathsf{B}^{-1}]}$ . Since the above facts are true for all users  $u\in \mathcal{M}^{(\ell +1,z)}$ , we can also conclude that the new pruned set of items  $\mathcal{N}^{(\ell +1,t_{a + 1},z)}$  at the next decision round  $t_{a + 1}$  is a superset of the best  $\mathsf{T}\mathsf{B}^{-1} - \left|\mathcal{O}_{\mathcal{M}^{(\ell +1,z)}}^{(\ell +1,t_{a + 1})}\right|$  items for every user  $u$ . This completes the second part of the induction proof.
Lemma 16. Conditioned on the events  $\mathcal{E}_2^{(\ell)},\mathcal{E}_3^{(\ell)}$  , with choice of  $\Delta_{\ell +1} = \epsilon_{\ell +1} / 88C$  conditions  $A - C$  will be satisfied at the beginning of the explore component of phase  $\ell$  for the different nice subsets  $\{\mathcal{M}^{(\ell +1,z)}\}_{z}$  and their corresponding set of active arms  $\{\mathcal{N}^{(\ell +1,z)}\}_{z}$  implying that the event  $\mathcal{E}_2^{(\ell +1)}$  will be true.
Proof. 1. Proof of condition A: Due to our induction hypothesis, condition A is true for the explore component of phase  $\ell$ . Hence, the explore component of phase  $\ell$  is implemented separately and asynchronously for each disjoint nice subset of users  $\mathcal{M}^{(\ell,i)} \in \mathcal{M}'^{(\ell)} \subseteq \mathcal{M}^{(\ell)}$  (recall that  $\mathcal{M}'^{(\ell)}$  corresponds to the nice subsets of users which do not fall into the edge case scenarios). Let us fix one such nice subset of users  $\mathcal{M}^{(\ell,i)} \in \mathcal{M}'^{(\ell)}$ . After the successful low rank matrix completion step (event  $\mathcal{E}_3^{(\ell)}$  is true), we find the connected components of a graph  $\mathcal{G}^{(\ell,i)}$  which in turn correspond to nice subsets of users as well (see Lemma 7). Since the above facts are true for all nice subsets of users in  $\mathcal{M}'^{(\ell)}$ , the nice subsets of users that progress to the  $(\ell+1)^{\text{th}}$  phase are disjoint. Since these set of users are not modified during the exploit component of phase  $\ell+1$ , condition A is true at the beginning of the explore component of phase  $\ell+1$ .
2. Proof of conditions  $B$  and  $C$ : Again, let us fix a subset of nice users  $\mathcal{M}^{(\ell +1,z)}$  that has progressed to phase  $\ell +1$  and was in turn a part of the nice subset of users  $\mathcal{M}^{(\ell ,i)}$  in phase  $\ell$ . In other words, the set of users  $\mathcal{M}^{(\ell +1,z)}$  corresponds to a connected component of the graph  $\mathcal{G}^{(\ell ,i)}$ . From Lemma 15, we can conclude that conditions B and C are true at the beginning of the explore component of phase  $\ell +1$  for users in  $\mathcal{M}^{(\ell +1,z)}$  with choice of  $\Delta_{\ell +1} = \epsilon_{\ell +1} / 88C$  where  $\epsilon_{\ell +1}$  was pre-determined. Therefore, the conditions B and C hold for all nice subsets of users  $\{\mathcal{M}^{(\ell +1,z)}\}_{z}$  that have progressed to phase  $\ell +1$ .
Hence, conditioned on the events  $\mathcal{E}_2^{(\ell)},\mathcal{E}_3^{(\ell)}$  , with choice of  $\Delta_{\ell +1} = \epsilon_{\ell +1} / 88C$  , the algorithm will be  $(\epsilon_{\ell +1},\ell +1)$  -good and the event  $\mathcal{E}_2^{(\ell +1)}$  will be true.
D.2 Analyzing the regret guarantee
Lemma 17. Consider a fixed decreasing sequence  $\{\epsilon_{\ell}\}_{\ell \geq 1}$  where  $\epsilon_1 = ||\mathbf{P}||_{\infty}$  and  $\epsilon_{\ell} = C^{\prime}2^{-\ell}\min \left(\| \mathbf{P}\|_{\infty},\frac{\sigma\sqrt{\mu}}{\log\mathsf{N}}\right)$  for  $\ell >1$  for some constant  $C^\prime >0$  . Let us denote the event  $\mathcal{E} = \bigcap_{\ell}\mathcal{E}_{2}^{(\ell)}\bigcap_{\ell}\mathcal{E}_{3}^{(\ell)}$  to imply that our algorithm is  $(\epsilon_{\ell},\ell)$  -good at all phases indexed by  $\ell$  and the explore components of all phases are successful with the length of the explore component of phase  $\ell$  being
$$
m _ {\ell} = O \Big (\frac {\sigma^ {2} \widetilde {\mu} ^ {3} \log (M \bigvee N)}{\Delta_ {\ell + 1} ^ {2}} \max  \left(1, \frac {N \tau}{M}\right) \log T) \Big) \Big).
$$
The above statement implies that for any nice subset of users  $\mathcal{M}^{(\ell, i)}$  that has progressed to the  $\ell^{\mathrm{th}}$  phase with active items  $\mathcal{N}^{(\ell, i)}$  at the beginning of the explore components, with  $m_{\ell}$  rounds, we can compute an estimate  $\widetilde{\mathbf{P}}_{\mathcal{M}^{(\ell, i)}, \mathcal{N}^{(\ell, i)}}$  of  $\mathbf{P}_{\mathcal{M}^{(\ell, i)}, \mathcal{N}^{(\ell, i)}}$  satisfying
In that case, the event  $\mathcal{E}$  is true with probability at least  $1 - \mathsf{CT}^{-2}$ .
Proof. Notice that
$$
\begin{array}{l} \Pr (\mathcal {E} ^ {c}) = 1 - \Pr \left(\bigcup_ {\ell} \mathcal {E} _ {2} ^ {(\ell) c} \bigcup_ {\ell} \mathcal {E} _ {3} ^ {(\ell) c}\right) \\ \geq 1 - \left(\Pr \left(\mathcal {E} _ {2} ^ {(1) c}\right) + \Pr \left(\mathcal {E} _ {3} ^ {(1) c} \mid \mathcal {E} _ {2} ^ {(1)}\right)\right) \\ - \sum_ {\ell > 1} \left(\Pr \left(\mathcal {E} _ {2} ^ {(\ell) c} \mid \bigcap_ {\ell^ {\prime} <   \ell} \left(\mathcal {E} _ {2} ^ {(\ell^ {\prime})} \cap \mathcal {E} _ {3} ^ {(\ell)}\right)\right) + \Pr \left(\mathcal {E} _ {3} ^ {(\ell) c} \mid \mathcal {E} _ {2} ^ {(\ell) c}, \bigcap_ {\ell^ {\prime} <   \ell} \left(\mathcal {E} _ {2} ^ {(\ell^ {\prime})} \cap \mathcal {E} _ {3} ^ {(\ell)}\right)\right)\right) \\ \geq 1 - \left(\Pr \left(\mathcal {E} _ {2} ^ {(1) c}\right) + \Pr \left(\mathcal {E} _ {3} ^ {(1) c} \mid \mathcal {E} _ {2} ^ {(1)}\right)\right) \\ - \sum_ {\ell > 1} \left(\Pr \left(\mathcal {E} _ {2} ^ {(\ell) c} \mid \mathcal {E} _ {2} ^ {(\ell - 1)}, \mathcal {E} _ {3} ^ {(\ell - 1)}\right) + \Pr \left(\mathcal {E} _ {3} ^ {(\ell) c} \mid \mathcal {E} _ {2} ^ {(\ell)}\right)\right) \geq 1 - C T ^ {- 2} \\ \end{array}
$$
where we used the following facts 1)  $\operatorname{Pr}(\mathcal{E}_2^{(1)c}) = 0$  and  $\operatorname{Pr}(\mathcal{E}_2^{(\ell)c} \mid \mathcal{E}_2^{(\ell-1)}, \mathcal{E}_3^{(\ell-1)}) = 0$  for all  $\ell$  (Lemma 16) 2)  $\operatorname{Pr}(\mathcal{E}_3^{(\ell)c} \mid \mathcal{E}_2^{(\ell)}) \leq \mathrm{CT}^{-2}$  implied from Lemma 4 with additional union bounds over the number of phases (at most the number of rounds  $\mathsf{T}$ ) and the number of disjoint nice subsets of users that have progressed in each phase (at most the number of clusters  $\mathsf{C}$ ). An important fact to keep in mind is that the above analysis is possible since the observations used to compute estimates are never repeated in Alg. 2 and we are able to avoid complex dependencies.
Now, we are ready to prove our main regret bound. Suppose we condition on the event  $\mathcal{E}$  as defined in Lemma 17. Conditioned on the event  $\mathcal{E}$ , let us denote by  $\rho_{u}$  to be some sequence of items recommended to the user  $u\in [\mathsf{M}]$  by our algorithm. The probability of this sequence of items being recommended is  $\operatorname*{Pr}(\cap_{u\in [\mathsf{M}]} \rho_u\mid \mathcal{E})$ .
D.2.1 Swapping argument
Let us fix a particular user  $u \in [\mathsf{M}]$  and a sequence of recommended items  $\rho_{u}$  such that  $\rho_{u}(t) \in [\mathsf{N}]$  is the item recommended to user  $u$  at round  $t$ . For sake of analysis, we will construct a permutation  $\theta_{u} : [\mathsf{T}] \to \{\rho_{u}(t)\}_{t \in [\mathsf{T}]}$  of the items  $\{\rho_{u}(t)\}_{t \in [\mathsf{T}]}$  with sequential modifications  $(\theta_{u}$  is initialized with  $\rho_{u}$ ). For any phase indexed by  $\ell$ , consider the exploit and explore components for a nice subset of users  $\mathcal{M}^{(\ell, i)}$ . The sequence of items recommended to user  $u$  during the explore component remains unchanged i.e. for any phase  $\ell$ ,  $\theta_{u}(t) = \rho_{u}(t)$  for all rounds  $t \in [t]$  such that  $t$  corresponds to the explore component of phase  $\ell$  for the user  $u$ .
Now, for the exploit component in phase  $\ell$ , recall that  $\mathcal{O}_{\mathcal{M}(\ell,i)}^{(\ell)} \setminus \mathcal{O}_{\mathcal{M}(\ell,i)}^{(\ell-1)}$  is the set of items chosen for recommendation particularly in the exploit component of phase  $\ell$  (we remove the super-script  $t$  since we refer to the end of the exploit components in phase  $\ell$  and phase  $\ell-1$  respectively). Suppose at round  $t$  in the explore component of phase  $\ell$ , the  $b^{\text{th}}$  item in the set  $\mathcal{O}_{\mathcal{M}(\ell,i)}^{(\ell)} \setminus \mathcal{O}_{\mathcal{M}(\ell,i)}^{(\ell-1)}$  was chosen
to be recommended to all users in  $\mathcal{M}^{(\ell, i)}$  but was found to be blocked for user  $u \in \mathcal{M}^{(\ell, i)}$  before it could be recommended B times. Let us also denote  $\mathcal{N}^{(\ell, t_a, i)}$  to be a set of active items for users in  $\mathcal{M}^{(\ell, i)}$  at round  $t$  (i.e.  $t_a$  was the previous decision round where it was decided whether possibility A or possibility B was true). Since the  $b^{\text{th}}$  item in the set  $\mathcal{O}_{\mathcal{M}^{(\ell, i)}}^{(\ell)} \setminus \mathcal{O}_{\mathcal{M}^{(\ell, i)}}^{(\ell-1)}$  was blocked for user  $u$ , instead we recommend any unblocked item for user  $u$  from the current set of active items  $\mathcal{N}^{(\ell, t_a, i)}$ . This is always possible; we showed in Lemma 15 (see eq. 23) that the active set of items always contain sufficient unblocked items for possible recommendations for remaining rounds for any user in the corresponding nice subset of users during the exploit component. Now there are two possibilities:
1.  $(b^{\mathrm{th}}$  item in the set  $\mathcal{O}_{\mathcal{M}^{(\ell, i)}}^{(\ell)} \setminus \mathcal{O}_{\mathcal{M}^{(\ell, i)}}^{(\ell-1)}$  was recommended in round  $t'$  in the explore component of previous phase  $\ell'$ ). We consider the active set of items  $\mathcal{N}^{(\ell', h)}$  (such that  $u \in \mathcal{M}^{(\ell', h)}$  in phase  $\ell'$ ) where  $\ell' < \ell$  is the phase index (and  $t'$  is the round index) when the  $b^{\mathrm{th}}$  item in  $\mathcal{O}_{\mathcal{M}^{(\ell, i)}}^{(\ell)} \setminus \mathcal{O}_{\mathcal{M}^{(\ell, i)}}^{(\ell-1)}$  (say  $a$ ) was recommended to the user  $u$  during the explore component. Let us denote the item that we have recommended as replacement to user  $u$  at round  $t$  by  $a'$ . Note that since  $a' \in \mathcal{N}^{(\ell, t_a, i)}$ , it must happen that  $a' \in \mathcal{N}^{(\ell', h)}$  since  $\mathcal{N}^{(\ell, t_a, i)} \subseteq \mathcal{N}^{(\ell', h)}$ . In that case, we have
$$
\rho_ {u} (t) = a ^ {\prime} \text {a n d} \rho_ {u} \left(t ^ {\prime}\right) = a
$$
We swap the above items so that in the modified sequence, we have
$$
\theta_ {u} (t) = a \text {a n d} \theta_ {u} \left(t ^ {\prime}\right) = a ^ {\prime}
$$
The goal of the swapping operation at round  $t$  in the exploit component is to modify the sequence of items such that 1) the item  $a$  chosen for recommendation at round  $t$  is assigned to round  $t$  in the exploit component 2) the item  $a$  actually recommended in round  $t' < t$  and phase  $\ell' < \ell$  is replaced by another item  $a'$  (actually recommended at round  $t > t'$  in phase  $\ell > \ell'$ ) such that both  $a, a'$  belongs to the same set of active items  $\mathcal{N}^{(\ell', h)}$  ( $u \in \mathcal{M}^{(\ell', h)}$  for some  $h$ ). We will also say that the item  $a$  chosen for recommendation is replaced by a swapping operation of length 1. This is because the chosen element for recommendation  $a$  was recommended in the explore component of a previous phase. A more precise definition of the length of a swapping operation is provided below.
2.  $(b^{\mathrm{th}}$  item in the set  $\mathcal{O}_{\mathcal{M}(\ell ,i)}^{(\ell)}\setminus \mathcal{O}_{\mathcal{M}(\ell ,i)}^{(\ell -1)}$  was recommended in the exploit component of previous phase): This is the more difficult case. As before, let us denote the  $b^{\mathrm{th}}$  item in the set  $\mathcal{O}_{\mathcal{M}(\ell ,i)}^{(\ell)}\setminus \mathcal{O}_{\mathcal{M}(\ell ,i)}^{(\ell -1)}$  by  $a$ . Of course, the item  $a$  could not have been chosen for recommendation in the exploit component of a previous phase (i.e.  $a$  cannot belong in the set  $\mathcal{O}_{\mathcal{M}(\ell ',j)}^{(\ell ')}\setminus \mathcal{O}_{\mathcal{M}(\ell ',j)}^{(\ell ' - 1)}$  for any  $\ell^{\prime} < \ell$  with  $u\in \mathcal{M}^{(\ell ',j)}$  in phase  $\ell '$ .) In that case, the item  $a$  was recommended in the exploit component of phase  $\ell$  as part of a swapping operation. With this intuition in mind, let us define length of a swapping operation precisely:
Definition 3 (Length of swapping operation). For a user  $u$ , suppose  $a_1 \in \mathcal{O}_{\mathcal{M}(\ell, i)}^{(\ell)} \setminus \mathcal{O}_{\mathcal{M}(\ell, i)}^{(\ell-1)}$  is an item chosen for recommendation in the exploit component of phase  $\ell$  but found to be blocked. In that case, we will say that  $a_1$  is replaced by a swapping operation of length  $\mathsf{L} + 1$  if there exists  $\mathsf{L} + 1$  items  $a_1, a_2, \ldots, a_{\mathsf{L}}, a_{\mathsf{L}+1}, \mathsf{L} + 1$  phases  $p_1 > p_2 > \dots > p_{\mathsf{L}} > p_{\mathsf{L}+1}$  and respective rounds  $t_1, t_2, \ldots, t_{\mathsf{L}}, t_{\mathsf{L}+1}$  such that  $I$ )  $a_1$  is chosen for recommendation in exploit component of phase  $p_1$  at round  $t_1$  2) for each  $2 \leq i < \mathsf{L}$ ,  $a_{i-1}$  has been recommended in the exploit component of phase  $p_i$  at round  $t_i$  when the intended item chosen for recommendation was  $a_i$  3)  $a_{\mathsf{L}}$  has been recommended in the explore component of phase  $p_{\mathsf{L}+1}$  at round  $t_{\mathsf{L}+1}$  3)  $a_{\mathsf{L}+1}$  is the item recommended to user  $u$  in place of  $a_1$  in phase  $p_1$  at round  $t_1$ .
As before,  $a_{\mathsf{L} + 1}$  belongs to current set of active items  $\mathcal{N}^{(\ell ,t_a,i)}$  at round  $t_1$  for users in  $\mathcal{M}^{(\ell ,i)}$  (recall that  $t_a$  is the decision round just prior to  $t_1$ ). In the above definition, note that  $\mathsf{L}$  must be finite since there are a finite number of components and the first phase only has an explore component (recall that exploit component in the first phase has zero rounds).
For a user  $u \in \mathcal{M}^{(\ell, i)}$ , suppose  $a_1 \in \mathcal{O}_{\mathcal{M}^{(\ell, i)}}^{(\ell)} \setminus \mathcal{O}_{\mathcal{M}^{(\ell, i)}}^{(\ell-1)}$  is an item chosen for recommendation in the exploit component of phase  $\ell$  but found to be blocked. Moreover suppose  $a_1$  must be replaced by a swapping operation of length  $L$  (see Definition 3 for notations). In that case, we make the following modifications to the permutation  $\theta_u$ :
$$
\theta_ {u} (t _ {i}) = a _ {i} \text {f o r a l l} 1 \leq i \leq \mathrm {L} + 1
$$
Lemma 18. Condition on the event  $\mathcal{E}$ . Consider the modified sequence of distinct items  $\{\theta_u(t)\}_{t\in [T]}$  for a certain fixed user  $u$ . For any phase  $\ell$  to which the user  $u$  has progressed as part of the nice subset  $\mathcal{M}^{(\ell ,i)}$ , consider the exploit component starting from round index  $t_{\text{exploit,start,}\ell}$  to  $t_{\text{exploit,end,}\ell}$ . In that case, the set of elements  $\{\theta_u(t)\mid t\in [t_{\text{exploit,start,}\ell},t_{\text{exploit,end,}\ell}]\}$  is equivalent to the set of elements  $\mathcal{O}_{\mathcal{M}^{(\ell ,i)}}^{\left(\ell\right)}\setminus \mathcal{O}_{\mathcal{M}^{(\ell ,i)}}^{\left(\ell -1\right)}$  repeated B times i.e. the golden items chosen for recommendation in the exploit component of phase  $\ell$  to users in  $\mathcal{M}^{(\ell ,i)}$ . Similarly, consider the exploit component starting from round index  $t_{\text{exploit,start,}\ell}$  to  $t_{\text{exploit,end,}\ell}$ . In that case, for any  $t\in [t_{\text{exploit,start,}\ell},t_{\text{exploit,end,}\ell}]$ , it must happen that  $\theta_u(t)\in \mathcal{N}^{(\ell ,i)}$  i.e. the set of active items for  $\mathcal{M}^{(\ell ,i)}$  during the explore component of phase  $\ell$ .
Proof. We start with the following claim.
Claim 1. For a user  $u \in \mathcal{M}^{(\ell, i)}$ , suppose  $a_1 \in \mathcal{O}_{\mathcal{M}^{(\ell, i)}}^{(\ell)} \setminus \mathcal{O}_{\mathcal{M}^{(\ell, i)}}^{(\ell-1)}$  is an item chosen for recommendation in the exploit component of phase  $\ell$  at round  $t_1$  but found to be blocked for user  $u$ . Moreover, suppose  $a_1$  has been replaced by a swapping operation of length  $\mathsf{L} + 1$  (see Definition 3 for notations). In that case, we already have  $\theta_u(t_i) = a_i$  for all  $2 \leq i \leq \mathsf{L}$ . We only need to modify the sequence by making the following two changes: 1)  $\theta_u(t_1) = a_1$  2)  $\theta_u(t_{\mathsf{L} + 1}) = a_{\mathsf{L} + 1}$ . Note that in the true sequence,  $a_{\mathsf{L} + 1}$  has been recommended in the round  $t_1$  replacing the intended item  $a_1$ .