SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left(\mathsf {T} - t _ {\ell}\right) \max  _ {y \in \mathcal {N} ^ {(\ell , i)}} \left(\mathbf {P} _ {u \pi_ {u} (1) | \mathcal {N} ^ {(\ell , i)}} - \mathbf {P} _ {u y}\right) = \widetilde {O} \Big (\sigma \mathsf {C} \sqrt {\mu^ {3} \log d _ {1}} \max  \left(\sqrt {\mathsf {T}}, \mathsf {T} \sqrt {\frac {\mathsf {C}}{\mathsf {M} \tau}}\right) \Big)
$$
where  $t_\ell$  is the round at which the explore component of phase  $\ell$  starts.
Proof. Note that by our induction hypothesis, we must have  $\mathsf{B}\left|\mathcal{N}^{(\ell ,i)}\right|\geq \mathsf{T} - t_{\ell}$  because the set of surviving items must contain sufficient unblocked items (for recommendation in the remaining rounds) for every user  $u\in \mathcal{M}^{(\ell ,i)}$ . Hence, by setting  $p = 1$  in eq. 3, with a certain number of rounds, we can obtain an estimate  $\mathbf{Q}$  of  $\mathbf{P}$  satisfying
$$
\left| \left| \mathbf {Q} _ {\mathcal {M} ^ {(\ell , i)}, \mathcal {N} ^ {(\ell , i)}} - \mathbf {P} _ {\mathcal {M} ^ {(\ell , i)}, \mathcal {N} ^ {(\ell , i)}} \right| \right| _ {\infty} = O \Big (\frac {\sigma \sqrt {\mu^ {3} \log d _ {1}}}{\sqrt {d _ {2}}} \Big).
$$
Hence, this implies that our choice of  $\Delta_{\ell +1}$  satisfies  $\Delta_{\ell +1} = O\left(\frac{\sigma\sqrt{\mu^3\log d_1}}{\sqrt{d_2}}\right)$  implying that  $\epsilon_{\ell} = O\left(\frac{\sigma C\sqrt{\mu^3\log d_1}}{\sqrt{d_2}}\right)$ . Now, there are two possibilities: 1) either  $d_2 = |\mathcal{M}^{(\ell ,i)}|$  implying that  $\mathsf{M} / (\tau \mathsf{C})\leq$
$\left|\mathcal{M}^{(\ell ,i)}\right|\leq \left|\mathcal{N}^{(\ell ,i)}\right|$ . In that case, we have that  $\epsilon_{\ell} = O\Big(\frac{\sigma\mathsf{C}^{1.5}\sqrt{\mu^3\log d_1}}{\sqrt{\mathsf{M}\tau}}\Big)$ . Now, because of our induction assumption, we will have
$$
\max  _ {y \in \mathcal {N} ^ {(\ell , i)}} \left(\mathbf {P} _ {u \pi_ {u} (1) | \mathcal {N} ^ {(\ell , i)}} - \mathbf {P} _ {u y}\right) \leq \epsilon_ {\ell} = O \Big (\frac {\sigma \mathsf {C} ^ {1 . 5} \sqrt {\mu^ {3} \log d _ {1}}}{\sqrt {\mathsf {M} \tau}} \Big).
$$
Therefore, for any user  $u \in \mathcal{M}^{(\ell, i)}$ , if we recommend arbitrary unblocked items in  $\mathcal{N}^{(\ell, i)}$  for the remaining rounds then we can bound the following quantity
$$
\left(\mathsf {T} - t _ {\ell}\right) \max  _ {y \in \mathcal {N} ^ {(\ell , i)}} \left(\mathbf {P} _ {u \pi_ {u} (1) | \mathcal {N} ^ {(\ell , i)}} - \mathbf {P} _ {u y}\right) = O \Big (\frac {\sigma \mathsf {T C} ^ {1 . 5} \sqrt {\mu^ {3} \log d _ {1}}}{\sqrt {\mathsf {M} \tau}} \Big).
$$
2) The second possibility is the following:  $d_{2} = \left|\mathcal{N}^{(\ell ,i)}\right|\geq \mathsf{B}^{-1}(\mathsf{T} - t_{\ell})$ . In that case, from our induction assumption, we have that  $(\mathsf{B} = \Theta (\log \mathsf{T}))$
$$
\max _ {y \in \mathcal {N} ^ {(\ell , i)}} \left(\mathbf {P} _ {u \pi_ {u} (1) | \mathcal {N} ^ {(\ell , i)}} - \mathbf {P} _ {u y}\right) \leq \epsilon_ {\ell} = \widetilde {O} \Big (\frac {\sigma \mathsf {C} \sqrt {\mu^ {3} \log d _ {1}}}{\sqrt {\mathsf {T} - t _ {\ell}}} \Big).
$$
$$
\begin{array}{l} \left(\mathsf {T} - t _ {\ell}\right) \max  _ {y \in \mathcal {N} ^ {(\ell , i)}} \left(\mathbf {P} _ {u \pi_ {u} (1) | \mathcal {N} ^ {(\ell , i)}} - \mathbf {P} _ {u y}\right) \leq \epsilon_ {\ell} (\mathsf {T} - t _ {\ell}) = \widetilde {O} \Big (\frac {\sigma \mathsf {C} \sqrt {\mu^ {3} \log d _ {1}}}{\sqrt {\mathsf {T} - t _ {\ell}}} \cdot (\mathsf {T} - t _ {\ell}) \Big) \\ = \widetilde {O} \left(\sigma \mathrm {C} \sqrt {\mu^ {3} \mathrm {T} \log d _ {1}}\right). \\ \end{array}
$$
Consider  $\mathcal{M}^{\prime (\ell)}\subseteq \mathcal{M}^{(\ell)}$  to be the family of nice subsets of users which do not fall into the edge case scenarios i.e. 1) there exists  $0\leq p\leq 1$  for which the theoretical bound in RHS in eq. 3 can be smaller than  $\Delta_{\ell +1}$  2) we have  $\left|\mathcal{N}^{(\ell ,i)}\right|\geq T^{1 / 3}$ . More precisely  $\mathcal{M}^{\prime (\ell)}$  corresponds to the set
$$
\left\{\mathcal {M} ^ {(\ell , i)} \in \mathcal {M} ^ {(\ell)} \text {w i t h a c t i v e i t e m s} \mathcal {N} ^ {(\ell , i)} \mid \Delta_ {\ell + 1} = \Omega \left(\frac {\sigma \sqrt {\mu^ {3} \log \max (\left| \mathcal {M} ^ {(\ell , i)} \right| , \left| \mathcal {N} ^ {(\ell , i)} \right|)}}{\sqrt {\min (\left| \mathcal {M} ^ {(\ell , i)} \right| , \left| \mathcal {N} ^ {(\ell , i)} \right|)}}\right) \right.
$$
$$
\left. \text {a n d} \left| \mathcal {N} ^ {(\ell , i)} \right| \geq T ^ {1 / 3} \right\}
$$
Suppose  $\mathsf{M} / (\tau \mathsf{C}) = \Omega (\mathsf{T}^{1 / 3})$ . As mentioned before, the event  $\mathcal{E}_3^{(\ell)}$  is true if the algorithm has successfully computed an estimate  $\widetilde{\mathbf{P}}^{(\ell)}\in \mathbb{R}^{\mathsf{M}\times \mathsf{N}}$  such that for all  $\mathcal{M}^{(\ell ,i)}\in \mathcal{M}'^{(\ell)}$
$$
\left| \left| \widetilde {\mathbf {P}} _ {\mathcal {M} ^ {(\ell , i)}, \mathcal {N} ^ {(\ell , i)}} ^ {(\ell)} - \mathbf {P} _ {\mathcal {M} ^ {(\ell , i)}, \mathcal {N} ^ {(\ell , i)}} \right| \right| _ {\infty} \leq \Delta_ {\ell + 1} \text {f o r a l l} \mathcal {M} ^ {(\ell , i)} \in \mathcal {M} ^ {\prime \ell} \tag {11}
$$
implying that for each of the distinct nice subsets  $\mathcal{M}^{(\ell, i)} \in \mathcal{M}'^{(\ell)}$ , after the explore component of phase  $\ell$ , the algorithm finds a good entry-wise estimate of the sub-matrix  $\mathbf{P}_{\mathcal{M}^{(\ell, i)}, \mathcal{N}^{(\ell, i)}}$ . In the following part of the analysis, we will repeatedly condition on the events  $\mathcal{E}_2^{(\ell)}$  (conditions A-C are satisfied at the beginning of the explore component of phase  $\ell$ ) and the event  $\mathcal{E}_3^{(\ell)}$  (eq. 10 is true for all nice subsets  $\mathcal{M}^{(\ell, i)} \in \mathcal{M}'^{(\ell)}$  in the explore component of phase  $\ell$ ). Note that the event  $\mathcal{E}_2^{(\ell)}$  is true due to the induction hypothesis and conditioned on the event  $\mathcal{E}_2^{(\ell)}$ , the event  $\mathcal{E}_3^{(\ell)}$  is true with probability at least  $1 - O(T^{-3}\mathsf{C})$  (after taking a union bound over  $\mathsf{C}$  clusters).
Fix any  $\mathcal{M}^{(\ell,i)} \subseteq \mathcal{M}'^{(\ell)}$  and condition on the events  $\mathcal{E}_2^{(\ell)}, \mathcal{E}_3^{(\ell)}$ . For each user  $u \in \mathcal{M}^{(\ell,i)}$ , once the algorithm has computed the estimate  $\widetilde{\mathbf{P}}_{\mathcal{M}^{(\ell,i)}, \mathcal{N}^{(\ell,i)}}^{(\ell)}$  in eq. 11, let us denote a set of good items for the user  $u$  by
$$
\mathcal {T} _ {u} ^ {(\ell)} \equiv \left\{j \in \mathcal {N} ^ {(\ell , i)} \mid \widetilde {\mathbf {P}} _ {u j} \geq \widetilde {\mathbf {P}} _ {u \widetilde {\pi} _ {u} \left(T B ^ {- 1} - \left| O _ {\mathcal {M} (\ell , i)} ^ {(\ell , t)} \right|\right) | \mathcal {N} ^ {(\ell , i)} - 2 \Delta_ {\ell + 1} \right\} \tag {12}
$$
$$
\mathcal {R} _ {u} ^ {(\ell)} \equiv \left\{j \in \mathcal {N} ^ {(\ell , i)} \mid \widetilde {\mathbf {P}} _ {u j} \geq \widetilde {\mathbf {P}} _ {u \widetilde {\pi} _ {u} (1) \mid \mathcal {N} ^ {(\ell , i)}} - 2 \Delta_ {\ell + 1} \right\} \tag {13}
$$
where  $t$  is the round index at the end of the explore component of phase  $\ell$  and  $|\mathcal{O}_{\mathcal{M}^{(\ell,i)}}^{(\ell,t)}|$  is the number of rounds in exploit components of phases that the user  $u \in \mathcal{M}^{(\ell,i)}$  has encountered so far. Note from our algorithm, that users in the same nice subset  $\mathcal{M}^{(\ell,i)}$  have encountered exactly the same number of rounds in exploit components of phases until the end of phase  $\ell$ . Recall that users in  $\mathcal{M}^{(\ell,i)}$  have been recommended the same set of items until blocked (unless blocked already for some user in which case the item in question has already been recommended B times) in the exploit components of phases until the end of phase  $\ell$  (this set of items chosen for recommendation in the exploit components up to end of phase  $\ell$  is denoted by  $\mathcal{O}_{\mathcal{M}^{(\ell,i)}}^{(\ell,t)}$ ). If we condition on the event  $\mathcal{E}_3^{(\ell)}$ , then we can show the following statement to be true (in the following lemma, we remove the superscript  $t$  in  $\mathcal{O}_{\mathcal{M}^{(\ell,i)}}^{(\ell,t)}$  for simplicity - the round index  $t$  corresponds to the end of phase  $\ell$  for users in  $\mathcal{M}^{(\ell,i)})$ :
Lemma 6. Condition on the events  $\mathcal{E}_2^{(\ell)},\mathcal{E}_3^{(\ell)}$  being true. Consider a nice subset of users  $\mathcal{M}^{(\ell ,i)}\in$ $\mathcal{M}'(\ell)$  and their corresponding set of active items  $\mathcal{N}^{(\ell ,i)}$  for which guarantees in eq. 11 holds. Let  $\mathcal{O}_{\mathcal{M}^{(\ell ,i)}}^{(\ell)}$  be the set of items that have been chosen for recommendation to users in  $\mathcal{M}^{(\ell ,i)}$  in exploit components of the first  $\ell$  phases. Denote  $\mathcal{Z} = [\mathsf{N}]\setminus \mathcal{O}_{\mathcal{M}^{(\ell ,i)}}^{(\ell)}$ . In that case, for every user  $u\in \mathcal{M}^{(\ell ,i)}$ , the items  $\pi_u(s)\mid \mathcal{Z}$  for all  $s\in [\mathsf{T}\mathsf{B}^{-1} - \left|\mathcal{O}_{\mathcal{M}^{(\ell ,i)}}^{(\ell)}\right|]$  must belong to the set  $\mathcal{T}_u^{(\ell)}$ .
Proof. Let us fix a user  $u \in \mathcal{M}^{(\ell, i)}$  with active set of arms  $\mathcal{N}^{(\ell, i)}$ . Let us also fix an item  $a \equiv \pi_u(t) \mid \mathcal{Z}$  for  $t \in [\mathrm{T}\mathsf{B}^{-1} - \left|\mathcal{O}_{\mathcal{M}^{(\ell, i)}}^{(\ell)}\right|]$ . Recall that  $\widetilde{\pi} \mid \mathcal{Z}$  is the permutation of the items sorted in descending order according to their estimated reward in  $\widetilde{\mathbf{P}}_{\mathcal{M}^{(\ell, i)}, \mathcal{N}^{(\ell, i)}}^{(\ell)}$ . Now there are two possibilities regarding the position of the arm  $a$  in the permutation  $\widetilde{\pi} \mid \mathcal{Z} - 1$ :  $\pi_u(t) \mid \mathcal{Z} \equiv \widetilde{\pi}_u(t_1) \mid \mathcal{Z}$  for some  $t_1 \leq t$  which implies that in the permutation  $\widetilde{\pi} \mid \mathcal{Z}$ , the position of the arm  $a$  is  $t_1$ . In that case, the arm  $a$  survives in the set  $\mathcal{T}_u^{(\ell)}$  by definition (see eq. 12) 2). Now, suppose that  $\pi_u(t) \mid \mathcal{Z} \equiv \widetilde{\pi}_u(t_1) \mid \mathcal{Z}$  for some  $t_1 \leq t$  which implies that in the permutation  $\widetilde{\pi} \mid \mathcal{Z}$ , the arm  $a$  has been shifted to the right (from  $\pi \mid \mathcal{Z}$ ). In that case, there must exist another item  $b \equiv \pi_u(t_2) \mid \mathcal{Z}$  for  $t_2 > t$  such that  $\pi_u(t_2) \mid \mathcal{Z} \equiv \widetilde{\pi}_u(t_3) \mid \mathcal{Z}$  for  $t_3 \leq t$ . Now, we will have
$$
\widetilde {\mathbf {P}} _ {u b} ^ {(\ell)} - \widetilde {\mathbf {P}} _ {u a} ^ {(\ell)} = \widetilde {\mathbf {P}} _ {u b} ^ {(\ell)} - \mathbf {P} _ {u b} + \mathbf {P} _ {u b} - \mathbf {P} _ {u a} + \mathbf {P} _ {u a} - \widetilde {\mathbf {P}} _ {u a} ^ {(\ell)} \leq 2 \Delta_ {\ell + 1}
$$
where we used the following facts a) conditioned on the event  $\mathcal{E}_3^{(\ell)}$ , we have  $\widetilde{\mathbf{P}}_{us}^{(\ell)} - \mathbf{P}_{us} \leq \Delta_{\ell + 1}$  for all  $u \in \mathcal{M}^{(\ell, i)}$ ,  $s \in \mathcal{N}^{(\ell, i)}$  b)  $\mathbf{P}_{ub} - \mathbf{P}_{ua} \leq 0$  by definition. Hence, this implies that  $a \in \mathcal{T}_u^{(\ell)}$ .
Corollary 1. Condition on the events  $\mathcal{E}_2^{(\ell)},\mathcal{E}_3^{(\ell)}$  being true. Consider a nice subset of users  $\mathcal{M}^{(\ell ,i)}\in$ $\mathcal{M}'(\ell)$  and their corresponding set of active items  $\mathcal{N}^{(\ell ,i)}$  for which guarantees in eq. 11 holds. Suppose  $t_\ell$  is the final round of the explore component of phase  $\ell$  for users in  $\mathcal{M}^{(\ell ,i)}$ . In that case, for user  $u$ , the set  $\mathcal{T}_u^{(\ell)}$  contains sufficient items that are unblocked and can be recommended for the remaining  $\top -t_{\ell}$  rounds.
Proof. From Lemma 11, we showed that  $\mathcal{T}_u^{(\ell)}$  comprises the set of items  $\pi_u(s) \mid \mathcal{Z}$  for all  $s \in [\mathsf{T}\mathsf{B}^{-1} - \left|\mathcal{O}_{\mathcal{M}^{(\ell,i)}}^{(\ell)}\right|]$  where  $\mathcal{Z} = [\mathsf{N}] \setminus \mathcal{O}_{\mathcal{M}^{(\ell,i)}}^{(\ell)}$ . Now, suppose the  $k^{\text{th}}$  item in the set  $\mathcal{H}_u \equiv \{\pi_u(t') \mid \mathcal{Z}\}_{t'=1}^{\mathsf{T}-\left|\mathcal{O}_{\mathcal{M}^{(\ell,i)}}^{(\ell)}\right|}$  has been recommended to user  $u b_k < \mathsf{B}$  times in previous phases. In that case, the number of allowed recommendations of items in  $\mathcal{T}_u^{(\ell)}$  is at least  $\mathsf{T}\mathsf{B}^{-1} - \mathsf{B}\left|\mathcal{O}_{\mathcal{M}^{(\ell,i)}}^{(\ell)}\right| - \sum_{k \in \mathcal{H}_u} b_k$  which is more than the remaining rounds that is at most  $\mathsf{T}\mathsf{B}^{-1} - \mathsf{B}\left|\mathcal{O}_{\mathcal{M}^{(\ell,i)}}^{(\ell)}\right| - \sum_{k \in \mathcal{H}_u} b_k (\mathsf{B}\left|\mathcal{O}_{\mathcal{M}^{(\ell,i)}}^{(\ell)}\right|)$  rounds have already been used up when the golden items were identified and recommended).
Consider a nice subset of users  $\mathcal{M}^{(\ell, i)} \in \mathcal{M}'^{(\ell)}$  and its corresponding active set of items  $\mathcal{N}^{(\ell, i)}$  for which eq. 11 holds true. At the end of the explore component of phase  $\ell$ , based on the estimate  $\widetilde{\mathbf{P}}_{\mathcal{M}^{(\ell, i)}, \mathcal{N}^{(\ell, i)}}^{(\ell)}$ , we construct a graph  $\mathcal{G}^{(\ell, i)}$  whose nodes are given by the users in  $\mathcal{M}^{(\ell, i)}$ . Now, we
draw an edge between two users  $u, v \in \mathcal{M}^{(\ell, i)}$  if  $\left| \widetilde{\mathbf{P}}_{ux}^{(\ell)} - \widetilde{\mathbf{P}}_{vx}^{(\ell)} \right| \leq 2\Delta_{\ell+1}$  for all considered items  $x \in \mathcal{N}^{(\ell, i)}$ .
Lemma 7. Condition on the events  $\mathcal{E}_2^{(\ell)},\mathcal{E}_3^{(\ell)}$  being true. Consider a nice subset of users  $\mathcal{M}^{(\ell ,i)}\in$ $\mathcal{M}'(\ell)$  and their corresponding set of active items  $\mathcal{N}^{(\ell ,i)}$  for which guarantees in eq. 11 holds. Consider the graph  $\mathcal{G}^{(\ell ,i)}$  formed by the users in  $\mathcal{M}^{(\ell ,i)}$  such that an edge exists between two users  $u,v\in \mathcal{M}^{(\ell ,i)}$  if  $\left|\widetilde{\mathbf{P}}_{ux}^{(\ell)} - \widetilde{\mathbf{P}}_{vx}^{(\ell)}\right|\leq 2\Delta_{\ell +1}$  for all considered items  $x\in \mathcal{N}^{(\ell ,i)}$  . Nodes in  $\mathcal{G}^{(\ell ,i)}$  corresponding to users in the same cluster form a clique. Also, users in each connected component of the graph  $\mathcal{G}^{(\ell ,i)}$  form a nice subset of users.
Proof. For any two users  $u, v \in \mathcal{M}^{(\ell,i)}$  belonging to the same cluster, consider an arm  $x \in \mathcal{N}^{(\ell,i)}$ . We must have
$$
\widetilde {\mathbf {P}} _ {u x} ^ {(\ell)} - \widetilde {\mathbf {P}} _ {v x} ^ {(\ell)} = \widetilde {\mathbf {P}} _ {u x} ^ {(\ell)} - \mathbf {P} _ {u x} + \mathbf {P} _ {u x} - \mathbf {P} _ {v x} + \mathbf {P} _ {v x} - \widetilde {\mathbf {P}} _ {v x} ^ {(\ell)} \leq 2 \Delta_ {\ell + 1}.
$$
Now, consider two users  $u, v \in \mathcal{M}^{(\ell, i)}$  that belongs to different clusters  $\mathcal{P}, \mathcal{Q}$  respectively. Note that since the event  $\mathcal{E}^{(\ell)}$  is true,  $\mathcal{M}^{(\ell, i)}$  is a union of clusters comprising  $\mathcal{P}, \mathcal{Q}$ . Furthermore, we have already established that nodes in  $\mathcal{G}^{(\ell, i)}$  (users in  $\mathcal{M}^{(\ell, i)}$ ) restricted to the same cluster form a clique. There every connected component of the graph  $\mathcal{G}^{(\ell, i)}$  can be represented as a union of a subset of clusters.
Lemma 8. Condition on the events  $\mathcal{E}_2^{(\ell)},\mathcal{E}_3^{(\ell)}$  being true. Consider a nice subset of users  $\mathcal{M}^{(\ell ,i)}\in$ $\mathcal{M}'(\ell)$  and their corresponding set of active items  $\mathcal{N}^{(\ell ,i)}$  for which guarantees in eq. 11 holds. In that case, for any subset  $\mathcal{V}\subseteq \mathcal{N}^{(\ell ,i)}$  and any  $s\in [|\mathcal{V}|]$ , for every user  $u\in \mathcal{M}^{(\ell ,i)}$ , we must have
$$
\widetilde {\mathbf {P}} _ {u \widetilde {\pi} _ {u} (1) | \mathcal {Y}} - \widetilde {\mathbf {P}} _ {u \widetilde {\pi} _ {u} (s) | \mathcal {Y}} - 6 \Delta_ {\ell + 1} \leq \mathbf {P} _ {u \pi_ {u} (1) | \mathcal {Y}} - \mathbf {P} _ {u \pi_ {u} (s) | \mathcal {Y}} \leq \widetilde {\mathbf {P}} _ {u \widetilde {\pi} _ {u} (1) | \mathcal {Y}} - \widetilde {\mathbf {P}} _ {u \widetilde {\pi} _ {u} (s) | \mathcal {Y}} + 6 \Delta_ {\ell + 1}.
$$
Proof. Since, we condition on the event  $\mathcal{E}_3^{(\ell)}$ , we must have computed an estimate  $\widetilde{\mathbf{P}}^{(\ell)}$ , an estimate of  $\mathbf{P}$  restricted to users in  $\mathcal{M}^{(\ell,i)}$  and items in  $\mathcal{N}^{(\ell,i)}$  such that
$$
\left| \left| \widetilde {\mathbf {P}} _ {\mathcal {M} ^ {(\ell , i)}, \mathcal {N} ^ {(\ell , i)}} ^ {(\ell)} - \mathbf {P} _ {\mathcal {M} ^ {(\ell , i)}, \mathcal {N} ^ {(\ell , i)}} \right| \right| _ {\infty} \leq \Delta_ {\ell + 1}.
$$
We can decompose the term being studied in the following manner:
$$
\begin{array}{l} \widetilde {\mathbf {P}} _ {u \pi_ {u} (1) | \mathcal {Y}} ^ {(\ell)} - \widetilde {\mathbf {P}} _ {u \pi_ {u} (s) | \mathcal {Y}} ^ {(\ell)} \\ = \widetilde {\mathbf {P}} _ {u \pi_ {u} (1) | \mathcal {Y}} ^ {(\ell)} - \widetilde {\mathbf {P}} _ {u \widetilde {\pi} _ {u} (1) | \mathcal {Y}} ^ {(\ell)} + \widetilde {\mathbf {P}} _ {u \widetilde {\pi} _ {u} (1) | \mathcal {Y}} ^ {(\ell)} - \widetilde {\mathbf {P}} _ {u \widetilde {\pi} _ {u} (s) | \mathcal {Y}} ^ {(\ell)} + \widetilde {\mathbf {P}} _ {u \widetilde {\pi} _ {u} (s) | \mathcal {Y}} ^ {(\ell)} - \widetilde {\mathbf {P}} _ {u \pi_ {u} (s) | \mathcal {Y}} ^ {(\ell)}. \\ \end{array}
$$
Let us bound the quantity  $\widetilde{\mathbf{P}}_{u\widetilde{\pi}_u(s)|\mathcal{Y}}^{(\ell)} - \widetilde{\mathbf{P}}_{u\pi_u(s)|\mathcal{Y}}^{(\ell)}$ . In order to analyze this quantity, we will consider a few cases. In the first case, suppose that  $\pi_u(s) \mid \mathcal{Y} = \widetilde{\pi}_u(t_1) \mid \mathcal{Y}$  for  $t_1 \geq s$ . In that case, we will have that  $\widetilde{\mathbf{P}}_{u\widetilde{\pi}_u(s)|\mathcal{Y}}^{(\ell)} - \widetilde{\mathbf{P}}_{u\pi_u(s)|\mathcal{Y}}^{(\ell)} \geq 0$ . In the second case, suppose  $\pi_u(s) \mid \mathcal{Y} = \widetilde{\pi}_u(t_2) \mid \mathcal{Y}$  for  $t_2 < s$  and  $\pi_u(t_3) \mid \mathcal{Y} = \widetilde{\pi}_u(s) \mid \mathcal{Y}$  for  $t_3 < s$ . In that case, we have
$$
\begin{array}{l} \widetilde {\mathbf {P}} _ {u \widetilde {\pi} _ {u} (s) | \mathcal {Y}} ^ {(\ell)} - \widetilde {\mathbf {P}} _ {u \pi_ {u} (s) | \mathcal {Y}} ^ {(\ell)} \\ = \widetilde {\mathbf {P}} _ {u \widetilde {\pi} _ {u} (s) | \mathcal {Y}} ^ {(\ell)} - \mathbf {P} _ {u \widetilde {\pi} _ {u} (s) | \mathcal {Y}} + \mathbf {P} _ {u \widetilde {\pi} _ {u} (s) | \mathcal {Y}} - \mathbf {P} _ {u \pi_ {u} (s) | \mathcal {Y}} + \mathbf {P} _ {u \pi_ {u} (s) | \mathcal {Y}} - \widetilde {\mathbf {P}} _ {u \pi_ {u} (s) | \mathcal {Y}} ^ {(\ell)} \\ = \widetilde {\mathbf {P}} _ {u \widetilde {\pi} _ {u} (s) | \mathcal {Y}} ^ {(\ell)} - \mathbf {P} _ {u \widetilde {\pi} _ {u} (s) | \mathcal {Y}} + \mathbf {P} _ {u \pi_ {u} (t _ {3}) | \mathcal {Y}} - \mathbf {P} _ {u \pi_ {u} (s) | \mathcal {Y}} + \mathbf {P} _ {u \pi_ {u} (s) | \mathcal {Y}} - \widetilde {\mathbf {P}} _ {u \pi_ {u} (s) | \mathcal {Y}} ^ {(\ell)} \geq - 2 \Delta_ {\ell + 1} \\ \end{array}
$$
where we used the fact  $\mathbf{P}_{u\pi_u(t_3)}|y - \mathbf{P}_{u\pi_u(s)}|y\geq 0$ . In the final case, we assume that  $\pi_u(s)\mid \mathcal{V} = \widetilde{\pi}_u(t_2)\mid \mathcal{V}$  for  $s > t_2$  and  $\pi_u(t_3)\mid \mathcal{V} = \widetilde{\pi}_u(s)\mid \mathcal{V}$  for  $t_3 > s$ . This means that both the items  $\pi_u(s)\mid \mathcal{V},\pi_u(t_3)\mid \mathcal{V}$  have been shifted to the left in the permutation  $\widetilde{\pi}_u\mid \mathcal{V}$ . Hence,
$$
\begin{array}{l} \widetilde {\mathbf {P}} _ {u \widetilde {\pi} _ {u} (s) | \mathcal {Y}} ^ {(\ell)} - \widetilde {\mathbf {P}} _ {u \pi_ {u} (s) | \mathcal {Y}} ^ {(\ell)} \\ = \widetilde {\mathbf {P}} _ {u \widetilde {\pi} _ {u} (s) | \mathcal {Y}} ^ {(\ell)} - \mathbf {P} _ {u \widetilde {\pi} _ {u} (s) | \mathcal {Y}} + \mathbf {P} _ {u \widetilde {\pi} _ {u} (s) | \mathcal {Y}} - \mathbf {P} _ {u \pi_ {u} (s) | \mathcal {Y}} \\ + \mathbf {P} _ {u \pi_ {u} (s) | \mathcal {Y}} - \widetilde {\mathbf {P}} _ {u \pi_ {u} (s) | \mathcal {Y}} ^ {(\ell)} \geq - 2 \Delta_ {\ell + 1} + \mathbf {P} _ {u \widetilde {\pi} _ {u} (s) | \mathcal {Y}} - \mathbf {P} _ {u \pi_ {u} (s) | \mathcal {Y}} = - 2 \Delta_ {\ell + 1} + \mathbf {P} _ {u \pi_ {u} (t _ {3}) | \mathcal {Y}} - \mathbf {P} _ {u \pi_ {u} (s) | \mathcal {Y}}. \\ \end{array}
$$
Hence there must exist an element  $\pi_u(t_4) \mid \mathcal{V}$  such that  $t_4 < s$  and  $\widetilde{\pi}_u(t_5) \mid \mathcal{V} = \pi_u(t_4) \mid \mathcal{V}$  for  $t_5 > s$ . In that case, we must have
$$
\begin{array}{l} \mathbf {P} _ {u \pi_ {u} (t _ {3}) | \mathcal {Y}} - \mathbf {P} _ {u \pi_ {u} (s) | \mathcal {Y}} \geq \mathbf {P} _ {u \pi_ {u} (t _ {3}) | \mathcal {Y}} - \mathbf {P} _ {u \pi_ {u} (t _ {4}) | \mathcal {Y}} \\ = \mathbf {P} _ {u \pi_ {u} (t _ {3}) | \mathcal {Y}} - \widetilde {\mathbf {P}} _ {u \pi_ {u} (t _ {3}) | \mathcal {Y}} ^ {(\ell)} + \widetilde {\mathbf {P}} _ {u \pi_ {u} (t _ {3}) | \mathcal {Y}} ^ {(\ell)} - \widetilde {\mathbf {P}} _ {u \pi_ {u} (t _ {4}) | \mathcal {Y}} ^ {(\ell)} + \widetilde {\mathbf {P}} _ {u \pi_ {u} (t _ {4}) | \mathcal {Y}} ^ {(\ell)} - \mathbf {P} _ {u \pi_ {u} (t _ {4}) | \mathcal {Y}} \\ \geq - 2 \Delta_ {\ell + 1} + \widetilde {\mathbf {P}} _ {u \pi_ {u} (t _ {3}) | \mathcal {Y}} ^ {(\ell)} - \widetilde {\mathbf {P}} _ {u \pi_ {u} (t _ {4}) | \mathcal {Y}} ^ {(\ell)} = - 2 \Delta_ {\ell + 1} + \widetilde {\mathbf {P}} _ {u \widetilde {\pi} _ {u} (s) | \mathcal {Y}} ^ {(\ell)} - \widetilde {\mathbf {P}} _ {u \widetilde {\pi} _ {u} (t _ {5}) | \mathcal {Y}} ^ {(\ell)} \geq - 2 \Delta_ {\ell + 1}. \\ \end{array}
$$
Therefore, in this case, we get that  $\widetilde{\mathbf{P}}_{u\widetilde{\pi}_u(s)}^{(\ell)} - \widetilde{\mathbf{P}}_{u\pi_u(s)}^{(\ell)}\geq -4\Delta_{\ell +1}$ . Again, we will have that
$$
\begin{array}{l} \widetilde {\mathbf {P}} _ {u \pi_ {u} (1) | \mathcal {Y}} ^ {(\ell)} - \widetilde {\mathbf {P}} _ {u \widetilde {\pi} _ {u} (1) | \mathcal {Y}} ^ {(\ell)} = \widetilde {\mathbf {P}} _ {u \pi_ {u} (1) | \mathcal {Y}} ^ {(\ell)} - \mathbf {P} _ {u \pi_ {u} (1) | \mathcal {Y}} ^ {(\ell)} \\ + \mathbf {P} _ {u \pi_ {u} (1) | \mathcal {Y}} ^ {(\ell)} - \mathbf {P} _ {u \widetilde {\pi} _ {u} (1) | \mathcal {Y}} ^ {(\ell)} + \mathbf {P} _ {u \widetilde {\pi} _ {u} (1) | \mathcal {Y}} ^ {(\ell)} - \widetilde {\mathbf {P}} _ {u \widetilde {\pi} _ {u} (1) | \mathcal {Y}} ^ {(\ell)} \geq - 2 \Delta_ {\ell + 1}. \\ \end{array}
$$
By combining the above arguments, we have that
$$
\widetilde {\mathbf {P}} _ {u \pi_ {u} (1) | \mathcal {Y}} ^ {(\ell)} - \widetilde {\mathbf {P}} _ {u \pi_ {u} (s) | \mathcal {Y}} ^ {(\ell)} \geq \widetilde {\mathbf {P}} _ {u \widetilde {\pi} _ {u} (1) | \mathcal {Y}} ^ {(\ell)} - \widetilde {\mathbf {P}} _ {u \widetilde {\pi} _ {u} (s) | \mathcal {Y}} ^ {(\ell)} - 6 \Delta_ {\ell + 1}.
$$
By a similar set of arguments involving triangle inequalities, we will also have
$$
\widetilde {\mathbf {P}} _ {u \pi_ {u} (1) | \mathcal {Y}} ^ {(\ell)} - \widetilde {\mathbf {P}} _ {u \pi_ {u} (s) | \mathcal {Y}} ^ {(\ell)} \leq \widetilde {\mathbf {P}} _ {u \widetilde {\pi} _ {u} (1) | \mathcal {Y}} ^ {(\ell)} - \widetilde {\mathbf {P}} _ {u \widetilde {\pi} _ {u} (s) | \mathcal {Y}} ^ {(\ell)} + 6 \Delta_ {\ell + 1}.
$$
This completes the proof of the lemma.
We now show the following lemma characterizing the union of good items for a connected component of the graph  $\mathcal{G}^{(\ell, i)}$ . Recall that  $\mathsf{T} - \left|\mathcal{O}_{\mathcal{M}^{(\ell, i)}}^{(\ell)}\right|$  counts the number of rounds excluding the ones used up in exploit component so far up to the  $\ell^{\mathrm{th}}$  phase.