SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \max  _ {j ^ {\prime}} \bar {\mu} _ {h, n} (\mathcal {C}, j ^ {\prime}) - \bar {\mu} _ {h, n} (\mathcal {C}, j) \geq \bar {\mu} _ {h, n} (\mathcal {C}, j _ {b}) - \bar {\mu} _ {h, n} (\mathcal {C}, j) \geq \mu (b, j _ {b}) - \mu (b, j) - \frac {\widetilde {\varepsilon} _ {h , n}}{2} \\ \geq 2. 5 \cdot \widetilde {\varepsilon} _ {h, n} > 2 \widetilde {\varepsilon} _ {h, n} \\ \end{array}
$$
Therefore,  $j$  is eliminated in  $\mathrm{GOOD}_{h,n}(\mathcal{C})$ , and thus eliminated in  $\mathrm{GOOD}_{h+1,n'}(\mathcal{C})$  for  $n' > n$ .
G.1 Proof of Theorem 5
Proof. Fix a phase  $h$  and a level  $n$ . It suffices to bound regret within a single pair  $(h, n)$ . For  $b \in [r]$ , let  $n_b = \lceil \log(1 / \varepsilon_{h,n}^2) \rceil$ . By Lemma 23, if  $n > n_b$ , all  $6\widetilde{\varepsilon}_{h,n_b}$ -suboptimal arms are eliminated from  $\mathrm{GOOD}_{h,n}(\mathcal{C})$  for any cluster  $\mathcal{C} \in \mathcal{P}_h$ . Therefore, regret of playing an action from  $\mathrm{GOOD}_{h,n}(\mathcal{C})$  is  $6\widetilde{\varepsilon}_{h,n_b}$ . In this case, regret is bounded by
$$
\begin{array}{l} \sum_ {b \in [ r ]} r (S + K) 2 ^ {(n + h) / 2} \cdot \omega (b) \cdot 6 \widetilde {\varepsilon} _ {h, n _ {b}} \leq \sum_ {b \in [ r ]} r (S + K) 2 ^ {(n + h) / 2} \cdot \omega (b) \cdot 6 \sqrt {\frac {\widetilde {\lg} _ {h}}{2 ^ {n _ {b}}}} \\ \leq 6 \sqrt {\widetilde {\lg} _ {h}} \sum_ {b \in [ r ]} r (S + K) 2 ^ {(n + h) / 2} \cdot \omega (b) \cdot \varepsilon_ {h, b} \\ \leq 6 \sqrt {\widetilde {\mathrm {l g}} _ {h}} \sum_ {b \in [ r ]} r (S + K) 2 ^ {(n + h) / 2} \cdot \omega (b) \cdot \frac {\varepsilon_ {h}}{r \omega (b)} \\ \leq 6 \sqrt {\widetilde {\lg_ {h}}} r (S + K) 2 ^ {(n + h) / 2} = \widetilde {O} \left(r (S + K) 2 ^ {h / 2}\right) \\ \end{array}
$$
If  $n \leq n_b$ , a similar argument shows that regret of playing an action from  $\mathrm{GOOD}_{h,n}(\mathcal{C})$  is  $6\widetilde{\varepsilon}_{h,n}$ . Therefore, regret is bounded by
$$
\begin{array}{l} \sum_ {b \in [ r ]} r (S + K) 2 ^ {(n + h) / 2} \cdot \omega (b) \cdot 6 \widetilde {\varepsilon} _ {h, n} \leq 6 \sum_ {b \in [ r ]} \omega (b) \cdot r (S + K) 2 ^ {(n + h) / 2} \sqrt {\frac {\widetilde {\lg} h}{2 ^ {n}}} \\ \leq 6 \sqrt {\tilde {\log_ {h}}} r (S + K) 2 ^ {h} = \widetilde {O} \left(r (S + K) 2 ^ {h / 2}\right) \\ \end{array}
$$
We conclude the overall regret is  $\widetilde{O}\left(\sqrt{r(S + K)T}\right)$  for the data collection stage by noting that  $T\geq r(S + K)2^{(h - 1)}$  as phase  $h - 1$  is executed completely. The same argument holds for analyzing the clustering stage when replacing  $r$  with  $r^3$ , so we conclude the regret is bounded by  $\widetilde{O}\left(\sqrt{r^3(S + K)T}\right)$ .
H Omitted Details in Section 5
In this section we discuss how to solve the more general low-rank bandit problem. Note that  $A$  has rank  $r$  if and only if there exist vectors  $w_{1},\ldots ,w_{S}\in \mathbb{R}^{r}$  and  $v_{1},\ldots ,v_{K}\in \mathbb{R}^{r}$  so that  $A(i,j) = w_i^\top v_j$ . We first consider  $r = 1$  and assume that every  $w_{i}$  is non-negative. In this case, we have
$$
\underset {j \in [ K ]} {\arg \max } A (i, j) = \underset {j \in [ K ]} {\arg \max } w _ {i} v _ {j} = \underset {j \in [ K ]} {\arg \max } v _ {j}. \tag {10}
$$
In other words, there exists an arm that is optimal for all contexts. The problem becomes simple as we can run the EXP4 algorithm [Auer et al., 2002] using constant experts that recommend the same arm for all contexts. Thus, we will have only  $K$  experts and have the following proposition:
Proposition 24. Consider the rank-1 bandit problem with  $r = 1$  and  $w_{i} \geq 0$  for every  $i \in [S]$ . The regret of the EXP4 algorithm run with  $K$  constant experts is bounded by  $O(\sqrt{KT\log K})$ .
However, the idea seems hard to generalize when  $w_{i} \in \mathbb{R}$ , as Eq. (10) does not hold anymore and we need exponentially many experts for EXP4. Next we introduce a new idea based on a reduction to context-lumpable bandits.
In the following we define constant  $B = \max_{i} \| w_{i} \|_{\infty}$ . To better illustrate the idea we first assume  $r = 1$  and consider the PAC setting. We create an  $\alpha$ -covering of  $[-B, B]$  and "cluster" each  $i$  into one of the segments. Specifically, we have  $\frac{2B}{\alpha}$  intervals
$$
\left[ - B, - B + \frac {1}{\alpha} \right], \left(- B + \frac {1}{\alpha}, - B + \frac {2}{\alpha} \right], \dots , \left(B - \frac {1}{\alpha}, B \right]
$$
and each  $w_{i}$  is assigned to the interval that contains it. Given  $\alpha$ , let  $\mathcal{R}_{\alpha}$  denote the number of intervals that have at least one context. Conceptually we can view contexts in the same interval as if they are in the same block in context-lumpable bandits, and we have  $\mathcal{R}_{\alpha}$  blocks analogously. For contexts  $i, i'$  in the same interval, they are indeed similar in the sense that we have  $|A(i,j) - A(i,j)| = O(\alpha)$  for every arm  $j$ . Intuitively, if  $\alpha$  is much smaller than  $\varepsilon$ , then Algorithm 1 can proceed normally as a rank- $\mathcal{R}_{\alpha}$  context-lumpable bandit problem. Consequently, we have the following theorem.
Theorem 25. For  $r = 1$ , by choosing  $\alpha = \Theta(\varepsilon)$ , Algorithm 1 uses  $\widetilde{O}\left(\mathcal{R}_{\alpha}(S + K) / \varepsilon^{2}\right)$  samples and outputs an  $\widetilde{O}(\varepsilon)$ -optimal policy.
Clearly we have  $\mathcal{R}_{\alpha} \leq \min \left\{\frac{1}{\alpha}, S\right\}$ , so the above theorem leads to a  $\widetilde{O}\left((S + K) / \varepsilon^3\right)$  sample complexity in the worst case. The idea can be generalized to regret minimization and  $r > 1$ . Specifically, we construct an  $\alpha$ -grid of  $[-B, B]^r$  so that  $\mathcal{R}_{\alpha} = O\left(\frac{1}{\alpha^r}\right)$  and run Algorithm 6 for regret minimization. Consequently, we have the following theorem:
Theorem 26. Let  $p = \frac{1}{3r + 2}$  and choose  $\alpha = (S + K)^p T^{-p}$ . Then regret of Algorithm 6 is bounded as  $Reg_{T} = \widetilde{O}\left((S + K)^{p}T^{1 - p}\right)$ .
Proof. Similar to previous analysis, the regret in a single phase  $h$  is
$$
\widetilde {O} \left(\varepsilon_ {h}\right) \cdot \min  \left\{T, \widetilde {O} \left(\frac {\mathcal {R} _ {\alpha} ^ {3} (S + K)}{\varepsilon_ {h} ^ {2}}\right) \right\} = \min  \left\{\widetilde {O} \left(\varepsilon_ {h} T\right), \widetilde {O} \left(\frac {\mathcal {R} _ {\alpha} ^ {3} (S + K)}{\varepsilon_ {h}}\right) \right\} \tag {11}
$$
Recall that  $\mathcal{R}_{\alpha} = O(1 / \alpha^{r})$ . Therefore, when  $\alpha = \Theta (\varepsilon_h)$ , we have the above regret is bounded by
$$
\widetilde {O} \left(\frac {\mathcal {R} _ {\alpha} ^ {3} (S + K)}{\alpha} + \alpha T\right) = \widetilde {O} \left(\frac {(S + K)}{\alpha^ {3 r + 1}} + \alpha T\right) = \widetilde {O} \left((S + K) ^ {p} T ^ {1 - p}\right)
$$
when choosing  $\alpha$  optimally as  $\alpha = (S + K)^{p}T^{-p}$ . Otherwise,  $\alpha = o(\varepsilon_h)$  and Eq. (11) can be bounded by
$$
\widetilde {O} \left(\frac {\mathcal {R} _ {\alpha} ^ {3} (S + K)}{\varepsilon_ {h}}\right) = \widetilde {O} \left(\frac {\mathcal {R} _ {\alpha} ^ {3} (S + K)}{\alpha}\right) = \widetilde {O} \left((S + K) ^ {p} T ^ {1 - p}\right).
$$
We finish the proof by noting that there are at most  $\log T$  phases.
The bound becomes non-trivial when  $S$  and  $K$  are large. For example, when  $S = K = \sqrt{T}$ , the bound is  $T^{1 - p / 2} = o(T)$  for any  $r$  while the  $\widetilde{O}\left(\sqrt{SKT}\right)$  bound given by EXP4 is vacuous. The factor 3 comes from the  $r^3$  term in the regret bound of Theorem 5. It is a promising direction to first improve the factor in context-lumpable bandits and extend it to low-rank bandits using the reduction introduced here.