SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\forall g \in [ G ], \quad \frac {\mathrm {U C B} _ {t} \left(\sigma_ {g}\right) ^ {\frac {2 p}{p + 1}}}{n _ {g , t}} \leq \frac {\mathrm {U C B} _ {t} \left(\sigma_ {X _ {t + 1}}\right) ^ {\frac {2 p}{p + 1}}}{n _ {X _ {t + 1} , t}}. \tag {10}
$$
On the one hand, from the leftmost inequality in (9),
$$
\forall g \in [ G ], \quad \frac {\sigma_ {g} ^ {\frac {2 p}{p + 1}}}{n _ {g , t}} \leq \frac {\mathbf {U C B} _ {t} \left(\sigma_ {g}\right) ^ {\frac {2 p}{p + 1}}}{n _ {g , t}}. \tag {11}
$$
On the other hand, from the rightmost inequality in (9),
$$
\frac {\mathrm {U C B} _ {t} \left(\sigma_ {X _ {t + 1}}\right) ^ {\frac {2 p}{p + 1}}}{n _ {X _ {t + 1} , t}} \leq \frac {\sigma_ {X _ {t + 1}} ^ {\frac {2 p}{p + 1}} + \Sigma_ {p} w _ {X _ {t + 1}} \left(n _ {X _ {t + 1} , t}\right)}{n _ {X _ {t + 1} , t}}. \tag {12}
$$
Therefore by combining both Inequalities (11) and (12) in Inequality (10),
$$
\forall g \in [ G ], \quad \frac {\sigma_ {g} ^ {\frac {2 p}{p + 1}}}{n _ {g , t}} \leq \frac {\sigma_ {X _ {t + 1}} ^ {\frac {2 p}{p + 1}} + \Sigma_ {p} w _ {X _ {t + 1}} (n _ {X _ {t + 1} , t})}{n _ {X _ {t + 1} , t}},
$$
which implies, after multiplying both sides by  $n_{g,t}n_{X_{t + 1},t}$  and summing over  $g\in [G]$
$$
n_{X_{t + 1},t}\underbrace{\sum_{g\in[G]}\sigma_{g}^{\frac{2p}{p + 1}}}_{= \Sigma_{p}}\leq \Big(\sigma_{X_{t + 1}}^{\frac{2p}{p + 1}} + \Sigma_{p}w_{X_{t + 1}}(n_{X_{t + 1},t})\Big)\underbrace{\sum_{g\in[G]}n_{g,t}}_{= t}.
$$
Dividing both sides by  $\Sigma_p > 0$ , and using the formula  $n_{g,t}^* = \frac{\sigma_g^{\frac{2p}{p + 1}}}{\Sigma_p} t$  (see Lemma 1) implies
$$
n _ {X _ {t + 1}, t} \leq n _ {X _ {t + 1}, t} ^ {*} + t w _ {X _ {t + 1}} (n _ {X _ {t + 1}, t}).
$$
Lemma 7 follows by subtracting  $tw_{X_{t + 1}}(n_{X_{t + 1},t})$  from both sides.
Lemma 7 states that the possible excess between the number of samples output by the algorithm and the optimal number of samples is not too big, and can be controlled by the width  $w$ . Since the width decreases in the number of samples, the function
$$
x \rightarrow x - t w _ {g} (x)
$$
must be increasing and has therefore an inverse function that is also increasing, which we denote  $W_g^t(x)$ . We introduce the following sequence, which mimics the behavior stated in Lemma 7:
$$
\tilde {n} _ {g, t} = n _ {g, t}
$$
$$
\text {F o r} t = 1, \dots , 2 G
$$
$$
\tilde {n} _ {g, t + 1} = \tilde {n} _ {g, t} + \mathbb {1} \left(\tilde {n} _ {g, t} \leq W _ {g} ^ {t} \left(n _ {g, t} ^ {*}\right)\right)
$$
$$
\text {F o r} t \geq 2 G
$$
The sequence is easier to analyze and upper bounds the true number of samples  $n$ :
Lemma 8.  $\tilde{n}\geq n$
Proof. By construction, the result holds for  $t = 1, \ldots, 2G$ . Assume for the sake of contradiction that the result does not hold for a  $g \in [G]$  and  $t + 1 > 2G$ , and take such a  $t$  minimal. For such a pair  $(g, t)$ :
$$
\begin{array}{l} 1 \geq \mathbb {1} (X _ {t + 1} = g) = n _ {g, t + 1} - n _ {g, t} \\ > \tilde {n} _ {g, t + 1} - \tilde {n} _ {g, t} \\ = \mathbb {1} \left(\tilde {n} _ {g, t} \leq W _ {g} ^ {t} \left(n _ {g, t} ^ {*}\right)\right) \\ = \mathbb {1} \left(n _ {g, t} \leq W _ {g} ^ {t} \left(n _ {g, t} ^ {*}\right)\right) \geq 0, \\ \end{array}
$$
where the first step follows from the definition of  $n$ . By minimality of  $t$ ,  $n_{g,t+1} > \tilde{n}_{g,t+1}$  and  $n_{g,t} = \tilde{n}_{g,t}$ , which induces the second step. The third step follows from the definition of  $\tilde{n}$ , and the last step follows once again from the minimality of  $t$ .
The strict inequality in the chain of inequalities implies that  $\mathbb{1}(X_{t + 1} = g) = 1$  and  $\mathbb{1}(n_{g,t}\leq W_g^t (n_{g,t}^*)) = 0$ , so that
$$
n _ {X _ {t + 1}, t} > W _ {g} ^ {t} \left(n _ {X _ {t + 1}, t} ^ {*}\right).
$$
By taking the inverse of the increasing function  $W_{g}^{t}$  on both sides, the previous inequality can be rewritten as
$$
n _ {X _ {t + 1}, t} - t w _ {X _ {t + 1}} \left(n _ {X _ {t + 1}, t}\right) > n _ {X _ {t + 1}, t} ^ {*},
$$
contradicting Lemma 7. Therefore the assumption is wrong and  $\tilde{n} \geq n$ , which completes the proof.
Lemma 9. For a fixed  $g$ , the sequence  $\{W_g^t (n_{g,t}^*)\}_{t\geq 1}$  is increasing. Consequently,
$$
\tilde {n} _ {g, t} \leq W _ {g} ^ {t} \left(n _ {g, t} ^ {*}\right) ^ {+} + 2
$$
Proof. For a fixed  $x > 0$  and  $t \geq 1$ ,
$$
(x - (t + 1) w _ {g} (x)) - (x - t w _ {g} (x)) = - w _ {g} (x) <   0,
$$
therefore the sequence of functions  $\{x\to x - tw_g(x)\}_{t\geq 1}$  is decreasing in  $t$ . Consequently, the sequence of its inverse functions  $\{x\to W_g^t (x)\}_{t\geq 1}$  is increasing in  $t$ :
$$
W _ {g} ^ {t + 1} \left(n _ {g, t + 1} ^ {*}\right) \geq W _ {g} ^ {t} \left(n _ {g, t + 1} ^ {*}\right). \tag {13}
$$
Moreover, the function  $W_{g}^{t}$  is increasing in  $\mathbb{R}_+$ :
$$
W _ {g} ^ {t} \left(n _ {g, t + 1} ^ {*}\right) \geq W _ {g} ^ {t} \left(n _ {g, t} ^ {*}\right). \tag {14}
$$
By combining Equations (13) and (14), we obtain
$$
W _ {g} ^ {t + 1} \left(n _ {g, t + 1} ^ {*}\right) \geq W _ {g} ^ {t} \left(n _ {g, t} ^ {*}\right),
$$
which proves that the sequence  $\{W_g^t (n_{g,t}^*)\}_{t\geq 1}$  is increasing, thus completing the proof for the first part of Lemma 9.
The second part follows by induction on  $t \geq 1$ . For  $t \leq 2G$ , the result holds immediately as  $\tilde{n}_{g,t} = n_{g,t}$  and  $n_{g,t} \leq 2$ . We assume the result holds for a  $t \geq 2G$ . We distinguish two cases:
-  $\tilde{n}_{g,t} \leq W_g^t\left(n_{g,t}^*\right)$ : This implies that:
$$
\begin{array}{l} \tilde {n} _ {g, t + 1} = \tilde {n} _ {g, t} + 1 \\ \leq W _ {g} ^ {t} \left(n _ {g, t} ^ {*}\right) + 1 \\ \leq W _ {p} ^ {t + 1, g} \left(n _ {g, t + 1} ^ {*}\right) + 1 \\ \leq W _ {p} ^ {t + 1, g} \left(n _ {g, t + 1} ^ {*}\right) ^ {+} + 2, \\ \end{array}
$$
where the first step stems from the definition of  $\tilde{n}$ , the second step stems from the assumption  $\tilde{n}_{g,t} \leq W_g^t (n_{g,t}^*)$ , and the third step stems from the first part of the proof.
-  $\tilde{n}_{g,t} > W_g^t (n_{g,t}^*)$ : This implies that:
$$
\begin{array}{l} \tilde {n} _ {g, t + 1} = \tilde {n} _ {g, t} \\ \leq W _ {g} ^ {t} \left(n _ {g, t} ^ {*}\right) ^ {+} + 2 \\ \leq W _ {p} ^ {t + 1, g} \left(n _ {g, t + 1} ^ {*}\right) ^ {+} + 2, \\ \end{array}
$$
where the first step stems from the definition of  $\tilde{n}$ , the second step stems from the induction hypothesis, and the third step stems from the first part of the proof.
Studying both cases concludes the induction and proves the inequality for all  $t \geq 1$ . This concludes the proof of Lemma 9.
Next, we derive an upper bound on  $W_{g}^{t}$ :
Lemma 10. For  $x > 0$ ,  $W_{g}^{t}(x) \leq \frac{x}{1 - t \frac{w_{g}(x)}{x}}$ .
Proof. Let  $x, y > 0$  with  $0 \leq y - tw_g(y) = x$ , so that  $y = W_g^t(x)$  by definition of  $W_g^t$ . We have:
$$
\begin{array}{l} \frac {x}{1 - t \frac {w _ {g} (x)}{x}} = \frac {y - t w _ {g} (y)}{1 - t \frac {w _ {g} (y - t w _ {g} (y))}{x - t w _ {g} (x)}} \\ = y \frac {1 - t \frac {w _ {g} (y)}{y}}{1 - t \frac {w _ {g} (y - t w _ {g} (y))}{y - t w _ {g} (y)}} \\ \geq y = W _ {g} ^ {t} (x), \\ \end{array}
$$
where the last step follows from the function  $y \to \frac{w_g(y)}{y}$  being decreasing, which concludes the proof.
We are now ready to prove Lemma 3.
Lemma 3. Variance-UCB collects a vector of samples  $\mathbf{n}$  such that for all  $g \in [G]$ , with probability at least  $1 - \tilde{O}(T^{-2})$ ,
Proof. Conditionally on  $\mathcal{A}_T$ , all the previous lemmas stated in this section hold. Consequently,
$$
\begin{array}{l} n _ {g, T} \leq \tilde {n} _ {g, T} \\ \leq 2 + W _ {g} ^ {t} \left(n _ {g, T} ^ {*}\right) ^ {+} \\ \leq 2 + \frac {n _ {g , T} ^ {*}}{1 - T \frac {w _ {g} (n _ {g , T} ^ {*})}{n _ {g , T} ^ {*}}} \\ \leq 2 + n _ {g, T} ^ {*} + 1 + T w _ {g} \left(n _ {g, T} ^ {*}\right) \\ = 3 + n _ {g, T} ^ {*} + \frac {4 C _ {T} p}{\Sigma_ {p} (p + 1)} \left(\sigma_ {g} + \frac {2 C _ {T}}{\sqrt {n _ {g , T} ^ {*}}}\right) ^ {\frac {p - 1}{p + 1}} \sqrt {n _ {g , T} ^ {*}}, \\ \end{array}
$$
where the first step stems from Lemma 8, the second step stems from Lemma 9, the third step stems from first order approximations and the last step stems from Lemma 10. From Assumption 2,  $\sigma > 0$ , therefore  $n^{*} = \Theta(T)$  and the right hand side is  $\Theta(\sqrt{T})$ .
A.4 Proof of Lemma 4
The goal of this section is to prove Lemma 4. We will do so by combining the optimality of  $\pmb{n}^*$  and the curvature properties of  $R_p$ .
Lemma 4. Let  $p < +\infty$  and  $\boldsymbol{n}^{\prime}\in \mathbb{R}_{+}^{G}$  such that  $\sum_{g\in [G]}n_g^\prime = T$ . Then,
$$
\left| \frac {R _ {p} (\pmb {n} ^ {\prime}) - R _ {p} (\pmb {n} _ {T} ^ {*})}{R _ {p} (\pmb {n} ^ {*})} - \frac {p + 1}{2} \sum_ {g \in [ G ]} \frac {(n _ {g} ^ {\prime} - n _ {g , T} ^ {*}) ^ {2}}{T n _ {g , T} ^ {*}} \right| \leq \frac {7 (p + 2) ^ {2} \Sigma_ {p} ^ {2}}{\sigma_ {\min} ^ {2}} \max _ {g} \left(\frac {n _ {g , T} ^ {*}}{n _ {g} ^ {\prime}}\right) ^ {3 p + 3} \frac {\| \pmb {n} ^ {\prime} - \pmb {n} _ {T} ^ {*} \| _ {\infty} ^ {3}}{T ^ {3}}.
$$
Proof. Since  $R_{p}$  is defined over the constrained set  $\{n \in \mathbb{R}_{+}^{G} | \sum_{g} n_{g} = T\}$  with empty interior, Taylor inequality can't be directly applied. To overcome this, we construct an alternative function that has the same curvature as  $R_{p}$  while being defined on a non-empty interior domain. Define
$$
\mathcal {K} := \left\{\boldsymbol {\lambda} \in [ 0, 1 ] ^ {G - 1} \Big | \sum_ {g} \lambda_ {g} \leq 1 \right\}. \tag {15}
$$
For each vector  $\pmb{n} \in \mathbb{R}_+^G$  with  $\sum_{t=1}^{T} n_g = T$ , we set for each  $g \in [G-1]$ $\lambda_g \coloneqq \frac{n_g}{T}$ . We have  $\lambda \in \mathcal{K}$ . Moreover, we have from (1):
$$
R _ {p} (\boldsymbol {n}) = \left\| \left\{\frac {\sigma_ {g} ^ {2}}{n _ {g}} \right\} _ {g = 1} ^ {G} \right\| _ {p} = \frac {1}{T} \left\| \left\{\frac {\sigma_ {1} ^ {2}}{\lambda_ {1}}, \dots , \frac {\sigma_ {G - 1} ^ {2}}{\lambda_ {G - 1}}, \frac {\sigma_ {G} ^ {2}}{1 - \lambda_ {1} - \dots - \lambda_ {G - 1}} \right\} \right\| _ {p}. \tag {16}
$$
where the last equality follows from the homogeneity property of the  $p$ -norm. Eq. (16) motivates the introduction of the re-scaled function  $r$ :
$$
r _ {p}: \boldsymbol {\lambda} \in \mathcal {K} \rightarrow r _ {p} (\boldsymbol {\lambda}) := \left\|\left\{\frac {\sigma_ {1} ^ {2}}{\lambda_ {1}}, \dots , \frac {\sigma_ {G - 1} ^ {2}}{\lambda_ {G - 1}}, \frac {\sigma_ {G} ^ {2}}{1 - \lambda_ {1} - \dots - \lambda_ {G - 1}} \right\}\right\| _ {p}, \tag {17}
$$
so that Eq. (16) can be written as
$$
R _ {p} (\boldsymbol {n}) = \frac {1}{T} r _ {p} (\boldsymbol {\lambda}). \tag {18}
$$
From the definition of  $\kappa$  in (15), the interior of  $\kappa$ , denoted  $\kappa^{\circ}$ , is non-empty and is equal to
$$
\mathcal {K} ^ {\circ} = \left\{\boldsymbol {\lambda} \in \mathcal {K} | \forall g \in [ G - 1 ], \lambda_ {g} > 0, \quad \sum_ {g \in [ G - 1 ]} \lambda_ {g} <   1 \right\}.
$$
Moreover, the function  $r_p$  is  $\mathcal{C}^3$  in  $\kappa^\circ$ . From Lemma 1,  $\lambda^{*}\in \mathcal{K}^{\circ}$ . Therefore from Taylor's theorem, we have for  $\lambda \in \mathcal{K}^{\circ}$ :
$$
\left| r _ {p} (\boldsymbol {\lambda}) - r _ {p} \left(\boldsymbol {\lambda} ^ {*}\right) - \left(\boldsymbol {\lambda} - \boldsymbol {\lambda} ^ {*}\right) \nabla r _ {p} \left(\boldsymbol {\lambda} ^ {*}\right) - \frac {1}{2} \langle \mathcal {H} \left(\boldsymbol {\lambda} ^ {*}\right) \left(\boldsymbol {\lambda} - \boldsymbol {\lambda} ^ {*}\right), \boldsymbol {\lambda} - \boldsymbol {\lambda} ^ {*} \rangle \right| \leq \| \boldsymbol {\lambda} - \boldsymbol {\lambda} ^ {*} \| _ {\infty} ^ {3} \sup  _ { \begin{array}{c} \boldsymbol {u} \in [ \boldsymbol {\lambda}, \boldsymbol {\lambda} ^ {\prime} ] \\ x, y, z \in \mathbb {N} \\ x + y + z = 3 \\ \{g, h, i \} \subset [ G - 1 ] \end{array} } \left| \frac {1}{x ! y ! z !} \frac {\partial^ {3} r _ {p} (\boldsymbol {u})}{\partial \lambda_ {g} \partial \lambda_ {h} \partial \lambda_ {i}} \right|,
$$
where  $\nabla$  is the gradient operator and  $\mathcal{H}$  is the hessian operator. We exploit the optimality of  $\lambda^{*}$  to derive simple expressions for  $\nabla r_{p}(\boldsymbol{\lambda}^{*})$ ,  $\mathcal{H}(\boldsymbol{\lambda}^{*})$ , and  $\frac{\partial^3r_p(\boldsymbol{u})}{\partial\lambda_g\partial\lambda_h\partial\lambda_i}$ . First, since  $\boldsymbol{\lambda}^{*}$  is optimal and an interior point of  $\mathcal{K}$ , the following equality holds:
$$
\nabla r _ {p} \left(\boldsymbol {\lambda} ^ {*}\right) = 0 _ {G}. \tag {20}
$$
In particular,  $\langle \pmb{\lambda} - \pmb{\lambda}^*, \nabla r_p(\pmb{\lambda}^*) \rangle = 0$ .  $\langle \mathcal{H}(\pmb{\lambda}^*)(\pmb{\lambda} - \pmb{\lambda}^*), \pmb{\lambda} - \pmb{\lambda}^* \rangle$  is simplified in the following lemma, which proof is deferred to A.6:
Lemma 11.  $\langle \mathcal{H}(\pmb{\lambda}^{*})(\pmb{\lambda} - \pmb{\lambda}^{*}), \pmb{\lambda} - \pmb{\lambda}^{*} \rangle = (p + 1)r_{p}(\pmb{\lambda}^{*})\sum_{g \in [G]}\frac{(\lambda_{g} - \lambda_{g}^{*})^{2}}{\lambda_{g}^{*}}.$
so that Inequality (19) is rewritten as
$$
\left| r _ {p} (\boldsymbol {\lambda}) - r _ {p} \left(\boldsymbol {\lambda} ^ {*}\right) - \frac {1}{2} (p + 1) r _ {p} \left(\boldsymbol {\lambda} ^ {*}\right) \sum_ {g \in [ G ]} \frac {\left(\lambda_ {g} - \lambda_ {g} ^ {*}\right) ^ {2}}{\lambda_ {g} ^ {*}} \right| \leq \| \boldsymbol {\lambda} - \boldsymbol {\lambda} ^ {*} \| _ {\infty} ^ {3} \sup  _ {\substack {\boldsymbol {u} \in [ \boldsymbol {\lambda}, \boldsymbol {\lambda} ^ {\prime} ] \\ x, y, z \in \mathbb {N} \\ x + y + z = 3 \\ \{g, h, i \} \subset [ G - 1 ]}} \left| \frac {1}{x ! y ! z !} \frac {\partial^ {3} r _ {p} (\boldsymbol {u})}{\partial \lambda_ {g} \partial \lambda_ {h} \partial \lambda_ {i}} \right|. \tag{21}
$$
It remains to bound  $\frac{1}{x!y!z!}\frac{\partial^3r_p(\boldsymbol{u})}{\partial\lambda_g\partial\lambda_h\partial\lambda_i}$ . To do this, the following lemma is applied (the proof is also deferred to A.6):
Lemma 12. The following inequality holds:
$$
\sup_{\substack{\boldsymbol {u}\in [\boldsymbol {\lambda},\boldsymbol{\lambda}^{\prime}]\\ x,y,z\in \mathbb{N}\\ x + y + z = 3\\ \{g,h,i\} \subset [G - 1]}}\left|\frac{1}{x!y!z!}\frac{\partial^{3}r_{p}(\boldsymbol{u})}{\partial\lambda_{g}\partial\lambda_{h}\partial\lambda_{i}}\right|\leq \frac{7(p + 2)^{2}}{(\min_{g}\lambda_{g}^{*})^{2}}\max_{g}\left(\frac{\lambda_{g}^{*}}{\lambda_{g}}\right)^{3p + 3}r_{p}(\boldsymbol{\lambda}^{*}).
$$
which implies after dividing both sides by  $r_p(\lambda)$
$$
\left| \frac {r _ {p} (\boldsymbol {\lambda}) - r _ {p} \left(\boldsymbol {\lambda} ^ {*}\right)}{r _ {p} \left(\boldsymbol {\lambda} ^ {*}\right)} - \frac {p + 1}{2} \sum_ {g \in [ G ]} \frac {\left(\lambda_ {g} - \lambda_ {g} ^ {*}\right) ^ {2}}{\lambda_ {g} ^ {*}} \right| \leq \frac {7 (p + 2) ^ {2}}{\left(\min  _ {g} \lambda_ {g} ^ {*}\right) ^ {2}} \max  _ {g} \left(\frac {\lambda_ {g} ^ {*}}{\lambda_ {g}}\right) ^ {3 p + 3} \| \boldsymbol {\lambda} - \boldsymbol {\lambda} ^ {*} \| _ {\infty} ^ {3}. \tag {22}
$$
By using the change of variable from  $\lambda$  to  $n^{\prime}$ , we have
$$
R _ {p} \left(\boldsymbol {n} ^ {\prime}\right) = \frac {1}{T} r _ {p} (\boldsymbol {\lambda}), \quad R _ {p} \left(\boldsymbol {n} ^ {*}\right) = \frac {1}{T} r _ {p} \left(\boldsymbol {\lambda} ^ {*}\right), \quad \| \boldsymbol {\lambda} - \boldsymbol {\lambda} ^ {*} \| _ {\infty} \leq \frac {\| \boldsymbol {n} ^ {\prime} - \boldsymbol {n} ^ {*} \| _ {\infty}}{T}
$$
$$
\frac {R _ {p} (\pmb {n} ^ {\prime}) - R _ {p} (\pmb {n} _ {T} ^ {*})}{R _ {p} (\pmb {n} ^ {*})} \leq \frac {p + 1}{2} \sum_ {g \in [ G ]} \frac {(n _ {g} ^ {\prime} - n _ {g , T} ^ {*}) ^ {2}}{T n _ {g , T} ^ {*}} + \frac {7 (p + 2) ^ {2} \Sigma_ {p} ^ {2}}{\sigma_ {\mathrm {m i n}} ^ {2}} \max _ {g} \left(\frac {n _ {g , T} ^ {*}}{n _ {g} ^ {\prime}}\right) ^ {3 p + 3} \frac {\| \pmb {n} ^ {\prime} - \pmb {n} _ {T} ^ {*} \| _ {\infty} ^ {3}}{T ^ {3}},
$$
hence proving Lemma 5.
A.5 Putting everything together
We are now ready to complete the proof of Theorem 1.
$$
R e g r e t _ {p, T} \left(V a r i a n c e - U C B, \boldsymbol {\mathcal {D}}\right) = \tilde {O} \left(T ^ {- 2}\right).
$$
Proof. First, notice that
$$
\begin{array}{l} \operatorname {R e g r e t} _ {p, T} (\operatorname {V a r i a n c e - U C B}) = \mathbb {E} _ {\pi} \left[ R _ {p} (\boldsymbol {n}) - R _ {p} ^ {*} \right] \\ = \mathbb {E} \left[ R _ {p} (\boldsymbol {n}) - R _ {p} ^ {*} \mid \mathcal {A} _ {T}\right) \mathbb {P} _ {\pi} \left(\mathcal {A} _ {T}\right) + \mathbb {E} \left[ R _ {p} (\boldsymbol {n}) - R _ {p} ^ {*} \mid \mathcal {A} _ {T} ^ {c} \right] \mathbb {P} _ {\pi} \left(\mathcal {A} _ {T} ^ {c}\right) \\ \leq \mathbb {E} \left[ R _ {p} (\boldsymbol {n}) - R _ {p} ^ {*} | \mathcal {A} _ {T} \right] + \| \boldsymbol {\sigma} ^ {2} \| _ {p} \mathbb {P} _ {\pi} \left(\mathcal {A} _ {T} ^ {c}\right), \\ \end{array}
$$
where the first step stems from the definition of regret introduced in Eq. (4), the second step stems from the law of total expectation, and the third step stems from both  $\mathbb{P}(\mathcal{A}_T) \leq 1$  and  $R_p(\pmb{n}) - R_p^* \leq \| \pmb{\sigma}^2\|_p$ . It remains to show that each term in the rightmost side is in  $\tilde{O}(T^{-2})$ .
First,  $\mathbb{P}_{\pi}(\mathcal{A}_T^c)\leq 2GT^{-2.5} = \tilde{O} (T^{-2})$  . Next, we have conditionally on  $\mathcal{A}_T$  ..
$$
\begin{array}{l} \frac {R _ {p} (\boldsymbol {n}) - R _ {p} (\boldsymbol {n} ^ {*})}{R _ {p} (\boldsymbol {n} ^ {*})} \leq \frac {p + 1}{2} \sum_ {g \in [ G ]} \frac {(n _ {g} - n _ {g , T} ^ {*}) ^ {2}}{T n _ {g , T} ^ {*}} + \frac {7 (p + 2) ^ {2} \Sigma_ {p} ^ {2}}{\sigma_ {\min} ^ {2}} \max  \left(\frac {n _ {g , T} ^ {*}}{n _ {g}}\right) ^ {3 p + 3} \frac {\| \boldsymbol {n} - \boldsymbol {n} ^ {*} \| _ {\infty} ^ {3}}{T ^ {3}} \\ \leq \frac {p + 1}{2} \frac {G \| \boldsymbol {n} - \boldsymbol {n} ^ {*} \| _ {\infty} ^ {2}}{T \min _ {g} n _ {g , T} ^ {*}} + \frac {7 (p + 2) ^ {2} \Sigma_ {p} ^ {2}}{\sigma_ {\min} ^ {2}} \max _ {g} \left(\frac {n _ {g , T} ^ {*}}{n _ {g}}\right) ^ {3 p + 3} \frac {\| n - n ^ {*} \| _ {\infty} ^ {3}}{T ^ {3}}, \\ \end{array}
$$
where the first inequality stems from Lemma 4, and the second inequality stems from  $(n_{g,T} - n_{g,T}^{*})^{2} \leq \| \pmb{n} - \pmb{n}^{*}\|_{\infty}^{2}$ . Since  $\sum_{g} n_{g,T} = T$ , from Lemma 3,
$$
\begin{array}{l} n _ {g, T} - n _ {g, T} ^ {*} = - \sum_ {h \neq g} n _ {h, T} - n _ {h, T} ^ {*} \\ \geq - 3 (G - 1) - \sum_ {h \neq g} \frac {4 C _ {T} p}{\Sigma_ {p} (p + 1)} \left(\sigma_ {g} + \frac {2 C _ {T}}{\sqrt {n _ {h , T} ^ {*}}}\right) ^ {\frac {p - 1}{p + 1}} \sqrt {n _ {h , T} ^ {*}} \\ \geq - 3 (G - 1) - G \max  _ {h} \frac {4 C _ {T} p}{\Sigma_ {p} (p + 1)} \left(\sigma_ {g} + \frac {2 C _ {T}}{\sqrt {n _ {h , T} ^ {*}}}\right) ^ {\frac {p - 1}{p + 1}} \sqrt {n _ {h , T} ^ {*}} \\ \geq - 3 G - \frac {4 G C _ {T} p}{\Sigma_ {p} (p + 1)} \left(\min  \sigma_ {g} + \frac {2 C _ {T}}{\sqrt {\operatorname* {m i n} _ {h} n _ {h , T} ^ {*}}}\right) ^ {\frac {p - 1}{p + 1}} \sqrt {\underset {h} {\min } n _ {h , T} ^ {*}}, \\ \end{array}
$$
where the first step stems from  $\sum_{h} n_{h,T} - n_{h,T}^{*} = \sum_{h} n_{h,T} - \sum_{h} n_{h,T}^{*} = T - T = 0$ , the second step stems from Lemma 3, and the last steps stem from taking the max over the sum. The last inequality implies
$$
\left\| \boldsymbol {n} - \boldsymbol {n} ^ {*} \right\| _ {\infty} \leq 3 G + \frac {4 G C _ {T} p}{\Sigma_ {p} (p + 1)} \left(\min  \sigma_ {g} + \frac {2 C _ {T}}{\sqrt {\min  _ {h} n _ {h , T} ^ {*}}}\right) ^ {\frac {p - 1}{p + 1}} \sqrt {\min  _ {h} n _ {h , T} ^ {*}}. \tag {23}
$$
In particular,  $\| \pmb {n} - \pmb{n}^{*}\|_{\infty} = \tilde{O} (\sqrt{\min_{h}n_{h,t}^{*}}) = \tilde{O} (\sqrt{T})$  and
$$
\max  _ {g} \frac {n _ {g , T} ^ {*}}{n _ {g , T}} \leq \frac {1}{1 - \frac {\| \boldsymbol {n} - \boldsymbol {n} ^ {*} \| _ {\infty}}{\min  _ {h} n ^ {*}}} = \frac {1}{1 - \tilde {O} (T ^ {- 0 . 5})} = \tilde {O} (1).
$$
$$
\begin{array}{l} \frac {p + 1}{2} \frac {G \| n - n ^ {*} \| _ {\infty} ^ {2}}{T \min _ {g} n _ {g , T} ^ {*}} + \frac {7 (p + 2) ^ {2} \Sigma_ {p} ^ {2}}{\sigma_ {\min} ^ {2}} \max _ {g} \left(\frac {n _ {g , T} ^ {*}}{n _ {g}}\right) ^ {3 p + 3} \frac {\| \boldsymbol {n} - \boldsymbol {n} ^ {*} \| _ {\infty} ^ {3}}{T ^ {3}} = \frac {p + 1}{2} \frac {G \tilde {O} (T)}{T \Theta (T)} + \frac {7 (p + 2) ^ {2} \Sigma_ {p} ^ {2}}{\sigma_ {\min} ^ {2}} \tilde {O} (1) \frac {\tilde {O} (T ^ {1 . 5})}{T ^ {3}} \\ = \tilde {O} (T ^ {- 1}). \\ \end{array}
$$
Recall from Lemma 1 that  $R_{p}^{*} = R_{p}(\boldsymbol{n}^{*}) = \Theta (T^{-1})$ . Thus by taking the conditional expectation on  $\mathcal{A}_T$ , we have
$$
\mathbb {E} [ R _ {p} (\boldsymbol {n}) - R _ {p} ^ {*} | \mathcal {A} _ {T} ] \leq R _ {p} ^ {*} \tilde {O} (T ^ {- 1}) = \tilde {O} (T ^ {- 2}),
$$
which concludes the proof of Theorem 1.
A.6 Proof of auxiliary lemmas
In this section, we prove Lemmas 11 and 12. To do so, simple expressions for the derivatives of  $r_p$  are needed. These are established in the following lemma:
Lemma 13. For  $g, h, i \in [G - 1]$ , we introduce the following functions in  $\mathcal{K}^{\circ}$ :
$$
H _ {g}: \boldsymbol {\lambda} \in \mathcal {K} \rightarrow \frac {\sigma_ {G} ^ {2 p}}{(1 - \lambda_ {1} - \ldots - \lambda_ {G - 1}) ^ {p + 1}} - \frac {\sigma_ {g} ^ {2 p}}{\lambda_ {g} ^ {p + 1}}
$$
$$
G _ {h, g} := \frac {1}{p + 1} \frac {\partial}{\partial \lambda_ {g}} H _ {h},
$$
$$
I _ {g, h, i} := \frac {1}{p + 2} \frac {\partial}{\partial \lambda_ {i}} G _ {g, h}.
$$
The following holds:
1.  $\nabla r_{p} = r_{p}^{1 - p}\mathbf{H}$
2.  $\mathcal{H}_{g,h} = (1 - p)r_p^{1 - 2p}H_gH_h + (p + 1)r_p^{1 - p}G_{g,h}$
$$
\begin{array}{l} \frac {\partial^ {3} r _ {p}}{\partial \lambda_ {g} \partial \lambda_ {h} \partial_ {i}} = (1 - p) (1 - 2 p) H _ {g} H _ {h} H _ {i} r _ {p} ^ {1 - 3 p} \\ + (1 - 2 p) (1 + p) \left(H _ {g} G _ {h, i} + H _ {h} G _ {i, g} + H _ {g} G _ {h, i}\right) \\ + r _ {p} ^ {1 - 2 p} + (p + 1) (p + 2) I _ {g, h, i} r _ {p} ^ {1 - p}. \\ \end{array}
$$
Proof. Fix a  $\lambda \in \mathcal{K}^{\circ}$  and  $g, h, i \in [G - 1]$ .
Expression of the gradient: On the one hand, the definition of  $r_p$  in (17) implies
$$
r _ {p} ^ {p} (\boldsymbol {\lambda}) = \left\| \left\{\frac {\sigma_ {1} ^ {2}}{\lambda_ {1}}, \dots , \frac {\sigma_ {G - 1} ^ {2}}{\lambda_ {G - 1}}, \frac {\sigma_ {G} ^ {2}}{1 - \lambda_ {1} - \dots - \lambda_ {G - 1}} \right\} \right\| _ {p} ^ {p} = \frac {\sigma_ {G} ^ {2 p}}{(1 - \lambda_ {1} - \dots - \lambda_ {G - 1}) ^ {p}} + \sum_ {h \leq G - 1} \frac {\sigma_ {h} ^ {2 p}}{\lambda_ {h} ^ {p}},
$$
$$
\frac {\partial}{\partial \lambda_ {g}} \left(r _ {p} ^ {p}\right) (\boldsymbol {\lambda}) = p \left[ \frac {\sigma_ {G} ^ {2 p}}{\left(1 - \lambda_ {1} - \dots - \lambda_ {G - 1}\right) ^ {p + 1}} - \frac {\sigma_ {g} ^ {2 p}}{\lambda_ {g} ^ {p + 1}} \right] = p H _ {g} (\boldsymbol {\lambda}). \tag {24}
$$
On the other hand, from the formula  $(f^p)' = pf'f^{p-1}$ ,
$$
\frac {\partial}{\partial \lambda_ {g}} \left(r _ {p} ^ {p}\right) (\boldsymbol {\lambda}) = p r _ {p} ^ {p - 1} (\boldsymbol {\lambda}) \frac {\partial}{\partial \lambda_ {g}} r _ {p} (\boldsymbol {\lambda}), \tag {25}
$$
Combining Equations (24) and (25) yields
$$
\frac {\partial}{\partial \lambda_ {g}} r _ {p} ^ {p} = p H _ {g} = p r _ {p} ^ {p - 1} \frac {\partial}{\partial \lambda_ {g}} r _ {p},
$$
$$
\frac {\partial}{\partial \lambda_ {g}} r _ {p} = r _ {p} ^ {1 - p} H _ {g}. \tag {26}
$$
Therefore  $\nabla r_{p} = \left(\frac{\partial}{\partial\lambda_{1}} r_{p},\dots ,\frac{\partial}{\partial\lambda_{G - 1}} r_{p}\right) = r_{p}^{1 - p}(H_{1},\dots ,H_{G - 1}) = r_{p}^{1 - p}\boldsymbol {H}$ , which proves the first Eq. of Lemma 13.
Expression of the Hessian: We have