SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \sum_ {s = 1} ^ {t} \ell_ {s} (W _ {*}) - \sum_ {s = 1} ^ {t} \ell (\widetilde {\mathbf {z}} _ {s}, \mathbf {y} _ {s}) \leq \frac {5}{4} (S _ {\mu} + S) + 4 (S _ {\mu} + S) \log \left(\frac {\sqrt {1 + 2 t}}{\delta}\right) 2 + \mu t \\ \leq 3 \log (1 + (1 + K) t) \left(\frac {5}{4} + 4 \log \left(\frac {\sqrt {1 + 2 t}}{\delta}\right)\right) + 2, \\ \end{array}
$$
where the last inequality is by the parameter setting  $\mu = 1 / t$  and, as a consequence,  $S_{\mu} = \log (1 + K) + 2\log (1 + (K + 1)t)$ . We have completed the proof.
C.1.4 Proof of Lemma 14
Proof of Lemma 14. Our proof starts with the observation made by [26, Proposition 2] that  $\widetilde{\mathbf{z}}_s$  is an aggregating forecaster [48, Chapter, 3.5] for the logistic function, which satisfies
$$
\ell (\widetilde {\mathbf {z}} _ {s}, \mathbf {y} _ {s}) \leq - \ln \left(\mathbb {E} _ {W \sim P _ {s}} \left[ e ^ {- \ell_ {s} (W)} \right]\right).
$$
Then, a direct calculation with the definition of Gaussian distribution  $P_{s}\sim \mathcal{N}(\overrightarrow{W}_{s},cH_{s}^{-1})$  gives
$$
\ell \left(\widetilde {\mathbf {z}} _ {s}, \mathbf {y} _ {s}\right) \leq - \ln \left(\mathbb {E} _ {W \sim P _ {s}} \left[ e ^ {- \ell_ {s} (W)} \right]\right) = - \ln \left(\frac {1}{Z _ {s}} \int_ {\mathbb {R} ^ {K d}} e ^ {- L _ {s} (W)} \mathrm {d} \overrightarrow {W}\right), \tag {40}
$$
where we define the loss function  $L_{s}(W) = \ell_{s}(W) + (2c)^{-1}\| \overrightarrow{W} -\overrightarrow{W}_{s}\|_{H_{s}}^{2}$  and  $Z_{s} = \sqrt{(2\pi)^{Kd}c|H_{s}^{-1}|}$  being the normalization factor.
Then, we consider the following quadratic approximation,
$$
\widetilde {L} _ {s} (W) = L _ {s} \left(W _ {s + 1}\right) + \left\langle \nabla L _ {s} \left(W _ {s + 1}\right), \overrightarrow {W} - \overrightarrow {W} _ {s + 1} \right\rangle + \frac {1}{2 c} \| \overrightarrow {W} - \overrightarrow {W} _ {s + 1} \| _ {H _ {s}} ^ {2}. \tag {41}
$$
According to Lemma 18, we have
$$
L _ {s} (W) \leq \widetilde {L} _ {s} (W) + e ^ {6 \| \overrightarrow {W} - \overrightarrow {W} _ {s + 1} \| _ {2} ^ {2}} \| \overrightarrow {W} - \overrightarrow {W} _ {s + 1} \| _ {\nabla \ell_ {s} (W _ {s + 1})} ^ {2}.
$$
Then, we can low bound the term in the expectation by
$$
\begin{array}{l} \mathbb {E} _ {W \sim P _ {s}} \left[ e ^ {- \ell_ {s} (W)} \right] \\ = \frac {1}{Z _ {s}} \int_ {\mathbb {R} ^ {K d}} e ^ {- L _ {s} (W)} \mathrm {d} W \\ \geq \frac {1}{Z _ {s}} \int_ {\mathbb {R} ^ {K d}} e ^ {- \widetilde {L} _ {s} (W) - e ^ {6 \| \overrightarrow {W} - \overrightarrow {W} _ {s + 1} \| _ {2} ^ {2}} \| \overrightarrow {W} - \overrightarrow {W} _ {s + 1} \| _ {\nabla \ell_ {s} (W _ {s + 1})} ^ {2}} \mathrm {d} W \\ = \frac {e ^ {- L _ {s} \left(W _ {s + 1}\right)}}{Z _ {s}} \int_ {\mathbb {R} ^ {K d}} \widetilde {f} _ {s + 1} (W) \cdot e ^ {- \langle \nabla L _ {s} \left(W _ {s + 1}\right), \vec {W} - \vec {W} _ {s + 1} \rangle} \mathrm {d} W, \tag {42} \\ \end{array}
$$
where we define the function  $\widetilde{f}_s:\mathcal{W}\mapsto \mathbb{R}$  as
$$
\widetilde {f} _ {s + 1} (W) = \exp \left(- \frac {1}{2 c} \| \overrightarrow {W} - \overrightarrow {W} _ {s + 1} \| _ {H _ {s}} ^ {2} - e ^ {6 \| \overrightarrow {W} - \overrightarrow {W} _ {s + 1} \| _ {2} ^ {2}} \| \overrightarrow {W} - \overrightarrow {W} _ {s + 1} \| _ {\nabla^ {2} \ell_ {s} (W _ {s + 1})} ^ {2}\right).
$$
Denote by  $\widetilde{Z}_{s+1} = \int_{W \in \mathbb{R}^{Kd}} \widetilde{f}_{s+1}(W) \, \mathrm{d}W \leq +\infty$  the normalization factor and  $\widetilde{P}_{s+1}$  the distribution whose density function is  $\widetilde{f}_{s+1}(W) / \widetilde{Z}_{s+1}$ , we can further rewrite the last line of the above displayed equation (42) as
$$
\mathbb {E} _ {W \sim P _ {s}} \left[ e ^ {- \ell_ {s} (W)} \right] \geq \frac {e ^ {- L _ {s} (W _ {s + 1})} \widetilde {Z} _ {s + 1}}{Z _ {s}} \mathbb {E} _ {W \sim \widetilde {P} _ {s + 1}} \left[ e ^ {- \langle \nabla L _ {s} (W _ {s + 1}), \overrightarrow {W} - \overrightarrow {W} _ {s + 1} \rangle} \right]
$$
$$
\begin{array}{l} \geq \frac {e ^ {- L _ {s} \left(W _ {s + 1}\right)} \tilde {Z} _ {s + 1}}{Z _ {s}} \exp \left(- \mathbb {E} _ {W \sim \tilde {P} _ {s + 1}} [ \langle \nabla L _ {s} (W _ {s + 1}), \vec {W} - \vec {W} _ {s + 1} \rangle ]\right) \\ = \frac {e ^ {- L _ {s} \left(W _ {s + 1}\right)} \widetilde {Z} _ {s + 1}}{Z _ {s}}, \tag {43} \\ \end{array}
$$
where the second inequality is due to the Jensen's inequality and the last equality comes from the fact that  $\widetilde{f}_{s + 1}(W) / \widetilde{Z}_{s + 1}$  is symmetry around  $\overrightarrow{W}_{s + 1}$  and thus  $\mathbb{E}_{W\sim \widetilde{F}_{s + 1}}[\langle \nabla L_s(W_{s + 1}),\overrightarrow{W} -\overrightarrow{W}_{s + 1}\rangle ] = 0$  Plugging (43) into (40), we arrive
$$
\ell \left(\widetilde {\mathbf {z}} _ {s}, \mathbf {y} _ {s}\right) \leq L _ {s} \left(W _ {s + 1}\right) + \ln Z _ {s} - \ln \widetilde {Z} _ {s + 1}, \tag {44}
$$
where we can further convert the last term  $-\ln \widetilde{Z}_{s + 1}$  to
$$
\begin{array}{l} - \ln \widetilde {Z} _ {s + 1} = - \ln \left(\int_ {W \in \mathbb {R} ^ {K d}} e ^ {- \frac {1}{2 c} \| \vec {W} - \vec {W} _ {s + 1} \| _ {H _ {s}} ^ {2} - e ^ {6 \| \vec {W} - \vec {W} _ {s + 1} \| _ {2} ^ {2}} \| \vec {W} - \vec {W} _ {s + 1} \| _ {\nabla^ {2} \ell_ {s} (W _ {s + 1})} ^ {2}} \mathrm {d} W\right) \\ = - \ln \left(\widehat {Z} _ {s + 1} \cdot \mathbb {E} _ {W \sim \widehat {P} _ {s + 1}} \left[ e ^ {- e ^ {6 \| \vec {W} - \vec {W} _ {s + 1} \| _ {2} ^ {2}} \| \vec {W} - \vec {W} _ {s + 1} \| _ {\nabla^ {2} \ell_ {s} (W _ {s + 1})} ^ {2}} \right]\right) \\ \leq - \ln \widehat {Z} _ {s + 1} + \mathbb {E} _ {W \sim \widehat {P} _ {s + 1}} \left[ e ^ {6 \| \overrightarrow {W} - \overrightarrow {W} _ {s + 1} \| _ {2} ^ {2}} \| \overrightarrow {W} - \overrightarrow {W} _ {s + 1} \| _ {\nabla^ {2} \ell_ {s} (W _ {s + 1})} ^ {2} \right] \\ = - \ln Z _ {s} + \underbrace {\mathbb {E} _ {W \sim \widehat {P} _ {s + 1}} \left[ e ^ {6 \| \overrightarrow {W} - \overrightarrow {W} _ {s + 1} \| _ {2} ^ {2}} \| \overrightarrow {W} - \overrightarrow {W} _ {s + 1} \| _ {\nabla^ {2} \ell_ {s} (W _ {s + 1})} ^ {2} \right]} _ {: = \Omega_ {s}} \tag {45} \\ \end{array}
$$
where  $\widehat{P}_{s+1} = \mathcal{N}(\overrightarrow{W}_{s+1}, cH_s^{-1})$  and  $\widehat{Z}_{s+1} = \int_{\mathbb{R}^{Kd}} e^{-\frac{1}{2c} \| \overrightarrow{W} - \overrightarrow{W}_{s+1}\|_{H_s}^2} \, \mathrm{d}W$ . In above, the first equality is by definition of  $\widetilde{Z}_{s+1}$  and the second one comes from the definition of  $\widehat{P}_{s+1}$ . The last second inequality is due to the Jensen's inequality such that  $\ln(\mathbb{E}[a]) \geq \mathbb{E}[\ln a]$  for the random variable  $a \in \mathbb{R}$ . The last equality is by the fact  $\widehat{Z}_{s+1} = \int_{\mathbb{R}^{Kd}} e^{-\frac{1}{2c} \| \overrightarrow{W} - \overrightarrow{W}_{s+1}\|_{H_s}^2} \, \mathrm{d}W = Z_s$ .
Combining (44) and (45) and taking the summation overt  $t$  iterations, we have
$$
\sum_ {s = 1} ^ {t} \ell \left(\widetilde {\mathbf {z}} _ {s}, \mathbf {y} _ {s}\right) = \sum_ {s = 1} ^ {t} L _ {s} \left(W _ {s + 1}\right) + \sum_ {s = 1} ^ {t} \Omega_ {s}. \tag {46}
$$
Following a similar argument in [27], we can bound the last term as follows. Specifically, by the Cauchy-Schwarz inequality, we can decompose the term  $\Omega_s$  as
$$
\Omega_ {s} \leq \underbrace {\sqrt {\mathbb {E} _ {W \sim \widehat {P} _ {s + 1}} \left[ e ^ {1 2 \| \overrightarrow {W} - \overrightarrow {W} _ {s + 1} \| _ {2} ^ {2}} \right]}} _ {\text {t e r m (a - 1)}} \underbrace {\sqrt {\mathbb {E} _ {W \sim \widehat {P} _ {s + 1}} \left[ \| \overrightarrow {W} - \overrightarrow {W} _ {s + 1} \| _ {\nabla^ {2} \ell_ {s} (W _ {s + 1})} ^ {4} \right]} _ {\text {t e r m (a - 2)}},
$$
Then, remarking that  $\widehat{P}_{s+1} = \mathcal{N}\left(W_{s+1}, cH_s^{-1}\right)$ , the key observation here is that  $\overrightarrow{W} - \overrightarrow{W}_{s+1}$  also follows the same distribution as
$$
\sum_ {i = 1} ^ {K d} \sqrt {c \lambda_ {i} \left(H _ {s} ^ {- 1}\right)} X _ {i} \mathbf {e} _ {i}, \quad \text {w h e r e} X _ {i} \stackrel {i. i. d.} {\sim} \mathcal {N} (0, 1), \forall i \in [ K d ], \tag {47}
$$
where  $\{\mathbf{e}_i\in \mathbb{R}^{Kd}\}_{i = 1}^{Kd}$  is a set of certain orthogonal basis and  $\lambda_{i}\coloneqq \lambda_{i}(H_{s}^{-1})$  denotes the  $i$ -th largest eigenvalue of  $H_{s}^{-1}$ . Then, we can bound the first term of term (a-1) by
$$
\begin{array}{l} \mathsf {t e r m} (\mathbf {a - 1}) \leq \sqrt {\mathbb {E} _ {X _ {i}} \left[ \Pi_ {i = 1} ^ {K d} e ^ {1 2 c \lambda_ {i} X _ {i} ^ {2}} \right]} \leq \sqrt {\Pi_ {i = 1} ^ {K d} \mathbb {E} _ {X _ {i}} \left[ e ^ {\frac {1 2 c}{\lambda} X _ {i} ^ {2}} \right]} \\ = \left(\mathbb {E} _ {X \sim \chi^ {2}} \left[ e ^ {\frac {1 2 c}{\lambda} X} \right]\right) ^ {\frac {K d}{2}} \leq \mathbb {E} _ {X \sim \chi^ {2}} \left[ e ^ {\frac {6 K d c}{\lambda} X} \right], \\ \end{array}
$$
where  $\chi$  is the chi-square distribution and the last second inequality is due to  $H_{s} \succcurlyeq \lambda I_{Kd}$  and thus  $\lambda_{i} \leq 1 / \lambda$ . The last inequality is due to the Jensen's inequality since  $Kd > 1$ . Then, by further requiring  $\lambda \geq 24Kdc$ , we have
$$
\operatorname {t e r m} (\mathbf {a} - 1) \leq \mathbb {E} _ {X \sim \chi^ {2}} \left[ e ^ {\frac {X}{4}} \right] \leq \sqrt {2}, \tag {48}
$$
where the last inequality is because the moment-generating function for  $\chi^2$ -distribution is bounded by  $\mathbb{E}_{X \sim \chi^2} [e^{tX}] \leq 1 / \sqrt{1 - 2t}$  for all  $t \leq 1/2$ .
Then, we consider the second on the upper bound of term (b). A direct calculation gives,
$$
\mathsf {t e r m} \left(\mathsf {a - 2}\right) = \mathbb {E} _ {W \sim \mathcal {N} \left(\mathbf {0}, \alpha H _ {s} ^ {- 1}\right)} \left[ \| (\nabla^ {2} \ell_ {s} (W _ {s + 1})) ^ {\frac {1}{2}} \overrightarrow {W} \| ^ {4} \right] = \mathbb {E} _ {W \sim \mathcal {N} \left(\mathbf {0}, \alpha \tilde {H} _ {s} ^ {- 1}\right)} \left[ \| \overrightarrow {W} \| _ {2} ^ {4} \right],
$$
where  $\widetilde{H}_s = (\nabla^2\ell_s(W_{s + 1}))^{-\frac{1}{2}}H_s(\nabla^2\ell_s(W_{s + 1}))^{-\frac{1}{2}}$ . Then, let  $\widetilde{\lambda}_i\coloneqq \lambda_i\Bigg(c\widetilde{H}_s^{-1}\Bigg)$  be the  $i$ -th largest eigenvalue of the matrix. A similar decomposition as (47) shows that
$$
\begin{array}{l} \mathbf {t e r m} (\mathbf {a - 2}) = \sqrt {\mathbb {E} _ {X _ {i} \sim \mathcal {N} (0 , 1)} \left[ \left\| \sum_ {i = 1} ^ {K d} \sqrt {\widetilde {\lambda} _ {i}} X _ {i} \mathbf {e} _ {i} \right\| _ {2} ^ {4} \right]} = \sqrt {\mathbb {E} _ {X _ {i} \sim \mathcal {N} (0 , 1)} \left[ \left(\sum_ {i = 1} ^ {K d} \widetilde {\lambda} _ {i} X _ {i} ^ {2}\right) ^ {2} \right]} \\ = \sqrt {\sum_ {i = 1} \sum_ {j = 1} \widetilde {\lambda} _ {i} \widetilde {\lambda} _ {j} \mathbb {E} _ {X _ {i} , X _ {j} \sim \mathcal {N} (0 , 1)} [ X _ {i} ^ {2} X _ {j} ^ {2} ]} \leq \sqrt {3 \sum_ {i = 1} ^ {K d} \sum_ {j = 1} ^ {K d} \widetilde {\lambda} _ {i} \widetilde {\lambda} _ {j}} = \sqrt {3} c \mathrm {T r} \big (\widetilde {H} _ {s} ^ {- 1} \big), \\ \end{array}
$$
where the last inequality is due to  $\mathbb{E}_{X_i,X_j\sim \mathcal{N}(0,1)}[X_i^2 X_j^2 ]\leq 3$  for all  $i,j\in [Kd]$  and the last equality comes from the fact that  $\sum_{i = 1}^{Kd}\widetilde{\lambda}_i = \mathrm{Tr}\bigl (c\widetilde{H}_s^{-1}\bigr)$ . The notation  $\operatorname {Tr}(A)$  is used to denote the trace of the matrix  $A$ .
Then, we proceed to bound the trace  $\mathrm{Tr}\big(\widetilde{H}_s^{-1}\big)$  with  $\widetilde{H}_s = H_s\nabla^2\ell_s(W_{s + 1})$ . Recall the definition of  $H_{s} = \lambda I_{Kd} + \sum_{\tau = 1}^{s - 1}\nabla^{2}\ell_{\tau}(W_{\tau +1})$ . We define  $M_{s + 1} = \lambda I_{Kd} / 2 + \sum_{\tau = 1}^{s}\nabla^{2}\ell_{\tau}(W_{\tau +1})$ . Under the condition  $\lambda \geq 2$  for any  $s\in [T]$ , we have  $\nabla^2\ell_s(W)\preccurlyeq I_{Kd}\leq \frac{\lambda}{2} I_{Kd}$  for any  $s\in [T]$  and  $W\in \mathcal{W}$ . Then, we have  $H_{s}\succcurlyeq M_{s + 1}$ . Then, we can bound the trace by
$$
\begin{array}{l} \operatorname {T r} \left(\widetilde {H} _ {s} ^ {- 1}\right) = \operatorname {T r} \left(H _ {s} ^ {- 1} \nabla^ {2} \ell_ {s} \left(W _ {s + 1}\right)\right) \leq \operatorname {T r} \left(M _ {s + 1} ^ {- 1} \nabla^ {2} \ell_ {s} \left(W _ {s + 1}\right)\right) \\ = \operatorname {T r} \left(M _ {s + 1} ^ {- 1} \left(M _ {s + 1} - M _ {s}\right)\right) \leq \log \frac {\left| M _ {s + 1} \right|}{\left| M _ {s} \right|}. \\ \end{array}
$$
where the last inequality is due to Lemma 4.5 of [16]. As a consequence, we have term (a-2)  $\leq \sqrt{3} c\ln(|M_{s+1}|/|M_s|)$ , which leads to
$$
\Omega_ {s} \leq \operatorname {t e r m} (\mathrm {a} - 1) \cdot \operatorname {t e r m} (\mathrm {a} - 2) \leq \sqrt {6} c \ln \left(\frac {\left| M _ {s + 1} \right|}{\left| M _ {s} \right|}\right) \tag {49}
$$
Combining (46) and (49), we obtain that
$$
\sum_ {s = 1} ^ {t} \ell (\widetilde {\mathbf {z}} _ {s}, \mathbf {y} _ {s}) - \sum_ {s = 1} ^ {t} \ell_ {s} (W _ {s + 1}) \leq \frac {1}{c} \sum_ {s = 1} ^ {t} \| \overrightarrow {W} _ {s} - \overrightarrow {W} _ {s + 1} \| _ {H _ {s}} ^ {2} + \sqrt {6} c \sum_ {s = 1} ^ {t} \ln \left(\frac {| M _ {s + 1} |}{| M _ {s} |}\right).
$$
W can further bound the last term of the displayed equality by  $\sum_{s=1}^{t} \ln \left( \frac{|M_{s+1}|}{|M_s|} \right) \leq \ln \left( \frac{|M_{t+1}|}{|\lambda/2 \cdot I_{Kd}|} \right) \leq K d \ln \left( 1 + \frac{(t+1)L}{2\lambda} \right)$ , which completes the proof.
C.1.5 Useful Lemmas
Proof of Lemma 15. The proof of Lemma 15 shares the same spirit as that of Lemma 6. Let  $\bar{H}_t = \sum_{s=1}^{t-1} \|\mathbf{z}_s\|_{\Sigma_s}^2$ . We can also defined the function
$$
M _ {t} (\xi) = \exp (\xi S _ {t} - \xi^ {2} \bar {H} _ {t})
$$
for any  $t \geq 1$  and  $\xi \in \mathbb{R}$ . For  $t = 0$ , we let  $M_0(\xi) = 0$ . Following the similar arguments in the proof of Lemma 7, one can show that the sequence  $\{M_t(\xi)\}_{t=1}^{\infty}$  is a non-negative super-martingale when  $\xi \leq \frac{1}{2}$ . Then, let  $h(\xi)$  be the density of the normal distribution with precision  $2\lambda$  truncated on the 1-dimensional ball  $\frac{1}{2}\mathcal{B}(1)$ . We can define
$$
\bar {M} _ {t} = \int_ {\xi} \exp \left(\xi S _ {t} - \xi^ {2} \bar {H} _ {t}\right) \mathrm {d} h (\xi).
$$
By the similar arguments in deriving (17), we have
$$
\Pr \left[ \sup  _ {t \in \mathbb {N}} \log \left(\bar {M} _ {t}\right) \geq \log \left(\frac {1}{\delta}\right) \right] = \Pr \left[ \sup  _ {t \in \mathbb {N}} \bar {M} _ {t} \geq \frac {1}{\delta} \right] \leq \delta . \tag {50}
$$
Following the arguments in the proof of [8, Theorem 1], for any  $t \geq 1$ , we have
$$
\bar {M} _ {t} \geq \exp (\xi S _ {t} - \xi^ {2} H _ {t}) \cdot \frac {N (g)}{N (h)}
$$
for any  $|\xi| \leq 1/4$ . In the above,  $g(\xi)$  is the normal distribution with precision  $2H_{t}$  truncated on the interval  $[-1/4, 1/4]$ . Then, let  $\xi_{0} = \frac{1}{4}\sqrt{\frac{\lambda}{H_{t}}},$  we have  $\xi_{0} \leq 1/4$  since  $H_{t} \geq \lambda$ . We can obtain that
$$
\log \left(\bar {M} _ {t}\right) \geq \xi_ {0} S _ {t} - \xi_ {0} ^ {2} H _ {t} + \log \left(\frac {N (g)}{N (h)}\right) = \frac {\sqrt {\lambda}}{4} \cdot \frac {S _ {t}}{\sqrt {H _ {t}}} - \frac {\lambda}{1 6} + \log \left(\frac {N (g)}{N (h)}\right). \tag {51}
$$
A combination of (50) and (51) shows that
$$
\Pr \left[ \forall t \geq 1, \frac {\sqrt {\lambda}}{4} \cdot \frac {S _ {t}}{\sqrt {H _ {t}}} - \frac {\lambda}{1 6} + \log \left(\frac {N (g)}{N (h)}\right) \leq \log \left(\frac {1}{\delta}\right) \right] \geq 1 - \delta ,
$$
$$
\Pr \left[ \forall t \geq 1, S _ {t} \leq \sqrt {H _ {t}} \left(\frac {\sqrt {\lambda}}{4} + \frac {4}{\sqrt {\lambda}} \log \left(\frac {N (h)}{\delta N (g)}\right)\right) \right] \geq 1 - \delta .
$$
We complete the proof with Lemma 6 of [8], which shows that  $\log \left(N(h) / N(g)\right) \leq \log \left((2\sqrt{H_t}) / \lambda\right)$ .
Lemma 16 (Proposition 4.1 of [49]). Let the  $\mathbf{w}_{t + 1}$  be the solution of the update rule
$$
\mathbf{w}_{t + 1} = \operatorname *{arg  min}_{\mathbf{w}\in \mathcal{V}}\eta \ell_{t}(\mathbf{w}) + \mathcal{D}_{\psi}(\mathbf{w},\mathbf{w}_{t}),
$$
where  $\mathcal{V} \subseteq \mathcal{W} \subseteq \mathbb{R}^d$  is a non-empty convex set and  $\mathcal{D}_{\psi}(\mathbf{w}_1, \mathbf{w}_2) = \psi(\mathbf{w}_1) - \psi(\mathbf{w}_2) - \langle \nabla \psi(\mathbf{w}_2), \mathbf{w}_1 - \mathbf{w}_2 \rangle$  is the Bregman Divergence w.r.t. a strictly convex and continuously differentiable function  $\psi: \mathcal{W} \mapsto \mathbb{R}$ . Further supposing  $\psi(\mathbf{w})$  is 1-strongly convex w.r.t. a certain norm  $\|\cdot\|$  in  $\mathcal{W}$ , then there exists a  $\mathbf{g}_t' \in \partial \ell_t(\mathbf{w}_{t+1})$  such that
$$
\langle \eta_ {t} \mathbf {g} _ {t} ^ {\prime}, \mathbf {w} _ {t + 1} - \mathbf {u} \rangle \leq \langle \nabla \psi (\mathbf {w} _ {t}) - \nabla \psi (\mathbf {w} _ {t + 1}), \mathbf {w} _ {t + 1} - \mathbf {u} \rangle
$$
for any  $\mathbf{u}\in \mathcal{W}$
Lemma 17. Let  $\ell (\mathbf{z},y) = \sum_{k = 0}^{K}\mathbb{1}\{y = k\} \cdot \log \left(\frac{1}{|\pmb{\sigma}(\mathbf{z})|_k}\right)$  and  $\mathbf{z}\in \mathbb{R}^K$  be a  $K$ -dimensional vector. Define  $\mathbf{z}^{\mu}\triangleq \pmb{\sigma}^{+}\bigl (\mathrm{smooth}_{\mu}(\pmb {\sigma}(\mathbf{z}))\bigr)$ , where  $\mathrm{smooth}_{\mu}(\mathbf{p}) = (1 - \mu)\mathbf{p} + \mu \mathbf{1} / (K + 1)$ . Then, for  $\mu \in [0,1 / 2]$ , we have
$$
\ell (\mathbf {z} ^ {\mu}, y) - \ell (\mathbf {z}, y) \leq 2 \mu
$$
for any  $y\in \{0\} \cup [K]$  . We also have  $\| \mathbf{z}^{\mu}\|_{\infty}\leq \log (K / \mu)$
Proof of Lemma 17. The proof of Lemma 17 is extracted from the proof of [26, Lemma 3] with a slight modification to the logistic loss used in this paper. According to the definition of  $\mathbf{z}^{\mu}$  and the fact that  $\sigma (\pmb{\sigma}^{+}(\mathbf{p})) = \mathbf{p}$  for any  $\mathbf{p}\in \{\mathbf{q}\in [0,1]^K ||\mathbf{q}||_1 < 1\}$ , we have  $\sigma (\mathbf{z}^{\mu}) = (1 - \mu)\sigma (\mathbf{z}) + \mu \mathbf{1} / (K + 1)$ . Then, we have
$$
\ell (\mathbf {z} ^ {\mu}, y) - \ell (\mathbf {z}, y) = \sum_ {k = 0} ^ {K} \mathbb {1} \{y = k \} \cdot \log \left(\frac {[ \boldsymbol {\sigma} (\mathbf {z}) ] _ {k}}{(1 - \mu) [ \boldsymbol {\sigma} (\mathbf {z}) ] _ {k} + \mu \mathbf {1} / (K + 1)}\right) \leq \log (1 - \mu) \leq 2 \mu ,
$$
where the first inequality is due to  $\sum_{k=0}^{K} \mathbb{1}\{y=k\}=1$  and the last inequality is due to  $\log(1/(1-a)) \leq a$  for  $a \in [0,1/2]$ . Besides, let  $\bar{\mathbf{p}} = \sigma(\mathbf{z})$ . According to the definition of  $\mathbf{z}^{\mu}$ , we have
$$
\| \mathbf {z} ^ {\mu} \| _ {\infty} = \max  _ {k \in [ K ]} \left\{\left| \log \left(\frac {(1 - \mu) \widetilde {p} _ {k} + \mu / (K + 1)}{(1 - K \mu / (K + 1)) - (1 - \mu) \| \widetilde {\mathbf {p}} \| _ {1}}\right) \right| \right\}.
$$
Since the term inside the logarithmic function can be bounded by
$$
\frac {\mu}{1 + K (1 - \mu)} \leq \log \left(\frac {(1 - \mu) \widetilde {p} _ {k} + \mu / (K + 1)}{(1 - K \mu / (K + 1)) - (1 - \mu) \| \widetilde {\mathbf {p}} \| _ {1}}\right) \leq 1 + \frac {K + 1}{\mu}
$$
and  $\mu /(1 + K(1 - \mu))\leq 1$  , we have  $\| \mathbf{z}^{\mu}\|_{\infty}\leq \log (1 + (K + 1) / \mu)$
Lemma 18. Let  $L_{s}(W) = \ell_{s}(W) + \frac{1}{2c}\| \overrightarrow{W} -\overrightarrow{W}_{s}\|_{H_{s}}^{2}$ . Then, for any  $W,W_{s + 1}\in \mathcal{W}$ , the quadratic approximation  $\widetilde{L}_s(W) = L_s(W_{s + 1}) + \langle \nabla L_s(W_{s + 1}),\overrightarrow{W} -\overrightarrow{W}_{s + 1}\rangle +\frac{1}{2c}\| \overrightarrow{W} -\overrightarrow{W}_{s + 1}\|_{H_s}^2$  defined as (41) satisfies
Proof of Lemma 18. According to [19, Lemma 4],  $\ell_s$  is a  $\sqrt{6}$ -self-concordant-like function. Then, by Lemma 5 and the fact  $c(x) \leq e^{x^2}$  for any  $x \geq 0$ , then for any  $W \in \mathcal{W}$ , we have
$$
\ell_ {s} (W) \leq \ell_ {s} \left(W _ {s + 1}\right) + \left\langle \nabla \ell_ {s} \left(W _ {s + 1}\right), \overrightarrow {W} - \overrightarrow {W} _ {s + 1} \right\rangle + e ^ {6 \| \overrightarrow {W} - \overrightarrow {W} _ {s + 1} \| _ {2} ^ {2}} \| \overrightarrow {W} - \overrightarrow {W} _ {s + 1} \| _ {\nabla^ {2} \ell_ {s} \left(W _ {s + 1}\right)} ^ {2}. \tag {52}
$$
Besides, since  $g_{s}(W) = \frac{1}{2c}\| \overrightarrow{W} -\overrightarrow{W}_{s}\|_{H_{s}}^{2}$  is a quadratic function, we have
$$
g _ {s} (W) = g _ {s} \left(W _ {s + 1}\right) + \left\langle \nabla g _ {s} \left(W _ {s + 1}\right), \overrightarrow {W} - \overrightarrow {W} _ {s + 1} \right\rangle + \frac {1}{2 c} \| \overrightarrow {W} - \overrightarrow {W} _ {s + 1} \| _ {H _ {s}} ^ {2}. \tag {53}
$$
Then combing (52) and (53), we can obtain an upper bound of  $L_{s}(W) = \ell_{s}(W) + g_{s}(W)$  by
$$
\begin{array}{l} L _ {s} (W) \leq L _ {s} (W _ {s + 1}) + \langle \nabla L _ {s} (W _ {s + 1}), \overrightarrow {W} - \overrightarrow {W} _ {s + 1} \rangle \\ + \frac {1}{2 c} \| \overrightarrow {W} - \overrightarrow {W} _ {s + 1} \| _ {H _ {s}} ^ {2} + e ^ {6 \| \overrightarrow {W} - \overrightarrow {W} _ {s + 1} \| _ {2} ^ {2}} \| \overrightarrow {W} - \overrightarrow {W} _ {s + 1} \| _ {\nabla^ {2} \ell_ {s} (W _ {s + 1})} ^ {2} \\ \leq \widetilde {L} _ {s} (W) + e ^ {6 \| \overrightarrow {W} - \overrightarrow {W} _ {s + 1} \| _ {2} ^ {2}} \| \overrightarrow {W} - \overrightarrow {W} _ {s + 1} \| _ {\nabla^ {2} \ell_ {s} (W _ {s + 1})} ^ {2}, \\ \end{array}
$$
We have complete the proof.
This section presents the proof of Proposition 1.
C.2.1 Main Proof
Proof of Proposition 1. Since  $\rho^{\top}\sigma(W_{*}\mathbf{x}) \leq \rho^{\top}\sigma(W_{t}\mathbf{x}) + |\rho^{\top}\sigma(W_{*}\mathbf{x}) - \rho^{\top}\sigma(W_{t}\mathbf{x})|$ , to show  $\widehat{r}_{t}^{\mathrm{DL}}(\mathbf{x}) = \rho^{\top}\sigma(W_{t}\mathbf{x}) + \epsilon_{t}^{\mathrm{fst}}(\mathbf{x}) + \epsilon_{t}^{\mathrm{snd}}(\mathbf{x})$  is an optimistic estimate, it is sufficient to prove that  $|\rho^{\top}\sigma(W_{*}\mathbf{x}) - \rho^{\top}\sigma(W_{t}\mathbf{x})| \leq \epsilon_{t}^{\mathrm{fst}}(\mathbf{x}) + \epsilon_{t}^{\mathrm{snd}}(\mathbf{x})$ . We have the following decomposition:
$$
\begin{array}{l} \left| \boldsymbol {\rho} ^ {\top} \boldsymbol {\sigma} \left(\boldsymbol {W} _ {*} \mathbf {x}\right) - \boldsymbol {\rho} ^ {\top} \boldsymbol {\sigma} \left(\boldsymbol {W} _ {t} \mathbf {x}\right) \right| \\ = \left| \sum_ {k = 1} ^ {K} \rho_ {k} \left(\sigma_ {k} \left(W _ {*} \mathbf {x}\right) - \sigma_ {k} \left(W _ {t} \mathbf {x}\right)\right) \right| \\ = \left| \sum_ {k = 1} ^ {K} \rho_ {k} \nabla \sigma_ {k} \left(W _ {t} \mathbf {x}\right) ^ {\top} \left(W _ {*} - W _ {t}\right) \mathbf {x} + \sum_ {k = 1} ^ {K} \rho_ {k} \| \left(W _ {*} - W _ {t}\right) \mathbf {x} \| _ {\Xi_ {k, t}} ^ {2} \right| \\ \leq \underbrace {\left| \sum_ {k = 1} ^ {K} \rho_ {k} \nabla \sigma_ {k} (W _ {t} \mathbf {x}) ^ {\top} (W _ {*} - W _ {t}) \mathbf {x} \right|} _ {\text {t e r m (a)}} + \underbrace {\left| \sum_ {k = 1} ^ {K} \rho_ {k} \| (W _ {*} - W _ {t}) \mathbf {x} \| _ {\Xi_ {k , t}} ^ {2} \right|} _ {\text {t e r m (b)}}, \\ \end{array}
$$
where  $\sigma_{k}:\mathbf{x}\mapsto [\pmb {\sigma}(\mathbf{x})]_{k}$  is the  $k$  -th output of the vector-valued function  $\pmb {\sigma}(\mathbf{x})$  and  $\Xi_{k,t} = \int_{\nu = 0}^{1}(1-$ $\nu)\nabla^{2}\sigma_{k}((W_{t} + \nu (W_{*} - W_{t}))\mathbf{x})\mathrm{d}\nu .$  In the above, the last equality is due to the integral formulation of the Taylor series.
Then, we proceed to analyze term (a) and term (b), respectively.