SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \leq \frac {1}{2 \gamma} \left(\| \widehat {\theta} _ {t} - \theta \| _ {A _ {t}} ^ {2} - \| \widehat {\theta} _ {t + 1} - \theta \| _ {A _ {t}} ^ {2} - \| \widehat {\theta} _ {t} - \widehat {\theta} _ {t + 1} \| _ {A _ {t}} ^ {2}\right) + \langle \nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}), \widehat {\theta} _ {t} - \widehat {\theta} _ {t + 1} \rangle \\ \leq \frac {1}{2 \gamma} \left(\| \widehat {\theta} _ {t} - \theta \| _ {A _ {t}} ^ {2} - \| \widehat {\theta} _ {t + 1} - \theta \| _ {A _ {t}} ^ {2} - \| \widehat {\theta} _ {t} - \widehat {\theta} _ {t + 1} \| _ {A _ {t}} ^ {2}\right) + 2 \gamma \| \nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}) \| _ {A _ {t} ^ {- 1}} ^ {2} + \frac {1}{2 \gamma} \| \widehat {\theta} _ {t} - \widehat {\theta} _ {t + 1} \| _ {A _ {t}} ^ {2} \\ = 2 \gamma \| \nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}) \| _ {A _ {t} ^ {- 1}} ^ {2} + \frac {1}{2 \gamma} \left(\| \widehat {\theta} _ {t} - \theta \| _ {A _ {t}} ^ {2} - \| \widehat {\theta} _ {t + 1} - \theta \| _ {A _ {t}} ^ {2}\right) \\ = 2 \gamma \| \nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}) \| _ {A _ {t} ^ {- 1}} ^ {2} + \frac {1}{2 \gamma} \left(\| \widehat {\theta} _ {t} - \theta \| _ {A _ {t - 1}} ^ {2} - \| \widehat {\theta} _ {t + 1} - \theta \| _ {A _ {t}} ^ {2}\right) + \frac {1}{2 \gamma} \left(\nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}) ^ {\top} (\widehat {\theta} _ {t} - \theta)\right) ^ {2}. \tag {34} \\ \end{array}
$$
In above, the first inequality is due to the update rule of the ONS algorithm and the Bregman proximal inequality [67, Lemma 5]. The second inequality is due to the AM-GM inequality. The last equality is due the definition of  $A_{t}$  such that  $\| \widehat{\theta}_t - \theta \|_{A_t}^2 -\| \widehat{\theta}_t - \theta \|_{A_{t - 1}}^2 = (\nabla \widehat{L}_t(\widehat{\theta}_t)^\top (\widehat{\theta}_t - \theta))^2$ .
Plugging (34) into (33) and taking the summation over the interval  $I = [s, e]$ , we have
$$
\begin{array}{l} \sum_ {t \in I} \widetilde {L} _ {t} (\widehat {\theta} _ {t}) - \sum_ {t \in I} \widetilde {L} _ {t} (\theta) \\ \leq \underbrace {2 \gamma \sum_ {t \in I} \| \nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}) \| _ {A _ {t} ^ {- 1}} ^ {2}} _ {\text {t e r m (a)}} + \frac {1}{2 \gamma} \| \widehat {\theta} _ {s} - \theta \| _ {A _ {s}} ^ {2} + \frac {1}{2 \gamma} \sum_ {t \in I} \left(\nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}) ^ {\top} (\widehat {\theta} _ {t} - \theta)\right) ^ {2} \tag {35} \\ + \underbrace{\sum_{t\in I}\langle\nabla\widetilde{L}_{t}(\widehat{\theta}_{t}) - \nabla\widehat{L}_{t}(\widehat{\theta}_{t}),\widehat{\theta}_{t} - \theta\rangle}_{\text{term (b)}} - \frac{1}{4(1 + \beta)}\sum_{t\in I}\mathbb{E}_{t}\left[\left(\nabla \widehat{L}_{t}(\widehat{\theta}_{t})^{\top}(\widehat{\theta}_{t} - \theta)\right)^{2}\right]. \\ \end{array}
$$
Now, we proceed to bound term (a) and term (b), respectively. Firstly, according to Lemma 7, we can show term (a) is bounded by
$$
\text {t e r m} (\mathbf {a}) \leq 2 d \gamma \log \left(1 + \frac {| I | R ^ {2}}{\lambda}\right).
$$
Then, we handle term (b) with the Bernstein-type self-normalized concentration inequality [68, Theorem 4]. Specifically, let  $\mathbb{E}_t[\cdot ] = \mathbb{E}_{S_t\sim \mathcal{D}_t}[\cdot |S_0,S_1,\ldots ,S_{t - 1}]$  be the expectation taken over the draw of  $S_{t}$  conditioned on the randomness until round  $t - 1$ . Denoting by  $Z_{t} = \langle \nabla \widetilde{L}_{t}(\widehat{\theta}_{t})-\nabla \widehat{L}_{t}(\widehat{\theta}_{t}),\widehat{\theta}_{t} - \theta \rangle$ , it is easy to check  $\{Z_{t}\}_{t = s}^{e}$  is a martingale difference sequence such that  $\mathbb{E}_t[Z_t] = 0$  and  $|Z_{t}|\leq 4SR$ . Lemma 8 with the choice  $\nu = 1 / (48(1 + \beta))$  indicates,
$$
\begin{array}{l} \text {t e r m} (\mathfrak {b}) \leq \frac {1}{2 4 (1 + \beta)} \sum_ {t \in I} \left(\left(\nabla \widehat {L} _ {t} \left(\widehat {\theta} _ {t}\right) ^ {\top} \left(\widehat {\theta} _ {t} - \theta\right)\right) ^ {2} + \mathbb {E} _ {t} \left[ \left(\nabla \widehat {L} _ {t} \left(\widehat {\theta} _ {t}\right) ^ {\top} \left(\widehat {\theta} _ {t} - \theta\right)\right) ^ {2} \right]\right) \\ + 4 8 (1 + \beta) \log \frac {\sqrt {2 | I | + 1}}{\delta} + 4 S R \sqrt {\log \frac {2 | I | + 1}{\delta^ {2}}}. \tag {36} \\ \end{array}
$$
Combining the upper bounds for term (a) and term (b) with (35), we have
$$
\begin{array}{l} \sum_ {t \in I} \widetilde {L} _ {t} (\widehat {\theta} _ {t}) - \sum_ {t \in I} \widetilde {L} _ {t} (\theta) \\ \leq 2 d \gamma \log \left(1 + \frac {| I | R ^ {2}}{\lambda}\right) + \frac {2 \lambda S ^ {2}}{\gamma} + 4 8 (1 + \beta) \log \frac {\sqrt {2 | I | + 1}}{\delta} + 4 S R \sqrt {\log \frac {2 | I | + 1}{\delta^ {2}}} \\ + \underbrace {\left(\frac {1}{2 \gamma} + \frac {1}{2 4 (1 + \beta)}\right) \sum_ {t \in I} \left(\nabla \widehat {L} _ {t} \left(\widehat {\theta} _ {t}\right) ^ {\top} \left(\widehat {\theta} _ {t} - \theta\right)\right) ^ {2} - \frac {5}{2 4 (1 + \beta)} \sum_ {t \in I} \mathbb {E} _ {t} \left[ \left(\nabla \widehat {L} _ {t} \left(\widehat {\theta} _ {t}\right) ^ {\top} \left(\widehat {\theta} _ {t} - \theta\right)\right) ^ {2} \right]} _ {\text {t e r m (c)}} \tag {37} \\ \end{array}
$$
Then, setting  $\gamma = 6(1 + \beta)$ , with probability at least  $1 - \delta$ , we can bound term (c) by
$$
\operatorname {t e r m} (\mathsf {c}) \leq \frac {1}{6 (1 + \beta)} \left(\frac {3}{4} \sum_ {t \in I} \nabla \widehat {L} _ {t} \left(\widehat {\theta} _ {t}\right) ^ {\top} \left(\widehat {\theta} _ {t} - \theta\right)\right) ^ {2} - \frac {5}{4} \sum_ {t \in I} \mathbb {E} _ {t} \left[ \left(\nabla \widehat {L} _ {t} \left(\widehat {\theta} _ {t}\right) ^ {\top} \left(\widehat {\theta} _ {t} - \theta\right)\right) ^ {2} \right]
$$
$$
\leq \frac {6 S ^ {2} R ^ {2}}{1 + \beta} \log \frac {\sqrt {2 | I | + 1}}{\delta} + \frac {S ^ {2} R ^ {2}}{1 + \beta} \sqrt {\log \frac {2 | I | + 1}{\delta^ {2}}},
$$
where the last inequality is due to Lemma 9. Combining the upper bound for term (c) and (37) and setting  $\lambda = 1$ , we have
$$
\begin{array}{l} \sum_ {t \in I} \widetilde {L} _ {t} (\widehat {\theta} _ {t}) - \sum_ {t \in I} \widetilde {L} _ {t} (\theta) \\ \leq 1 2 d (1 + \beta) \log \left(1 + | I | R ^ {2}\right) + \frac {S ^ {2}}{3 \beta + 3} \\ + \left(4 8 (\beta + 1) + \frac {6 S ^ {2} R ^ {2}}{1 + \beta}\right) \log \frac {\sqrt {2 | I | + 1}}{\delta} + \left(4 S R + \frac {S ^ {2} R ^ {2}}{\beta + 1}\right) \sqrt {\log \frac {2 | I | + 1}{\delta^ {2}}} \\ = \mathcal {O} (d \beta \log (| I | / \delta)), \\ \end{array}
$$
with probability at least  $1 - 2\delta$  , which completes the proof.
Lemma 5. When all the minimizers  $\theta_t^* = \arg \min_{\theta \in \mathbb{R}^d} L_t(\theta)$  lie in the interior of  $\Theta$ , then we have
$$
\| \nabla \widetilde {L} _ {t} (\theta_ {t} ^ {*}) \| _ {2} ^ {2} = \frac {2 R ^ {2} \ln ((d + 1) / \delta)}{N _ {0}} + o \left(\frac {\log (d / \delta)}{N _ {0}}\right).
$$
Proof of Lemma 5. Since all the minimizers  $\theta_t^* = \arg \min_{\theta \in \mathbb{R}^d} L_t(\theta)$  lie in the interior of  $\Theta$ , then we have  $\nabla L_t(\theta_t^*) = 0$  and the term  $\|\nabla \widetilde{L}_t(\theta_t^*)\|_2^2$  can be rewritten as
$$
\begin{array}{l} \| \nabla \widetilde {L} _ {t} (\theta_ {t} ^ {*}) \| _ {2} ^ {2} = \| \nabla \widetilde {L} _ {t} (\theta_ {t} ^ {*}) - \nabla L _ {t} (\theta_ {t} ^ {*}) \| _ {2} ^ {2} \\ = \frac {1}{4} \| \mathbb {E} _ {\mathbf {x} \sim S _ {0}} [ (\sigma (\mathbf {x} ^ {\top} \boldsymbol {\theta} _ {t} ^ {*}) - 1) \phi (\mathbf {x}) ] - \mathbb {E} _ {\mathbf {x} \sim \mathcal {D} _ {0}} [ (\sigma (\mathbf {x} ^ {\top} \boldsymbol {\theta} _ {t} ^ {*}) - 1) \phi (\mathbf {x}) ] \| _ {2} ^ {2} \\ = \frac {1}{4} \left\| \frac {1}{N _ {0}} \sum_ {n = 1} ^ {N _ {0}} Z _ {n} \right\| _ {2} ^ {2}, \tag {38} \\ \end{array}
$$
where  $\{Z_n\}_{n=1}^{N_0}$  with  $Z_n = (\sigma(\phi(\mathbf{x}_n)^\top \theta_t^*) - 1)\phi(\mathbf{x}_n) - \mathbb{E}_{\mathbf{x} \sim \mathcal{D}_0}[(\sigma(\phi(\mathbf{x})^\top \theta_t^*) - 1)\phi(\mathbf{x})]$  are  $N_0$  independent random vectors. We can check that  $\mathbb{E}_{\mathbf{x}_n \sim \mathcal{D}_0}[Z_n] = 0$  and  $\| Z_n\|_2 \leq 2R$  for any  $\mathbf{x}_n \in \mathcal{X}$ . Then by the Hoeffding's inequality for the random vector [69, Theorem 6.1.1], with probability at least  $1 - \delta$ , we have
$$
\left\| \frac {1}{N _ {0}} \sum_ {n = 1} ^ {N _ {0}} Z _ {n} \right\| _ {2} \leq \frac {2 R \ln ((d + 1) \delta)}{3 N _ {0}} + \sqrt {\frac {8 R ^ {2} \ln ((d + 1) / \delta)}{N _ {0}}}. \tag {39}
$$
Plugging (39) into (38) obtains  $\| \nabla \widetilde{L}_t(\theta_t^*)\| _2^2 = \frac{2R^2\ln((d + 1) / \delta)}{N_0} +o\left(\frac{\log(d / \delta)}{N_0}\right).$
Lemma 6. Let  $\nabla \widehat{L}_t(\widehat{\theta}_t) = \frac{1}{2}\mathbb{E}_{\mathbf{x}\sim S_0}[(\sigma (\phi (\mathbf{x})^\top \widehat{\theta}_t) - 1)\phi (\mathbf{x})] + \frac{1}{2}\mathbb{E}_{\mathbf{x}\sim S_t}[\sigma (\phi (\mathbf{x})^\top \widehat{\theta}_t)\phi (\mathbf{x})]$ . We have
$$
2 \nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}) \nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}) ^ {\top} \preccurlyeq \mathbb {E} _ {\mathbf {x} \sim S _ {0}} [ \mathbf {x x} ^ {\top} ] + \mathbb {E} _ {\mathbf {x} \sim S _ {t}} [ \phi (\mathbf {x}) (\phi (\mathbf {x}) ^ {\top} ].
$$
Proof of Lemma 6. For notation simplicity, we denote by  $\mathbf{a}_t = \mathbb{E}_{\mathbf{x} \sim S_0}[(\sigma(\mathbf{x}^\top \widehat{\theta}_t) - 1)\mathbf{x}]$  and  $\mathbf{b}_t = \mathbb{E}_{\mathbf{x} \sim S_t}[\sigma(\mathbf{x}^\top \widehat{\theta}_t)\mathbf{x}]$ . Then, according to the definition of  $\nabla \widehat{L}_t(\widehat{\theta}_t)$ , we have
$$
\nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}) \nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}) ^ {\top} = \frac {1}{4} (\mathbf {a} _ {t} + \mathbf {b} _ {t}) (\mathbf {a} _ {t} + \mathbf {b} _ {t}) ^ {\top} \preccurlyeq \frac {1}{2} (\mathbf {a} _ {t} \mathbf {a} _ {t} ^ {\top} + \mathbf {b} _ {t} \mathbf {b} _ {t} ^ {\top}),
$$
where the last inequality is due to Lemma 16. We can further bound the R.H.S. of the above inequality by
$$
\begin{array}{l} \mathbf {a} _ {t} \mathbf {a} _ {t} ^ {\top} = \left(\mathbb {E} _ {\mathbf {x} \sim S _ {0}} [ (\sigma (\phi (\mathbf {x}) ^ {\top} \widehat {\theta} _ {t}) - 1) \phi (\mathbf {x}) ]\right) \left(\mathbb {E} _ {\mathbf {x} \sim S _ {0}} [ (\sigma (\phi (\mathbf {x}) ^ {\top} \widehat {\theta} _ {t}) - 1) \phi (\mathbf {x}) ]\right) ^ {\top} \\ \preccurlyeq \mathbb {E} _ {\mathbf {x} \sim S _ {0}} [ (\sigma (\mathbf {x} ^ {\top} \widehat {\boldsymbol {\theta}} _ {t}) - 1) ^ {2} \phi (\mathbf {x}) \phi (\mathbf {x}) ^ {\top} ] \preccurlyeq \mathbb {E} _ {\mathbf {x} \sim S _ {0}} [ \phi (\mathbf {x}) \phi (\mathbf {x}) ^ {\top} ]. \\ \end{array}
$$
In above the first inequality is due to the fact that  $\mathbb{E}[\mathbf{a}]\mathbb{E}[\mathbf{a}]^{\top}\preccurlyeq \mathbb{E}[\mathbf{aa}^{\top}]$  for any random vector  $\mathbf{a}\in \mathbb{R}^d$ . The second inequality comes from  $(\sigma (\mathbf{x}^{\top}\widehat{\theta}_t) - 1)^2\leq 1$  since the output value of the sigmoid function  $\sigma$  is bounded in  $(0,1)$ . A similar arguments show  $\mathbf{b}_t\mathbf{b}_t^\top \leq \mathbb{E}_{\mathbf{x}\sim S_t}[\phi (\mathbf{x})\phi (\mathbf{x})^\top ]$ .
Combining it with above displayed inequalities completes the proof.
Lemma 7. The predictions returned by the ONS algorithm satisfies,
$$
\sum_ {t \in I} \| \nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}) \| _ {A _ {t} ^ {- 1}} ^ {2} \leq d \log \left(1 + \frac {| I | R ^ {2}}{\lambda}\right).
$$
Proof of Lemma 7. The proof follows the standard arguments in the analysis of the online Newton step algorithm [13, Theorem 4.3]. We present the proof here for self-containedness.
$$
\begin{array}{l} \sum_ {t \in I} \| \nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}) \| _ {A _ {t} ^ {- 1}} ^ {2} = \sum_ {t \in I} \nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}) ^ {\top} A _ {t} ^ {- 1} \nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}) \\ = \sum_ {t \in I} \operatorname {t r a c e} \left(A _ {t} ^ {- 1} \nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}) \nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}) ^ {\top}\right) \\ = \sum_ {t \in I} \operatorname {t r a c e} \left(A _ {t} ^ {- 1} \left(A _ {t} - A _ {t - 1}\right)\right) \\ \leq \sum_ {t \in I} \log \frac {| A _ {t} |}{| A _ {t - 1} |} = \log \frac {| A _ {e} |}{| A _ {s} |}, \\ \end{array}
$$
where the last inequality is due to the fact that  $\mathrm{trace}(A^{-1}(A - B))\leq \log (|A| / |B|)$  for any positive definite matrix satisfying  $A\succcurlyeq B\succ 0$  [13, Lemma 4.5].
By definition, we have  $A_{s} = \lambda I_{d}\in \mathbb{R}^{d\times d}$  and we have  $A_{e} = \lambda I_{d} + \sum_{t\in I}\nabla \widehat{L}_{t}(\widehat{\theta}_{t})\nabla \widehat{L}_{t}(\widehat{\theta}_{t})^{\top}$ . Furthermore, since  $\| \nabla \widehat{L}_t(\widehat{\theta}_t)\| _2\leq R$ , we have  $|A_e|\leq (|I|R^2 +\lambda)^d$ . In such a case, we can bound the above inequality by
$$
\sum_ {t \in I} \| \nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}) \| _ {A _ {t} ^ {- 1}} ^ {2} \leq \log \frac {| A _ {e} |}{| A _ {s} |} \leq \log \frac {(| I | R ^ {2} + \lambda) ^ {d}}{\lambda^ {d}} \leq d \log \left(1 + \frac {| I | R ^ {2}}{\lambda}\right)
$$
and finish the proof.
Lemma 8. Let  $Z_{t} = \langle \nabla \widetilde{L}_{t}(\widehat{\theta}_{t}) - \nabla \widehat{L}_{t}(\widehat{\theta}_{t}),\widehat{\theta}_{t} - \theta \rangle$  . Then, with probability at least  $1 - \delta$  , we have
$$
\begin{array}{l} \sum_ {t \in I} Z _ {t} \leq 2 \nu \sum_ {t \in I} \left(\nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}) ^ {\top} (\widehat {\theta} _ {t} - \theta)\right) ^ {2} + 2 \nu \sum_ {t \in I} \mathbb {E} _ {t} \left[ \left(\nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}) ^ {\top} (\widehat {\theta} _ {t} - \theta)\right) ^ {2} \right] \\ + \frac {1}{\nu} \log \frac {\sqrt {2 | I | + 1}}{\delta} + 4 S R \sqrt {\log \frac {2 | I | + 1}{\delta^ {2}}} \\ \end{array}
$$
for any  $\delta \in (0,1)$  and  $\nu > 0$ .
Proof of Lemma 8. We can check that  $\{Z_t\}_{t=s}^e$  is a martingale difference sequence such that  $\mathbb{E}_t[Z_t] = 0$  since  $\widehat{\theta}_t$  is independent of  $S_t$ . Further noting that  $|Z_t| \leq \| \nabla \widetilde{L}_t(\widehat{\theta}_t) - \nabla \widehat{L}_t(\widehat{\theta}_t)\|_2\|\widehat{\theta}_t - \theta\|_2 \leq 4SR$ , a direct application of Lemma 17 shows that
$$
\sum_ {t \in I} Z _ {t} \leq \nu \sum_ {t \in I} Z _ {t} ^ {2} + \nu \sum_ {t \in I} \mathbb {E} _ {t} \left[ Z _ {t} ^ {2} \right] + \frac {1}{\nu} \log \frac {\sqrt {2 | I | + 1}}{\delta} + 4 S R \sqrt {\log \frac {2 | I | + 1}{\delta^ {2}}}, \tag {40}
$$
with probability at least  $1 - \delta$  for any  $\nu > 0$ . Then, for any  $t \in [T]$ , we can further bound the term  $Z_t^2$  and  $\mathbb{E}_t[Z_t^2]$  by
$$
Z _ {t} ^ {2} = \left\langle \nabla \widetilde {L} _ {t} \left(\widehat {\theta} _ {t}\right) - \nabla \widehat {L} _ {t} \left(\widehat {\theta} _ {t}\right), \widehat {\theta} _ {t} - \theta \right\rangle^ {2} \leq 2 \left\langle \nabla \widetilde {L} _ {t} \left(\widehat {\theta} _ {t}\right), \widehat {\theta} _ {t} - \theta \right\rangle^ {2} + 2 \left\langle \nabla \widehat {L} _ {t} \left(\widehat {\theta} _ {t}\right), \widehat {\theta} _ {t} - \theta \right\rangle^ {2} \tag {41}
$$
$$
\mathbb {E} _ {t} [ Z _ {t} ^ {2} ] = \mathbb {E} _ {t} [ \langle \nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}), \widehat {\theta} _ {t} - \theta \rangle^ {2} ] - \langle \nabla \widetilde {L} _ {t} (\widehat {\theta} _ {t}), \widehat {\theta} _ {t} - \theta \rangle^ {2},
$$
where the first inequality is due to the fact  $(a - b)^2 \leq 2a^2 + 2b^2$  for any  $a, b \in \mathbb{R}$ . Then, combining the above two displayed inequalities and taking the summation over  $T$  iterations, we have
$$
\sum_ {t \in I} Z _ {t} ^ {2} + \sum_ {t \in I} \mathbb {E} _ {t} \left[ Z _ {t} ^ {2} \right] \leq \sum_ {t \in I} Z _ {t} ^ {2} + 2 \sum_ {t \in I} \mathbb {E} _ {t} \left[ Z _ {t} ^ {2} \right] \leq 2 \left\langle \nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}), \widehat {\theta} _ {t} - \theta \right\rangle^ {2} + 2 \mathbb {E} _ {t} \left[ \left\langle \nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}), \widehat {\theta} _ {t} - \theta \right\rangle^ {2} \right]. \tag {42}
$$
We complete the proof by plugging (42) into (40).
Lemma 9. With probability at least  $1 - \delta$ , we have
$$
\frac {3}{4} \sum_ {t \in I} (\nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}) ^ {\top} (\widehat {\theta} _ {t} - \theta)) ^ {2} - \frac {5}{4} \sum_ {t \in I} \mathbb {E} _ {t} \left[ (\nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}) ^ {\top} (\widehat {\theta} _ {t} - \theta)) ^ {2} \right] \leq \Lambda_ {T} (\delta)
$$
for any  $\delta \in (0,1)$ , where  $\Lambda_I(\delta) = 32S^2 R^2\log \frac{\sqrt{2|I| + 1}}{\delta} +4S^2 R^2\sqrt{\log\frac{2|I| + 1}{\delta^2}}$
Proof of Lemma 9. We can check  $\{Y_t\}_{t=s}^e$  is a martingale difference sequence such that  $|Y_t| \leq 4S^2 R^2$ . As a consequence, a direct application of Lemma 17 implies,
$$
\begin{array}{l} \sum_ {t \in I} Y _ {t} = \sum_ {t \in I} (\nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}) ^ {\top} (\widehat {\theta} _ {t} - \theta)) ^ {2} - \sum_ {t \in I} \mathbb {E} _ {t} \left[ (\nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}) ^ {\top} (\widehat {\theta} _ {t} - \theta)) ^ {2} \right] \\ \leq 2 \nu \sum_ {t \in I} \left(\left(\nabla \widehat {L} _ {t} \left(\widehat {\theta} _ {t}\right) ^ {\top} \left(\widehat {\theta} _ {t} - \theta\right)\right) ^ {4} + \mathbb {E} _ {t} \left[ \left(\nabla \widehat {L} _ {t} \left(\widehat {\theta} _ {t}\right) ^ {\top} \left(\widehat {\theta} _ {t} - \theta\right)\right) ^ {4} \right]\right) \\ + \frac {1}{\nu} \log \frac {\sqrt {2 | I | + 1}}{\delta} + 4 S ^ {2} R ^ {2} \sqrt {\log \frac {2 | I | + 1}{\delta^ {2}}} \\ \leq \frac {1}{4} \sum_ {t \in I} \left(\left(\nabla \widehat {L} _ {t} \left(\widehat {\theta} _ {t}\right) ^ {\top} \left(\widehat {\theta} _ {t} - \theta\right)\right) ^ {2} + \mathbb {E} _ {t} \left[ \left(\nabla \widehat {L} _ {t} \left(\widehat {\theta} _ {t}\right) ^ {\top} \left(\widehat {\theta} _ {t} - \theta\right)\right) ^ {2} \right]\right) \\ + 3 2 S ^ {2} R ^ {2} \log \frac {\sqrt {2 | I | + 1}}{\delta} + 4 S ^ {2} R ^ {2} \sqrt {\log \frac {2 | I | + 1}{\delta^ {2}}}, \\ \end{array}
$$
where the first inequality follows the proof of Lemma 8 and the second inequality is due to the parameter setting  $\nu = 1 / (32S^2 R^2)$  and the condition that  $(\nabla \widehat{L}_t(\widehat{\theta}_t)^\top (\widehat{\theta}_t - \theta))^2\leq 4S^2 R^2$ . Rearranging the above inequality, we have
$$
\begin{array}{l} \frac {3}{4} \sum_ {t \in I} (\nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}) ^ {\top} (\widehat {\theta} _ {t} - \theta)) ^ {2} - \frac {5}{4} \sum_ {t \in I} \mathbb {E} _ {t} \left[ (\nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}) ^ {\top} (\widehat {\theta} _ {t} - \theta)) ^ {2} \right] \\ \leq 3 2 S ^ {2} R ^ {2} \log \frac {\sqrt {2 | I | + 1}}{\delta} + 4 S ^ {2} R ^ {2} \sqrt {\log \frac {2 | I | + 1}{\delta^ {2}}}, \\ \end{array}
$$
which completes the proof.
D.3 Regret Guarantee for Meta-Algorithm (Lemma 10)
We have the following guarantee on the meta-algorithm, which ensures that for any interval  $I_i \in \mathcal{C}$ , the final prediction is comparable with the prediction of the base-learner  $\mathcal{E}_i$  with an  $\mathcal{O}(\log T)$  cost.
Lemma 10. For any interval  $I_{i} \in \mathcal{C}$  and  $\delta \in (0,1)$ , with probability at least  $1 - \delta$ , the final prediction returned by our meta-learner satisfies
$$
\sum_ {t \in I _ {i}} \widetilde {L} _ {t} (\widehat {\theta} _ {t}) - \sum_ {t \in I _ {i}} \widetilde {L} _ {t} (\widehat {\theta} _ {t, i}) \leq \mathcal {O} \left(\beta \log (T / \delta) + \beta \ln K\right),
$$
where  $K = |\mathcal{C}|$  is the number of the intervals contained in the set  $\mathcal{C}$ .
D.3.1 Main Proof
This part presents the proof of Lemma 10. Our analysis for the meta-learner is based on that of the strongly adaptive algorithm [70], where the problem of tracking base-learners on the corresponding intervals is converted to a sleeping expert problem [71]. The main challenge of our problem is that the feedback function is noisy. Thus, we require to provide tight analysis on the generalization gap between  $\widehat{L}_t$  and  $\widetilde{L}_t$  without ruining the  $\mathcal{O}(\log T)$  regret guarantee. By twisting the meta-algorithm from Hedge [42] used in the previous work to Adapt-ML-Prod [43], we show that the generalization gap can be cancelled by the negative term introduced by the exp-concavity of the loss functions (see how to bound term (b) in (48)).
Proof of Lemma 10. To prove Lemma 10, we show that the final prediction  $\widehat{\theta}_t$  generated by rule  $\widehat{\theta}_t = \sum_{\mathcal{E}_i \in \mathcal{A}_t} p_{t,i} \widehat{\theta}_{t,i}$  with the weight update procedure (25), (26) and (27) is identical to an algorithm learning in the standard prediction with expert advice (PEA) setting [72], where all base-learners submit their predictions at each time  $t \in [T]$ .
Lemma 11. For any base-learner  $\mathcal{E}_i$ , let  $\mathcal{E}_i'$  be a surrogate of  $\mathcal{E}_i$ , whose prediction  $\bar{\theta}_{t,i} = \widehat{\theta}_{t,i}$  is the same as that of  $\mathcal{E}_i$  for any  $t \in I_i$  and  $\bar{\theta}_{t,i} = \widehat{\theta}_t$  for other iterations. Then, for any time  $t \in [T]$ , the surrogate meta-learner predicts as
$$
\bar {\theta} _ {t} = \sum_ {i \in [ K ]} \bar {p} _ {t, i} \bar {\theta} _ {t, i} \quad a n d \quad \bar {p} _ {t, i} = \frac {\bar {\varepsilon} _ {t - 1 , i} \bar {v} _ {t - 1 , i}}{\sum_ {i \in [ K ]} \bar {\varepsilon} _ {t - 1 , i} \bar {v} _ {t - 1 , i}}, \tag {43}
$$
where the potential  $\bar{v}_{t,i}$  and the learning rate  $\bar{\varepsilon}_{t,i}$  is defined as
$$
\bar {v} _ {t, i} = \left(\bar {v} _ {t - 1, i} \cdot \left(1 + \bar {\varepsilon} _ {t - 1, i} \bar {m} _ {t, i}\right)\right) ^ {\frac {\bar {\varepsilon} _ {t , i}}{\bar {\varepsilon} _ {t - 1 , i}}} f o r a l l t \in [ T ], \tag {44}
$$
$$
\bar {\varepsilon} _ {t, i} = \min  \left\{\frac {1}{2}, \sqrt {\frac {\ln K}{1 + \sum_ {\tau = 1} ^ {t} \bar {m} _ {\tau , j} ^ {2}}} \right\} f o r a l l t \in [ T ] \tag {45}
$$
with  $\bar{m}_{t,i} = \langle \widehat{L}_t(\bar{\theta}_t),\bar{\theta}_t - \bar{\theta}_{t,i}\rangle$ $\bar{v}_{0,i} = 1 / K$  and  $\bar{\varepsilon}_{0,i} = \min \{1 / 2,\ln K\}$  for all  $i\in [K]$ . Then, we have  $\bar{\theta}_t = \widehat{\theta}_t$  for any  $t\in [T]$ .
We have the following guarantee for the final prediction generated by the surrogate algorithm defined by (43).
Lemma 12. With probability at least  $1 - 2\delta$ , for any  $\delta \in (0,1)$ , the prediction returned by the surrogate meta-algorithm defined as (43) ensures,
$$
\sum_ {t = 1} ^ {T} \widetilde {L} _ {t} (\bar {\theta} _ {t}) - \sum_ {t = 1} ^ {T} \widetilde {L} _ {t} (\bar {\theta} _ {t, j}) \leq C _ {T} ^ {(2)} (\delta) = \mathcal {O} \Bigl (\beta \log (T / \delta) + \beta \ln K \Bigr)
$$
for any base-algorithm  $\mathcal{E}_i^\prime$ ,  $i \in [K]$ , where  $C_T^{(2)}(\delta) = \frac{16(1 + \beta)S^2R^2(C_T')^2}{\ln K} + \frac{1}{6(1 + \beta)} + 2C_T' + \left(24(1 + \beta) + \frac{11S^2R^2}{1 + \beta}\right)\log \frac{\sqrt{2T + 1}}{\delta} + \left(4SR + \frac{S^2R^2}{2(1 + \beta)}\right)\sqrt{\log\frac{2T + 1}{\delta^2}}$  and  $C_T' = 3\ln K + \ln \left(1 + \frac{K}{2e}(1 + \ln (T + 1))\right) = O(\ln K + \ln \ln T)$ .
Then, according to the relationship between  $\mathcal{E}_i$  and  $\mathcal{E}_i'$ , we can bound the regret of the final prediction  $\widehat{\theta}_t$  with respect to that of any base-learner  $\mathcal{E}_i$  over the interval  $I_i$  by