SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Since the bounded random variable  $\Phi_1^{(i)}$  is sub-Gaussian with parameter  $\sigma(\Phi_1^{(i)})$ , then the error in the mean vector estimate  $z_1^{-(i)}$  is sub-Gaussian with parameter that satisfies
$$
\left(\eta_ {1} ^ {\rightarrow^ {(i)}}\right) ^ {2} = \frac {\sigma \left(\Phi_ {1} ^ {(i)}\right) ^ {2}}{n _ {1}} \leq \frac {\kappa^ {2} \sigma_ {1} ^ {2 ^ {(i)}}}{n _ {1}} = \kappa^ {2} s _ {1} ^ {(i)}.
$$
If  $z_{j-1}^{(i)} = \tau_{j-1}^{\infty(i)} - \tau_{j-1}^{(i)}$  is sub-Gaussian with parameter  $\eta_{j-1}^{(i)} \leq \kappa \sqrt{s_{j-1}^{(i)}}$  for any  $i = 1, 2, \ldots, m$ , then using the recursions in (6) we have that
$$
\begin{array}{l} z _ {j} ^ {\rightharpoonup (i)} = \tau_ {j} ^ {\infty (i)} - \tau_ {j} ^ {\rightharpoonup (i)} = \tau_ {j - 1} ^ {\infty (i)} + w _ {j} ^ {(i)} - \tau_ {j} ^ {(i)} - \frac {s _ {j} ^ {(i)}}{s _ {j - 1} ^ {\rightharpoonup (i)} + s _ {j} ^ {(i)} + d _ {j} ^ {2 (i)}} \left(\tau_ {j - 1} ^ {\rightharpoonup (i)} - \tau_ {j} ^ {(i)}\right) \\ = \tau_ {j - 1} ^ {\infty} ^ {(i)} + w _ {j} ^ {(i)} - \tau_ {j - 1} ^ {\rightharpoonup} ^ {(i)} + \left(1 - \frac {s _ {j} ^ {(i)}}{s _ {j - 1} ^ {\rightharpoonup} ^ {(i)} + s _ {j} ^ {(i)} + d _ {j} ^ {2 ^ {(i)}}}\right)\left(\tau_ {j - 1} ^ {\rightharpoonup} ^ {(i)} - \tau_ {j} ^ {(i)}\right) \\ = \tau_ {j - 1} ^ {\infty} ^ {(i)} + w _ {j} ^ {(i)} - \tau_ {j - 1} ^ {\rightarrow (i)} - \frac {s _ {j} ^ {\rightarrow (i)}}{s _ {j} ^ {(i)}} \left(\tau_ {j} ^ {(i)} - \tau_ {j - 1} ^ {\rightarrow (i)}\right) \\ \end{array}
$$
since  $\pmb{w}_j = \pmb{\tau}_j^\infty -\pmb{\tau}_{j - 1}^\infty$  . If  $\pmb {v}_j = \pmb {\tau}_j - \pmb {\tau}_j^\infty$  , the error in the mean vector estimate is given by
$$
\begin{array}{l} z _ {j} ^ {\rightharpoonup (i)} = \tau_ {j - 1} ^ {\infty} ^ {(i)} + w _ {j} ^ {(i)} - \tau_ {j - 1} ^ {\rightharpoonup (i)} - \frac {s _ {j} ^ {\rightharpoonup (i)}}{s _ {j} ^ {(i)}} \left(\tau_ {j} ^ {\infty (i)} + v _ {j} ^ {(i)} - \tau_ {j - 1} ^ {\rightharpoonup (i)}\right) \\ = \tau_ {j - 1} ^ {\infty} ^ {(i)} + w _ {j} ^ {(i)} - \tau_ {j - 1} ^ {\rightarrow (i)} - \frac {s _ {j} ^ {\rightarrow (i)}}{s _ {j} ^ {(i)}} \left(\tau_ {j - 1} ^ {\infty} ^ {(i)} + w _ {j} ^ {(i)} + v _ {j} ^ {(i)} - \tau_ {j - 1} ^ {\rightarrow (i)}\right) \\ = \left(1 - \frac {s _ {j} ^ {\rightarrow (i)}}{s _ {j} ^ {(i)}}\right) z _ {j - 1} ^ {\rightarrow (i)} + \left(1 - \frac {s _ {j} ^ {\rightarrow (i)}}{s _ {j} ^ {(i)}}\right)\left(w _ {j} ^ {(i)}\right) - \frac {s _ {j} ^ {\rightarrow (i)}}{s _ {j} ^ {(i)}} v _ {j} ^ {(i)} \\ \end{array}
$$
where  $w_{j}^{(i)}$  and  $v_{j}^{(i)}$  are sub-Gaussian with parameter  $\sigma(w_{j}^{(i)})$  and  $\sigma\left(\Phi_{j}^{(i)}\right) / \sqrt{n_{j}}$ , respectively. Therefore, we have that  $z_{j}^{\rightarrow (i)}$  is sub-Gaussian with parameter  $\eta_{j}^{\rightarrow (i)}$  that satisfies
$$
\left(\eta_ {j} ^ {\rightarrow (i)}\right) ^ {2} = \left(1 - \frac {s _ {j} ^ {\rightarrow (i)}}{s _ {j} ^ {(i)}}\right) ^ {2} \left(\eta_ {j - 1} ^ {\rightarrow (i)}\right) ^ {2} + \left(1 - \frac {s _ {j} ^ {\rightarrow (i)}}{s _ {j} ^ {(i)}}\right) ^ {2} \sigma \left(w _ {j} ^ {(i)}\right) ^ {2} + \left(\frac {s _ {j} ^ {\rightarrow (i)}}{s _ {j} ^ {(i)}}\right) ^ {2} \frac {\sigma \left(\Phi_ {j} ^ {(i)}\right) ^ {2}}{n _ {j}}
$$
since  $\pmb{z}_{j-1}^{\rightarrow}, \pmb{w}_j$ , and  $\pmb{v}_j$  are independent. Using that  $\eta_{j-1}^{(i)} \leq \kappa \sqrt{s_{j-1}^{(i)}}$  and the definition of  $\kappa$ , we have that
$$
\begin{array}{l} \left(\eta_ {j} ^ {\rightharpoonup (i)}\right) ^ {2} \leq \left(1 - \frac {s _ {j} ^ {\rightharpoonup (i)}}{s _ {j} ^ {(i)}}\right) ^ {2} \kappa^ {2} s _ {j - 1} ^ {\rightharpoonup (i)} + \left(1 - \frac {s _ {j} ^ {\rightharpoonup (i)}}{s _ {j} ^ {(i)}}\right) ^ {2} \kappa^ {2} d _ {j} ^ {2 (i)} + \left(\frac {s _ {j} ^ {\rightharpoonup (i)}}{s _ {j} ^ {(i)}}\right) ^ {2} \kappa^ {2} \frac {\sigma_ {j} ^ {2 (i)}}{n _ {j}} \\ \leq \left(1 - \frac {s _ {j} ^ {\neg (i)}}{s _ {j} ^ {(i)}}\right) ^ {2} \kappa^ {2} \left(\left(\frac {1}{s _ {j} ^ {\neg (i)}} - \frac {1}{s _ {j} ^ {(i)}}\right) ^ {- 1} + d _ {j} ^ {2 (i)}\right) + \frac {\left(s _ {j} ^ {\neg (i)}\right) ^ {2}}{s _ {j} ^ {(i)}} \kappa^ {2} \tag {16} \\ = \left(1 - \frac {s _ {j} ^ {\rightarrow (i)}}{s _ {j} ^ {(i)}}\right) \kappa^ {2} s _ {j} ^ {\rightarrow (i)} + \kappa^ {2} \frac {\left(s _ {j} ^ {\rightarrow (i)}\right) ^ {2}}{s _ {j} ^ {(i)}} \\ \end{array}
$$
where (16) is obtained using the second recursion in (6).
The inequality in (15) is obtained using the union bound together with the Chernoff bound (concentration inequality) [31] for the random variables  $z_{j}^{\rightarrow (i)}$  that are sub-Gaussian with parameter  $\eta_j^{\rightarrow (i)}$ .
Now, we prove by induction that, for any  $j$ ,  $\| \sqrt{s_{j}^{-}} \|_{\infty} \leq M / \sqrt{n_{j}^{-}}$  where the ESSs satisfy  $n_1^{\neg} = n_1$  and  $n_{j}^{\neg} \geq n_{j} + n_{j-1}^{\neg} \frac{\|\pmb{\sigma}_{j}^{2}\|_{\infty}}{\|\pmb{\sigma}_{j}^{2}\|_{\infty} + n_{j-1}^{\neg}\|\pmb{d}_{j}^{2}\|_{\infty}}$  for  $j \geq 2$ . For  $j = 1$ , using the definition of  $s_{j}^{-}$  in the
second recursion in (6), we have that for any component  $i$
$$
\left(s _ {1} ^ {\rightarrow (i)}\right) ^ {- 1} = \left(s _ {1} ^ {(i)}\right) ^ {- 1} = \frac {n _ {1}}{\sigma_ {1} ^ {2 (i)}} \geq \frac {n _ {1}}{M ^ {2}}.
$$
Then, vector  $s_1^{\rightarrow}$  satisfies
$$
\| \sqrt {\boldsymbol {s} _ {1} ^ {\rightarrow}} \| _ {\infty} \leq \frac {M}{\sqrt {n _ {1}}} = \frac {M}{\sqrt {n _ {1} ^ {\rightarrow}}}.
$$
If  $\| \sqrt{s_{j - 1}^{\rightarrow}}\|_{\infty}\leq M / \sqrt{n_{j - 1}^{\rightarrow}}$  then we have that for any component  $i$
$$
\begin{array}{l} \left(s _ {j} ^ {\rightarrow (i)}\right) ^ {- 1} = \frac {1}{s _ {j} ^ {(i)}} + \frac {1}{s _ {j - 1} ^ {\rightarrow (i)} + d _ {j} ^ {2 (i)}} \geq \frac {1}{s _ {j} ^ {(i)}} + \frac {1}{\frac {M ^ {2}}{n _ {j - 1}} + d _ {j} ^ {2 (i)}} \geq \frac {1}{M ^ {2}} \left(n _ {j} + \frac {1}{\frac {1}{n _ {j - 1}} + \frac {d _ {j} ^ {2 (i)}}{M ^ {2}}}\right) \\ \geq \frac {1}{M ^ {2}} \left(n _ {j} + \frac {1}{\frac {1}{n _ {j - 1} ^ {2}} + \frac {\| \boldsymbol {d} _ {j} ^ {2} \| _ {\infty}}{\| \boldsymbol {\sigma} _ {j} ^ {2} \| _ {\infty}}}\right) \\ \end{array}
$$
by using the second recursion in (6) and the induction hypothesis. Then, vector  $s_j$  satisfies
$$
\left\| \sqrt {\boldsymbol {s} _ {j} ^ {\rightarrow}} \right\| _ {\infty} \leq \frac {M}{\sqrt {n _ {j} + n _ {j - 1} ^ {\rightarrow} \frac {\left\| \boldsymbol {\sigma} _ {j} ^ {2} \right\| _ {\infty}}{\left\| \boldsymbol {\sigma} _ {j} ^ {2} \right\| _ {\infty} + n _ {j - 1} ^ {\rightarrow} \left\| \boldsymbol {d} _ {j} ^ {2} \right\| _ {\infty}}}}. \tag {17}
$$
The inequality in (8) is obtained because the minimax risk is bounded by the smallest minimax risk as shown in [24, 25, 30] so that
$$
R \left(\mathcal {U} _ {j} ^ {\rightharpoonup}\right) \leq R _ {j} ^ {\infty} + \left(\left\| \boldsymbol {\tau} _ {j} ^ {\infty} - \boldsymbol {\tau} _ {j} ^ {\rightharpoonup} \right\| _ {\infty} + \left\| \boldsymbol {\lambda} _ {j} ^ {\rightharpoonup} \right\| _ {\infty}\right)\left\| \boldsymbol {\mu} _ {j} ^ {\infty} \right\| _ {1}
$$
that leads to (8) using (15), (17), and the fact that  $1 \leq \sqrt{2 \log \left( \frac{2m}{\delta} \right)}$ .
Proof. To obtain bound in (9), we proceed by induction. For  $j = 1$ , using the expression for the ESS in (8), we have that
$$
n _ {1} ^ {\leftarrow} = n _ {1} \geq n.
$$
If (9) holds for the  $(j - 1)$ -task, then for the  $j$ -th task, we have that
$$
n _ {j} ^ {\rightarrow} \geq n _ {j} + n _ {j - 1} ^ {\rightarrow} \frac {\| \pmb {\sigma} _ {j} ^ {2} \| _ {\infty}}{\| \pmb {\sigma} _ {j} ^ {2} \| _ {\infty} + n _ {j - 1} ^ {\rightarrow} \| \pmb {d} _ {j} ^ {2} \| _ {\infty}} \geq n + n _ {j - 1} ^ {\rightarrow} \frac {1}{1 + n _ {j - 1} ^ {\rightarrow} d ^ {2}} = n \left(1 + \frac {1}{\frac {n}{n _ {j - 1} ^ {\rightarrow}} + n d ^ {2}}\right)
$$
where the second inequality is obtained because  $n_j \geq n$ ,  $\|\pmb{\sigma}_j^2\|_{\infty} \leq 1$ , and  $\|\pmb{d}_j^2\|_{\infty} \leq d^2$ . Using that  $n_{j-1}^{\rightarrow} \geq n\left(1 + \frac{(1+\alpha)^{2j-3}-1-\alpha}{\alpha(1+\alpha)^{2j-3}+\alpha}\right)$ , the ESS of the  $j$ -th task satisfies
$$
\begin{array}{l} n _ {j} ^ {\rightarrow} \geq n \left(1 + \frac {1}{\frac {n}{n \left(1 + \frac {(1 + \alpha) ^ {2 j - 3} - 1 - \alpha}{\alpha (1 + \alpha) ^ {2 j - 3} + \alpha}\right)} + n d ^ {2}}\right) = n \left(1 + \frac {1}{\frac {\alpha (1 + \alpha) ^ {2 j - 3} + \alpha}{(1 + \alpha) ^ {2 j - 2} - 1} + n d ^ {2}}\right) \\ = n \left(1 + \frac {1}{\frac {\alpha (1 + \alpha) ^ {2 j - 3} + \alpha}{(1 + \alpha) ^ {2 j - 2} - 1} + \frac {\alpha^ {2}}{\alpha + 1}}\right) \tag {18} \\ = n \left(1 + \frac {(1 + \alpha) ^ {2 j - 1} - 1 - \alpha}{\alpha (1 + \alpha) ^ {2 j - 2} + \alpha (\alpha + 1) + \alpha^ {2} (1 + \alpha) ^ {2 j - 2} - \alpha^ {2}}\right) \\ \end{array}
$$
where (18) is obtained because  $nd^2 = \frac{\alpha^2}{\alpha + 1}$  since  $\alpha = \frac{nd^2}{2}\left(\sqrt{1 + \frac{4}{nd^2}} + 1\right)$ .
Now, we obtain bounds for the ESS if  $nd^2 < \frac{1}{j^2}$ . In the following, the constant  $\phi$  represents the golden ratio  $\phi = 1.618 \ldots$
If  $nd^2 < \frac{1}{j^2} \Rightarrow \sqrt{nd^2} \leq \alpha \leq \sqrt{nd^2}\phi \leq \frac{\phi}{j} \leq 1$  because  $\alpha = \frac{nd^2}{2}\left(\sqrt{1 + \frac{4}{nd^2}} + 1\right) = \sqrt{nd^2}\frac{\sqrt{nd^2 + 4} + \sqrt{nd^2}}{2}$ , then we have that  $n_j \rightarrow \infty$  satisfies
$$
n _ {j} ^ {\rightarrow} \geq n \left(1 + \frac {1}{\alpha} \frac {\alpha (2 j - 2)}{2 + \alpha (2 j - 1)}\right) = n \left(1 + \frac {2 j - 2}{2 + \alpha (2 j - 1)}\right)
$$
where the first inequality follows because  $(1 + \alpha)^{2j - 2} \geq 1 + \alpha (2j - 2)$ . Using  $\alpha \leq \frac{\phi}{j}$ , we have that
$$
n _ {j} ^ {\rightarrow} \geq n \left(1 + \frac {2 j - 2}{2 + \frac {\phi}{j} (2 j - 1)}\right) \geq n \left(1 + \frac {2 j - 2}{2 + 2 \phi - \frac {\phi}{j}}\right) \geq n \left(1 + \frac {j - 1}{1 + \phi}\right).
$$
E More efficient recursions for forward and backward learning
The Rauch-Tung-Striebel smoother recursions [34] allow to obtain forward and backward mean and MSE vectors directly from those vectors for the succeeding task. Specifically, for each  $j$ -th task, the mean vector  $\tau_{j}^{\overline{=} k}$  together with the MSE vector  $s_{j}^{\overline{=} k}$  can be obtained using those vectors for the succeeding task  $\tau_{j+1}^{\overline{=} k}, s_{j+1}^{\overline{=} k}$  as
$$
\boldsymbol {\tau} _ {j} ^ {\rightleftarrows k} = \boldsymbol {\tau} _ {j} ^ {\rightleftarrows} + \frac {\boldsymbol {s} _ {j} ^ {\rightleftarrows}}{\boldsymbol {s} _ {j} ^ {\rightleftarrows} + \boldsymbol {d} _ {j + 1} ^ {2}} \left(\boldsymbol {\tau} _ {j + 1} ^ {\rightleftarrows k} - \boldsymbol {\tau} _ {j} ^ {\rightleftarrows}\right)
$$
$$
\boldsymbol {s} _ {j} ^ {\rightleftharpoons k} = \left(\frac {1}{\boldsymbol {s} _ {j} ^ {\rightleftharpoonup}} + \left(\boldsymbol {d} _ {j + 1} ^ {2} + \left(\frac {\boldsymbol {1}}{\boldsymbol {s} _ {j + 1} ^ {\rightleftharpoonup k}} - \frac {1}{\boldsymbol {s} _ {j} ^ {\rightleftharpoonup} + \boldsymbol {d} _ {j + 1} ^ {2}}\right) ^ {- 1}\right) ^ {- 1}\right) ^ {- 1}.
$$
The above recursions provide the same mean vector estimate as the recursions in (10) in the paper since they are obtained using the Rauch-Tung-Striebel smoother recursions instead of fixed-lag smoother recursions [34].
Proof. To obtain bound in (11) we first prove that the mean vector estimate and the MSE vector given by (10) satisfy
$$
\mathbb {P} \left\{\left| \tau_ {k} ^ {\infty (i)} - \tau_ {j} ^ {\overline {{=}} k (i)} \right| \leq \kappa \sqrt {2 s _ {j} ^ {\overline {{=}} k (i)} \log \left(\frac {2 m}{\delta}\right)} \right\} \geq (1 - \delta) \tag {19}
$$
for any component  $i = 1,2,\ldots ,m$ . Then, we prove that  $\| s_j^{\overline{= k}}\|_{\infty}\leq M / \sqrt{n_j^{\overline{= k}}}$  for  $j\in \{1,2,\dots ,k\}$ , where the ESSs satisfy  $n_k^{\overline{= k}} = n_k^{-}$  and  $n_j^{\overline{= k}}\geq n_j^{-} + n_{j + 1}^{-k}\frac{\|\sigma_j^2\|_{\infty}}{\|\sigma_j^2\|_{\infty} + n_{j + 1}^{-k}\|d_{j + 1}^2\|_{\infty}}$  for  $j\geq 2$ .
To obtain inequality (19), we prove that each component  $i = 1,2,\ldots ,m$  of the error in the mean vector estimate  $z_{j}^{\overline{= k}^{(i)}} = \tau_{j}^{\infty (i)} - \tau_{j}^{\overline{= k}^{(i)}}$  is sub-Gaussian with parameter  $\eta_j^{\overline{= k}^{(i)}}\leq \kappa \sqrt{s_j^{\overline{= k}^{(i)}}}$ . Analogously to the proof of Theorem 1, it is proven that each component in the error of the backward mean vector  $\pmb{\tau}_{j + 1}^{-k}$  is sub-Gaussian with parameters satisfying  $\pmb{\eta}_{j + 1}^{-k}\preceq \kappa \sqrt{\pmb{s}_{j + 1}^{-k}}$ . The error in the forward and backward mean vector estimate is given by
$$
z _ {j} ^ {\overleftarrow {=} k ^ {(i)}} = \tau_ {j} ^ {\infty^ {(i)}} - \tau_ {j} ^ {\overleftarrow {=} k ^ {(i)}} = \tau_ {j} ^ {\infty^ {(i)}} - \tau_ {j} ^ {\rightarrow^ {(i)}} - \frac {s _ {j} ^ {\rightarrow^ {(i)}}}{s _ {j} ^ {\rightarrow^ {(i)}} + s _ {j + 1} ^ {\rightarrow^ {(k)} (i)} + d _ {j + 1} ^ {2 (i)}} \left(\tau_ {j + 1} ^ {\overleftarrow {=} k ^ {(i)}} - \tau_ {j} ^ {\overleftarrow {=} (i)}\right)
$$
where the second equality is obtained using the recursion for  $\tau_{j}^{\equiv k^{(i)}}$  in (10). Adding and subtracting  $\frac{s_j^{-(i)}}{s_j^{-(i)} + s_{j + 1}^{-k^{(i)}} + d_{j + 1}^2^{(i)}}\tau_{j + 1}^{\infty^{(i)}}$ , we have that
$$
\begin{array}{l} z _ {j} ^ {\overleftarrow {=} k ^ {(i)}} = z _ {j} ^ {\overleftarrow {-} (i)} - \frac {s _ {j} ^ {\overrightarrow {-} (i)}}{s _ {j} ^ {\overrightarrow {-} (i)} + s _ {j + 1} ^ {\overleftarrow {-} k ^ {(i)}} + d _ {j + 1} ^ {2 ^ {(i)}}} \left(\tau_ {j + 1} ^ {\infty^ {(i)}} - \tau_ {j + 1} ^ {\infty^ {(i)}} + \tau_ {j + 1} ^ {\overleftarrow {-} k ^ {(i)}} - \tau_ {j} ^ {\overleftarrow {-} (i)}\right) \\ = z _ {j} ^ {\rightarrow (i)} - \frac {s _ {j} ^ {\rightarrow (i)}}{s _ {j} ^ {\rightarrow (i)} + s _ {j + 1} ^ {\rightarrow k (i)} + d _ {j + 1} ^ {2 (i)}} \left(\tau_ {j} ^ {\infty (i)} + w _ {j + 1} ^ {(i)} - z _ {j + 1} ^ {\rightarrow k (i)} - \tau_ {j} ^ {\rightarrow (i)}\right) \\ \end{array}
$$
since  $\pmb{w}_j = \pmb{\tau}_j^\infty -\pmb{\tau}_{j - 1}^\infty$  and  $z_{j}^{\neg (i)} = \tau_{j}^{\infty (i)} - \tau_{j}^{\neg (i)}$ . Then, we have that
$$
\begin{array}{l} z _ {j} ^ {\rightarrow k (i)} = z _ {j} ^ {\rightarrow (i)} - \frac {s _ {j} ^ {\rightarrow (i)}}{s _ {j} ^ {\rightarrow (i)} + s _ {j + 1} ^ {\rightarrow k (i)} + d _ {j + 1} ^ {2 (i)}} \left(z _ {j} ^ {\rightarrow (i)} + w _ {j + 1} ^ {(i)} - z _ {j + 1} ^ {\rightarrow k (i)}\right) \tag {20} \\ = \left(1 - \frac {s _ {j} ^ {\rightarrow (i)}}{s _ {j} ^ {\rightarrow (i)} + s _ {j + 1} ^ {\rightarrow k (i)} + d _ {j + 1} ^ {2 (i)}}\right) z _ {j} ^ {\rightarrow (i)} - \frac {s _ {j} ^ {\rightarrow (i)}}{s _ {j} ^ {\rightarrow (i)} + s _ {j + 1} ^ {\rightarrow k (i)} + d _ {j + 1} ^ {2 (i)}} \left(w _ {j + 1} ^ {(i)} - z _ {j + 1} ^ {\rightarrow k (i)}\right) \\ \end{array}
$$
where  $z_{j}^{\rightarrow (i)}, z_{j+1}^{\rightarrow k(i)}$ , and  $w_{j+1}^{(i)}$  are sub-Gaussian with parameters  $\eta_{j}^{\rightarrow (i)} \leq \kappa \sqrt{s_{j}^{\rightarrow (i)}}, \eta_{j+1}^{\rightarrow k(i)} \leq \kappa \sqrt{s_{j+1}^{\rightarrow k(i)}}$ , and  $\sigma(w_{j}^{(i)})$ , respectively. Since  $z_{j}^{\rightarrow}, z_{j+1}^{\rightarrow k}$ , and  $w_{j+1}$  are independent, we have that  $z_{j}^{\rightarrow k(i)}$  given by (20) is sub-Gaussian with parameter that satisfies
$$
\begin{array}{l} \left(\eta_ {j} ^ {\overleftarrow {\rightleftharpoons} k ^ {(i)}}\right) ^ {2} = \left(1 - \frac {s _ {j} ^ {\overleftarrow {\rightleftharpoons} (i)}}{s _ {j} ^ {\overleftarrow {\rightleftharpoons} (i)} + s _ {j + 1} ^ {\overleftarrow {\rightleftharpoons} k ^ {(i)}} + d _ {j + 1} ^ {2 ^ {(i)}}}\right) ^ {2} \left(\eta_ {j} ^ {\overleftarrow {\rightleftharpoons} (i)}\right) ^ {2} \\ + \left(\frac {s _ {j} ^ {\rightarrow (i)}}{s _ {j} ^ {\rightarrow (i)} + s _ {j + 1} ^ {\leftarrow k (i)} + d _ {j + 1} ^ {2 (i)}}\right) ^ {2} \left(\sigma \left(w _ {j} ^ {(i)}\right) ^ {2} + \left(\eta_ {j + 1} ^ {\leftarrow k (i)}\right) ^ {2}\right) \\ \leq \left(1 - \frac {s _ {j} ^ {\rightarrow (i)}}{s _ {j} ^ {\rightarrow (i)} + s _ {j + 1} ^ {\rightarrow k (i)} + d _ {j + 1} ^ {2 (i)}}\right) ^ {2} \kappa^ {2} s _ {j} ^ {\rightarrow (i)} \\ + \left(\frac {s _ {j} ^ {\rightarrow (i)}}{s _ {j} ^ {\rightarrow (i)} + s _ {j + 1} ^ {\rightarrow k (i)} + d _ {j + 1} ^ {2 (i)}}\right) ^ {2} \kappa^ {2} \left(d _ {j + 1} ^ {(i)} + s _ {j + 1} ^ {\rightarrow k (i)}\right) \\ \end{array}
$$
Using the second recursion in (10) we have that the sub-Gaussian parameter satisfies
$$
\begin{array}{l} \left(\eta_ {j} ^ {\overline {{\rightleftharpoons}} k ^ {(i)}}\right) ^ {2} \leq \left(1 - \frac {s _ {j} ^ {\overline {{\rightleftharpoons}} k ^ {(i)}}}{s _ {j + 1} ^ {\overline {{\rightleftharpoons}} k ^ {(i)}} + d _ {j + 1} ^ {2 ^ {(i)}}}\right) ^ {2} \kappa^ {2} \left(\frac {1}{s _ {j} ^ {\overline {{\rightleftharpoons}} k ^ {(i)}}} - \frac {1}{s _ {j + 1} ^ {\overline {{\rightleftharpoons}} k ^ {2 ^ {(i)}}} + d _ {j + 1} ^ {2 ^ {(i)}}}\right) ^ {- 1} \\ + \frac {\left(s _ {j} ^ {\overleftarrow {k} k ^ {(i)}}\right) ^ {2}}{s _ {j + 1} ^ {- k ^ {(i)}} + d _ {j + 1} ^ {2 ^ {(i)}}} \kappa^ {2} \\ = \left(\frac {s _ {j + 1} ^ {- k (i)} + d _ {j + 1} ^ {2 (i)} - s _ {j} ^ {\overline {{\equiv}} k (i)}}{s _ {j + 1} ^ {- k (i)} + d _ {j + 1} ^ {2 (i)}}\right) \kappa^ {2} s _ {j} ^ {\overline {{\equiv}} k (i)} \\ + \frac {\left(s _ {j} ^ {\rightleftharpoons k ^ {(i)}}\right) ^ {2}}{s _ {j + 1} ^ {\leftarrow k ^ {(i)}} + d _ {j + 1} ^ {2 ^ {(i)}}} \kappa^ {2} = \kappa^ {2} s _ {j} ^ {\rightleftharpoons k ^ {(i)}}. \\ \end{array}
$$
The inequality in (19) is obtained using the union bound together with the Chernoff bound (concentration inequality) [31] for the random variables  $z_{j}^{\overline{= k}^{(i)}}$  that are sub-Gaussian with parameter  $\eta_{\overline{j}}^{\overline{= k}^{(i)}}$ .
Now, we prove that, for any  $j$ ,  $\| \sqrt{\pmb{s}_j^{\overline{\equiv}k}} \| \leq M / \sqrt{n_j^{\overline{\equiv}k}}$  where the ESSs satisfy  $n_k^{\overline{\equiv} k} = n_k^{\overline{\cdot}}$  and  $n_j^{\overline{\equiv} k} \geq n_j^{\overline{\cdot}} + n_{j+1}^{-k} \frac{\|\pmb{\sigma}_j^2\|_\infty}{\|\pmb{\sigma}_j^2\|_\infty + n_{j+1}^{-k}\|\pmb{d}_{j+1}^2\|_\infty}$  for  $j \geq 2$ . Analogously to the proof of Theorem 1, we prove that the backward MSE vector  $\pmb{s}_{j+1}^{\overline{k}}$  satisfies  $\| \sqrt{\pmb{s}_{j+1}^{\overline{k}}} \|_\infty \leq M / \sqrt{n_{j+1}^{\overline{k}}}$ . Then, using that  $\| \sqrt{\pmb{s}_{j+1}^{\overline{k}}} \|_\infty \leq M / \sqrt{n_{j+1}^{\overline{k}}}$ , we have that for every component  $i$
$$
\begin{array}{l} \left(s _ {j} ^ {\rightleftarrows k (i)}\right) ^ {- 1} = \frac {1}{s _ {j} ^ {\rightleftarrows (i)}} + \frac {1}{s _ {j + 1} ^ {\leftarrow k (i)} + d _ {j + 1} ^ {2 (i)}} \geq \frac {n _ {j} ^ {\rightarrow}}{\sigma_ {j} ^ {2 (i)}} + \frac {1}{\frac {M ^ {2}}{n _ {j + 1} ^ {\leftarrow k}} + d _ {j + 1} ^ {2 (i)}} \\ \geq \frac {1}{M ^ {2}} \left(n _ {j} ^ {\rightarrow} + \frac {1}{\frac {1}{n _ {j + 1} ^ {- k}} + \frac {d _ {j + 1} ^ {2}}{M ^ {2}}}\right) \geq \frac {1}{M ^ {2}} \left(n _ {j} ^ {\rightarrow} + \frac {1}{\frac {1}{n _ {j + 1} ^ {- k}} + \frac {\| d _ {j + 1} ^ {2} \| _ {\infty}}{\| \boldsymbol {\sigma} _ {j} ^ {2} \| _ {\infty}}}\right). \\ \end{array}
$$
$$
\left\| \sqrt {\boldsymbol {s} _ {j} ^ {\overleftarrow {=} k}} \right\| _ {\infty} \leq \frac {M}{\sqrt {n _ {j} ^ {\rightarrow} + \frac {1}{\frac {1}{n _ {j + 1} ^ {\rightarrow} k} + \frac {\| d _ {j + 1} ^ {2} \| _ {\infty}}{\| \sigma_ {j} ^ {2} \| _ {\infty}}}}}. \tag {21}
$$
The inequality in (11) is obtained because the minimax risk is bounded by the smallest minimax risk as shown in [24, 25, 30] so that
$$
R \left(\mathcal {U} _ {j} ^ {\rightleftharpoons k}\right) \leq R _ {j} ^ {\infty} + \left(\| \boldsymbol {\tau} _ {j} ^ {\infty} - \boldsymbol {\tau} _ {j} ^ {\rightleftharpoons k} \| _ {\infty} + \| \boldsymbol {\lambda} _ {j} ^ {\rightleftharpoons k} \| _ {\infty}\right)\left\| \boldsymbol {\mu} _ {j} ^ {\infty} \right\| _ {1}
$$
that leads to (11) using (19), (21), and the fact that  $1 \leq \sqrt{2 \log \left( \frac{2m}{\delta} \right)}$ .
G Proof of Theorem 4
Proof. To obtain bound in (12), we use the ESS obtained with forward learning in Theorem 2 and obtained with backward learning. Analogously to the proof of Theorem 2, we prove that the ESS obtained at backward learning satisfies
$$
n _ {j + 1} ^ {\leftarrow k} \geq n _ {j + 1} + n _ {j + 2} ^ {\leftarrow k} \frac {\| \boldsymbol {\sigma} _ {j + 1} ^ {2} \| _ {\infty}}{\| \boldsymbol {\sigma} _ {j + 1} ^ {2} \| _ {\infty} + n _ {j + 2} ^ {\leftarrow k} \| \boldsymbol {d} _ {j + 2} ^ {2} \| _ {\infty}} \geq n \left(1 + \frac {(1 + \alpha) ^ {2 (k - j) - 1} - 1 - \alpha}{\alpha (1 + \alpha) ^ {2 (k - j) - 1} + \alpha}\right).
$$
Therefore, the ESS obtained with forward an backward learning satisfies
$$
\begin{array}{l} n _ {j} ^ {\overrightarrow {= k}} \geq n _ {j} ^ {\overleftarrow {}} + n \left(1 + \frac {(1 + \alpha) ^ {2 (k - j) - 1} - 1 - \alpha}{\alpha (1 + \alpha) ^ {2 (k - j) - 1} + \alpha}\right) \left(1 + \frac {n \left(1 + \frac {(1 + \alpha) ^ {2 (k - j) - 1} - 1 - \alpha}{\alpha (1 + \alpha) ^ {2 (k - j) - 1} + \alpha}\right)}{n d ^ {2}}\right) ^ {- 1} \\ = n _ {j} ^ {\rightarrow} + n \frac {(1 + \alpha) ^ {2 (k - j)} - 1}{\alpha (1 + \alpha) ^ {2 (k - j) - 1} + \alpha} \left(1 + \frac {\alpha^ {2}}{\alpha + 1} \left(1 + \frac {(1 + \alpha) ^ {2 (k - j) - 1} - 1 - \alpha}{\alpha (1 + \alpha) ^ {2 (k - j) - 1} + \alpha}\right)\right) ^ {- 1} \\ \end{array}
$$
where the second equality follows because  $nd^2 = \frac{\alpha^2}{\alpha + 1}$  since  $\alpha = \frac{nd^2}{2}\left(\sqrt{1 + \frac{4}{nd^2}} + 1\right)$ . Then, we have that
$$
\begin{array}{l} n _ {j} ^ {\rightleftharpoons k} \geq n _ {j} ^ {\leftarrow} + n \frac {(1 + \alpha) ^ {2 (k - j)} - 1}{\alpha (1 + \alpha) ^ {2 (k - j) - 1} + \alpha} \\ \cdot \left(\frac {\left((1 + \alpha) ^ {2 (k - j) - 1} + 1\right) (\alpha + 1 + \alpha^ {2}) + \alpha \left((1 + \alpha) ^ {2 (k - j) - 1} - 1 - \alpha\right)}{(\alpha + 1) \left((1 + \alpha) ^ {2 (k - j) - 1} + 1\right)}\right) ^ {- 1} \\ \geq n _ {j} ^ {\rightarrow} + n \frac {(1 + \alpha) ^ {2 (k - j)} - 1}{\alpha (1 + \alpha) ^ {2 (k - j) - 1} + \alpha} \frac {(\alpha + 1) ((1 + \alpha) ^ {2 (k - j) - 1} + 1)}{(1 + \alpha) ^ {2 (k - j) + 1} + 1}. \\ \end{array}
$$
Now, we obtain bounds for the ESS depending on the value value of  $nd^2$ . Such bounds are obtained similarly as in Theorem 2 and we also denote by  $\phi$  the golden ratio  $\phi = 1.618 \dots$
1. If  $nd^2 < \frac{1}{j^2} \Rightarrow \sqrt{nd^2} \leq \alpha \leq \sqrt{nd^2}\phi \leq \frac{\phi}{j} \leq 1$  because
$$
\alpha = n d ^ {2} \frac {\sqrt {1 + \frac {4}{n d ^ {2}}} + 1}{2} = \sqrt {n d ^ {2}} \frac {\sqrt {n d ^ {2} + 4} + \sqrt {n d ^ {2}}}{2}
$$
then we have that  $n_j \stackrel{=k}{=} k$  satisfies
$$
n _ {j} ^ {\rightleftarrows^ {k}} \geq n _ {j} ^ {\rightleftarrows} + n \frac {1}{\alpha} \frac {\alpha (2 (k - j))}{2 + \alpha 2 (k - j)} = n _ {j} ^ {\rightleftarrows} + n \frac {k - j}{1 + \alpha (k - j)} \geq n _ {j} ^ {\rightleftarrows} + n \frac {k - j}{1 + \frac {\phi}{j} (k - j)}
$$
where the first inequality follows because  $(1 + \alpha)^{2(k - j) - 1} \geq 1 + \alpha (2(k - j) - 1)$  and the second inequality is obtained using  $\alpha \leq \frac{\phi}{j}$ .
2. If  $\frac{1}{j^2} \leq nd^2 < 1 \Rightarrow \frac{1}{j} \leq \sqrt{nd^2} \leq \alpha \leq \sqrt{nd^2}\phi$  because
then we have that  $n_j \stackrel{=k}{=} 1$  satisfies
$$
n _ {j} ^ {\rightleftharpoons k} \geq n _ {j} ^ {\rightleftharpoonup} \frac {n}{\alpha} \frac {(1 + \alpha) ^ {2 (k - j)} - 1}{(1 + \alpha) ^ {2 (k - j)} + 1} \geq n _ {j} ^ {\rightleftharpoonup} \frac {n}{\alpha} \frac {(1 + \sqrt {n d ^ {2}}) ^ {2 (k - j)} - 1}{(1 + \sqrt {n d ^ {2}}) ^ {2 (k - j)} + 1}
$$