SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left\| \boldsymbol {\theta} _ {t + 1} - \bar {\boldsymbol {\theta}} _ {t + 1} \right\| \\ = \left\| \boldsymbol {\theta} _ {t} ^ {j} - \bar {\boldsymbol {\theta}} _ {t} ^ {j} \right\| + \frac {2 \eta}{n} \sqrt {n (\rho + \sqrt {\frac {\nu}{m}})} \sqrt {n} \sqrt {\left(1 - \frac {\eta \lambda_ {0}}{4 n}\right) ^ {t} \hat {\mathcal {R}} (\boldsymbol {\theta} _ {0}) + \frac {4 n (1 - \mu \gamma) ^ {K} D}{\eta \lambda_ {0}}} \\ + \frac {2 \eta}{n} \sqrt {n} \left(1 - \frac {\eta \lambda_ {0}}{4 n}\right) ^ {\frac {t}{2}} \frac {1 6}{\lambda_ {0}} \\ \times \left(\left(\frac {3}{2} \| \mathbf {K} \| + 2 n ^ {2} \tilde {n} + 2 \sqrt {n} n \tilde {n}\right) \sqrt {(1 - \mu \gamma) ^ {K} D} + \left(2 \sqrt {n} n ^ {2} \tilde {n} + 2 \sqrt {n} n \tilde {n}\right) \sqrt {\hat {\mathcal {R}} (\boldsymbol {\theta} _ {0})}\right) \\ + \frac {2 \eta}{n} \sqrt {n} \left(2 \sqrt {n} \tilde {n} + 4 \sqrt {n} n \tilde {n}\right) \frac {8 n}{\lambda_ {0}} \sqrt {\frac {4 n C (1 - \mu \gamma) ^ {K} D}{\eta \lambda_ {0}}} \\ + \frac {2 \eta}{n} 2 \sqrt {\left(1 - \frac {\eta \lambda_ {0}}{4 n}\right) ^ {t} (1 - \mu \gamma) ^ {K} D}. \\ \end{array}
$$
Conducting telescoping sum yields:
$$
\begin{array}{l} \left\| \boldsymbol {\theta} _ {t + 1} - \bar {\boldsymbol {\theta}} _ {t + 1} \right\| \\ = \frac {2 \eta}{n} \sum_ {s = 0} ^ {t + 1} n \sqrt {\left(\rho + \sqrt {\frac {\nu}{m}}\right)} \sqrt {\left(1 - \frac {\eta \lambda_ {0}}{4 n}\right) ^ {s} \hat {\mathcal {R}} ^ {j} \left(\boldsymbol {\theta} _ {0} ^ {j}\right) + \frac {4 n C (1 - \mu \gamma) ^ {K} D}{\eta \lambda_ {0}}} \\ + \frac {2 \eta}{n} \sum_ {s = 0} ^ {t + 1} \sqrt {n} \left(1 - \frac {\eta \lambda_ {0}}{4 n}\right) ^ {\frac {s}{2}} \frac {1 6}{\lambda_ {0}} \\ \times \left(\left(\frac {3}{2} \| \mathbf {K} \| + 2 n ^ {2} \tilde {n} + 2 \sqrt {n} n \tilde {n}\right) \sqrt {(1 - \mu \gamma) ^ {K} D} + \left(2 \sqrt {n} n ^ {2} \tilde {n} + 2 \sqrt {n} n \tilde {n}\right) \sqrt {\hat {\mathcal {R}} (\boldsymbol {\theta} _ {0})}\right) \\ + \frac {2 \eta}{n} \sum_ {s = 0} ^ {t + 1} \left(2 n \tilde {n} + 4 n ^ {2} \tilde {n}\right) \frac {8 n}{\lambda_ {0}} \sqrt {\frac {4 n C (1 - \mu \gamma) ^ {K} D}{\eta \lambda_ {0}}} \\ + \frac {2 \eta}{n} \sum_ {s = 0} ^ {t + 1} 2 \sqrt {\left(1 - \frac {\eta \lambda_ {0}}{4 n}\right) ^ {s} (1 - \mu \gamma) ^ {K} D} \\ \leq n \sqrt {\tilde {n}} \left(\frac {1 6}{\lambda_ {0}} \sqrt {\hat {\mathcal {R}} (\boldsymbol {\theta} _ {0})}\right) + \eta T \left(\sqrt {\tilde {n}} + \left(2 n \tilde {n} + 4 n ^ {2} \tilde {n}\right) \frac {1 6}{\lambda_ {0}}\right) \sqrt {\frac {4 n C (1 - \mu \gamma) ^ {K} D}{\eta \lambda_ {0}}} \\ + \frac {2 5 6 \sqrt {n}}{\lambda_ {0} ^ {2}} \left(\left(\frac {3}{2} \| \mathbf {K} \| + 2 n ^ {2} \tilde {n} + 2 \sqrt {n} n \tilde {n}\right) \sqrt {\left(1 - \mu \gamma\right) ^ {K} D} + \left(2 \sqrt {n} n ^ {2} \tilde {n} + 2 \sqrt {n} n \tilde {n}\right) \sqrt {\hat {\mathcal {R}} (\boldsymbol {\theta} _ {0})}\right) \\ + \frac {3 2}{\lambda_ {0}} \sqrt {(1 - \mu \gamma) ^ {K} D} \\ \end{array}
$$
Recall that  $\tilde{n} = \left(\frac{48nB_y}{\lambda_0\sqrt{m}} +\sqrt{\frac{\nu}{m}}\right)$  and  $C = \frac{\lambda_0^2\eta^2}{24n^4} +\frac{4\eta^2\left(\frac{1}{2}\|\mathbf{K}\| + \frac{\lambda_0}{8}\right)^2}{\frac{3}{8}n^2}$ , we conclude that:
$$
\begin{array}{l} \left\| \boldsymbol {\theta} _ {t + 1} - \bar {\boldsymbol {\theta}} _ {t + 1} \right\| ^ {2} \\ \leq O \left(\frac {n ^ {2}}{\lambda_ {0} ^ {2}} \left(\frac {4 8 n B _ {y}}{\lambda_ {0} \sqrt {m}} + \sqrt {\frac {\nu}{m}}\right) + \eta^ {2} T ^ {2} \left(\frac {4 8 n B _ {y}}{\lambda_ {0} \sqrt {m}} + \sqrt {\frac {\nu}{m}}\right) ^ {2} \frac {n ^ {4}}{\lambda_ {0} ^ {2}} \frac {n (1 - \frac {1}{\kappa}) ^ {K} D}{\eta \lambda_ {0}} \frac {\eta^ {2} (\lambda_ {\max } + \lambda_ {0}) ^ {2}}{n ^ {2}}\right) \\ + O \left(\frac {n}{\lambda_ {0} ^ {4}} n ^ {4} \left(\frac {4 8 n B _ {y}}{\lambda_ {0} \sqrt {m}} + \sqrt {\frac {\nu}{m}}\right) ^ {2} \left(1 - \frac {1}{\kappa}\right) ^ {K} D + \frac {n}{\lambda_ {0} ^ {4}} n ^ {5} \left(\frac {4 8 n B _ {y}}{\lambda_ {0} \sqrt {m}} + \sqrt {\frac {\nu}{m}}\right) ^ {2} + \frac {(1 - \frac {1}{\kappa}) ^ {K} D}{\lambda_ {0} ^ {2}}\right) \\ = O \left(\frac {n ^ {2}}{\lambda_ {0} ^ {2}} \left(\frac {4 8 n B _ {y}}{\lambda_ {0} \sqrt {m}} + \sqrt {\frac {\nu}{m}}\right) + \eta^ {3} T ^ {2} \left(\frac {4 8 n B _ {y}}{\lambda_ {0} \sqrt {m}} + \sqrt {\frac {\nu}{m}}\right) ^ {2} \frac {n ^ {3} (1 - \frac {1}{\kappa}) ^ {K} D}{\lambda_ {0} ^ {3}} (\lambda_ {\max } + \lambda_ {0}) ^ {2}\right) \\ + O \left(\frac {n ^ {6}}{\lambda_ {0} ^ {4}} \left(\frac {4 8 n B _ {y}}{\lambda_ {0} \sqrt {m}} + \sqrt {\frac {\nu}{m}}\right) ^ {2} + \frac {(1 - \frac {1}{\kappa}) ^ {K} D}{\lambda_ {0} ^ {2}}\right) \\ \end{array}
$$
Theorem 6. For Algorithm 2, the following statement holds true for any  $j \in [d]$ ,  $t \in [T]$ :
$$
\begin{array}{l} \sup  _ {\boldsymbol {\alpha} \in \Delta^ {N}} \left(h _ {T} ^ {j} (\boldsymbol {\alpha}) - h _ {T} ^ {\mathrm {k l s}, j} (\boldsymbol {\alpha})\right) ^ {2} \\ \leq 2 \left(\frac {4 8 n B _ {y}}{\lambda_ {0} \sqrt {m}}\right) ^ {2} \left(\frac {4 8 n B _ {y}}{\lambda_ {0}} + \sqrt {\nu}\right) ^ {2} + O \left(\frac {n ^ {2}}{\lambda_ {0} ^ {2}} \left(\frac {4 8 n B _ {y}}{\lambda_ {0} \sqrt {m}} + \sqrt {\frac {\nu}{m}}\right) + \frac {n ^ {6}}{\lambda_ {0} ^ {4}} \left(\frac {4 8 n B _ {y}}{\lambda_ {0} \sqrt {m}} + \sqrt {\frac {\nu}{m}}\right) ^ {2}\right) \\ + O \left(\eta^ {3} T ^ {2} \left(\frac {4 8 n B _ {y}}{\lambda_ {0} \sqrt {m}} + \sqrt {\frac {\nu}{m}}\right) ^ {2} \frac {n ^ {3} (1 - \frac {1}{\kappa}) ^ {K} D}{\lambda_ {0} ^ {3}} (\lambda_ {\max } + \lambda_ {0}) ^ {2} + \frac {(1 - \frac {1}{\kappa}) ^ {K} D}{\lambda_ {0} ^ {2}}\right) \\ \end{array}
$$
If we choose  $K \geq \kappa \log \left(\frac{\eta^3 T^2 n^3 D}{\lambda_0^3}\right)$ , then we have:
$$
\sup  _ {\boldsymbol {\alpha} \in \Delta^ {N}} \left(h _ {T} ^ {j} (\boldsymbol {\alpha}) - h _ {T} ^ {\mathrm {k l s}, j} (\boldsymbol {\alpha})\right) ^ {2} \leq O \left(\max  \left\{\frac {\operatorname {p o l y} _ {3} (n)}{\sqrt {m}}, \frac {\operatorname {p o l y} _ {8} (n)}{m} \right\}\right).
$$
Proof. We consider the following canonical decomposition:
$$
\left(h ^ {j} (\boldsymbol {\alpha}) - h ^ {\mathrm {k l s}, j} (\boldsymbol {\alpha})\right) ^ {2} \leq 2 \left(h _ {t} ^ {j} (\boldsymbol {\alpha}) - h _ {t} ^ {\mathrm {r f}, j} (\boldsymbol {\alpha})\right) ^ {2} + 2 \left(h _ {t} ^ {\mathrm {r f}, j} (\boldsymbol {\alpha}) - h _ {t} ^ {\mathrm {k l s}, j} (\boldsymbol {\alpha})\right) ^ {2}
$$
For the first term, we
$$
\begin{array}{l} \left\| h _ {t} ^ {j} (\boldsymbol {\alpha}) - h _ {t} ^ {\mathrm {r f}, j} (\boldsymbol {\alpha}) \right\| ^ {2} \\ \leq 2 \left\| \phi_ {t} ^ {j} (\boldsymbol {\alpha}) ^ {\top} \boldsymbol {\theta} _ {t} ^ {j} - \phi_ {0} ^ {j} (\boldsymbol {\alpha}) ^ {\top} \boldsymbol {\theta} _ {t} ^ {j} \right\| ^ {2} + 2 \left\| \phi_ {0} ^ {j} (\boldsymbol {\alpha}) ^ {\top} (\boldsymbol {\theta} _ {t} ^ {j} - \bar {\boldsymbol {\theta}} _ {t} ^ {j}) \right\| ^ {2} \\ \leq 2 \rho^ {2} (\sqrt {m} \rho + \sqrt {\nu}) ^ {2} + O \left(\frac {n ^ {2}}{\lambda_ {0} ^ {2}} \left(\frac {4 8 n B _ {y}}{\lambda_ {0} \sqrt {m}} + \sqrt {\frac {\nu}{m}}\right) + \frac {n ^ {6}}{\lambda_ {0} ^ {4}} \left(\frac {4 8 n B _ {y}}{\lambda_ {0} \sqrt {m}} + \sqrt {\frac {\nu}{m}}\right) ^ {2}\right) \\ + O \left(\eta^ {3} T ^ {2} \left(\frac {4 8 n B _ {y}}{\lambda_ {0} \sqrt {m}} + \sqrt {\frac {\nu}{m}}\right) ^ {2} \frac {n ^ {3} (1 - \frac {1}{\kappa}) ^ {K} D}{\lambda_ {0} ^ {3}} (\lambda_ {\max } + \lambda_ {0}) ^ {2} + \frac {(1 - \frac {1}{\kappa}) ^ {K} D}{\lambda_ {0} ^ {2}}\right) \\ \end{array}
$$
where we plug in Proposition 5 (e) to bound  $\left\| \phi_t^j (\pmb {\alpha})^\top \pmb {\theta}_t^j -\phi_0^j (\pmb {\alpha})^\top \pmb {\theta}_t^j\right\|$ , and use Lemma 13 to bound  $\left\| \phi_0^j (\pmb {\alpha})^\top (\pmb {\theta}_t^j -\bar{\pmb{\theta}}_t^j)\right\|$ . Plugging in  $\rho = \frac{48nB_y}{\lambda_0\sqrt{m}}$  will complete bounding first term:
$$
\begin{array}{l} \left\| h _ {t} ^ {j} (\boldsymbol {\alpha}) - h _ {t} ^ {\mathrm {r f}, j} (\boldsymbol {\alpha}) \right\| ^ {2} \\ \leq 2 \left(\frac {4 8 n B _ {y}}{\lambda_ {0} \sqrt {m}}\right) ^ {2} \left(\frac {4 8 n B _ {y}}{\lambda_ {0}} + \sqrt {\nu}\right) ^ {2} + O \left(\frac {n ^ {2}}{\lambda_ {0} ^ {2}} \left(\frac {4 8 n B _ {y}}{\lambda_ {0} \sqrt {m}} + \sqrt {\frac {\nu}{m}}\right) + \frac {n ^ {6}}{\lambda_ {0} ^ {4}} \left(\frac {4 8 n B _ {y}}{\lambda_ {0} \sqrt {m}} + \sqrt {\frac {\nu}{m}}\right) ^ {2}\right) \\ + O \left(\eta^ {3} T ^ {2} \left(\frac {4 8 n B _ {y}}{\lambda_ {0} \sqrt {m}} + \sqrt {\frac {\nu}{m}}\right) ^ {2} \frac {n ^ {3} (1 - \frac {1}{\kappa}) ^ {K} D}{\lambda_ {0} ^ {3}} (\lambda_ {\max } + \lambda_ {0}) ^ {2} + \frac {(1 - \frac {1}{\kappa}) ^ {K} D}{\lambda_ {0} ^ {2}}\right) \\ \end{array}
$$
For the second term, we use the intermediate result from [21]
$$
\left\| \mathbf {h} _ {t} ^ {\mathrm {r f}, j} (\boldsymbol {\alpha}) - \mathbf {h} _ {t} ^ {\mathrm {k l s}, j} (\boldsymbol {\alpha}) \right\| \leq \sqrt {\frac {\nu}{m}} B _ {y} \frac {n}{\lambda_ {0}} \left(\frac {2 4 n}{\lambda_ {0}} + \frac {1}{2}\right)
$$
Lemma 14 (Approximation of Lipschitz functions on the ball [2, Proposition 6]). For  $R$  larger than a constant  $c$  that depends only on  $N$ , for any function  $f^{\star}:\mathbb{R}^{N}\to \mathbb{R}$  such that for all  $\mathbf{x},\tilde{\mathbf{x}}\in \mathbb{B}_q^N$ ,  $\sup_{\mathbf{x}\in \mathbb{B}_q^N}|f^\star (\mathbf{x})|\leq \Lambda$  and  $|f^{\star}(\mathbf{x}) - f^{\star}(\tilde{\mathbf{x}})|\leq \Lambda \| \mathbf{x} - \tilde{\mathbf{x}}\| _q$ , there exists  $h\in \mathcal{H}$ , such that  $\| h\|_{\mathcal{H}}^2\leq R$  and
$$
\sup  _ {\mathbf {x} \in \mathbb {B} _ {q} ^ {d} (1)} | f ^ {\star} (\mathbf {x}) - h (\mathbf {x}) | \leq A (R), \quad A (R) = c \Lambda \left(\frac {\sqrt {R}}{\Lambda}\right) ^ {- \frac {2}{d - 2}} \ln \left(\frac {\sqrt {R}}{\Lambda}\right).
$$
Theorem 7 ([32, Theorem 1]). Let  $\mathcal{F} = \left\{f:\mathbb{S}^{d - 1}\to [0,b]\right\}$  for some  $b < \infty$ . Then with probability at least  $1 - e^{-\nu}, \nu > 0$ , for all  $f\in \mathcal{F}$  simultaneously
$$
\| f \| _ {2} ^ {2} \leq \| f \| _ {n} ^ {2} + O \left(\| f \| _ {n} \left(\widehat {\Re} (\mathcal {F}) + \sqrt {\frac {b \nu}{n}}\right) + \widehat {\Re} ^ {2} (\mathcal {F}) + \frac {b \nu}{n}\right),
$$
where the worst-case empirical Rademacher complexity is defined as
$$
\widehat {\mathfrak {R}} (\mathcal {F}) = \sup  _ {\mathbf {x} _ {1}, \dots , \mathbf {x} _ {n} \in \mathbb {S} ^ {d - 1}} \mathbb {E} _ {\varepsilon} \sup  _ {f \in \mathcal {F}} \left| \frac {1}{n} \sum_ {i = 1} ^ {n} \varepsilon_ {i} f (\mathbf {x} _ {i}) \right|. \quad (\varepsilon_ {1}, \dots , \varepsilon_ {n} \stackrel {{\mathrm {i i d}}} {{\sim}} \operatorname {u n i f} \{\pm 1 \})
$$
Lemma 15. [Rademacher complexity of kernel predictor class [3] Lemma 22] Consider the following function class with bounded RKHS norm:
$$
\mathcal {F} = \left\{h (\mathbf {x}) = \sum_ {i = 1} ^ {n} c _ {i} k (\mathbf {x}, \mathbf {x} _ {i}): h \in \mathcal {H}, \| h \| _ {\mathcal {H}} \leq B \right\}.
$$
Then its empirical Rademacher complexity is bounded by:
$$
\widehat {\Re} (\mathcal {F}) \leq O \left(\frac {B}{\sqrt {n}}\right).
$$
Lemma 16 (Excess risk of virtual KLS predictor learnt on optimal KLS approximator). Let  $h_*^{\mathrm{kl},j}$  be the optimal approximator of  $f^{*,j}$  in RKHS, and  $\tilde{h}_t^{\mathrm{kl},j}(\alpha)$  be KLS predictor trained by GD on sample  $\{\alpha_i, h_*^{\mathrm{kl},j}(\alpha_i)\}_{i=1}^n$ . Then the following excess risk bound holds with probability at least  $1 - e^{-\nu}$ :
$$
\mathbb {E} \left\| \tilde {h} _ {t + 1} ^ {\mathrm {k l s}, j} (\pmb {\alpha}) - h _ {*} ^ {\mathrm {k l s}, j} (\pmb {\alpha}) \right\| ^ {2} \leq \left(1 - \frac {2 \lambda_ {0} ^ {2} \eta}{n}\right) ^ {t} + O \left(\left(1 - \frac {2 \lambda_ {0} ^ {2} \eta}{n}\right) ^ {t} \left(\frac {R}{\sqrt {n}} + \sqrt {\frac {\nu}{n}}\right) + \frac {R ^ {2}}{n} + \frac {\nu}{n}\right),
$$
Proof. Given a fixed function  $h_{*}^{\mathrm{kl},j}$  with  $\left\| h_{*}^{\mathrm{kl},j}\right\|_{\mathcal{H}}^{2} \leq R$ , we consider the following class:
$$
\widetilde {\mathcal {H}} = \left\{\tilde {h} (\mathbf {x}) = h (\mathbf {x}) - h _ {*} ^ {\mathrm {k l s}, j} (\mathbf {x}): h \in \mathcal {H}, \| h \| _ {\mathcal {H}} \leq B \right\}
$$
According to constant shift property of Rademacher complexity [33] and Lemma 15, we know
$$
\widehat {\Re} (\widetilde {\mathcal {H}}) \leq O \left(\frac {B}{\sqrt {n}} + \frac {\sqrt {R}}{\sqrt {n}}\right).
$$
According to Theorem 7, we have
$$
\begin{array}{l} \mathbb {E} \left\| \tilde {h} _ {t} ^ {\mathrm {k l s}, j} (\pmb {\alpha}) - h _ {*} ^ {\mathrm {k l s}, j} (\pmb {\alpha}) \right\| ^ {2} \leq \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| \tilde {h} _ {t} ^ {\mathrm {k l s}, j} (\pmb {\alpha} _ {i}) - h _ {*} ^ {\mathrm {k l s}, j} (\pmb {\alpha} _ {i}) \right\| ^ {2} \\ + O \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \left\| \tilde {h} _ {t} ^ {\mathrm {k l s}, j} (\boldsymbol {\alpha} _ {i}) - h _ {*} ^ {\mathrm {k l s}, j} (\boldsymbol {\alpha} _ {i}) \right\| ^ {2} \left(\frac {B + \sqrt {R}}{\sqrt {n}} + \sqrt {\frac {\nu}{n}}\right) + \frac {(B + \sqrt {R}) ^ {2}}{n} + \frac {\nu}{n}\right), \\ \end{array}
$$
Now, it suffices to bound (i) empirical risk term  $\frac{1}{n}\sum_{i=1}^{n}\left\|\tilde{h}_t^{\mathrm{kls},j}(\pmb{\alpha}_i)-h_*^{\mathrm{kls},j}(\pmb{\alpha}_i)\right\|^2$  and (ii) RKHS norm of  $\tilde{h}_t^{\mathrm{kls},j}$ .
For (i), we define  $\widehat{\mathcal{R}}_t = \frac{1}{n}\sum_{i=1}^{n}\left\|\tilde{h}_t^{\mathrm{kls},j}(\pmb{\alpha}_i) - h_*^{\mathrm{kls},j}(\pmb{\alpha}_i)\right\|^2$  and recall the updating rule of kernel OLS predictor:
$$
\begin{array}{l} \tilde {\mathbf {h}} _ {t + 1} ^ {\mathrm {k l s}, j} - \tilde {\mathbf {h}} _ {*} ^ {\mathrm {k l s}, j} = \mathbf {K} \mathbf {c} _ {t + 1} ^ {j} - \tilde {\mathbf {h}} _ {*} ^ {\mathrm {k l s}, j} = \mathbf {K} \mathbf {c} _ {t} ^ {j} - \mathbf {K} \frac {2 \eta}{n} \mathbf {K} ^ {\top} (\tilde {\mathbf {h}} _ {t} ^ {\mathrm {k l s}, j} - \tilde {\mathbf {h}} _ {*} ^ {\mathrm {k l s}, j}) - \tilde {\mathbf {h}} _ {*} ^ {\mathrm {k l s}, j} \\ = \left(\mathbf {I} - \frac {2 \eta}{n} \mathbf {K} ^ {\top} \mathbf {K}\right) (\tilde {\mathbf {h}} _ {t} ^ {\mathrm {k l s}, j} - \tilde {\mathbf {h}} _ {*} ^ {\mathrm {k l s}, j}) \\ = \left(\mathbf {I} - \frac {2 \eta}{n} \mathbf {K} ^ {\top} \mathbf {K}\right) ^ {t} (\tilde {\mathbf {h}} _ {0} ^ {\mathrm {k l s}, j} - \tilde {\mathbf {h}} _ {*} ^ {\mathrm {k l s}, j}). \\ \end{array}
$$
Hence we know that:
$$
\widehat {\mathcal {R}} _ {t + 1} = \left(\mathbf {I} - \frac {2 \eta}{n} \mathbf {K K} ^ {\top}\right) ^ {t} \widehat {\mathcal {R}} _ {0}.
$$
(ii) RKHS norm of  $\tilde{h}_t^{\mathrm{kls},j}$
$$
\begin{array}{l} \left\| \tilde {h} _ {t} ^ {\mathrm {k l s}, j} \right\| _ {\mathcal {H}} = \left\| \left(\mathbf {I} - \left(\mathbf {I} - \frac {2 \eta}{n} \mathbf {K}\right) ^ {t}\right) \mathbf {y} _ {*} \right\| _ {\mathbf {K} ^ {- 1}} \\ = \left\| \left(\mathbf {I} - \left(\mathbf {I} - \frac {2 \eta}{n} \mathbf {K}\right) ^ {t}\right) \mathbf {K} ^ {- 1} \mathbf {K} \mathbf {c} _ {*} \right\| \\ \leq \left\| \mathbf {I} - \left(\mathbf {I} - \frac {2 \eta}{n} \mathbf {K}\right) ^ {t} \right\| \left\| \mathbf {K} ^ {- 1} \mathbf {K c} _ {*} \right\| \\ = \left\| \mathbf {I} - \left(\mathbf {I} - \frac {2 \eta}{n} \mathbf {K}\right) ^ {t} \right\| \left\| \tilde {h} _ {*} ^ {\mathrm {k l s}, j} \right\| _ {\mathcal {H}} \leq \sqrt {R}. \\ \end{array}
$$
Lemma 17. [21, Lemma 8] Let  $f_{t}^{\kappa}, \tilde{f}_{t}^{\kappa}$  be GD-trained KLS predictors (as discussed in appendix B.2) given training samples  $(\mathbf{x}_i, y_i)_{i=1}^n$  and  $(\mathbf{x}_i, \tilde{y}_i)_{i=1}^n$  respectively, where  $y_i = f^{\star}(\mathbf{x}_i) + \varepsilon_i$  and  $\tilde{y}_i = h(\mathbf{x}_i) + \varepsilon_i$  with  $\|h\|_{\mathcal{H}}^2 \leq R$  characterized by lemma 14. Then, with  $A(R)$  defined in lemma 14, for any  $t \in \mathbb{N}$ ,
$$
\left\| f _ {t} ^ {\kappa} - \tilde {f} _ {t} ^ {\kappa} \right\| _ {n} ^ {2} \leq A (R).
$$
Proof. We notice the following risk decomposition:
$$
\begin{array}{l} \mathbb {E} \left\| h _ {T} ^ {j} (\boldsymbol {\alpha}) - w _ {j} ^ {*} (\boldsymbol {\alpha}) \right\| ^ {2} \leq 4 \mathbb {E} \left\| h _ {T} ^ {j} (\boldsymbol {\alpha}) - h _ {T} ^ {\mathrm {k l s}, j} (\boldsymbol {\alpha}) \right\| ^ {2} + 4 \mathbb {E} \left\| h _ {T} ^ {\mathrm {k l s}, j} (\boldsymbol {\alpha}) - \tilde {h} _ {T} ^ {\mathrm {k l s}, j} (\boldsymbol {\alpha}) \right\| ^ {2} \\ + 4 \mathbb {E} \left\| \tilde {h} _ {t} ^ {\mathrm {k l s}, j} (\boldsymbol {\alpha}) - h _ {*} ^ {\mathrm {k l s}, j} (\boldsymbol {\alpha}) \right\| ^ {2} + 4 \mathbb {E} \left\| h _ {*} ^ {\mathrm {k l s}, j} (\boldsymbol {\alpha}) - w _ {j} ^ {*} (\boldsymbol {\alpha}) \right\| ^ {2} \\ \leq O \left(\max  \left\{\frac {\operatorname {p o l y} _ {3} (n)}{\sqrt {m}}, \frac {\operatorname {p o l y} _ {8} (n)}{m} \right\}\right) + O \left(\left(1 + \frac {c}{\lambda_ {0} ^ {2}}\right) A (R) ^ {2} + \frac {c}{\sqrt {n}}\right) \\ + \left(1 - \frac {2 \lambda_ {0} ^ {2} \eta}{n}\right) ^ {T} + O \left(\frac {R}{n}\right) + O \left(\Lambda \left(\frac {\sqrt {R}}{\Lambda}\right) ^ {- \frac {2}{N - 2}} \ln \left(\frac {\sqrt {R}}{\Lambda}\right)\right), \\ \end{array}
$$
where respectively we plug in the NN-KLS coupling result (Theorem 6), Lemma 17, excess risk of KLS (Lemma 16) and approximation error of RKHS to Lipschitz function (Lemma 14). By choosing  $m \geq \Omega(n^{8 + \frac{2}{2 + N}})$ ,  $T \geq \Omega\left(\frac{n}{N\lambda_0^2\eta}\log(n)\right)$  and optimizing over  $R$ , we recover the rate:
$$
\mathbb {E} \left\| h _ {t} ^ {j} (\boldsymbol {\alpha}) - w _ {j} ^ {*} (\boldsymbol {\alpha}) \right\| ^ {2} \leq O \left(\Lambda^ {2} n ^ {- \frac {2}{2 + N}}\right).
$$
Hence by summing over all coordinates  $j \in [d]$  we conclude the result:
$$
\mathbb {E} \left\| \mathbf {h} _ {t} (\boldsymbol {\alpha}) - \mathbf {w} ^ {*} (\boldsymbol {\alpha}) \right\| ^ {2} \leq O \left(\Lambda^ {2} d n ^ {- \frac {2}{2 + N}}\right)
$$
where in our case lemma 1 implies  $\Lambda = \kappa^{*}$ . Finally we lower bound  $\lambda_0$  using [29, Theorem 3.2] which implies that  $\lambda_0 = \Omega (\mathrm{polylog}(n,N)N)$  with probability at least  $1 - n^{2}e^{-O(\sqrt{N})}$  for a distribution on a simplex.
C Regret Analysis of Algorithm 4
In this section, we are going to provide the proof for Theorem 4. Let us first introduce the following definitions that will be used in our proof.
Definition 2 (Covering and Packing Numbers). An  $\varepsilon$ -cover of a set  $S$  w.r.t. some metric  $\|\cdot\|$  is a set  $\{\mathbf{x}_1', \ldots, \mathbf{x}_n'\} \subseteq S$  such that for each  $\mathbf{x} \in S$  there exists  $i \in \{1, \ldots, n\}$  such that  $\|\mathbf{x} - \mathbf{x}_i'\| \leq \varepsilon$ . The covering number  $\mathcal{N}(S, \varepsilon, \|\cdot\|)$  is the smallest cardinality of a  $\varepsilon$ -cover. An  $\varepsilon$ -packing of a set  $S$  w.r.t. some metric  $\|\cdot\|$  is a set  $\{\mathbf{x}_1', \ldots, \mathbf{x}_m'\} \subseteq S$  such that for any distinct  $i, j \in \{1, \ldots, m\}$ , we have  $\|\mathbf{x}_i' - \mathbf{x}_j'\| > \varepsilon$ . The packing number  $\mathcal{M}(S, \varepsilon, \|\cdot\|)$  is the largest cardinality of a  $\varepsilon$ -packing.
It is well known that  $\mathcal{M}(S,2\varepsilon ,\| \cdot \|)\leq \mathcal{N}(S,\varepsilon ,\| \cdot \|)\leq \mathcal{M}(S,\varepsilon ,\| \cdot \|)$
Definition 3 (Metric dimension). The metric space  $(\mathcal{X},\| \cdot \|)$  had metric dimension  $d$ , if there exists a constant  $C_d$  such that for all  $\varepsilon >0$ ,  $\mathcal{X}$  has an  $\varepsilon$ -cover at most  $C_d\varepsilon^{-d}$ .
Proof. We start from the standard analysis framework for online non-parametric regression [13, 18, 20]. Let  $S_{t}$  be the value of the variable  $S$  at the end of time  $t$ . Hence  $S_{0} = \varnothing$ . The functions  $\pi_t: \mathcal{X} \to \{1, \dots, t\}$  for  $t = 1, 2, \dots$  map each data point  $\mathbf{x}$  to its closest center (in norm  $\| \cdot \|$ ) in  $S_{t-1}$ ,
$$
\pi_ {t} (\boldsymbol {\alpha}) = \left\{ \begin{array}{l l} \arg \min  _ {s \in S _ {t - 1}} \| \boldsymbol {\alpha} - \boldsymbol {\alpha} _ {s} \| & \text {i f} S _ {t - 1} \neq \varnothing \\ t & \text {o t h e r w i s e .} \end{array} \right.
$$
The set  $T_{s}$  contain all data points  $\alpha_{t}$  that at time  $t$  belonged to the ball with center  $\alpha_{s}$  and radius  $\varepsilon_{t}$ ,
$$
T _ {s} = \left\{t: \| \boldsymbol {\alpha} _ {t} - \boldsymbol {\alpha} _ {s} \| \leq \varepsilon_ {t}, t = s, \dots , T \right\}.
$$
Finally,  $\mathbf{w}_s^\star$  is the best fixed prediction for all examples  $(\alpha_{t},\mathbf{w}_{t})$  such that  $t\in T_s$
$$
\mathbf {w} _ {s} ^ {\star} = \underset {\mathbf {w} \in \mathcal {W}} {\arg \min } \sum_ {t \in T _ {s}} \ell_ {t} (\mathbf {w}) = \frac {1}{| T _ {s} |} \sum_ {t \in T _ {s}} \mathbf {w} _ {t}. \tag {30}
$$
We proceed by decomposing the regret into estimation and approximation cumulative errors,
$$
R _ {T} (f) = \sum_ {t = 1} ^ {T} \left(\ell_ {t} (\hat {\mathbf {w}} _ {t}) - \ell_ {t} \big (f (\boldsymbol {\alpha} _ {t}) \big)\right) = \sum_ {t = 1} ^ {T} \left(\ell_ {t} (\hat {\mathbf {w}} _ {t}) - \ell_ {t} \big (\mathbf {w} _ {\pi_ {t} (\boldsymbol {\alpha} _ {t})} ^ {\star} \big)\right) + \sum_ {t = 1} ^ {T} \left(\ell_ {t} \big (\mathbf {w} _ {\pi_ {t} (\boldsymbol {\alpha} _ {t})} ^ {\star} \big) - \ell_ {t} \big (f (\boldsymbol {\alpha} _ {t}) \big)\right).
$$
The estimation term is bounded as
$$
\sum_ {t = 1} ^ {T} \left(\ell_ {t} (\hat {\mathbf {w}} _ {t}) - \ell_ {t} \left(\mathbf {w} _ {\pi_ {t} (\boldsymbol {\alpha} _ {t})} ^ {\star}\right)\right) = \sum_ {s \in S _ {T}} \sum_ {t \in T _ {s}} \left(\ell_ {t} (\hat {\mathbf {w}} _ {t}) - \ell_ {t} \left(\mathbf {w} _ {s} ^ {\star}\right)\right)
$$
We study the cumulative regret within a fixed ball  $T_{s}$ . We use  $T_{s}(t)$  to denote the set of all points belong to ball with center  $\mathbf{w}_{s}$ , at time  $t$ . That is,
$$
T _ {s} (t) = \{j: j \in T _ {s}, j \leq t \}.
$$
One property of  $T_{s}(t)$  is that, if  $\mathbf{x}_t \in T_s$ , then  $|T_{s}(t)| = |T_{s}(t - 1)| + 1$ . We further define  $\tilde{\mathbf{w}}_t = \frac{1}{|T_s(t)|} \sum_{j \in T_s(t)} \mathbf{w}^*(\alpha_j)$ . According to [6, Lemma 3.1], we have:
$$
\sum_ {t \in T _ {s}} \ell_ {t} (\hat {\mathbf {w}} _ {t}) - \ell_ {t} (\mathbf {w} _ {s} ^ {\star}) \leq \sum_ {t \in T _ {s}} \ell_ {t} (\hat {\mathbf {w}} _ {t}) - \ell_ {t} (\tilde {\mathbf {w}} _ {t})
$$
Recall that  $\hat{\mathbf{w}}_t = \frac{1}{|T_s(t - 1)|}\sum_{j\in T_s(t - 1)}\mathbf{w}_j$ . Then,
$$
\begin{array}{l} \mathbb {E} \left[ \ell_ {t} (\hat {\mathbf {w}} _ {t}) - \ell_ {t} (\tilde {\mathbf {w}} _ {t}) \right] \leq 4 D \mathbb {E} \left\| \hat {\mathbf {w}} _ {t} - \ell_ {t} (\tilde {\mathbf {w}} _ {t}) \right\| \\ = 4 D \mathbb {E} \left\| \frac {1}{\left| T _ {s} (t - 1) \right|} \sum_ {j \in T _ {s} (t - 1)} \mathbb {I} \left\{Z _ {j} = 1 \right\} \mathbf {w} _ {j} - \frac {1}{\left| T _ {s} (t) \right|} \sum_ {j \in T _ {s} (t)} \mathbf {w} ^ {*} (\boldsymbol {\alpha} _ {j}) \right\| \\ \leq 4 D \mathbb {E} \left\| \frac {1}{\left| T _ {s} (t - 1) \right|} \sum_ {j \in T _ {s} (t - 1)} \mathbb {I} \left\{Z _ {j} = 1 \right\} \left(\mathbf {w} _ {j} - \mathbf {w} ^ {*} (\boldsymbol {\alpha} _ {j})\right) \right\| \\ + 4 D \mathbb {E} \left\| \frac {1}{\left| T _ {s} (t - 1) \right|} \sum_ {j \in T _ {s} (t - 1)} \mathbb {I} \left\{Z _ {j} = 1 \right\} \mathbf {w} ^ {*} (\boldsymbol {\alpha} _ {j}) - \frac {1}{\left| T _ {s} (t) \right|} \sum_ {j \in T _ {s} (t)} \mathbf {w} ^ {*} (\boldsymbol {\alpha} _ {j}) \right\| \\ \end{array}
$$
According to the convergence rate of GD on strongly-convex and smooth function, the first term is bounded by  $p(1 - \frac{1}{\kappa})^{K}\bar{D}$ . For second term: