SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} _ {W, S} \mathcal {L} _ {S} (W) - \mathbb {E} _ {S} \mathcal {L} _ {S} \left(w ^ {*}\right) \leq \mathbb {E} _ {S} \left[ \mathbb {E} _ {V _ {1: K}} \left(\frac {1}{K} \sum_ {k = 1} ^ {K} \langle W _ {k} - w ^ {*}, V _ {k} \rangle\right) \right] \\ \leq \frac {\left\| W _ {1} - w ^ {*} \right\| ^ {2}}{2 \eta K} + \frac {\eta}{2 K} \sum_ {k = 1} ^ {K} \mathbb {E} \left\| V _ {k} \right\| ^ {2}. \\ \end{array}
$$
Since the loss function is  $L$ -Lipschitz, we can further obtain:
$$
\mathbb {E} _ {W, S} \mathcal {L} _ {S} (W) - \mathcal {L} _ {\mu} (w ^ {*}) \leq \frac {\| W _ {1} - w ^ {*} \| ^ {2}}{2 \eta K} + \frac {\eta}{2} L ^ {2}.
$$
We have assumed that the loss function is convex and  $\beta$ -smooth. Therefore, following Theorem 4.7 in Hardt et al. [50], we have the following upper bound for the generalization gap of SGD if the step size is smaller than  $\frac{2}{\beta}$ :
$$
| \operatorname {g e n} (\mu , S G D) | \leq \frac {L ^ {2} K}{m} \eta .
$$
So, we can conclude
$$
R _ {\text {e x c e s s}} (\mathbf {S G D}, u) = \mathbb {E} _ {W, S} [ \mathcal {L} _ {\mu} (W) ] - \inf  _ {w \in \mathcal {W}} \mathcal {L} _ {\mu} (w) \leq \frac {\| \phi - w ^ {*} \| ^ {2}}{2 \eta K} + \left(\frac {L ^ {2}}{2} + \frac {L ^ {2} K}{m}\right) \eta .
$$
B.3 Excess risk bound for SGLD
Theorem B.5 (Excess risk bound for SGLD of meta-parameter  $u = (\eta, \phi)$ ). Suppose  $\mathcal{W} = \mathbb{R}^d$ , let  $w^{*} \stackrel{\text{def}}{=} \arg \min_{w \in \mathcal{W}} \mathcal{L}_{\mu}(w)$  be the hypothesis that achieves the minimum population risk in  $\mathcal{W}$ . For SGLD algorithm with  $K$  adaptation starting from  $W_1 = \phi$ :  $W_{k+1} = W_k - \eta \nabla \ell(W_k, Z_k) + \xi_k$ , where  $Z_k$  is a sample randomly selected from data set  $S$ ,  $\xi_k$  is the independently injected isotropic Gaussian noise  $\mathcal{N}(0, \sigma^2 \mathbf{1}_d)$ . Assume the loss function  $\ell(\cdot, z)$  is convex and  $L$ -Lipschitz for all  $z \in \mathcal{Z}$ . And let the output of the algorithm be  $W = \frac{1}{K} \sum_{k=1}^{K} W_k$ . Then we have the following excess risk bound:
$$
R _ {e x c e s s} (S G L D, u) \stackrel {{d e f}} {{=}} \mathbb {E} _ {W, S} [ \mathcal {L} _ {\mu} (W) ] - \inf  _ {w \in \mathcal {W}} \mathcal {L} _ {\mu} (w) \leq \frac {\| \phi - w ^ {*} \| ^ {2}}{2 \eta K} + \frac {\eta}{2} L ^ {2} + \frac {d \sigma^ {2}}{2 \eta} + \sqrt {\frac {K}{4 m}} \frac {\eta L}{\sigma}.
$$
Proof. Analogous to the proof of SGD, we can decompose the excess risk as
$$
\begin{array}{l} \mathbb {E} _ {W, S} \left[ \mathcal {L} _ {\mu} (W) - \min  _ {w} \mathcal {L} _ {\mu} (w) \right] = \mathbb {E} _ {W, S} \left[ \mathcal {L} _ {\mu} (W) - \mathcal {L} _ {S} (W) \right] + \mathbb {E} _ {W, S} \mathcal {L} _ {S} (W) - \min  _ {w} \mathcal {L} _ {\mu} (w) \\ = \mathbb {E} _ {W, S} \left[ \mathcal {L} _ {\mu} (W) - \mathcal {L} _ {S} (W) \right] + \mathbb {E} _ {W, S} \mathcal {L} _ {S} (W) - \mathbb {E} _ {S} \left[ \mathcal {L} _ {S} \left(w ^ {*}\right) \right] \\ = \operatorname {g e n} (\mu , \mathrm {S G L D}) + \mathbb {E} _ {W, S} \mathcal {L} _ {S} (W) - \mathbb {E} _ {S} \mathcal {L} _ {S} \left(w ^ {*}\right). \\ \end{array}
$$
According to Pensia et al. [38], we have  $|gen(\mu, \mathrm{SGLD})| \leq \sqrt{\frac{1}{4m} \frac{K\eta^2L^2}{\sigma^2}}$ .
In addition, we have the update for SGLD:  $W_{k + 1} = W_k - \eta \nabla \ell (W_k,Z_k) + \xi_k$ , where  $W_{1:K}$  is related to the randomness introduced by the sampling path  $Z_{1:K}$  from  $S$  and the injected noise  $\xi_{1:K}$ .
Denote  $V_{k} = \nabla \ell (W_{k},Z_{k}) + \frac{\xi_{k}}{\eta}$ , then we have  $\mathbb{E}[V_k|W_k] = \mathbb{E}[V_k|V_{1:k - 1}] = \nabla \mathcal{L}_S(W_k)$ .
Conduct similar proof as the above SGD algorithm and use Lemma 14.1 in [51], we can obtain that
$$
\begin{array}{l} \mathbb {E} _ {W, S} \mathcal {L} _ {S} (W) - \mathbb {E} _ {S} \mathcal {L} _ {S} \left(w ^ {*}\right) \leq \mathbb {E} _ {S} \left[ \mathbb {E} _ {V _ {1: K}} \left(\frac {1}{K} \sum_ {k = 1} ^ {K} \langle W _ {k} - w ^ {*}, V _ {k} \rangle\right) \right] \\ \leq \frac {\| W _ {1} - w ^ {*} \| ^ {2}}{2 \eta K} + \frac {\eta}{2 K} \sum_ {k = 1} ^ {K} \mathbb {E} \| V _ {k} \| ^ {2}. \\ \end{array}
$$
Since the loss function is  $L$ -Lipschitz, we have
$$
\mathbb {E} _ {W, S} \mathcal {L} _ {S} (W) - \mathcal {L} _ {\mu} (w ^ {*}) \leq \frac {\| W _ {1} - w ^ {*} \| ^ {2}}{2 \eta K} + \frac {\eta}{2} (L ^ {2} + d \frac {\sigma^ {2}}{\eta^ {2}}).
$$
Combining the aforementioned generalization gap bound of SGLD, we conclude the proof.
B.4 Excess risk bound for Regularized Loss Minimization (RLM)
Theorem B.6 (Excess risk bound for RLM of meta-parameter  $u = (\beta, \phi)$ ). Suppose  $\mathcal{W} = \mathbb{R}^d$ , let  $w^{*} \stackrel{\text{def}}{=} \arg \min_{w \in \mathcal{W}} \mathcal{L}_{\mu}(w)$  be the hypothesis that achieves the minimum population risk in  $\mathcal{W}$ . Let us denote the output of RLM with Tikhonov regularization as  $W = \arg \min_{w \in \mathcal{W}} \mathcal{L}_S(w) + \frac{1}{\beta} \| w - \phi \|^2$ . Suppose the loss function  $\ell: \mathcal{W} \times \mathcal{Z} \to [0, 1]$  is convex and  $L$ -Lipschitz, so the excess risk of the above algorithm is bounded by:
$$
R _ {e x c e s s} (R L M, u) \stackrel {{d e f}} {=} \mathbb {E} _ {W, S} \left[ \mathcal {L} _ {\mu} (W) \right] - \inf  _ {w \in \mathcal {W}} \mathcal {L} _ {\mu} (w) \leq \frac {\| \phi - w ^ {*} \| ^ {2}}{\beta} + \frac {2 L ^ {2} \beta}{m}.
$$
Proof. Here, since the algorithm is deterministic, we can consider  $P_{W|S,u}$  as a delta distribution centered on  $w_S^* = \mathcal{A}(u,S)$ . Then, the bound becomes
$$
\begin{array}{l} \mathbb {E} _ {W, S} \left[ \mathcal {L} _ {\mu} (W) \right] = \mathbb {E} _ {S} \left[ L _ {\mu} \left(w _ {S} ^ {*}\right) \right] = \mathbb {E} _ {S} \left[ L _ {\mu} (\mathcal {A} (u, S) \right] \\ \leq \inf  _ {w \in \mathcal {W}} \mathcal {L} _ {\mu} (w) + \frac {1}{\beta} \| \phi - w ^ {*} \| ^ {2} + \frac {2 L ^ {2} \beta}{m}, \\ \end{array}
$$
which is equivalent to Corollary 13.8 in [51] with  $\phi = \mathbf{0},\beta = \frac{1}{\lambda}$
To prove the above bound, we use the same decomposition of excess risk:
$$
\begin{array}{l} \mathbb {E} _ {W, S} \left[ \mathcal {L} _ {\mu} (W) - \min  _ {w} \mathcal {L} _ {\mu} (w) \right] = \mathbb {E} _ {S} \left[ L _ {\mu} \left(w _ {S} ^ {*}\right) - \mathcal {L} _ {S} \left(w _ {S} ^ {*}\right) \right] + \mathbb {E} _ {S} \mathcal {L} _ {S} \left(w _ {S} ^ {*}\right) - \min  _ {w} \mathcal {L} _ {\mu} (w) \\ = \mathbb {E} _ {S} \left[ L _ {\mu} \left(w _ {S} ^ {*}\right) - \mathcal {L} _ {S} \left(w _ {S} ^ {*}\right) \right] + \mathbb {E} _ {S} \mathcal {L} _ {S} \left(w _ {S} ^ {*}\right) - \mathbb {E} _ {S} \left[ \mathcal {L} _ {S} \left(w ^ {*}\right) \right] \\ = \operatorname {g e n} (\mu , \mathrm {R L M}) + \mathbb {E} _ {S} \mathcal {L} _ {S} \left(w _ {S} ^ {*}\right) - \mathbb {E} _ {S} \mathcal {L} _ {S} \left(w ^ {*}\right). \\ \end{array}
$$
From Corollary 13.6 in [51], the stability of RLM rule satisfies
$$
g e n (\mu , R L M) = \mathbb {E} _ {S} \left[ L _ {\mu} \left(w _ {S} ^ {*}\right) - \mathcal {L} _ {S} \left(w _ {S} ^ {*}\right) \right] \leq \frac {2 L ^ {2} \beta}{m},
$$
which means the RLM rule using Tikhonov regularization described above is on average-replace-one-stable with rate  $\frac{2L^2\beta}{m}$ . From the definition of this algorithm, we have for any  $w \in \mathcal{W}$ ,
$$
\mathcal {L} _ {S} (w _ {S} ^ {*}) \leq \mathcal {L} _ {S} (w _ {S} ^ {*}) + \frac {1}{\beta} \| w _ {S} ^ {*} - \phi \| ^ {2} \leq \mathcal {L} _ {S} (w) + \frac {1}{\beta} \| w - \phi \| ^ {2}.
$$
Take expectation w.r.t  $S$ , we can obtain  $\mathbb{E}_S\mathcal{L}_S(w_S^*)\leq \mathcal{L}_\mu (w) + \frac{1}{\beta}\| w - \phi \| ^2,\forall w\in \mathcal{W}$ . Finally, combining the stability bound and letting  $w = w^{*}$ , we conclude the proof.
B.5 Unified form of excess risk upper bound
Lemma B.7. Assume that  $\mathcal{L}_{\mu_t}(\cdot)$  has  $\alpha$ -quadratic growth (defined in Appendix A.2)  $\forall t \in [T]$  and that for a given base learner  $\mathcal{A}$ , we have  $R_{excess}(\mathcal{A}, u_t) \leq \frac{b_t + c_t \| \phi_t - w_t^* \|^2}{\beta_t} + d_t \beta_t + e_t$ , with  $b_t, e_t \geq 0, c_t, d_t > 0$ . Then we have  $R_{excess}(\mathcal{A}, u_t) \leq f_t(u_t)$ , where  $f_t(u_t)$  is given by eq. (1).
Proof. Let  $\bar{w}_t = \mathbb{E}W_t$  be the expected output, and  $\xi_0 \stackrel{\mathrm{def}}{=} \| w_t - \bar{w}_t \|^2$  characterizes the randomness of the algorithm output, which should be small enough.
Then, using Titu's lemma, Jensen's inequality, and the  $\alpha$ -quadratic growth assumption, we get:
$$
\begin{array}{l} \left\| \phi_ {t} - w _ {t} ^ {*} \right\| ^ {2} = \left\| \phi_ {t} - \bar {w} _ {t} + \bar {w} _ {t} - w _ {t} ^ {*} \right\| ^ {2} \\ \leq 2 \| \phi_ {t} - \bar {w} _ {t} \| ^ {2} + 2 \| \bar {w} _ {t} - w _ {t} ^ {*} \| ^ {2} \\ = 2 \| \phi_ {t} - \bar {w} _ {t} \| ^ {2} + 2 \| \mathbb {E} W _ {t} - w _ {t} ^ {*} \| ^ {2} \\ \leq 2 \| \phi_ {t} - \bar {w} _ {t} \| ^ {2} + 2 \mathbb {E} \| W _ {t} - w _ {t} ^ {*} \| ^ {2} \\ \leq 2 \| \phi_ {t} - \bar {w} _ {t} \| ^ {2} + \frac {4 \mathbb {E} [ \mathcal {L} _ {\mu_ {t}} (W _ {t}) - \mathcal {L} _ {\mu_ {t}} (w _ {t} ^ {*}) ]}{\alpha} \\ \leq 2 \| \phi_ {t} - \bar {w} _ {t} \| ^ {2} + \frac {4 \left(\frac {b _ {t} + c _ {t} \| \phi_ {t} - w _ {t} ^ {*} \| ^ {2}}{\beta_ {t}} + d _ {t} \beta_ {t} + e _ {t}\right)}{\alpha}. \\ \end{array}
$$
Rearrange and put the upper bound of  $\| \phi_t - w_t^*\|^2$  into the excess risk bound, replace  $\sqrt{c_t / d_t}\beta_t' = \alpha \beta_t - 4c_t$  with  $\beta_{t} > 4c_{t} / \alpha$  and use again the Titu's lemma, we get
$$
R _ {\mathrm {e x c e s s}} (\mathcal {A}, u _ {t}) \leq 4 \alpha \sqrt {c _ {t} d _ {t}} \left(\frac {\| \phi_ {t} - w _ {t} \| ^ {2} + \xi_ {0} + \frac {b _ {t}}{4 c _ {t}} + \frac {4 d _ {t} c _ {t}}{\alpha^ {2}} + \frac {e _ {t}}{\alpha}}{\beta_ {t} ^ {\prime}} + \frac {2 \sqrt {d _ {t} c _ {t}}}{\alpha^ {2}} + \frac {\beta_ {t} ^ {\prime}}{4 \alpha^ {2}} + \frac {e _ {t}}{4 \alpha \sqrt {d _ {t} c _ {t}}}\right)
$$
It's easy to relate  $\kappa_t, \Delta_t, \epsilon_t, \epsilon_0$  to the terms in the above bound. Consequently, we have  $\epsilon_0 = 2\alpha \xi_0 = 2\alpha \| w_t - \mathbb{E}W_t \|^2$ ,  $\kappa_t = 2\sqrt{c_t d_t}$ ,  $a = \frac{1}{2\alpha}$ ,  $b = 2\alpha$ ,  $\epsilon_t = \frac{\alpha b_t}{2c_t} + \frac{8d_t c_t}{\alpha} + 2e_t$ , and  $\Delta_t = \frac{4\sqrt{c_t d_t}}{\alpha} + \frac{e_t}{2\sqrt{c_t d_t}}$ .
We prove Proposition E.1 and Proposition E.4 without using this Lemma and provide more details in the corresponding derivations.
Theorem B.8. For any  $t \in [T]$ , assume that  $\mathcal{L}_{\mu_t}(\cdot)$  has  $\alpha$ -quadratic growth (see Definition A.2), then whenever the base learner is SGD, SGLD, RLM, or the Gibbs algorithm, there exists  $f_t(\cdot)$  that gives
$$
R _ {e x c e s s} (\mathcal {A}, u _ {t}) \leq f _ {t} (u _ {t}) = \kappa_ {t} \Bigl (a \beta_ {t} + \frac {b \| \phi_ {t} - w _ {t} \| ^ {2} + \epsilon_ {t} + \epsilon_ {0}}{\beta_ {t}} + \Delta_ {t} \Bigr), \kappa_ {t}, \epsilon_ {t}, \beta_ {t}, \Delta_ {t} \in \mathbb {R} ^ {+}, a, b, \epsilon_ {0} > 0.
$$
The meta-parameter  $u_{t} = (\beta_{t},\phi_{t})$  decomposes into an initialization or bias  $\phi_t$  and a learning rate  $\beta_{t}$  (if  $\mathcal{A}\in \{SGD,SGLD\}$ ) or a regularization coefficient (if  $\mathcal{A}\in \{RLM,Gibbs\}$ ). Moreover,  $w_{t}$  denotes the (single) output for the (possibly randomized) base learner, and  $\epsilon_0\stackrel {\text{def}}{=}2\alpha ||w_t - \mathbb{E}W_t||^2$  characterizes the randomness of the output. Finally,  $a$  and  $b$  are constants, and  $\epsilon_t,\kappa_t,\Delta_t$  are functions of the task sample size  $m_t$  that characterize the base learner (see Appendix B and D).
Proof. According to Theorem B.3, Theorem B.4, Theorem B.5 and Theorem B.6, we have
$$
R _ {\text {e x c e s s}} \left(\operatorname {G i b b s}, \left(\eta_ {t}, \phi_ {t}\right)\right) \leq \frac {\eta_ {t}}{2 m _ {t}} + m _ {t} ^ {- 1 / 4} L ^ {1 / 2} d ^ {1 / 4} + \frac {d ^ {1 / 2} m _ {t} ^ {1 / 2} L}{2 \eta_ {t}} \| \phi_ {t} - w _ {t} ^ {*} \| ^ {2},
$$
$$
R _ {\text {e x c e s s}} (\mathbf {S G D}, (\eta_ {t}, \phi_ {t})) \leq \frac {\| \phi_ {t} - w _ {t} ^ {*} \| ^ {2}}{2 \eta_ {t} K _ {t}} + \left(\frac {L ^ {2}}{2} + \frac {L ^ {2} K _ {t}}{m _ {t}}\right) \eta_ {t},
$$
$$
R _ {\mathrm {e x c e s s}} (\mathrm {S G L D}, (\eta_ {t}, \phi_ {t})) \leq \frac {\| \phi_ {t} - w _ {t} ^ {*} \| ^ {2}}{2 \eta_ {t} K _ {t}} + \frac {\eta_ {t}}{2} L ^ {2} + \frac {d \sigma^ {2}}{2 \eta_ {t}} + \sqrt {\frac {K _ {t}}{4 m _ {t}}} \frac {\eta_ {t} L}{\sigma},
$$
$$
R _ {\text {e x c e s s}} \left(\operatorname {R L M}, \left(\eta_ {t}, \phi_ {t}\right)\right) \leq \frac {\left\| \phi_ {t} - w _ {t} ^ {*} \right\| ^ {2}}{\eta_ {t}} + \frac {2 L ^ {2} \eta_ {t}}{m _ {t}}.
$$
<sup>4</sup>We assumed quadratic growth for  $\ell (\cdot ,z),\forall z\in \mathcal{Z}$  in the submitted version, which is stronger than the current assumption and is unnecessary.
Combining Lemma B.7, we observe that  $b_{t} = 0$ ,  $c_{t} = \frac{d^{1/2}m_{t}^{1/2}L}{2}$ ,  $d_{t} = \frac{1}{2m_{t}}$ ,  $e_{t} = m_{t}^{-1/4}L^{1/2}d^{1/4}$  for Gibbs algorithm;  $b_{t} = 0$ ,  $c_{t} = \frac{1}{2K_{t}}$ ,  $d_{t} = \frac{L^{2}}{2} + \frac{L^{2}K_{t}}{m_{t}}$ ,  $e_{t} = 0$  for SGD;  $b_{t} = \frac{d\sigma^{2}}{2}$ ,  $c_{t} = \frac{1}{2K_{t}}$ ,  $d_{t} = \frac{L^{2}}{2} + \sqrt{\frac{K_{t}}{4m_{t}}} \frac{L}{\sigma}$ ,  $e_{t} = 0$  for SGLD; and  $b_{t} = 0$ ,  $c_{t} = 1$ ,  $d_{t} = \frac{2L^{2}}{m_{t}}$ ,  $e_{t} = 0$  for RLM. Therefore, simply adding the transformation  $\eta_{t} = \sqrt{c_{t}/d_{t}}\beta_{t}/\alpha + 4c_{t}/\alpha$  inside the algorithm, the algorithms mentioned above have a unified form of excess risk upper bound as defined in eq. (1).
B.6 Convexity of the unified upper bound
Lemma B.9.  $\forall t\in [T]$  , the cost function that has the following form:
$$
f _ {t} (u _ {t}) = f _ {t} (\beta_ {t}, \phi_ {t}) = \kappa_ {t} (a \beta_ {t} + \frac {b \| \phi_ {t} - w _ {t} \| ^ {2} + \epsilon_ {t} + \epsilon_ {0}}{\beta_ {t}} + \Delta_ {t}), \beta_ {t}, \kappa_ {t}, \epsilon_ {t}, \Delta_ {t} \in \mathbb {R} ^ {+}, a, b, \epsilon_ {0} > 0,
$$
is convex w.r.t  $u_{t} = (\beta_{t},\phi_{t})$  , where  $\phi_t,w_t\in \mathcal{W}\subseteq \mathbb{R}^d$
$$
\partial_ {\beta_ {t}} f _ {t} = \kappa_ {t} (a - \frac {b \| \phi_ {t} - w _ {t} \| ^ {2} + \epsilon_ {t} + \epsilon_ {0}}{\beta_ {t} ^ {2}})
$$
$$
\partial_ {\phi_ {t}} f _ {t} = \kappa_ {t} \frac {2 b (\phi_ {t} - w _ {t})}{\beta_ {t}}
$$
The Hessian matrix of  $f_{t}(\beta_{t},\phi_{t})$ :
$$
\nabla^ {2} f _ {t} = \left( \begin{array}{c c} 2 \kappa_ {t} \frac {b \| \phi_ {t} - w _ {t} \| ^ {2} + \epsilon_ {t} + \epsilon_ {0}}{\beta_ {t} ^ {3}} & \frac {- 2 b (\phi_ {t} - w _ {t})}{\beta_ {t} ^ {2}} \kappa_ {t} \\ \frac {- 2 b (\phi_ {t} - w _ {t})}{\beta_ {t} ^ {2}} \kappa_ {t} & \kappa_ {t} \frac {2 b}{\beta_ {t}} \mathbf {1} _ {d \times d} \end{array} \right)
$$
For any  $h = (h_0, h_d) \in \mathbb{R} \times \mathbb{R}^d$ , and  $\beta_t > 0, \phi \in \mathbb{R}^d$  we have:
$$
(\nabla^ {2} f (u _ {t}) h, h) = \kappa_ {t} \frac {2 (\epsilon_ {t} + \epsilon_ {0}) h _ {0} ^ {2} + 2 b \| h _ {0} (\phi_ {t} - w _ {t}) - \beta_ {t} h _ {d} \| ^ {2}}{\beta_ {t} ^ {3}} \geq 0
$$
, so  $f_{t}$  is convex w.r.t  $u_{t}$ .
Such transformation is not unique, e.g.,  $\beta_{t}$  can be scaled by  $\sqrt{\alpha}$ . It depends on the concrete implementation.
C Table of Notations
Table 2: Summary of major notations
Notation	Description
T	length of task sequence
N	number of slots (environments)
t	index of task at time t
n,k	k-th task insider n-th slot
Mn,M	number of tasks inside slot
mt, mn,k,m	number of examples for each task
τt	environment distribution
μt	task distribution
Z = X × Y	example space
A	base leaner/learning algorithm
W ⊂ Rd	hypothesis/model parameter space
wt, wn,k	task-model, algorithm output
U	meta-parameter space
ut, un,k	meta-parameter
u* = (β*, φ*)	optimal meta-parameter in hindsight
φt, φn,k	model initialization/bias
βt, βn,k	learning rate/regularization coefficient of A
γt, γn,k	learning rate of meta-parameter
ρ	hopping learning rate
Kt, Kn,k	number of gradient steps for SGD
D	diameter of W
D̂	diameter of all the algorithm outputs space Ŵ
D̂	diameter of all meta-parameters
V	variance of all the algorithm outputs
D̂n, D̂n	diameter in n-th slot
D̂max = max{D̂n}N=1	maximum of slot diameters
Vn	slot variance
Gn	maximum of cost function gradient norm in n-th slot
P*	path length
L	Lipschitz constant
α	quadratic-growth parameter
ε0	randomness of algorithm output
εv	parameter for controlling range of βt
B = [√(ε0/a), √(bD̂n2 + ε + ε0)/a]	range of βn,k, where ε0 = ε0 + εv
γn,0 = D̂n/Gn√k	theoretically optimal value
ρ = √(2P* D̂max/∑n=1N Gn2)	theoretically optimal hopping learning rate
D Missing Proofs in Section 5
D.1 AER bound in static environments
Algorithm 2 Meta-OGD (Static Environment)
Require: Convex set  $\mathcal{W},T,\phi_1 = \mathbf{0}\in \mathcal{W},\beta_1 > 0$  , initial learning rate  $\gamma$  for  $t\gets 1$  to  $T$  do Sample task distribution:  $\mu_t\sim \tau_t$  . Sample dataset  $S_{t}\sim \mu_{t}$  . Select meta parameter  $u_{t} = (\beta_{t},\phi_{t})\in \mathcal{U}$  . Learn base parameter  $w_{t} = \mathcal{A}(u_{t},S_{t})$  - Estimate excess risk upper bound  $f_{t}(u_{t})$  in Eq. (1); Update learning rate of the meta-parameter:  $\gamma_{t} = \gamma /\sqrt{t}$  Update the meta-parameter:  $u_{t + 1} = \Pi_{\mathcal{U}}(u_t - \gamma_t\nabla f_t(u_t)),\nabla f_t(u_t) = (a\kappa_t - \frac{b\kappa_t\|\phi_t - w_t\|^2 + \kappa_t\epsilon_t + \kappa_t\epsilon_0}{\beta_t^2},\frac{2b\kappa_t(\phi_t - w_t)}{\beta_t});$  i.e.:  $\phi_{t + 1} = (1 - \frac{2b\kappa_t\gamma_t}{\beta_t})\phi_t + \frac{2b\kappa_t\gamma_t}{\beta_t} w_t;\beta_{t + 1} = \beta_t - \gamma_t(a\kappa_t - \frac{\kappa_t(b\|\phi_t - w_t\|^2 + \epsilon_t + \epsilon_0)}{\beta_t^2});$  end
Theorem D.1 (AER bound in static environment). Let us consider the static environment. If the excess risk's upper bound of the base learner  $\mathcal{A}(\beta_t, \phi_t)$  can be formulated as the aforementioned unified cost function form in Lemma B.6. Then, running Meta-OGD (Algo. 2) as the meta learner, the AER is bounded as:
$$
\begin{array}{l} A E R _ {\mathcal {A}} ^ {T} \leq (2 \sqrt {a (b V ^ {2} + \epsilon + \epsilon_ {0})}) \sum_ {t = 1} ^ {T} \frac {\kappa_ {t}}{T} + \sum_ {t = 1} ^ {T} \frac {\kappa_ {t} \Delta_ {t}}{T} \\ + \frac {3 \kappa_ {m a x}}{2 \tilde {\epsilon} _ {0}} \sqrt {T} \left(\sqrt {(\hat {D} ^ {2} + \frac {b \hat {D} ^ {2} + \epsilon_ {m a x}}{a}) (4 a b ^ {2} \tilde {\epsilon} _ {0} \hat {D} ^ {2} + a ^ {2} (\epsilon_ {m a x} + b \hat {D} ^ {2}) ^ {2})}\right), \\ \end{array}
$$
where  $\hat{D}$  is the diameter of the base learner outputs,  $\phi^{*} = \sum_{t=1}^{T} \frac{\kappa_{t}}{\sum_{t=1}^{T} \kappa_{t}} w_{t}$  is the optimal initialization (or bias) in hindsight,  $V^{2} = \sum_{t=1}^{T} \frac{\kappa_{t}}{\sum_{t=1}^{T} \kappa_{t}} \| \phi^{*} - w_{t} \|^{2}$  is the variance of the base learner outputs, and  $\epsilon = \frac{\sum_{t=1}^{T} \kappa_{t} \epsilon_{t}}{\sum_{t=1}^{T} \kappa_{t}}$ ,  $\kappa_{max} = \max \left\{\kappa_{1:T}\right\}$ ,  $\epsilon_{max} = \max \left\{\epsilon_{1:T}\right\}$ ,  $\tilde{\epsilon}_{0} = \epsilon_{0} + \epsilon_{v}$ .  $\epsilon_{0}$  is the across tasks maximum corresponding term in eq.(1), and  $\epsilon_{v} \in \mathbb{R}^{+}$ . If  $\epsilon_{v} > bV^{2} + \epsilon$ , we replace  $\epsilon_{0}$  in the bound with  $\tilde{\epsilon}_{0}$ . Let  $\mathcal{B} \stackrel{\text{def}}{=} \left[\sqrt{\tilde{\epsilon}_{0}/a}, \sqrt{(b\hat{D}^{2} + \epsilon + \tilde{\epsilon}_{0})/a}\right]$ , then, the meta-parameter set in Algo. 2 is  $\mathcal{U} \stackrel{\text{def}}{=} \mathcal{B} \times \mathcal{W}$ .
Proof. At first, in a static environment, all the tasks share the same optimal prior information  $u^{*} = \arg \min_{u\in \mathcal{U}}\sum_{t = 1}^{T}f_{t}(u)$  (the comparator in static regret). As proved in Lemma B.6, the cost functions are convex, so the sum of  $T$  convex functions  $\sum_{t = 1}^{T}f_{t}(u)$  is convex. Then we have the minimum attained at  $u^{*} = (\beta^{*},\phi^{*})$ . By solving  $\min_u\sum_{t = 1}^T f_t(u)$ , we get  $\phi^{*} = \sum_{t = 1}^{T}\frac{\kappa_{t}}{\sum_{t = 1}^{T}\kappa_{t}} w_{t}$  and  $\beta^{*} = \sqrt{(bV^{2} + \epsilon + \epsilon_{0}) / a}$ , where  $\epsilon = \frac{\sum_{t = 1}^{T}\kappa_{t}\epsilon_{t}}{\sum_{t = 1}^{T}\kappa_{t}}$  and  $V^{2} = \sum_{t = 1}^{T}\frac{\kappa_{t}}{\sum_{t = 1}^{T}\kappa_{t}}\| \phi^{*} - w_{t}\|^{2}$ .
According to Algo.  $2,\phi_{t}$  is determined by the previous base learner outputs  $w_{1:t - 1}$  and the initialization  $\phi_1 = \mathbf{0}$ . As long as  $\frac{2b\kappa_t\gamma_t}{\beta_t}\leq 1,\forall t$  we have  $\phi_t$  is always constrained within the ball (of diameter  $\hat{D}$ ) that contains  $\hat{\mathcal{W}}$ . So we have  $0\leq \| \phi_t - w_t\| \leq \hat{D}$  and  $V^{2} = \sum_{t = 1}^{T}\frac{\kappa_{t}}{\sum_{t = 1}^{T}\kappa_{t}}\| \phi^{*} - w_{t}\|^{2}\leq \hat{D}^{2}$ . Moreover, we constrain  $\beta_{t}\in \left[\sqrt{\tilde{\epsilon}_{0} / a},\sqrt{(b\hat{D}^{2} + \epsilon + \tilde{\epsilon}_{0}) / a}\right]$ , where  $\tilde{\epsilon}_0 = \epsilon_0 + \epsilon_v$  is related to the randomness of the base learner output  $\epsilon_0$  and a factor  $\epsilon_v$  to control the range of  $\beta_{t}$ . We will see later that  $\epsilon_v$  can affect the gradient norm of the meta-parameter.
To derive the static regret bound, we need to use the following result first:
$$
\begin{array}{l} \left\| u _ {t + 1} - u ^ {*} \right\| ^ {2} = \left\| \Pi_ {\mathcal {U}} \left(u _ {t} - \gamma_ {t} \nabla f _ {t} (u _ {t})\right) - u ^ {*} \right\| \leq \left\| u _ {t} - \gamma_ {t} \nabla f _ {t} - u ^ {*} \right\| ^ {2} \\ \leq \left\| u _ {t} - u ^ {*} \right\| ^ {2} + \gamma_ {t} ^ {2} \left\| \nabla f _ {t} \right\| ^ {2} - 2 \gamma_ {t} \left\langle \nabla f _ {t}, \left(u _ {t} - u ^ {*}\right) \right\rangle . \\ \end{array}
$$
So we have  $\langle \nabla f_t,(u_t - u^*)\rangle \leq \frac{\|u_t - u^*\|^2 - \|u_{t + 1} - u^*\|^2}{2\gamma_t} +\frac{\gamma_t\|\nabla f_t\|^2}{2}$ .
Moreover, with  $\beta_{t}\in \left[\sqrt{\tilde{\epsilon}_{0} / a},\sqrt{(b\hat{D}^{2} + \epsilon + \tilde{\epsilon}_{0}) / a}\right]$ , we have
$$
| \partial_ {\beta_ {t}} f _ {t} | = | \kappa_ {t} (a - \frac {b \| \phi_ {t} - w _ {t} \| ^ {2} + \epsilon_ {t} + \epsilon_ {0}}{\beta_ {t} ^ {2}}) | \leq \frac {a \kappa_ {t} (b \hat {D} ^ {2} + \epsilon_ {\max})}{\tilde {\epsilon} _ {0}},
$$
$$
| \partial_ {\phi} f _ {t} | = \frac {2 b \kappa_ {t} \| \phi_ {t} - w _ {t} \|}{\beta_ {t}} \leq \frac {2 b \kappa_ {t} \hat {D}}{\sqrt {\tilde {\epsilon} _ {0} / a}}.
$$
Hence,  $\| \nabla f_t\| ^2\leq \frac{a^2\kappa_t^2(b\hat{D}^2 + \epsilon_{\max})^2}{\tilde{\epsilon}_0^2} +\frac{4ab^2\kappa_t^2\hat{D}^2}{\tilde{\epsilon}_0}$
In addition,  $\| u_{t} - u^{*}\|^{2} = \| \phi_{t} - \phi^{*}\|^{2} + |\beta_{t} - \beta^{*}|^{2}\leq \hat{D}^{2} + \frac{b\hat{D}^{2} + \epsilon_{\max}}{a}$
So the static regret is bounded by: