SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In *Psychology of learning and motivation*, volume 24, pages 109–165. Elsevier, 1989.
[2] Gail A Carpenter and Stephen Grossberg. A massively parallel architecture for a self-organizing neural pattern recognition machine. Computer vision, graphics, and image processing, 37(1): 54-115, 1987.
[3] Sachin Ravi and Alex Beatson. Amortized bayesian meta-learning. In International Conference on Learning Representations, 2018.
[4] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, 2017.
[5] Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. Continual learning: A comparative study on how to defy forgetting in classification tasks. arXiv preprint arXiv:1909.08383, 2(6):2, 2019.
[6] Chen Zeno, Itay Golan, Elad Hoffer, and Daniel Soudry. Task agnostic continual learning using online variational bayes. arXiv preprint arXiv:1803.10123, 2018.
[7] Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint arXiv:1810.11910, 2018.
[8] Xu He, Jakub Sygnowski, Alexandre Galashov, Andrei A Rusu, Yee Whye Teh, and Razvan Pascanu. Task agnostic continual learning via meta learning. arXiv preprint arXiv:1906.05201, 2019.
[9] Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta-learning. In International Conference on Machine Learning, pages 1920-1930. PMLR, 2019.
[10] Antreas Antoniou, Massimiliano Patacchiola, Mateusz Ochal, and Amos Storkey. Defining benchmarks for continual few-shot learning. arXiv preprint arXiv:2004.11967, 2020.
[11] Mengye Ren, Renjie Liao, Ethan Fetaya, and Richard Zemel. Incremental few-shot learning with attention attractor networks. Advances in neural information processing systems, 32, 2019.
[12] Sung Whan Yoon, Do-Yeon Kim, Jun Seo, and Jaekyun Moon. Xtarnet: Learning to extract task-adaptive representation for incremental few-shot learning. In International Conference on Machine Learning, pages 10852-10860. PMLR, 2020.
[13] Massimo Caccia, Pau Rodriguez, Oleksiy Ostapenko, Fabrice Normandin, Min Lin, Lucas Page-Caccia, Issam Hadj Laradji, Irina Rish, Alexandre Lacoste, David Vázquez, et al. Online fast adaptation and knowledge accumulation (osaka): a new approach to continual learning. Advances in Neural Information Processing Systems, 33:16532-16545, 2020.
[14] Ghassen Jerfel, Erin Grant, Tom Griffiths, and Katherine A Heller. Reconciling meta-learning and continual learning with online mixtures of tasks. Advances in Neural Information Processing Systems, 32, 2019.
[15] Qiang Zhang, Jinyuan Fang, Zaiqiao Meng, Shangsong Liang, and Emine Yilmaz. Variational continual bayesian meta-learning. Advances in Neural Information Processing Systems, 34: 24556-24568, 2021.
[70] Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient-based meta-learning as hierarchical bayes. In International Conference on Learning Representations, 2018.
[71] Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. In Advances in Neural Information Processing Systems, pages 7332–7342, 2018.
[72] Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. Advances in neural information processing systems, 32, 2019.
[73] Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7201-7211, 2022.
[74] Qi Chen and Mario Marchand. Algorithm-dependent bounds for representation learning of multi-source domain adaptation. In International Conference on Artificial Intelligence and Statistics, pages 10368-10394. PMLR, 2023.
[75] Changjian Shui, Qi Chen, Jun Wen, Fan Zhou, Christian Gagné, and Boyu Wang. A novel domain adaptation theory with jensen-shannon divergence. Knowledge-Based Systems, 257: 109808, 2022.
A Preliminaries 17
A.1 Definitions 17
A.2 Online Convex Optimization (OCO) 17
B Missing Proofs in Section 3 19
B.1 Excess risk bound for Gibbs algorithm 19
B.2 Excess risk bound for SGD 21
B.3 Excess risk bound for SGLD 23
B.4 Excess risk bound for Regularized Loss Minimization (RLM) 24
B.5 Unified form of excess risk upper bound 25
B.6 Convexity of the unified upper bound 26
C Table of Notations 27
D Missing Proofs in Section 5 28
D.1 AER bound in static environments 28
D.2 AER bound in possibly shifting environments (Proof of Theorem 5.1) 31
E Missing Proofs in Section 6 35
E.1 Cost function for Gibbs base learner 35
E.2 Static AER of Gibbs 37
E.3 Proof of Theorem 6.1 (Dynamic AER of Gibbs) 38
E.4 Cost function for SGD base leaner 39
E.5 Static AER of SGD 40
E.6 Proof of Theorem 6.2 (Dynamic AER of SGD) 41
F Experiments 43
F.1 Experimental settings 43
F.2 Additional results 45
F.3 Experimental details 47
F.4 Bayes Online Changing-point Detection 48
G Additional Discussion 50
G.1 Learning-forgetting w.r.t meta-updates 50
G.2 Real-world examples for practicality and necessity of CML 50
G.3 More details on DCML 50
G.4 Limitations 51
G.5Broader Impacts 51
H Additional Related Works 52
H.1 Summary and comparison of related settings 52
H.2 Continual Learning 52
H.3 Meta-Continual Learning 53
H.4 ContinuaMeta-Learning 53
H.5 Meta-Learning 54
In the appendix, we assume that all sets are convex subsets of  $\mathbb{R}^d$ , and we use  $\|\cdot\|$  to denote the Euclidean norm.
A.1 Definitions
Definition A.1 (Lipschitzness). A function  $f: \mathcal{W} \to \mathbb{R}$  is  $L$ -Lipschitz w.r.t norm  $\| \cdot \|$  if for all  $w_1, w_2 \in \mathcal{W}$ ,  $|f(w_1) - f(w_2)| \leq L \| w_1 - w_2 \|$
Definition A.2 (Quadratic Growth). A function  $f: \mathcal{W} \to \mathbb{R}$  has  $\alpha$ -quadratic-growth w.r.t  $\| \cdot \|$  for  $\alpha > 0$  if for any  $w \in \mathcal{W}$ , we have:
$$
\frac {\alpha}{2} \| w - w ^ {*} \| ^ {2} \leq f (w) - f (w ^ {*}),
$$
where  $w^{*}$  denotes the global minimum point of  $f$  which is closest to  $w$ .
Definition A.3 (Convex). A function  $f: \mathcal{W} \to \mathbb{R}$  is convex w.r.t norm  $\| \cdot \|$  if  $f$  is everywhere sub-differentiable and if  $\forall x, y \in \mathcal{W}$ ,
$$
f (y) \geq f (x) + \langle \nabla f (x), y - x \rangle ,
$$
where  $\nabla f(x)$  is a subgradient of  $f$  at  $x$ .
Definition A.4 (Strongly Convex). A function  $f: \mathcal{W} \to \mathbb{R}$  is  $\alpha$ -strongly convex w.r.t norm  $\| \cdot \|$  if  $f$  is everywhere sub-differentiable and if  $\forall x, y \in \mathcal{W}$ ,
$$
f (y) \geq f (x) + \langle \nabla f (x), y - x \rangle + \frac {\alpha}{2} \| y - x \| ^ {2},
$$
Definition A.5 (Smoothness). A function  $f: \mathcal{W} \to \mathbb{R}$  is  $\beta$ -smooth w.r.t norm  $\| \cdot \|$  if  $f$  is everywhere sub-differentiable and if  $\forall x, y \in \mathcal{W}$ ,
$$
f (y) \leq f (x) + \langle \nabla f (x), y - x \rangle + \frac {\beta}{2} \| y - x \| ^ {2},
$$
Definition A.6 (Bregman divergence). Let  $R: \mathcal{U} \to \mathbb{R}$  be an everywhere sub-differentiable strongly convex regularization function. Then the Bregman divergence w.r.t function  $R$  for any  $x, y \in \mathcal{U}$  is defined as:
$$
B _ {R} (x \| y) \stackrel {\text {d e f}} {=} R (x) - R (y) - \langle \nabla R (y), (x - y) \rangle .
$$
A.2 Online Convex Optimization (OCO)
A.2.1 Definitions of regrets
Definition A.7 (Static regret). The static regret of an OCO algorithm on the action set  $\mathcal{U}$  w.r.t a sequence of cost functions  $\{f_t:\mathcal{U}\to \mathbb{R}\}_{t = 1}^T$  is defined as:
$$
R _ {T} \stackrel {\text {d e f}} {=} \sum_ {t = 1} ^ {T} f _ {t} (u _ {t}) - \min  _ {u \in \mathcal {U}} \sum_ {t = 1} ^ {T} f _ {t} (u)
$$
Definition A.8 (Traditional dynamic regret [40]). The dynamic regret of an OCO algorithm on the action set  $\mathcal{U}$  w.r.t a sequence of cost functions  $\{f_t: \mathcal{U} \to \mathbb{R}\}_{t=1}^T$  and the comparator sequence  $\psi_{1:T}, \forall \psi_t \in \mathcal{U}$  is defined as:
$$
R _ {T} \stackrel {\text {d e f}} {=} \sum_ {t = 1} ^ {T} f _ {t} (u _ {t}) - \sum_ {t = 1} ^ {T} f _ {t} (\psi_ {t})
$$
A.2.2 Static regret may be vacuous in shifting environments
Proof. For a shifting environment that has  $u_{T}^{*} = \arg \min_{u}\frac{1}{T}\sum_{t = 1}^{T}f_{t}(u)$  changing with  $T$ , we can prove that the static regret may be vacuous as  $T$  goes to infinity.
Let us denote  $F_{T} = \sum_{t = 1}^{T}f_{t}(u_{T}^{*}),F_{T - 1} = \sum_{t = 1}^{T - 1}f_{t}(u_{T - 1}^{*}),F_{0} = 0.$
Assume that the cost functions are non-negative, which is usually the case. Whenever we have  $f_{t}(u_{T}^{*}) - f_{t}(u_{T - 1}^{*}) \geq a > 0$ , we can obtain
$$
F _ {T} - F _ {T - 1} = \sum_ {t = 1} ^ {T - 1} \left(f _ {t} \left(u _ {T} ^ {*}\right) - f _ {t} \left(u _ {T - 1} ^ {*}\right)\right) + f _ {T} \left(u _ {T} ^ {*}\right) \geq a (T - 1) + f _ {T} \left(u _ {T} ^ {*}\right).
$$
The above inequality holds for any  $T \geq 1$ , so we can rewrite it as  $F_{t} - F_{t-1} \geq a(t-1) + f_{t}(u_{t}^{*})$ . By summing each side from  $t = 1$  to  $t = T$ , we have  $F_{T} \geq F_{0} + \frac{aT(T-1)}{2} + \sum_{t=1}^{T} f_{t}(u_{t}^{*}) \geq \frac{aT(T-1)}{2}$ . Consequently, we have  $\frac{1}{T} F_{T} = \frac{1}{T} \sum_{t=1}^{T} f_{t}(u_{T}^{*}) \geq \frac{a(T-1)}{2}$ , which means the static regret will be vacuous when  $T \to \infty$ .
A.2.3 Online algorithms
Definition A.9 (Follow The Leader (FTL)). Given a sequence of strongly convex cost functions  $\{f_t: \mathcal{U} \to \mathbb{R}\}_{t \geq 1}$ , FTL plays for  $t > 1$ :
$$
u_{t} = \operatorname *{arg  min}_{u\in \mathcal{U}}\sum_{i = 1}^{t - 1}f_{i}(u).
$$
Definition A.10 (Follow The Regularized Leader (FTRL)). Given a sequence of convex cost functions  $\{f_t : \mathcal{U} \to \mathbb{R}\}_{t \geq 1}$ , a strongly convex regularization function  $R : \mathcal{U} \to \mathbb{R}$ , and the starting point  $u_1 = \operatorname*{arg\min}_{u \in \mathcal{U}} R(u)$ , FTRL plays for  $t > 1$ :
$$
u_{t} = \operatorname *{arg  min}_{u\in \mathcal{U}}\sum_{i = 1}^{t - 1}f_{i}(u) + R(u).
$$
Khodak et al. [31] write FTRL as  $u_{t} = \arg \min_{u\in \mathcal{U}}\sum_{i = 1}^{t - 1}f_{i}(u) + B_{R}(u\| u_{1})$ , while we follow Shalev-Shwartz et al. [41] and Hazan et al. [40].
Definition A.11 (Online Mirror Descent (OMD)). Given a sequence of convex cost functions  $\{f_t: \mathcal{U} \to \mathbb{R}\}_{t \geq 1}$ , a strongly convex regularization function  $R: \mathcal{U} \to \mathbb{R}$ , and  $u_1 = \nabla R^*(0)$ , OMD plays for  $t > 1$ :
$$
u _ {t} = \nabla R ^ {*} (- \eta \sum_ {i = 1} ^ {t - 1} \nabla f _ {i} (u _ {i}))
$$
where  $R^{*}(\tilde{u}) = \max_{u}\langle u,\tilde{u}\rangle -R(u)$  is the Fenchel conjugate of  $R(u)$ .
There exists an equivalent linearized version of the above FTRL and lazy OMD through the linearization of the convex cost functions.
Definition A.12 (Lazy OMD). Given a sequence of convex cost functions  $\{f_t: \mathcal{U} \to \mathbb{R}\}_{t \geq 1}$ , a strongly convex regularization function  $R: \mathcal{U} \to \mathbb{R}$ , and  $\nabla R(\tilde{u}_1) = 0$ ,  $u_1 = \operatorname*{arg min}_{u \in \mathcal{U}} B_R(u \| \tilde{u}_1)$ ,
then lazy OMD and FTRL play for  $t > 1$ :
$$
\tilde {u} _ {t} = \underset {u} {\arg \min } \left\langle \eta \nabla f _ {t - 1} \left(u _ {t - 1}\right), u \right\rangle + B _ {R} \left(u \| u _ {t - 1}\right)
$$
$$
u_{t} = \operatorname *{arg  min}_{u\in \mathcal{U}}B_{R}(u\| \tilde{u}_{t}).
$$
The proof of the equivalence can be found in Hazan et al. [40].
B Missing Proofs in Section 3
Definition B.1. Consider any base learner  $\mathcal{A}$  that takes a fixed prior information  $u$  with a data set  $S \sim \mu^m$  as input and outputs  $W = \mathcal{A}(S, u) \sim P_{W|S,u}$ . Define the expected generalization gap of algorithm  $\mathcal{A}$  as
$$
\operatorname {g e n} (\mu , \mathcal {A}) \stackrel {\mathrm {d e f}} {=} \mathbb {E} _ {\mathcal {A}, S} [ \mathcal {L} _ {\mu} (\mathcal {A} (S, u)) - \mathcal {L} _ {S} (\mathcal {A} (S, u)) ] = \mathbb {E} _ {W, S} [ \mathcal {L} _ {\mu} (W) - \mathcal {L} _ {S} (W) ]
$$
and the expected excess risk as
$$
R _ {\text {e x c e s s}} (\mathcal {A}, u) \stackrel {{\text {d e f}}} {{=}} \mathbb {E} _ {\mathcal {A}, S} \left[ \mathcal {L} _ {\mu} (\mathcal {A} (S, u)) - \mathcal {L} _ {\mu} (w ^ {*}) \right] = \mathbb {E} _ {W, S} \left[ \mathcal {L} _ {\mu} (W) - \mathcal {L} _ {\mu} (w ^ {*}) \right],
$$
where  $w^{*} = \arg \min_{w\in \mathcal{W}}\mathcal{L}_{\mu}(w)$  is the hypothesis that achieves the minimum true risk.
B.1 Excess risk bound for Gibbs algorithm
Lemma B.2. Let  $Q$  be an arbitrary distribution on  $\mathcal{W}$  and let  $\beta > 0$  be the inverse temperature that balances fitting and generalization. Then, we can jointly denote  $u = (\beta, Q)$ . Let  $S \sim \mu^m$  be a sample of examples. The solution to the optimization problem
$$
P _ {W | S, u} ^ {*} = \arg \inf _ {P _ {W | S}} \left\{\mathbb {E} _ {W, S} [ L _ {S} (W) ] + \frac {1}{\beta} \mathbb {E} _ {S} D _ {K L} (P _ {W | S} \| Q) \right\}
$$
is given by the Gibbs algorithm which satisfies
$$
d P _ {W | S, u} ^ {*} (w) = \frac {e ^ {- \beta L _ {S} (w)} d Q (w)}{\mathbb {E} _ {W \sim Q} \left[ e ^ {- \beta L _ {S} (W)} \right]}.
$$
See Proof of Theorem 5 in [48].
Theorem B.3 (Excess risk bound for Gibbs algorithm of meta-parameter  $u = (\beta, \phi)$ ). Suppose  $\mathcal{W} = \mathbb{R}^d$  and assume that  $\ell(\cdot, z) \in [0,1]$  is  $L$ -Lipschitz for all  $z \in \mathcal{Z}$ . Let  $W_g \sim P_{W_g|S,u}$  denote the output of the Gibbs algorithm applied on data set  $S$  with meta-parameter  $u = (\beta, Q)$ . Let  $w^*$  be the hypothesis that achieves the minimum true risk among  $\mathcal{W}$ . The excess risk of  $W_g$  satisfies
$$
\begin{array}{l} R _ {e x c e s s} (G i b b s, u) \stackrel {{d e f}} {=} \mathbb {E} [ \mathcal {L} _ {\mu} (W _ {g}) ] - \inf  _ {w \in \mathcal {W}} \mathcal {L} _ {\mu} (w) \\ \leq \frac {\beta}{2 m} + \inf _ {\sigma > 0} \left(\sigma L \sqrt {d} + \frac {1}{\beta} D _ {K L} (\mathcal {N} (w ^ {*}, \sigma^ {2} \mathbf {1} _ {d})) \| Q)\right). \\ \end{array}
$$
Specifically, if we assume  $Q$  is the Gaussian distribution  $\mathcal{N}(\phi, \sigma^2\mathbf{1}_d)$  with  $\sigma = m^{-1/4}d^{-1/4}L^{-1/2}$  and denote  $u = (\beta, \phi)$ , then we have
$$
R _ {e x c e s s} (G i b b s, u) \leq \frac {\beta}{2 m} + m ^ {- 1 / 4} L ^ {1 / 2} d ^ {1 / 4} + \frac {d ^ {1 / 2} m ^ {1 / 2} L}{2 \beta} \| \phi - w ^ {*} \| ^ {2}.
$$
Proof. The original proof can be found in Corollary 3 [48], we provide the proof below with more details and some tiny modifications. Consider an arbitrary data-free distribution related to the unknown minimum  $w^{*}$  of the true risk, i.e.,  $\mathcal{N}(w^{*},\sigma^{2}\mathbf{1}_{d})$ . Then for any  $\sigma >0$ , we have:
$$
\begin{array}{l} \mathbb {E} _ {W _ {g}, S} \left[ \mathcal {L} _ {\mu} \left(W _ {g}\right) \right] = \mathbb {E} _ {W _ {g}, S} \left[ \mathcal {L} _ {S} \left(W _ {g}\right) \right] + \operatorname {g e n} (\mu , \text {G i b b s}) \\ \leq \mathbb {E} _ {W _ {g}, S} \left[ \mathcal {L} _ {S} \left(W _ {g}\right) \right] + \frac {\beta}{2 m} \\ \leq \mathbb {E} _ {W _ {g}, S} \left[ \mathcal {L} _ {S} \left(W _ {g}\right) \right] + \frac {1}{\beta} \mathbb {E} _ {S} D _ {\mathrm {K L}} \left(P _ {W _ {g} | S, u} \| Q\right) + \frac {\beta}{2 m} \\ \leq \int_ {\mathcal {W}} \mathbb {E} _ {S} \mathcal {L} _ {S} (w) \mathcal {N} (w; w ^ {*}, \sigma^ {2} \mathbf {1} _ {d}) d w + \frac {1}{\beta} D _ {K L} (\mathcal {N} (w ^ {*}, \sigma^ {2} \mathbf {1} _ {d})) \| Q) + \frac {\beta}{2 m} \\ = \int_ {\mathcal {W}} \mathcal {L} _ {\mu} (w) \mathcal {N} (w; w ^ {*}, \sigma^ {2} \mathbf {1} _ {d}) d w + \frac {1}{\beta} D _ {K L} (\mathcal {N} (w ^ {*}, \sigma^ {2} \mathbf {1} _ {d})) \| Q) + \frac {\beta}{2 m} \\ \end{array}
$$
<sup>3</sup>In some definitions, the output of the Gibbs algorithm is the optimal distribution itself. Here, we consider a sampling from this distribution.
The first step is obtained with previous work's results that Gibbs algorithm is  $(\frac{2\beta}{m},0)$ -differentially private [49]. The second inequality makes use of the non-negativity of KL divergence. The last inequality derives naturally from the fact that the output of the Gibbs algorithm follows the optimal distribution for the objective function in Lemma B.2.
Since  $\ell (.,z)$  is  $L$ -Lipschitz for all  $z\in \mathcal{Z}$ , we can obtain the following by combining the Jensen's inequality:
$$
\left| \mathcal {L} _ {\mu} (w) - \mathcal {L} _ {\mu} \left(w ^ {*}\right) \right| \leq \mathbb {E} _ {Z \sim \mu} [ \left| \ell \left(w, Z\right) - \ell \left(w ^ {*}, Z\right) \right| ] \leq L \| w - w ^ {*} \|, \forall w \in \mathcal {W}.
$$
$$
\begin{array}{l} \int_ {\mathcal {W}} \mathcal {L} _ {\mu} (w) \mathcal {N} (w; w ^ {*}, \sigma^ {2} \mathbf {1} _ {d}) d w \leq \int_ {\mathcal {W}} (\mathcal {L} _ {\mu} (w ^ {*}) + L \| w - w ^ {*} \|) \mathcal {N} (w; w ^ {*}, \sigma^ {2} \mathbf {1} _ {d}) d w \\ \leq \mathcal {L} _ {\mu} (w ^ {*}) + L \sigma \sqrt {d}. \\ \end{array}
$$
So we have for any  $\sigma > 0$ ,
$$
\begin{array}{l} \mathbb {E} \left[ \mathcal {L} _ {\mu} \left(W _ {g}\right) \right] - \inf  _ {w \in \mathcal {W}} \mathcal {L} _ {\mu} (w) \leq \frac {\beta}{2 m} + \inf  _ {\sigma > 0} \left(\sigma L \sqrt {d} + \frac {1}{\beta} D _ {K L} \left(\mathcal {N} \left(w ^ {*}, \sigma^ {2} \mathbf {1} _ {d}\right)\right) \| Q\right) \\ = \frac {\beta}{2 m} + \inf  _ {\sigma > 0} \left(\sigma L \sqrt {d} + \frac {1}{\beta} D _ {K L} \left(\mathcal {N} \left(w ^ {*}, \sigma^ {2} \mathbf {1} _ {d}\right)\right) \| \mathcal {N} \left(\phi , \sigma^ {2} \mathbf {1} _ {d}\right)\right) \\ = \frac {\beta}{2 m} + \inf  _ {\sigma > 0} \left(\sigma L \sqrt {d} + \frac {1}{2 \beta \sigma^ {2}} \| \phi - w ^ {*} \| ^ {2}\right) \\ \leq \frac {\beta}{2 m} + m ^ {- 1 / 4} L ^ {1 / 2} d ^ {1 / 4} + \frac {d ^ {1 / 2} m ^ {1 / 2} L}{2 \beta} \| \phi - w ^ {*} \| ^ {2} \\ \end{array}
$$
The last three lines are obtained with the assumption that  $Q$  is the Gaussian distribution  $\mathcal{N}(\phi, \sigma^2\mathbf{1}_d)$  and setting  $\sigma = m^{-1/4}d^{-1/4}L^{-1/2}$ .
B.2 Excess risk bound for SGD
Theorem B.4 (Excess Risk Bound for SGD of meta-parameter  $u = (\eta, \phi)$ ). Suppose  $\mathcal{W} = \mathbb{R}^d$ , let  $w^* \stackrel{\text{def}}{=} \arg \min_{w \in \mathcal{W}} \mathcal{L}_\mu(w)$  be the hypothesis that achieves the minimum population risk in  $\mathcal{W}$ . Consider the SGD algorithm with  $K$  updates starting from  $W_1 = \phi$ :  $W_{k+1} = W_k - \eta \nabla \ell(W_k, Z_k)$ , where  $Z_k$  is a sample randomly selected from data set  $S$ . Assume the loss function  $\ell(\cdot, z)$  is convex,  $\beta$ -smooth and  $L$ -Lipschitz for all  $z \in \mathcal{Z}$ . Let the output of the algorithm be  $W = \frac{1}{K} \sum_{k=1}^{K} W_k$ . Then, the excess risk bound is given by:
$$
R _ {e x c e s s} (S G D, u) \stackrel {{d e f}} {=} \mathbb {E} _ {W, S} [ \mathcal {L} _ {\mu} (W) ] - \inf  _ {w \in \mathcal {W}} \mathcal {L} _ {\mu} (w) \leq \frac {\| \phi - w ^ {*} \| ^ {2}}{2 \eta K} + (\frac {L ^ {2}}{2} + \frac {L ^ {2} K}{m}) \eta .
$$
Proof. The excess risk can be decomposed as
$$
\begin{array}{l} \mathbb {E} _ {W, S} \left[ \mathcal {L} _ {\mu} (W) - \min  _ {w} \mathcal {L} _ {\mu} (w) \right] = \mathbb {E} _ {W, S} \left[ \mathcal {L} _ {\mu} (W) - \mathcal {L} _ {S} (W) \right] + \mathbb {E} _ {W, S} \mathcal {L} _ {S} (W) - \min  _ {w} \mathcal {L} _ {\mu} (w) \\ = \mathbb {E} _ {W, S} \left[ \mathcal {L} _ {\mu} (W) - \mathcal {L} _ {S} (W) \right] + \left[ \mathbb {E} _ {W, S} \mathcal {L} _ {S} (W) - \mathbb {E} _ {S} \mathcal {L} _ {S} \left(w ^ {*}\right) \right] \\ = \operatorname {g e n} (\mu , \operatorname {S G D}) + \mathbb {E} _ {W, S} \mathcal {L} _ {S} (W) - \mathbb {E} _ {S} \mathcal {L} _ {S} \left(w ^ {*}\right). \\ \end{array}
$$
The first term is the generalization gap defined in Definition B.1. Note that the second term is not the optimization error  $\epsilon_{opt}^{W}\stackrel {\mathrm{def}}{=}|\mathbb{E}_{W,S}[\mathcal{L}_S(W)] - \mathbb{E}_{W,S}[\mathcal{L}_S(w_S^*)]|$  used in [38], which is the expected gap between the SGD output  $W$  and the ERM output  $w_{S}^{*}$ . The optimization error indicates how close the SGD output can approximate the possibly intractable exact ERM solution, which is an upper bound of the absolute value of the second term  $|\mathbb{E}_{W,S}\mathcal{L}_S(W) - \mathbb{E}_S[\mathcal{L}_S(w^*)]|\leq \epsilon_{opt}^{W}$  (can be derived with Lemma 5.1 in [50]).
Here, we directly make use of the convexity and obtain the gap compared to the true minimum. It's easy to bound  $\mathbb{E}_{W,S}\mathcal{L}_S(W) - \mathbb{E}_S\mathcal{L}_S(w^*)$  using the method provided in Shalev-Shwartz and Ben-David [51] with some tiny modifications.
Since we have  $W_{k + 1} = W_k - \eta \nabla \ell (W_k,Z_k)$ ,  $W_{1:K}$  is related to the randomness introduced by the sampling path  $Z_{1:K}$  from  $S$ . Moreover, for the convex loss function, we have:
$$
\begin{array}{l} \mathbb {E} _ {W, S} \mathcal {L} _ {S} (W) - \mathbb {E} _ {S} \mathcal {L} _ {S} \left(w ^ {*}\right) = \mathbb {E} _ {S} \left[ \mathbb {E} _ {W \sim P _ {W | S, u}} \mathcal {L} _ {S} (W) - \mathcal {L} _ {S} \left(w ^ {*}\right) \right] \\ = \mathbb {E} _ {S} \left[ \mathbb {E} _ {W _ {1: K}} \mathcal {L} _ {S} \left(\frac {1}{K} \sum_ {k = 1} ^ {K} W _ {k}\right) - \mathcal {L} _ {S} \left(w ^ {*}\right) \right] \\ \leq \mathbb {E} _ {S} \left[ \mathbb {E} _ {W _ {1: K}} \frac {1}{K} \sum_ {k = 1} ^ {K} \mathcal {L} _ {S} \left(W _ {k}\right) - \mathcal {L} _ {S} \left(w ^ {*}\right) \right] \\ \leq \mathbb {E} _ {S} \left[ \mathbb {E} _ {W _ {1: K}} \frac {1}{K} \sum_ {k = 1} ^ {K} \langle W _ {k} - w ^ {*}, \nabla \mathcal {L} _ {S} (W _ {k}) \rangle \right] \\ = \mathbb {E} _ {S} \left[ \mathbb {E} _ {V _ {1: K - 1}} \frac {1}{K} \sum_ {k = 1} ^ {K} \langle W _ {k} - w ^ {*}, \nabla \mathcal {L} _ {S} (W _ {k}) \rangle \right] \\ = \mathbb {E} _ {S} \left[ \frac {1}{K} \sum_ {k = 1} ^ {K} \mathbb {E} _ {V _ {1: k - 1}} \langle W _ {k} - w ^ {*}, \nabla \mathcal {L} _ {S} (W _ {k}) \rangle \right], \\ \end{array}
$$
where  $V_{k} = \nabla \ell (W_{k},Z_{k})$ . The inequalities are obtained by using Jensen's inequality and the last two equalities are obtained with the fact that  $W_{1:K}$  is determined by  $V_{1:K - 1}$ .
Moreover, we have  $\mathbb{E}[V_k|W_k] = \mathbb{E}[V_k|V_{1:k - 1}] = \nabla \mathcal{L}_S(W_k)$ . The first equality comes from the update rule, where  $W_{k}$  is determined by the previous  $k - 1$  gradients and the meta parameter  $u = (\eta ,\phi)$ . Note that  $u$  is not a random variable and hence no randomness needs to be considered. The second equality holds because  $Z_{k}$  is uniformly sampled from  $S$ , so  $V_{k}$  is an unbiased estimator of  $\nabla \mathcal{L}_S(W_k)$ .
$$
\begin{array}{l} \mathbb {E} _ {V _ {1: K}} \left(\frac {1}{K} \sum_ {k = 1} ^ {K} \langle W _ {k} - w ^ {*}, V _ {k} \rangle\right) = \frac {1}{K} \sum_ {k = 1} ^ {K} \mathbb {E} _ {V _ {1: K}} [ \langle W _ {k} - w ^ {*}, V _ {k} \rangle ] \\ = \frac {1}{K} \sum_ {k = 1} ^ {K} \mathbb {E} _ {V _ {1: k}} [ \langle W _ {k} - w ^ {*}, V _ {k} \rangle ] \\ = \frac {1}{K} \sum_ {k = 1} ^ {K} \mathbb {E} _ {V _ {1: k - 1}} \mathbb {E} _ {V _ {k}} [ \langle W _ {k} - w ^ {*}, V _ {k} \rangle | V _ {1: k - 1} ] \\ = \frac {1}{K} \sum_ {k = 1} ^ {K} \mathbb {E} _ {V _ {1: k - 1}} [ \langle W _ {k} - w ^ {*}, \mathbb {E} _ {V _ {k}} [ V _ {k} | V _ {1: k - 1} ] \rangle ] \\ = \frac {1}{K} \sum_ {k = 1} ^ {K} \mathbb {E} _ {V _ {1: k - 1}} [ \langle W _ {k} - w ^ {*}, \nabla \mathcal {L} _ {S} (W _ {k}) \rangle ] \\ \end{array}
$$
By using Lemma14.1 in Shalev-Shwartz and Ben-David [51], we can obtain that