SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proposition E.1. Suppose  $\mathcal{W} \subset \mathbb{R}^d$ , let  $w_t^*$  be the hypothesis that achieves the minimum population risk among  $\mathcal{W}$ . Suppose  $\ell \in [0,1]$ , and  $\ell(\cdot, z)$  is  $L$ -Lipschitz  $\forall z \in \mathcal{Z}$ . Assume that  $\mathcal{L}_{\mu_t}(\cdot)$  has  $\alpha$ -quadratic-growth for all  $t \in [T]$ . Let  $w_t = \mathcal{A}(\phi_t, \beta_t)$  denote the output of the Gibbs algorithm applied on dataset  $S_t$ , and  $Q_t$  be the prior distribution and  $\beta_t$  the inverse temperature of  $t$ -th task. Assume  $Q_t$  is a gaussian distribution  $\mathcal{N}(\phi_t, \sigma_t^2 \mathbf{1}_d)$  with  $\sigma_t = m_t^{-1/4} d^{-1/4} L^{-1/2}$ . To meet the general form of the cost function, we replace  $\beta_t$  with  $\beta_t'$  by  $\alpha \beta_t - 2\sqrt{dm_t} L = m_t^{\frac{3}{4}} d^{\frac{1}{4}} \sqrt{L} \beta_t'$ . Hence, the upper bound of the excess risk of the Gibbs algorithm is given by
$$
f _ {t} (\beta_ {t} ^ {\prime}, \phi_ {t}) = \left(\frac {d ^ {\frac {1}{4}} \sqrt {L}}{m _ {t} ^ {\frac {1}{4}}}\right) \left(\frac {\beta_ {t} ^ {\prime}}{2 \alpha} + 1 + \frac {2 d ^ {\frac {1}{4}} \sqrt {L}}{\alpha m _ {t} ^ {\frac {1}{4}}} + \frac {2 \alpha \| \phi_ {t} - w _ {t} \| ^ {2} + \epsilon_ {0} + \frac {2}{\alpha} \sqrt {\frac {d}{m _ {t}}} L + 2 \frac {\sqrt {L} d ^ {\frac {1}{4}}}{m _ {t} ^ {\frac {1}{4}}}}{\beta_ {t} ^ {\prime}}\right).
$$
Proof. From Theorem B.3, we have:
$$
R _ {\text {e x c e s s}} \left(\operatorname {G i b b s}, u _ {t}\right) \leq \frac {\beta_ {t}}{2 m _ {t}} + m _ {t} ^ {- 1 / 4} L ^ {1 / 2} d ^ {1 / 4} + \frac {d ^ {1 / 2} m _ {t} ^ {1 / 2} L}{2 \beta_ {t}} \left\| \phi_ {t} - w _ {t} ^ {*} \right\| ^ {2}.
$$
Let  $\bar{w}_t = \mathbb{E}W_t$  be the expected output of the Gibbs algorithm, and let  $\epsilon_0 = 2\alpha \| w_t - \bar{w}_t\|^2$  be the randomness of the algorithm, which should be a small. Then, using Titu's lemma and the  $\alpha$ -quadratic growth assumption on the loss function, we get
$$
\begin{array}{l} \left\| \phi_ {t} - w _ {t} ^ {*} \right\| ^ {2} = \left\| \phi_ {t} - \bar {w} _ {t} + \bar {w} _ {t} - w _ {t} ^ {*} \right\| ^ {2} \\ \leq 2 \| \phi_ {t} - \bar {w} _ {t} \| ^ {2} + 2 \| \bar {w} _ {t} - w _ {t} ^ {*} \| ^ {2} \\ \leq 2 \| \phi_ {t} - \bar {w} _ {t} \| ^ {2} + 2 \| \mathbb {E} W _ {t} - w _ {t} ^ {*} \| ^ {2} \\ \leq 2 \| \phi_ {t} - \bar {w} _ {t} \| ^ {2} + 2 \mathbb {E} \| W _ {t} - w _ {t} ^ {*} \| ^ {2} \\ \leq 2 \| \phi_ {t} - \bar {w} _ {t} \| ^ {2} + \frac {4 \mathbb {E} \left[ \mathcal {L} _ {\mu_ {t}} \left(W _ {t}\right) - \mathcal {L} _ {\mu_ {t}} \left(w _ {t} ^ {*}\right) \right]}{\alpha} \\ \leq 2 \| \phi_ {t} - \bar {w} _ {t} \| ^ {2} + \frac {4 \left(\frac {\beta_ {t}}{2 m _ {t}} + m _ {t} ^ {- 1 / 4} L ^ {1 / 2} d ^ {1 / 4} + \frac {d ^ {1 / 2} m _ {t} ^ {1 / 2} L}{2 \beta_ {t}} \| \phi_ {t} - w _ {t} ^ {*} \| ^ {2}\right)}{\alpha}. \\ \end{array}
$$
Then, we have  $\| \phi_t - w_t^*\|^2 \leq \frac{2\alpha\|\phi_t - \bar{w}_t\|^2 + \frac{2\beta_t}{m_t} + 4m_t^{-1/4}L^{1/2}d^{1/4}}{\alpha - \frac{2d^{1/2}m_t^{1/2}L}{\beta_t}}$ .
Consequently, we can obtain:
$$
R _ {\mathrm {e x c e s s}} (\mathbf {G i b b s}, u _ {t}) \leq \frac {\beta_ {t}}{2 m _ {t}} + m _ {t} ^ {- 1 / 4} L ^ {1 / 2} d ^ {1 / 4} + \frac {d ^ {1 / 2} m _ {t} ^ {1 / 2} L (\alpha \| \phi_ {t} - \bar {w} _ {t} \| ^ {2} + \frac {\beta_ {t}}{m _ {t}} + 2 m _ {t} ^ {- 1 / 4} L ^ {1 / 2} d ^ {1 / 4})}{\alpha \beta_ {t} - 2 d ^ {1 / 2} m _ {t} ^ {1 / 2} L}.
$$
Replace  $\alpha \beta_{t} - 2d^{1 / 2}m_{t}^{1 / 2}L = m_{t}^{3 / 4}d^{1 / 4}L^{1 / 2}\beta_{t}^{\prime}$ , we have
$$
\begin{array}{l} R _ {\text {e x c e s s}} (\mathbf {G i b b s}, u _ {t}) \leq \frac {\left(m _ {t} ^ {3 / 4} d ^ {1 / 4} L ^ {1 / 2} \beta_ {t} ^ {\prime} + 2 d ^ {1 / 2} m _ {t} ^ {1 / 2} L\right) / \alpha}{2 m _ {t}} + m _ {t} ^ {- 1 / 4} L ^ {1 / 2} d ^ {1 / 4} \\ + \frac {\alpha \| \phi_ {t} - \bar {w} _ {t} \| ^ {2} + \frac {(m _ {t} ^ {3 / 4} d ^ {1 / 4} L ^ {1 / 2} \beta_ {t} ^ {\prime} + 2 d ^ {1 / 2} m _ {t} ^ {1 / 2} L) / \alpha}{m _ {t}} + 2 m _ {t} ^ {- 1 / 4} L ^ {1 / 2} d ^ {1 / 4}}{m _ {t} ^ {1 / 4} d ^ {- 1 / 4} L ^ {- 1 / 2} \beta_ {t} ^ {\prime}} \\ = \frac {1}{2 \alpha} m _ {t} ^ {- 1 / 4} d ^ {1 / 4} L ^ {1 / 2} \beta_ {t} ^ {\prime} + \frac {1}{\alpha} m _ {t} ^ {- 1 / 2} d ^ {1 / 2} L + m _ {t} ^ {- 1 / 4} L ^ {1 / 2} d ^ {1 / 4} \\ + \frac {\alpha \| \phi_ {t} - \bar {w} _ {t} \| ^ {2} + \frac {1}{\alpha} m _ {t} ^ {- 1 / 4} d ^ {1 / 4} L ^ {1 / 2} \beta_ {t} ^ {\prime} + \frac {2}{\alpha} m _ {t} ^ {- 1 / 2} d ^ {1 / 2} L + 2 m _ {t} ^ {- 1 / 4} L ^ {1 / 2} d ^ {1 / 4}}{m _ {t} ^ {1 / 4} d ^ {- 1 / 4} L ^ {- 1 / 2} \beta_ {t} ^ {\prime}} \\ = \frac {1}{2 \alpha} m _ {t} ^ {- 1 / 4} d ^ {1 / 4} L ^ {1 / 2} \beta_ {t} ^ {\prime} + m _ {t} ^ {- 1 / 4} L ^ {1 / 2} d ^ {1 / 4} + \frac {2}{\alpha} m _ {t} ^ {- 1 / 2} d ^ {1 / 2} L \\ + \frac {\alpha \| \phi_ {t} - \bar {w} _ {t} \| ^ {2} + \frac {2}{\alpha} m _ {t} ^ {- 1 / 2} d ^ {1 / 2} L + 2 m _ {t} ^ {- 1 / 4} L ^ {1 / 2} d ^ {1 / 4}}{m _ {t} ^ {1 / 4} d ^ {- 1 / 4} L ^ {- 1 / 2} \beta_ {t} ^ {\prime}} \\ = \frac {d ^ {\frac {1}{4}} \sqrt {L}}{m _ {t} ^ {\frac {1}{4}}} \left(\frac {\beta_ {t} ^ {\prime}}{2 \alpha} + 1 + \frac {2}{\alpha} m _ {t} ^ {- 1 / 4} d ^ {1 / 4} L ^ {1 / 2} \right. \\ \left. + \frac {\alpha \| \phi_ {t} - \bar {w} _ {t} \| ^ {2} + \frac {2}{\alpha} m _ {t} ^ {- 1 / 2} d ^ {1 / 2} L + 2 m _ {t} ^ {- 1 / 4} L ^ {1 / 2} d ^ {1 / 4}}{\beta_ {t} ^ {\prime}}\right) \\ \leq \frac {d ^ {\frac {1}{4}} \sqrt {L}}{m _ {t} ^ {\frac {1}{4}}} \left(\frac {\beta_ {t} ^ {\prime}}{2 \alpha} + 1 + \frac {2 d ^ {\frac {1}{4}} \sqrt {L}}{\alpha m _ {t} ^ {\frac {1}{4}}} + \frac {\alpha \| \phi_ {t} - \bar {w} _ {t} \| ^ {2} + \frac {2}{\alpha} \sqrt {\frac {d}{m _ {t}}} L + 2 \frac {\sqrt {L} d ^ {\frac {1}{4}}}{m _ {t} ^ {\frac {1}{4}}}}{\beta_ {t} ^ {\prime}}\right) \\ \leq \frac {d ^ {\frac {1}{4}} \sqrt {L}}{m _ {t} ^ {\frac {1}{4}}} \left(\frac {\beta_ {t} ^ {\prime}}{2 \alpha} + 1 + \frac {2 d ^ {\frac {1}{4}} \sqrt {L}}{\alpha m _ {t} ^ {\frac {1}{4}}} + \frac {2 \alpha \| \phi_ {t} - w _ {t} \| ^ {2} + \epsilon_ {0} + \frac {2}{\alpha} \sqrt {\frac {d}{m _ {t}}} L + 2 \frac {\sqrt {L} d ^ {\frac {1}{4}}}{m _ {t} ^ {\frac {1}{4}}}}{\beta_ {t} ^ {\prime}}\right) \\ = f _ {t} \left(\beta_ {t} ^ {\prime}, \phi_ {t}\right). \\ \end{array}
$$
E.2 Static AER of Gibbs
Theorem E.2. Consider the Gibbs algorithm as described in Proposition E.1 to be the base learner. Apply the Meta-OGD (Algo. 2) on the cost function  $f_{t}(\beta_{t}^{\prime},\phi_{t})$ . Further, assume that each task uses the same sample number  $m$ . Then
$$
A E R _ {G i b b s} ^ {T} \in \mathcal {O} \left(\left((V + 1) + \frac {1}{\sqrt {T}}\right) \frac {1}{m ^ {\frac {1}{4}}}\right).
$$
Proof. From Lemma B.6, we know that  $f_{t}$  is convex w.r.t  $u$ , so we can directly apply OGD. Following Proposition E.1, we have
To apply Theorem D.1, we first find  $a = \frac{1}{2\alpha}$ ,  $b = 2\alpha$ ,  $\epsilon_{t} = \frac{2}{\alpha}\sqrt{\frac{d}{m_{t}}}L + 2\frac{\sqrt{L}d^{\frac{1}{4}}}{m_{t}^{\frac{1}{4}}}$ ,  $\Delta_{t} = 1 + \frac{2d^{\frac{1}{4}}\sqrt{L}}{\alpha m_{t}^{\frac{1}{4}}}$ ,  $\epsilon_{0} = 2\alpha \| w_{t} - \bar{w}_{t}\|^{2}$ , and  $\kappa_{t} = \frac{d^{\frac{1}{4}}\sqrt{L}}{m_{t}^{\frac{1}{4}}}$ . Since we have assumed  $m_{t} = m$ ,  $\forall t \in [T]$ , then we have
$$
\kappa = \kappa_ {t} = \kappa_ {\mathrm {m a x}} = \frac {d ^ {1 / 4} \sqrt {L}}{m ^ {1 / 4}}, \epsilon = \epsilon_ {t} = \epsilon_ {\mathrm {m a x}} = \frac {2}{\alpha} \sqrt {\frac {d}{m}} L + 2 \frac {\sqrt {L} d ^ {\frac {1}{4}}}{m ^ {\frac {1}{4}}}.
$$
Taking the above values into Theorem D.1, we have:
$$
\begin{array}{l} \mathrm {A E R} _ {\mathrm {G i b b s}} ^ {T} \leq (\sqrt {2 (2 \alpha V ^ {2} + \epsilon + \epsilon_ {0}) / \alpha} + 1) \sum_ {t = 1} ^ {T} \frac {d ^ {1 / 4} \sqrt {L}}{T m ^ {1 / 4}} + \sum_ {t = 1} ^ {T} \frac {2 \sqrt {d} L}{T \alpha \sqrt {m}} \\ + \frac {3 d ^ {1 / 4} \sqrt {L}}{2 \tilde {\epsilon} _ {0} \sqrt {T} m ^ {1 / 4}} \sqrt {((1 + 4 \alpha^ {2}) \hat {D} ^ {2} + 4 \sqrt {d / m} L + \frac {4 \alpha \sqrt {L} d ^ {1 / 4}}{m ^ {1 / 4}})} \\ \times \sqrt {8 \alpha \tilde {\epsilon} _ {0} \hat {D} ^ {2} + (\frac {1}{\alpha^ {2}} \sqrt {\frac {d}{m}} L + \frac {\sqrt {L} d ^ {\frac {1}{4}}}{\alpha m ^ {\frac {1}{4}}} + \hat {D} ^ {2}) ^ {2}}. \\ \end{array}
$$
Since we have assumed the bounded gradient norm for the cost function, i.e.,  $L^{\prime}$ -Lipschitz, then we can obtain:
$$
\hat {D} ^ {2} \leq \frac {\tilde {\epsilon} _ {0} L ^ {\prime} - a \kappa \epsilon}{a b \kappa},
$$
which implies  $\tilde{\epsilon}_0\geq a\kappa \epsilon /L'$ , where  $a\kappa \epsilon /L^{\prime}\in \mathcal{O}(1 / \sqrt{m})$ . So it suffices to ensure  $\tilde{\epsilon}_0 > \Omega (1 / \sqrt{m})$
For the Gibbs algorithm, by the law of large numbers, we have  $\epsilon_0 = 2\alpha \| w_t - \mathbb{E}_{S_t,\mathcal{A}}W_t\|^2 \to 2\alpha \| w_t - \mathbb{E}_A W_t\|^2$  as  $m \to \infty$ . In addition, since the variance of Gaussian is  $\sigma = m^{-1/4} d^{-1/4} L^{-1/2}$ , we have  $\epsilon_0 \in o_m(1)$ .
So if we set  $\epsilon_v = bV^2 < bV^2 + \epsilon$  with constant  $V$ , then  $\tilde{\epsilon}_0 = \epsilon_0 + \epsilon_v$  satisfies the above condition. Consequently, we have
$$
\mathrm {A E R} _ {\mathrm {G i b b s}} ^ {T} \in \mathcal {O} \left(\left((V + 1) + \frac {1}{\sqrt {T}}\right) \frac {1}{m ^ {\frac {1}{4}}}\right).
$$
E.3 Proof of Theorem 6.1 (Dynamic AER of Gibbs)
Theorem E.3. Let  $\mathcal{W} \subset \mathbb{R}^d$ . Assume that  $\ell(\cdot, z) \in [0,1]$  is  $L$ -Lipschitz  $\forall z \in \mathcal{Z}$ . Assume that  $\mathcal{L}_{\mu_t}(\cdot)$  has  $\alpha$ -quadratic-growth for all  $t \in [T]$ . Let  $w_t = A_{Gibbs}(\beta_t, \phi_t, S_t)$  be the output of the Gibbs algorithm (Definition B.2) on  $S_t$ , where  $\phi_t$  is the mean of a prior Gaussian  $\mathcal{N}(\phi_t, \sigma_t^2 \mathbf{1}_d)$  with  $\sigma_t = m_t^{-1/4} d^{-1/4} L^{-1/2}$  and  $\beta_t$  is the inverse temperature. Consider Algo. 1 (DCML) and further assume that each slot has equal length  $M$  and each task uses the sample number  $m$ . Then
$$
A E R _ {G i b b s} ^ {T} \in \mathcal {O} \left(\left((1 + \frac {1}{N} \sum_ {n = 1} ^ {N} V _ {n} + \frac {1}{\sqrt {M}}) + \frac {\sqrt {P ^ {*}}}{M \sqrt {N}}\right) \frac {1}{m ^ {\frac {1}{4}}}\right).
$$
Proof. From Proposition E.1, we first obtain  $a = \frac{1}{2\alpha}$ ,  $b = 2\alpha$ ,  $\epsilon_{n,k} = \frac{2}{\alpha}\sqrt{\frac{d}{m_{n,k}}} L + 2\frac{\sqrt{L}d^{\frac{1}{4}}}{m_{n,k}^{\frac{1}{4}}}$ ,
$\Delta_{n,k} = 1 + \frac{2d^{\frac{1}{4}}\sqrt{L}}{\alpha m_{n,k}^{\frac{4}{4}}}$ , and  $\kappa_{n,k} = \frac{d^{\frac{1}{4}}\sqrt{L}}{m_{n,k}^{\frac{4}{4}}}$ . Moreover,  $\kappa_{n}^{\max} = \frac{d^{1/4}\sqrt{L}}{m_{n}^{*1/4}}$ ,  $\epsilon_{n}^{\max} = \frac{2}{\alpha}\sqrt{\frac{d}{m_{n}^{*}}} L + 2\frac{\sqrt{L}d^{\frac{1}{4}}}{m_{n}^{*}\frac{1}{4}}$ ,
where  $m_{n}^{*} = \min \{m_{n,k}\}_{k = 1}^{M_{n}}$
For simplicity, let us assume that each environment slot has the same number of tasks, i.e.,  $M_{n} = M$ ,  $T = NM$ , and each task has the same number of samples, i.e.,  $m_{n,k} = m$ . So we have  $\kappa_{n,k} = \kappa_{n}^{\max} = \kappa = d^{1/4}L^{1/2}m^{-1/4}$  and  $\epsilon_{n,k} = \epsilon_{n}^{\max} = \epsilon = \frac{2}{\alpha}\sqrt{d/m}L + 2\sqrt{L}d^{1/4}m^{-1/4}$ .
Taking these values into Theorem 5.1, we can obtain:
$$
\begin{array}{l} \mathrm {A E R} _ {\text {G i b b s}} ^ {T} \leq \frac {1}{N} \sum_ {n = 1} ^ {N} \left(\sqrt {2 \left(2 \alpha V _ {n} ^ {2} + \epsilon_ {n} + \epsilon_ {0}\right) / \alpha}\right) \frac {d ^ {\frac {1}{4}} \sqrt {L}}{m ^ {\frac {1}{4}}} + \frac {d ^ {\frac {1}{4}} \sqrt {L}}{m ^ {\frac {1}{4}}} + \frac {2 \sqrt {d L}}{\alpha \sqrt {m}} \\ + \frac {3}{2 \tilde {\epsilon} _ {0} N M} \frac {d ^ {\frac {1}{4}} \sqrt {L}}{m ^ {\frac {1}{4}}} \sqrt {M - 1} \times \\ \sum_ {n = 1} ^ {N} \sqrt {(1 + 4 \alpha^ {2}) \hat {D} _ {n} ^ {2} + 4 \sqrt {\frac {d}{m}} L + \frac {4 \alpha \sqrt {L} d ^ {\frac {1}{4}}}{m ^ {\frac {1}{4}}}} \sqrt {(\hat {D} _ {n} ^ {2} + \sqrt {\frac {d}{m}} L \frac {1}{\alpha^ {2}} + \frac {\sqrt {L} d ^ {\frac {1}{4}}}{\alpha m ^ {\frac {1}{4}}}) ^ {2} + 8 \alpha \tilde {\epsilon} _ {0} \hat {D} _ {n} ^ {2}} \\ + \frac {1}{N M} \sqrt {\tilde {\epsilon} _ {0} (1 + 4 \alpha^ {2}) \hat {D} _ {\max } ^ {2} + 4 \sqrt {\frac {d}{m}} L + \frac {4 \alpha \sqrt {L} d ^ {1 / 4}}{m ^ {1 / 4}}} \times \frac {d ^ {1 / 4} \sqrt {L}}{m ^ {\frac {1}{4}}} \\ \times \sqrt {2 P ^ {*} \sum_ {n = 1} ^ {N} \frac {\left(\hat {D} _ {n} ^ {2} + \frac {1}{\alpha^ {2}} \sqrt {d / m} L + \frac {\sqrt {L} d ^ {\frac {1}{4}}}{\alpha m ^ {\frac {1}{4}}}\right) ^ {2}}{\tilde {\epsilon} _ {0} ^ {2}} + \frac {8 \alpha \hat {D} _ {n} ^ {2}}{\tilde {\epsilon} _ {0}}}. \\ \end{array}
$$
$$
\hat {D} _ {n} ^ {2} \leq \frac {\tilde {\epsilon} _ {0} L ^ {\prime} - a \kappa \epsilon}{a b \kappa},
$$
For the Gibbs algorithm, by the law of large numbers, we have  $\epsilon_0 = 2\alpha \| w_t - \mathbb{E}_{S_t,\mathcal{A}}W_t\|^2 \to 2\alpha \| w_t - \mathbb{E}_A W_t\|^2$  as  $m \to \infty$ . In addition, since the variance of Gaussian is  $\sigma = m^{-1/4}d^{-1/4}L^{-1/2}$ , we have  $\epsilon_0 \in o_m(1)$ . So if we set  $\epsilon_v = bV_n^2 < bV_n^2 + \epsilon$  with constant  $V_n$ ,  $\tilde{\epsilon}_0 = \epsilon_0 + \epsilon_v$  satisfies the above condition. Consequently, we have
$$
\mathrm {A E R} _ {\mathbf {G i b b s}} ^ {T} \in \mathcal {O} \left(\left((1 + \frac {1}{N} \sum_ {n = 1} ^ {N} V _ {n} + \frac {1}{\sqrt {M}}) + \frac {\sqrt {P ^ {*}}}{M \sqrt {N}}\right) \frac {1}{m ^ {\frac {1}{4}}}\right).
$$
E.4 Cost function for SGD base leaner
Proposition E.4. Consider  $\mathcal{W} \subset \mathbb{R}^d$ . Let  $w_t^*$  be the hypothesis that achieves the minimum population risk among  $\mathcal{W}$ . Suppose  $\ell(\cdot, z)$  is convex,  $\beta$ -smooth, and  $L$ -Lipschitz for all  $z \in \mathcal{Z}$ . Assume that  $\mathcal{L}_{\mu_t}(\cdot)$  has  $\alpha$ -quadratic-growth  $\forall t \in [T]$ . Let  $w_t$  denote the output of the SGD algorithm applied on dataset  $S_t$  and  $\phi_t$  be the initialization of  $t$ -th task,  $\eta_t$  is the learning rate, and  $K_t$  is the gradient updates number. To meet the unified form in Eq. (1), let  $\eta_t' = (K_t \alpha \eta_t - 2) \kappa_t$ . Then, the upper bound of the excess risk is given by
$$
f (\eta_ {t} ^ {\prime}, \phi_ {t}) = \kappa_ {t} \left(\frac {2 \alpha \| \phi_ {t} - w _ {t} \| ^ {2} + \epsilon_ {0} + 2 (L ^ {2} + 2 L ^ {2} \frac {K _ {t}}{m _ {t}}) / (K _ {t} \alpha)}{\eta_ {t} ^ {\prime}} + \frac {1}{2} \eta_ {t} ^ {\prime} + 2 \kappa_ {t}\right),
$$
where  $\kappa_{t} = \sqrt{(1 / (K_{t}\alpha) + 2 / (m_{t}\alpha)} L.$
From Theorem .B.4, we have the following excess risk bound for SGD:
$$
R _ {\text {e x c e s s}} (\mathrm {S G D}, u _ {t}) \leq \frac {\| \phi_ {t} - w _ {t} ^ {*} \| ^ {2}}{2 \eta_ {t} K _ {t}} + \left(\frac {L ^ {2}}{2} + \frac {L ^ {2} K _ {t}}{m _ {t}}\right) \eta_ {t}.
$$
Let  $\bar{w}_t = \mathbb{E}W_t$  be the expected output of the SGD algorithm of  $t$ -th task, and  $w_{t}$  is the actual output. Let  $\epsilon_0 = 2\alpha \| w_t - \bar{w}_t\|^2$  be the randomness of the algorithm, which should be small. With the quadratic growth property, we have
$$
\begin{array}{l} \left\| \phi_ {t} - w _ {t} ^ {*} \right\| ^ {2} = \left\| \phi_ {t} - \bar {w} _ {t} + \bar {w} _ {t} - w _ {t} ^ {*} \right\| ^ {2} \\ \leq 2 \| \phi_ {t} - \bar {w} _ {t} \| ^ {2} + 2 \| \bar {w} _ {t} - w _ {t} ^ {*} \| ^ {2} \\ \leq 2 \| \phi_ {t} - \bar {w} _ {t} \| ^ {2} + 2 \mathbb {E} \| W _ {t} - w _ {t} ^ {*} \| ^ {2} \\ \leq 2 \| \phi_ {t} - \bar {w} _ {t} \| ^ {2} + \frac {4 \mathbb {E} \left[ \mathcal {L} _ {\mu_ {t}} \left(W _ {t}\right) - \mathcal {L} _ {\mu_ {t}} \left(w _ {t} ^ {*}\right) \right]}{\alpha} \\ \leq 2 \| \phi_ {t} - \bar {w} _ {t} \| ^ {2} + \frac {4 \left(\frac {\| \phi_ {t} - w _ {t} ^ {*} \| ^ {2}}{2 \eta_ {t} K _ {t}} + \left(\frac {L ^ {2}}{2} + \frac {L ^ {2} K _ {t}}{m _ {t}}\right) \eta_ {t}\right)}{\alpha}. \\ \end{array}
$$
Therefore, we can obtain
$$
\begin{array}{l} \left\| \phi_ {t} - w _ {t} ^ {*} \right\| ^ {2} \leq \frac {2 \alpha \left\| \phi_ {t} - \bar {w} _ {t} \right\| ^ {2} + (2 L ^ {2} + 4 L ^ {2} \frac {K t}{m _ {t}}) \eta_ {t}}{\alpha - \frac {2}{K _ {t} \eta_ {t}}}, \\ R _ {\mathrm {e x c e s s}} (\mathrm {S G D}, u _ {t}) \leq \frac {2 \alpha \| \phi_ {t} - w _ {t} \| ^ {2} + \epsilon_ {0} + (L ^ {2} + 2 L ^ {2} \frac {K _ {t}}{m _ {t}}) \eta_ {t}}{K _ {t} \alpha \eta_ {t} - 2} + (\frac {L ^ {2}}{2} + \frac {L ^ {2} K _ {t}}{m _ {t}}) \eta_ {t}. \\ \end{array}
$$
$$
f (\eta_ {t}, \phi_ {t}) = \frac {2 \alpha \| \phi_ {t} - w _ {t} \| ^ {2} + \epsilon_ {0} + (L ^ {2} + 2 L _ {m _ {t}} ^ {2} \frac {K _ {t}}{\eta_ {t}}) \eta_ {t}}{K _ {t} \alpha \eta_ {t} - 2} + (\frac {L ^ {2}}{2} + \frac {L ^ {2} K _ {t}}{m _ {t}}) \eta_ {t},
$$
and replace  $\eta_t' = (K_t \alpha \eta_t - 2) \kappa_t$ ,  $\kappa_t = \sqrt{\frac{(L^2 + 2L^2 \frac{K_t}{m_t})}{K_t \alpha}}$ , we have
$$
\begin{array}{l} f (\eta_ {t} ^ {\prime}, \phi_ {t}) = \frac {2 \alpha \| \phi_ {t} - w _ {t} \| ^ {2} + \epsilon_ {0} + (L ^ {2} + 2 L ^ {2} \frac {K _ {t}}{m _ {t}}) (\eta_ {t} ^ {\prime} / \kappa_ {t} + 2) / (K _ {t} \alpha)}{\eta_ {t} ^ {\prime} / \kappa_ {t}} \\ + \left(\frac {L ^ {2}}{2} + L ^ {2} \frac {K _ {t}}{m _ {t}}\right) \left(\eta_ {t} ^ {\prime} / \kappa_ {t} + 2\right) / \left(K _ {t} \alpha\right) \\ = \kappa_ {t} \left(\frac {2 \alpha \| \phi_ {t} - w _ {t} \| ^ {2} + \epsilon_ {0} + 2 (L ^ {2} + 2 L ^ {2} \frac {K _ {t}}{m _ {t}}) / (K _ {t} \alpha)}{\eta_ {t} ^ {\prime}} + \frac {1}{2} \eta_ {t} ^ {\prime} + 2 \kappa_ {t}\right) \\ \end{array}
$$
E.5 Static AER of SGD
Theorem E.5. Consider the SGD algorithm as described in Proposition E.4 to be the base learner. Apply the Meta-OGD(Algo. 2) algorithm on the cost function  $f_{t}(\eta_{t}^{\prime},\phi_{t})$ . Further, assume that for each task, we have the same sample number  $m$  and that SGD performs the same number of gradient steps  $K$ . Then
$$
A E R _ {S G D} ^ {T} \in \mathcal {O} \left((V + \frac {1}{\sqrt {T}}) \sqrt {\frac {1}{K} + \frac {1}{m}}\right).
$$
Proof. According to Proposition E.4, we have  $\kappa_{t} = \sqrt{\frac{(L^{2} + 2L^{2}\frac{K_{t}}{m_{t}})}{K_{t}\alpha}}$ ,  $a = 1/2$ ,  $b = 2\alpha$  and  $\epsilon_{t} = \frac{2(L^{2} + 2L^{2}\frac{K_{t}}{m_{t}})}{K_{t}\alpha} = 2\kappa_{t}^{2}$ ,  $\Delta_{t} = 2\kappa_{t}$ ,  $\epsilon_{0} = 2\alpha \|w_{t} - \bar{w}_{t}\|^{2}$ .
For simplicity, we assume the SGD step and sample number are the same for different tasks. Hence, we have  $K_{t} = K$ ,  $m_{t} = m$ ,  $\kappa_{t} = \kappa_{\max} = \sqrt{\frac{1}{K\alpha} + \frac{2}{m\alpha}} L = \kappa$  and  $\epsilon_{t} = \epsilon = \epsilon_{\max} = 2\kappa^{2}$ .
Put these variable into the bound of Theorem D.1, we have:
$$
\begin{array}{l} A E R _ {\mathrm {S G D}} ^ {T} \leq (\sqrt {2 (2 \alpha V ^ {2} + (\frac {2}{K \alpha} + \frac {4}{m \alpha}) L ^ {2} + \epsilon_ {0}))}) \sqrt {\frac {1}{K \alpha} + \frac {2}{m \alpha}} L + 2 (\frac {1}{K \alpha} + \frac {2}{m \alpha}) L ^ {2} \\ + \frac {3}{2 \tilde {\epsilon} _ {0} \sqrt {T}} \sqrt {\frac {1}{K \alpha} + \frac {2}{m \alpha}} L \times \\ \left(\sqrt {\left((1 + 4 \alpha) \hat {D} ^ {2} + 4 \left(\frac {1}{K \alpha} + \frac {2}{m \alpha}\right) L ^ {2}\right) \left(8 \alpha^ {2} \tilde {\epsilon} _ {0} \hat {D} ^ {2} + \left(\left(\frac {1}{K \alpha} + \frac {2}{m \alpha}\right) L ^ {2} + \alpha \hat {D} ^ {2}\right) ^ {2}\right)}\right). \\ \end{array}
$$
which implies  $\tilde{\epsilon}_0 \geq a\kappa \epsilon / L'$ , where  $a\kappa \epsilon / L' \in \mathcal{O}(m^{-3/2} + K^{-3/2})$ . So it suffices to ensure  $\tilde{\epsilon}_0 \in \Omega(m^{-3/2} + K^{-3/2})$ .
For SGD, by the law of large numbers, we have  $\epsilon_0 = 2\alpha \| w_t - \mathbb{E}_{S_t,\mathcal{A}}W_t\|^2 \to 2\alpha \| w_t - \mathbb{E}_{\mathcal{A}}W_t\|^2$  as  $m \to \infty$ . In addition,  $2\alpha \| w_t - \mathbb{E}_{\mathcal{A}}W_t\|^2 \to 0$  as  $K \to \infty$ , so we have  $\epsilon_0 \in o_m(1) + o_K(1)$ .
Then, if we set  $\epsilon_v = bV^2 < bV^2 + \epsilon$ , s.t.  $\tilde{\epsilon}_0 = \epsilon_0 + \epsilon_v$  satisfies the above condition, we have
$$
\mathrm {A E R} _ {\mathrm {S G D}} ^ {T} \leq \mathcal {O} \left((V + o _ {m} (1) + o _ {K} (1) + \frac {\frac {1}{o _ {m} (1) + o _ {K} (1) + V ^ {2}}}{\sqrt {T}}) \sqrt {\frac {1}{K} + \frac {1}{m}}\right).
$$
Considering a constant variance, it gives the following bound that has the same complexity factor and rate as Theorem 3.2 in [31] (Khodak et al. [31] used OGD as the base learner, where each step only takes one sample, i.e.,  $K = m$ ).
$$
\mathrm {A E R} _ {\mathrm {S G D}} ^ {T} \in \mathcal {O} \left(\left(V + \frac {1}{\sqrt {T}}\right) \sqrt {\frac {1}{K} + \frac {1}{m}}\right).
$$
If we have  $V \to 0$ , we can set  $\epsilon_v = bV^2 + \epsilon + 1 / T^{1/4} > bV^2 + \epsilon$ , s.t.  $\tilde{\epsilon}_0 = \epsilon_0 + \epsilon_v$  satisfies the above condition. In this case, we have:
$$
\begin{array}{l} \mathrm {A E R} _ {\mathrm {S G D}} ^ {T} \leq \mathcal {O} \left(\left(V + o _ {m} (1) + o _ {K} (1) + \frac {1}{\sqrt {T}} + \frac {\frac {1}{o _ {m} (1) + o _ {K} (1) + V ^ {2} + 1 / T ^ {1 / 4}}}{\sqrt {T}}\right) \sqrt {\frac {1}{K} + \frac {1}{m}}\right) \\ = \mathcal {O} \left(\left(\frac {1}{\sqrt [ 4 ]{T}} + o _ {m} (1) + o _ {K} (1)\right) \sqrt {\frac {1}{K} + \frac {1}{m}}\right), \text {w h e n} V \rightarrow 0, \\ \end{array}
$$
which has an additional term  $o_{m}(1) + o_{K}(1)$  compared to Theorem 3.2 in [31]. This is because the variance  $V^2$  (task similarity) in our paper is calculated via the average SGD iterates. A similar term will be introduced using the same output for estimating  $V$  in [31].
E.6 Proof of Theorem 6.2 (Dynamic AER of SGD)
Theorem E.6. Let  $\mathcal{W} \subset \mathbb{R}^d$ . Consider that  $\ell(\cdot, z)$  is convex,  $\beta$ -smooth,  $L$ -Lipschitz for all  $z \in \mathcal{Z}$ . Assume that  $\mathcal{L}_{\mu_t}(\cdot)$  has  $\alpha$ -quadratic-growth for all  $t \in [T]$ . Let SGD be the base learner for each task where it outputs  $w_t = A_{SGD}(\eta_t, \phi_t, S_t)$  with learning rate  $\eta_t$  and initialization  $\phi_t$ . Consider the Algo. 1 (DCML) on the SGD cost function. Further, assume each slot has equal length  $M$ , each task has the sample number  $m$ , and that SGD performs the same number of updating steps  $K$ . Then
$$
A E R _ {S G D} ^ {T} \in \mathcal {O} \left(\left(\frac {1}{N} \sum_ {n = 1} ^ {N} V _ {n} + \frac {1}{\sqrt {M}} + \frac {\sqrt {P ^ {*}}}{M \sqrt {N}}\right) \sqrt {\frac {1}{K} + \frac {1}{m}}\right).
$$
Proof. For simplicity, we assume the SGD step and sample number are the same for different tasks. Hence,  $K_{n,k} = K$  and  $m_{n,k} = m$ . Based on Proposition E.4, we have  $a = 1/2$ ,  $b = 2\alpha$ . Then, we get  $\kappa_{n,k} = \sqrt{\frac{1}{K\alpha} + \frac{2}{m\alpha}} L = \kappa \epsilon_{n,k} = \epsilon = 2\kappa^2$ , and  $\Delta_{n,k} = 2\kappa_{n,k}$ .
Put these variable into the bound of Theorem 5.1, we can obtain: