SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Analogously, we consider the shadow parameters as  $\{\pmb{\theta}_1^{(k),*},\pmb{\theta}_2^{(k),*}\} \sim \{\widetilde{\pmb{\theta}}_1^{(\tau),*},\widetilde{\pmb{\theta}}_2^{(\tau),*}\}_{\tau \in [k]}$  where each pair  $\{\widetilde{\pmb{\theta}}_1^{(\tau),*},\widetilde{\pmb{\theta}}_2^{(\tau),*}\}$  is separately trained on past received rewards of the optimal arm(s)  $\{r_{\tau ',i^*}\}_{\tau '\in [\tau ],\mathcal{T}_{\tau ',i^*}\in \Omega_k^*}$  and past exploration scores of the optimal arm(s)  $\{e_{\tau ',i^*}\}_{\tau '\in [\tau ],\mathcal{T}_{\tau ',i^*}\in \Omega_k^*}$  with  $J_{\theta}$ -iteration GD starting from the random initialization  $\{\pmb {\theta}_1^{(0)},\pmb {\theta}_2^{(0)}\}$ .
Corollary D.4. For the constants  $\rho \in (0, \mathcal{O}(\frac{1}{L}))$  and  $\xi_1 \in (0, 1)$ , given the past records  $\mathcal{P}_{k-1}$  we suppose  $m, \eta_1, J$  satisfy the conditions in Theorem 5.2, and randomly draw the parameters  $\{\pmb{\theta}_1^{(k),*}, \pmb{\theta}_2^{(k),*}\} \sim \{\widetilde{\pmb{\theta}}_1^{(\tau),*}, \widetilde{\pmb{\theta}}_2^{(\tau),*}\}_{\tau \in [k]}$ . For the optimal arm batch  $\Omega_k^* \subset \Omega_{task}^k$ , consider its union set with the collection of past optimal arms  $\mathcal{P}_{k-1}^* \cup \{\Omega_k^*, r_k^*\}$  are generated by a fixed policy when witness the candidate arms  $\{\Omega_{task}^{(\tau)}\}_{\tau \in [k]}$ , with  $\mathcal{P}_{k-1}^*$  being the collection chosen by this policy. Then, with probability at least  $1 - \delta$ , we have
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {T} _ {k, i} \sim \mathcal {P} (\mathcal {T})} \left[ | f _ {2} \left(\phi (\frac {[ \nabla \pmb {\theta} f _ {1} (\pmb {\chi} _ {k} ^ {s , *}) ; \nabla \pmb {\theta} f _ {1} (\pmb {\chi} _ {k} ^ {q , *}) ]}{c _ {g} ^ {\prime} L}, \pmb {\chi} _ {k} ^ {q, *}\); \pmb {\theta} _ {2} ^ {(k - 1), *}\right) - \left(r _ {\tau} - f _ {1} (\pmb {\chi} _ {k} ^ {q, *}; \pmb {\theta} _ {1} ^ {(k - 1), *})\right) | \left| \Omega_ {t a s k} ^ {(k)}, \mathcal {P} _ {k - 1} ^ {*} \right. \right] \\ \leq \frac {1}{\sqrt {k}} \cdot \left(\sqrt {2 \xi_ {2}} + \frac {3 L}{\sqrt {2}} + (1 + \gamma_ {1}) \sqrt {2 \log (\frac {k}{\delta})}\right) + \Gamma_ {k} \\ \end{array}
$$
This corollary is a directly application of Lemma D.3 and can be obtained with a similar proof as in Corollary D.2.
D.4 Ancillary Lemmas
Applying  $\mathcal{P}_{k-1}$  as the training data, we have the following properties for the over-parameterized FC network  $f(\cdot; \theta)$  after GD.
Lemma D.5. For the constants  $\rho \in (0, \mathcal{O}(\frac{1}{L}))$  and  $\xi_1 \in (0, 1)$ , given the past records  $\mathcal{P}_{k-1}$  up to time step  $k$ , we suppose  $m, \eta_1, J_1$  satisfy the conditions in Theorem 5.2. Then, with probability at least  $1 - \delta$ , given a sample-label pair  $(x, r)$ , we have
$$
\left| f (\boldsymbol {x}; \boldsymbol {\theta} ^ {(k)}) \right| \leq \gamma_ {1} = 2 + \mathcal {O} \left(\frac {k ^ {3} L}{\rho \sqrt {m}} \log m\right) + \mathcal {O} \left(\frac {L ^ {2} k ^ {4}}{\rho^ {4 / 3} m ^ {1 / 6}} \log^ {1 1 / 6} (m)\right).
$$
Proof. The LHS of the inequality could be written as
$$
\begin{array}{l} | f (\boldsymbol {x}; \boldsymbol {\theta}) | \leq | f (\boldsymbol {x}; \boldsymbol {\theta}) - f (\boldsymbol {x}; \boldsymbol {\theta} ^ {(0)}) - \langle \nabla_ {\boldsymbol {\theta} ^ {(0)}} f (\boldsymbol {x}; \boldsymbol {\theta} ^ {(0)}), \boldsymbol {\theta} - \boldsymbol {\theta} ^ {(0)} \rangle | \\ + | f (\boldsymbol {x}; \boldsymbol {\theta} ^ {(0)}) + \left\langle \nabla_ {\boldsymbol {\theta} ^ {(0)}} f (\boldsymbol {x}; \boldsymbol {\theta} ^ {(0)}), \boldsymbol {\theta} - \boldsymbol {\theta} ^ {(0)} \right\rangle |. \\ \end{array}
$$
Here, we could bound the first term on the RHS with Lemma D.7. Applying Lemma D.8 on the second term, and recalling  $\| \pmb{\theta} - \pmb{\theta}^{(0)}\|_2 \leq \omega$ , would give
$$
\begin{array}{l} | f (\boldsymbol {x}; \boldsymbol {\theta}) | \leq 2 + \| \nabla_ {\boldsymbol {\theta} ^ {(0)}} f (\boldsymbol {x}; \boldsymbol {\theta} ^ {(0)}) \| _ {2} \| \boldsymbol {\theta} - \boldsymbol {\theta} ^ {(0)} \| _ {2} + \\ \mathcal {O} \left(\omega^ {1 / 3} L ^ {2} \sqrt {m \log (m)}\right) \cdot \| \boldsymbol {\theta} - \boldsymbol {\theta} ^ {(0)} \| _ {2} \\ \leq 2 + \mathcal {O} (L) \cdot \| \boldsymbol {\theta} - \boldsymbol {\theta} ^ {(0)} \| _ {2} + \mathcal {O} (L ^ {2} \sqrt {m \log (m)}) (\| \boldsymbol {\theta} - \boldsymbol {\theta} ^ {(0)} \| _ {2}) ^ {\frac {4}{3}}. \\ \end{array}
$$
Then, applying the conclusion of Lemma D.6 would lead to
$$
\begin{array}{l} | f (\boldsymbol {x}; \boldsymbol {\theta}) | \leq 2 + \mathcal {O} (L) \cdot \mathcal {O} \left(\frac {k ^ {3}}{\rho \sqrt {m}} \log m\right) + \mathcal {O} \left(L ^ {2} \sqrt {m \log (m)}\right) \left(\mathcal {O} \left(\frac {k ^ {3}}{\rho \sqrt {m}} \log m\right)\right) ^ {\frac {4}{3}} \\ = 2 + \mathcal {O} \left(\frac {k ^ {3} L}{\rho \sqrt {m}} \log m\right) + \mathcal {O} \left(\frac {L ^ {2} k ^ {4}}{\rho^ {4 / 3} m ^ {1 / 6}} \log^ {1 1 / 6} (m)\right) = \gamma_ {1}. \\ \end{array}
$$
Lemma D.6 (Theorem 1 from [3]). For any  $0 < \xi_{1} \leq 1$ ,  $0 < \rho \leq \mathcal{O}\left(\frac{1}{L}\right)$ . Given the past records  $\mathcal{P}_{k-1}$ , suppose  $m, \eta_{1}, J$  satisfy the conditions in Theorem 5.2, then with probability at least  $1 - \delta$ , we could have
1.  $\mathcal{L}(\pmb {\theta})\leq \xi_{1}$  after  $J$  iterations of  $GD$
2. For any  $j\in [J],\| \pmb{\theta}^{(j)} - \pmb{\theta}^{(0)}\| \leq \mathcal{O}\left(\frac{k^3}{\rho\sqrt{m}}\log m\right).$
In particular, Lemma D.6 above provides the convergence guarantee for  $f(\cdot; \theta)$  after certain rounds of GD training on the past records  $\mathcal{P}_{k-1}$ .
Lemma D.7 (Lemma 4.1 in [11]). Assume a constant  $\omega$  such that  $\mathcal{O}(m^{-3/2}L^{-3/2}[\log(TnL^2/\delta)]^{3/2}) \leq \omega \leq \mathcal{O}(L^{-6}[\log m]^{-3/2})$  and  $n$  training samples. With randomly initialized  $\pmb{\theta}^{(0)}$ , for parameters  $\pmb{\theta}, \pmb{\theta}'$  satisfying  $\|\pmb{\theta}-\pmb{\theta}^{(0)}\|, \|\pmb{\theta}-\pmb{\theta}^{(0)}\| \leq \omega$ , we have
$$
\left| f (\boldsymbol {x}; \boldsymbol {\theta}) - f (\boldsymbol {x}; \boldsymbol {\theta} ^ {\prime}) - \left\langle \nabla_ {\boldsymbol {\theta} ^ {\prime}} f (\boldsymbol {x}; \boldsymbol {\theta} ^ {\prime}), \boldsymbol {\theta} - \boldsymbol {\theta} ^ {\prime} \right\rangle \right| \leq \mathcal {O} \left(\omega^ {1 / 3} L ^ {2} \sqrt {m \log (m)}\right) \| \boldsymbol {\theta} - \boldsymbol {\theta} ^ {\prime} \|
$$
with the probability at least  $1 - \delta$ .
Lemma D.8. Assume  $m, \eta_1, J$  satisfy the conditions in Theorem 5.2 and  $\pmb{\theta}^{(0)}$  is randomly initialized. Then, with probability at least  $1 - \delta$  and given an arm  $\| \pmb{x}\| _2 = 1$ , we have
1.  $|f(\pmb{x};\pmb{\theta}^{(0)})|\leq 2$
2.  $\| \nabla_{\pmb{\theta}^{(0)}}f(\pmb {x};\pmb{\theta}^{(0)})\| _2\leq \mathcal{O}(L)$
Proof. The conclusion (1) is a direct application of Lemma 7.1 in [3]. Suppose the parameters of the  $L$ -layer FC network are  $\pmb{\theta} = \{\pmb{\theta}_1, \dots, \pmb{\theta}_L\}$ . For conclusion (2), applying Lemma 7.3 in [3], for each layer  $\pmb{\theta}_l \in \{\pmb{\theta}_1, \dots, \pmb{\theta}_L\}$ , we have
$$
\left\| \nabla_ {\boldsymbol {\theta} _ {l}} f (\boldsymbol {x}; \boldsymbol {\theta} ^ {(0)}) \right\| _ {2} = \left\| \left(\boldsymbol {\theta} _ {L} \boldsymbol {D} _ {L - 1} \dots \boldsymbol {D} _ {l + 1} \boldsymbol {\theta} _ {l + 1}\right) \cdot \left(\boldsymbol {D} _ {l + 1} \boldsymbol {\theta} _ {l + 1} \dots \boldsymbol {D} _ {1} \boldsymbol {\theta} _ {1}\right) \cdot \boldsymbol {x} ^ {\intercal} \right\| _ {2} = \mathcal {O} (\sqrt {L}).
$$
where  $D$  is the diagonal matrix corresponding to the activation function. Then, we could have the conclusion that
$$
\| \nabla_ {\boldsymbol {\theta} ^ {(0)}} f (\boldsymbol {x}; \boldsymbol {\theta} ^ {(0)}) \| _ {2} = \sqrt {\sum_ {l \in [ L ]} \| \nabla_ {\boldsymbol {\theta} _ {l}} f (\boldsymbol {x} ; \boldsymbol {\theta} ^ {(0)}) \| _ {2} ^ {2}} = \mathcal {O} (L).
$$
Lemma D.9 (Theorem 5 in [3]). Assume  $m, \eta_1, J$  satisfy the conditions in Theorem 5.2 and  $\pmb{\theta}^{(0)}$  being randomly initialized. Then, with probability at least  $1 - \delta$ , and for all parameter  $\pmb{\theta}$  such that  $\| \pmb{\theta} - \pmb{\theta}^{(0)} \|_2 \leq \omega$ , we have
$$
\left\| \nabla_ {\boldsymbol {\theta}} f (\boldsymbol {x}; \boldsymbol {\theta}) - \nabla_ {\boldsymbol {\theta} ^ {(0)}} f (\boldsymbol {x}; \boldsymbol {\theta} ^ {(0)}) \right\| _ {2} \leq \mathcal {O} \left(\omega^ {1 / 3} L ^ {3} \sqrt {\log (m)}\right)
$$
Lemma D.10. Assume  $m, \eta_1$  satisfy the condition in Theorem 5.2. For notation brevity, suppose the training sample-label pairs are  $\{\pmb{x}_{\tau}, r_{\tau}\}_{\tau \in [k]}$ . With the probability at least  $1 - \delta$ , we have
$$
\sum_ {\tau \in [ k ]} | f (\boldsymbol {x} _ {\tau}; \boldsymbol {\theta} ^ {(\tau)}) - r _ {\tau} | \leq \sum_ {\tau \in [ k ]} | f (\boldsymbol {x} _ {\tau}; \boldsymbol {\theta} ^ {(k)}) - r _ {\tau} | + \frac {3 L \sqrt {2 k}}{2}
$$
Proof. With the notation from Lemma 4.3 in [11], set  $R = \frac{k^3\log(m)}{\delta}$ ,  $\nu = R^2$ , and  $\epsilon = \frac{LR}{\sqrt{2\nu k}}$ . Then, considering the loss function to be  $\mathcal{L}(\pmb{\theta}) \coloneqq \sum_{\tau \in [k]} |f(\pmb{x}_{\tau};\pmb{\theta}) - r_{\tau}|$  would complete the proof.
Lemma D.11. Consider a randomly initialized  $L$ -layer ReLU fully-connected network  $f(\cdot; \theta_0)$ . For any  $0 < \xi_2 \leq 1$ ,  $0 < \rho \leq \mathcal{O}\left(\frac{1}{L}\right)$ . Let there be two sets of training samples  $\mathcal{P}_k, \mathcal{P}_k'$  with the unit-length and the  $\rho$ -separateness assumption, and let  $\theta$  be the trained parameter on  $\mathcal{P}_k$  while  $\theta'$  is the trained parameter on  $\mathcal{P}_k'$ . Suppose the conditions in Theorem 5.2 are satisfied. Then, with probability at least  $1 - \delta$ , we have
$$
| f (\boldsymbol {x}; \boldsymbol {\theta}) - f (\boldsymbol {x}; \boldsymbol {\theta} ^ {\prime}) | \leq \left(1 + \mathcal {O} \left(\frac {k L ^ {3} \log^ {5 / 6} (m)}{\rho^ {1 / 3} m ^ {1 / 6}}\right)\right) \cdot \mathcal {O} \left(\frac {k ^ {3} L}{\rho \sqrt {m}} \log (m)\right) + \mathcal {O} \left(\frac {k ^ {4} L ^ {2} \log^ {1 1 / 6} (m)}{\rho^ {4 / 3} m ^ {1 / 6}}\right)
$$
when given a new sample  $\pmb{x} \in \mathbb{R}^d$ .
Proof. First, based on the conclusion from Theorem 1 from [3] and regarding the  $t$  samples, the trained the parameters satisfy  $\| \pmb{\theta} - \pmb{\theta}_0\|_2, \| \pmb{\theta}' - \pmb{\theta}_0\|_2 \leq \mathcal{O}\left(\frac{k^3}{\rho\sqrt{m}}\log(m)\right) = \omega$  where  $\pmb{\theta}_0$  is the randomly initialized parameter. Then, we could have
$$
\begin{array}{l} \left\| \nabla_ {\boldsymbol {\theta}} f (\boldsymbol {x}; \boldsymbol {\theta}) \right\| _ {2} \leq \left\| \nabla_ {\boldsymbol {\theta} _ {0}} f (\boldsymbol {x}; \boldsymbol {\theta} _ {0}) \right\| _ {2} + \left\| \nabla_ {\boldsymbol {\theta}} f (\boldsymbol {x}; \boldsymbol {\theta}) - \nabla_ {\boldsymbol {\theta} _ {0}} f (\boldsymbol {x}; \boldsymbol {\theta} _ {0}) \right\| _ {2} \\ \leq \left(1 + \mathcal {O} \left(\frac {k L ^ {3} \log^ {5 / 6} (m)}{\rho^ {1 / 3} m ^ {1 / 6}}\right)\right) \cdot \mathcal {O} (L) \\ \end{array}
$$
w.r.t. the conclusion from Theorem 1 and Theorem 5 of [3]. Then, regarding the Lemma 4.1 from [11], we would have
$$
\left| f (\boldsymbol {x}; \boldsymbol {\theta}) - f (\boldsymbol {x}; \boldsymbol {\theta} ^ {\prime}) - \left\langle \nabla_ {\boldsymbol {\theta} ^ {\prime}} f (\boldsymbol {x}; \boldsymbol {\theta} ^ {\prime}), \boldsymbol {\theta} - \boldsymbol {\theta} ^ {\prime} \right\rangle \right| \leq \mathcal {O} \left(\omega^ {1 / 3} L ^ {2} \sqrt {m \log (m)}\right) \cdot \| \boldsymbol {\theta} - \boldsymbol {\theta} ^ {\prime} \| _ {2}.
$$
Therefore, the our target could be reformed as
$$
\begin{array}{l} \left| f (\boldsymbol {x}; \boldsymbol {\theta}) - f (\boldsymbol {x}; \boldsymbol {\theta} ^ {\prime}) \right| \leq \| \nabla_ {\boldsymbol {\theta} ^ {\prime}} f (\boldsymbol {x}; \boldsymbol {\theta} ^ {\prime}) \| _ {2} \| \boldsymbol {\theta} - \boldsymbol {\theta} ^ {\prime} \| _ {2} + \mathcal {O} \left(\omega^ {1 / 3} L ^ {2} \sqrt {m \log (m)}\right) \cdot \| \boldsymbol {\theta} - \boldsymbol {\theta} ^ {\prime} \| _ {2} \\ \leq \left(1 + \mathcal {O} \left(\frac {k L ^ {3} \log^ {5 / 6} (m)}{\rho^ {1 / 3} m ^ {1 / 6}}\right)\right) \cdot \mathcal {O} (L) \cdot \omega + \mathcal {O} \left(\omega^ {4 / 3} L ^ {2} \sqrt {m \log (m)}\right) \\ \end{array}
$$
Substituting the  $\omega$  with its value would complete the proof.
Corollary D.12. Following a similar settings as in Lemma D.11, consider a randomly initialized  $L$ -layer fully-connected network  $f(\cdot; \theta_0)$  with Sigmoid activation. For any  $0 < \xi_2 \leq 1$ ,  $0 < \rho \leq \mathcal{O}\left(\frac{1}{L}\right)$ . Let there be two sets of training samples  $\mathcal{P}_k, \mathcal{P}_k'$  with the unit-length and the  $\rho$ -separateness assumption, and let  $\pmb{\theta}$  be the trained parameter on  $\mathcal{P}_k$  while  $\pmb{\theta}'$  is the trained parameter on  $\mathcal{P}_k'$ . Suppose the conditions in Theorem 5.2 are satisfied. Then, with probability at least  $1 - \delta$ , we have
Proof. This corollary is an intuitive extension of Lemma D.11. Since the result from Theorem 1 of [3] also applies to Lipschitz-smooth (i.e., Sigmoid) activation functions, combining the proof of Lemma D.11 and the result from Lemma 7 in [47] will give the conclusion.
D.5 Regret Bound for Uniform Sampling
Lemma D.13 (Regret Bound for the Uniform Sampling Approach). When applying the uniform sampling as in most meta-learning frameworks, we denote the corresponding sampled task series as  $\Omega_u(K)$ . We will have  $R_u(K) = \mathbb{E}_{\mathcal{T} \sim \mathcal{P}(\mathcal{T}), \boldsymbol{x} \sim \mathcal{D}_{\mathcal{T}}}\left[\mathcal{L}(\boldsymbol{x}; \mathcal{I}(\mathcal{T}, \Theta_u^{(K)})) - \mathcal{L}(\boldsymbol{x}; \mathcal{I}(\mathcal{T}, \Theta^{(K),*}))\right]$ . where  $\Theta_u^{(K)}$  refers to the meta-parameters trained with uniform sampling. With  $\|\Theta_u^{(K)} - \Theta^{(K),*}\|_2 \leq \omega$ , we have the regret bound for the uniform sampling as
$$
\begin{array}{l} R _ {u} (K) = \mathbb {E} _ {\mathcal {T} \sim \mathcal {P} (\mathcal {T}), \boldsymbol {x} \sim \mathcal {D} _ {\mathcal {T}}} \left[ \mathcal {L} (\boldsymbol {x}; \mathcal {I} (\mathcal {T}, \boldsymbol {\Theta} _ {u} ^ {(K)})) - \mathcal {L} (\boldsymbol {x}; \mathcal {I} (\mathcal {T}, \boldsymbol {\Theta} ^ {(K), *})) \right] \\ \leq \sqrt {m _ {\mathcal {F}} L _ {\mathcal {F}}} \cdot \omega + \mathcal {O} \left(\omega^ {4 / 3} L _ {\mathcal {F}} ^ {3} \sqrt {m _ {\mathcal {F}} \log (m _ {\mathcal {F}})}\right) + \mathcal {O} \left(\sqrt {\frac {L _ {\mathcal {F}}}{m _ {\mathcal {F}}}}\right) \\ \leq \min  \left\{\mathcal {O} \left(K L _ {\mathcal {F}} + \frac {K ^ {4 / 3} L _ {\mathcal {F}} ^ {1 1 / 3} \sqrt {\log (m _ {\mathcal {F}})}}{m _ {\mathcal {F}} ^ {1 / 6}} + \sqrt {\frac {L _ {\mathcal {F}}}{m _ {\mathcal {F}}}}\right), 1 \right\} \\ \end{array}
$$
Proof. Here, for the simplicity of notation, we denote  $\Theta = \mathcal{I}(\mathcal{T},\Theta)$ , and neglect the expectation terms. Note that the difference between adapted meta-parameters and the original meta-parameters is
small enough and can be well-bounded. We will then have
$$
\begin{array}{l} R _ {u} (K) = \mathbb {E} _ {\mathcal {T} \sim \mathcal {P} (\mathcal {T}), \boldsymbol {x} \sim \mathcal {D} _ {\mathcal {T}}} \left[ \mathcal {L} (\boldsymbol {x}; \mathcal {I} (\mathcal {T}, \boldsymbol {\Theta} _ {u} ^ {(K)})) - \mathcal {L} (\boldsymbol {x}; \mathcal {I} (\mathcal {T}, \boldsymbol {\Theta} ^ {(K), *})) \right] \\ = \widetilde {\mathcal {L}} \left(\widetilde {\boldsymbol {\Theta}} _ {u} ^ {(K)}\right) - \widetilde {\mathcal {L}} \left(\widetilde {\boldsymbol {\Theta}} ^ {(K), *}\right) \\ \end{array}
$$
where the two sets of meta-parameters are trained with uniformly sampled tasks and the optimal tasks, and  $\widetilde{\Theta}$  is used to denote the adapted meta-parameters  $\mathcal{I}(\mathcal{T},\Theta)$  for simplicity. With any convex loss function (e.g.,  $L_{2}$  loss or cross-entropy loss) under the over-parameterization settings, we will have the generalization loss being almost convex w.r.t. the meta-parameters as in Lemma D.14, which leads to
$$
\begin{array}{l} \widetilde {\mathcal {L}} (\widetilde {\Theta} _ {u} ^ {(K)}) - \widetilde {\mathcal {L}} (\widetilde {\Theta} ^ {(K), *}) \leq \langle \nabla_ {\widetilde {\Theta}} \widetilde {\mathcal {L}} (\widetilde {\Theta} _ {u} ^ {(K)}), \widetilde {\Theta} _ {u} ^ {(K)} - \widetilde {\Theta} ^ {(K), *} \rangle + \epsilon \\ \leq \| \nabla_ {\widetilde {\Theta}} \widetilde {\mathcal {L}} (\widetilde {\Theta} _ {u} ^ {(K)}) \| _ {2} \| \widetilde {\Theta} _ {u} ^ {(K)} - \widetilde {\Theta} ^ {(K), *} \| _ {2} + \epsilon \\ \leq \| \nabla_ {\widetilde {\Theta}} \widetilde {\mathcal {L}} \left(\widetilde {\Theta} _ {u} ^ {(K)}\right) \| _ {2} \| \Theta_ {u} ^ {(K)} - \Theta^ {(K), *} \| _ {2} + \eta_ {1} \cdot \mathcal {O} (\sqrt {m _ {\mathcal {F}} L _ {\mathcal {F}}}) + \epsilon \\ \stackrel {\mathrm {(i)}} {\leq} \sqrt {m _ {\mathcal {F}} L _ {\mathcal {F}}} \cdot \omega + \mathcal {O} (\omega^ {4 / 3} L _ {\mathcal {F}} ^ {3} \sqrt {m _ {\mathcal {F}} \log (m _ {\mathcal {F}})}) + \mathcal {O} (\sqrt {\frac {L _ {\mathcal {F}}}{m _ {\mathcal {F}}}}) \\ \stackrel {\mathrm {(i i)}} {\leq} \mathcal {O} \left(K L _ {\mathcal {F}} + \frac {K ^ {4 / 3} L _ {\mathcal {F}} ^ {1 1 / 3} \sqrt {\log (m _ {\mathcal {F}})}}{m _ {\mathcal {F}} ^ {1 / 6}} + \sqrt {\frac {L _ {\mathcal {F}}}{m _ {\mathcal {F}}}}\right) \\ \stackrel {\mathrm {(i i i)}} {\Longrightarrow} \widetilde {\mathcal {L}} (\widetilde {\Theta} _ {u} ^ {(K)}) - \widetilde {\mathcal {L}} (\widetilde {\Theta} ^ {(K), *}) \leq \min  \left\{\mathcal {O} \left(K L _ {\mathcal {F}} + \frac {K ^ {4 / 3} L _ {\mathcal {F}} ^ {1 1 / 3} \sqrt {\log (m _ {\mathcal {F}})}}{m _ {\mathcal {F}} ^ {1 / 6}} + \sqrt {\frac {L _ {\mathcal {F}}}{m _ {\mathcal {F}}}}\right), 1 \right\} \\ \end{array}
$$
where  $\epsilon = \mathcal{O}(\omega^{4/3}L_{\mathcal{F}}^3\sqrt{m_{\mathcal{F}}\log(m_{\mathcal{F}})}) > 0$ , and  $\|\Theta_u^{(K),*} - \Theta_u^{(K)}\|_2 \leq \omega$ . Here, the first inequality is due to Lemma D.14 and the convexity of the loss function. The third inequality is due to the upper bound for meta-model gradients (Lemma D.15). The (i) is due to Lemma D.16 and sufficiently small learning rate  $\eta_1 \leq \mathcal{O}\left(\frac{1}{m_{\mathcal{F}}}\right)$ . Based on Lemma D.15, we will have  $\|\nabla_{\Theta}\mathcal{L}(\boldsymbol{x};\Theta_K^{(J),*})\|_2, \|\nabla_{\Theta}\mathcal{L}(\boldsymbol{x};\Theta_K^{(J)})\|_2 \leq \mathcal{O}(\sqrt{m_{\mathcal{F}}L_{\mathcal{F}}})$ . Since we have  $\eta_1, \eta_2 \leq \mathcal{O}\left(\frac{1}{m}\right)$ , starting from randomly initialized  $\Theta^{(0)}$ , the parameter shift caused by GD can be upper bounded by  $\|\Theta_u^{(K),*} - \Theta_u^{(K)}\|_2 \leq \omega = \mathcal{O}(K \cdot \sqrt{\frac{L_{\mathcal{F}}}{m_{\mathcal{F}}}})$ . The implication (iii) is because the loss function  $\mathcal{L}(\cdot; \cdot)$  has the value range [0, 1].
Here, we notice that the RHS of the regret bound in Lemma D.13 has two terms. Although the second term can be reduced to  $\mathcal{O}(1)$  with sufficiently large meta-model width  $m_{\mathcal{F}} > \mathcal{O}(\mathrm{Poly}(K,L,\rho^{-1}))$ , the first term tends to grow along with more iterations  $K$  and the larger meta-model width  $m_{\mathcal{F}}$ . The reason is that the radius for the parameter shift during meta-training  $\omega$  can be as large as  $\mathcal{O}\left(\frac{1}{\sqrt{m_{\mathcal{F}}}}\right)$ , which means that it cannot cancel out the effects of gradient norms, which have the order of  $\mathcal{O}(\sqrt{m_{\mathcal{F}}})$ . In this case, we will not be able to include a  $m_{\mathcal{F}}$  term to the denominator to scale down the regret with  $m_{\mathcal{F}}$ , and make the upper bound narrower than 1.
Lemma D.14. Given an arbitrary sample  $\pmb{x}$  and its label, let  $\tilde{\mathcal{L}}(\Theta) = \mathcal{L}(\pmb{x};\Theta)$ . Suppose  $m_{\mathcal{F}}, \eta_1, \eta_2$  satisfy the conditions in Theorem 5.2. With probability at least  $1 - \mathcal{O}(KL_{\mathcal{F}}^2) \cdot \exp[-\Omega(m_{\mathcal{F}}\omega^{2/3}L_{\mathcal{F}})]$  over randomness of  $\Theta^{(0)}$ , for all  $k \in [K]$ , and  $\Theta, \Theta'$  satisfying  $\| \Theta - \Theta^{(0)} \|_2 \leq \omega$  and  $\| \Theta' - \Theta^{(0)} \|_2 \leq \omega$  with  $\omega \leq \mathcal{O}(L_{\mathcal{F}}^{-6}[\log m_{\mathcal{F}}]^{-3/2})$ , it holds uniformly that
$$
\widetilde {\mathcal {L}} (\boldsymbol {\Theta}) - \widetilde {\mathcal {L}} (\boldsymbol {\Theta} ^ {\prime}) \leq \langle \nabla_ {\boldsymbol {\Theta}} \widetilde {\mathcal {L}} (\boldsymbol {\Theta}), \boldsymbol {\Theta} - \boldsymbol {\Theta} ^ {\prime} \rangle + \epsilon .
$$
with  $\epsilon = \mathcal{O}(\omega^{4 / 3}L_{\mathcal{F}}^3\sqrt{\log m_{\mathcal{F}}})$  being a small constant.
Proof. This proof follows an analogous approach as the proof of Lemma 4.2 in [11]. Let  $\nabla_{\mathcal{F}}\widetilde{\mathcal{L}} (\Theta^{\prime})$  be the derivative of  $\widetilde{\mathcal{L}}$  with respect to  $\mathcal{F}(\boldsymbol {x};\boldsymbol {\Theta})$ . Then, it holds that  $|\nabla_{\mathcal{F}}\widetilde{\mathcal{L}} (\Theta^{\prime})|\leq \mathcal{O}(1)$  based on
Lemma D.15. Then, by convexity of  $\widetilde{\mathcal{L}}$ , we have
$$
\begin{array}{l} \widetilde {\mathcal {L}} (\boldsymbol {\Theta} ^ {\prime}) - \widetilde {\mathcal {L}} (\boldsymbol {\Theta}) \\ \stackrel {\mathrm {(i)}} {\geq} \nabla_ {\mathcal {F}} \widetilde {\mathcal {L}} (\boldsymbol {\Theta}) \cdot \left(\mathcal {F} (\boldsymbol {x}; \boldsymbol {\Theta} ^ {\prime}) - \mathcal {F} (\boldsymbol {x}; \boldsymbol {\Theta})\right) \\ \geq \nabla_ {\mathcal {F}} \widetilde {\mathcal {L}} (\boldsymbol {\Theta} ^ {\prime}) \cdot \langle \nabla_ {\boldsymbol {\Theta}} \mathcal {F} (\boldsymbol {x}; \boldsymbol {\Theta}), \boldsymbol {\Theta} ^ {\prime} - \boldsymbol {\Theta} \rangle \\ - \left| \nabla_ {\mathcal {F}} \tilde {\mathcal {L}} \left(\Theta^ {\prime}\right) \right| \cdot \left| \mathcal {F} (\boldsymbol {x}; \Theta^ {\prime}) - \mathcal {F} (\boldsymbol {x}; \Theta) - \langle \nabla \mathcal {F} (\boldsymbol {x}; \Theta), \Theta^ {\prime} - \Theta \rangle \right| \\ \geq \left\langle \nabla_ {\Theta} \widetilde {\mathcal {L}} (\Theta), \Theta^ {\prime} - \Theta \right\rangle - | \nabla_ {\mathcal {F}} \widetilde {\mathcal {L}} (\Theta^ {\prime}) | \cdot | \mathcal {F} (\boldsymbol {x}; \Theta^ {\prime}) - \mathcal {F} (\boldsymbol {x}; \Theta) - \langle \nabla \mathcal {F} (\boldsymbol {x}; \Theta), \Theta^ {\prime} - \Theta \rangle | \\ \stackrel {\text {(i i i)}} {\geq} \langle \nabla_ {\Theta} \tilde {\mathcal {L}} (\Theta), \Theta^ {\prime} - \Theta \rangle - \mathcal {O} \left(\omega^ {4 / 3} L _ {\mathcal {F}} ^ {3} \sqrt {m _ {\mathcal {F}} \log (m _ {\mathcal {F}})}\right) \\ \geq \langle \nabla_ {\Theta} \widetilde {\mathcal {L}} (\Theta), \Theta^ {\prime} - \Theta \rangle - \epsilon \\ \end{array}
$$
where (i) is due to the convexity of the loss function  $\mathcal{L}$ , (ii) is an application of triangle inequality, and (iii) is the application of and Lemma D.16. Finally, denoting  $\epsilon = \mathcal{O}(\omega^{4/3} L_{\mathcal{F}}^3 \sqrt{m_{\mathcal{F}} \log m_{\mathcal{F}}})$  will complete the proof.
Lemma D.15. Suppose  $m_{\mathcal{F}}, \eta_1, \eta_2$  satisfy the conditions in Theorem 5.2. With probability at least  $1 - \mathcal{O}(KL_{\mathcal{F}}) \cdot \exp(-\Omega(m_{\mathcal{F}}\omega^{2/3}L_{\mathcal{F}}))$  over the random initialization,  $\Theta$  satisfying  $\| \Theta - \Theta^{(0)} \|_2 \leq \omega$  with  $\omega \leq \mathcal{O}(L_{\mathcal{F}}^{-9/2}[\log m_{\mathcal{F}}]^{-3})$ , it holds uniformly that