SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\gamma \lesssim \frac {1}{L _ {F}} \min  \left\{\min  \left\{1, \frac {p _ {\mathrm {o u t}}}{\sqrt {1 - p _ {\mathrm {o u t}}}} \right\} \frac {p _ {\mathrm {i n}} p _ {\mathrm {o u t}} \sqrt {B _ {1}} \sqrt {S _ {1}}}{(1 - p _ {\mathrm {o u t}}) \sqrt {1 - p _ {\mathrm {i n}}}} \frac {1}{L _ {F}}, \min  \left\{1, \frac {p _ {\mathrm {o u t}}}{\sqrt {1 - p _ {\mathrm {o u t}}}} \right\} \frac {p _ {\mathrm {i n}} \sqrt {S _ {1}} \sqrt {B _ {2}}}{\sqrt {1 - p _ {\mathrm {i n}}}}, 1, \frac {p _ {\mathrm {o u t}} p _ {\mathrm {i n}} \sqrt {S _ {1}}}{5 L _ {F} (1 - p _ {\mathrm {i n}}) ^ {3 / 2}} \right\}.
$$
This can be simplified as an upper bound
$$
\gamma \lesssim \frac {1}{L _ {F}} \min  \left\{\frac {p _ {\text {i n}} p _ {\text {o u t}} \sqrt {S _ {1}}}{\sqrt {1} - p _ {\text {i n}}}, \frac {p _ {\text {i n}} p _ {\text {o u t}} \sqrt {S _ {1}} \sqrt {B _ {1}}}{\sqrt {1} - p _ {\text {o u t}}}, \frac {p _ {\text {i n}} p _ {\text {o u t}} ^ {2} \sqrt {S _ {1}} \sqrt {B _ {1}}}{\sqrt {1} - p _ {\text {i n}} \sqrt {1} - p _ {\text {o u t}}}, 1 \right\}.
$$
Now we consider two sets of hyperparameters depending on the size of  $n$  Case 1: For  $n = \mathcal{O}(\varepsilon^{-2/3})$ , we choose the following set of hyperparameters
$$
B _ {1} = B _ {2} = n, \quad p _ {\mathrm {o u t}} = 1, \quad S _ {1} = \tilde {L} _ {F} ^ {2} \varepsilon^ {- 2}, \quad S _ {2} = \tilde {L} _ {F} \varepsilon^ {- 1}, \quad p _ {\mathrm {i n}} = \tilde {L} _ {F} ^ {- 1} \varepsilon .
$$
Then we have  $\gamma \lesssim \frac{1}{L_F} \min \left\{\frac{p_{\mathrm{in}} \sqrt{S_1}}{\sqrt{1 - p_{\mathrm{in}}}}, 1\right\} = \frac{1}{L_F}$ , we have the total sample complexity of
$$
B _ {2} S _ {2} T = \frac {B _ {2} S _ {2} T \gamma}{\gamma} \stackrel {(3 9)} {=} \frac {F (\pmb {x} ^ {0}) - F ^ {\star}}{\varepsilon^ {2}} \frac {B _ {2} S _ {2}}{\gamma} = \frac {(F (\pmb {x} ^ {0}) - F ^ {\star}) n \tilde {L} _ {F} L _ {F}}{\varepsilon^ {3}}
$$
Figure 4: Performance of BSGD vs. E-BSGD on the few-shot sinusoid regression task.
Case 2: For  $n = \Omega(\varepsilon^{-2/3})$ , we choose the following set of hyperparameters
$$
B _ {1} = n, \quad B _ {2} = \sqrt {n}, \quad p _ {\mathrm {o u t}} = \frac {1}{\sqrt {n}}.
$$
In this case, (46) is stronger than (47) which requires  $S_{1} \gtrsim \frac{\tilde{L}_{F}^{2}}{n\varepsilon^{2}}$
$$
S _ {1} = S _ {2} = \max \left\{\tilde {\sigma} _ {\mathrm {b i a s}} ^ {1 / 2} \varepsilon^ {- 1 / 2}, \frac {\sigma_ {\mathrm {i n}} ^ {2}}{n \varepsilon^ {2}} \right\}, p _ {\mathrm {i n}} = 1
$$
Then we have  $\gamma \lesssim \frac{1}{L_F} \min \left\{\frac{p_{\mathrm{out}} \sqrt{n} \sqrt{S_1}}{\sqrt{1 - p_{\mathrm{out}}}}, 1\right\} = \frac{1}{L_F}$ , we have the total sample complexity of
$$
B _ {2} S _ {2} T = \frac {B _ {2} S _ {2} T \gamma}{\gamma} \stackrel {(3 9)} {=} \frac {F (\pmb {x} ^ {0}) - F ^ {\star}}{\varepsilon^ {2}} \frac {B _ {2} S _ {2}}{\gamma} = (F (\pmb {x} ^ {0}) - F ^ {\star}) \max \left\{\frac {\sqrt {n} \tilde {\sigma} _ {\mathrm {b i a s}} ^ {1 / 2}}{\varepsilon^ {2 . 5}}, \frac {\sigma_ {\mathrm {i n}} ^ {2}}{\sqrt {n} \varepsilon^ {4}} \right\}.
$$
F Missing Details from Section 5
F.1 Application of First-order MAML
Over the past few years, the MAML framework [11] has become quite popular for few-shot supervised learning and meta reinforcement learning tasks. The first-order Model-Agnostic Meta-Learning (MAML) can be formulated mathematically as follows:
$$
\min  _ {\boldsymbol {x}} \mathbb {E} _ {i \sim p, \mathcal {D} _ {\text {q u e r y}} ^ {i}} \ell_ {i} \left(\mathbb {E} _ {\mathcal {D} _ {\text {s u p p}} ^ {i}} (\boldsymbol {x} - \alpha \nabla \ell_ {i} (\boldsymbol {x}, \mathcal {D} _ {\text {s u p p}} ^ {i})) , \mathcal {D} _ {\text {q u e r y}} ^ {i}\right)
$$
where  $\alpha$  is the step size,  $\mathcal{D}_{\mathrm{supp}}^i$  and  $\mathcal{D}_{\mathrm{query}}^i$  are meta-training and meta-testing data respectively and  $\ell_i$  being the loss function of task  $i$ . Stated in the CSO framework,  $f_{\xi}(\boldsymbol{x}) \coloneqq \ell_i(\boldsymbol{x}, \mathcal{D}_{\mathrm{query}}^i)$  and  $g_{\eta}(\boldsymbol{x}, \xi) \coloneqq \boldsymbol{x} - \alpha \nabla \ell_i(\boldsymbol{x}, \mathcal{D}_{\mathrm{supp}}^i)$  where  $\xi = (i, \mathcal{D}_{\mathrm{query}}^i)$  and  $\eta = \mathcal{D}_{\mathrm{supp}}^i$ .
In this context, lots of popular choices for  $f_{\xi}$  are smooth. For illustration purposes, we now discuss a widely used sine-wave few-shot regression task as appearing from the work of Finn et al. [11], where the goal is to do a few-shot learning of a sine wave,  $A\sin (t - \phi)$ , using a neural network  $\Phi_x(t)$  with smooth activations, where  $A$  and  $\phi$  represent the unknown amplitude and phase, and  $x$  denotes the model weight. Each task  $i$  is characterized by  $(A^i,\phi^i,\mathcal{D}_{\mathrm{query}}^i)$ . In the first-order MAML training process, we randomly select a task  $i$ , and draw training data  $\eta = \mathcal{D}_{\mathrm{supp}}^i$ . Define the loss function for a given dataset  $\mathcal{D}$  as  $\ell_i(\Phi_x;\mathcal{D}) = \frac{1}{2}\mathbb{E}_{t\sim \mathcal{D}}\left\| A^i\sin (t - \phi^i) - \Phi_x(t)\right\| _2^2$ . We then establish the outer function  $f_{i}(\boldsymbol {x}) = \ell_{i}(\Phi_{\boldsymbol{x}};\mathcal{D}_{\mathrm{query}}^{i})$  and inner function  $g_{\eta}(\pmb {x}) = \pmb {x} - \alpha \nabla_{\pmb{x}}\ell_{i}(\Phi_{\pmb{x}};\mathcal{D}_{\mathrm{supp}}^{i})$ . As  $f_{i}$  is smooth, our results are applicable.
In Figure 4, we show the results of BSGD and E-BSGD applied to this problem. In this experiment, the amplitude  $A$  is drawn from a uniform distribution  $\mathcal{U}(0.1,5)$  and the phase  $\phi$  is drawn from  $\mathcal{U}(0,\pi)$ . Both  $\mathcal{D}_{\mathrm{supp}}$  and  $\mathcal{D}_{\mathrm{query}}$  are independently drawn from  $\mathcal{U}(-5,5)$ . The step size is set to
$\alpha = 0.01$ . The batch size is fixed to 10. The performances of BSGD and E-BSGD are very close. This is not surprising because finetuning step size  $\alpha$  is chosen to be small which significantly reduces the variance of  $g_{\eta}$ , making the bias of meta gradient to be very small  $(\mathcal{O}(\alpha^{2}))$ . Therefore, we observe similar performance of BSGD and E-BSGD. Similar trend also holds for BSpiderBoost and NestedVR compared to their extrapolated variants.
F.2 Application of Deep Average Precision Maximization
The areas under precision-recall curve (AUPRC) has an unbiased point estimator that maximizes average precision (AP) [26, 34]. Let  $S_{+}$  and  $S_{-}$  be the set of positive and negative samples and  $S = S_{-} \cup S_{+}$ . Let  $h_{\boldsymbol{w}}(\cdot)$  be a classifier parameterized with  $\boldsymbol{w}$  and  $\ell$  be a surrogate function, such as logistic or sigmoid. A smooth surrogate objective for maximizing average precision can be formulated as [33]:
$$
F (\boldsymbol {w}) = - \frac {1}{| \mathcal {S} _ {+} |} \sum_ {\boldsymbol {x} _ {i} \in \mathcal {S} _ {+}} \frac {\sum_ {\boldsymbol {x} \in \mathcal {S} _ {+}} \ell \left(h _ {\boldsymbol {w}} (\boldsymbol {x}) - h _ {\boldsymbol {w}} \left(\boldsymbol {x} _ {i}\right)\right)}{\sum_ {\boldsymbol {x} \in \mathcal {S}} \ell \left(h _ {\boldsymbol {w}} (\boldsymbol {x}) - h _ {\boldsymbol {w}} \left(\boldsymbol {x} _ {i}\right)\right)}
$$
This problem can be seen as a conditional stochastic optimization problem with  $g_{i}(\boldsymbol{w}) = \left[\sum_{\boldsymbol{x} \in S_{+}} \ell(h_{\boldsymbol{w}}(\boldsymbol{x}) - h_{\boldsymbol{w}}(\boldsymbol{x}_{i})) \right]$ , where  $f_{i}: \mathbb{R} \times \mathbb{R} \backslash \{0\} \to \mathbb{R}$  is defined as  $f_{i}(\boldsymbol{y}) = -\frac{[\boldsymbol{y}]_{1}}{[\boldsymbol{y}]_{2}}$  where  $[\boldsymbol{y}]_k$  denotes the  $k$ th coordinate of a vector  $\boldsymbol{y} \in \mathbb{R} \times \mathbb{R} \backslash \{0\}$ . During the stochastic optimization of this objective, we draw uniformly at random  $\xi := x_i$  (drawn from the set  $S_{+}$ ) as a positive sample and  $\eta | \xi = [\mathcal{F}_{x_1}, \mathcal{F}_{x_2}]$  where set  $x_1$  is drawn uniformly at random from  $S_{+}$  and  $x_2$  is drawn uniformly at random from  $S$  and functional  $\mathcal{F}_{\boldsymbol{x}}(\boldsymbol{w}) := \ell(h_w(\boldsymbol{x}) - h_w(\boldsymbol{x}_i))$ . Note that  $f_{i} \in C^{\infty}$  is smooth with gradient
$$
\nabla f _ {i} (\boldsymbol {y}) = \left[ \begin{array}{c} - \frac {1}{[ \boldsymbol {y} ] _ {2}} \\ \frac {[ \boldsymbol {y} ] _ {1}}{([ \boldsymbol {y} ] _ {2}) ^ {2}} \end{array} \right].
$$
Therefore, our results from Sections 3 and 4 again apply.
F.3 Necessity of Additional Smoothness Conditions
Throughout the paper, we assume bounded moments (Assumption 1) and a smoothness condition (Assumption 2) to derive our extrapolation technique. However, it is worth noting that the technique itself does not explicitly depend on higher-order derivatives. Our theoretical framework does not address the behavior of extrapolation in the absence of these smoothness constraints. In this section, we investigate the application of extrapolation to two non-smooth functions:
- ReLU function given by  $q(x) = \max \{x, 0\}$ ;
- Perturbed quadratics represented as  $q(x) = x^2 / 2 + \mathrm{TriangleWave}(\mathbf{x}) + 1$ . The function TriangleWave(x) has a period of 2 and spans the range [-1,1], defined as:
$$
\operatorname {T r i a n g l e W a v e} (x) = 2 \left| 2 \left(\frac {x}{2} - \left\lfloor \frac {x}{2} + \frac {1}{2} \right\rfloor\right) \right| - 1
$$
Visual representations of these functions can be found in Figure 5c. We set  $s = 0$  and consider a random variable  $\delta \sim \mathcal{N}(10,100)$  with  $m = 1$ . We then apply first-, second-, and third-order extrapolation. The outcomes are depicted in Figure 5. Remarkably, both the ReLU and the perturbed quadratic functions do not conform to the differentiability assumptions inherent to our stochastic extrapolation schemes. Nonetheless, as indicated by Figure 5a and Figure 5b, our proposed second- and third-order extrapolation techniques yield a superior approximation of  $q(\mathbb{E}[\delta])$ .
Figure 5c
Figure 5: (a) Fig. 5a: Error in estimating  $q(s + \mathbb{E}[\delta])$  for our proposed first-, second-, and third-order extrapolation schemes applied to ReLU  $q(x) = \max \{x,0\}$ ,  $s = 0$ ,  $\delta \sim \mathcal{N}(10,100)$ ,  $m = 1$ . (b) Fig 5b: Error in estimating  $q(s + \mathbb{E}[\delta])$  for our proposed first-, second-, and third-order extrapolation schemes applied to a perturbed quadratic  $q(x) = x^2 /2 + \mathrm{TriangleWave}(\mathrm{x}) + 1$ ,  $s = 0$ ,  $\delta \sim \mathcal{N}(10,100)$ ,  $m = 1$ . The TriangleWave(x) has a period of 2 and spans the range [-1,1], i.e.  $2|2\left(\frac{x}{2} -\lfloor \frac{x}{2} +\frac{1}{2}\rfloor\right)| - 1$ . (c) Fig 5c: The ReLU and perturbed quadratic used in the Fig. 5a and 5b along with quadratic curves.