SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $z_{1}$  is the first dimension of  $Z$ ,  $e$  is the confounding variable and  $\delta$ ,  $\gamma$  are noises. In this experiment, we solve the instrumental variable regression using BSGD, BSpiderBoost, NestedVR and their extrapolated variants described in Sections 3 and 4. In each pair of experiments, the samples used per iteration are fixed same, i.e.: 1) BSGD uses  $m = 2$  and E-BSGD uses  $m = 1$ ; 2) For BSpiderBoost and E-BSpiderBoost, we use cycle length of 10, small batch and large batch in Spider to be 10 and 100 respectively, and we choose inner batch sizes  $m = 2$  for BSpiderBoost and  $m = 1$  for E-BSpiderBoost; 3) For NestedVR and E-NestedVR, we fix the outer batch size to 10 and 5 respectively, and choose fix the inner Spider Cycle to be 10 with large batch 100 and small batch 10. The results are presented in Figure 3. As is quite evident, the extrapolation variants achieve faster convergence in all 3 cases, confirming our theoretical findings.
In this paper, we consider the conditional stochastic optimization CSO problem and its finite-sum variant FCCO. Due to the interplay between nested structure and stochasticity, most of the existing gradient estimates suffer from large biases and have large sample complexity of  $\mathcal{O}(\varepsilon^{-5})$ . We propose stochastic extrapolation-based algorithms that tackle this bias problem and improve the sample complexities for both these problems. While we focus on nonconvex objectives, our proposed algorithms can also be beneficial when used with strongly convex, convex objectives. We also believe that similar ideas could also prove helpful for multi-level stochastic optimization problems [41] with nested dependency.
We would like to thank Caner Turkmen, Sai Praneeth Karimireddy, and Martin Jaggi for helpful initial discussions surrounding this project.
[37] Mengdi Wang, Ethan X Fang, and Han Liu. Stochastic compositional gradient descent: algorithms for minimizing compositions of expected-value functions. Mathematical Programming, 161:419-449, 2017.
[38] Zhe Wang, Kaiyi Ji, Yi Zhou, Yingbin Liang, and Vahid Tarokh. Spiderboost and momentum: Faster variance reduction algorithms. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 2403-2413, 2019. URL https://proceedings.neurips.cc/paper/2019/hash/512c5cad6c37edb98ae91c8a76c3a291-AAbstract.html.
[39] Christopher Stroude Withers. Bias reduction by taylor series. Communications in Statistics-Theory and Methods, 16(8):2369-2383, 1987.
[40] Yu M Yermol'yev. A general stochastic programming problem. 1971.
[41] Junyu Zhang and Lin Xiao. Multilevel composite stochastic optimization via nested variance reduction. SIAM Journal on Optimization, 31(2):1131-1157, 2021.
A Missing Pseudocodes 15
B Missing Details from Section 1 15
B.1 Other Related Work 15
C Missing Details from Section 2 17
D Stationary Point Convergence Proofs from Section 3 (CSO) 21
D.1 Helpful Lemmas 21
D.2 Convergence of BSGD 23
D.3 Convergence of E-BSGD 25
D.4 Convergence of BSpiderBoost 26
D.5 Convergence of E-BSpiderBoost 30
E Stationary Point Convergence Proofs from Section 4 (FCCO) 31
E.1 E-BSpiderBoost for FCCO problem 31
E.2 Convergence of NestedVR 32
E.3 Convergence of E-NestedVR 41
F Missing Details from Section 5 46
F.1 Application of First-order MAML 46
F.2 Application of Deep Average Precision Maximization 47
F.3 Necessity of Additional Smoothness Conditions 47
A Missing Pseudocodes
We present pseudocodes of E-BSGD and E-BSpiderBoost scheme in Algorithms 2 and 3 respectively.
Algorithm 2 E-BSGD
1: Input:  $\pmb{x}^0 \in \mathbb{R}^d$ , step-size  $\gamma$ , batch sizes  $m$
2: for  $t = 0,1,\ldots ,T - 1$  do
3: Draw one sample  $\xi$  and compute extrapolated gradient  $G_{\mathrm{E - BSGD}}^{t + 1}$  from (7)
4:  $\pmb{x}^{t + 1}\gets \pmb{x}^t -\gamma G_{\mathrm{E - BSGD}}^{t + 1}$
5: end for
6: Output:  $x^{s}$  picked uniformly at random from  $\{x^t\}_{t=0}^{T-1}$
B Missing Details from Section 1
B.1 Other Related Work
CSO. Dai et al. [5] proposed a primal-dual stochastic approximation algorithm to solve a min-max reformulation of CSO, employing the kernel embedding techniques. However, this method requires convexity of  $f_{\xi}$  and linearity of  $g_{\eta}$ , which are not satisfied by general applications when neural networks are involved. Goda and Kitade [13] showed that a special class of CSO problems can be
Algorithm 3 E-BSpiderBoost
1: Input:  $x^0 \in \mathbb{R}^d$ , step-size  $\gamma$ , batch sizes  $B_1, B_2$ , Probability  $p_{\mathrm{out}}$   
2: for  $t = 0, 1, \ldots, T - 1$  do  
3: Draw  $\chi_{\mathrm{out}}$  from Bernoulli  $(p_{\mathrm{out}})$   
4: if  $(t = 0)$  or  $(\chi_{\mathrm{out}} = 1)$  then ▷ Large batch  
5: Draw  $\mathcal{B}_1$  outer samples  $\{\xi_1, \dots, \xi_{B_1}\}$   
6: Compute extrapolated gradient  $G_{\mathrm{E-BSGD}}^{t+1}$  with (7)  
 $G_{\mathrm{E-BSB}}^{t+1} = \frac{1}{B_1} \sum_{\xi \in \mathcal{B}_1} G_{\mathrm{E-BSGD}}^{t+1}$   
7: else ▷ Small batch  
8: Draw  $\mathcal{B}_2$  outer samples  $\{\xi_1, \dots, \xi_{B_2}\}$   
9: Compute extrapolated gradient  $G_{\mathrm{E-BSGD}}^{t+1}$  with (7)  
 $G_{\mathrm{E-BSB}}^{t+1} = G_{\mathrm{E-BSB}}^{t} + \frac{1}{B_2} \sum_{\xi \in \mathcal{B}_2} (G_{\mathrm{E-BSGD}}^{t+1} - G_{\mathrm{E-BSGD}}^{t})$   
10: end if  
11:  $x^{t+1} = x^t - \gamma G_{\mathrm{E-BSB}}^{t+1}$   
12: end for  
13: Output:  $x^s$  picked uniformly at random from  $\{x^t\}_{t=0}^{T-1}$
unbiased, e.g., when  $f_{\xi}$  measures the squared error between some  $u(\xi)$  and  $\mathbb{E}_{\eta|\xi}[g_{\eta}(\pmb{x};\xi)]$ , giving rise to this objective function  $\mathbb{E}_{\xi}[(u(\xi) - \mathbb{E}_{\eta|\xi}[g_{\eta}(\pmb{x};\xi)]^2]$ . However, they did not show any improvement over the sample complexity of BSGD (i.e.,  $\mathcal{O}(\varepsilon^{-6})$ ). Hu et al. [16] also analyzed lower bounds on the minimax error for the CSO problem and showed that for a specific class of biased gradients with  $\mathcal{O}(\varepsilon)$  bias (same bias as BSGD) and variance  $\mathcal{O}(1)$  the bound achieved by BSpiderBoost is tight. However, these lower bounds are not applicable in settings such as ours (and also to [17]) where the bias is smaller than the BSGD bias.
Variance Reduction. The reduction of variance in stochastic optimization is a crucial approach to decrease sample complexity, particularly when dealing with finite-sum formulations of the form  $\min_{\boldsymbol{x}} \frac{1}{n} \sum_{i=1}^{n} f_i(\boldsymbol{x})$ . Pioneering works such as Stochastic Average Gradient (SAG) [30], Stochastic Variance Reduced Gradient (SVRG) [21, 28], and SAGA [7, 29] improved the iteration complexity from  $\mathcal{O}(\varepsilon^{-4})$  in Stochastic Gradient Descent (SGD) to  $\mathcal{O}(\varepsilon^{-2})$ . Subsequent research, including Stochastic Path-Integrated Differential Estimator (SPIDER) [10] and Stochastic Recursive Gradient Algorithm (SARAH) [25], expanded the application of these techniques to both finite-sum and online scenarios, where  $n$  is large or possibly infinite. These methods boast an improved sample complexity of  $\min(\sqrt{n}\varepsilon^{-2}, \varepsilon^{-3})$ . SpiderBoost [38], achieves the same near-optimal complexity performance as SPIDER, but allows a much larger step size and hence runs faster in practice than SPIDER. In this paper, we use a probabilistic variant of SpiderBoost as the variance reduction module for CSO and FCCO problems. We highlight that alternative techniques, such as SARAH, can also be applied and offer similar guarantees.
Bias Correction. One of the classic problems in statistics is to design procedures to reduce the bias of estimators. Well-established general bias correction techniques, such as the jackknife [32], bootstrap [8], Taylor series [39, 14], have been extensively studied and applied in various contexts [20]. However, these methods are predominantly examined in relation to standard statistical distributions, with limited emphasis on their adaptability to optimization problems. Our proposed extrapolation-based approach is derived from sample-splitting methods [14], specifically tailored and analyzed for optimization problems involving unknown distributions.
Stochastic Composition Optimization. Finally, a closely related class of problems, called stochastic composition optimization, has been extensively studied (e.g., [40, 9, 36, 37]) in the literature where the goal is:
$$
\min  _ {\boldsymbol {x} \in \mathbb {R} ^ {d}} \mathbb {E} _ {\xi} \left[ f _ {\xi} \left(\mathbb {E} _ {\eta} \left[ g _ {\eta} (\boldsymbol {x}) \right]\right) \right]. \tag {12}
$$
Despite having nested expectations in their formulations (CSO) and (12) are fundamentally different: a) in stochastic composite optimization the inner randomness  $\eta$  is conditionally dependent on the outer randomness  $\xi$  and b) in CSO the inner random function  $g_{\eta}(\pmb{x},\xi)$  depends on both  $\xi$  and  $\eta$ . These differences lead to quite different sample complexity bounds for these problems, as explored in Hu et al. [15]. In fact, Zhang and Xiao [41] presented a near optimal complexity of  $\mathcal{O}(\min(\varepsilon^{-3},\sqrt{n\varepsilon^{-2}}))$
for stochastic composite optimization problems using nested variance reduction. While Wang et al. [36] also use the "extrapolation" technique, their motivation and formula are significantly different from ours and cannot reduce the bias in the CSO problem.
C Missing Details from Section 2
Lemma 1 (Moments of  $\mathcal{D}_m$ ) The moments of  $\delta \in \mathcal{D}_m$  are bounded as follows
$$
\mathbb {E} \big [ (\delta - \mathbb {E} [ \delta ]) ^ {2} \big ] = \frac {\sigma_ {2}}{m}, \quad | \mathbb {E} \big [ (\delta - \mathbb {E} [ \delta ]) ^ {3} \big ] | = \frac {\sigma_ {3}}{m ^ {2}}, \quad \mathbb {E} \big [ (\delta - \mathbb {E} [ \delta ]) ^ {4} \big ] = \frac {\sigma_ {4}}{m ^ {3}} + \frac {3 (m - 1) \sigma_ {2} ^ {2}}{m ^ {3}}.
$$
More generally, for  $k \geq 2$ ,  $|\mathbb{E}[(\delta - \mathbb{E}[\delta])^k]| = \mathcal{O}(m^{-\lceil k/2 \rceil})$ .
Proof: Define  $\hat{\delta} = \delta -\mathbb{E}[\delta ]$  as the centered random variable. Now
$$
\mathbb {E} \left[ \left(\delta - \mathbb {E} [ \delta ]\right) ^ {k} \right] = \mathbb {E} \left[ \hat {\delta} ^ {k} \right].
$$
So we focus on  $\mathbb{E}[\hat{\delta}^k ]$  in the remainder of the proof. For  $k = 2$
$$
| \mathbb {E} [ \hat {\delta} ^ {2} ] | = \frac {1}{m ^ {2}} | \mathbb {E} [ \sum_ {i = 1} ^ {m} \hat {\delta} _ {i} ] ^ {2} | = \frac {1}{m ^ {2}} \left| \mathbb {E} \left[ \sum_ {i} \hat {\delta} _ {i} ^ {2} + 2 \sum_ {i <   j} \hat {\delta} _ {i} \hat {\delta} _ {j} \right] \right| = \frac {\sigma_ {2}}{m}.
$$
For  $k = 3$
$$
\begin{array}{l} | \mathbb {E} [ \hat {\delta} ^ {3} ] | = \frac {1}{m ^ {3}} | \mathbb {E} [ \sum_ {i = 1} ^ {m} \hat {\delta} _ {i} ] ^ {3} | \\ = \frac {1}{m ^ {3}} \left| \mathbb {E} \left[ \sum_ {i} \hat {\delta} _ {i} ^ {3} + 3 \sum_ {i \neq j} \hat {\delta} _ {i} ^ {2} \hat {\delta} _ {j} + 6 \sum_ {i <   j <   k} \hat {\delta} _ {i} \hat {\delta} _ {j} \hat {\delta} _ {k} \right] \right| \\ = \frac {\sigma_ {3}}{m ^ {2}}. \\ \end{array}
$$
For  $k = 4$
$$
\begin{array}{l} | \mathbb {E} [ \hat {\delta} ^ {4} ] | = \frac {1}{m ^ {4}} | \mathbb {E} [ \sum_ {i = 1} ^ {m} \hat {\delta} _ {i} ] ^ {4} | \\ = \frac {1}{m ^ {4}} \left| \mathbb {E} \left[ \sum_ {i} \hat {\delta} _ {i} ^ {4} + 4 \sum_ {i \neq j} \hat {\delta} _ {i} ^ {3} \hat {\delta} _ {j} + 6 \sum_ {i <   j} \hat {\delta} _ {i} ^ {2} \hat {\delta} _ {j} ^ {2} + 2 4 \sum_ {i <   j <   k <   l} \hat {\delta} _ {i} \hat {\delta} _ {j} \hat {\delta} _ {k} \hat {\delta} _ {l} \right] \right| \\ = \frac {1}{m ^ {4}} \left| m \mathbb {E} [ \hat {\delta} _ {i} ^ {4} ] + 6 \frac {m (m - 1)}{2} \mathbb {E} [ \hat {\delta} _ {i} ^ {2} ] \mathbb {E} [ \hat {\delta} _ {j} ^ {2} ] \right| \\ = \frac {\sigma_ {4}}{m ^ {3}} + \frac {3 (m - 1) \sigma_ {2} ^ {2}}{m ^ {3}}. \\ \end{array}
$$
For  $k = 5$
$$
\begin{array}{l} | \mathbb {E} [ \hat {\delta} ^ {5} ] | = \frac {1}{m ^ {5}} | \mathbb {E} [ \sum_ {i = 1} ^ {m} \hat {\delta} _ {i} ] ^ {5} | \\ = \frac {1}{m ^ {5}} \left| \mathbb {E} \left[ \sum_ {i} \hat {\delta} _ {i} ^ {5} + 1 0 \sum_ {i \neq j} \hat {\delta} _ {i} ^ {3} \hat {\delta} _ {j} ^ {2} \right] \right| \\ = \frac {1}{m ^ {5}} \left| m \mathbb {E} [ \hat {\delta_ {i} ^ {5}} ] + 1 0 m (m - 1) \mathbb {E} [ \hat {\delta_ {i} ^ {3}} ] \mathbb {E} [ \hat {\delta_ {j} ^ {2}} ] \right| \\ = \frac {\sigma_ {5}}{m ^ {4}} + \frac {1 0 (m - 1) \sigma_ {3} \sigma_ {2}}{m ^ {4}}. \\ \end{array}
$$
For general  $k > 0$ , we expand the following term as a function of  $m$
$$
\left| \mathbb {E} [ \hat {\delta} ^ {k} ] \right| = \frac {1}{m ^ {k}} \left| \mathbb {E} [ \sum_ {i = 1} ^ {m} \hat {\delta} _ {i} ] ^ {k} \right|.
$$
As  $\mathbb{E}[\hat{\delta}_i] = 0$  and  $\hat{\delta}_i$  and  $\hat{\delta}_j$  are independent for different  $i$  and  $j$ , the outcome has the following form
$$
\left| \mathbb {E} \left[ \hat {\delta} ^ {k} \right] \right| = \frac {1}{m ^ {k}} \mathcal {O} \left(\sum_ {\substack {2 a _ {2} + 3 a _ {3} + \dots + k a _ {k} = k \\ a _ {i} \geq 0 \forall i}} m ^ {\sum_ {i = 2} ^ {k} a _ {i}} \sigma_ {2} ^ {a _ {2}} \sigma_ {3} ^ {a _ {3}} \dots \sigma_ {k} ^ {a _ {k}}\right) \tag{13}
$$
where  $\sum_{i=2}^{k} a_i$  is the count of independent  $\{\hat{\delta}_i\}$  used in  $\sigma_2^{a_2} \sigma_3^{a_3} \cdots \sigma_4^{a_4}$ . Among the terms in (13), the dominating one in terms of  $m$  is one with largest  $\sum_{i=2}^{k} a_i$ , i.e.
$$
| \mathbb {E} [ \hat {\delta} ^ {k} ] | = \left\{ \begin{array}{l l} \frac {1}{m ^ {k}} \mathcal {O} (m ^ {k / 2}) \sigma_ {2} ^ {k / 2} & \text {i f k e v e n}, \\ \frac {1}{m ^ {k}} \mathcal {O} (m ^ {\lfloor k / 2 \rfloor}) \sigma_ {2} ^ {\lfloor k / 2 \rfloor - 1} \sigma_ {3} & \text {i f k o d d}. \end{array} \right.
$$
Then, we can simplify the upper right-hand side with
$$
\left. \mid \mathbb {E} [ \hat {\delta} ^ {k} ] \right| = \mathcal {O} \left(m ^ {- k + \lfloor k / 2 \rfloor}\right),
$$
which gives all the desired results.
Proposition 2 (First-order Guarantee) Assume that  $\mathcal{D}_m$  and  $q(\cdot)$  satisfy Assumption 1 and 2 respectively with  $k = 1$ . Then,  $\forall s \in \mathbb{R}$ ,  $\left|\mathbb{E}\left[\mathcal{L}_{\mathcal{D}_m}^{(1)}q(s)\right] - q(s + \mathbb{E}[\delta])\right| \leq a_2\sigma_2 / (2m)$ .
Proof: Let  $h = \mathbb{E}[\delta]$ . If the function  $q \in \mathcal{C}^2$ , then the Taylor expansion at  $s + h$  with remainders leads to
$$
\mathbb {E} [ q (s + \delta) ] = q (s + h) + q ^ {\prime} (s + h) \mathbb {E} [ \delta - h ] + \frac {1}{2} \mathbb {E} [ q ^ {\prime \prime} (\phi_ {1}) (\delta - h) ^ {2} ]
$$
where  $\phi_1$  between  $s + h$  and  $s + \delta$ . Then the error of extrapolation becomes
$$
\left| \mathbb {E} [ q (s + \delta) ] - q (s + h) \right| = \left| \frac {1}{2} \mathbb {E} [ q ^ {\prime \prime} (\phi_ {1}) (\delta - h) ^ {2} ] \right| \leq \frac {a _ {2}}{2} \mathbb {E} [ (\delta - h) ^ {2} ].
$$
By Assumption 2 and Lemma 1, we have that
$$
\left| \mathbb {E} \left[ q (s + \delta) \right] - q (s + h) \right| \leq \frac {a _ {2}}{2} \mathbb {E} \left[ (\delta - h) ^ {2} \right] = \frac {a _ {2}}{2} \mathbb {E} \left[ (\delta - h) ^ {2} \right] = \frac {a _ {2} \sigma_ {2}}{2 m}.
$$
Proposition 1 (Second-order Guarantee) Assume that distribution  $\mathcal{D}_m$  and  $q(\cdot)$  satisfies Assumption 1 and 2 respectively with  $k = 2$ . Then, for all  $s \in \mathbb{R}$ ,  $\left|\mathbb{E}\left[\mathcal{L}_{\mathcal{D}_m}^{(2)}q(s)\right] - q(s + \mathbb{E}[\delta])\right| \leq \frac{4a_3\sigma_3 + 9a_4\sigma_2^2}{48m^2} + \frac{5a_4}{96}\frac{\sigma_4 - 3\sigma_2^2}{m^3}$ .
Proof: Let  $h = \mathbb{E}[\delta]$ . If the function  $q \in \mathcal{C}^4$ , then the Taylor expansion at  $s + h$  with remainders leads to
$$
\begin{array}{l} \mathbb {E} [ q (s + \delta_ {1}) ] = q (s + h) + q ^ {\prime} (s + h) \mathbb {E} [ \delta_ {1} - h ] + \frac {q ^ {\prime \prime} (s + h)}{2} \mathbb {E} [ (\delta_ {1} - h) ^ {2} ] + \frac {q ^ {(3)} (s + h)}{6} \mathbb {E} [ (\delta_ {1} - h) ^ {3} ] \\ + \frac {1}{2 4} \mathbb {E} \left[ q ^ {(4)} \left(\phi_ {1}\right) \left(\delta_ {1} - h\right) ^ {4} \right] \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} [ q (s + \delta_ {2}) ] = q (s + h) + q ^ {\prime} (s + h) \mathbb {E} [ \delta_ {2} - h ] + \frac {q ^ {\prime \prime} (s + h)}{2} \mathbb {E} [ (\delta_ {2} - h) ^ {2} ] + \frac {q ^ {(3)} (s + h)}{6} \mathbb {E} [ (\delta_ {2} - h) ^ {3} ] \\ + \frac {1}{2 4} \mathbb {E} \left[ q ^ {(4)} \left(\phi_ {2}\right) \left(\delta_ {2} - h\right) ^ {4} \right] \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} \left[ q \left(s + \frac {\delta_ {1} + \delta_ {2}}{2}\right) \right] = q (s + h) + q ^ {\prime} (s + h) \mathbb {E} \left[ \frac {\delta_ {1} + \delta_ {2}}{2} - h \right] + \frac {q ^ {\prime \prime} (s + h)}{2} \mathbb {E} \left[ \left(\frac {\delta_ {1} + \delta_ {2}}{2} - h\right) ^ {2} \right] \\ + \frac {q ^ {(3)} (s + h)}{6} \mathbb {E} \big [ \big (\frac {\delta_ {1} + \delta_ {2}}{2} - h \big) ^ {3} \big ] + \frac {1}{2 4} \mathbb {E} \big [ q ^ {(4)} (\phi_ {3}) \big (\frac {\delta_ {1} + \delta_ {2}}{2} - h \big) ^ {4} \big ] \\ \end{array}
$$
where  $\phi_1, \phi_2, \phi_3$  between  $s + h$  and  $s + \delta_1$ ,  $s + \delta_2$ ,  $s + \delta_3$  respectively.
As  $\mathbb{E}[\delta - h] = 0$ , the error of extrapolation becomes
$$
\begin{array}{l} | \mathbb {E} [ \mathcal {L} _ {\mathcal {D} _ {m}} ^ {2} q (s) ] - q (s + h) | \\ \leq \left| 2 \mathbb {E} \left[ \frac {q ^ {(3)} (s + h)}{6} \left(\frac {\delta_ {1} + \delta_ {2}}{2} - h\right) ^ {3} \right] - \frac {1}{2} \left(\mathbb {E} \left[ \frac {q ^ {(3)} (s + h)}{6} (\delta_ {1} - h) ^ {3} \right] + \mathbb {E} \left[ \frac {q ^ {(3)} (s + h)}{6} (\delta_ {2} - h) ^ {3} \right]\right) \right| \\ + \left| 2 \mathbb {E} \left[ \frac {q ^ {(4)} (\phi_ {3})}{2 4} \left(\frac {\delta_ {1} + \delta_ {2}}{2} - h\right) ^ {4} \right] - \frac {1}{2} \left(\mathbb {E} \left[ \frac {q ^ {(4)} (\phi_ {1})}{2 4} (\delta_ {1} - h) ^ {4} \right] + \mathbb {E} \left[ \frac {q ^ {(4)} (\phi_ {2})}{2 4} (\delta_ {2} - h) ^ {4} \right]\right) \right| \\ \leq \frac {a _ {3}}{6} \left| 2 \mathbb {E} \left[ \left(\frac {\delta_ {1} + \delta_ {2}}{2} - h\right) ^ {3} \right] - \frac {1}{2} \left(\mathbb {E} [ (\delta_ {1} - h) ^ {3} ] + \mathbb {E} [ (\delta_ {2} - h) ^ {3} ]\right) \right| \\ + \left| 2 \mathbb {E} \left[ \frac {q ^ {(4)} (\phi_ {3})}{2 4} \left(\frac {\delta_ {1} + \delta_ {2}}{2} - h\right) ^ {4} \right] - \frac {1}{2} \left(\mathbb {E} \left[ \frac {q ^ {(4)} (\phi_ {1})}{2 4} (\delta_ {1} - h) ^ {4} \right] + \mathbb {E} \left[ \frac {q ^ {(4)} (\phi_ {2})}{2 4} (\delta_ {2} - h) ^ {4} \right]\right) \right| \\ \leq \frac {a _ {3}}{6} \left| 2 \mathbb {E} \left[ \left(\frac {\delta_ {1} + \delta_ {2}}{2} - h\right) ^ {3} \right] - \frac {1}{2} \left(\mathbb {E} [ (\delta_ {1} - h) ^ {3} ] + \mathbb {E} [ (\delta_ {2} - h) ^ {3} ]\right) \right| \\ + \left| 2 \mathbb {E} \left[ \frac {\left| q ^ {(4)} (\phi_ {3}) \right|}{2 4} \left(\frac {\delta_ {1} + \delta_ {2}}{2} - h\right) ^ {4} \right] + \frac {1}{2} \left(\mathbb {E} \left[ \frac {\left| q ^ {(4)} (\phi_ {1}) \right|}{2 4} \left(\delta_ {1} - h\right) ^ {4} \right] + \mathbb {E} \left[ \frac {\left| q ^ {(4)} (\phi_ {2}) \right|}{2 4} \left(\delta_ {2} - h\right) ^ {4} \right]\right) \right| \\ \leq \frac {a _ {3}}{6} \left| 2 \mathbb {E} \left[ \left(\frac {\delta_ {1} + \delta_ {2}}{2} - h\right) ^ {3} \right] - \frac {1}{2} \left(\mathbb {E} [ (\delta_ {1} - h) ^ {3} ] + \mathbb {E} [ (\delta_ {2} - h) ^ {3} ]\right) \right| \\ + \frac {a _ {4}}{2 4} \left| 2 \mathbb {E} \left[ \left(\frac {\delta_ {1} + \delta_ {2}}{2} - h\right) ^ {4} \right] + \frac {1}{2} \left(\mathbb {E} [ (\delta_ {1} - h) ^ {4} ] + \mathbb {E} [ (\delta_ {2} - h) ^ {4} ]\right) \right|. \\ \end{array}
$$
where the second inequality uses the upper bound on  $q^{(3)}(\cdot)$  (Assumption 2) and the third inequality uses  $(\delta - h)^4$  is non-negative and the last inequality uses the uniform bound on  $q^{(4)}(\cdot)$  (Assumption 2).
$$
\begin{array}{l} | \mathbb {E} [ \mathcal {L} _ {\mathcal {D} _ {m}} ^ {2} q (s) ] - q (s + h) | \\ \leq \frac {a _ {3}}{1 2} | \mathbb {E} (\delta_ {1} - h) ^ {3} | + \frac {a _ {4}}{2 4} \left(2 \mathbb {E} \left(\frac {\delta_ {1} + \delta_ {2}}{2} - h\right) ^ {4} + \mathbb {E} (\delta_ {1} - h) ^ {4}\right) \\ \leq \frac {a _ {3} \sigma_ {3}}{1 2 m ^ {2}} + \frac {a _ {4}}{2 4} \left(\frac {\sigma_ {4}}{4 m ^ {3}} + \frac {3 (2 m - 1) \sigma_ {2} ^ {2}}{4 m ^ {3}} + \frac {\sigma_ {4}}{m ^ {3}} + \frac {3 (m - 1) \sigma_ {2} ^ {2}}{m ^ {3}}\right) \\ \leq \frac {a _ {3} \sigma_ {3}}{1 2 m ^ {2}} + \frac {a _ {4}}{2 4} \left(\frac {9 \sigma_ {2} ^ {2}}{2 m ^ {2}} + \frac {5 (\sigma_ {4} - 3 \sigma_ {2} ^ {2})}{4 m ^ {3}}\right) \\ \leq \frac {4 a _ {3} \sigma_ {3} + 9 a _ {4} \sigma_ {2} ^ {2}}{4 8 m ^ {2}} + \frac {5 a _ {4}}{9 6} \frac {\sigma_ {4} - 3 \sigma_ {2} ^ {2}}{m ^ {3}}. \\ \end{array}
$$
we first use that  $\mathbb{E}[(\delta_1 - h)^3] = \mathbb{E}[(\delta_2 - h)^3] = 4\mathbb{E}[(\frac{\delta_1 + \delta_2}{2} -h)^3 ]$  and the uses the bound on moments in Lemma 1. Note that  $\mathbb{E}\left(\frac{\delta_1 + \delta_2}{2} -h\right)^4$  can be seen as the 4th order moments of a batch size of  $2m$
Proposition 3 Assume  $q \in \mathcal{C}^6$ . Then  $\mathcal{L}_{\mathcal{D}_m}^{(3)}$  as defined below is a third-order extrapolation operator.
Proof: Let  $h = \mathbb{E}[\delta]$ . If  $q \in \mathcal{C}^{2k}$ , then  $q$  has the following Taylor expansion
$$
\begin{array}{l} \mathbb {E} [ q (s + \delta) ] = \underbrace {q (s + h)} _ {\text {z e r o o r d e r t e r m}} + q ^ {\prime} (s + h) \mathbb {E} [ \delta - h ] + \underbrace {\frac {q ^ {\prime \prime} (s + h)}{2} \mathbb {E} [ (\delta - h) ^ {2} ]} _ {\text {s e c o n d o r d e r t e r m}} + \dots . \\ + \frac {q ^ {(2 k - 1)} (s + h)}{(2 k - 1) !} \mathbb {E} [ (\delta - h) ^ {2 k - 1} ] + \frac {1}{2 k !} \mathbb {E} [ q ^ {(2 k)} (\phi) (\delta - h) ^ {2 k} ]. \\ \end{array}
$$
Eliminate the third order term in the Taylor expansion. Consider the following affine combination which
$$
\mathcal {F} _ {\mathcal {D} _ {m}} ^ {(3)} q: s \mapsto \alpha_ {1} \mathcal {L} _ {\mathcal {D} _ {m}} ^ {(2)} q (s) + \alpha_ {2} \mathcal {L} _ {\mathcal {D} _ {2 m}} ^ {(2)} q (s).
$$
We determine  $\alpha_{1}$  and  $\alpha_{2}$  by expanding  $\mathcal{L}_{\mathcal{D}_m}^{(2)}q(s)$  and  $\mathcal{L}_{\mathcal{D}_{2m}}^{(2)}q(s)$  and analyze the coefficients of terms:
- (Affine). Taylor expansion of  $\mathcal{F}_{\mathcal{D}_m}^{(3)}q(s)$  at  $s + h$  should have zero order term  $q(s + h)$ , i.e.
$$
\alpha_ {1} q (s + h) + \alpha_ {2} q (s + h) = q (s + h).
$$
- (Eliminate third term). Taylor expansion of  $\mathcal{F}_{\mathcal{D}_m}^{(3)}q(s)$  at  $s + h$  should have third order term  $\mathbb{E}[(\delta -h)^3 ]$ . That is,
$$
\alpha_ {1} \mathbb {E} [ (\delta_ {1} - h) ^ {3} ] + \alpha_ {2} \mathbb {E} \left[ \left(\frac {\delta_ {1} + \delta_ {2}}{2} - h\right) ^ {3} \right] = 0.
$$
This is equivalent to
$$
\alpha_ {1} \mathbb {E} [ (\delta_ {1} - h) ^ {3} ] + \frac {\alpha_ {2}}{4} \mathbb {E} \left[ (\delta_ {1} - h) ^ {3} \right] = 0.
$$
Therefore,  $\alpha_{1}$  and  $\alpha_{2}$  can be determined through the following linear system
$$
\alpha_ {1} + \alpha_ {2} = 1
$$
$$
\alpha_ {1} + \frac {1}{4} \alpha_ {2} = 0.
$$
The solution is  $\alpha_{1} = -\frac{1}{3}$  and  $\alpha_{2} = \frac{4}{3}$ .
For  $k = 3$  order extrapolation, consider the following
$$
\mathcal {L} _ {\mathcal {D} _ {m}} ^ {(3)} q: s \mapsto \alpha_ {1} ^ {\prime} \mathcal {F} _ {\mathcal {D} _ {m}} ^ {(3)} q (s) + \alpha_ {2} ^ {\prime} \mathcal {F} _ {\mathcal {D} _ {2 m}} ^ {(3)} q (s) + \alpha_ {3} ^ {\prime} \mathcal {F} _ {\mathcal {D} _ {3 m}} ^ {(3)} q (s).
$$