SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lie He*
lie.he@epfl.ch
Shiva Prasad Kasiviswanathan
kasivisw@gmail.com
In this paper, we study the conditional stochastic optimization (CSO) problem which covers a variety of applications including portfolio selection, reinforcement learning, robust learning, causal inference, etc. The sample-averaged gradient of the CSO objective is biased due to its nested structure, and therefore requires a high sample complexity for convergence. We introduce a general stochastic extrapolation technique that effectively reduces the bias. We show that for nonconvex smooth objectives, combining this extrapolation with variance reduction techniques can achieve a significantly better sample complexity than the existing bounds. Additionally, we develop new algorithms for the finite-sum variant of the CSO problem that also significantly improve upon existing results. Finally, we believe that our debiasing technique has the potential to be a useful tool for addressing similar challenges in other stochastic optimization problems.
In this paper, we investigate the conditional stochastic optimization (CSO) problem as presented by Hu et al. [16], which is formulated as follows:
$$
\min  _ {\boldsymbol {x} \in \mathbb {R} ^ {d}} F (\boldsymbol {x}) = \mathbb {E} _ {\xi} \left[ f _ {\xi} \left(\mathbb {E} _ {\eta | \xi} \left[ g _ {\eta} (\boldsymbol {x}; \xi) \right]\right) \right], \tag {CSO}
$$
where  $\xi$  and  $\eta$  represent two random variables, with  $\eta$  conditioned on  $\xi$ . The  $f_{\xi}:\mathbb{R}^{p}\to \mathbb{R}$  and  $g_{\eta}:\mathbb{R}^d\to \mathbb{R}^p$  denote a stochastic function and a mapping respectively. The inner expectation is calculated with respect to the conditional distribution of  $\eta |\xi$ . In line with the established CSO framework [16, 15], throughout this paper, we assume access to samples from the distribution  $\mathbb{P}(\xi)$  and the conditional distribution  $\mathbb{P}(\eta |\xi)$ .
Many machine learning tasks can be formulated as a CSO problem, such as policy evaluation and control in reinforcement learning [6, 24], and linearly-solvable Markov decision process [5]. Other examples of the CSO problem include instrumental variable regression [23] and invariant learning [16]. Moreover, the widely-used Model-Agnostic Meta-Learning (MAML) framework, which seeks to determine a meta-initialization parameter using metadata for related learning tasks that are trained through gradient-based algorithms, is another example of a CSO problem. In this context, tasks  $\xi$  are drawn randomly, followed by the drawing of samples  $\eta | \xi$  from the specified task [11]. It is noteworthy that the standard stochastic optimization problem  $\min_{\boldsymbol{x}} \mathbb{E}_{\xi} [f_{\xi}(\boldsymbol{x})]$  represents a degenerate case of the CSO problem, achieved by setting  $g_{\eta}$  as an identity function.
In numerous prevalent CSO problems, such as first-order MAML (FO-MAML) [11], the outer random variable  $\xi$  only takes value in a finite set (say in  $\{1,\dots ,n\}$ ). These problems can be reformulated to have a finite-sum structure in the outer loop and referred to as Finite-sum Coupled Compositional Optimization (FCCO) problem in [33, 19]. In this paper, we also study this problem, formulated as:
$$
\min  _ {\boldsymbol {x} \in \mathbb {R} ^ {d}} F _ {n} (\boldsymbol {x}) = \frac {1}{n} \sum_ {i = 1} ^ {n} f _ {i} \left(\mathbb {E} _ {\eta | i} \left[ g _ {\eta} (\boldsymbol {x}; i) \right]\right). \tag {FCCO}
$$
The FCCO problem also has broad applications in machine learning for optimizing average precision, listwise ranking losses, neighborhood component analysis, deep survival analysis, deep latent variable models [33, 19].
*Work initiated during internship at Amazon.
Although the CSO and FCCO problems are widespread, they present challenges for optimization algorithms. Based on the special composition structure of CSO, using chain rule, under mild conditions, the full gradient of CSO is given by
$$
\nabla F (\boldsymbol {x}) = \mathbb {E} _ {\xi} \left[ \left(\mathbb {E} _ {\eta | \xi} [ \nabla g _ {\eta} (\boldsymbol {x}; \xi) ]\right) ^ {\top} \nabla f _ {\xi} \left(\mathbb {E} _ {\eta | \xi} [ g _ {\eta} (\boldsymbol {x}; \xi) ]\right) \right].
$$
Constructing an unbiased stochastic estimator for the gradient is generally computationally expensive (and even impossible). A straightforward estimation of  $\nabla F(\pmb{x})$  is to estimate  $\mathbb{E}_{\xi}$  with 1 sample of  $\xi$ , estimate  $\mathbb{E}_{\eta|\xi}[g_{\eta}(\cdot)]$  with a set  $H_{\xi}$  of  $m$  independent and identically distributed (i.i.d.) samples drawn from the conditional distribution  $\mathbb{P}(\eta|\xi)$ , and  $\mathbb{E}_{\eta|\xi}[\nabla g_{\eta}(\cdot)]$  with a different set  $\tilde{H}_{\xi}$  of  $m$  i.i.d. samples drawn from the same conditional distribution, i.e.,
$$
\nabla \hat {F} _ {m} (\boldsymbol {x}) := \left(\frac {1}{m} \sum_ {\tilde {\eta} \in \tilde {H} _ {\xi}} \nabla g _ {\tilde {\eta}} (\boldsymbol {x}; \xi)\right) ^ {\top} \nabla f _ {\xi} \left(\frac {1}{m} \sum_ {\eta \in H _ {\xi}} g _ {\eta} (\boldsymbol {x}; \xi)\right). \tag {1}
$$
Note that  $\nabla \hat{F}_m(\pmb{x})$  consists of two terms. The first term,  $(1 / m)\sum_{\tilde{\eta}\in \tilde{H}_{\xi}}\nabla g_{\tilde{\eta}}(\mathbf{x};\xi)$ , is an unbiased estimate of  $\mathbb{E}_{\eta |\xi}[\nabla g_{\eta}(\pmb{x};\xi)]$ . However, the second term is generally biased, i.e.,
$$
\mathbb {E} _ {\eta | \xi} \left[ \nabla f _ {\xi} \left(\frac {1}{m} \sum_ {\eta \in H _ {\xi}} g _ {\eta} (\boldsymbol {x}; \xi)\right) \right] \neq \nabla f _ {\xi} \left(\mathbb {E} _ {\eta | \xi} \left[ g _ {\eta} (\boldsymbol {x}; \xi) \right]\right).
$$
Consequently,  $\nabla \hat{F}_m(\pmb{x})$  is a biased estimator of  $\nabla F(\pmb{x})$ . To reach the  $\varepsilon$ -stationary point of  $F(\pmb{x})$  (Definition 1), the bias has to be sufficiently small.
Optimization with biased gradients converges only to a neighborhood of the stationary point. While the bias diminishes with increasing batch size, it also introduces additional sample complexity. For nonconvex objectives, Biased Stochastic Gradient Descent (BSGD) requires a total sample complexity of  $\mathcal{O}(\varepsilon^{-6})$  to reach an  $\varepsilon$ -stationary point [16]. This contrasts with standard stochastic optimization, where sample-averaged gradients are unbiased with a sample complexity of  $\mathcal{O}(\varepsilon^{-4})$  [12, 3]. This discrepancy has spurred a multitude of proposals aimed at reducing the sample complexities of both CSO and FCCO problems. Hu et al. [16] introduced Biased SpiderBoost (BSSpiderBoost), which, based on the variance reduction technique SpiderBoost from Wang et al. [38], reduces the variance of  $\xi$  to achieve a sample complexity of  $\mathcal{O}(\varepsilon^{-5})$  for the CSO problem. Hu et al. [17] proposed multi-level Monte Carlo (MLMC) gradient methods V-MLMC and RT-MLMC to further enhance the sample complexity to  $\mathcal{O}(\varepsilon^{-4})$ . The SOX [33] and MSVR-V2 [19] algorithms concentrated on the FCCO problem and improved the sample complexity to  $\mathcal{O}(n\varepsilon^{-4})$  and  $\mathcal{O}(n\varepsilon^{-3})$ , respectively.
Our Contributions. In this paper, we improve the sample complexities for both the CSO and FCCO problems (see Table 1). To facilitate a clear and concise presentation, we will suppress the dependence on specific problem parameters throughout the ensuing discussion.
(a) Our main technical tool in this paper is an extrapolation-based scheme that mitigates bias in gradient estimations. Considering a suitably differentiable function  $q(\cdot)$  and a random variable  $\delta \sim \mathcal{D}$ , we show that we can approximate the value of  $q(\mathbb{E}[\delta])$  via extrapolation from a limited number of evaluations of  $q(\delta)$ , while maintaining a minimal bias. In the context of CSO and FCCO problems, this scheme is used in gradient estimation, where the function  $q$  corresponds to  $\nabla f_{\xi}$  and the random variable  $\delta$  corresponds to  $g_{\eta}$ .
(b) For the CSO problem, we present novel algorithms that integrate the above extrapolation-based scheme with BSGD and BSpiderBoost algorithms of Hu et al. [16]. Our algorithms, referred to as E-BSGD and E-BSpiderBoost, achieve a sample complexity of  $\mathcal{O}(\varepsilon^{-4.5})$  and  $\mathcal{O}(\varepsilon^{-3.5})$  respectively, in order to attain an  $\varepsilon$ -stationary point for nonconvex smooth objectives. Notably, the sample complexity of E-BSpiderBoost improves the best-known sample complexity of  $\mathcal{O}(n\varepsilon^{-4})$  for the CSO problem from Hu et al. [17].
(c) For the FCCO problem $^2$  we propose a new algorithm that again combines the extrapolation-based scheme with a multi-level variance reduction applied to both inner and outer parts of the problem. Our algorithm, referred to as E-NestedVR, achieves a sample complexity of  $\mathcal{O}(n\varepsilon^{-3})$  if  $n \leq \varepsilon^{-2/3}$  and  $\mathcal{O}(\max\{\sqrt{n}\varepsilon^{-2.5}, \varepsilon^{-4}/\sqrt{n}\})$  if  $n > \varepsilon^{-2/3}$  for nonconvex smooth objectives and second-order extrapolation scheme. Our bound is never worse than the  $\mathcal{O}(n\varepsilon^{-3})$  bound of MSVR-V2 algorithm of Jiang et al. [19] and is in fact better if  $n = \Omega(\varepsilon^{-2/3})$ . As an illustration, when  $n = \Theta(\varepsilon^{-1.5})$ , our bound of  $\mathcal{O}(\varepsilon^{-3.25})$  is significantly better than the MSVR-V2 bound of  $\mathcal{O}(\varepsilon^{-4.5})$ .
2 For the FCCO problem we focus on  $n = \mathcal{O}(\varepsilon^{-2})$  case, for  $n = \Omega (\varepsilon^{-2})$  we can just treat the FCCO problem as a CSO problem and get an  $\mathcal{O}(\varepsilon^{-3.5})$  sample complexity bound via our E-BSpiderBoost algorithm.
Problem	Old Bounds	Our Bounds
Algorithm	Bound	Algorithm	Bound
CSO	BSGD [16]	O(ε-6)	E-BSGD	O(ε-4.5)
CSO	BSpiderBoost [16]	O(ε-5)	E-BSpiderBoost	O(ε-3.5)
CSO	RT-MLMC [17]	O(ε-4)
FCCO	MSVR-V2 [19]	O(nε-3)	E-NestedVR	{O(nε-3) if n ≤ ε-2/3
O(max{√n/ε2.5, 1/√nε4}), if n > ε-2/3
Table 1: Sample complexities needed to reach  $\varepsilon$ -stationary point for FCCO and CSO problems with nonconvex smooth objectives. Assumptions are comparable, but our results require an additional mild regularity on  $f_{\xi}$  and  $g_{\eta}$ . For FCCO also see Footnote 2. Note that  $\Omega(\varepsilon^{-3})$  is a sample complexity lower bound for standard stochastic nonconvex optimization [3], and hence, also for the problems considered in this paper.
In terms of proof techniques, our approach diverges from conventional analyses for the CSO and FCCO problems in that we focus on explicitly bounding the bias and variance terms of the gradient estimator to establish the convergence guarantee. Compared to previous results, our improvements do require an additional mild regularity assumption on  $f_{\xi}$  and  $g_{\eta}$  mainly that  $\nabla f_{\xi}$  is 4th order differentiable. Firstly, as we discuss in Remark 2 most common instantiations of CSO/FCCO framework such as: 1) invariant logistic regression [16], 2) instrumental variable regression [23], 3) first-order MAML for sine-wave few shot regression [11] and other problems, 4) deep average precision maximization [26, 34], tend to satisfy this assumption. Secondly, we highlight that the bounds derived from previous studies do not improve when incorporating this additional regularity assumption. Thirdly,  $\Omega(\varepsilon^{-3})$  remains the lower bound for stochastic optimization even under the arbitrary smoothness constraint [2], demonstrating that our improvement is non-trivial. Our results show that, this regularity assumption, which seems to practically valid, can be exploited through a novel extrapolation-based bias reduction technique to provide substantial improvements in sample complexity. $^{3}$
We defer some additional related work to Appendix B and conclude with some preliminaries.
Notation. Vectors are denoted by boldface letters. For a vector  $\pmb{x}$ ,  $\| \pmb{x} \|_2$  denotes its  $\ell_2$ -norm. A function with  $k$  continuous derivatives is called a  $\mathcal{C}^k$  function. We use  $a \lesssim b$  to denote that  $a \leq Cb$  for some constant  $C > 0$ . We consider expectation over various randomness:  $\mathbb{E}_{\xi}[\cdot]$  denotes expectation over the random variable  $\xi$ ,  $\mathbb{E}_{\eta|\xi}[\cdot]$  denotes expectation over the conditional distribution of  $\eta|\xi$ . Unless otherwise specified, for a random variable  $X$ ,  $\mathbb{E}[X]$  denotes expectation over the randomness in  $X$ . We focus on nonconvex objectives in this paper and use the following standard convergence criterion for nonconvex optimization [18].
Definition 1 (ε-stationary point) For a differentiable function  $F(\cdot)$ , we say that  $\pmb{x}$  is a first-order ε-stationary point if  $\| \nabla F(\pmb {x})\| ^2\leq \varepsilon^2$
For notational convenience, in the rest of this paper, we omit the dependence on  $\xi$  (or  $i$  in the FCCO context) in the function  $g$  and use  $g_{\eta}(\pmb{x})$  to represent  $g_{\eta}(\pmb{x};\xi)$ .
2 Stochastic Extrapolation as a Tool for Bias Correction
In this section, we present an approach for tackling the bias problem as appears in optimization procedures such as BSGD, BSpiderBoost, etc. Importantly, our approach addresses a general problem appearing in optimization settings and could be of independent interest. All missing details from this section are presented in Appendix C.
For ease of presentation, we start by considering the 1-dimensional case and assume a function  $q: \mathbb{R} \to \mathbb{R}$ , a constant  $s \in \mathbb{R}$ . Let  $\delta$  be a random variable drawn from an arbitrary distribution  $\mathcal{D}$  over  $\mathbb{R}$ . In Sections 3 and 4, we apply these ideas to the CSO and FCCO problems where the random variable  $\delta$  is played by  $g_{\eta}(\cdot)$  and function  $q$  is played by  $\nabla f_{\xi}$ . Informally stated, our goal in this section will be to
$$
\text {E f f e c t i n l y a p p r o x i m a t e} q (s + \mathbb {E} [ \delta ]) \text {w i t h f e w e v a l u a t i o n s o f} \{q (s + \delta) \} _ {\delta \sim \mathcal {D}}.
$$
Higher-order smoothness conditions have also been exploited in standard stochastic optimization for performance gains [4].
An interesting case is when  $s = 0$ , where we are approximating  $q(\mathbb{E}[\delta])$  with evaluations of  $\{q(\delta)\}_{\delta \sim \mathcal{D}}$ . Now, if  $q$  is an affine function, then  $q(s + \mathbb{E}[\delta]) = \mathbb{E}[q(s + \delta)]$ . However, the equality does not hold true for general  $q$ , and there exists a bias, i.e.,  $|q(s + \mathbb{E}[\delta]) - \mathbb{E}[q(s + \delta)]| > 0$ . In this section, we introduce a stochastic extrapolation-based method, where we use an affine combination of biased stochastic estimates, to achieve better approximation.
Suppose  $q \in \mathcal{C}^{2k}$  is a continuous differentiable up to  $2k$ -th derivative and let  $h = \mathbb{E}[\delta]$ . We expand  $q(s + \delta)$ , the most straightforward approximation of  $q(s + \mathbb{E}[\delta])$ , using Taylor series at  $s + h$ , and take expectation,
$$
\begin{array}{l} \mathbb {E} [ q (s + \delta) ] = q (s + h) + q ^ {\prime} (s + h) \mathbb {E} [ \delta - h ] + \frac {q ^ {\prime \prime} (s + h)}{2} \mathbb {E} [ (\delta - h) ^ {2} ] + \frac {q ^ {(3)} (s + h)}{6} \mathbb {E} [ (\delta - h) ^ {3} ] \\ + \dots + \frac {q ^ {(2 k - 1)} (s + h)}{(2 k - 1) !} \mathbb {E} \big [ (\delta - h) ^ {(2 k - 1)} \big ] + \frac {1}{(2 k) !} \mathbb {E} \big [ q ^ {(2 k)} (\phi_ {\delta}) (\delta - h) ^ {2 k} \big ], \\ \end{array}
$$
where  $\phi_{\delta}$  between  $s + \delta$  and  $s + h$ . While  $\mathbb{E}[q(s + \delta)]$  matches  $q(s + h)$  in the first 2 terms, the third term is no longer zero. The approximation error (bias) is
$$
| \mathbb {E} [ q (s + \delta) ] - q (s + h) | = | \frac {q ^ {\prime \prime} (s + h)}{2} \mathbb {E} [ (\delta - h) ^ {2} ] + \dots + \frac {1}{(2 k) !} \mathbb {E} [ q ^ {(2 k)} (\phi_ {\delta}) (\delta - h) ^ {2 k} ] |.
$$
In order to analyze the upper bound, we make the following assumption on  $\mathcal{D}$  and  $q$ .
Assumption 1 (Bounded moments) For all  $\delta \sim \mathcal{D}$  has bounded higher-order moments:  $\sigma_l \coloneqq |\mathbb{E}[(\delta - \mathbb{E}[\delta])^l]| < \infty$  for  $l = 2, 3, \ldots, 2k$ .
Assumption 2 (Bounded derivatives) The  $q \in \mathcal{C}^{2k}$  and has bounded derivatives, i.e.,  $a_{l} := \sup_{s \in \operatorname{dom}(q)} |q^{(l)}(s)| < \infty$  for  $l = 1, 2, \ldots, 2k$ .
In addition, we consider a sample averaged distribution  $\mathcal{D}_m$  derived from  $\mathcal{D}$  as follows.
Definition 2 Given a distribution  $\mathcal{D}$  satisfying Assumption 1 and  $m\in \mathbb{N}^+$ , we define the distribution  $\mathcal{D}_m$  that outputs  $\delta$  where  $\delta = \frac{1}{m}\sum_{i = 1}^{m}\delta_{i}$  with  $\delta_i\stackrel {i.i.d.}{\sim}\mathcal{D}$ .
The moments of such distribution  $\mathcal{D}_m$  decrease with batch size  $m$  as  $k\geq 2$ ,  $|\mathbb{E}[(\delta -\mathbb{E}[\delta])^k ]| = \mathcal{O}(m^{-\lceil k / 2\rceil})$  (see Lemma 1). Our desiderata would be to construct a scheme that uses some samples from the distribution  $\mathcal{D}_m$  to construct an approximation of  $q(s + \mathbb{E}[\delta])$  that satisfies the following requirement.
Definition 3 (kth-order Extrapolation Operator) Given a function  $q: \mathbb{R} \to \mathbb{R}$  satisfying Assumption 2 and distribution  $\mathcal{D}_m$  satisfying Assumption 1, we define a kth-order extrapolation operator  $\mathcal{T}_{\mathcal{D}_m}^{(k)}$  as an operator from  $\mathcal{C}^{2k} \to \mathcal{C}^{2k}$  that given  $N = N(k)$  i.i.d. samples  $\delta_1, \ldots, \delta_N$  from  $\mathcal{D}_m$  satisfies  $\forall s \in \mathbb{R}: |\mathbb{E}[\mathcal{T}_m^{(k)}q(s)] - q(s + \mathbb{E}[\delta])| = \mathcal{O}(m^{-k})$ .
We now propose a sequence of operators  $\mathcal{L}_{\mathcal{D}_m}^{(1)}, \mathcal{L}_{\mathcal{D}_m}^{(2)}, \mathcal{L}_{\mathcal{D}_m}^{(3)}, \ldots$  that satisfy the above definition. The  $\mathcal{L}_{\mathcal{D}_m}^{(k)}q(s)$  is designed to ensure its Taylor expansion at  $s + h$  has a form of  $q(s + h) + \mathcal{O}(\mathbb{E}[(\delta - h)^{2k}])$ . The remainder  $\mathcal{O}(\mathbb{E}[(\delta - h)^{2k}])$  is bounded by  $\mathcal{O}(m^{-k})$  due to Lemma 1.
A First-order Extrapolation Operator. We define the simplest operator
$$
\mathcal {L} _ {\mathcal {D} _ {m}} ^ {(1)} q: s \mapsto [ q (s + \delta) ] \qquad \text {w h e r e} \delta \stackrel {{i. i. d.}} {{\sim}} \mathcal {D} _ {m}.
$$
In Proposition 2 (Appendix C), we show that  $\mathcal{L}_{\mathcal{D}_m}^{(1)}$  is a first-order extrapolation operator.
A Second-order Extrapolation Operator. We define the following linear operator  $\mathcal{L}_{\mathcal{D}_m}^{(2)}$  which transforms  $q \in \mathcal{C}^4$  into  $\mathcal{L}_{\mathcal{D}_m}^{(2)} q$  which has lesser bias (but similar variance, as shown later).
Definition 4 ( $\mathcal{L}_{\mathcal{D}_m}^{(2)}$  Operator) Given  $\mathcal{D}_m$  and  $q$ , define the following operator,
$$
\underline {{\mathcal {L}}} _ {\mathcal {D} _ {m}} ^ {(2)} q: s \mapsto \left[ 2 \cdot q \left(s + \frac {\delta_ {1} + \delta_ {2}}{2}\right) - \frac {q \left(s + \delta_ {1}\right) + q \left(s + \delta_ {2}\right)}{2} \right] \quad \text {w h e r e} \delta_ {1}, \delta_ {2} \stackrel {{i. i. d.}} {{\sim}} \mathcal {D} _ {m}.
$$
Note that if the function  $q$  is only  $L_{q}$ -Lipschitz continuous, then  $|\mathbb{E}[q(s + \delta)] - q(s + \mathbb{E}[\delta])| \leq \sqrt{L_q^2\mathbb{E}[|\delta - \mathbb{E}[\delta]|]^2} \leq \frac{L_q\sqrt{\sigma_2}}{m^{1 / 2}}$ . Therefore, in this case,  $q(s + \delta)$  does not satisfy the first-order guarantee.
Figure 1: The Fig. 1a investigates the estimation errors of  $\mathcal{L}^{(\cdot)}q(s)$  with their number of observations. The Fig. 1b compares the biases of  $\mathbb{E}[\mathcal{L}^{(\cdot)}q(s)]$  with increasing inner batch size  $m$ .
Note that  $\frac{\delta_1 + \delta_2}{2}$  is same as sampling from  $\mathcal{D}_{2m}$ . The absolute difference in the Taylor expansion of  $\mathcal{L}_{\mathcal{D}_m}^{(2)}q$  at  $s + h$  differs from  $q(s + h)$  as,
$$
\left. \mathcal {O} \left(\left| \mathbb {E} \left[ 2 \left(\frac {\delta_ {1} + \delta_ {2}}{2} - h\right) ^ {3} - \frac {1}{2} \left((\delta_ {1} - h) ^ {3} + (\delta_ {2} - h) ^ {3}\right) \right] \right|\right) = \mathcal {O} \left(\left| \left(\mathbb {E} [ (\delta - h) ^ {3} ] \right| \right|\right) \text {f o r} \delta^ {i. i. d}. \quad (3) \right.
$$
The bias error of this scheme can be bounded through the following proposition.
Proposition 1 (Second-order Guarantee) Assume that distribution  $\mathcal{D}_m$  and  $q(\cdot)$  satisfies Assumption 1 and 2 respectively with  $k = 2$ . Then, for all  $s \in \mathbb{R}$ ,  $\left| \mathbb{E}\left[\mathcal{L}_{\mathcal{D}_m}^{(2)}q(s)\right] - q(s + \mathbb{E}[\delta])\right| \leq \frac{4a_3\sigma_3 + 9a_4\sigma_2^2}{48m^2} + \frac{5a_4}{96}\frac{\sigma_4 - 3\sigma_2^2}{m^3}$ .
Remark 1 While extrapolation is motivated by Taylor expansion which requires smoothness, higher order derivatives are not explicitly computed. Appendix F.3 empirically shows that applying extrapolation to non-smooth functions achieves similar bias correction. Relaxing the smoothness conditions is a direction for future work.
The above proposition shows that  $\mathcal{L}_{\mathcal{D}_m}^{(2)}$  is in fact a second-order extrapolation operator with  $k = 2$  under Definition 3. We will use this operator when we consider the CSO and FCCO problems later. Now, focusing on variance, we can relate the variance of  $\mathcal{L}_{\mathcal{D}_m}^{(2)}q(s)$  in terms of the variance of  $q(s + \delta)$ . In particular, a consequence of Lemma 2 is that
$$
\mathbb {E} \left[ \left(\mathcal {L} _ {\mathcal {D} _ {m}} ^ {(2)} q (s) - \mathbb {E} [ \mathcal {L} _ {\mathcal {D} _ {m}} ^ {(2)} q (s) ]\right) ^ {2} \right] = \mathcal {O} (\mathbb {E} [ (q (s + \delta) - \mathbb {E} [ q (s + \delta) ]) ^ {2} ]).
$$
Extension of  $\mathcal{L}_{\mathcal{D}_m}^{(2)}$  to Higher-dimensional Case. If  $q: \mathbb{R}^p \to \mathbb{R}^\ell$  is a vector-valued function, then there is a straightforward extension of Definition 4. Now, for distribution  $\mathcal{D}$  over  $\mathbb{R}^p$  and corresponding sampled averaged distribution  $\mathcal{D}_m$ , and  $s \in \mathbb{R}^p$
$$
\mathcal {L} _ {\mathcal {D} _ {m}} ^ {(2)} q: \boldsymbol {s} \mapsto \left[ 2 \cdot q \left(\boldsymbol {s} + \frac {\delta_ {1} + \delta_ {2}}{2}\right) - \frac {q \left(\boldsymbol {s} + \delta_ {1}\right) + q \left(\boldsymbol {s} + \delta_ {2}\right)}{2} \right] \quad \text {w h e r e} \boldsymbol {\delta} _ {1}, \boldsymbol {\delta} _ {2} \stackrel {{\mathrm {i . i . d .}}} {{\sim}} \mathcal {D} _ {m}. \tag {4}
$$
Higher-order Extrapolation Operators. The idea behind the construction of  $\mathcal{L}_{\mathcal{D}_m}^{(2)}$  can be generalized to higher  $k$ 's. For example, in Proposition 3, we construct a third-order extrapolation operator  $\mathcal{L}_{\mathcal{D}_m}^{(3)}$  through higher degree Taylor series approximation
$$
\mathcal {L} _ {\mathcal {D} _ {m}} ^ {(3)} q: s \mapsto \left(- \frac {1}{3 6} \mathcal {L} _ {\mathcal {D} _ {m}} ^ {(2)} + \frac {5}{9} \mathcal {L} _ {\mathcal {D} _ {2 m}} ^ {(2)} - \frac {3}{4} \mathcal {L} _ {\mathcal {D} _ {3 m}} ^ {(2)} - \frac {1 6}{9} \mathcal {L} _ {\mathcal {D} _ {4 m}} ^ {(2)} + 3 \mathcal {L} _ {\mathcal {D} _ {6 m}} ^ {(2)}\right) q (s).
$$
While this idea of expressing the  $k$ -th order operator as an affine combination of lower-order operators works for every  $k$ , explicit constructions soon become tedious.
In Fig. 1, we empirically demonstrate the effectiveness of extrapolation in stochastic estimation. In Fig. 1a, we choose  $q(s) = s^2 / 2$ ,  $\delta \sim \mathcal{N}(10, 100)$ . For both  $\mathcal{L}_{\mathcal{D}_{6m}}^{(2)}q(s)$  and  $\mathcal{L}_{\mathcal{D}_m}^{(3)}q(s)$ , their estimation errors converge to 0 with increasing number of estimates. This coincides with Proposition 1 as  $a_3 = 0$