Title: Debiasing Conditional Stochastic Optimization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Optimization

Bilevel Optimization

Stochastic Optimization
Abstract: In this paper, we study the conditional stochastic optimization (CSO) problem which  covers a variety of applications including  portfolio selection, reinforcement learning, robust learning, causal inference, etc. The sample-averaged gradient of the CSO objective is biased due to its nested structure, and therefore requires a high sample complexity for convergence. We introduce a general stochastic extrapolation technique that effectively reduces the bias. We show that for nonconvex smooth objectives, combining this extrapolation with variance reduction techniques can achieve a significantly better sample complexity than the existing bounds. Additionally, we  develop new algorithms for the  finite-sum variant of the CSO problem that also significantly improve upon existing results. Finally, we believe that our debiasing technique has the potential to be a useful tool for addressing similar challenges in other stochastic optimization problems.

Debiasing Conditional Stochastic Optimization

Abstract

In this paper, we study the conditional stochastic optimization (CSO) problem which covers a variety of applications including portfolio selection, reinforcement learning, robust learning, causal inference, etc. The sample-averaged gradient of the CSO objective is biased due to its nested structure, and therefore requires a high sample complexity for convergence. We introduce a general stochastic extrapolation technique that effectively reduces the bias. We show that for nonconvex smooth objectives, combining this extrapolation with variance reduction techniques can achieve a significantly better sample complexity than the existing bounds. Additionally, we develop new algorithms for the finite-sum variant of the CSO problem that also significantly improve upon existing results. Finally, we believe that our debiasing technique has the potential to be a useful tool for addressing similar challenges in other stochastic optimization problems.

1 Introduction

In this paper, we investigate the conditional stochastic optimization (CSO) problem as presented by Hu et al. [16], which is formulated as follows:

where ξ and η represent two random variables, with η conditioned on ξ . The f ξ : R p → R and g η : R d → R p denote a stochastic function and a mapping respectively. The inner expectation is calculated with respect to the conditional distribution of η | ξ . In line with the established CSO framework [16, 15], throughout this paper, we assume access to samples from the distribution P ( ξ ) and the conditional distribution P ( η | ξ ) .

Many machine learning tasks can be formulated as a CSO problem, such as policy evaluation and control in reinforcement learning [6, 24], and linearly-solvable Markov decision process [5]. Other examples of the CSO problem include instrumental variable regression [23] and invariant learning [16]. Moreover, the widely-used Model-Agnostic Meta-Learning (MAML) framework, which seeks to determine a meta-initialization parameter using metadata for related learning tasks that are trained through gradient-based algorithms, is another example of a CSO problem. In this context, tasks ξ are drawn randomly, followed by the drawing of samples η | ξ from the specified task [11]. It is noteworthy that the standard stochastic optimization problem min x E ξ [ f ξ ( x )] represents a degenerate case of the CSO problem, achieved by setting g η as an identity function.

In numerous prevalent CSO problems, such as first-order MAML (FO-MAML) [11], the outer random variable ξ only takes value in a finite set (say in { 1 , . . . , n } ). These problems can be reformulated to have a finite-sum structure in the outer loop and referred to as Finite-sum Coupled Compositional Optimization ( FCCO ) problem in [33, 19]. In this paper, we also study this problem, formulated as:

The FCCO problem also has broad applications in machine learning for optimizing average precision, listwise ranking losses, neighborhood component analysis, deep survival analysis, deep latent variable models [33, 19].

Although the CSO and FCCO problems are widespread, they present challenges for optimization algorithms. Based on the special composition structure of CSO, using chain rule, under mild conditions, the full gradient of CSO is given by

Constructing an unbiased stochastic estimator for the gradient is generally computationally expensive (and even impossible). A straightforward estimation of ∇ F ( x ) is to estimate E ξ with 1 sample of ξ , estimate E η | ξ [ g η ( · )] with a set H ξ of m independent and identically distributed (i.i.d.) samples drawn from the conditional distribution P ( η | ξ ) , and E η | ξ [ ∇ g η ( · )] with a different set ˜ H ξ of m i.i.d. samples drawn from the same conditional distribution, i.e.,

Note that ∇ ˆ F m ( x ) consists of two terms. The first term, (1 /m ) ∑ ˜ η ∈ ˜ H ξ ∇ g ˜ η ( x ; ξ ) , is an unbiased estimate of E η | ξ [ ∇ g η ( x ; ξ )] . However, the second term is generally biased, i.e.,

̸

Consequently, ∇ ˆ F m ( x ) is a biased estimator of ∇ F ( x ) . To reach the ε -stationary point of F ( x ) (Definition 1), the bias has to be sufficiently small.

Optimization with biased gradients converges only to a neighborhood of the stationary point. While the bias diminishes with increasing batch size, it also introduces additional sample complexity. For nonconvex objectives, Biased Stochastic Gradient Descent (BSGD) requires a total sample complexity of O ( ε -6 ) to reach an ε -stationary point [16]. This contrasts with standard stochastic optimization, where sample-averaged gradients are unbiased with a sample complexity of O ( ε -4 ) [12, 3]. This discrepancy has spurred a multitude of proposals aimed at reducing the sample complexities of both CSO and FCCO problems. Hu et al. [16] introduced Biased SpiderBoost (BSpiderBoost), which, based on the variance reduction technique SpiderBoost from Wang et al. [38], reduces the variance of ξ to achieve a sample complexity of O ( ε -5 ) for the CSO problem. Hu et al. [17] proposed multi-level Monte Carlo (MLMC) gradient methods V-MLMC and RT-MLMC to further enhance the sample complexity to O ( ε -4 ) . The SOX [33] and MSVR-V2 [19] algorithms concentrated on the FCCO problem and improved the sample complexity to O ( nε -4 ) and O ( nε -3 ) , respectively.

Our Contributions. In this paper, we improve the sample complexities for both the CSO and FCCO problems (see Table 1). To facilitate a clear and concise presentation, we will suppress the dependence on specific problem parameters throughout the ensuing discussion.

(a) Our main technical tool in this paper is an extrapolation-based scheme that mitigates. bias in gradient estimations. Considering a suitably differentiable function q ( · ) and a random variable δ ∼ D , we show that we can approximate the value of q ( E [ δ ]) via extrapolation from a limited number of evaluations of q ( δ ) , while maintaining a minimal bias. In the context of CSO and FCCO problems, this scheme is used in gradient estimation, where the function q corresponds to ∇ f ξ and the random variable δ corresponds to g η .

(b) For the CSO problem, we present novel algorithms that integrate the above extrapolation-based scheme with BSGD and BSpiderBoost algorithms of Hu et al. [16]. Our algorithms, referred to as E-BSGD and E-BSpiderBoost, achieve a sample complexity of O ( ε -4 . 5 ) and O ( ε -3 . 5 ) respectively, in order to attain an ε -stationary point for nonconvex smooth objectives. Notably, the sample complexity of E-BSpiderBoost improves the best-known sample complexity of O ( nε -4 ) for the CSO problem from Hu et al. [17].

(c) For the FCCO problem 2 we propose a new algorithm that again combines the extrapolationbased scheme with a multi-level variance reduction applied to both inner and outer parts of the problem. Our algorithm, referred to as E-NestedVR, achieves a sample complexity of O ( nε -3 ) if n ≤ ε -2 / 3 and O (max { √ nε -2 . 5 , ε -4 / √ n } ) if n > ε -2 / 3 for nonconvex smooth objectives and second-order extrapolation scheme. Our bound is never worse than the O ( nε -3 ) bound of MSVR-V2 algorithm of Jiang et al. [19] and is in fact better if n = Ω( ε -2 / 3 ) . As an illustration, when n = Θ( ε -1 . 5 ) , our bound of O ( ε -3 . 25 ) is significantly better than the MSVR-V2 bound of O ( ε -4 . 5 ) .

Table 1: Sample complexities needed to reach ε -stationary point for FCCO and CSO problems with nonconvex smooth objectives. Assumptions are comparable, but our results require an additional mild regularity on f ξ and g η . For FCCO also see Footnote 2. Note that Ω( ε -3 ) is a sample complexity lower bound for standard stochastic nonconvex optimization [3], and hence, also for the problems considered in this paper.

In terms of proof techniques, our approach diverges from conventional analyses for the CSO and FCCO problems in that we focus on explicitly bounding the bias and variance terms of the gradient estimator to establish the convergence guarantee. Compared to previous results, our improvements do require an additional mild regularity assumption on f ξ and g η mainly that ∇ f ξ is 4 th order differentiable. Firstly, as we discuss in Remark 2 most common instantiations of CSO/FCCO framework such as: 1) invariant logistic regression [16], 2) instrumental variable regression [23], 3) first-order MAML for sine-wave few shot regression [11] and other problems, 4) deep average precision maximization [26, 34], tend to satisfy this assumption. Secondly, we highlight that the bounds derived from previous studies do not improve when incorporating this additional regularity assumption. Thirdly, Ω( ε -3 ) remains the lower bound for stochastic optimization even under the arbitrary smoothness constraint [2], demonstrating that our improvement is non-trivial. Our results show that, this regularity assumption, which seems to practically valid, can be exploited through a novel extrapolation-based bias reduction technique to provide substantial improvements in sample complexity. 3

We defer some additional related work to Appendix B and conclude with some preliminaries.

Notation. Vectors are denoted by boldface letters. For a vector x , ∥ x ∥ 2 denotes its ℓ 2 -norm. A function with k continuous derivatives is called a C k function. We use a ≲ b to denote that a ≤ Cb for some constant C > 0 . We consider expectation over various randomness: E ξ [ · ] denotes expectation over the random variable ξ , E η | ξ [ · ] denotes expectation over the conditional distribution of η | ξ . Unless otherwise specified, for a random variable X , E [ X ] denotes expectation over the randomness in X . We focus on nonconvex objectives in this paper and use the following standard convergence criterion for nonconvex optimization [18].

Definition 1 ( ε -stationary point) For a differentiable function F ( · ) , we say that x is a first-order ε -stationary point if ∥∇ F ( x ) ∥ 2 ≤ ε 2 .

For notational convenience, in the rest of this paper, we omit the dependence on ξ (or i in the FCCO context) in the function g and use g η ( x ) to represent g η ( x ; ξ ) .

2 Stochastic Extrapolation as a Tool for Bias Correction

In this section, we present an approach for tackling the bias problem as appears in optimization procedures such as BSGD, BSpiderBoost, etc. Importantly, our approach addresses a general problem appearing in optimization settings and could be of independent interest. All missing details from this section are presented in Appendix C.

For ease of presentation, we start by considering the 1-dimensional case and assume a function q : R → R , a constant s ∈ R . Let δ be a random variable drawn from an arbitrary distribution D over R . In Sections 3 and 4, we apply these ideas to the CSO and FCCO problems where the random variable δ is played by g η ( · ) and function q is played by ∇ f ξ . Informally stated, our goal in this section will be to

An interesting case is when s = 0 , where we are approximating q ( E [ δ ]) with evaluations of { q ( δ ) } δ ∼D . Now, if q is an affine function, then q ( s + E [ δ ]) = E [ q ( s + δ )] . However, the equality does not hold true for general q , and there exists a bias, i.e., | q ( s + E [ δ ]) -E [ q ( s + δ )] | > 0 . In this section, we introduce a stochastic extrapolation -based method, where we use an affine combination of biased stochastic estimates, to achieve better approximation.

Suppose q ∈ C 2 k is a continuous differentiable up to 2 k -th derivative and let h = E [ δ ] . We expand q ( s + δ ) , the most straightforward approximation of q ( s + E [ δ ]) , using Taylor series at s + h , and take expectation,

where ϕ δ between s + δ and s + h . While E [ q ( s + δ )] matches q ( s + h ) in the first 2 terms, the third term is no longer zero. The approximation error (bias) is

In order to analyze the upper bound, we make the following assumption on D and q .

Assumption 1 (Bounded moments) For all δ ∼ D has bounded higher-order moments: σ l := | E [( δ -E [ δ ]) l ] | < ∞ for l = 2 , 3 , . . . 2 k.

Assumption 2 (Bounded derivatives) The q ∈ C 2 k and has bounded derivatives, i.e., a l := sup s ∈ dom ( q ) | q ( l ) ( s ) | < ∞ for l = 1 , 2 , . . . , 2 k .

In addition, we consider a sample averaged distribution D m derived from D as follows.

Definition 2 Given a distribution D satisfying Assumption 1 and m ∈ N + , we define the distribution D m that outputs δ where δ = 1 m ∑ m i =1 δ i with δ i i.i.d. ∼ D .

The moments of such distribution D m decrease with batch size m as k ≥ 2 , | E [( δ -E [ δ ]) k ] | = O ( m -⌈ k/ 2 ⌉ ) (see Lemma 1). Our desiderata would be to construct a scheme that uses some samples from the distribution D m to construct an approximation of q ( s + E [ δ ]) that satisfies the following requirement.

Definition 3 ( k th-order Extrapolation Operator) Given a function q : R → R satisfying Assumption 2 and distribution D m satisfying Assumption 1, we define a k th-order extrapolation operator T ( k ) D m as an operator from C 2 k → C 2 k that given N = N ( k ) i.i.d. samples δ 1 , . . . , δ N from D m satisfies ∀ s ∈ R : | E [ T ( k ) m q ( s )] -q ( s + E [ δ ]) | = O ( m -k ) .

We now propose a sequence of operators L (1) D m , L (2) D m , L (3) D m , . . . that satisfy the above definition. The L ( k ) D m q ( s ) is designed to ensure its Taylor expansion at s + h has a form of q ( s + h )+ O ( E [( δ -h ) 2 k ]) . The remainder O ( E [( δ -h ) 2 k ]) is bounded by O ( m -k ) due to Lemma 1.

A First-order Extrapolation Operator. We define the simplest operator

In Proposition 2 (Appendix C), we show that L (1) D m is a first-order extrapolation operator. 4

A Second-order Extrapolation Operator. We define the following linear operator L (2) D m which transforms q ∈ C 4 into L (2) D m q which has lesser bias (but similar variance, as shown later).

Definition 4 ( L (2) D m Operator) Given D m and q , define the following operator,

10

0

10

1

10

2

10

3

10

4

10

5

Number of estimates

10

0

10

1

10

2

10

3

|

q

(

s

+ [ ])

Avg

(

( )

q

(

s

))|

|

q

(

s

+ [ ])

[

(1)

12

m

q

(

s

)]|

(1)

12

m

q

(

s

)

(2)

6

m

q

(

s

)

(3)

m

q

(

s

)

(a) q ( s ) = s 2 / 2 , δ ∼ N (10 , 100) , m = 1 .

10

0

10

1

10

2

10

7

10

5

10

3

10

1

10

1

|

q

(

s

+ [ ])

[

( )

m

q

(

s

)|

Burn-in

[

(1)

12

m

q

(

s

)]

[

(2)

6

m

q

(

s

)]

[

(3)

m

q

(

s

)]

m

(b) q ( s ) = s 4 , p ( δ ) = δ/ 2 where δ ∈ [0 , 2] .

Figure 1: The Fig. 1a investigates the estimation errors of L ( · ) q ( s ) with their number of observations. The Fig. 1b compares the biases of E [ L ( · ) q ( s )] with increasing inner batch size m .

Note that δ 1 + δ 2 2 is same as sampling from D 2 m . The absolute difference in the Taylor expansion of L (2) D m q at s + h differs from q ( s + h ) as,

The bias error of this scheme can be bounded through the following proposition.

Proposition 1 (Second-order Guarantee) Assume that distribution D m and q ( · ) satisfies Assumption 1 and 2 respectively with k = 2 . Then, for all s ∈ R , ∣ ∣ ∣ E [ L (2) D m q ( s ) ] -q ( s + E [ δ ]) ∣ ∣ ∣ ≤ 4 a 3 σ 3 +9 a 4 σ 2 2 48 m 2 + 5 a 4 96 σ 4 -3 σ 2 2 m 3 .

Remark 1 While extrapolation is motivated by Taylor expansion which requires smoothness, higher order derivatives are not explicitly computed. Appendix F .3 empirically shows that applying extrapolation to non-smooth functions achieves similar bias correction. Relaxing the smoothness conditions is a direction for future work.

The above proposition shows that L (2) D m is in fact a second-order extrapolation operator with k = 2 under Definition 3. We will use this operator when we consider the CSO and FCCO problems later. Now, focusing on variance, we can relate the variance of L (2) D m q ( s ) in terms of the variance of q ( s + δ ) . In particular, a consequence of Lemma 2 is that

Extension of L (2) D m to Higher-dimensional Case. If q : R p → R ℓ is a vector-valued function, then there is a straightforward extension of Definition 4. Now, for distribution D over R p and corresponding sampled averaged distribution D m , and s ∈ R p

Higher-order Extrapolation Operators. The idea behind the construction of L (2) D m can be generalized to higher k 's. For example, in Proposition 3, we construct a third-order extrapolation operator L (3) D m through higher degree Taylor series approximation

While this idea of expressing the k -th order operator as an affine combination of lower-order operators works for every k , explicit constructions soon become tedious.

In Fig. 1, we empirically demonstrate the effectiveness of extrapolation in stochastic estimation. 5 In Fig. 1a, we choose q ( s ) = s 2 / 2 , δ ∼ N (10 , 100) . For both L (2) D 6 m q ( s ) and L (3) D m q ( s ) , their estimation errors converge to 0 with increasing number of estimates. This coincides with Proposition 1 as a 3 = 0

and a 4 = 0 for quadratic q . In contrast, biased first order method only converges to a neighborhood. In Fig. 1b, we consider q ( s ) = s 4 and p ( δ ) = δ/ 2 where δ ∈ [0 , 2] . All three methods are biased and their biases decrease with m , i.e. O ( m -k ) for k th order method. Depending on the constants (e.g. a i , σ i ), a higher-order extrapolation method may need decently large m (burn-in phase) to outperform lower-order methods.

3 Applying Stochastic Extrapolation in the CSO Problem

In this section, we apply the extrapolation-based scheme from the previous section to reduce the bias in the CSO problem. We focus on variants of BSGD and their accelerated version BSpiderBoost based on our second-order approximation operator (Definition 4). Let H ξ , ˜ H ξ , and H ′ ξ indicate different sets, each of which contains m i.i.d. random variables/samples drawn from the conditional distribution P ( η | ξ ) . Remember that, as mentioned earlier, we use g η ( x ) to represent g η ( x ; ξ ) .

Extrapolated BSGD. At time t , BSGD constructs a biased estimator of ∇ F ( x t ) using one sample ξ and 2 m i.i.d. samples from the conditional distribution as in (1)

To reduce this bias, we apply the second-order extrapolation operator from (4). At time t , we define D t +1 g ,ξ to be the distribution of the random variable 1 m ∑ η ∈ H ξ g η ( x t ) . Then we apply L (2) D t +1 g ,ξ by setting q to ∇ f ξ and s = 0 , i.e.

where 1 m ∑ η ∈ H ξ g η ( x t ) and 1 m ∑ η ′ ∈ H ′ ξ g η ′ ( x t ) are i.i.d. drawn from D t +1 g ,ξ . In Algorithm 2 (Appendix A), we present our extrapolated BSGD (E-BSGD) scheme, where we replace ∇ f ξ ( 1 m ∑ η ∈ H ξ g η ( x t )) in (5) by L (2) D t +1 g ,ξ ∇ f ξ (0) resulting in this following gradient estimate:

Extrapolated BSpiderBoost. BSpiderBoost, proposed by Hu et al. [16], uses the variance reduction methods for nonconvex smooth stochastic optimization developed by Fang et al. [10], Wang et al. [38]. BSpiderBoost builds upon BSGD and has two kinds of updates: a large batch and a small batch update. In each step, it decides which update to apply based on a random coin. With probability p out , it selects a large batch update with B 1 outer samples of ξ . With remaining probability 1 -p out , it selects a small batch update where the gradient estimator will be updated with gradient information in the current iteration generated with B 2 outer samples of ξ and the information from the last iteration. Formally, it constructs a gradient estimate as follows,

We propose our extrapolated BSpiderBoost scheme (formally defined in Algorithm 3, Appendix A) by replacing the BSGD gradient estimates in (8) with E-BSGD.

Sample Complexity Analyses of E-BSGD and E-BSpiderBoost. We adopt the standard assumptions used in the literature [27, 35, 33, 41]. All proofs are deferred to Appendix D.

Assumption 3 (Lower bound) F is lower bounded by F ⋆ .

Assumption 4 (Bounded variance) Assume that g η and ∇ g η have bounded variances, i.e., for all ξ in the support of P ( ξ ) and x ∈ R p , σ 2 g := E η | ξ [ ∥ ∥ g η ( x ; ξ ) -E η | ξ [ g η ( x ; ξ )] ∥ ∥ 2 2 ] < ∞ and ζ 2 g := E η | ξ [ ∥ ∥ ∇ g η ( x ; ξ ) -E η | ξ [ ∇ g η ( x ; ξ )] ∥ ∥ 2 2 ] < ∞ .

Assumption 5 (Lipschitz continuity/smoothness of f ξ and g η ) For all ξ in the support of P ( ξ ) , f ξ ( · ) is C f -Lipschitz continuous (i.e., ∥ f ξ ( x ) -f ξ ( x ′ ) ∥ 2 ≤ C f ∥ x -x ′ ∥ 2 ∀ x , x ′ ∈ R p ) and L f -Lipschitz smooth (i.e., ∥∇ f ξ ( x ) -∇ f ξ ( x ′ ) ∥ 2 ≤ L f ∥ x -x ′ ∥ 2 , ∀ x , x ′ ∈ R p ) for any ξ . Similarly, for all ξ in the support of P ( ξ ) and η in the support of P ( η | ξ ) , g η ( · ; ξ ) is C g -Lipschitz continuous and L g -Lipschitz smooth.

The smoothness of f ξ and g η naturally implies the smoothness of F . Zhang and Xiao [41, Lemma 4.2] show that Assumption 5 ensures F is: 1) C F -Lipschitz continuous with C F = C f C g ; and 2) L F -Lipschitz smooth with L F = L g C f + C 2 g L f . We denote ˜ L F = ζ g C f + σ g C g L f . Moreover, Assumption 5 also guarantees that f ξ and g η have bounded gradients. In addition, f ξ and g η are assumed to satisfy the following regularity condition in order to apply our extrapolation-based scheme from Section 2.

Assumption 6 (Regularity) For all ξ in the support of P ( ξ ) , ∇ f ξ is 4 th-order differentiable with bounded derivatives (i.e., a l := sup g ∈ R p ∥ ∥ ∇ ( l ) f ξ ( g ) ∥ ∥ 2 < ∞ for l = 1 , 2 , 3 , 4 , ∀ x ∈ R p ) and g η has bounded moments upto 4th-order (i.e., σ k = sup x ∈ R d sup ξ E η | ξ [ ∑ p i =1 [ g η ( x ) -E η | ξ [ g η ( x )] ] k i ] < ∞ , k = 1 , 2 , 3 , 4 ).

Remark 2 The core piece of Assumption 6 is the 4 th order differentiability of ∇ f ξ as other parts can be easily satisfied through appropriate boundedness assumptions. This condition though is satisfied by common instantiations of CSO/FCCO. We discuss some examples including invariant logistic regression, instrumental variable regression, first-order MAML for sine-wave few-shot regression task, deep average precision maximization in Section 5. Therefore, our improvements in sample complexity apply to all these problems.

Consider some time t > 0 . Let G t +1 be a stochastic estimate of ∇ F ( x t ) where x t is the current iterate. The next iterate x t +1 := x t -γG t . Let E [ · ] denote the conditional expectation, where we condition on all the randomness until time t . We consider the bias and variance terms coming from our gradient estimate. Formally, we define the following two quantities

Our idea of getting to an ε -stationary point (Definition 1) will be to ensure that E t +1 bias and E t +1 var are bounded. The main technical component of our analyses is in fact analyzing these bias and variance terms for the various gradient estimates considered. For this purpose, we first analyze the bias and variance terms for the (original) BSGD (Lemma 5) and BSpiderBoost (Lemma 7) algorithms, which are then used to get the corresponding bounds for our E-BSGD (Lemma 6) and E-BSpiderBoost (Lemma 8) algorithms. Through these bias and variance bounds, we establish the following main results of this section.

Theorem 3 [E-BSGD Convergence] Consider the (CSO) problem. Suppose Assumptions 3, 4, 5, 6 hold true and L F , C F , ˜ L F , C g , F ⋆ are constants and C e ( f ; g ) := 8 a 3 σ 3 +18 a 4 σ 2 2 +5 a 4 σ 4 96 defined in Corollary 1 are associated with second order extrapolation in the CSO problem. Let step size γ ≤ 1 / (2 L F ) . Then the output x s of E-BSGD (Algorithm 2) satisfies: E [ ∥∇ F ( x s ) ∥ 2 2 ] ≤ ε 2 , for nonconvex F , if the inner batch size m = Ω( C e C g ε -1 / 2 ) , and the number of iterations

The E-BSGD takes O ( ε -4 ) iterations to converge and compute O ( ε -0 . 5 ) gradients per iteration. Therefore, its resulting sample complexity is O ( ε -4 . 5 ) which is more efficient than O ( ε -6 ) of BSGD. Similar improvements can be observed for E-BSpiderBoost in Theorem 4.

Theorem 4 [E-BSpiderBoost Convergence] Consider the (CSO) problem under the same assumptions as Theorem 3. Let step size γ ≤ 1 / (13 L F ) . Then the output x s of E-BSpiderBoost (Algorithm 3)

satisfies: E [ ∥∇ F ( x s ) ∥ 2 2 ] ≤ ε 2 , for nonconvex F , if the inner batch size m = O ( C e C g ε -0 . 5 ) , the hyperparameters of the outer loop of E-BSpiderBoost B 1 = ( ˜ L 2 F /m + C 2 F ) ε -2 , B 2 = √ B 1 , p out = 1 /B 2 , and the number of iterations

The resulting sample complexity of E-BSpiderBoost is O ( ε -3 . 5 ) , which improves O ( ε -5 ) bound of BSpiderBoost [16] and O ( ε -4 ) bound of V-MLMC/RT-MLMC [17].

4 Applying Stochastic Extrapolation in the FCCO Problem

In this section, we apply the extrapolation-based scheme from Section 2 to the FCCO problem. We focus on case where n = O ( ε -2 ) . For larger n , we can treat the FCCO problem as a CSO problem and get an O ( ε -3 . 5 ) bound from Theorem 4. All missing details are presented in Appendix E.

Now, a straightforward algorithm for FCCO is to use the finite-sum variant of SpiderBoost (or SPIDER) [10, 38] in Algorithm 3. In this case, if we choose the outer batch sizes to be B 1 = n , B 2 = √ n and the inner batch size to be m = max { ε -2 /n, ε -1 / 2 } . The resulting sample complexity of E-BSpiderBoost now becomes, O (max { √ n/ε 2 . 5 , 1 / √ nε 4 } ) , which recovers O ( ε -3 . 5 ) bound as in Theorem 4 for n = Θ( ε -2 ) . However, when n is small, such as n = O (1) , the sample complexity degenerates to O ( ε -4 ) which is worse than the Ω( ε -3 ) lower bound of stochastic optimization [3]. We leave the details to Theorem 8. We still use Assumptions 3, 4, 5, 6 for the analysis of FCCO problem, replacing the role of ξ with i .

Algorithm 1 E-NestedVR

Extrapolated NestedVR. We now introduce a nested variance reduction algorithm E-NestedVR which reaches low sample complexity for all choices of n . Missing proofs from this section are presented in Appendix E. For the stochasticities in the FCCO problem, our idea is to use two nested SpiderBoost variance reduction components: one for the outer random variable i and the other for the inner random variable η | i . In each outer (resp. inner) SpiderBoost step, we choose large batch B 1 (resp. S 1 ) with probability p out (resp. p in); otherwise we choose small batch. Let H i denote a set of m i.i.d. samples drawn from the conditional distribution P ( η | i ) . Similarly, let ˜ H i denote another set of m i.i.d. samples drawn from the same conditional distribution. For each given i , we approximate E η | i [ g η ( x t )] with y t +1 i from distribution D t +1 y ,i where,

Similarly, we approximate E ˜ η | i [ ∇ g ˜ η ( x t )] with z t +1 i defined as follows

0

2e5

4e5

10

4

10

2

10

0

Sub-optimality

BSGD

E-BSGD

0

2e5

4e5

BSpiderBoost

E-BSpiderBoost

0

2e5

4e5

NestedVR

E-NestedVR

(a) CSO: Large n = 50000

0

1e6

2e6

10

2

10

0

Sub-optimality

E-BSGD

E-BSpiderBoost

E-NestedVR

V-MLMC

(b) FCCO: small n = 50

where ϕ t i is the last time i is visited before time t . If i is not selected at time t , then y t +1 i = y t i and z t +1 i = z t i . Note that we use independent samples for y t +1 i and z t +1 i .

Finally, we present E-NestedVR in Algorithm 1 where second-order extrapolation operator L (2) · is applied to each occurrence of ∇ f i . We now analyze its convergence guarantee. Our analysis works by first looking at the effect of multi-level variance reduction without the extrapolation (that we refer to as NestedVR, Theorem 10, Appendix E.2), and then showing how extrapolation could further help to drive down the sample complexity.

Theorem 5 [E-NestedVR Convergence] Consider the (FCCO) problem. Under the same assumptions as Theorem 3.

If n = O ( ε -2 / 3 ) , then we choose the hyperaparameters of E-NestedVR (Algorithm 1) as B 1 = B 2 = n, p out = 1 , S 1 = ˜ L 2 F ε -2 , S 2 = ˜ L F ε -1 , p in = ˜ L -1 F ε, γ = O ( 1 L F ) .

If n = Ω( ε -2 / 3 ) , then we choose the hyperaparameters of E-NestedVR as B 1 = n, B 2 = √ n, p out = 1 / √ n, S 1 = S 2 = max { C e C g ε -1 / 2 , ˜ L 2 F / ( nε 2 ) } , p in = 1 , γ = O ( 1 L F ) .

Then the output x s of E-NestedVR satisfies: E [ ∥∇ F ( x s ) ∥ 2 2 ] ≤ ε 2 , for nonconvex F with iterations T = Ω ( L F ( F ( x 0 ) -F ⋆ ) ε -2 ) .

From Theorem 5, E-NestedVR has a sample complexity of O ( nε -3 ) in the small n regime ( n = O ( ε -2 / 3 ) ) and O (max { √ n/ε 2 . 5 , 1 / √ nε 4 } ) in the large n regime ( n = Ω( ε -2 / 3 ) ). Therefore, in the large n regime, this improves the O ( nε -3 ) sample complexity of MSVR-V2 [19].

5 Applications

In this section, we demonstrate the numerical performance of our proposed algorithms. We focus on the application of invariant logistic regression here. In Appendix F, we discuss performance of our proposed algorithms on other common CSO/FCCO applications.

5.1 Application of Invariant Risk Minimization

Invariant learning has wide applications in machine learning and related areas [22, 1]. Invariant logistic regression [16] is formulated as follows:

where a and b represent a sample and its corresponding label, and η is a noisy observation of the sample a . This first part can be considered as a CSO objective, with f ξ ( y ) := log(1+exp( -by )) and g η ( x ; ξ ) := η ⊤ x . As the loss f ξ ∈ C ∞ is smooth, our results from Sections 3 and 4 are applicable.

An ℓ 2 -regularizer is added to ensure the existence of an unique minimizer. Since the gradient of the penalization term is unbiased, we only have to consider the bias of the data-dependent term. We generate a synthetic dataset with d = 10 dimensions. The minimizer is drawn from Gaussian distribution x ⋆ ∼ N (0 , 1) ∈ R d . We draw invariant samples { ( a i , b i ) } i where a i ∼ N (0 , 1) ∈ R d and compute b i = sgn ( a ⊤ i x ⋆ ) and its perturbed observation η ∼ N ( a i , 100) ∈ R d .

We consider drawing ξ from a large set ( n = 50000 ) and a small set ( n = 50 ) as CSO and FCCO problems respectively. As baselines, we implemented the BSGD and BSpiderBoost methods

0.0

0.5

1.0

1e6

10

2

10

0

10

2

Sub-optimality

BSGD

E-BSGD

0.0

0.5

1.0

BSpiderBoost

E-BSpiderBoost

0.0

0.5

1.0

1e6

NestedVR

E-NestedVR

(a) CSO: Large n = 10000

0.0

0.5

1.0

1e6

20

40

60

Sub-optimality

1e-1

1e-2

E-BSGD

E-BSpiderBoost

E-NestedVR

V-MLMC

(b) FCCO: small n = 50

from [16], V-MLMC approach from [17], and NestedVR approach from Appendix E.2 which achieves the same complexity as MSVR-V2 [19] for the FCCO problem. RT-MLMC shares the same sample complexity as V-MLMC, and is thus omitted from the experiment [17]. The results are shown in Fig. 2. In the CSO setting, we compare biased gradient methods with their extrapolated variants (BSGD vs. E-BSGD, BSpiderBoost vs. E-BSpiderBoost, and NestedVR vs. E-NestedVR). The extrapolated versions of BSGD, BSpiderBoost, and NestedVR consistently reach lower error than their non-extrapolated counterparts, as is evident in Figure 2a. In this case, the performance of BSpiderBoost is similar to BSGD as also noted by the authors of these techniques [16], and a drawback of BSpiderBoost seems to be that it is much harder to tune in practice. However, it is clear that E-BSGD outperforms BSGD, and E-BSpiderBoost outperforms BSpiderBoost, respectively. In the FCCO setting, we compare extrapolation based methods and MLMC based methods. Figure 2a, shows that E-NestedVR outperforms all other extrapolated algorithms, including the V-MLMC approach of [17], matching our theoretical findings.

5.2 Application of Instrumental Variable Regression

Instrumental variable regression is a popular technique in econometrics that aims to estimate the causal effect of input X on a confounded outcome Y with an instrument variable Z , which, for a given X , is conditionally independent of Y . As noted by [23], the instrumental variable regression is a special case of the CSO problem. The instrumental variable regression problem is phrased as:

where ξ = ( Y, Z ) , η = X . This can be viewed in the CSO framework with f ξ ( · ) = ℓ ( Y, · ) . We choose ℓ ( y, ˆ y ) = log cosh( y -ˆ y ) as regression loss function and g X ( w ) = w ⊤ X to be a linear regressor. In this case, f ξ ∈ C ∞ with ∇ f ξ (ˆ y ) = tanh(ˆ y -Y ) , and our results from Sections 3 and 4 apply. We generate the data similar to [31]

where z 1 is the first dimension of Z , e is the confounding variable and δ , γ are noises. In this experiment, we solve the instrumental variable regression using BSGD, BSpiderBoost, NestedVR and their extrapolated variants described in Sections 3 and 4. In each pair of experiments, the samples used per iteration are fixed same, i.e.: 1) BSGD uses m = 2 and E-BSGD uses m = 1 ; 2) For BSpiderBoost and E-BSpiderBoost, we use cycle length of 10, small batch and large batch in Spider to be 10 and 100 respectively, and we choose inner batch sizes m = 2 for BSpiderBoost and m = 1 for E-BSpiderBoost; 3) For NestedVR and E-NestedVR, we fix the outer batch size to 10 and 5 respectively, and choose fix the inner Spider Cycle to be 10 with large batch 100 and small batch 10. The results are presented in Figure 3. As is quite evident, the extrapolation variants achieve faster convergence in all 3 cases, confirming our theoretical findings.

6 Concluding Remarks

In this paper, we consider the conditional stochastic optimization CSO problem and its finite-sum variant FCCO. Due to the interplay between nested structure and stochasticity, most of the existing gradient estimates suffer from large biases and have large sample complexity of O ( ε -5 ) . We propose stochastic extrapolation-based algorithms that tackle this bias problem and improve the sample complexities for both these problems. While we focus on nonconvex objectives, our proposed algorithms can also be beneficial when used with strongly convex, convex objectives. We also believe that similar ideas could also prove helpful for multi-level stochastic optimization problems [41] with nested dependency.

References

Fabio Anselmi, Joel Z Leibo, Lorenzo Rosasco, Jim Mutch, Andrea Tacchetti, and Tomaso Poggio. Unsupervised learning of invariant representations. Theoretical Computer Science , 633:112-121, 2016.

Yossi Arjevani, Yair Carmon, John C. Duchi, Dylan J. Foster, Ayush Sekhari, and Karthik Sridharan. Second-order information in non-convex stochastic optimization: Power and limitations. In Jacob D. Abernethy and Shivani Agarwal, editors, Conference on Learning Theory, COLT 2020, 9-12 July 2020, Virtual Event [Graz, Austria] , volume 125 of Proceedings of Machine Learning Research , pages 242-299. PMLR, 2020. URL http: //proceedings.mlr.press/v125/arjevani20a.html .

Yossi Arjevani, Yair Carmon, John C Duchi, Dylan J Foster, Nathan Srebro, and Blake Woodworth. Lower bounds for non-convex stochastic optimization. Mathematical Programming , pages 1-50, 2022.

Sébastien Bubeck, Qijia Jiang, Yin Tat Lee, Yuanzhi Li, and Aaron Sidford. Near-optimal method for highly smooth convex optimization. In Alina Beygelzimer and Daniel Hsu, editors, Conference on Learning Theory, COLT 2019, 25-28 June 2019, Phoenix, AZ, USA , volume 99 of Proceedings of Machine Learning Research , pages 492-507. PMLR, 2019. URL http: //proceedings.mlr.press/v99/bubeck19a.html .

Bo Dai, Niao He, Yunpeng Pan, Byron Boots, and Le Song. Learning from conditional distributions via dual embeddings. In Aarti Singh and Xiaojin (Jerry) Zhu, editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS 2017, 20-22 April 2017, Fort Lauderdale, FL, USA , volume 54 of Proceedings of Machine Learning Research , pages 1458-1467. PMLR, 2017. URL http://proceedings.mlr.press/v54/ dai17a.html .

Bo Dai, Albert Shaw, Lihong Li, Lin Xiao, Niao He, Zhen Liu, Jianshu Chen, and Le Song. SBEED: convergent reinforcement learning with nonlinear function approximation. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018 , volume 80 of Proceedings of Machine Learning Research , pages 1133-1142. PMLR, 2018. URL http://proceedings.mlr.press/v80/dai18c.html .

Aaron Defazio, Francis R. Bach, and Simon Lacoste-Julien. SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger, editors, Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada , pages 1646-1654, 2014. URL https://proceedings.neurips.cc/paper/2014/hash/ ede7e2b6d13a41ddf9f4bdef84fdc737-Abstract.html .

Bradley Efron. Bootstrap methods: another look at the jackknife . Springer, 1992.

Yuri M Ermoliev and Vladimir I Norkin. Sample average approximation method for compound stochastic optimization problems. SIAM Journal on Optimization , 23(4):2231-2263, 2013.

Cong Fang, Chris Junchi Li, Zhouchen Lin, and Tong Zhang. SPIDER: near-optimal nonconvex optimization via stochastic path-integrated differential estimator. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada , pages 687-697, 2018. URL https://proceedings.neurips.cc/paper/2018/ hash/1543843a4723ed2ab08e18053ae6dc5b-Abstract.html .

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017 , volume 70 of Proceedings of Machine Learning Research , pages 1126-1135. PMLR, 2017. URL http://proceedings.mlr.press/v70/finn17a.html .

Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization , 23(4):2341-2368, 2013.

Takashi Goda and Wataru Kitade. Constructing unbiased gradient estimators with finite variance for conditional stochastic optimization. ArXiv preprint , abs/2206.01991, 2022. URL https: //arxiv.org/abs/2206.01991 .

Yanjun Han, Jiantao Jiao, and Tsachy Weissman. Minimax estimation of divergences between discrete distributions. IEEE Journal on Selected Areas in Information Theory , 1(3):814-823, 2020.

Yifan Hu, Xin Chen, and Niao He. Sample complexity of sample average approximation for conditional stochastic optimization. SIAM Journal on Optimization , 30(3):2103-2133, 2020.

Yifan Hu, Siqi Zhang, Xin Chen, and Niao He. Biased stochastic first-order methods for conditional stochastic optimization and applications in meta learning. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 612, 2020, virtual , 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ 1cdf14d1e3699d61d237cf76ce1c2dca-Abstract.html .

Yifan Hu, Xin Chen, and Niao He. On the bias-variance-cost tradeoff of stochastic optimization. Advances in Neural Information Processing Systems , 34:22119-22131, 2021.

Prateek Jain, Purushottam Kar, et al. Non-convex optimization for machine learning. Foundations and Trends® in Machine Learning , 10(3-4):142-363, 2017.

Wei Jiang, Gang Li, Yibo Wang, Lijun Zhang, and Tianbao Yang. Multi-block-single-probe variance reduced estimator for coupled compositional optimization. ArXiv preprint , abs/2207.08540, 2022. URL https://arxiv.org/abs/2207.08540 .

Jiantao Jiao and Yanjun Han. Bias correction with jackknife, bootstrap, and taylor series. IEEE Transactions on Information Theory , 66(7):4392-4418, 2020.

Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In Christopher J. C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger, editors, Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States , pages 315-323, 2013. URL https://proceedings.neurips.cc/paper/2013/hash/ ac1dd209cbcc5e5d1c6e28598e8cbbe8-Abstract.html .

Youssef Mroueh, Stephen Voinea, and Tomaso A. Poggio. Learning with group invariant features: A kernel perspective. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada , pages 1558-1566, 2015. URL https://proceedings.neurips. cc/paper/2015/hash/6602294be910b1e3c4571bd98c4d5484-Abstract.html .

Krikamol Muandet, Arash Mehrjou, Si Kai Lee, and Anant Raj. Dual instrumental variable regression. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 612, 2020, virtual , 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ 1c383cd30b7c298ab50293adfecb7b18-Abstract.html .

Ofir Nachum and Bo Dai. Reinforcement learning via fenchel-rockafellar duality. ArXiv preprint , abs/2001.01866, 2020. URL https://arxiv.org/abs/2001.01866 .

Lam M. Nguyen, Jie Liu, Katya Scheinberg, and Martin Takác. SARAH: A novel method for machine learning problems using stochastic recursive gradient. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017 , volume 70 of Proceedings of Machine Learning Research , pages 2613-2621. PMLR, 2017. URL http://proceedings.mlr.press/ v70/nguyen17b.html .

Qi Qi, Youzhi Luo, Zhao Xu, Shuiwang Ji, and Tianbao Yang. Stochastic optimization of areas under precision-recall curves with provable convergence. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 1752-1765, 2021. URL https://proceedings.neurips.cc/paper/2021/hash/ 0dd1bc593a91620daecf7723d2235624-Abstract.html .

Qi Qi, Youzhi Luo, Zhao Xu, Shuiwang Ji, and Tianbao Yang. Stochastic optimization of areas under precision-recall curves with provable convergence. Advances in Neural Information Processing Systems , 34:1752-1765, 2021.

Sashank J. Reddi, Ahmed Hefny, Suvrit Sra, Barnabás Póczos, and Alexander J. Smola. Stochastic variance reduction for nonconvex optimization. In Maria-Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016 , volume 48 of JMLR Workshop and Conference Proceedings , pages 314-323. JMLR.org, 2016. URL http://proceedings.mlr.press/v48/reddi16.html .

Sashank J Reddi, Suvrit Sra, Barnabás Póczos, and Alex Smola. Fast incremental method for nonconvex optimization. ArXiv preprint , abs/1603.06159, 2016. URL https://arxiv.org/ abs/1603.06159 .

Mark Schmidt, Nicolas Le Roux, and Francis Bach. Minimizing finite sums with the stochastic average gradient. Mathematical Programming , 162(1):83-112, 2017.

Rahul Singh, Maneesh Sahani, and Arthur Gretton. Kernel instrumental variable regression. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 4595-4607, 2019. URL https://proceedings.neurips.cc/paper/2019/hash/ 17b3c7061788dbe82de5abe9f6fe22b3-Abstract.html .

John Tukey. Bias and confidence in not quite large samples. Ann. Math. Statist. , 29:614, 1958.

Bokun Wang and Tianbao Yang. Finite-sum coupled compositional stochastic optimization: Theory and applications. In International Conference on Machine Learning , pages 2329223317. PMLR, 2022.

Guanghui Wang, Ming Yang, Lijun Zhang, and Tianbao Yang. Momentum accelerates the convergence of stochastic AUPRC maximization. In Gustau Camps-Valls, Francisco J. R. Ruiz, and Isabel Valera, editors, International Conference on Artificial Intelligence and Statistics, AISTATS 2022, 28-30 March 2022, Virtual Event , volume 151 of Proceedings of Machine Learning Research , pages 3753-3771. PMLR, 2022. URL https://proceedings.mlr. press/v151/wang22b.html .

Guanghui Wang, Ming Yang, Lijun Zhang, and Tianbao Yang. Momentum accelerates the convergence of stochastic auprc maximization. In International Conference on Artificial Intelligence and Statistics , pages 3753-3771. PMLR, 2022.

Mengdi Wang, Ji Liu, and Ethan X. Fang. Accelerating stochastic composition optimization. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain , pages 1714-1722, 2016. URL https://proceedings.neurips.cc/paper/2016/hash/ 92262bf907af914b95a0fc33c3f33bf6-Abstract.html .

Mengdi Wang, Ethan X Fang, and Han Liu. Stochastic compositional gradient descent: algorithms for minimizing compositions of expected-value functions. Mathematical Programming , 161:419-449, 2017.

Zhe Wang, Kaiyi Ji, Yi Zhou, Yingbin Liang, and Vahid Tarokh. Spiderboost and momentum: Faster variance reduction algorithms. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 2403-2413, 2019. URL https://proceedings.neurips.cc/paper/2019/hash/ 512c5cad6c37edb98ae91c8a76c3a291-Abstract.html .

Christopher Stroude Withers. Bias reduction by taylor series. Communications in StatisticsTheory and Methods , 16(8):2369-2383, 1987.

Yu M Yermol'yev. A general stochastic programming problem. 1971.

Junyu Zhang and Lin Xiao. Multilevel composite stochastic optimization via nested variance reduction. SIAM Journal on Optimization , 31(2):1131-1157, 2021.

x

Appendix

A Missing Pseudocodes

We present pseudocodes of E-BSGD and E-BSpiderBoost scheme in Algorithms 2 and 3 respectively.

Algorithm 2 E-BSGD

1:

Input:

x

0

∈

R

d

, step-size

2:

3:

4:

5:

6:

for

t

= 0

,

1

, . . . , T

-

1

Draw one sample

x

t

+1

t

end for

Output:

s

picked uniformly at random from

B Missing Details from Section 1

B.1 Other Related Work

CSO. Dai et al. [5] proposed a primal-dual stochastic approximation algorithm to solve a min-max reformulation of CSO, employing the kernel embedding techniques. However, this method requires convexity of f ξ and linearity of g η , which are not satisfied by general applications when neural networks are involved. Goda and Kitade [13] showed that a special class of CSO problems can be

ξ

do and compute extrapolated gradient

t

+1

E-BSGD

γG

x

-

γ

, batch sizes

m

{

x

t

}

T

-

1

t

=0

G

t

+1

E-BSGD

from (7)

Algorithm 3 E-BSpiderBoost 1: Input: x 0 ∈ R d , step-size γ , batch sizes B 1 , B 2 , Probability p out 2: for t = 0 , 1 , . . . , T -1 do 3: Draw χ out from Bernoulli ( p out ) 4: if ( t = 0 ) or ( χ out = 1 ) then ▷ Large batch 5: Draw B 1 outer samples { ξ 1 , . . . , ξ B 1 } 6: Compute extrapolated gradient G t +1 E-BSGD with (7) G t +1 E-BSB = 1 B 1 ∑ ξ ∈B 1 G t +1 E-BSGD 7: else ▷ Small batch 8: Draw B 2 outer samples { ξ 1 , . . . , ξ B 2 } 9: Compute extrapolated gradient G t +1 E-BSGD with (7) G t +1 E-BSB = G t E-BSB + 1 B 2 ∑ ξ ∈B 2 ( G t +1 E-BSGD -G t E-BSGD ) 10: end if 11: x t +1 = x t -γG t +1 E-BSB 12: end for 13: Output: x s picked uniformly at random from { x t } T -1 t =0

unbiased, e.g., when f ξ measures the squared error between some u ( ξ ) and E η | ξ [ g η ( x ; ξ ] , giving rise to this objective function E ξ [( u ( ξ ) -E η | ξ [ g η ( x ; ξ ]) 2 ] . However, they did not show any improvement over the sample complexity of BSGD (i.e., O ( ε -6 ) ). Hu et al. [16] also analyzed lower bounds on the minimax error for the CSO problem and showed that for a specific class of biased gradients with O ( ε ) bias (same bias as BSGD) and variance O (1) the bound achieved by BSpiderBoost is tight. However, these lower bounds are not applicable in settings such as ours (and also to [17]) where the bias is smaller than the BSGD bias.

Variance Reduction. The reduction of variance in stochastic optimization is a crucial approach to decrease sample complexity, particularly when dealing with finite-sum formulations of the form min x 1 n ∑ n i =1 f i ( x ) . Pioneering works such as Stochastic Average Gradient (SAG) [30], Stochastic Variance Reduced Gradient (SVRG) [21, 28], and SAGA [7, 29] improved the iteration complexity from O ( ε -4 ) in Stochastic Gradient Descent (SGD) to O ( ε -2 ) . Subsequent research, including Stochastic Path-Integrated Differential Estimator (SPIDER) [10] and Stochastic Recursive Gradient Algorithm (SARAH) [25], expanded the application of these techniques to both finite-sum and online scenarios, where n is large or possibly infinite. These methods boast an improved sample complexity of min( √ nε -2 , ε -3 ) . SpiderBoost [38], achieves the same near-optimal complexity performance as SPIDER, but allows a much larger step size and hence runs faster in practice than SPIDER. In this paper, we use a probabilistic variant of SpiderBoost as the variance reduction module for CSO and FCCO problems. We highlight that alternative techniques, such as SARAH, can also be applied and offer similar guarantees.

Bias Correction. One of the classic problems in statistics is to design procedures to reduce the bias of estimators. Well-established general bias correction techniques, such as the jackknife [32], bootstrap [8], Taylor series [39, 14], have been extensively studied and applied in various contexts [20]. However, these methods are predominantly examined in relation to standard statistical distributions, with limited emphasis on their adaptability to optimization problems. Our proposed extrapolationbased approach is derived from sample-splitting methods [14], specifically tailored and analyzed for optimization problems involving unknown distributions.

Stochastic Composition Optimization. Finally, a closely related class of problems, called stochastic composition optimization, has been extensively studied (e.g., [40, 9, 36, 37]) in the literature where the goal is:

Despite having nested expectations in their formulations (CSO) and (12) are fundamentally different: a) in stochastic composite optimization the inner randomness η is conditionally dependent on the outer randomness ξ and b) in CSO the inner random function g η ( x , ξ ) depends on both ξ and η . These differences lead to quite different sample complexity bounds for these problems, as explored in Hu et al. [15]. In fact, Zhang and Xiao [41] presented a near optimal complexity of O (min( ε -3 , √ nε -2 ))

for stochastic composite optimization problems using nested variance reduction. While Wang et al. [36] also use the "extrapolation" technique, their motivation and formula are significantly different from ours and cannot reduce the bias in the CSO problem.

C Missing Details from Section 2

Lemma 1 (Moments of D m ) The moments of δ ∈ D m are bounded as follows

Proof: Define ˆ δ = δ -E [ δ ] as the centered random variable. Now

So we focus on E [ ˆ δ k ] in the remainder of the proof. For k = 2 ,

For k = 3 ,

For k = 4 ,

For k = 5 ,

̸

̸

̸

For general k > 0 , we expand the following term as a function of m

As E [ ˆ δ i ] = 0 and ˆ δ i and ˆ δ j are independent for different i and j , the outcome has the following form

where ∑ k i =2 a i is the count of independent { ˆ δ i } used in σ a 2 2 σ a 3 3 · · · σ a 4 4 . Among the terms in (13), the dominating one in terms of m is one with largest ∑ k i =2 a i , i.e.

Then, we can simplify the upper right-hand side with

which gives all the desired results.

□

Proposition 2 (First-order Guarantee) Assume that D m and q ( · ) satisfy Assumption 1 and 2 respectively with k = 1 . Then, ∀ s ∈ R , ∣ ∣ ∣ E [ L (1) D m q ( s ) ] -q ( s + E [ δ ]) ∣ ∣ ∣ ≤ a 2 σ 2 / (2 m ) .

Proof: Let h = E [ δ ] . If the function q ∈ C 2 , then the Taylor expansion at s + h with remainders leads to

where ϕ 1 between s + h and s + δ . Then the error of extrapolation becomes

By Assumption 2 and Lemma 1, we have that

This completes the proof.

□

Proposition 1 (Second-order Guarantee) Assume that distribution D m and q ( · ) satisfies Assumption 1 and 2 respectively with k = 2 . Then, for all s ∈ R , ∣ ∣ ∣ E [ L (2) D m q ( s ) ] -q ( s + E [ δ ]) ∣ ∣ ∣ ≤ 4 a 3 σ 3 +9 a 4 σ 2 2 2 + 5 a 4 σ 4 -3 σ 2 2 3 .

Proof: Let h = E [ δ ] . If the function q ∈ C 4 , then the Taylor expansion at s + h with remainders leads to

where ϕ 1 , ϕ 2 , ϕ 3 between s + h and s + δ 1 , s + δ 2 , s + δ 3 respectively.

As E [ δ -h ] = 0 , the error of extrapolation becomes

where the second inequality uses the upper bound on q (3) ( · ) (Assumption 2) and the third inequality uses ( δ -h ) 4 is non-negative and the last inequality uses the uniform bound on q (4) ( · ) (Assumption 2).

Then

we first use that E [( δ 1 -h ) 3 ] = E [( δ 2 -h ) 3 ] = 4 E [( δ 1 + δ 2 2 -h ) 3 ] and the uses the bound on moments in Lemma 1. Note that E ( δ 1 + δ 2 2 -h ) 4 can be seen as the 4th order moments of a batch size of 2 m . □

Proposition 3 Assume q ∈ C 6 . Then L (3) D m as defined below is a third-order extrapolation operator.

Proof: Let h = E [ δ ] . If q ∈ C 2 k , then q has the following Taylor expansion

Eliminate the third order term in the Taylor expansion. Consider the following affine combination which

We determine α 1 and α 2 by expanding L (2) D m q ( s ) and L (2) D 2 m q ( s ) and analyze the coefficients of terms:

(Eliminate third term). Taylor expansion of F (3) D m q ( s ) at s + h should have third order term E [( δ -h ) 3 ] . That is,

Therefore, α 1 and α 2 can be determined through the following linear system

The solution is α 1 = -1 3 and α 2 = 4 3 .

For k = 3 order extrapolation, consider the following

We determine α ′ 1 , α ′ 2 and α ′ 3 by satisfying the following two conditions

(Affine). Taylor expansion of L (3) D m q ( s ) at s + h should have zero order term q ( x + h ) , i.e.

This is equivalent to

Taylor expansion of L q ( s ) at s + h should have 4th order term E [( δ -h ) 4 ] . That is

This is equivalent to

D m .

Therefore, α ′ 1 , α ′ 2 and α ′ 3 can be determined through the following linear system

The solution is α ′ 1 = 1 12 , α ′ 2 = -4 3 and α ′ 3 = 9 4 . Then consider the Taylor expansion of L (3) D m q ( s ) at s + h with (2), we can

where the first inequality uses the fact that L (3) D m is an affine mapping and the last inequality uses Lemma 1. Therefore, L (3) D m is a 3rd-order extrapolation operator. We can expand it into

Lemma 2 (Variance Bound) Assume that q : R p → R ℓ is in C 4 and D m is the distribution in Assumption 1. Suppose that the variance of q ( s + δ ) is bounded as

Then the variance of extrapolation L (2) D m q ( s ) is upper bounded by

Proof: Let us use the definition of L (2) D m q ( s ) :

This completes the proof.

□

D Stationary Point Convergence Proofs from Section 3 (CSO)

In this section, we provide the convergence proofs for the CSO problem. We start by establishing some helpful lemmas in Appendix D.1. In Appendix D.2, we reanalyze the BSGD algorithm to obtain explicit bias and variance bounds, which are then useful when we analyze E-BSGD in Appendix D.3. Similarly, we reanalyze BSpiderBoost in Appendix D.4 and use the resulting bias and variance bounds for the analysis of E-BSpiderBoost in Appendix D.5.

Note that throughout our analyses, we define E t +1 [ ·| t ] as the expectation of randomness at time t +1 conditioning on the randomness until time t . When there is no ambiguity, we use E [ · ] instead of E t +1 [ ·| t ] .

D.1 Helpful Lemmas

Lemma 3 (Sufficient Decrease) Suppose Assumption 5 holds true and γ ≤ 1 2 L F then

where E [ · ] denote conditional expectation over the randomness at time t conditioned on all of the past randomness until time t .

Proof: In this proof, we use E [ · ] to denote conditional expectation over the randomness at time t conditioned on all the past randomness until time t .

Let us expand F ( x t +1 ) and apply the L F -smoothness of F

Expand the middle term with

Combine with the inequality

By taking γ ≤ 1 2 L F , we have that

Re-arranging the terms we get the desired inequality.

A consequence of Lemma 3 is the following result.

Lemma 4 (Descent Lemma) Suppose Assumption 5 holds true. By taking γ ≤ 1 2 L F , we have,

where the expectation is taken over all randomness from t = 0 to T .

Proof: We denote the conditional expectation at time t in the descent lemma (Lemma 3) as E t +1 [ ·| t ] which conditions on all past randomness until time t . Then the descent lemma can be written as

□

If we additionally consider the randomness at time t -1 , and apply E t [ ·| t -1] to both sides

By the law of iterative expectations, we have E t [ E t +1 [ ·| t ] | t -1 ] = E t E t +1 [ ·| t -1]

Similarly, we can apply E t -1 [ ·| t -2] , E t -2 [ ·| t -3] , . . . , E 2 [ ·| 1] and finally E 1 [ · ]

Now that both sides of the inequality have no randomness, we can simplify the notation by applying E t +1 . . . [ E t [ · ]] to both sides and by denoting

Then the descent lemma becomes

Now we can sum the descent lemmas from t = 0 to T -1

After simplification and division by T , we get

The following corollary is a consequence of Proposition 1.

Corollary 1 Assume ∇ f ξ in CSO satisfies

Let's further assume that the higher order moments of g η ( · ) are bounded,

where [ · ] i refers to the i -th coordinate of a vector. Consider the L (2) D t +1 g ,ξ ∇ f ξ (0) defined in (6) , then

Proof: The Proposition 1 gives the following upper bound

For simplicity, we can relax the upper bound to

D.2 Convergence of BSGD

In this section, we reanalyze the BSGD algorithm of [16] to obtain bounds on bias and variance of its gradient estimates. Theorem 6 shows that BSGD achieves an O ( ε -6 ) sample complexity.

Lemma 5 (Bias and Variance of BSGD) The bias and variance of BSGD are

Proof: Denote G t +1 = G t +1 BSGD (5) and denote E [ · ] as the conditional expectation E t +1 [ ·| t ] which conditions on all past randomness until time t . Note that the ∇ g ˜ η can be estimated without bias, i.e.

Then let's first look at the bias of BSGD

For the first inequality, we take the expectation outside the norm and bound ∇ g with C

On the other hand, the variance of BSGD can be decomposed into inner variance and outer variance

The inner variance is bounded as follows

The outer variance is independent of the inner batch size and can be bounded by

Therefore, the variance is bounded as follows

This completes the proof.

Theorem 6 (BSGD Convergence) Consider the (CSO) problem. Suppose Assumptions 3, 4, 5 holds true. Let step size γ ≤ 1 / (2 L F ) . Then for BSGD, x s picked uniformly at random among { x t } T -1 t =0 satisfies: E [ ∥∇ F ( x s ) ∥ 2 2 ] ≤ ε 2 , for nonconvex F , if the inner batch size m = Ω ( σ 2 bias ε -2 ) and the number of iterations T = Ω ( ( F ( x 0 ) -F ⋆ ) L F ( σ 2 in / m + σ 2 out ) ε -4 ) , where σ 2 in = ζ 2 g C 2 f + σ 2 g C 2 g L 2 f , σ 2 out = C 2 F , and σ 2 bias = σ 2 g C 2 g L 2 f .

Proof: Denote G t +1 = G t +1 BSGD (1). Using descent lemma (Lemma 4) and bias-variance bounds of BSGD (Lemma 5)

Then we can minimize the right-hand size by optimizing γ to

which is smaller than the bound of step size γ ≤ 1 2 L F if T is greater than the following constant which does not rely on the target precision ε

Then the upper bound of gradient becomes

By taking inner batch size of at least

and iteration T greater than

By picking x s uniformly at random among { x t } T -1 t =0 , we get the desired guarantee.

The resulting sample complexity of BSGD to get to an ε -stationary point is O ( ε -6 ) .

D.3 Convergence of E-BSGD

In this section, we analyze the sample complexity of Algorithm 2 (E-BSGD) for the CSO problem.

Lemma 6 (Bias and Variance of E-BSGD) The bias and variance of E-BSGD are

Proof: Denote G t +1 = G t +1 E-BSGD (7). Like previously (Lemma 5), let E [ · ] denote the conditional expectation E t +1 [ ·| t ] which conditions on all past randomness until time t . In E-BSGD, we apply extrapolation to ∇ f ξ ( · ) . The bias can be estimated with the help of Corollary 1 as

Since the variance of BSGD in Lemma 5 is upper bounded by σ 2 in m + σ 2 out , then Lemma 2 gives

we have that

This proves the claimed bounds.

□

Theorem 3 [E-BSGD Convergence] Consider the (CSO) problem. Suppose Assumptions 3, 4, 5, 6 hold true and L F , C F , ˜ L F , C g , F ⋆ are constants and C e ( f ; g ) := 8 a 3 σ 3 +18 a 4 σ 2 2 +5 a 4 σ 4 96 defined in Corollary 1 are associated with second order extrapolation in the CSO problem. Let step size γ ≤ 1 / (2 L F ) . Then the output x s of E-BSGD (Algorithm 2) satisfies: E [ ∥∇ F ( x s ) ∥ 2 2 ] ≤ ε 2 , for nonconvex F , if the inner batch size m = Ω( C e C g ε -1 / 2 ) , and the number of iterations

Proof: The proof is very similar to Theorem 6. Denote G t +1 = G t +1 E-BSGD (7). Using descent lemma (Lemma 4) and bias-variance bounds of E-BSGD (Lemma 6)

□

Then we optimize γ to we can enforce

we have that

which is smaller than the bound of step size γ ≤ 1 2 L F if T is greater than the following constant which does not rely on the target precision ε

Then the gradient norm has the following upper bound.

In order to reach ε -stationary point, i.e.

By taking inner batch size of at least

and iteration T greater than

By picking x s uniformly at random among { x t } T -1 t =0 , we get the desired guarantee. □

D.4 Convergence of BSpiderBoost

In this section, we reanalyze the BSpiderBoost algorithm of [16] to obtain bounds on bias and variance of its gradient estimates. Theorem 6 shows that BSpiderBoost achieves an O ( ε -5 ) sample complexity.

Let G t +1 BSB as the BSpiderBoost gradient estimate

Lemma 7 (Bias and Variance of BSpiderBoost) If γ ≤ min { 1 2 L F , √ B 2 6 L F } , then the bias and variance of BSpiderBoost are

Proof: Denote G t +1 = G t +1 BSB (8). Like previously (Lemma 5), let E [ · ] denote the conditional expectation E t +1 [ ·| t ] which conditions on all past randomness until time t . Denote G t +1 L and G t +1 S as the large batch and small batch in BSpiderBoost separately, i.e.,

The bias of BSpiderBoost can be decomposed to its distance to BSGD and the distance from BSGD to the full gradient, i.e.,

where the last inequality uses the bias of BSGD from Lemma 5. Then the second term can be bounded as follows

By taking the expectation of randomness of G t

Note that ∥ ∥ E [ G 1 BSGD ] -E [ G 1 ] ∥ ∥ 2 2 = 0 as the first iteration always chooses the large batch. Then as we always use large batch at t = 0 we know that

Therefore combine (14) and (15) we can upper bound the bias

Variance. Now we consider the variance,

where the last equality is because the large batch in BSpiderBoost is similar to BSGD.

Finally, we expand the variance at small batch size epoch

The outer variance T out can be upper bounded as

The inner variance can be bounded by

Then we have that

To sum up, the variance is bounded by

Then averaging over time and now we redefine E [ · ] = E T . . . [ E 0 [ · ]]

If we take γ ≤ √ B 2 6 L , then 14(1 -p out ) L 2 F γ 2 B ≤ 1 2 , therefore

Then with (16), we can bound the bias by

Theorem 7 (BSpiderBoost Convergence) Consider the (CSO) problem. Suppose Assumptions 3, 4, 5 holds true. Let step size γ ≤ 1 / (13 L F ) . Then for BSpiderBoost, x s picked uniformly at random among { x t } T -1 t =0 satisfies: E [ ∥∇ F ( x s ) ∥ 2 2 ] ≤ ε 2 , for nonconvex F , if the inner batch size m = Ω( σ 2 bias ε -2 ) , the hyperparameters of the outer loop of BSpiderBoost are B 1 = ( σ 2 in /m + σ 2 out ) ε -2 , B 2 = O ( ε -1 ) , p out = 1 /B 2 , and the number of iterations T = Ω ( L F ( F ( x 0 ) -F ⋆ ) ε -2 ) , where σ 2 in = ζ 2 g C 2 f + σ 2 g C 2 g L 2 f , σ 2 out = C 2 F , and σ 2 bias = σ 2 g C 2 g L 2 f .

Proof: Denote G t +1 = G t +1 BSB (8). Using descent lemma (Lemma 4) and bias-variance bounds of BSpiderBoost (Lemma 7)

where the last inequality use γ ≤ 1 2 L F . Use the variance estimation of G t +1 and choose B 2 p out = 1

Now we can let γ ≤ 1 13 L F such that 84 L 2 F γ 2 ≤ 1 2

In order for the right-hand side to be ε 2 , the inner batch size

and the outer batch size

The step size γ is upper bounded by min { 1 2 L F , √ B 2 6 L F , 1 13 L F } . As B 2 ≥ 1 , we can take γ = 1 13 L F . So we need iteration T greater than

By picking x s uniformly at random among { x t } T -1 t =0 , we get the desired guarantee.

The resulting sample complexity of BSpiderBoost to get to an ε -stationary point is O ( ε -5 ) .

□

D.5 Convergence of E-BSpiderBoost

In this section, we analyze the sample complexity of Algorithm 3 (E-BSpiderBoost) for the CSO problem.

Lemma 8 (Bias and Variance of E-BSpiderBoost) The bias and variance of E-BSpiderBoost are

where σ 2 in := ζ 2 g C 2 f + σ 2 g C 2 g L 2 f , σ out = C 2 F , and ˜ σ 2 bias = C 2 g C 2 e with C 2 e defined in Corollary 1.

Proof: Denote G t +1 = G t +1 E-BSB (9). Like previously (Lemma 5), let E [ · ] denote the conditional expectation E t [ ·| t ] which conditions on all past randomness until time t . Let G t +1 = G t +1 E-BSB be the E-BSpiderBoost update. We expand the bias as follows

From Lemma 6, we know that

The distance between E [ G t +1 E-BSGD ] and E [ G t +1 ] can be bounded as follows.

Taking expectation with respect to G t

where ∥ ∥ E [ G 1 E-BSGD ] -E [ G 1 ] ∥ ∥ 2 2 = 0 . By averaging over time we have

Then the bias is bounded by

Variance. Since the extrapolation only gives a constant overhead given Lemma 2

Then the variance is bounded by

Theorem 4 [E-BSpiderBoost Convergence] Consider the (CSO) problem under the same assumptions as Theorem 3. Let step size γ ≤ 1 / (13 L F ) . Then the output x s of E-BSpiderBoost (Algorithm 3) satisfies: E [ ∥∇ F ( x s ) ∥ 2 2 ] ≤ ε 2 , for nonconvex F , if the inner batch size m = O ( C e C g ε -0 . 5 ) , the hyperparameters of the outer loop of E-BSpiderBoost B 1 = ( ˜ L 2 F /m + C 2 F ) ε -2 , B 2 = √ B 1 , p out = 1 /B 2 , and the number of iterations

Proof: Denote G t +1 = G t +1 E-BSB (9). Using descent lemma (Lemma 4) and bias-variance bounds of E-BSpiderBoost (Lemma 8)

where the last inequality use γ ≤ 1 2 L F . Use the variance estimation of G t +1 and choose B 2 p out = 1

Now we can let γ ≤ 1 13 L F such that 84 L 2 F γ 2 ≤ 1 2

In order to make the right-hand side ε 2 , the inner batch size

The step size γ is upper bounded by min { 1 2 L F , √ B 2 6 L F , 1 13 L F } . As B 2 ≥ 1 , we can take γ = 1 13 L F . So we need iteration T greater than

By picking x s uniformly at random among { x t } T -1 t =0 , we get the desired guarantee.

□

E Stationary Point Convergence Proofs from Section 4 (FCCO)

In this section, we provide the convergence proofs for the FCCO problem. We start by analyzing a variant of BSpiderBoost (Algorithm 3) for this case in Appendix E.1. In Appendix E.2, we present a multi-level variance reduction approach (called NestedVR) that applies variance reduction in both outer (over the random variable i ) and inner (over the random variable η | i ) loops. In Appendix E.3, we analyze E-NestedVR. As in the case of CSO analyses, our proofs go via bounds on bias and variance terms of these algorithms.

E.1 E-BSpiderBoost for FCCO problem

Theorem 8 Consider the (FCCO) problem. Suppose Assumptions 3, 4, 5, 6 holds true. Let step size γ = O (1 /L F ) . Then the output of E-BSpiderBoost (Algorithm 3) satisfies: E [ ∥∇ F ( x s ) ∥ 2 2 ] ≤ ε 2 , for nonconvex F , if the inner batch size m = Ω(max { C e C g ε -1 / 2 , σ 2 in n -1 ε -2 } ) , the hyperparameters of the outer loop of E-BSpiderBoost B 1 = n, B 2 = √ n, p out = 1 /B 2 , and the number of iterations T = Ω ( L F ( F ( x 0 ) -F ⋆ ) ε -2 ) . The resulting sample complexity is

Remark 9 The sample complexity depends on the relation between n and ε

When n = O (1) , we have a complexity of O ( ε -4 ) . This happens because we did not apply variance reduction for the inner loop.

and the outer batch size

When n = Θ( ε -2 / 3 ) , E-BSpiderBoost has same performance as MSVR-V2 [19] of O ( nε -3 ) = O ( ε -11 / 3 ) .

When n = Θ( ε -1 . 5 ) , E-BSpiderBoost achieves a better sample complexity of O ( ε -3 . 25 ) than O ( ε -4 . 5 ) from MSVR-V2 [19].

When n = Θ( ε -2 ) , we recover O ( ε -3 . 5 ) sample complexity as in Theorem 4.

Proof: Denote G t +1 = G t +1 E-BSB (9). As we are using the finite-sum variant of SpiderBoost for the outer loop of the CSO problem, we only need to change the (17) and (18) to reflect that the outer variance is 0 now instead of σ 2 out B 1 in the general CSO case. More concretely, we update (17) to

and change (18) to

Then our analysis only has to start from the updated version of (19)

We would like all terms on the right-hand side to be bounded by ε 2 . From 2˜ σ 2 bias m 4 ≤ ε 2 we know that

From ( 1 Tp out +1) 84 B 1 σ 2 in m ≤ ε 2 , we know that

From 2( F ( x 0 ) -F ⋆ ) γT ≤ ε 2 , we can choose that

Now the total sample complexity for E-BSpiderBoost for the FCCO problem becomes

By picking x s uniformly at random among { x t } T -1 t =0 , we get the desired guarantee. □

E.2 Convergence of NestedVR

NestedVR Algorithm. We start by describing the NestedVR construction. We maintain states y t +1 i and z t +1 i to approximate

In iteration t +1 , if i is selected, then the state y t +1 i is updated as follows

where ϕ t i is the last time node i is visited. If i is not selected, then

In this case, y t +1 i was never used to compute ∇ f i ( y t +1 i ) because i is not selected at the time t +1 . We use the following quantities

We use G t +1 NVR as the actual updates,

We can also use the following quantity ˆ G t +1 NVR as an auxiliary

Here we use ˜ y t i to represent an i.i.d. copy of y t i where i is selected at time t .

The iterate x t +1 is therefore updated

Lemma 9 The error between G t +1 NVR and ˆ G t +1 NVR can be upper bounded as follows

Proof: In this proof, we ignore the subscript in G t +1 NVR and ˆ G t +1 NVR , we bound the error between G t +1 and associated ˆ G t +1 where

Let's only consider the expectation over the randomness of ∇ g ˜ η ,

Then we can bound the error as follows

Unroll the recursion gives

Lemma 10 (Staleness) Define the staleness of iterates at time t as Ξ t := 1 n ∑ n j =1 ∥ x t -ϕ t j ∥ 2 2 and let G t +1 be the gradient estimate, then

Proof: Like previously (Lemma 5), let E [ · ] denote the expectation conditioned on all previous randomness until t -1 . It is clear that Ξ 0 = 0 , so we only consider t > 0 . We upper bound E [Ξ t ] as follows,

Then we can expand E [Ξ t ] as follows

where we use Cauchy-Schwarz inequality with coefficient β > 0 . By the definition of ϕ t j ,

By taking β = 2 n/B 2 , we have that (1 + 1 β )(1 -B 2 n ) ≤ 1 -B 2 2 n and thus

Note that E [Ξ 0 ] = 0 .

The following lemma describes how the inner variable changes inside the variance.

Lemma 11 Denote E t +1 y := E [ ∥ ∥ y t +1 i -E η | i [ g η ( x t )] ∥ ∥ 2 2 ] to be the error from inner variance and p out T ≤ 1 . Then

Meanwhile, E 1 y = E [ ∥ ∥ y 1 i -E η | i [ g η ( x 0 )] ∥ ∥ 2 2 ] = σ 2 g S 1 .

Proof:

Note that when p in = 1 and S 1 = S 2 = m , we recover the following

Proof: For t ≥ 2 , E i [ E p in [ E η | i [ ∥ ∥ y t +1 i -˜ y t i ∥ ∥ 2 2 ]]] can be upper bounded as follows

Then for summing up t = 1 to T -1

Finally, the error has the following upper bound

Lemma 13 (Bias and Variance of NestedVR) If the step size γ satisfies,

then the variance and bias of NestedVR are

Proof: Notations. Let us define the following terms,

Note that the ˜ G t computed at time t +1 has same expectation as G t

Computing the bias. First consider the two cases in the outer loop

We expand A t +1 2 as follows

where we use (24) in the last equality. Now we take expectation with respect to randomness at t such that G t is a random variable, then

while at initialization we always use large batch

Therefore, when we average over time t

On the other hand, let us consider the upper bound on A t +1 1

From Lemma 11 we know that

From Lemma 10 we know that

Therefore, the bias has the following bound

Note that when p in = 1 and S 1 = S 2 = m , then this bias recovers BSpiderBoost in (16)

Computing the variance. Let us decompose the variance into 3 parts:

where E t +1 var,out and E t +1 var,in are the variance of outer loop and inner loop.

Inner Variance. For t ≥ 1 , we expand the inner variance

We bound the outer variance as

For t = 0 , as we only use large and small batch in the

Therefore, average over time t = 0 , . . . T -1 gives

Let us first apply Lemma 12

Then we apply Lemma 11 on the bound of 1 T ∑ T -1 t =0 E t +1 y

From Lemma 10, we plug in the upper bound of 1 T ∑ T -1 t =0 Ξ t

Finally, we add the upper bound on with E 1 var,in with (29)

Outer Variance. Now we consider the outer variance for t ≥ 1

Compared to (27) we know that the upper bound of is smaller than that of E t +1 var,in . Besides, whereas E 1 var,out = 0 as we use large batch at t = 0 . Therefore, the upper bound of E t +1 var is upper bounded by 2*(30).

Variance of ∇ g ˜ η . From Lemma 9, we know that

Finally, we use E [ ∥ ∥ G t +1 ∥ ∥ 2 2 ] = ∥ ∥ E [ G t +1 ] ∥ ∥ 2 2 + E t +1 var .

By taking step size γ to satisfy

which can be simplified to

Then the coefficient of 1 T ∑ T -1 t =0 E t +1 var is bounded by 1 2

The the variance has the following bound

Theorem 10 Consider the (FCCO) problem. Suppose Assumptions 3, 4, 5 holds true. Let step size γ = O ( 1 √ nL F ) . Then for NestedVR, x s picked uniformly at random among { x t } T -1 t =0 satisfies: E [ ∥∇ F ( x s ) ∥ 2 2 ] ≤ ε 2 , for nonconvex F , if the hyperparameters of the inner loop S 1 = O ( ˜ L 2 F ε -2 ) , S 2 = O ( ˜ L F ε -1 ) , p in = O (1 /S 2 ) , the hyperparameters of the outer loop B 1 = n, B 2 = √ n, p out = 1 /B 2 , and the number of iterations

In fact, it reaches this sample complexity for all p in p out √ 1 -p in ≲ ε .

The resulting sample complexity is

Proof: Using descent lemma (Lemma 4) and bias-variance bounds of NestedVR (Lemma 13)

Compute T 0 . In order to let T 0 ≤ ε 2 , we require that

Compute T 1 . In order to let T 1 to be smaller than ε 2 , we need

Compute T 2 . In order to let the coefficient of 1 T ∑ T -1 t =0 ∥ ∥ E [ G t +1 ] ∥ ∥ 2 2 in T 2 to be less than 1 4 , i.e.

Compute T 3 . Let us now focus on T 3 and notice that the middle term 2(1 -p out ) 2 p out

which requires γ

Using Lemma 13 we have that

Compute T 3 , 3 : As we already know that S 1 = O ( ε -2 ) and T ≥ 1 and B 1 p out ≥ 1 . This imposes no more constraints, i.e.

Compute T 3 , 2 : As S 1 = O ( ε -2 ) and B 1 = n and B 2 = B 1 p out, then it requires

Compute T 3 , 1 : In order to satisfy the following

Now we go back to T 3 and compare the other two coefficients

As γL F ≤ 1 2 ≲ 2(1 -p out ) 2 p out we can safely ignore γL F . On the other hand, from (32) we know that the first term is also have

Constraints from the Bias-Variance Lemma (Lemma 13). By setting B 1 = n and S 1 = O ( ˜ L 2 F ε 2 ) , this constraint translates to

which is weaker than (33).

Summary on the Limit on γ . Combine (33) and (34) and γ ≤ 1 2 L F , we have a final limit on step size γ

Then the total sample complexity of NestedVR can be computed as

( # of iters T ) × ( Avg. outer batch size B 2 = B 1 p out ) × ( Avg. inner batch size S 2 = S 1 p in ) .

This sample complexity has the following requirement

The lower bound nε -3 is reached when in (35) we have

That is, p out p in √ 1 -p in ≲ ε .

In particular, we can choose the following hyperparameters to reach O ( nε -3 ) sample complexity

The step size γ can be chosen as and the iteration complexity

Putting these together gives the claimed sample complexity bound. By picking x s uniformly at random among { x t } T -1 t =0 , we get the desired guarantee. □

E.3 Convergence of E-NestedVR

In this section, we analyze the sample complexity of Algorithm 1 (E-NestedVR) for the FCCO problem with

Lemma 14 (Bias and Variance of E-NestedVR) If the step size γ satisfies

then the variance and bias of E-NestedVR are

Proof: Note that this proof is very similar to NestedVR so we highlight the differences. Let G t +1 = G t +1 E-NVR (36) be the E-NestedVR update and define

We expand the bias by inserting E i,p in ,η | i [ G t +1 i ]

Consider A t +1 1 . The term A t +1 1 captures the difference between full gradient and extrapolated gradient

The first term A t +1 1 , 1 can be upper bounded through smoothness of f ξ , for t ≥ 1

For t = 0 , A 1 1 , 1 = 0 , then

On the other hand, with Lemma 6

where D t +1 y ,S 1 ,i is the distribution of 1 S 1 ∑ η ∈S 1 g η ( x t ) and D t +1 y ,S 2 ,i is the distribution of

Thus the A t +1 1 has the following upper bound

Consider A t +1 2 . Let us expand A t +1 2 through recursion

For t = 0 , we have that A 1 2 = 0 , then average over time gives

Therefore, the bias has the following bound

Using Lemma 11

Using Lemma 10 we have that

Variance. Combine the variance of NestedVR in Lemma 13 and Lemma 2 gives

Theorem 5 [E-NestedVR Convergence] Consider the (FCCO) problem. Under the same assumptions as Theorem 3.

If n = O ( ε -2 / 3 ) , then we choose the hyperaparameters of E-NestedVR (Algorithm 1) as B 1 = B 2 = n, p out = 1 , S 1 = ˜ L 2 F ε -2 , S 2 = ˜ L F ε -1 , p in = ˜ L -1 F ε, γ = O ( 1 L F ) .

If n = Ω( ε -2 / 3 ) , then we choose the hyperaparameters of E-NestedVR as B 1 = n, B 2 = √ n, p out = 1 / √ n, S 1 = S 2 = max { C e C g ε -1 / 2 , ˜ L 2 F / ( nε 2 ) } , p in = 1 , γ = O ( 1 L F ) .

Then the output x s of E-NestedVR satisfies: E [ ∥∇ F ( x s ) ∥ 2 2 ] ≤ ε 2 , for nonconvex F with iterations

Proof: Denote. G t +1 = G t +1 E-NVR (36). Using descent lemma (Lemma 3) and bias-variance of E-NestedVR (Lemma 14)

As we would like the right-hand side to be bounded by either 1 T ∑ T -1 t =0 ∥ ∥ E [ G t +1 ]] ∥ ∥ 2 2 or ε 2 .

Bound on 2( F ( x 0 ) -F ⋆ ) γT with ε 2 , i.e.

Coefficient of 1 T ∑ T -1 t =0 ∥ ∥ E [ G t +1 ] ∥ ∥ 2 2 is bounded by 1 4 , i.e.

which can be achieved by choosing the following step size

Bound on the variance. First notice from (40) and γ ≤ 1 2 L F ,

Therefore, we only need to consider the upper bound on

Bound on 2(1 -p in ) 2 ˜ L 2 F S 1 with ε 2

Bound C 2 e S 4 2 with ε 2 . This leads to

We impose the constraints for each term

These can be simplified as

Constraints from Lemma 14

which can be translated to

Constraint from sufficient decrease lemma:

We simplify the conditions noticing that 1) (48) is weaker than (46); 2) (45) and (51) are weaker than (52). Combine all the constraints on γ , i.e. (43), (44), (49), (50), (52)

This can be simplified as an upper bound

Now we consider two sets of hyperparameters depending on the size of n Case 1: For n = O ( ε -2 / 3 ) , we choose the following set of hyperparameters

Then we have γ ≲ 1 L F min { p in √ S 1 √ 1 -p in , 1 } = 1 L F , we have the total sample complexity of

0

25000

50000

75000

1.05

1.10

1.15

1.20

1.25

Suboptimality

BSGD

m=1

m=10

m=100

0

25000

50000

E-BSGD

m=1

m=10

m=100

75000

Case 2: For n = Ω( ε -2 / 3 ) , we choose the following set of hyperparameters

In this case, (46) is stronger than (47) which requires S 1 ≳ ˜ L 2 F nε 2

Then we have γ ≲ 1 L F min { p out √ n √ S 1 √ 1 -p out , 1 } = 1 L F , we have the total sample complexity of

By picking x uniformly at random among { x } , we get the desired guarantee. □

s t T -1 t =0

F Missing Details from Section 5

F.1 Application of First-order MAML

Over the past few years, the MAML framework [11] has become quite popular for few-shot supervised learning and meta reinforcement learning tasks. The first-order Model-Agnostic Meta-Learning (MAML) can be formulated mathematically as follows:

where α is the step size, D i supp and D i query are meta-training and meta-testing data respectively and ℓ i being the loss function of task i . Stated in the CSO framework, f ξ ( x ) := ℓ i ( x , D i query ) and g η ( x , ξ ) := x -α ∇ ℓ i ( x , D i supp ) where ξ = ( i, D i query ) and η = D i supp .

In this context, lots of popular choices for f ξ are smooth. For illustration purposes, we now discuss a widely used sine-wave few-shot regression task as appearing from the work of Finn et al. [11], where the goal is to do a few-shot learning of a sine wave, A sin( t -ϕ ) , using a neural network Φ x ( t ) with smooth activations, where A and ϕ represent the unknown amplitude and phase, and x denotes the model weight. Each task i is characterized by ( A i , ϕ i , D i query ) . In the first-order MAML training process, we randomly select a task i , and draw training data η = D i supp . Define the loss function for a given dataset D as ℓ i (Φ x ; D ) = 1 2 E t ∼D ∥ ∥ A i sin( t -ϕ i ) -Φ x ( t ) ∥ ∥ 2 2 . We then establish the outer function f i ( x ) = ℓ i (Φ x ; D i query ) and inner function g η ( x ) = x -α ∇ x ℓ i (Φ x ; D i supp ) . As f i is smooth, our results are applicable.

In Figure 4, we show the results of BSGD and E-BSGD applied to this problem. In this experiment, the amplitude A is drawn from a uniform distribution U (0 . 1 , 5) and the phase ϕ is drawn from U (0 , π ) . Both D supp and D query are independently drawn from U ( -5 , 5) . The step size is set to

α = 0 . 01 . The batch size is fixed to 10. The performances of BSGD and E-BSGD are very close. This is not surprising because finetuning step size α is chosen to be small which significantly reduces the variance of g η , making the bias of meta gradient to be very small ( O ( α 2 ) ). Therefore, we observe similar performance of BSGD and E-BSGD. Similar trend also holds for BSpiderBoost and NestedVR compared to their extrapolated variants.

F.2 Application of Deep Average Precision Maximization

The areas under precision-recall curve (AUPRC) has an unbiased point estimator that maximizes average precision (AP) [26, 34]. Let S + and S -be the set of positive and negative samples and S = S -∪ S + . Let h w ( · ) be a classifier parameterized with w and ℓ be a surrogate function, such as logistic or sigmoid. A smooth surrogate objective for maximizing average precision can be formulated as [33]:

This problem can be seen as a conditional stochastic optimization problem with g i ( w ) = [ ∑ x ∈S + ℓ ( h w ( x ) -h w ( x i )) , ∑ x ∈S ℓ ( h w ( x ) -h w ( x i ))] and f i : R × R \{ 0 } → R is defined as f i ( y ) = -[ y ] 1 [ y ] 2 where [ y ] k denotes the k th coordinate of a vector y ∈ R × R \{ 0 } . During the stochastic optimization of this objective, we draw uniformly at random ξ := x i (drawn from the set S + ) as a positive sample and η | ξ = [ F x 1 , F x 2 ] where set x 1 is drawn uniformly at random from S + and x 2 is drawn uniformly at random from S and functional F x ( w ) := ℓ ( h w ( x ) -h w ( x i )) . Note that f i ∈ C ∞ is smooth with gradient

Therefore, our results from Sections 3 and 4 again apply.

F.3 Necessity of Additional Smoothness Conditions

Throughout the paper, we assume bounded moments (Assumption 1) and a smoothness condition (Assumption 2) to derive our extrapolation technique. However, it is worth noting that the technique itself does not explicitly depend on higher-order derivatives. Our theoretical framework does not address the behavior of extrapolation in the absence of these smoothness constraints. In this section, we investigate the application of extrapolation to two non-smooth functions:

ReLU function given by q ( x ) = max { x, 0 } ;

Perturbed quadratics represented as q ( x ) = x 2 / 2 + TriangleWave(x) + 1 . The function TriangleWave( x ) has a period of 2 and spans the range [-1,1], defined as:

Visual representations of these functions can be found in Figure 5c. We set s = 0 and consider a random variable δ ∼ N (10 , 100) with m = 1 . We then apply first-, second-, and third-order extrapolation. The outcomes are depicted in Figure 5. Remarkably, both the ReLU and the perturbed quadratic functions do not conform to the differentiability assumptions inherent to our stochastic extrapolation schemes. Nonetheless, as indicated by Figure 5a and Figure 5b, our proposed secondand third-order extrapolation techniques yield a superior approximation of q ( E [ δ ]) .

10

0

10

1

10

2

10

3

10

4

Number of estimates

10

0

10

1

10

2

10

3

|

q

(

s

+ [ ])

Avg

(

( )

q

(

s

))|

ReLU

(1)

12

m

q

(

s

)

(2)

6

m

q

(

s

)

(3)

m

q

(

s

)

10

0

10

1

10

2

10

3

10

4

Number of estimates

10

0

10

1

10

2

10

3

|

q

(

s

+ [ ])

Avg

(

( )

q

(

s

))|

Perturbed quadratics

(1)

12

m

q

(

s

)

(2)

6

m

q

(

s

)

(3)

m

q

(

s

)

5.0

2.5

0.0

2.5

5.0

x

0

5

10

15

q

Perturbed quad.

Quadratic

ReLU

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers CSO problems and its finite-sum variant FCCO, and proposed new algorithms incorporating extrapolation and variance reduction. The proposed algorithms are shown to enjoy a better sample size in the inner layer which induces an improved sample complexity compared to existing CSO literature. The authors also furthered their study in the FCCO regime, the proposed algorithm enjoys a better complexity in moderately large $n$ regime.\n\nSTRENGTHS:\n1. Improved sample complexities for CSO and FCCO problems\n2. The algorithm design incorporates extrapolation which is new to the CSO literature I think, and it brings an improvement in the sample complexity in the inner layer, which reveals some innovation.\n\nWEAKNESSES:\n1. The additional regularity assumption (compared to [15]) on the bounded higher-order gradients/moments, even though authors tried to rationalize it, reduces the significance of the result.\n2. You mentioned \"... $\\Omega(\\epsilon^{-3})$ is a sample complexity lower bound for standard stochastic nonconvex optimization [2]...\" in Table 1, but as far as I can see, with your higher-order Lipschitz condition, the statement may not hold. Your function class is more restricted compared to the setting in [2]. I think authors may refer to  Could you please comment more on it?\n   - Carmon, Yair, et al. \"Lower bounds for finding stationary points ii: first-order methods.\" Mathematical Programming 185.1-2 (2021): 315-355.\n   - Arjevani, Yossi, et al. \"Second-order information in non-convex stochastic optimization: Power and limitations.\" Conference on Learning Theory. PMLR, 2020.\n3. Some writing issues, I believe the paper's writing style could benefit from a few adjustments to enhance its readability, there .\n   - Line 141, $T_m$ (vs. $T_{D_m}$)\n   - Line 148, \"lesser\"\n   - Line 173, \"to (be) $\\nabla$...\"\n   - Line 224, \"... some time $t$...\", I think such statements can be further revised and formalized.\n\nWith that, I think the paper provides some interesting results, especially in terms of the algorithm design, but I am not that convinced on the significance of the results (regarding the a bit unfair comparison) and feel that the writing can be further revised for a better flow.\n\nQUESTIONS:\n1. For clarification, with the extrapolation framework you need the constant term $s$, but when applied to CSO, you just set $s=0$, I am not sure why it is important to include the $s$ into your framework if there is no nontrivial instantiation. Could you please comment more on it?\n2. For clarification, in Theorem 3, you set $m=O(\\epsilon^{-0.5})$, but in Line 247, you mentioned the straightforward extension of E-BSpiderBoost requires a different $m$ by taking a maximum, I am wondering why the setting of $m$ in Theorem 3 does not directly apply here.\n\nLIMITATIONS:\nN/A.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a stochastic extrapolation technique with variance reduction to obtain improved sample complexity bounds for conditional stochastic optimization problem. Numerical study also support the effectiveness of the proposed algorithm.\n\nSTRENGTHS:\n- The stochastic extrapolation is a useful tool for CSO problem. The technique is new for solving the CSO problem, though it requires higher-order smoothness assumption of the loss function. I think it could be potentially useful for other stochastic optimization formulations. I am also curious about how the authors come up with this approach? Do authors inspired from any existing literature?\n- The overall presentation is very clear and the theoretical results with technique assumptions are presented in detail.\n- Numerical study is very strong to support the effectiveness of the algorithm.\n\nWEAKNESSES:\n- From my perspective, some notations could be confusing. For example, in Theorem 2 the authors use $\\Omega(\\cdot)$ notation for batch size $m$ and number of iterations $T$, but in line 235 the authors use $\\mathcal{O}$ instead. I would encourage the authors check those notations throughout the paper to make it consistent.\n- Please check and revise the first sentence in line 287.\n- In the theoretical analysis, the authors present iteration complexity together with sample compelxity per-iteration. However, in related work comparison, the authors only compare the overall total sample complexity of baseline methods. I encourage the authors also check and compare iteration complexity together with sample compelxity per-iteration. \n- In Table 1, I think the authors should also add V-MLMC method (from [16]) for comparison. Also, in numerical study, I am confusing why the authors do not compare with RT-MLMC method but only V-MLMC method?\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper highlights that optimizing conditional stochastic optimization problems is challenging as the sample-averaged gradient is biased. The biased gradient increases the sample complexity needed to achieve convergence. To account for the bias, the paper proposes stochastic extrapolation. The method shows that the bias can be expressed as a sum of Taylor series expansion terms and the bias can be estimated by constructing estimates for each of the terms in the expansion. The key insight is that the size of the $k^{th}$ order term of the series is of size $O(1/m^k)$ where $m$ is the batch size. They propose the $k^{th}$ order extrapolation operator which computes up to the $k^{th}$ order term in the Taylor series and uses the terms to correct for the bias. The paper then uses the the stochastic extrapolation technique to bias correct the gradients for the conditional stochastic optimization problem and finite-sum coupled compositional optimization problem. They then prove convergence guarantees for optimization algorithms applied on the two problem types, showing stochastic extrapolation improves the sample complexity of the algorithms. Finally, they perform a numeric experiment on invariant logistic regression and show the extrapolated version of the algorithms achieves lower sub-optimality.\n\nSTRENGTHS:\nThe paper proposes a fairly novel approach to address the issue of biased sub-gradients. The problem and approach are described clearly and intuitively. The paper is also able to highlight some important optimization problems that satisfy the regularity conditions leveraged by the paper. The paper’s significance is being able to leverage these regularity assumptions in order to prove stronger sample complexity bounds.\n\nWEAKNESSES:\nThe paper is hard to read since authors i) often reference the appendix with limited context, ii) do not point out if the reference is in the appendix or not, and iii) do not define notation. \n\t- As an example for the first point, Lemma 1, Lemma 2, and Theorem 7 are all in the appendix but it is not noted. \n\t- For the second point, Lemma 2 also is referenced in line 155 to quantify the variance of the operator applied to the function, but it’s unclear what Lemma 2 shows precisely. In most cases, it seems that the authors are trying to make a statement and reference the proof, however, the language is not clear that is the case. As evidence, when referencing Lemma 2, the authors state “a consequence of Lemma 2 is that” when it could be more clear by stating “Lemma 2 (in Appendix Section B) proves”. \n\t-For the third point, it is hard to find a definition for V in Lemma 2. \nThe above three points make reading and verifying the statements in the paper challenging. \n\nThe paper has minimal numeric studies. For example, it would be interesting to see how the approach performs for problems that do not satisfy the regularity conditions. It may also be interesting to see the improvement as the level of bias varies due to underlying problem parameters.\n\nQUESTIONS:\n1. The paper mentions a burn-in rate for higher order extrapolation. Is there any intuition of controls the burn-in rate or if the burn-in rate is theoretically quantifiable?\n2. Does the stochastic extrapolation perform better or worse when the number of samples is small? The numerics seem to indicate the performance is the same for BSGD and BSpiderBoost when the # of samples is small.\n\nLIMITATIONS:\nFrom a technical perspective, the authors do note that their proposed approach requires the target optimization problem to satisfy certain regularity conditions. However, it is unclear what is the cost of applying the stochastic extrapolation and if there is any trade-offs.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study stochastic optimization problems with nested conditional expectations, with access to first-order samples from the involved functions. With such an oracle access, it is non-trivial to build unbiased estimates of the gradient, hence normal stochastic optimization methods don't work. \n\nThey authors show that under a third-order differentiability assumption, which they argue to be \"mild\", one could adapt existing extrapolation techniques to this problem and obtain an estimate of the gradient with a reduced bias. This in turn results in substantial improvement in the convergence rate of algorithm fed with this bias-reduced estimate. The authors demonstrate this using state-of-the-art algorithms, both with and without finite-sum variance reduction.\n\nSTRENGTHS:\nThe claimed improvements are substantial. The paper is modular, in the sense that there is one clear goal: reducing the bias of the gradient estimate, and that is carried out separately. Then, the result is fed into existing best algorithms. This makes the contribution very clear.\n\nWEAKNESSES:\nThere seems to be some issue in the last term of Eq (2): since $\\phi_\\delta$ depends on $\\delta$, we cannot take it outside expectation, neither can we decompose the resulting expectation in such a way that $E[(\\delta - h)^{2k-1}]$ comes out and multiplies the expectation of the term involving $\\phi_\\delta$. I am not sure if this will have a major adverse effect on the rest of the results in this section, but I will keep looking deeper in the proofs.\n\nIn terms of related work, I am surprised that the use of extrapolation for bias reduction in Stochastic Composite Optimization is missed. The central work here is:\nM. Wang and J. Liu. Accelerating Stochastic Composition Optimization. 2016\nThe authors need to discuss the relationship between the techniques.\n\nI also find it problematic that the authors call the assumption of third-order differentiability a \"mild assumption\". Whether this is mild or not has to be left to the reader to decide. Currently, the introduction and Table 1 give an incorrect impression. This needs to be made clear way earlier in the introduction, and also laid out clearly in the caption of Table 1. Also, the assumption $n = O(\\epsilon^{-2})$ is certainly worth more explanation that the footnote given.\n\nQUESTIONS:\nPlease add any comments you may have about the points raised under \"Weaknesses\", including the relationship to the work on accelerated stochastic composition optimization, as well as the issue with Eq (2).\n\nLIMITATIONS:\nThe paper can be improved in terms of clarifying the limitations of its results, as explained under \"Weaknesses\" above.\n\nThis is a theoretical paper. As such, greater societal impact is not addressable at this level.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the conditional stochastic optimization (CSO) problem. A noted drawback in CSO is the bias in its sample-averaged gradient, an offshoot of its nested structure, which necessitates an extensive sample complexity to achieve convergence.\nThe authors introduces a new stochastic extrapolation technique address this bias effectively. They propose that by merging this extrapolation with variance reduction techniques, improvements in sample complexity can be realized compared to existing results for non-convex smooth objectives. The authors claim that the proposed methods improve the sample complexities for both the CSO and FCCO under some reguarlity.\n\nSTRENGTHS:\n- Overall writing is clear, and organization is well-structured.\n- The paper introduces an extrapolation-based scheme to alleviate bias in gradient estimations. The proposed method is used for gradient estimations for CSO and FCCO problems.\n- For CSO problem, the authors claim that the proposed methods, E-BSGD and E-BSpiderBoost, achieve an improved sample complexity compared to the existing bounds under some additional reguarlity.\nFor FCCO problem, the authors proposed an algorithm, E-NestedVR, combining the extrapolation-based scheme with a multi-level variance reduction which achieves an improved (or at least matching) sample complexity compared to the existing results. \n- The techniques of the anlysis are interesting, and worth credit.\n\nWEAKNESSES:\n- While the new bounds show improved rates, the results are based on additional regularity assumptions. The authors acknowledge the discrepancy in the set of assumptions used in the current paper and those in the existing results. They state that \"the bounds derived from previous studies do not improve when incorporating this additional regularity assumption\" of the 3rd-order differentiability, and hence argue that their results present an improved bound. \nHowever, I am not sure whether such an assertion is valid. The new regularity condition imposed by this paper is used for the authors' technical convenience, not required by the existing methods. Hence, it is very possible and often obvious that the existing methods do not improve even with the new additional regularity assumption because these methods did not even need such regularity. \n- So I am not sure whether the claim of the \"improved\" rate is valid. Aren't they rather apples and oranges? For example, can you precisely pinpoint where Assumtpino 6 is needed? What rate (just a ballpark) would you get if you do not have this assumption?\n\nQUESTIONS:\nPlease see the comment in the weakness section.\n\nLIMITATIONS:\nPlease see the comment in the weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-27"}