On the Bias-Variance-Cost Tradeoff of Stochastic Optimization

1 Introduction . Solving modern machine learning tasks relies heavily on stochastic gradient descent ( SGD ) and many of its variants . Whilst the vanilla SGD depends crucially on unbiased gradient oracles , constructing unbiased gradient estimators can be very expensive or even impossible for many emerging machine learning applications . Such applications include GANs with regularization [ 25 ] , distributionally robust optimization [ 26 , 14 ] , conditional stochastic optimization [ 20 , 19 ] , meta-learning [ 30 , 11 ] , min-max optimization [ 32 , 33 ] , and bilevel optimization [ 16 ] , just to name a few . As an alternative , one often resorts to some natural biased gradient estimators . Existing work on biased gradient methods [ 17 , 1 , 18 , 7 , 23 ] focuses on the iteration complexity given biased stochastic oracles while ignoring the possibility of constructing refined stochastic oracles with lower bias at a higher cost , e.g. , more samples or more computation power . In this paper , we build a unified framework to study the fundamental tradeoff among the bias , the variance , and the cost . We study optimization problems of the general form min x∈Rd F ( x ) , ( 1 ) and assume that one does not have access to unbiased gradient estimator of F ( x ) via simple Monte Carlo sampling or simulation . Instead we assume that one can construct a sequence of approximations ∗Department of Industrial and Enterprise Systems Engineering , University of Illinois at Urbana-Champaign . †Optimization and Decision Intelligence ( ODI ) Group , Department of Computer Science , ETH Zürich . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . of F ( x ) , denoted as { F l ( x ) } ∞l=0 . Unlike the original objective F ( x ) , unbiased function value and gradient estimators of each F l ( x ) ( correspondingly biased estimators of F ( x ) ) are accessible through some oracles . However , the cost for querying such oracles increases as the approximation accuracy increases . We measure the bias level using the approximation error either in term of the function values |F ( x ) − F l ( x ) | or in terms of the gradient ∥∇F l ( x ) −∇F ( x ) ∥22 . For simplicity , throughout the paper , we assume F and F l are smooth , namely , they have Lipschitz continuous gradients . More precisely , without loss of generality , we assume that ( i ) for any given x , the function approximation error satisfies |F l ( x ) − F ( x ) | = O ( 2−al ) for some constant a > 0 under the convex setting , or the gradient approximation error satisfies ∥∇F l ( x ) −∇F ( x ) ∥22 = O ( 2−al ) under the nonconvex setting ; ( ii ) for each l ∈ N , there exists a stochastic oracle ( SOl ) that returns an unbiased estimator H l ( x , ζl ) of ∇F l ( x ) − ∇F l−1 ( x ) given x with bounded variance : V ( H l ( x , ζl ) ) = O ( 2−bl ) for some constant b > 0 ; ( iii ) the cost to query the oracle SOl is Cl = O ( 2cl ) for some constant c > 0 . Assumption ( ii ) indicates that the variance of the stochastic estimator of the function difference decreases with l. Intuitively , as function F l is close to function F l−1 , one can construct highly correlated gradient estimators for ∇F l ( x ) and ∇F l−1 ( x ) such that the variance of their difference is small.As we will show later , these constants a , b , c play a crucial role in the complexity of the problems of our interest . 1.1 Motivational Examples . Conditional Stochastic Optimization ( CSO ) In CSO [ 19 ] , the objective function has the following form : F ( x ) : = Eξfξ ( Eη|ξgη ( x , ξ ) ) , where ξ and η are random vectors . In general , it is not easy to directly obtain unbiased gradient estimators from samples of P ( ξ ) and P ( η|ξ ) , because of the nonlinearity of fξ ( · ) . Instead , we can construct a sequence of approximation functions given by F l ( x ) = Eζl= [ ξ , η1 , ... , η2l ] [ fξ ( 1 2l ∑2l j=1 gηj ( x , ξ ) ) ] , l = 0 , 1 , . . . , where { ηj } 2 l j=1 are independent and identically distributed ( i.i.d . ) random vectors from P ( η|ξ ) . Unlike F ( x ) , unbiased estimators for ∇F l ( x ) can be easily obtained directly through samples . Under mild conditions , we can show that assumptions ( i ) - ( iii ) hold true ; in particular , we have a = 1 , b = 1 , c = 1 . Distributionally Robust Optimization ( DRO ) DRO minimizes the expected loss with respect to the worst distribution in an uncertainty set U ( P ) for a given distribution P . The objective of DRO has the form : minx∈X FP ( x ) : = supQ∈U ( P ) Eξ∼Qℓ ( x ; ξ ) . Because of the maximal operator over the uncertainty set , it is hard to obtain unbiased gradient estimator using samples from P . Instead one constructs a series of approximation functions given by F l ( x ) = ∑2l i=1 q ∗ i ℓ ( x ; ξi ) , where { ξi } 2 l i=1 are i.i.d . samples from P , Pn denotes the empirical distribution , and q∗ = { q∗i } 2 l i=1 attains the maximum of supq∈∆ 2l , q∈U ( Pn ) ∑2l i=1 qiℓ ( x ; ξi ) for a 2 l-dimensional simplex ∆2l . For CVaR and χ2-penalty DRO problems , Levy et al . [ 26 ] showed that assumptions ( i ) - ( iii ) hold true with a = 2 , b = 1 , c = 1 for CVaR DRO and a = 1 , b = 1 , c = 1 for χ2-penalized DRO . 1.2 Multilevel Monte Carlo ( MLMC ) Techniques . A naive approach to solve ( 1 ) is to perform SGD on the approximation function FL ( x ) with a level L such that the bias is small and within a targeted accuracy ϵ . We call such a framework L-SGD . This encapsulates several existing algorithms under different contexts , such as BSGD [ 20 ] in conditional stochastic optimization , multi-step MAML [ 21 ] in meta-learning , and biased gradient method [ 26 ] in distributionally robust optimization . Despite its simplicity , L-SGD generally fails to achieve the smallest total cost since it requires expensive stochastic oracles to ensure small bias . On the other hand , the multi-level Monte Carlo ( MLMC ) sampling technique , originally designed for stochastic simulation [ 15 ] , is amenable to obtain better gradient estimators by exploiting the bias-cost tradeoff . The idea is to query fewer oracles with higher cost and smaller bias , and more oracles with lower cost and larger bias ; thus , the cost can be effectively reduced . There exist several variations of MLMC with randomization , including the MLMC estimator with randomized truncation ( denoted as RT-MLMC ) [ 5 ] , the MLMC estimator with importance sampling ( denoted RU-MLMC ) , the Russian roulette estimator ( denoted as RR-MLMC ) that uses randomized telescope [ 22 ] . These MLMC techniques have been recently explored in different optimization contexts . For instance , Blanchet and Glynn [ 5 ] combined the MLMC idea with sample average approximation to address stochastic optimization . Dereich and Müller-Gronbach [ 10 ] integrated the vanilla MLMC ( denoted as V-MLMC ) gradient estimator with Robbins-Monro and Polyak-Ruppert stochastic approximation schemes , and analyzed their convergences . Recently , the RT-MLMC technique was applied to stochastic compositional optimization with strongly convex objectives in Blanchet et al . [ 4 ] and distributional robust optimization in Levy et al . [ 26 ] , Ghosh and Squillante [ 14 ] . Beatson and Adams [ 3 ] studied the asymptotic behaviors of stochastic gradient descent with RT-MLMC and RRMLMC estimators . Concurrently , Asi et al . [ 2 ] applied RT-MLMC to estimating proximal points and Moreau-Yoshida envelope gradient , minimization of maximal loss , and proximal gradient methods . 1.3 Our Contributions . In this paper , we provide a systematic comparison among these MLMC techniques when combined with stochastic gradient descent . Our primary focus is on the analysis of their total computation cost for achieving an ϵ-optimal solution for convex problems and an ϵ-stationary point for nonconvex problems , which is largely missing in the literature . Our contributions are three-fold : Non-asymptotic analysis of MLMC gradient methods . We analyze the non-asymptotic performances and computation complexities of various MLMC gradient methods under the general stochastic optimization framework in all regimes , e.g. , different convexity assumptions and different combinations of a , b , c. Our main results are summarized in Table 1 . Previous results either focus on specific applications [ 4 , 26 ] or only asymptotic behaviors [ 3 ] in restricted regimes . Theoretical comparisons and new insights . Our comparative study of different MLMC techniques yields several interesting findings : ( 1 ) When b > c , i.e. , the variance decays faster than the increase of the cost , all three randomized MLMC gradient methods and V-MLMC ( only when a ≥ min { b , c } ) nearly match the complexity of classical unbiased SGD , implying that the problem is no harder than classical stochastic optimization , regardless of the bias . ( 2 ) When b ≤ c , the unbiased MLMC constructions , RU-MLMC and RR-MLMC , are no longer applicable since either the expected periteration cost or the variance of the gradient estimator goes to infinity . In this case , RT-MLMC is the most favorable approach without requiring a mini-batch . ( 3 ) In all regimes with 0 < a < ∞ ( i.e. , bias exists ) , using the naive L-SGD is strictly sub-optimal in terms of the total cost , demonstrating the importance of the exploitation of the bias-variance-cost tradeoff . Improved sample complexity for conditional stochastic optimization . When applied to CSO problems ( resp . a = b = c = 1 ) , we show that the RT-MLMC and the vanilla MLMC gradient methods significantly improve over existing results reported in the literature . In particular , for nonconvex smooth CSO problems , RT-MLMC and vanilla MLMC achieve Õ ( ϵ−4 ) sample complexity , which is even better than the best-known Õ ( ϵ−5 ) achieved by the variance reduction technique [ 20 ] . 1.4 Preliminaries . A function f ( · ) : Rd → R is L-Lipschitz continuous if |f ( x ) − f ( y ) | ≤ L∥x − y∥2 holds for any x , y ∈ Rd . A function f ( · ) is S-smooth on Rd , if it is continuously differentiable on Rd and it holds that ∥∇f ( x ) − ∇f ( y ) ∥2 ≤ S∥x − y∥2 for any x , y ∈ Rd . For any x , y ∈ Rd , and a continuously differentiable function f , if it holds that f ( x ) − f ( y ) −∇f ( y ) ⊤ ( x− y ) ≥ µ2 ∥x− y∥ 2 2 , we say f is µ-strongly convex when µ > 0 and f is convex when µ = 0 . Let x∗ ∈ argminx F ( x ) . We say x is an ϵ-optimal solution if F ( x ) − F ( x∗ ) ≤ ϵ , and x is an ϵ-stationarity point of F if ∥∇F ( x ) ∥22 ≤ ϵ. Polyak-Łojasiewicz ( PL ) condition [ 24 ] is a generalization of strong convexity such that ∥∇F ( x ) ∥22 ≥ 2µ ( F ( x ) − F ( x∗ ) ) for µ > 0 . By convention we let ∇F−1 ( x ) = 0 . For simplicity , throughout the paper , we assume the desired accuracy ϵ > 0 is small enough so that ϵ−1 ≥ log ( ϵ−1 ) ≥ 1 . We use O ( · ) to hide constants that does not depend on the desired accuracy ϵ. Õ further hides the log ( ϵ−1 ) term .

Summary: This paper provides a systematic study of the bias-variance-cost tradeoff for several MLMC gradient methods and L-SGD, under the assumption that (i) there exist a sequence of obj. approximations; (ii)  there exist stochastic oracles which generate unbiased estimates of the gradient for the obj. approximations, and unbiased estimates of the difference between function approximation gradients. They assume three parameters a, b, c which respectively describe the bias, variance of the stochastic oracle, and cost to query the oracle, and provide insights on which class of algorithms work better under different configurations of a, b, c. They also show that for conditional stochastic optimization problems, MLMC methods can significantly reduce the sample complexity.