On the Bias-Variance-Cost Tradeoff of Stochastic Optimization

1 Introduction . Solving modern machine learning tasks relies heavily on stochastic gradient descent ( SGD ) and many of its variants . Whilst the vanilla SGD depends crucially on unbiased gradient oracles , constructing unbiased gradient estimators can be very expensive or even impossible for many emerging machine learning applications . Such applications include GANs with regularization [ 25 ] , distributionally robust optimization [ 26 , 14 ] , conditional stochastic optimization [ 20 , 19 ] , meta-learning [ 30 , 11 ] , min-max optimization [ 32 , 33 ] , and bilevel optimization [ 16 ] , just to name a few . As an alternative , one often resorts to some natural biased gradient estimators . Existing work on biased gradient methods [ 17 , 1 , 18 , 7 , 23 ] focuses on the iteration complexity given biased stochastic oracles while ignoring the possibility of constructing refined stochastic oracles with lower bias at a higher cost , e.g. , more samples or more computation power . In this paper , we build a unified framework to study the fundamental tradeoff among the bias , the variance , and the cost . We study optimization problems of the general form min x∈Rd F ( x ) , ( 1 ) and assume that one does not have access to unbiased gradient estimator of F ( x ) via simple Monte Carlo sampling or simulation . Instead we assume that one can construct a sequence of approximations ∗Department of Industrial and Enterprise Systems Engineering , University of Illinois at Urbana-Champaign . †Optimization and Decision Intelligence ( ODI ) Group , Department of Computer Science , ETH Zürich . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . of F ( x ) , denoted as { F l ( x ) } ∞l=0 . Unlike the original objective F ( x ) , unbiased function value and gradient estimators of each F l ( x ) ( correspondingly biased estimators of F ( x ) ) are accessible through some oracles . However , the cost for querying such oracles increases as the approximation accuracy increases . We measure the bias level using the approximation error either in term of the function values |F ( x ) − F l ( x ) | or in terms of the gradient ∥∇F l ( x ) −∇F ( x ) ∥22 . For simplicity , throughout the paper , we assume F and F l are smooth , namely , they have Lipschitz continuous gradients . More precisely , without loss of generality , we assume that ( i ) for any given x , the function approximation error satisfies |F l ( x ) − F ( x ) | = O ( 2−al ) for some constant a > 0 under the convex setting , or the gradient approximation error satisfies ∥∇F l ( x ) −∇F ( x ) ∥22 = O ( 2−al ) under the nonconvex setting ; ( ii ) for each l ∈ N , there exists a stochastic oracle ( SOl ) that returns an unbiased estimator H l ( x , ζl ) of ∇F l ( x ) − ∇F l−1 ( x ) given x with bounded variance : V ( H l ( x , ζl ) ) = O ( 2−bl ) for some constant b > 0 ; ( iii ) the cost to query the oracle SOl is Cl = O ( 2cl ) for some constant c > 0 . Assumption ( ii ) indicates that the variance of the stochastic estimator of the function difference decreases with l. Intuitively , as function F l is close to function F l−1 , one can construct highly correlated gradient estimators for ∇F l ( x ) and ∇F l−1 ( x ) such that the variance of their difference is small.As we will show later , these constants a , b , c play a crucial role in the complexity of the problems of our interest . 1.1 Motivational Examples . Conditional Stochastic Optimization ( CSO ) In CSO [ 19 ] , the objective function has the following form : F ( x ) : = Eξfξ ( Eη|ξgη ( x , ξ ) ) , where ξ and η are random vectors . In general , it is not easy to directly obtain unbiased gradient estimators from samples of P ( ξ ) and P ( η|ξ ) , because of the nonlinearity of fξ ( · ) . Instead , we can construct a sequence of approximation functions given by F l ( x ) = Eζl= [ ξ , η1 , ... , η2l ] [ fξ ( 1 2l ∑2l j=1 gηj ( x , ξ ) ) ] , l = 0 , 1 , . . . , where { ηj } 2 l j=1 are independent and identically distributed ( i.i.d . ) random vectors from P ( η|ξ ) . Unlike F ( x ) , unbiased estimators for ∇F l ( x ) can be easily obtained directly through samples . Under mild conditions , we can show that assumptions ( i ) - ( iii ) hold true ; in particular , we have a = 1 , b = 1 , c = 1 . Distributionally Robust Optimization ( DRO ) DRO minimizes the expected loss with respect to the worst distribution in an uncertainty set U ( P ) for a given distribution P . The objective of DRO has the form : minx∈X FP ( x ) : = supQ∈U ( P ) Eξ∼Qℓ ( x ; ξ ) . Because of the maximal operator over the uncertainty set , it is hard to obtain unbiased gradient estimator using samples from P . Instead one constructs a series of approximation functions given by F l ( x ) = ∑2l i=1 q ∗ i ℓ ( x ; ξi ) , where { ξi } 2 l i=1 are i.i.d . samples from P , Pn denotes the empirical distribution , and q∗ = { q∗i } 2 l i=1 attains the maximum of supq∈∆ 2l , q∈U ( Pn ) ∑2l i=1 qiℓ ( x ; ξi ) for a 2 l-dimensional simplex ∆2l . For CVaR and χ2-penalty DRO problems , Levy et al . [ 26 ] showed that assumptions ( i ) - ( iii ) hold true with a = 2 , b = 1 , c = 1 for CVaR DRO and a = 1 , b = 1 , c = 1 for χ2-penalized DRO . 1.2 Multilevel Monte Carlo ( MLMC ) Techniques . A naive approach to solve ( 1 ) is to perform SGD on the approximation function FL ( x ) with a level L such that the bias is small and within a targeted accuracy ϵ . We call such a framework L-SGD . This encapsulates several existing algorithms under different contexts , such as BSGD [ 20 ] in conditional stochastic optimization , multi-step MAML [ 21 ] in meta-learning , and biased gradient method [ 26 ] in distributionally robust optimization . Despite its simplicity , L-SGD generally fails to achieve the smallest total cost since it requires expensive stochastic oracles to ensure small bias . On the other hand , the multi-level Monte Carlo ( MLMC ) sampling technique , originally designed for stochastic simulation [ 15 ] , is amenable to obtain better gradient estimators by exploiting the bias-cost tradeoff . The idea is to query fewer oracles with higher cost and smaller bias , and more oracles with lower cost and larger bias ; thus , the cost can be effectively reduced . There exist several variations of MLMC with randomization , including the MLMC estimator with randomized truncation ( denoted as RT-MLMC ) [ 5 ] , the MLMC estimator with importance sampling ( denoted RU-MLMC ) , the Russian roulette estimator ( denoted as RR-MLMC ) that uses randomized telescope [ 22 ] . These MLMC techniques have been recently explored in different optimization contexts . For instance , Blanchet and Glynn [ 5 ] combined the MLMC idea with sample average approximation to address stochastic optimization . Dereich and Müller-Gronbach [ 10 ] integrated the vanilla MLMC ( denoted as V-MLMC ) gradient estimator with Robbins-Monro and Polyak-Ruppert stochastic approximation schemes , and analyzed their convergences . Recently , the RT-MLMC technique was applied to stochastic compositional optimization with strongly convex objectives in Blanchet et al . [ 4 ] and distributional robust optimization in Levy et al . [ 26 ] , Ghosh and Squillante [ 14 ] . Beatson and Adams [ 3 ] studied the asymptotic behaviors of stochastic gradient descent with RT-MLMC and RRMLMC estimators . Concurrently , Asi et al . [ 2 ] applied RT-MLMC to estimating proximal points and Moreau-Yoshida envelope gradient , minimization of maximal loss , and proximal gradient methods . 1.3 Our Contributions . In this paper , we provide a systematic comparison among these MLMC techniques when combined with stochastic gradient descent . Our primary focus is on the analysis of their total computation cost for achieving an ϵ-optimal solution for convex problems and an ϵ-stationary point for nonconvex problems , which is largely missing in the literature . Our contributions are three-fold : Non-asymptotic analysis of MLMC gradient methods . We analyze the non-asymptotic performances and computation complexities of various MLMC gradient methods under the general stochastic optimization framework in all regimes , e.g. , different convexity assumptions and different combinations of a , b , c. Our main results are summarized in Table 1 . Previous results either focus on specific applications [ 4 , 26 ] or only asymptotic behaviors [ 3 ] in restricted regimes . Theoretical comparisons and new insights . Our comparative study of different MLMC techniques yields several interesting findings : ( 1 ) When b > c , i.e. , the variance decays faster than the increase of the cost , all three randomized MLMC gradient methods and V-MLMC ( only when a ≥ min { b , c } ) nearly match the complexity of classical unbiased SGD , implying that the problem is no harder than classical stochastic optimization , regardless of the bias . ( 2 ) When b ≤ c , the unbiased MLMC constructions , RU-MLMC and RR-MLMC , are no longer applicable since either the expected periteration cost or the variance of the gradient estimator goes to infinity . In this case , RT-MLMC is the most favorable approach without requiring a mini-batch . ( 3 ) In all regimes with 0 < a < ∞ ( i.e. , bias exists ) , using the naive L-SGD is strictly sub-optimal in terms of the total cost , demonstrating the importance of the exploitation of the bias-variance-cost tradeoff . Improved sample complexity for conditional stochastic optimization . When applied to CSO problems ( resp . a = b = c = 1 ) , we show that the RT-MLMC and the vanilla MLMC gradient methods significantly improve over existing results reported in the literature . In particular , for nonconvex smooth CSO problems , RT-MLMC and vanilla MLMC achieve Õ ( ϵ−4 ) sample complexity , which is even better than the best-known Õ ( ϵ−5 ) achieved by the variance reduction technique [ 20 ] . 1.4 Preliminaries . A function f ( · ) : Rd → R is L-Lipschitz continuous if |f ( x ) − f ( y ) | ≤ L∥x − y∥2 holds for any x , y ∈ Rd . A function f ( · ) is S-smooth on Rd , if it is continuously differentiable on Rd and it holds that ∥∇f ( x ) − ∇f ( y ) ∥2 ≤ S∥x − y∥2 for any x , y ∈ Rd . For any x , y ∈ Rd , and a continuously differentiable function f , if it holds that f ( x ) − f ( y ) −∇f ( y ) ⊤ ( x− y ) ≥ µ2 ∥x− y∥ 2 2 , we say f is µ-strongly convex when µ > 0 and f is convex when µ = 0 . Let x∗ ∈ argminx F ( x ) . We say x is an ϵ-optimal solution if F ( x ) − F ( x∗ ) ≤ ϵ , and x is an ϵ-stationarity point of F if ∥∇F ( x ) ∥22 ≤ ϵ. Polyak-Łojasiewicz ( PL ) condition [ 24 ] is a generalization of strong convexity such that ∥∇F ( x ) ∥22 ≥ 2µ ( F ( x ) − F ( x∗ ) ) for µ > 0 . By convention we let ∇F−1 ( x ) = 0 . For simplicity , throughout the paper , we assume the desired accuracy ϵ > 0 is small enough so that ϵ−1 ≥ log ( ϵ−1 ) ≥ 1 . We use O ( · ) to hide constants that does not depend on the desired accuracy ϵ. Õ further hides the log ( ϵ−1 ) term .

Summary: This paper consider stochastic optimization when we only have access to a sequence of biased functions that converge to the true function. They apply various Multilevel Monte-Carlo methods (MLMC) to handle the bias-variance tradeoff. Convergence rate and computational complexity are computed for both convex and non-convex objective. They also demonstrate the superiority of MLMC gradient methods when applied to conditional stochastic optimization.