On the Bias-Variance-Cost Tradeoff of Stochastic Optimization
Authors: Yifan Hu
Xin Chen
Niao He
Venue: NeurIPS 2021 Poster (2021)
Keywords: stochastic optimization
multilevel Monte Carlo
bias variance and cost tradeoff
biased first-order methods

Abstract: We consider stochastic optimization when one only has access to biased stochastic oracles of the objective, and obtaining stochastic gradients with low biases comes at high costs. This setting captures a variety of optimization paradigms widely used in machine learning, such as conditional stochastic optimization, bilevel optimization, and distributionally robust optimization. We examine a family of multi-level Monte Carlo (MLMC) gradient methods that exploit a delicate trade-off among the bias, the variance, and the oracle cost. We provide a systematic study of their convergences and total computation complexities for strongly convex, convex, and nonconvex objectives, and demonstrate their superiority over the naive biased stochastic gradient method. Moreover, when applied to conditional stochastic optimization, the MLMC gradient methods significantly improve the best-known sample complexity in the literature. 

On the Bias- Vassaiaaanee-sst Tradeoff of Stochastic Optimization
Yifan Hu"
Xin Chen" UICC
UICC yifaahuu@@llinois..oo
xinchen@illinois.co edu
Niao He
ETH Zürich niao he@ singiinf... eth........hehh
Abstract
We consider stochastic optimization when one only has access biased stochastic oracles of the objective, and obtaining stochastic gradients with low biases comes high costs. This setting captures variety optimization paradigms widely used in machine learning, such as conditional stochastic optimization, bilevel optimization, and distributionally robust optimization. We examine a family of multi-level Monte Carlo (MLMC) gradient methods that exploit delicate trade-off among the bias, the variance, and the oracle cost. We provide a systematic study their convergences and total computation complexities for strongly convex, convex, and nonconvex objectives, and demonstrate their superiority over the naive biased stochastic gradient method. Moreover, when applied conditional stochastic optimization, the MLMC gradient methods significantly improve the best-known sample complexity in the literature.
Introduction
Solving modern machine learning tasks relies heavily on stochastic gradient descent (SGD) and many of its variants. Whilst the vanilla SGD depends crucially on unbiased gradient oracles, constructing unbiased gradient estimators learning applications. Such applications include GANs with regularization [25], distributionally robust optimization [26, 14], conditional stochastic optimization [20, 19], meta-learning [30, 11], min-max optimization [32, 33J, and bilevel optimization [16], just to name a few.
be very expensive
impossible for many emerging machine
natural biased gradient estimators. Existing work biased gradient methods [17, 1, 18, 7, 23] focuses on the iteration complexity given biased stochastic oracles while ignoring the possibility of constructing refined stochastic oracles with lower bias at a higher cost, e.g., more samples or more computation power. In this paper, we build unified framework study the fundamental tradeoff among the bias, the variance, and the cost.
As alternative, often resorts to
We study optimization problems of the general form
min F(x),
xER4
and assume that one does not have access to unbiased gradient estimator of F(x) via simple Monte Carlo sampling simulation. Instead assume that one construct sequence of approximations
DDpaatmeen of Industrial and Enterprise Systems Engineering, University of Illinois at Urbana-Chanl Chaaaannn Optimizzaiin and Decision Intelligence (ODI) Group, Department Computer Science, ETH Zürich.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

of F(x), denoted as (x)) Unlike the original objective F(r), unbiased function value and gradient estimators of each F= (correspondingly biased estimators of F(x)) are accessible through some oracles. However, the cost for querying such oracles increases as the approximation accuracy increases. We measure the bias level using the approximation error either in term of the function values Fxx) - (x)) or in terms of the gradient || (x)) (=))). For simplicity, throughout the paper, we assume F and F' are smooth, namely, they have Lipschitz continuous gradients.
More precisely, without loss of generality, we assume that (i) for any given 2, the function approxi- mation error satisfies [F" - F(x)] O(2 al) for some constant > 0 under the convex setting, or the gradient approximation error satisfies V - F = O22-%) under the nonconvex setting; (ii) for each E N. there exists a stochastic oracle (SO') that returns unbiased estimator H' (a,<') of (x) - VF (x) given r with bounded variance: (HH = 0(2 "") for some constant > 0; (iii) the cost to query the oracle SO' is C = ( for some constant c > 0. Assumption (ii) indicates that the variance of the stochastic estimator of the function difference decreases with I. Intuitively, as function F' is close to function F'-1, one can construct highly correlated gradient estimators for (1)()) and VF-+(x) such that the variance of their difference is small. As we will show later, these constants a, b, e play a crucial role in the complexity of the problems of our interest.
1.1 Motivational Examples
Conditional Stochastic Optimization (CSO) In CSO [19], the objective function has the following form: F(x) := (Effe(Eneenn {)), where and 7 random vectors. In general, it is not easy to directly obtain unbiased gradient estimators from samples of P()) and P(vlE), because of the nonlinearity of fe(.). Instead, we can construct a sequence of approximation functions given by [fe( Σラ= 9ng (x. [)) 1, where {n; {n,)3²= are independent and identically distributed (i.i.d)) random vectors from P(nlE). Unlike F(z), unbiased estimators for (z) can be easily obtained directly through samples. Under mild conditions, we can show that assumptions (i)-(ii) hold true; in particular, we have 1,6 c,
F'(x) = Re
Distributionally Robust Optimization (DRO) DRO minimizes the expected loss with respect to the worst distribution in an uncertainty set (P)) for a given distribution P. The objective of DRO has the form: e Fp(z)) := supQ Ee oz- E). Because of the maximal operator over the uncertainty set, it is hard to obtain unbiased gradient estimator using samples from P. Instead one constructs a series of approximation functions given by F'xx) = Σ 9, f(r; Es), where {sr)²' are i..d. samples from P, P,, denotes the empirical distribution, and q" {q; ) attains the maximum of Supqees.. (P.) Σ q:((x;; {,) for a 2 diddeennioonal simplex 4g. For CVaR and x2 pe-petyy DRO problems, Levy et al. [26] showed that assumptions (i)- hold true with 2, b 1, =1 for CVaR DRO and a = 1, = 1, e = 1 for x2 2ppenalized DRO.
1.2 Multilevel Monte Carlo (MLMC) Techniques
A naive approach to solve (1) is to perform SGD on the approximation function (z)) with a level L
such that the bias small and within a targeted accuracy e. We call such a framework L-SGD. This encapsulates several existing algorithms under different contexts, such as BSGD [20] in conditional stochastic optimization, multi-step MAML [21] in meta-lcarning, and biased gradient method [26] in distributionally robust optimization. Despite its simplicity, L-SGD generally fails to achieve the smallest total cost since it requires expensive stochastic oracles to ensure small bias.
On the other hand, the multi-level Monte Carlo (MLMC) sampling technique, originally designed for stochastic simulation [15], amenable to obtain better gradient estimators by exploiting the bias-cost tradeoff. The idea is to query fewer oracles with higher cost and smaller bias, and more oracles with lower cost and larger bias; thus, the cost can be effectively reduced. There exist several variations of MLMC with randomization, including the MLMC estimator with randomized truncation (denoted as RT-MI-M)) [5], the MLMC estimator with importance sampling (denoted RU-MLMC), the Russian roulette estimator (denoted as RR-MLMC) that uses randomized telescope [22].

---

Table I: Summary of the total costs of the algorithms for finding e-optimal solution for (strongly) convex F and c-stationary point for nonconvex smooth F. Here the decrease rate the bias, b is decrease rate of the variance, e is the increase rate of oracle cost, and O represents the order hiding logarithmic factors. N.A. stands for not applicable. Let nı c/x/- 22 = max + (c-b)/a, b)/a, c/a).
Algorithms
V-MLMC
RT-MLMC
RU-MLMC RR-MLMC
L-SGD
Convexity
Bias
O(c)
O(e)
O(e)
Mini-batch
Yes
No
No
No
Variance
(())
((1)
O(1)
O(1)
when <
Per-iter cost when c <
((+*t)
0(1)
O(1)
O(e7%/)
O()-11)
0(67)
O(e-))
Strongly Convex
Total Cost c b
Oct-1-11)
0(67)
O16-2-0/0
Convex
O(c-)
O(c
04e-4-20/0
Nonconvex
Olt 0(E-22)
Ole 1(011100-)/
N.A.
Ole O(e-1-0/0)
Strongly Convex
Total Cost if 2 h
O O11-1
Ole (c- (--00/)) 67/0)
N.A.
O(E-2-0/9
Convex
2222222)
0 b1/a)
N.A.
(ee-4-20/a)
Nonconvex
These MLMC techniques have been recently explored in different optimization contexts. For instance, Blanchet and Glynn [5] combined the MLMC idea with sample average approximation to address stochastic optimization. Dereich and Müiler-Gronbach [10] integrated the vanilla MLMC (denoted as V-MLMC) gradient estimator with Robbins-Monro and Polyak-Rupper stochastic approximation schemes, and analyzed their convergences. Recently, the RT-MLMC technique was applied to stochastic compositional optimization with strongly convex objectives in Blanchet et al. [4] and distributional robust optimization Levy et al. [26]. Ghosh and Squillante [14]. Beatson and Adams [3] studied the asymptotic behaviors of stochastic gradient descent with RT-MLMC and RR- MIMC estimators. Concurrently, Asi et al. [2] applied RT-MLMC to estimating proximal points and Moreau- Moreau-Yoshida envelope gradient, minimization of maximal loss, and proximal gradient methods.
1.3 Our Contributions
In this paper, we provide a systematic comparison among these MLMC techniques when combined with stochastic gradient descent. Our primary focus is the analysis of their total computation cost for achieving an c-optimal solution for convex problems and an e-stationary point for nonconvex problems, which is largely missing in the literature. Our contributions are three-fold:
Non-asymptotic analysis of MIMC gradient methods. We analyze the non-asymptotic per- formances and computation complexities of various MLMC gradient methods under the general stochastic optimization framework in all regimes, e.g., different convexity assumptions and different combinations of a, b, c. Our main results summarized in Table 1. Previous results either focus specific applications [4, 26] or only asymptotic behaviors [3] in restricted regimes.
Theoretical comparisons and new insights. Our comparative study of different MLMC techniques yields several interesting findings: (1) When > c, i.e., the variance decays faster than the increase of the cost, all three randomized MLMC gradient methods and V-MLMC (only when 2 min (b, c)) nearly match the complexity of classical unbiased SGD, implying that the problem is no harder than classical stochastic optimization, regardless of the bias. (2) When < the unbiased MLMC constructions, RU-MLMC and RR-MLMC, are no longer applicable since either the expected per- iteration cost the variance of the gradient estimator goes infinity. In this case, RT-MLMC is the most favorable approach without requiring a mini-batch (3) In all regimes with 0 < < oo (i.e., bias exists), using the naive L-SGD is strictly sub-optimal in terms of the total cost, demonstrating the importance of the exploitation of the bias-variance-con tradeoff.
Improved sample complexity for conditional stochastic optimization. When applied to CSO prob- lems (resp. = c 1), we show that the RT-MLMC and the vanilla MLMC gradient methods significantly improve over existing results reported in the literature. In particular, for nonconvex

---

smooth CSO problems, RT-MLMC and vanilla MLMC achieve ((==*) sample complexity, which is better than the best-known O(c-5) achieved by the variance reduction technique [20].
1.4 Preliminaries
A function ((-) Rd IR is L-Lipschitz continuous if f(y) < L||e y/12 holds for any 2, E Rd A function f(-) is S-smooth R if it is continuously differentiable R and it holds that f(x) ( s S1Iz yll for any z, R For any z, E R4, and continuously differentiable function f. if it holds that f(x) f() f (a -) $ || y
we say f is je-strongly convex when μ > 0 and f is convex when μ 0. Let x= E arg min, Fx). F(x*) S E, and z is an e-stationarity point of F if generalization of strong convexity F(x*)) for μ 0. By convention we let (f) = 0. For simplicity, throughout the paper, we assume the desired accuracy c > small enough so that -1 l)) 2 1. We use (-) to hide constants that does not depend on the desired accuracy e.
We say x is an e-optimal solution if Fz))
F>>)33) < 6. Polyak-t.jjsiewiin
(PI.) condition [24]
such that [>)(()) 2 2/x(F(2)
O further hides the log(e-1) term.
2
MLMC Gradient Methods
2.1 Assumptions
We first formally restate the problem setting and assumptions we use throughout the paper.
Assumption 2.1. We assume that there exist constants a, b, Mas Mb, Mes o > 0 such that for any x E Rdd and l E N, the following hold.
(a). The function value upproximation error is bounded: |F - F(x) s B M.2-ol
(b). There exits stochastic oracle, SO', that for given returns stochastic estimators h 5²) and Hx, ( such that
Ex,( <²) = (F) (() S0²,
(2)
EH' (2) = VF VF' V(H (²) S VV:= M12
(3)
(c). The cost to query SO' is bounded: C S M,24.
(d). The objective F and the approximation function F' are S
Assumptions 2.1 (a) and (c) imply that obtaining unbiased gradient estimator of more accurate approximation functions F (namely smaller bias from the true gradient) requires higher cost. The variance decay of the difference estimator H' (2x, (+) in Assumption 2b) is the key assumption for MLMC methods. Intuitively, since VF V VFl-1(x) (x) becomes very small for large I, constructing
highly correlated estimators of F and V F (z) using the same samples ç' will likely yield small variance of their difference.
In the nonconvex setting, since we care about e-stationarity point rather than function values, we use the following assumption approximation Assumption 2.2. There exists Ma > 0 such that for any E N, it holds
of the gradient.
| (+))) < Bi == M,2-al
Remark 2.1 (Relationship between the assumptions). Our main results in the strongly convex case can be easily extended to the PL condition [24]. Particularly, under PL condition, Assumption (a)(a) and Assumption 2.2 are exchangeable for biased gradient-based methods to achieve the same convergence and total cost results. We defer the related discussion Appendix D. In general, these two assumptions do not imply each other. Both of them hold for convex CSO problems, we will show in Section In general, these two assumptions reflect two different ways of constructing approximations. Constructing uniform function approximation first then taking gradient is more suitable for problem when the form of the true gradient is unknown, for instance, DRO problems. On the other hand, directly constructing gradient approximations is more suitable when one knows the form of the true gradient and particularly when the approximated gradient does not have corresponding objective function.

---

2.2 SGD with MLMC Estimators
In the sequel, we construct direct biased gradient estimator and four different multilevel Monte Carlo gradient estimators based on the oracle SO', These estimators will be fed in the generic stochastic gradient descent (SGD) algorithm given as follows.
Algorithm SGD Framework
Input: Number of iterations T, stepsizes {"r } initialization point 21-
{m)
1: for t= 1 to T do
2: Construct gradient estimator v(2)) of VF())).
3: Update T++1 = It. 7e((x))
4: end for
Output: {
Due to potential computing limitation, one might only have access to the stochastic oracles of the first L levels of approximation functions (=))))) for some L > O. Let (ni)f- denote some batch sizes.
L-SGD estimator: at a query point 2, query oracle so at point x for nt, times to obtain (x ()) and then construct
LGD (x) == 1 E h= h ct).
(4)
V-MLMC estimator: at a query point x, query the oracle So' at x for ny times to obtain {1l (z, C) for l = O, ..., L, and then construct
V.MLMC == El 1 EE Hx, C!).
(5)
RT-MLMC estimator: at query point z, first sample a random level according to the probability distribution Q = { such that P( = 1) = qr. where Ei qi = 1, then query only oracle SO' at once to obtains 11' (t).
RT-MMMM
(x) (2)
% H (*)
(6)
RU-MLMC estimator: at query point x, first sample random level according to the probability distribution Q2 = {9] 1 such that P(t = =) = Qt where Ei a = 1, then query only oracle SO' once to obtains H1' (xx(x,(')
& RU-MLMC (x) = " (I,
RR-MLMC estimator: at query point x, first sample a random level L according to the probability distribution Q2 {q - such that P(L = qi where E qi 1, then query oracle So' point x once for .., to obtain (H' (sr, (tr,()) Ho
M.RR-MLMC (x:) = E (M,
(8)
where Pt 1-E1 and ~1=0 0.
Note that the V-MLMC estimator requires querying the stochastic oracles for each level / = 0,..., L. with some fixed L. The other three MLMC estimators only query the stochastic oracle from a random level or random subset of levels, with appropriate reweighing. As immediate observation, summarize the bias, variance, and (expected) computation cost in Table 2 when æ E Rd is independent from a for A being L-SGD, V-MLMC, RT-MLMC, RU-MLMC, RR-MLMC. In the table, CAA denotes the expected computation cost for constructing estimator A. See Lemmas B.1 and B.2 for a detailed derivation.
Both RU-MLMC and RR-MLMC give unbiased gradient estimators of F, whereas the other three estimators are biased. As result, combining SGD with L-SGD estimator, V-MLMC estimator, and RT-MLMC estimator will only lead to a stationary point of the approximation function (x), assuming the variance of these estimators are bounded.
5

---

Table 2: Bias, Variance, and Cost of Gradient Estimators.
Estimators A
Expectation Ev (x))
Variance V(o
Cost Cike
L-SGD
VF (x)
n2 0²
nrC1
V-MLMC
VF (x)
Σ 21 Vi
RT-MLMC
Vr (1)
21=0% VV
RU-MLMIC
VF())
Ei 'Vi
Ein
RR-MIMC
Fx))
Σ qr.
(izoriVi
El= qr.
EEiz Ci)
Remark 2.2 (Conditions and computation costs of RU-MLMC and RR-MLMC). For both RU- MLMC and RR-MLMC, invoking the definitions of Cj and V₁, it can be easily seen that when S c, either the variance the expected per-iteration cost will be unbounded. Hence, these two unbiased MLMC estimators are only useful in the restrictive regime when b > c. In fact, when
c, by (11) CA = O(1), for A being Heer
setting qt x 2- (2-6)))//2 we immediately have that V(u
either RU-MLMC or RR-MLMC. As result, the convergence rates of SGD based RU-MLMC and RR-MLMC estimators can be directly obtained from the standard results for unbiased SGD; see, e.g. [6, 29, 13]. In particular, the iteration complexities (hence, expected total costs) of RU-MLMC or RR-MLMC to achieve an e-optimal solution are (1/cc) for strongly convex F and (1)2²) for general convex F. under appropriately chosen stepsize. For nonconvex smooth objective F, the expected total cost for achieving an e-stationary point such that [[[VF[<)||]] S i O(L/e*). For completeness, we provide detailed analysis in Appendix B about the bias, the variance, and the per-iteration cost and in Appendix C.4 about the total cost.
For ease of notation, let 24 denote the output of SGD using estimator A after T iterations, where A corresponds to L-SGD, V-MLMC, or RT-MLMC. Under Assumption 2.2 we have the decomposition of errors the (strongly) convex case:
E F F(24) F(x*)
(TA)
FL (at))
Frroe of Algorithm A
Approximation Errors
SGD on
F4(x4) FL (x*)
+
F4 (x")=-F(F
by by optimality at
Approximation Error Bi.
where x the minimizer of (z). The decomposition suggests that as long as the level L is large enough, e.g., L [a log 24) such that 2Br. S </22, and the number of iteration T is large enough such that expected error of SGD on p4 is smaller than e/2, the output of these methods will be e-optimal in the convex setting.
Remark 2.3 (Computation costs of L-SGD). For L-SGD, the total cost is given by T- CT) With the above choice of L, we have C₁, O(E O(e-//). From standard analysis of SGD (see Theorem A.1 in Appendix A), if the variance of the gradient estimator is ((), we have T = O mge when FL is strongly convex [6], and we have T = O mz.e²) when FL is convex and smooth. See Appendix C.1 for detailed results on L-SGD. Therefore, the total computation costs for L-SGD to achieve an e-optimal solution are O(e O 1-c/a) and O(E-2-e/q respectively in the (strongly) convex settings.
3 Total Cost Analysis of V-MLMC and RT-MLMC
This section mainly focuses on the total cost analysis of the two biased MLMC gradient methods, V-MLMC and RT-MLMC. Due the page limit, we only demonstrate the regime when b 2 c, i.e., the decrease rate of the variance is larger than or equal to the increase rate of the cost. Full results can be found in Appendix C. Let 24 be selected uniformly from a-1 ,..,,,.. RT-MLMC.
.... 24. for A being V-MLMC or
Theorem 3.1 (Expected Total cost of RT-MLMC). For RT-MLMC with a distribution Q {qi such that qi x 2- (bbo)))1/ when b 2 c, expected total cost satisfies the followings.
If Assumption 2.1 holds and FL is strongly convex, let a = 1/(1 + 1/2/122) and L = Then RT.MIMC is an c-optimal point of after T = O(c-l) iterations. x..
[1 log(4M,, The total cost of RT-MLMC satisfies C' = ((e-)).

---

If Assumption 2.1 holds and FL. is convex. let stepsizes "t (11/VT) and L 1/a [ogM ", Then ST &RT-MMMM is an e-optimal point of after T Ole iterations, and the total cost of RT-MLMC satisfies C' - C(=(())))
If Assumption (.)( and 2.2 hold, let stepsizes "t OLL/VT) and L [1/a a 9), Then IT ations. The total cost of RT-MLMC satisfies C O((A).
GRI-MLMC is an e-stationarity point of F after T O iter-
Remark 3.1 (Selection of {a oo)) We use the RT-MLMC. for convex objectives to illustrate how we select the gt- To minimize the total cost for achieving e-optimality, conceptually, we are solving the following optimization problem:
min (m)i
TRT TRT-MIMG MLMC. MLM.CRRTTTE CRM-MLMC
iter
L (a)
(10)
s.t.
-a=1, 20 0,..., L.
1=0
where the constraints are to make sure that Q= {1r f is distribution. Problem (10) is nonconvex and we cannot directly solve Later in Section C, we show that under properly selected stepsizes, holds that
T-R--MMM MLMC
Oier
oc V(URT-MLMC) CTRT--MMM MLMC
12
vi/a).
Optimizing the Lagrange function over {m } gives x (b+tc))1/2 Although they might not be the optimal solution of (10), we have shown that such choice of qi can greatly reduce the total cost significantly comparing to L-SGD.
Note that when b > c, the total cost of RT-MLMC gets rid of the dependence on log(E Similar RU-MLMC and RR-MLMC, RT-MLMC achieves an ((1) expected per-iteration cost and O(1) variance.
Theorem 3.2 (Total cost for V-MLMC). When 2 c, for V-MLMC with batch sizes ny [2 (h+0)1/2 Nor some N O, its total cost satisfies the followings.
If Assumption 2.1 holds and FL is strongly convex, let stepsizes "tt = 1/Sf. = V.MMMC is an e-optimal solution of F after and N = O) Then IT T- =-oole 1)) iterations. The total cost of V-MLMC satisfies C Ö(E max(11e/a)).
If d'Assumption / holds and FL is convex, let stepsizes "It r2)( I. 11/alog(4Maa (4Mac and N Ö(c-)). Then SV-MIMC an c-optimal solution of after T O(c-1) iterations. B)
The total cost of V-MLMC satisfies C = (00xx())x()
If Assumption 2. 2.I(b)c)(d) and 2.2 hold, let stepsizes 7 = 1/(Sp), L = [1[1//log(MMa and N = ((E=)) Then &Y-MLMC is e-stationarity point of F after T O(E -2) iterations. The total cost of V-MLMC satisfies
Ö(E 2-2 2-2max().e/a))
Remark 3.2. Optimizing the total cost with respect {ni) in the continuous space shows that
n4 2- (b++e)1/2 Since mini-batch size nı has to be integer numbers, set n4 [2 [2-(++e)5/2 for some constant N 0. Since 2- (b++))//2 N + 1, the per-iteration cost satisfies
L. Σ
N-MLMC Citer
2lc-b11/2
2el (ibbsc) O(N) + ((--)
(11)
where [1/a log (40 as specified in the theorem. The first term (11) represents the desired balanced per-iteration cost of MLMC, while the second term refers to the cost incurred by rounding to integer numbers. When e < and N O(1), under the specified second term dominates the per-iteration cost. At the same time, since the variance of V-MLMC is O(N-1)= = O(1), the iteration complexity and the per-iteration cost of V-MLMC are the same as L-SGD. Therefore V-MLMC cannot reduce the total cost comparing to L-SGD when N is small.
the theorem,
7

---

Using large N = OE reduces the variance to Oee) and V-MLMC behaves like gradient descent. Thus the iteration complexity of V-MLMC reduced by O ") in the strongly convex case and the convex case and the total cost becomes Oie max(L./a)) and O(c-1-max(1e (../()))))) ), respectively. In any case, V-MLMC is always O min (1.e/a)) better than L-GGD.
One may also use N k<2(b+o)L/2 with certain k so that 2- 2 is integer for any I E [Z]. Then there will be no extra cost incurred by rounding to integers, i.e., the second term on the right hand side of (11) becomes 0. In such cases, the total cost of V-MLMC matches that of RT-MLMC and is always ((== better than L-SGD.
The drawback is that V-MLMC has to use very large N and thus large mini-batch {mi for small I. Note that L-SGD cannot use large batch to reduce the total cost.
When b > c, the total cost of V-MLMC and nonconvex smooth case by setting N = O(E 1) or N O in the convex case or the nonconvex case, respectively. The logarithmic term in the strongly convex case comes from the iteration complexity. When b- = c, ((N) in (11) becomes O (N log(c=)) and when < c, O(N) in (11) becomes O(Ne (e-b))22g Note that the variance will increase by the same amount, which explains why the cost of the MLMC methods increases gets larger when b S c. Detailed discussions are in Appendix C.3.
get rid of the dependence on log(e-)) in the convex
Remark 3.3 (Applicability of MLMC gradient methods). When b c, RU-MLMC and MR-MLMC are the most favorable among the four MLMC methods since they have unbiased gradient estimators and do not need to specify L in advance. When b S c, only RT-MLMC and V-MLMC applicable. They introduce bias to avoid the high computation cost. RT-MLMC is the most versatile algorithm among the four MLMC methods since it has restrictions on a, b, c and does not need any mini- batch. It suits the situations when the per-iteration budget is limited, e.g., when one only has limited samples or computation power. V-MLMC uses large mini-batches that lead to very small variance. Therefore a constant stepsize (1) is sufficient to guarantee convergence.
Remark 3.4 (Unbiased gradient estimator is not necessarily the best). When b I c, RU-MLMC and RR-MLMC can still build up unbiased gradient estimators. However, the variance and the expected cost cannot stay finite the same time. As result, the expected total cost for achieving e-optimality becomes infinite. Such observation suggests that unbiased gradient estimator are not necessarily the best. It further strengthens the importance of balancing the bias, variance, and cost.
Applications to Conditional Stochastic Optimization
Conditional stochastic optimization has been utilized to model and solve many applications in machine learning, including the optimal control Anearly-solvable Markov decision process [8]. policy evaluation and control in reinforcement learning [8, 9, 28], meta-learning [20]. instrumental variable regression [27]. Previously, Hu et al. [20] considered biased SGD method and biased variance reduction methods using SPIDER [12]. Note that the biased SGD method they used can be treated as
a special case of the L-SGD in our paper.
In the following, we show that MLMC methods can significantly reduce the sample complexity and achieve better results than the biased variance reduction methods proposed in their paper. Recall the definition of CSO problem and its approximation function F= (xx)
2' Σ Eng (2,, (1)] j=1
min F(x) = (Bn)en F( = EgiE ZER
in Tng
fee (2)
(12)
where g¹ ~ P(E) and (r/)3² P(rla)) Denote çl (5', tu 13"1" and Imyong (zc, ct = (n2
ni + 11) Eyzn, In. (z.5) for some S ni S n2. For each query on a query point x, SO' returns (x' <²), 11 ()) such that
(((x,()= = Vin Te
1
1
(13)
H' <-) Va (12((1) fel (0 (z. <\))
(x.(,)
8

---

Table 3: Comparison of Algorithms on CSO
Convexity of F
V-MLMC
RT-MLMC
RU-MIMC/RR-MIME
[20]
Strongly Convex
O(e-))
O(E
N.A.
Convex
O
O
N.A.
O
Nonconvex Smooth
O(c=t)
Ō(e=*)
N.A.
0)e-9)
f becomes Oie
when further applying variance reduction.
We denote the cost as the total number of samples y/ Thus the cost query oracle SO' is Cj 2'. To apply the MLMC methods, it remains to verify that Assumption 2.1 and Assumption 2.2 hold. We follow Hu al. [20] and make some assumptions the CSO problem.
Assumption 4.1. We assume that the followings hold for CSO problems.
0 = supzer B (2, E0, E)112 <
+oc.
fe(-) is Sy Sy-smooth and L.p-lipschitz continuous;
E) is Sg -mmoooth and Lg -ipschitt continuous.
Proposition 4.1. Under Assumption 4.1, we have the following results:
The functions F and F (S + SyL for any L N. holds that (x)
12 and (F) - F(x) S Syo
The variance of the oracle so! satisfies
Var(h' (xx <²)) I T.3 and Var Var(H (ar, (x, <)) S S 1
Proposition 4.1 implies that under Assumption 4.1, CSO problems have the following parameter:
1, = 1.
Since 5 = c, RU-MLMC and RR-MLMC are not applicable. By Theorem 3.2 and Theorem 3.1, we have the total sample complexity of RT-MLMC and V-MLMC, which is summarized in Table 3.
Corollary 4.1. Under Assumption 4. for strongly convex CSO problem, the sample complexity of V-MLMC and RT-MLMC finding e-optimal solution is O) for convex CSO problem, the sample complexity of V-MLMC and RT-MLMC is Ō(E-)); for nonconvex smooth CSO problem, the sample complexity of V-MLMC and RT-MLMC for finding E-stationary point ())
Remark 4.1. The sample complexities of V-MLMC and RT-MLMC better than the lower bounds proved in Hu et al. [20]. The reason is that Hu et al. [20] assumed access to a stochastic gradient oracle (black-box oracle) that returns biased gradient of CSO with O(e) bias and variance ((1) at query cost of (c- ). In contrast, our paper shows that using RT-MLMC, one can construct a specific gradient estimator (white-box oracle) that has the same Occ) bias and O(1) variance, but the cost to query such oracle is of order (log(e-))) Since the iteration complexity (which only depends the bias and the variance) stays the same, the reduced cost query the oracle leads reduced total cost. Such observation further strengthens the importance of taking into account the oracle cost when studying the complexity of biased oracle models.
5 Numerical Experiments
In numerical experiments, we apply four MLMC gradient methods and LSGD on three problems, a synthetic problem with biased oracles, invariant least square, and invariant absolute regression.
The synthetic problem with biased oracles is of the form: mingeR4 F(x) = 1 |æ 2 12, where
= lim,, 2n and stochastic observation of can be obtained via some simulation process
with cost n. Thus the approximation function is F* (x) 211 Ez 1|2. Let 2,, (1 + bias/n)
where Σ, the output of simulation such that 2, N(zn,02Ta) Here bias hyperparameter that controls the bias.

---

277
LSG0) -- 5,44 0.1, Zreer 2.3043
27
KIMGNC -- C, LB 0.2, www. 0.040
5 MTT.NNC -1 0.000, Bror- 0.1020
01. 1E000 0.5. 2,0000, Cree 02123
10
101
16-
30
1920
6,550
MMUMC
0.01 Teee 16:150
57 MLMC
5.01,
14,4753
RI 0.6, LR 2,001 Dver 36.776
347
347
24
20
10
30000 20000
20293 200.0
10300
20003 28031
43300 30003
20031
Figure 1: Top row: synthetic problem. Bottom invariant least square. "LR": learning rate stepsizes. "Error": average error of last iterate. Each subfigure represents the best average last iterate emror method can achieve with truncation level L selected within {0, (O....; 103. geometry distribution with parameter within (0.1...., 0. and stepsizes. Each line in the subfigure represents an instance of the method with different random initialization. For the synthetic problem, we do not any mini-batches, while for invariant least square, we apply bati-b on randomized MLMC methods to control variance. The performance is measured by the average error of last iterate.
The invariant least square is special case of CSO problem with linear function and is of the form:
min (a. xER"
b).
where f' loss function, E (a, b) represents the feature label pair, 7 represents a perturbed feature. For f. consider absolute loss and square loss. For simplicity, let 1 N(a, o²La). We generate 2000 sample from ç. i... 2000 feature label pairs such that a follows some multivariate normal distribution and b
x with x" generated via multivariate normal distribution. The MLMC gradient constructed following (13).
Figure 1 summarizes the optimal parameter setup that achieves the smallest average error over certain number of trials under given total budget for quadratic program and invariant least square. More detailed setup and report on comparison and discussion on RR-MIMC and invariant absolute regression are given in Appendix F.
We observe that V-MLMC has the smallest variance suggested by the theory while the variances of RT-MLMC, RU-MLMC, and RR-MLMC are larger than the variance of LSGD due to the extra randomness caused by additional sampling. Thus we use mini-batch for randomized MLMC methods to obtain a stable training process while using larger stepsizes.
In practice, we observe that sometimes LSGD outperforms MLMC gradient methods, especially when we are only aiming for low accuracy solution. This may be caused by the high variance of MLMC methods and possibly large hidden constant factor in the complexity bounds. When using the same truncation level, we observe that RT-MLMC generally converges faster than LSGD in the early stages but suffers more from the high variance later stages.
We observe that biased MLMC methods generally outperform unbiased MLMC methods. It further justifies the importance of bias, variance, cost tradeoff as we mentioned in Remark 3.4.
6 Conclusion
This paper provides systematic study of the wlas-variance-cost tradeoff several MLMC gradient methods under a generic biased oracle model for stochastic optimization, shedding light on their superiority and limitations under different situations. For future work, combing variance reduction techniques with MLMC gradient estimators should further reduce the total costs in the nonconvex smooth setting. Second, the current MLMC techniques focus on constructing gradient estimators whose bias level maintains the same at each iteration. Another possible approach is to construct MLMC gradient estimators that adaptively reduce the bias. Lastly, it remains interesting to explore the power of MLMC gradient methods on other applications, for instance, quantization in distributed optimization and federated learning. It also remains open to characterize the conditions when MLMC can greatly outperform LSGD numerically in real-world applications.
10

---

Acknowledgments and Disclosure of Funding
We thank all reviewers and the area chair for the suggestions the paper. We thank the reviewer of our last NeurIPS submission [20] for pointing out the potential of multilevel Monte Carlo methods on conditional stochastic optimization. This work is supported by NSF CRII 1755829.
References
[1] Ahmad Ajallocian and Sebastian U Stich. On the convergence of sgd with biased gradients. arXiv preprint arXiv:2008. 000000, 2020.
[2] Hilal Asi, Yair Carmon, Arun Jambulapati, Yujia Jin, and Aaron Sidford. Stochastic bias-reduced gradient methods. Advances in Neural Information Processing Systems, 34, 2021.
[3] Alex Beatson and Ryan P. Adams. Efficient optimization of loops and limits with randomized telescoping sums. In 36th International Conference on Machine Learning, ICML 2019, pages 534-543, 2019.
[4] Jose Blanchet, Donald Goldfarb, Garud Iyengar, Fengpei Li, and Chaoxu Zhou. Unbiased simulation for optimizing stochastic function compositions. arXiv preprint 171.v:1711.07555 2017.
[5] Jose H Blanchet and Peter W Glynn. Unbiased monte carlo for optimization and functions of expectations via multi-level randomization. In 20/5 Winter Simulation Conference (WSC), pages 1656-3667. IEEE, 2015.
[6] Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. Siam Review, 60(2):223-31, 2018.
[7] Jie Chen and Ronny Luss. Stochastic gradient descent with biased but consistent gradient estimators. arXiv preprint 180iv: 18888817788,,,,, 2018.
[8] Bo Dai, Niao He, Yunpeng Pan, Byron Boots, and Le Song. Learning from conditional distributions via dual embeddings. In Artificial Intelligence and Statistics, pages 1458-1467, 2017.
[9] Bo Dai, Albert Shaw, Lihong Li, Lin Xiao, Niao He, Zhen Liu, Jianshu Chen, and Le Song. SBEED: Convergent reinforcement learning with nonlinear function approximation. In Interna- tional Conference on Machine Learning, pages 1125-1134, 2018.
[10] Steffen Dereich and Thomas Müllee-Gronnach. General multilevel adaptations for stochastic approximation algorithms of robbins-monro and polyak-ruppert type. Numerische Mathematik, 142(2):279-328, 2019.
[11] Alireza Fallah, Aryan Mokhtari, and Asuman E. Ozdaglar. On the convergence theory of bradient-based model-agnostic meta-learning algorithms. In International Conference Artificial Intelligence and Statistics, pages 1082-1092, 2019.
[12] Cong Fang, Chris Junchi Li, Zhouchen Lin, and Tong Zhang. Spider: Near-optimal non- convex optimization via stochastic path-integrated differential estimator. In Advances in Neural Information Processing Systems, pages 689-699, 2018.
[13] Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23(4):2341-2366 2013.
[14] Soumyadip Ghosh and Mark Squillante. Unbiased gradient estimation for distributionally robust learning. arXiv preprint arXiv:2012.12367, 2020.
[15] Michael B Giles. Multilevel monte carlo path simulation. Operations research, 56(3):607-617, 2008.
[16] Mingyi Hong, Hoi-To Wai, Zhaoran Wang, and Zhuoran Yang. A two-timescale framework for bilevel optimization: Complexity analysis and application to actor-critie, arXiv preprint arXiv:2007.05170, 2020.
11

---

[17] Bin Hu, Peter Seiler, and Laurent Lessard. Analysis of biased stochastic gradient descent using sequential semidefinite programs. Mathematical Programming. pages 1-26, 2020.
[18] Xiaowei Hu, LA Prashanth, András György, and Csaba Szepesvári. (bandit) convex optimization with biased noisy gradient oracles. In Artificial Intelligence and Statistics, pages $19-828, 2016.
[19] Yifan Hu, Xin Chen, and Niao He. Sample complexity of sample average approximation for conditional stochastic optimization. SIAM Journal on Optimization, 30(3):1133-2133, 2020. 2020.
[20] Yifan Hu, Siqi Zhang, Xin Chen, and Niao He. Biased stochastic first-order methods for conditional stochastic optimization and applications in meta learning. Advances in Neural Information Processing Systems, 33, 2020.
[21] Kaiyi Ji, Junjie Yang, and Yingbin Liang. Multi-step model-agnostic meta-learning: Conver- gence and improved algorithms. arXiv preprini arXiv:2002.07836, 2020.
[22] Herman Kahn. Use of different Monte Carlo sampling techniques. Rand Corporation, 1955.
[23] Belhal Karimi, Blazej Miasojedow, Eric Moulines, and Hoi-To Wai. Non-asymptotic analysis of biased stochastic approximation scheme. In Conference Learning Theory, pages 1944-1974, 2019.
[24] Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximal- gradient methods under the fojaakk-ojasiiwicz condition. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 795-81 811. Springer, 2016.
[25] Qi Lei, Jason Lee, Alex Dimakis, and Constantinos Daskalakis. Sgd learns mayer networks in wgans. In International Conference on Machine Learning, pages 5799-5808. PMLR, 2020.
[26] Daniel Levy, Yair Carmon, John C Duchi, and Aaron Sidford. Large-scale methods for distributionally robust optimization. Advances in Neural Information Processing Systems, 33, 2020.
[27] Krikamol Muandet, Arash Mehrjou, Si Kai Lee, and Anant Raj. Dual IV: A single stage instrumental variable regression. arXiv preprint arXiv:1910.12358, 2019.
[28] Ofir Nachum and Bo Dai. Reinforcement learning via fenchel- duality, arXiv preprint wwXiv:2001.0/866 2020.
[29] Arkadi Nemirovski, Anatoli Juditsky. Guanghui Lan, and Alexander Shapiro. Robust stochastic approximation approach to stochastic programming. SIAM Journal on optimization, 19(4): 1574-1609, 2009.
[30] Aravind Rajeswaran, Chelsea Finn, Sham Kakade, and Sergey Levine. Meta-learning with implicit gradients. arXiv preprint (9t1::9999.443339 2019.
[31] Alexander Rakhlin, Ohad Shamir, and Karthik Sridharan. Making gradient descent optimal for strongly convex stochastic optimization. In Proceedings of the 29th International Conference on Machine Learning, pages 1571 2012.
[32] Junchi Yang, Siqi Zhang, Negar Kiyavash, and Niao Hc. A catalyst framework for minimax optimization. Advances in Neural Information Processing Systems, 2020.
[33] Siqi Zhang, Junchi Yang, Cristóbal Guzmán, Negar Kiyavash, and Niao He. The complexity of nonconvex--ttronlyycoom minimax optimization. arXiv preprint arXiv:2 15888, 2021.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're rou're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "STTiWgOUG-g", "reviewer": "Reviewer_Jayx", "summary": "This paper provides a systematic study of the bias-variance-cost tradeoff for several MLMC gradient methods and L-SGD, under the assumption that (i) there exist a sequence of obj. approximations; (ii)  there exist stochastic oracles which generate unbiased estimates of the gradient for the obj. approximations, and unbiased estimates of the difference between function approximation gradients. They assume three parameters a, b, c which respectively describe the bias, variance of the stochastic oracle, and cost to query the oracle, and provide insights on which class of algorithms work better under different configurations of a, b, c. They also show that for conditional stochastic optimization problems, MLMC methods can significantly reduce the sample complexity.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall this paper is well-written, and provides useful insights for practitioners regarding which MLMC method to use for real-world problems. The illustration on the CSO problem also demonstrates the value of MLMC. One question I have is that, in practice, since F(x) is unknown, how do you get an estimate of a, in order to know which is the proper MLMC method? I assume b and c can be estimated in some way since they only involve the function approximations, not the underlying true function. \n\nOn Page 5 there seem to be some typos: L-SGD estimator: ... for n_L times to obtain {h^L}, not gradient of h? Same error also exists in Eq. (4); V-MLMC: ... to obtain {H}, not gradient of H.\n\nIt would also be helpful to provide some experimental comparisons of these MLMC methods, to (i) validate your theoretical results, and (ii) give convincing empirical evidence of the superiority and limitations of different methods under different situations. \n\nI personally expect to see the application of MLMC methods in min-max optimization, which you mentioned as part of the future directions. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No. I do not think it is related to any negative societal impact. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "ItP0_EoVjTx", "reviewer": "Reviewer_Q47Z", "summary": "This paper describes various MLMC debiasing schemes for computing stochastic gradient estimates, and applies them to conditional stochastic optimization problems. Here, unlike other more-standard problems in optimization it is impossible to compute true gradients of the objective function: the objective is an expected value over infinitely many points. However, as this paper observes, approximate minimizers to these problems may nevertheless be computed. The main ideas underlying this result are the use of sampling and a multi-level Monte Carlo (MLMC) to generate low-bias stochastic gradient estimates: these are then combined with stochastic gradient descent. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper studies a very well-studied problem and achieves improved theoretical guarantees for it. I found the analysis highly technical and difficult to follow, but the results seem correct to me. The analyses of four different schemes for bias reduction may be helpful for future work on related problems, although the reason for including these is unclear (as the parameters a,b,c in the paper have specified values in the application to CSO)\n\nHowever, the paper's result seems to essentially derive from combining SGD with standard MLMC-based gradient estimation schemes (for example, the ones in [4]). The utility of biased oracles in stochastic methods has been observed several times before. While some of the bias-reduced estimators given here are new, the bias and variance analyses of these estimators follows a standard template laid out in previous work [4,25] (indeed, the calculations of Lemmas 4 and 5 may be obtained by slight modifications of the calculations in Appendix C of [25]).\n\nIn addition, while the application to CSO is compelling I did not completely understand how the results compare to those of [19]. In particular, the paper claims a result that contradicts a lower bound of [19] and justifies this by stating that the oracle of [19] is more restricted than the setup considered here. Some more description of this behavior and a justification of the stronger access model presented here would be helpful . ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "-", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "IeRneWxKy_", "reviewer": "Reviewer_voJd", "summary": "This paper consider stochastic optimization when we only have access to a sequence of biased functions that converge to the true function. They apply various Multilevel Monte-Carlo methods (MLMC) to handle the bias-variance tradeoff. Convergence rate and computational complexity are computed for both convex and non-convex objective. They also demonstrate the superiority of MLMC gradient methods when applied to conditional stochastic optimization. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The setting that one only have access to a sequence of approximations to the objective function is well motivated with some important applications like conditional stochastic optimization and distributionally robust optimization, so the problem that this paper studied is of interest to the optimization community. The idea of introducing MLMC to gradient based optimization is not new [1], so the main contribution of this paper is not the algorithms but the theoretical analysis of convergence rate and computational complexity. They make systematical convergence and complexity analysis for both convex and non-convex objectives with clear dependency on the assumptions parameters, which is novel to this area to the best of my knowledge. Although I didn’t fully check the detail proofs, the paper is in-general well-written and reasonably easy to follow. Here are some points that I would like to hear the response from the authors:\n\n1. Numerical example is missing in Section 4. Sometimes the MLMC based method may involved a huge constant term in either the convergence rate or the complexity (especially the MLMC method that involve randomized level), so it would be better to have some numerical results to confirm the practical usefulness. \n\n2. Consider the conditional stochastic optimization problem, say min_x E[f(E[g(Y, x)|Z])]. With some smooth conditions on f and g, we are actually able to come up with unbiased estimator of E[f(E[g(Y, x)|Z])] or its gradient by randomized MLMC [2], because they are of the form h(E[W]). So perhaps we don’t really need to follow the current setting (sequence of approximated functions) to solve the problem? \n\n[1]: Jose Blanchet, Donald Goldfarb, Garud Iyengar, Fengpei Li, and Chaoxu Zhou. Unbiased simulation for optimizing stochastic function compositions. arXiv preprint arXiv:1711.07564, 2017.\n[2]: Jose H Blanchet and Peter W Glynn. Unbiased monte carlo for optimization and functions of expectations via multi-level randomization. In 2015 Winter Simulation Conference (WSC), pages 3656–3667. IEEE, 2015.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "950GwxKD8_", "reviewer": "Reviewer_FGif", "summary": "The paper studies a family of MLMC methods for gradient estimation and systematically analyses their performances under different stochastic optimization settings (strongly convex, convex, and nonconvex). In each setting, the order of the total cost is given in terms of the decay rates of the bias and variance, and the corresponding increasing rates of the oracle cost. It is shown in the CSO problems that some of the MLMC methods have lower computational costs compared to existing state-of-the-art (e.g., [19]).", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The paper provides a comprehensive and systematic study on the computational costs of MLMC gradient descent methods, which is not available in the literature as far as I know and presents a good contribution.\n\nQuality: The theoretical claims are well supported with detailed proofs.\n\nIt would be good if the authors can provide examples where b<c and b>c, corresponding to table 1. This would give stronger evidence that each regime is worth studying. For example, in Section 1.1, the DRO example, can you provide the values of a,b and c and which regimes it could correspond to in table 1? \n\nIt seems that there is a sharp phase transition for the performance of RU-MLMC and RR-MLMC from c<b to c>=b, as seen in table 1. When c<b, they are the best. But when c>=b, they are not applicable at all. This seems to suggest some important considerations on the constant in the big-O notation that depends on these parameters. Discussions on this issue would be helpful.\n\nClarity: The paper is well organized and the main results are presented clearly. But some arguments could be explained more carefully:\n\nIn Theorem 1, it is not very clear to me why n_L should be chosen in the stated way. In remark 4, it is explained that this is a result of “optimizing the total cost with respect to n_L in the continuous space”, but I think the author(s) could provide a more concrete argument. Similarly, there should be some discussion of the choice of q_L in Theorem 2.\n\nLine 240-241: If my understanding is correct, it seems that if 1>c/a, then V-MLMC is O(\\epsilon^{-c/a}) better instead of O(\\epsilon^{-1}) better then L-SGD (comparing line 240 and remark 3). \n\nTable 2: in the last column of the row for RU-MLMC, should n_l be q_l instead?\n\nSignificance: The study is theoretically significant and insightful in guiding the choice of MLMC gradient schemes. It also points to approaches that can outperform state-of-the-art methods.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please see the comments and concerns in the quality and clarity parts above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides a systematic description and analysis of multilevel Monte-Carlo (MLMC) gradient estimators for stochastic convex optimization and describes application of these estimators to conditional stochastic optimization. Despite some reservations regarding the novelty of some of the developments in the paper, the reviewers appreciated its generality and agreed it could be of value to the NeurIPS community. Furthermore, the reviewers agreed that the experimental results outlined in the author response would be a useful combination to the paper.

In light of this, I recommend that the paper be accepted to NeurIPS and encourage the authors to carefully revise the paper to address the issues brought up during the review. In particular, the authors should include a comprehensive report of their experimental results, consulting the NeurIPS submission checklist to ensure that all the pertinent information is included. Moreover, in cases where experiments indicate the negative finding, that MLMC does not outperform standard bias SGD in practice, I encourage the authors to clearly and transparently highlight this conclusion, as it is as valuable to researchers as a positive finding.

Decision: Accept (Poster)