Sharper Rates and Flexible Framework for Nonconvex SGD with Client and Data Sampling
Authors: Alexander Tyurin
Lukang Sun
Konstantin Pavlovich Burlachenko
Peter Richtárik
Venue: Accepted by TMLR ()

Abstract: We revisit the classical problem of finding an approximately stationary point of the average of $n$ smooth and possibly nonconvex functions. The optimal complexity of stochastic first-order methods in terms of the number of gradient evaluations of individual functions is $\mathcal{O}\left(n + n^{1/2}\varepsilon^{-1}\right)$, attained by the optimal SGD methods SPIDER (Fang et al., 2018) and PAGE (Li et al., 2021), for example, where $\varepsilon$ is the error tolerance. However, i) the big-$\mathcal{O}$ notation hides crucial dependencies on the smoothness constants associated with the functions, and ii) the rates and theory in these methods assume simplistic sampling mechanisms that do not offer any flexibility. In this work we remedy the situation. First, we generalize the PAGE (Li et al., 2021) algorithm so that it can provably work with virtually any (unbiased) sampling mechanism. This is particularly useful in federated learning, as it allows us to construct and better understand the impact of various combinations of client and data sampling strategies. Second, our analysis is sharper as we make explicit use of certain novel inequalities  that capture the intricate interplay between the smoothness constants and the sampling procedure. Indeed, our analysis is better even for the simple sampling procedure analyzed in the PAGE (Li et al., 2021) paper. However, this already improved bound can be further sharpened by a different sampling scheme which we propose. In summary, we provide the most general and most accurate analysis of optimal SGD in the smooth nonconvex regime. Finally, our theoretical findings are supposed with carefully designed experiments.

Published in Transactions on Machine Learning Research (10/2023)
Sharper Rates and Flexible Framework for Nonconvex SGD with Client and Data Sampling
Alexander Tyurin KAUST, Saudi Arabia
alexandetuurin@@@aalll
Lukang Sun KAUST, Saudi Arabia
nikung.sun@kkuus..cdu.uuk
Konstantin Burlachenko KAUST, Saudi Arabia
burkachenkokk@@mall.comm
Peter Richtárik KAUST, Saudi Arabia
richtorik@gmail.comm
Reviewed on OpenReview: https: // openreview. net/ net/fore forum? id- J6TWFFE
Abstract
We revisit the classical problem of finding an approximately stationary point of the average of n smooth and possibly nonconvex functions. The optimal complexity of stochastie first- order methods in terms of the number of gradient evaluations of individual functions is O (n22g=)=)1) attained by the optimal SGD methods SPIDER (Fang al., 2018) and PAGE (Li et al., 2021), for example, where & is the error tolerance. However, i) the big-O notation hides crucial dependencies on the smoothness constants associated with the functions, and ii) the rates and theory in these methods assume simplistic sampling mechanisms that do not offer any flexibility. In this work we remody the situation. First, we generalize the PAGE algorithm so that it can provably work with virtually any (unbiased) sampling mechanism. This is particularly useful in federated learning, better understand the impact of various combinations of client and data sampling strategies. Second, our analysis is sharper as we make explicit use of certain novel incqualitics that capture the intricate interplay between the smoothness constants and the sampling procedure. Indeed, our analysis better even for the simple sampling procedure analyzed in the PAGE paper. However, this already improved bound can be further sharpened by a different sampling scheme which we propose. In summary, we provide the most general and most accurate analysis of optimal SGD in the smooth nonconvex regime. Finally, theoretical findings supported by carefully designed experiments.
it allows us to construct and
1 Introduction
In this paper, consider the minimization of the average of smooth functions (1) in the nonconvex setting in the regime when the number of functions is very large. In this regime, calculation of the exact gradient can be infeasible and the classical gradient descent method (GD) (Nesterov, 2018) can not be applied. The structure of the problem generic, and such problems arise in many applications, including machine learning (Bishop & Nasrabadi, 2006) and computer vision (Goodfellow al., 2016). Problems of this form are the basis of empirical risk minimization (ERM), which is the prevalent paradigm for training supervised machine learning models.

---

Published in Transactions on Machine Learning Research (10/2023)
1.1 Finite-sum sum optimization in the smooth nonconvex regime
We consider the finite-sum optimization problem
{
= (()
min EE
(1)
where fi Id + R is a smooth (and possibly nonconvex) function for all i E [n] := nn. We are interested in randomized algorithms that find an e-stationary point of (1) by relurning a random point & such that E [()))) S e. The main efficiency metric of gradient- based algorithms for finding such a point is the (expocted) number of gradient evaluations fia we will refer to it as the complerity of an algorithm.
1.2 Related work
The area of algorithmic research devoted to designing methods for solving the ERM problem (1) in the smooth nonconvex regime is one of the most highly developed and most competitive in optimization.
The path to optimality. Let us provide lightning-speed overview of recent progress. The complexity of GD for solving (1) is O (n but this was subsequently improved by more elaborate stochastic methods, including SAGA, SVRG and SCSG (Defazio et al., 2014; Johnson & Zhang, 2013; Lei et al., 2017; Horváth & Richtárik, 2019), which enjoy the better complexity O (n + n2) Further progress was obtained by methods such as SNVRG and Geom-SARAH (Zhon et al., 2018; Horváth et al., 2020), improving the complexity to O (n + g1) Finally, the methods SPIDER, SpiderBoost, SARAH and PAGE (Fang et al., 2018; Wang et al., 2019; Nguyen et al., 2017; Li et al., 2021), among others, shaved-off certain logarithmic factors and obtained the optimal complexity O (n + n 1/222-1 )). matching lower bounds (Li et al., 2021).
Optimal, but hiding a sccret. While may look that this the end of the road, the starting point of our work is the observation that the big-O notation in the above resulls hides important and typically very large data dato-dependent constants. For instance, it is rarely noted that the more precise complexity of GD is O (L__n=-)) while the complexity of the optimal methods, for instance PAGE, is O (n + -==1/ where L- S L4 are different and often very large smoothness constants. Morcover, it is casy to generate examples of problems (see Example 1) in which the ratio L+/L- is as large as one desires.
Client and data sampling in federated learning. Furthermore, several modern applications, notably foderated learning (Koncčný et al., 2016; McMahan et al., 2017), depend on claborate client and data sampling mechanisms, which are not properly understood. However, optimal SGD methods were considered in combination with very simple mechanisms only, such sampling random function fa several times independently with replacement (Li et al., 2021). We thus believe that an in-depth study of sampling mechanisms for optimal methods will be of interest to the federated learning community. There exists prior work on analyzing non-optimal SGD variants with flexible mechanisms For example, using the "arbitrary sampling" paradigm, originally proposed by Richtárik & Takáč (2016) in the study of randomized coordinate descent methods, Horváth & Richtárik (2019) and Qian et al. (2021) analyzed SVRG, SAGA, and SARAH methods, and showed that it is possible to improve the dependence of these methods on the smoothness constants via carefully crafted sampling strategies. Further, Zhao & Zhang (2014) investigated the stratified sampling, but only provided the analysis for vanilla SGD, and in the convex case.
1.3 Summary of contributions
Specifically, in the original paper (Li et al., 2021), the optimal (w...t. and E) optimization method PAGE was analyzed with a simple uniform mini-batch sampling with replacement. We analyze PAGE with virtually any (unbiased) sampling mechanism using novel Definition 1. Morcover, we show that some samplings can improve the convergence rate O (n+ + L,n1//2-11) of PAGE. We oblain the new stale-of-the-arl convergence rates in the considered setting (see Table 2).
We improve the analysis of PAGE using a new quantity, the weighted Hessian Variance L (or L+,,,, that is well-defined if the functions f. are smooth. It is important that we get strietly better theoretical guarantees without any additional assumptions on the structure of (1). We show that, when the functions
2

---

Published in Transactions on Machine Learning Research (10/2023)
Table 1: The constants A, B, we and SS] that characterize the samplings in Definition
Sampling scheme
A
W1
B
S|
Reference
Uniform With Replacement
1/7
1/10
7/7
Sec. E.3
Importance
1/7
1/4
S T
Sec. E.3
Nice
(=
Unsform Without Replacement)
r(n-13
B-
1/1
7(n
T
Sec. E.1
21
Independent
Σ Ps
Sec. E.2
E T-PL
T-P4
Estended Nice
ST
Sec. E.4
T[n-1]
7(a-1)
We provide brief explanation of the samplings (more mathematically rigorous explanations provided the corresponding sections from "Reference"): 1) Unéform With Replacement is a sampling, where we take exactly 7 sampling, where we take exactly
clements from set with replacement. 2) Importance with replacement, and new sampled element is taken non-uniformly, 3) Nice is sampling, where we take exactly elements from set without replacement. 4) Independent is sampling, where we take each element indepeudently with predefined probability. 5) Entended Nice is sampling, where we repeat
jtl element 4 times before applying
Nice sampling.
Notation: n # of data points; = batch size; q = probability to sample eth data point in the multinomial distribution; pt probability sample th data point the bernoulli distribution; apply the Nice sampling.
of times repeat 2th data point before
fi are "similar" in the sense of the weighted Hessian Variance, PAGE enjoys faster convergence rates (sce Table 2). Also, unlike (Szlendak et al., 2022), we introduce weights w; that can play crucial role some samplings. Moreover, the experiments in Sec 5 agree with theoretical results.
Our framework is flexible and can be generalized to the composition of samplings. These samplings naturally emerge in federated learning (Konečný et al., 2016; McMahan et al., 2017), and we show that our framework can be helpful in the analysis of problems from federated learning.
2 Assumptions
We need the following standard assumptions from nonconvex optimization. Assumption 1. There erists f E R such that f(x) 2 f* for all æ E Rd.
Assumption 2. There erisls L_ 2 0 such that. [>)( fw)) S L- || vll for all x, Rdd
Assumption 3. For all i e [n], there exists constant L > 0 such that [VVf(()) - vf((i)l s L, || bll for all y E Rd.
Note that Assumption 2 follows from Assumption 3 with L. 1 E L.
2.1 Tight variance control of general sampling estimators
In Algorithm 1 (a generalization of PAGE), we form an estimator of the gradient V via subsampling. In our search for achieving the combined goal of providing a general (in terms of the range sampling techniques cater for) and refined (in terms of the sharpness of our results, even when compared known results using the same sampling technique) analysis of PAGE, we have identified several powerful tools, the first of which is Defiitionn 1. Let
}
S" = (w) (wI...., Wn) E R W1. Wn 2 0,
be the standard simpler and
probability space.
Definition (S(A, (w:)=)) and Weighted AB Inequality). Consider any random mapping S Ed x Rd Ω Rdd which will call "sampling", such that E [[((a),.... ((;:,,).......) 1 "E= 0; for all
3

---

Published in Transactions on Machine Learning Research (10/2023)
01,..., On RR, and there exist A, 2 0 and weights (w1, "Ul,,..,,0.... Wn) E S" such that
E ((1 km:w)
Σ aj
||s
ai
(2)
n
for all a1,...,, On E Rd. The collection of such samplings will denote as S(A, B, {ws
The main purpose of sampling S E S(A, B, 1 is to estimate the mean Σi= 0₁ using some random subsets (possibly containing some elements more than once) of the set (ai,..., Un J. We refer to Table 1, where we provide examples of samplings that satisfy this definition. One of the simplest examples is the sampling
For simplicity, we denote S ((a))=)) ((....... 0g)) =
S(ai....,, anj = Ox*
(3)
where x E [n] is uniformly sampled from the set [n] (the Uniform With Replacement sampling with T -1 from Table 1).
We now define the cardinality S of sampling S (w;
Definition 2 (Cardinality of Sampling). Let us take S E S(A, B, (wr )=))) and define the function
(a Qn) : Rd x ... x Rd + Rd such that Sw an) == S(a1..... aniw). If the function
(a an) depends only on a subset A(w) of the arguments (a1...., an), where A(w) : Ω - aa))
we define [S] == [[]
In terms of our problem (1); the cardinality of a sampling represents the number of the gradients Vf that a sampling calculates in each draw. Note that it can be easily calculated (see Table for details). In the case of (3), one can see that [S] 1.
Definition 1 is most closely related to two independent works: (Horváth & Richtárik, 2019) and (Szlendak et al., 2022). Horváth & Richtárik (2019) analyzed several non-optimal SGD methods for "arbitrary samplings"; these are random set-valued mappings S with values being the subsets of [n]. The distribution of a such a sampling is uniquely determined by assigning probabilities to all 2n subsets [n]. In particular, they consider the samplings S(a an) = 1/2 Eies a1 P4 == Prob(i E S), [S] = [S], some 2 O, wi,...,, Wn, 2 and ==0. Our new set S(A, B, (w; includes these samplings. Recently, Szlendak et al. (2022) studicd a similar inequality, but in the context of communicttion-fffcien distributed training with randomized gradient compression operators. They explicitly set out to study correlated compressors, and for this reason introduced the second term in the right hand side; i.e., they considered the possibility of B being nonzero, in this way they obtain tighter inequality, which they can use in their analysis. However, their incquality only involves uniform weights [w,}. See Table 1 for overview of several samplings and the values A, B and (w,) that belong to S(A, B, (wi) from Definition 1. Our Definition 1 offers the tightest known way to control of the variance of the sampling estimator, and our analysis can lake advanlage of it.
2.2 Sampling dependent smoothness constants
We now define two smoothness constants that depend on the weights {wi) of sampling S and on the functions ft.
Definition 3. Given a sampling S e B, (w)) let L be a constant for which
Σ
[v] - V < 12 || w112
(4)
for all x, E Rd. For (Wi...., w.) = (1/n, (/.,...., 1/n1, we define L+ == L. Definition Given a sampling S E (( A, B, (wy ( let I, be a constant for which
1 ΣΣ
[V/((>) V [v) v/(ii < 12 1 -113,
Var, y E RRd.
(5)
For (Wi,-.. (W),..., w.) 1//.....///), we define L T.L.w-
4

---

Published in Transactions on Machine Learning Research (10/2023)
Table 2: The complexity of methods and samplings from Table and Sec
Sampling scheme
Complexity
Comment
2is
SVRG method x L,
Independent (Horvåth & Richtárik,, 2019)
o +
v
Uniform With Replocement (Li al., 2021)
Uniform With Replacement (new)
n 0
Importance
n+
01
max
The functions fi are splitted into groups
Stratified
n
Notation: n # of data points; c = error tolerance; I.,, Lig Iu 14 and I, are smoothness constants such that L S Ein Li L L1 and L $ Liige = # of groups in the Stratified sampling.
One can interpret Definition 3 as weighted mean-squared smoothness property (Arjevani et al., 2019), and Definition 4 as weighted Hessian variance (Szlendak et al., 2022) that captures the similarity between the functions fi- The constants It,0 and Lt." help us better understand the structure of the optimization problem (1) in connection with a particular choice of sampling scheme. Note that Definitions 3, and Definition are connected with the weights {w:)/=1-
The next result states that L and L well-defined and finite provided the functions fi are L; smooth for all i E [n].
Theorem 5. If Assumption 3 holds, then L,,,0 -.,,000 = E - La satisfy Def. 3 and 4.
Indeed, from Assumption and the inequality ( 2 we get
1
- - < 1 Σ nuw.
13 1| y/1
nww
thus we can take 12, - - nw: I... The proof for I22,,0 is the same.
From the proof, that, the obtained result is not tight.
that ignore [V+ vf(i)] when estimating 12, However, by doing
2.3 Clarifications and Discussion
In the literature, including (Li et al., 2021), it is usually considered that one of the following inequalities holds. For all E [n],
[V)((>) V(((i)l < Ta || wll.
Va, E Rd
for some Le do (see Assumption 3). The inequality
Σ V(((() V((I) 12 ||
vll
Vx,y E Rd,
holds for some L4 < 0c. These incqualities are standard smoothness assumptions (the latter follows from the former for some L3 = L?). Under the last inequality, Li et al. (2021) get the theoretical bound O (n+ -11 that depend on L+.
i) We want to emphasize that we also only consider Assumption However, further we show that our thcoretical bounds depend on L220 (and I4,w) instcad of L,, and we provide the examples when L (and L+,w) are much smaller than L+ (see Table 2). We get this improvement without additional assumptions on the problem (1)-
5

---

Published in Transactions on Machine Learning Research (10/2023)
ii) The weights w; depend only choice of sampling, and play an essential role in the Importance sampling, for instance. In See 3.3, we show that if we take the Importance sampling with = T4/ 1+/E= 21=1 T
for all i E [n], then wy L./ Ei=1 L, for all i E [n], and it leads to the fact that L2, S L+,0 S // 2 Li.
One can see that a good choice of a sampling can lead to good weights wa such that the incqualities Definitions 3 and 4 are satisfied with small 1 and L4,w- It holds without additional assumptions about. our problem.
3 A General and Refined Theoretical Analysis of PAGE
In the Algorithm 1, provide the description of the PAGE method. The description of the method can be found in (Li et al., 2021, Sec. 2.1) The choice of PAGE as the basc method is driven by the simplicity of the proof in the original paper. However, we believe that other methods, including SPIDER and SARAH, can also admit samplings from Definition 1.
Algorithm PAGE
Input: initial point x. Rd, stepsize 7 0, probability p E (0, vf(x²)
gd
for t = 0.1..... do
4: x++1_ = x² - "g"
5:
Generate a random sampling function St
o' S (((((=+t)-VA((()))) - Vf(a+t)
with probability p
g++1
with probability 1 -
7: end for
In this section, provide theoretical results for Algorithm 1. Let us define Ao :- -(x0)-1' Theorom 6. Suppose that Assumptions 1, 2, 3 hold and the samplings S' E S(A, B, fwili
Then Algorithm 1 (PAGE) has the convergence rate E [[(((())) L2 (1-+v² 1- ((A-B)IZ, BL"
- 200.
where 7 S
To reach an E-stationary point, it is enough to do 2=220(1-- ((A- 12,,, + BI2
(6)
iterations of Algorithm To deduce the gradient complexity, we provide the following corollary. Corollary Suppose that the assumptions of Thm hold. Let us take p == 18 expected number of gradient colculations fs) of Algorithm 1 equals
[S]
Then the complerity (the
N = (n (ST)) nn سي (L- +
((A B) L2 BLL,.))
Ts
Proof. At each iteration, the expected # gradient calculations equals pa + (1- S) < 2|S]. expected number of gradient calculations equals 2 to get an e-stationary point.
Thus the total
The original result from (Li et al., 2021) states that the complexity of PAGE with batch size r is
L- (L-+ v" ( n Ao E -(LL+ T (11)) 2
AOVTLy
Norig
for all T E nI...
In the following examples, will that Cor 1 can guarantee better complexities than (7). These improvements we get using refined and tighter analysis of the gradient estimator g++l in See D. In particular, we provide a better upper bound to the variance E ||g ()(() which makes our dependencies on smoothness constants tighter.

---

Published in Transactions on Machine Learning Research (10/2023)
3.1 Uniform With Replacement sampling
Let us do samity check and subslitute the parameters of the sampling that the original paper uses. We take the Uniform With Replacement sampling (see Sec E.3) with batch size 7 (note that 2 [S]), A= = =B=1/ 1/r and wi = 1/2 for all E [x] (see Table 1) and get the complexity Nuniform = e (" - [T_ + "
for all E 21,2,... mJ. Next, let us fix T S max VoLL [ -- 1]. and, finally, obtain that Nuniform
θ (n+ Ao max(vab= Let us compare it with (7). With the same sampling, our analysis provides
better complexity; indeod, note that max L-} } S VnL+ (see Lemma 2 in Szlendak et al. (2022)). Moreover, Szlendak et al. (2022) provides examples of the optimization problems when L+ is small and L+ is large, so the difference can be arbitrary large.
Note that the original analysis of PAGE (Li et al., 2021) also based on Assumptions 1, 2 and 3, and we do not have additional assumptions on the structure of the problem (1). We have better theoretical guarantees in the same setting.
3.2 Nice sampling (= Uniform Without Replacement) sampling
Next, consider the Nice sampling (see Sec F.1) and get that the complexity Nuice ( [ +
(n- ( )+ Unlike the Uniform With Replacement sampling, for e small enough, θ n +++ ~ (L-
the Nice sampling recovers the complexity of GD for n, which equal (oonL_.
3.3 Importance sampling
Let us consider the Importance sampling (see Sec E.3) that justifies the introduction of the weights wy We get the complexity
vn l.t )) s 20(m) [-_ 1- Do7
(0(+ n+
Do maxt wn L--) )
N Nimportance = n E
7
for T max (
and the results from Sec F to obtain N Nimportace = θ n dova( Ein (See Sec G). In Example 2, we consider the optimization task where
{vr, ( 1]. Now, take qi wi E. by
Li
1 E Li is vn times smaller than L4. Thus the complexity Nimportance can be at least vn times smaller that the complexity Norig. However, we do not guarantee that Nimportance I Norig in general.
3.4 The power of B >>0 0
This is the first analysis of optimal SGD, which uses B > 0. In all previous examples, the constant == B > 0.
If A = B, then the complexity
N θ n
Au |S|
T__
+
n S
BT2
V
thus the complexity N does not depend on L w which greater of equal to L It can even be possible that L is close or equal to zero, while L? ar is arbitrary large. Unlike Definition 3 of L2 Definition of L2 2 captures the voriance of gradient differences. Imagine that (wi,..., wn) = (1/n,... 1/1) and fi = f for all i [n]. then the the left-hand side of (5) equals zero, and we can take L = 0. In general, we can not take L2 = 0 in (4) if fi f for all E [n).
3.5 Analysis under Pt Condition
The previous results can be extended the optimization problems that satisfy the Polyak-Lojasiewits condition. Under this assumption, Algorithm enjoys linear convergence rate.
7

---

Published in Transactions on Machine Learning Research (10/2023)
Assumption 4. There erists μ 0 such that the function f satisfy (Polgak-Lojastewics) PL-condition:
[V 2p(J(x) Vo E IR,
where f infzeR= f(x) -0.
Using Assumption 4, we improve the convergence rate of PAGE.
Theorem 7. Suppose that Assumptions 1, 3, 4 and the samplings S e S(A, B, {wi ) Then Algorithm (PAGE) has the convergence rate E [f(x²)] f (1 (1-74) Do. where 7 S
min { (1=+v2/ 2(1p)) ((A B) 12, 1))
2p
Algorithm PAGE with compositionn of samplings
1: Input: initial point _º E Rd, stepsize 7 > O, probability p E (0, 9d = vf(x²)
for = 1,.....77 do
r+1 x² '
with probability
4: c+1
10 with probability 1-p
5:
if ct11 = then
6:
g++1= (V((x(+
FL Interpretation: With the small probability, calculate the full gradients V on the nodes and collect them" */
else
Generate samplings S for all i E [n] h5*1 8 ((V vis("")) for all i [n]
1 FL Interpretation: Calculate the mini-batches h++1 on the nodes */
Generate sampling St and set g++l= = =g + S' ((h) ((h(-1}31) [;=1)
10:
/ FL Interpretation: Collect hi+1 only from the sampled nodes */
11:
end
12: end for
4 Composition of Samplings: Application to Federated Learning
In Sec 3, we analyze the PAGE method with samplings that belong to Definition 1. Now, let assume that the functions fi have the finite-sum form, i.e, fa(x) En fiz(x), thus we optimization problem
min
- mi (1((x)
ZER
Another way to get the problem is to assume that we split the functions fi into groups of sizes m,, Let us consider (8) instead of (1).
The problem (8) occurs many applications, including distributed optimization and federated learning (Koncčný al., 2016; McMahan al., 2017). In fodcrated learning, many devices and machines (nodes) store local datasets that they do not share with other nodes. The local datasets are represented by functions fi and all nodes solve the common optimization problem (8). Due to privacy reasons and communication bottlenecks (Kairouz et al., 2021), it is infeasible to store and compute the functions fi locally in one machine.
In general, when we solve (1) in one machine, we have the freedom of choosing a sampling S for the functions fir which we have shown in Sec 3. However, in federated learning, a sampling of nodes or the functions fi is dictated by hardware limits or network quality (Kairouz et al., 2021). Still, each 7th node can choose sampling S, sample the functions fiy. As a result, have composition of the sampling S and the samplings S; (see A Algorithm 2).
8

---

Published in Transactions on Machine Learning Research (10/2023)
Assumption 5. For all j E [m:] E [n]. there exists Lipschitz constant Lı such that [() Vir(W>ill Laj || vll for all Rd.
We now introduce the counterpart of Definitions 3 and 4. Definition 8. For all E [n] and any sampling S; E 3(A,, B., (w define constant L,,, such that
Vfz
fu)(l) Li,,,u || yll
for all y E Rd. Definition 9. For all i E [n]] and any sampling S; S(A,, B, (wm)))=)) define constant Litt,w such that
my
[V)(())
-V -
VA(D)) 12 || - yl
m.
MgWii
for all r, y E Rd.
Let us provide the counterpart of Thm for Algorithm 2.
Theorem 10. Suppose that Assumptions 1, 2, 3, 5 hold and the samplings S E S(A, B, (wi)i=) and the samplings S E S(Ar, Br, {urg (wng)f=)1 for all [n]. Moreover, B B 1. Then Algorithm 2 has the convergence
rate ||V|<>>1
20o 7
where 7 S (L- + Le (C, +n))
A (1-B)
((A: B.)bi.... B,L.,t.....
nay
and
Cg = (A- B)L BL
The obtained theorem provides general framework that helps analyze the convergence rates of the composition of samplings that belong to Definition 1. We discuss the obtained result in different contexts.
4.1 Federated Learning
For simplicity, let assume that the samplings St and S are Uniform With Replacement samplings with batch sizes T and t; for all i E accordingly, then to get e-stationary point, it is enough to do
T θ ( Ao L-
pr Em F. 12 +13) )13))
V
iterations. Note that 2 θ Ao (L- V LPL3) for all Ti 2 1 for all i i [n]. It means that after some point, there is no benefit in increasing batch sizes т, In order to balance 1 Er I and L2, one can take 7; θ (Li=/11). The constant 12 captures the infra-varianc inside ith node, while L22 captures the inter-variance between nodes. If the intro-variance small with respect to the inter-variance, then our theory suggests taking small batch sizes and vice versa.
p7
4.2 Stratified sampling
Let us provide another example that is closely related to (Zhao & Zhang, 2014). Let us consider (1) and use variation of the Stratified sampling (Zhao & Zhang, 2014): split the functions f. into g "/m groups, where m is the number of functions in each group. Thus we get the problem (8) with f(x) a > En f1z(z). Let us assume that we always sample all groups, thus with batch sizes for all [n]. Applying Thm 10, we get the convergence rate
0, and the sampling S are Nice samplings
T groupp := 0
(
-20
__ +
1- pgTi
1
13.11 ))

---

Published in Transactions on Machine Learning Research (10/2023)
At each iteration, the algorithm calculates gr gradients, thus we should take p g tm to get the complexity Ngroup = + (n + ggm) = θ (n + du (InT + vin Ei= T )+ Let us take TI S
12 to obtain the complexity Ngroup =
@ n Ao max { v Σ. 4 1 Com-
max
9h-
paring the complexity Ngroup with the complexity Nuniform from Sec 3, one can see that if split the functions Ji in a "right way", such that L is small for i E [n]] (see Example 3), then we can get at least vn/vg times improvement with the Stratified sampling. However, we can be "unlucky" with a splitting of the functions; thus, we do not guarantee improvement in general.
5 Experiments
0.98c
0.96;
0.5; L. 0.06;
0.97;
0.90: L. 1.36
Leive beleerel Beah like
Univee dio Resh like
Brat
Nurbba krrdbro 3200
16-1
par
30-1
10-
30-
10-
30"1
Talan 10le refgradies calzzittrre
10-11
1000 1AMM calddaine
13.
Figure 1: Comparison of samplings and methods quadratic optimization tasks with various L+
We now provide experiments' with synthetic quadratic optimization tasks, where the functions fú, in general, are nonconvex quadratic functions. Note that our goal here is check whether the dependencies that our theory predicts correct for the problem (1). The procodures that generate synthotic quadratic optimization lasks give control over the choice of smoothness constants. All parameters, including the step sizes, are chosen as suggested by the corresponding theory. In the plots, we represent the relation between the norm of gradients and the number of gradient calculations fia-
5.1 Quadratic optimization tasks with various Hessian Variances L+
Using Algorithm 3 (see Appendix), generated various quadratic optimization tasks with different smoothness constants L1 E [0, 1.0] and fixed L_ x 1.0 (see Fig. 1). We choose d = 10, 1000, regularization A x= 0.001, and the noise scale E f0,0.1, 0.5, 1}. According to Sec 3 and Table 2, the gradient complexity of original PAGE mothod ("Vanilla PAGE" in Fig. 1) is proportional to L+. While the gradient complexity of the new analysis with the Uniform With Replacement sampling (*Uniform With Replacement" in Fig. 1) is proportional L1, which is always less or equal L.. In Fig. 1, one can see that the smaller L1 with respect L4, the better the performance of "Uniform With Replacement." Moreover, we provide experiments with for all E [n]. This sampling has the the Importance sampling ("Importance" in Fig. 1) with qi -"" L.
best performance in all regimes.
5.2 Quadratic optimization tasks with various local Lipschitz constatns L;
Using Algorithm 4 (see Appendix), we synthesized various quadratic optimization tasks with different. smoothness constants Lg (see Fig. 2). We choose d = 10, n = 1000, the regularization X = 0.001, and the noise scale s (0,0.1, 0.5, 10.03. We generated tasks in such way that the difference between max; L, and min; L4 increases. First, one can see that the Uniform With Replacement sampling with the new analysis ("Uniform With Replacement" in Fig. 2) has better performance even in the cases of significant variations of L, Next, we see the stability of the Importance sampling ("Importance" in Fig. 2) with respect to this variations.
"Our code: http:://tthib.c///sstrrresssssse iag.rraeeeeeraeeeiioioooooooiaao
10

---

Published in Transactions on Machine Learning Research (10/2023)
nin 0.98: max 0.98
min 0.98: maxi, 1.62
minl 0.98; maxk 4.34
mini.
6.95; maxi 64.47
rebh Pratenian Brah Littn
OWi sot Intve Balleaee Brah lilk
Jelin hestet dracs thes
rock caee
Narbea eftandrn 2006 greft
13
13
10-
15-1
10-
1906 TASA 2000 SAAK Vole
15650 TASS 2005 THESE
TODIS T4RSS TOAS "AAE
Figure 2: Comparison of samplings and methods on quadratic optimization tasks with various T.-.
mink:
28.75
2500100864.0
10-3
Replacement Importance
10'
Uniform Replacemmnnt Importance
10-3
10-
500
102
10
10a
10
10a
10
0.0
0.2
0.4
0.6
1.0
10
0.5
0.2
0.6
0.8
1.
calcuiation
w8a dataset
australion dataset
Figure 3: Comparison of samplings on nonconvex machine learning tasks with LIBSVM datasets.
5.3
Nonconvex classification problem with LIBSVM datasets
We now solve nonconvex machine learning tasks and compare samplings on LIBSVM datasets (Chang & Lin, 2011) (sce details in See A.2). As in previous sections, PAGE with the Importance sampling performs better in Fig. 3, especially in the australian dataset where the variation of L, is large.
References
Yossi Arjevani, Yair Carmon, John C Duchi, Dylan J Foster, Nathan Srebro, and Blake Woodworth. Lower bounds for non-convex stochastic optimization. arXiv preprint orXiv:1912.02365 2019.
Christopher M Bishop and Nasser M Nasrabadi. Puttern recognition and machine learning, volume 4. Springer, 2006.
Konstantin Burlachenko, Samuel Horvéth, and Peter Richtárik. Fl pytorch: optimization research simulator for foderated learning. In Proceedings of the 2nd ACM International Workshop on Distributed Machine Learning, pp. 1-7, 2021.
Chil-Chung Chang and Chil--enn Lin. LIBSVM: library for support vector machines. ACM Transactions Intelligent Systems and Technology (TIST), 2(3):1-27, 2011.
Aaron Delazio, Francis Bach, and Simon Lacoste-Julien. SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems, 27, 2014.
Cong Fang, Chris Junchi Li, Zhouchen Lin, and Tong Zhang. SPIDER: Near-optimal non-convex optimization via stochastic path integrated differential estimator. In NewrIPS Information Processing Systems, 2018
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
Samuel Horváth and Peter Richtárik. Nonconvex variance reduced optimization with arbitrary sampling. In International Conference on Machine Learning, pp. PMLR, 2019.
11

---

Published in Transactions on Machine Learning Research (10/2023)
Samuel Horváth, Lihua Lei, Peter Richtárik, and Michael I Jordan. Adaptivity of stochastic gradient methods for nonconvex optimization. arXiv preprinl wrXiv22002.05359 2020.
Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. Advances in Newral Information Processing Systems, 26, 2013.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trends@ in Machine Learning. 14(1-2):1-210, 2021.
Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. ar preprinl 2016.
Lihua Lei, Cheng Ju, Jianbo Chen, and Michael T Jordan. Non-convex finite-sum optimization via SCSG methods. Advances in Neural Information Processing Systems, 30, 2017.
Zhize Li, Hongyan Bao, Xiangliang Zhang, and Peter Richtárik. PAGE: A simple and optimal probabilistic gradient estimator for nonconvex optimization. In International Conference on Machine Learning, pp- 6286 6295. PMLR, 2021.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication- efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, pp. 1273-1282. PMLR, 2017.
Yurii Nesterov. Lectures on Convex Optimization, volume 137. Springer, 2018
Lam Nguyen, Jie Liu, Katya Scheinberg, and Martin Takáč. SARAH: A novel method for machine learning problems using stochastic recursive gradient. In The 34th International Conference on Machine Learning, 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performanee deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Xun Qian, Zheng Qu, and Peter Richtárik. L-SVRG and L-Katyusha with arbitrary sampling. Journal of Machine Learning Research, 22:1-49, 2021.
Peter Richtárik and Martin Takáë. Parallel coordinate descent methods for big data optimization. Mathemationl Programming, 156(1):433 484, 2016.
Rafal Szlendak, Alexander Tyurin, and Peter Richtárik. Permutation compressors for provably faster distributed nonconvex optimization. International Conference on Learning Representations (ICLR), 2022.
Zhe Wang, Kaiyi Ji, Yi Zhou, Yingbin Liang, and Vahid Tarokh. SpiderBoost and momentum: Faster variance reduction algorithms. Advances in Newral Information Processing Systems, 32, 2019.
Peilin Zhao and Tong Zhang, Accelerating minibatch stochastic gradient descent using stratified sampling. arxiv preprinl Xiv: 140.:400.300 2014.
Dongruo Zhou, Pan Xu, and Quanquan Gu. Stochastic nested variance reduction for nonconvex optimization.
A duances in Neural Information Processing Systems, 31, 2018.
12

---

Published in Transactions on Machine Learning Research (10/2023)
Contents
1 Introduction
1
1.1
Finite-sum optimization in the smooth nonconvex regime
2
1.2 Related work
2
1.3
Summary of contributions
Assumptions
3
2.1
Tight variance control of general sampling estimators
3
2.2
dampling- smoothness constants
4
2.3
Clarifications and Discussion
5
3
A General and Refined Theoretical Analysis of PAGE
6
3.1 Uniform With Replacement sampling
7
3.2 Nice sampling (= Uniform Without Replacement) sampling
3.3 Importance sampling
3.4 The power of B >
3.5 Analysis under PL Condition
7
4
Composition of Samplings: Application to Federated Learning
8
4.1 Federated Learning.
9
4.2 Stratified sampling
Experiments
5.1
Quadratic optimization tasks with various Hessian Variances L
5.2
Quadratic optimization tasks with various local Lipschitz constatns L;
10
5.3
Nonconvex classification problem with LIBSVM datasets
11
A Extra Experiments and Details
15
A.1 Quadratic optimization tasks with various batch sizes.
15
A.2 Details on experiments with LIBSVM datasets
A.3 Federated learning experiments with LIBSVM dataset
A.4 Computing environment
17
B Auxiliary fact: Variance Decomposition
18
C Examples of Optimization Problems
18
D Missing Proofs
19
E Derivations of the Parameters for the Samplings
21
13

---

Published in Transactions on Machine Learning Research (10/2023)
E.1 Nice sampling
21
E.2 Independent sampling
................................
22
E.3 Importance and Uniform With Replacement sampling.
................................
23
E.4 Extended Nice sampling
24
F The Optimal Choice of w;
25
G The Complexity of Algorithm 1 with the Importance sampling
25
H Missing Proofs: The Composition of Samplings
25
Artificial Quadratic Optimization Tasks
29
14

---

Published in Transactions on Machine Learning Research (10/2023)
A Extra Experiments and Details
A.1 Quadratic optimization tasks with various batch sizes.
In this section, we consider the same sctup as in See 5.1. In Figure 4, we fix L=, and show that the Importance sampling has better convergence rales with different batch sizes. Note that with large batches, the competitors reduce to the GD method, and the difference not significant.
L 0.1; 0.96: L. 0.97
10a
Number of functions 1000
10-1
10
Step:
10
PAGE Stop:
PAGE 1.0
Varil PAGE Datch: Step:
15-4
Importa
Step
Importa
Step:
Importanee Banch: 105-0he
Importansn now, Bareno 1000 Step:
20
16"
10"
gradient calculations (log scale)
Figure 4: Comparison of samplings and methods with various batch sizes.
A.2 Details on experiments with LIBSVM datasets
We compare the samplings on practical machine learnings with LIBSVM datasets (Chang & Lin, 2011) (under the 3-clause BSD license). Parameters of Algorithm 1 are chosen as suggested in Thm 6 and Cor 1. We take the parameters for Uniform With Replacement and Importance samplings from Table 1 with q: = Σ
We consider the logistic regression task with nonconvex regularization (Wang et al., 2019)
f(11, = 1 "
exp (a, xy.) log EEe(112) exp (a xny)
{xy) fayli
Σ
+
+Y Σ
Σ (E(2) A=1
min #1 #RR
where (.)x is an indexing operation, a, E Rdd the feature of 1th sample, y: E [1,2] is the label of 4th sample, constant X 0.001. We fix batch size J 1 and take w8a dataset (dimension 300, number of samples 49,749) and australian dataset (dimension = 14, number of samples 690) from LIBSVM.
0020
2030
500
30'
30
12'
10*
307
107
30
Figure 5: The distribution of Lipschitz constants L;
15

---

Published in Transactions on Machine Learning Research (10/2023)
107
Unilerm With Reglacoment [1p
10a
Unitomm Replacement [rann
Unitorm With Replacement [7)
Unitom With Replacement Tears 77]
10
Uniferm With Reglacoment Roppemortttt
Uniform Replacement Frann
Unitorm With Replacement 35)
Unitom With Replacement TTeas 35]
19904
20
Uniform With Replacement (1))
10
Uriffnn Replacement Teane
Importance 11
importance [Taee #11
-00
10
importance [qaee-])
Importance T 211
Importance [taae=2:]
10+
10-
Importatice Frans
Importance [66 56]
Importance [5pwe
10c
102
0.0
0.2
0.4
0.6
0.5
1.0
0.0
0.2
0.4
0.6
0.8
1.0
grastent calculations
gradient calculations
(a) Tolienta
1, #elients n
10
(b) Tolienata = 3, #clients n 10
105
Uniffrrm With Replacement [1]
10a
Uniform With Replacement [qaaec]]]
Uniform With Replacement [1gara-]]
Uniform With Replacement [sson=
10
Unitorm With Reglacement 22]
10a
Uniform With Replacement [aaaee
Uniform With Replacement [15)
Uniform With Replacement [sson 35]
1904700
Unitorm With Replacement (5))
10
Uniform With Replacement To 56]
Importance 11 11
Importance (1
20'
Importance Tooss 7)
100
Importance [Tpeeu
Impartance Too 211
Importance (Tyen=21)
10a
10:
Importanic [Tp))
Importance 300 56)
importance (7pen366)
100
102
0.0
0.2
0.4
0.5
0.8
1.0
6.0
62
0.4
0.6
0.8
of grasient calculations
gradient calculations
(c) Totients 6, #clients n = 10
(d) Talsenaa = 9, #clients n = 10
Figure 6: Comparison of methods on australian dataset from LIBSVM
For the logistic regression, the Lipschitz constants L, can be estimated. The distribution of Lipschitz constants
I across datapoints for that two datasets is presented in Fig. 5. We use Thm 5 to obtain I22,. and 122
A.3 Federated learning experiments with LIBSVM dataset
In this experiment?, we compare the Uniform With Replacement sampling and the Importance sampling on the logistic regression task from Sec. A.2 in a distributed environment. The training of the models is carried on australian dataset from LIBSVM. The dataset is reshuffled with uniform distribution, and then it is split across n 10 clients. In all experiments, we use Algorithm 2 with theoretical stepsizes according to Theorem 10. We take the paramcters of the Uniform With Replacement and Importance samplings from Table 1 with
0i
According to Algorithm 2, we have the samplings St that sample clients, and the samplings S that sample data from the local datasets of clients. Algorithm allows mixed sampling strategies that satisfy Assumption For simplicity, we consider that the samplings St and S are of the same type.
For the logistic regression, the Lipschitz constants L, and L,j of the gradients of functions fi(x)) and fiy (x) can be estimated. As in Sec A.2, we use Thm 5 to obtain the constants 1 L2 L² and L3 - The results of experiments are provided in Fig. 6. We denote by Tpointe the batch size of the samplings S for all E [m], and by Telients the batch size of the sampling St. The number of gradient calculations in Fig. stands for the total number of gradient calculations in all clients.
We demonstrate results for different values of the batch sizes Tolients and Tpoints. As in previous experiments, the Importance sampling has better empirical performance than the Uniform With Replacement sampling. In addition to it, we observe that plots with small batch sizes Tpoints converge faster.
2Qur code:
http:///ssaareeeee.........eeeer eeeperinent_...
16

---

Published in Transactions on Machine Learning Research (10/2023)
A.4 Computing environment
The code was written in Python 3.6.8 using PyTorch 1.9 (Paszke et al., 2019) and optimization research simulator FL PyTorch (Burlachenko et al., 2021). The distributed environment was emulated on a machine with (R) XR) Gold 6226R CPU @ 2.90GHz and 64 cores.
17

---

Published in Transactions on Machine Learning Research (10/2023)
B
Auxiliary fact: Variance Decomposition
We use the following auxiliary fact in our proofs: Let us take a random vector & E IR, then
[[K] 2 E ||[[]]]?
(9)
C Examples of Optimization Problems
Example 1. For simplicity, let us assume that is even. Let us consider the optimization problem (1) with
[xx) xx² + x²² for i {1,-.. n/2] and fi(x) = - -x² + ga² for ie {n/2+1, , where x E R and 6 b20 0. Then f(x) _x²² and
Vf(x) vf(w)l = 6².
L sup azy
x - yl
Moreover;
1 Σ4 [Vf(()) V f1)))11
1 ((a b)² 2 + (a-b))).
L sup azy
yll
and we can take arbitrary large.
Example 2. Let us assume that n 2 2 and consider the optimization problem (1) with fi(x) = =²²²² and b
f((x) = 0 for i E {2....,n where E IR and b 2 0. Then f(x)
=
2n
V V (v))]
b
L_ = sup
zzy
x y|l
n2
[ (x) (fi(v)
b
Li sup
71
11 xzy
x v
n2
and
E ===1 V Vf((2)11
1
sup
|| yl
Vn
zty
Example 3. Let us consider the optimization problem (8) with f(x) = 1 Σi 1 Σ
fz(()) and f(z(z)
ha x for all i [g] and j E [m], where æ E and b 2 0 and b, = for all E (2.....9) Then f(x)
b
2
2g
[V vf(w]] b
L_ = sup zzy
1 vl g
Σ9
(x)
Vir( 1IV vS(D)12 (
1
1 sup
gm
b4.
| - w|
9 g²
z
and
Th E VV)(()
( [V v =0 Vie In].
L sup xzy
|| 2/
Substituting the smoothness constants to the complexity Nuniform from Sec 3 and Nunitorm from See 4, one can show that
-((1 nn+ Do max ( (Vnl+, 1.m) 1.m)) = A -0(1 20(2t Aoynby
Nuuiform = =0
5o
and
Ao max
Vn (vi Ee
Agh2
N Ngroup = + n
= n
E
The complexity N Ngroup is vT/vg times better than the complexity Nuniform-
18

---

Published in Transactions on Machine Learning Research (10/2023)
D Missing Proofs
Lemma 1. Suppose that Assumption 2 holds and let x²+1 = x²' -7g'- Then for any g¹ E IRdd and y > 0, we
have
f(x s f(x²) - ? 2 [ 27 2
I-
|| - =z'll² + 7 "11o v
(10)
2
Proof.
Using Assumption 2, we have
f(x ) - +(x') (Vf(x))) x²+1 - -x²)= + L-
2 || - 2211
- - (V)(()) ).g²) - 2 -101t1--001 22'1?
=
-
Next, due to (x,1) || y/1² - 1 || - =lull,, obtain
L- 1(x S {(x') - 2/2 | {1_ 27 ||0 ||a4t1- -011 + llg' 2110
Theorem 6. Suppose that Assumplions 1, 2, 3 hold and the samplings
St
E S(A, B, (wx)
Then Algorithm 1 (PAGE) has the convergence rate E [[( s
2 7T
where
7 I
B) (T- + - V ((A-B) 12,,. + BI2,w)
Proof. We start with the estimation of the variance of the noise:
E [[10 [| ||]
= -11-p)E g + S
= -1 S ([V - -10 [(
(vS(*(1) - (1- [f
where we used the unbiasedness of the sampling. Using Definition 1, we have
E || ++
A n
$ (1 p)
- B
nZw;
+ (1 p) ||g \f(
Using the definition of L+w and Liww, get
E |1g +1
\| ||? P]
( - v.(0+|)11-- - [V/(ttt) - vS(an)112
5 (1 - p)
A
i=1 n2w;
+ (1 p)
l|g - vf(z')]
= (1 - p)
(A B) Σ n2ws
|ff(tttt)) - -
(11)
+ B Σ
[ V)(a')) -
[i/l n2ws
+ (1 p) "lg' - -V
< (1 - p) ((A - (A)-)) 2 + BLL w || x(12 + (1 - p) 11 V
19

---

Published in Transactions on Machine Learning Research (10/2023)
We now continue the proof using Lemma 1. We add (10) with 2/ x (11), and take expectation to get
|( - + 7 110 (
E
2p
f (x')- - - - v/ (x || -
L-
- -x'1]= +
llgt 211
4 (x')11
E
2
I E [1 p1 /6 v (z)) + (1-p) p) ((A B3 1 - +LL
2p
(12)
E
ʃ (x")
1" + 2
/ v/ (>("')) || vs (x')11
2p
1
L- (1
ph
((A - B) L.,.. + BL )
||
- 1'1
2y
2
2p
< E
f (x")
f* + 2
|وgs - vs
(+')))
vf (x
2p
where the last inequality holds due to
1
L
(1-ph 2p
((A - B) L2,w + BL2,w) 2 0 by choosing stepsize
27
2
L
+
p
((A B) L3,,. + +BL
Now, if we define L : 1 (x²) - f + 2
11g
v (>x))² )1² then (12) can written in the form
2p
E S
E (
Summing up from t == to - 1, we get
T-1 Σ E (+)))
E ( E [P)
Then according to the output of the algorithm, i.e., ãr is randomly chosen from {x {x')re heem and Do f (I²) - fo + 16 Vf (z = (z9) - fo def Do, we have
20. 7T
E |VF
Theorem 7. Suppose that Assumplions 1, 2, 3, 4 and the samplings St E S(A, B, {wi) ) Then Algorithm (PAGE) has the convergence rate E [f(=")] - f S (1 - 7H) -1 min {(L / 20-r) ((A - L3 + BL2..))
(1)) Do. where 7 S
2p
Proof. From the proof of Thm 6, we know that
E - v
(1 p) ((1 - B) 12 + BL2 ||_-1
x011 +(1 - p) 1lg -
(13)
20

---

Published in Transactions on Machine Learning Research (10/2023)
Using Lemma 1, we add (10) with 1 (13), and take expectation to get
E f( ) + ?
19 t \1(
p
(x)) 1" 2 V/ (1))
1
--
||0 1-4t1-000 --'1-=0111= x011++2 2211 Elg v
E
27
2
1 [1 (1-p))19 (>())) 1| (1 -1p) p) ((A B) L3 + +LL || -
P
fx²) f + (1 - P ? / vf(²)) 112 [[
E
2
(1 L- 1--1--1----- (1-1)7
((A - B) L3 BL] || -
27 2 p
E f (x²) f + (1
p | vs ())))) ? 2 [|
2 p
where the last inequality holds due
21 4 (1-p)7 ((A B) 12,,0 + BI2,a) 2 0 by choosing stepsize
2(1 -p)
i b- +
((A B) 12 + BL2
Next, using Assumption 4 and & S 2 have
f(alt))- - 8r + ? -
p
E - f 2 llss Vf v
(x')-
Unrolling the recursion and considering that g" vf(x"), we complete the proof of theorem.
E Derivations of the Parameters for the Samplings
E.1 Nice sampling
Let be a random subset uniformly chosen from [n]] with a fixed cardinality т. Let us fix ai...., on E Rd. A sampling S(a)...., an) := 1/2 Eies di is called the Nice sampling, where p. = Prob(i E S).
Let us bound E [[a an) 1 Σn aill"] and find parameters from Definition 1. Note that |S| = |
T. We introduce auxiliary random variables
{1
I i ies
Xi =
0 herwise
21

---

Published in Transactions on Machine Learning Research (10/2023)
Due to p; Prob (i S) 7, we have
27
F.
Xidl
Pi
iES
||xeael 11
+
X(a)))
itj
[Xal ||.11 2
+
Σ
E Kxx x²)l (a),
7²
itj
||::
1 +
T
Σ
(ais ag)
11 1)T
ifj
1
T-1
||a||
| 11
aj
TVT
n(n - 1)T
n - 7
Σ
||:il +
ύ-1 1
T(n 1)
n(n 1)T
i=1
where we use E [x+]] [xi]
and E [xxx]]
z m(n-1))
when # j.
Finally, we have
E
04
iES
iES
7
1
|a.11
T-1
1
7(n
1)
n
1)7
1
8
||:11
1|
7(n 1)
7 i=1
Thus we have A = B p-t and w; + for all i e [n].
r(n-15
E.2 Independent sampling
Let us define i.i.d. random variables
Xs = L
with probability Ps
with probability 1 1- Pis
for all E [n] and take S fie [n] Ix = 1}. We now fix aj..... an E Rdd, sampling S(a....., = n Eies as is called the Independent sampling, where pa Prob(i E S).
1
P.
22

---

Published in Transactions on Machine Learning Research (10/2023)
We get
1
F
Σ 4
=
Xia
Ag
72 1=1
iES
E
[xx]
E [x [x,]
n2p
|aı1|
n pipp
(ass az}
11
02
iデj
||:1| 2|
+
||all
n2pp
n²
Σ
(
||a:11
π
Pi
Thus have A
B and W)
for all E [n].
Σ
E.3 Importance and Uniform With Replacement sampling
Let us fix r > 0. For all l E [r], we define i..dd random variables
1 with probability ?
2 with probability d2
Xk
n with probability on.
where (g1...., Qn) S" (simple simplex). A sampling
S(a an)
ax
n4x+
is called the Importance sampling. The Imporlance sampling reduces to the Uniform With Replacement sampling when q; 1/n for all e [n]. Note that [S] S T.
Let us bound the variance
0xk
1
E
nixi
71
i=1
E
ax
1
E
ax*
ai
Oxe'
1
Σ a4
naxx
ndxx
ndxx
Using the independents and unbiasedness of the random variables, the last term vanishes and we get
E
axa
1
>a
a
ax
n
a
ngxx
n1
T1qx
71
1
E
XM
70
K
a;
4
a
nq
23

---

Published in Transactions on Machine Learning Research (10/2023)
22
|
2
0g
71
72
Thus we have A = B 1 and wA &i for all é E [n].
E.4 Extended Nice sampling
In this section, analyze the extension of Nice sampling. First, we 4 times repeat each vector de then we use the Nice sampling. We define
Σ
m/ d1
1 1sish
a2
4+1sis +
0j ==
Σニ an
Ej 4 Si Σラ= 4
ni
where a; Rd and 4 2 1 for all i [n]. Then have
N
1
(x()))
az(z).
72
N
1=1
where := : ==Ez= 1j. Also, we denote N₂ := Σラ= '--
For some 7>6 O, we apply the Nice sampling method:
N
1
0:
1
= S(a
X:ā
N
iES
where
iES otherwise
Xi
P; Prob (i E S),
and S is a random set with cardinality from [N]. The sampling S(a a.) is called the Extended Nice sampling.
We now ready to bound the variance. Using the results for the Nice sampling, we obtain
E
S an)
Σ
a1(x))
n 1=1
1
E
S
an)
ā. (z)
N
7 1
1
||
0
7(n
1) N
7(n
1)
N
n
N
2
Nr
||a 11
N 2
Σ
N₂
||22|
7(m
1)
nl
ntz
1= +1
N
N
||ax11
n ー 7
N
ntn
T(n
7 1=1
i=
i= 1+1
24

---

Published in Transactions on Machine Learning Research (10/2023)
N 1
|ai 1
N 1 nl2
||a2|| +...4
nln laanl²)
N
n 7
1
Σ
7(n- 1)
(n1 n
7(n 1)
n
n - 7
Σ n² ||all?
n 7
1 Σ
T(n - 1)
7(n 1)
where W; N. Thus we have A = a-r and wi = to for E [n]]
-(n
F The Optimal Choice of wi
Let us consider L2,w0 and L2.w. In Sec 2, we show that one can take L = L2 - nm,
-
L. Let us
minimize 1 - i=1 ww, L3 with respect to the weights w, such that wT..... Wn 2 and Σi=1 wi 1. Using the method of Lagrange multipliers, we can construct a Lagrangian
1
-1 -1)
m Σ
C(w, A)
L
x
new;
1=1
Next, calculate partial derivatives
0c
13 x 0vi E [n]
Ow;
n² n²w?
and get
L
2
Using Ei= W; 1, we can show that the weights wi problem. Moreover,
Ji
are the solutions of the minimization
Σ L.
2
L2
L3
2,00
T
TW
G The Complexity of Algorithm 1 with the Importance sampling
The expected number of gradient calculations V.r of Algorithm 1 with the Imporlance sampling, the optimal w.² from Sec. F, and r S max
vuL+..
1 equals
L.
Ag
N Nimportance
0
n
L
nuw, 12
T
A
Vn Σ Li
0
n
L_
+
8
A
Dovn ( En Ix))
Ez
00
n
VnL+,w-
E
Dovn ( E Li)
0 n
H Missing Proofs: The Composition of Samplings
Lemma Let us assume that random sampling function belongs Definition 1 with some A, and weights wiy and random sampling function S, belongs to Definition with some As, B₁ and weights wij for
25

---

Published in Transactions on Machine Learning Research (10/2023)
all i E [n]. Moreover, B < 1. Then
E
S (S1 (Q11....., 01m,) ) ..... Sn (ap1.... .... Cnm, ))
4
m:
am1
jml
B)
A
mi
|a
B;
77
72
m
my Mij
m71 j=1
im
A
1
my
Oij
B
aijj
71
nu
m
m
j=1
where aig Rd for all? j E [m:] and E [n].
Proof. We denote ā.
S; (a.1 ................................ asm;) and a; = my
ΣØ=1 Osg. Using (9), we have
E
S (1)
an)
ai
1=1
=
Es
(a)....., an)
0g
1=1
n1
Es
((ai....., an)
0
+ E
a
a
71
Next, using Definition for the sampling S, we get
1
E
S 0
2x)
A
E
||0,1|
BE
T:W
i=1
04
1=
Due to (9), we obtain
11
E
a1
........00
an)
0:
71
1
F
||ā: a:11
+
1
||a]| 2
nW;;
1=1
BE
03
B
i=1
1
+E
1=1
1=1
26

---

Published in Transactions on Machine Learning Research (10/2023)
1
E.
|
a:l
+
A
||a]]
2
11
71
7
+
(1
B)F
Σ 0
11
1
0g
B
0g
71
i=1
1
1
1
F.
||ā. - -a11?]
+
1
2
77
1=1
(1
B)
n
n
+
[00 - az
ai
n
n
A
+
(1 B)
E
a
axll
1
Σ
1
||
11
B
1
4
n
n
nww;
Using Definition 1 for the samplings S,, we have
E
S
01
an)
a3
i=1
A
(1
B)
1
m
A
||
112
Bg
||ab1?
m
1=1
j=1
1
12
1
||
2 B
1
a
71
nuw
n
Theorem 10. Suppose that Assumptions 1, 2, 3, 5 hold and the samplings S E S(A, B, (w:)==)) and the samplings S E S(Ai, Bi, { for all i E [m]. Moreover, B < 1. Then Algorithm 2 has the convergence
mile F. | | ( ||*] < 27p, where y S [I__ + 1 (Li + +cg)
-1
A
(1- (1-B) B)
L:
((A: Bi) + BILT,
naw
n
and
cy (A B) BL
Proof. We start with the estimation of the variance of the noise:
E |1g ( |1²]
|o' + (1.. (1V// -f((((())))) -
=(1- -1--))
(1 (1-p) p) is ((s,
-[,()))))) -VS( --(-))) -1- "(1-ol-- {|(
where we used the unbiasedness of the composition of samplings. Using Lemma 2, we have
E | ( - )1
< (1 -p) p)
Σ
A
1
B)
A
1
V (ef( -B -
new;
m;
m&W
1=1
==1
27

---

Published in Transactions on Machine Learning Research (10/2023)
A - B -
n
p) |1gs v
Using Definitions 3, 4, 8 and 9, we get
|1g +1 vf(x ()( ||
E
A
(1 (1-B)
A [A-B. B,
1 [iiz(attt)
$ (1 p)
new;;
m
myWliy
1
m4
Vis(2'/1 (f((=tt1) - V -
+
Bi
m7
Wi
A
[ (
TWV
i=1
- | - -
+B
1=1
+ (1 - P)) ||gs v)(<')||
< (1 -p) p)
1
(1-B)
((A; - B,) mn B.I.I.,oD.) + (A - B)L?,,. + BL?
| x²12
120
i=1
+ (1 p) ||g
f>(("))1
From this point the proof of theorem repeats the proof of Thm with
A
(1 B) ((A; B:) ww: + B, + (A B)I + BL2
+
instead of
(A B) I.,,.. + BT2
28

---

Published in Transactions on Machine Learning Research (10/2023)
Artificial Quadratic Optimization Tasks
In this section, we provide algorithms that we use to generate artificial optimization tasks for experiments. Algorithm 3 and Algorithm allow us to control the smoothness constants L and Li, accordingly, via the noise scales.
Algorithm Generate quadratic optimization task with controlled L (homogeneity)
1: Parameters: number nodes dimension d, regularizer X, and noise scale 8.
2: for = l....,, n do
3: Generate random noises of = =1 + af and u/ sC ii.dd E3 50 NormalDistributionn)) 1)
Take vector 6, (-11+ 0/........0 E Rd
5: Take the initial tridiagonal matrix
A; U9
-1
e Rdxd
-1
6: end for
7: Take the mean matrices A -. As
Find the minimum cigenvalue {min(A)
for 1...,.., n do
10: Update matrix A; A; + (A Amin (A))I
11: end for
12: Take starting point _0 = (Vd,0, 0, .... 0)
13: Output: matrices A,, An, vectors ...... bn starting point n0
Algorithm Generate quadratic optimization task with controlled L;
1: Parameters: number nodes n, dimension d, regularizer X, and noise scale s.
for = 1,...,, n do
Generate random noises Ui
s&;, where ii.d. & ~ IxponeeniafDDistibttttto...) Generate random noises D/ sEt ii.d. g NormalDistrriattion( 1)
Take vector 6, (-1 v/f,0. 0,.,0...,000 e Rd
6: Take the initial tridiagonal matrix
A; V/1
-1
E ddxd
-1
-1
7: end for
8: Take starting point x = (Vd, ........00)
Output: matrices AI, An, vectors bj. ... bns starting point z
29


Reviews:
[{"review_id": "NnytbMtYuX", "reviewer": "Reviewer_jZxM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n1. The paper provides the most general and accurate analysis of optimal SGD in the smooth nonconvex regime. \n2. Theoretical findings are supposed with experiments.\n\nWeakness\n1. The paper is worse organized.  E.g., the introduction seems too short, the background and the setting of the studied problem are not clear. The title of Section 2 is “Assumptions”, while only a small part of the assumptions. In addition, I do not understand the purpose of Section 2.3 ‘Clarifications and Discussion’. Section 3 presents the main results of the paper, but the algorithm and theorems lack some explanation and discussion. For example, how does Algorithm 1 work? In Section 4, the background of federated learning is not introduced. Also, there lacks an explanation of Algorithm 2. \n\n2. It seems the paper is incremental compared with the previous works, especially in the technique aspect.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the classical problem of finding an approximately stationary point of the average of smooth and possibly nonconvex functions. The theoretical analysis for PAGE algorithm is provided for a general (possibly unbiased) sampling scheme by carefully analyzing the related constant (e.g., smoothness parameter) , which extends the results in previous works. The proposed method is applied to solving federated learning problems. The experiments show that important samplings can improve the convergence rate.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. As mentioned in the weakness, the paper is hard to follow. I would suggest the authors re-organized the paper and polish the language.\n2. Some notations and definitions are not introduced. For example, $L_{-}$ and $L_{+}$ appear in the introduction, while they are introduced in Section 2.2. Algorithm 1 is mentioned in Section 2.1, while it is introduced in Section 3. In addition,  $\\|\\|\\cdot\\|\\|$ is not defined. And the definition of smoothness is not introduced. Please go through the paper carefully and make sure the paper is self-consistent.\n", "broader_impact_concerns": "no", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gIZdRyWqVp", "reviewer": "Reviewer_dCj7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n1.  The paper's central idea is intriguing. The introduction of $L_{+, w}$ and $L_{\\pm, w}$ and the subsequent complexity bounds offer fresh perspectives. Notably, the new complexity bound for uniform sampling significantly outperforms its predecessor, especially when all $f_i$ are similar.\n\n2. The paper's analytical framework is versatile, accommodating a variety of sampling methods.\n\n**Weaknesses:**\n\n1.  Discussion on results is not sufficient: While Section 3 discusses three sampling techniques—uniform, Nice, and Importance—and mentions superior bounds for uniform and Importance compared to the original PAGE bounds, it still falls short in comparative analysis. How are these bounds compare to each other? Under what circumstances should a particular sampling method be preferred? Is there a way to determine the \"optimal\" or \"best\" sampling technique?\n\n**Additional questions and comments:**\n\n1. In Section 3.3, the assertion $N_{importance} \\leq N_{orig}$ needs elucidation. Do authors mean $\\frac{1}{n}\\sum L_i \\leq L_{+}$? This isn't immediately apparent to me.\n\n2. The motivation \"However, in federated learning, a sampling of nodes or the functions $f_i$ is dictated by hardware limits or network quality\" in Section 4 does not sound very convincing to me. If these factors are indeed the driving forces, the \"meta\" sampling $S$ might not necessarily be unbiased. However, Algorithm 2 presupposes an unbiased $S$.\n\n3. How will results in Section 4.1 compare to the state-of-the-art?\n\n4. The last sentence in Section 4.2 seems to suggest a $\\sqrt{n/g}$ times enhancement exclusively in the context of Example 3. Or are we able to achieve such improvement for all cases?\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper delves into variance reduction techniques aimed at identifying approximate stationary points in nonconvex optimization. The contributions are summarized as follows.\n\n1. **Introduction of the \"Weighted AB Inequality\"**: The paper's first contribution lies in Definition 1, where the authors introduce the \"weighted AB inequality\" applicable to a spectrum of sampling methods. This inequality delineates an unbiased sampling method designed to estimate the average of \\( (a_1,..., a_n) \\) using three parameters: A, B, and \\( {w_i} \\). These parameters subsequently appear in the convergence bounds of PAGE, tightening its analysis.\n\n2. **Refined Complexity Bound for PAGE**: Compared with the previously established complexity bounds of PAGE, this paper offers a more nuanced complexity bound, improving dependency on the smoothness parameter. Specifically, the paper advances from \\( \\sqrt{n}L_{+} \\) to \\( \\max \\{ \\sqrt{n}L_{\\pm}, L_{-} \\} \\).\n\n3. **Expansion to Other Sampling Techniques**: The paper ventures into alternative sampling strategies, such as importance sampling. Furthermore, it broadens its scope to accommodate scenarios where each \\( f_i \\) adopts a finite-sum structure.\n\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper's writing requires refinement in terms of grammar, formality, clarity, and overall structure.\n\n1. **Grammar and Typos**: The manuscript contains several grammatical errors and typographical mistakes. Examples include:\n   - \"the ratio... is as large one desires\"\n   - \"our theoretical ﬁndings are supposed with carefully designed experiments.\"\n   - \"help us better to understand the structure of the optimization problem\"\n   \n   These are just a few instances. A comprehensive review of the entire paper is recommended to address all such issues.\n\n2. **Formality and Tone**: Certain phrases and sentences do not align with the formal and scientific tone expected of a research paper. Noteworthy examples are:\n   - \"Optimal, but hiding a secret.\"\n   - \"All in all, let us consider (8) instead of (1).\"\n   \n\n3. **Clarity of Expressions and Notations**: Some sections of the paper lack clarity in their presentation. For instance:\n   - In Section 3.4, \"This is the first analysis of optimal SGD\". The context of \"optimal SGD\" is not clearly defined.\n   - In Section 2, the notations \\( w_i \\) and \\( \\omega \\) are visually similar, which can be confusing for readers.\n\n4. **Structural Consistency**: The paper could benefit from a more coherent structure. For example, while Sections 3.1 to 3.4 discuss different sampling techniques, Sections 3.4 and 3.5 are parallel but discuss different things. \n\nI strongly advise the authors to undertake a thorough revision to enhance the paper's readability and overall quality.\n\n", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UOU5DVxvaO", "reviewer": "Reviewer_fnTe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n1) The dependence on L is often ignored. There are sure cases that ${L_+}$ can be much larger than $L_-$ or other notation of smoothness, thus it is meaningful to make improvement by dedicatedly investigating the structure of the finite sum. \n\n2) The improvement does not require strong assumptions. The Hessian variance structure looks fine to me.\n\nWeaknesses: \n\n1) The tighter results in some sampling methods, such as importance sampling and stratified sampling, require some extra knowledge to achieve. The importance sampling requires an optimal weighting, and the stratified sampling requires \"good\" grouping. These would be difficult to satisfy for general problems and would take extra complexity. \n\n2) More explanation of the sampling procedures (nice, extended nice, etc) should be in place so the audience can easily apply them. Regarding the \"Nice\" sampling, is it any different from the usually known as uniform sampling without replacements? What is the difference between independent sampling and uniform with replacement?\n\n3) The Hessian variance structure looks crucial. Even though it is a fair assumption, it is kind of misleading to say this paper \"does not have additional assumptions on the structure of the problem\" and is in \"the same setting\" as (Li et al., 2021) (Section 3.1, Page 6).\n\n4) The presentation can be improved by giving some insight on the analysis, i.e., how the improved results are achieved. I could not find much insight in the abstract, introduction or anywhere except the proof part. \n\nOverall, I appreciate the contribution on the tighter complexities.  \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper has obtained better dependence on smoothness modules, mainly by refining proofs and also incorporating multiple different sampling strategies. The performances of some methods have been tested on a couple of synthetic problems. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All three reviews appreciate the contribution of relaxing the dependency of the smoothness constant, which is largely ignored in the community. Furthermore, the paper gives examples to show the constant $L_{\pm}$ in their bound can be much smaller than $L_+$ in the existing bounds, and the gap can be a factor of $\sqrt{n}$. Furthermore, their framework is versatile, accommodating a variety of sampling methods, and has applications to federated learning with composition of sampling.

Meanwhile, as mentioned by all the reviewers, there are several issues regarding the organization and the readability. Here are some suggested changes:

- add some insights to sketch how to get better dependency on the smoothness constants
- more explanation of the sampling procedures
- improve the presentation and writing quality
Decision: Accept (poster)