EF21 with Bells & Whistles: Practical Algorithmic Extensions of Modern Error Feedback

1 INTRODUCTION In this paper , we consider the nonconvex distributed/federated optimization problem of the form min 𝑥∈R𝑑 { ︂ 𝑓 ( 𝑥 ) def = 1𝑛 𝑛∑︀ 𝑖=1 𝑓𝑖 ( 𝑥 ) } ︂ , ( 1 ) where 𝑛 denotes the number of clients/workers/devices/nodes connected with a server/master and client 𝑖 has an access to the local loss function 𝑓𝑖 only . The local loss of each client is allowed to have the online/expectation form 𝑓𝑖 ( 𝑥 ) = E𝜉𝑖∼𝒟𝑖 [ 𝑓𝜉𝑖 ( 𝑥 ) ] , ( 2 ) or the finite-sum form 𝑓𝑖 ( 𝑥 ) = 1 𝑚 𝑚∑︀ 𝑗=1 𝑓𝑖𝑗 ( 𝑥 ) . ( 3 ) Problems of this structure appear in federated learning ( Konečný et al. , 2016 ; Kairouz , 2019 ) , where training is performed directly on the clients ’ devices . In a quest for state-of-the-art performance , machine learning practitioners develop elaborate model architectures and train their models on enormous data sets . Naturally , for training at this scale to be possible , one needs to rely on distributed computing ( Goyal et al. , 2017 ; You et al. , 2020 ) . Since in recent years remarkable empirical successes were obtained with massively over-parameterized models ( Arora et al. , 2018 ) , which puts an extra strain on the communication links during training , recent research activity and practice focuses on developing distributed optimization methods and systems capitalizing on ( deterministic or randomized ) lossy communication compression techniques to reduce the amount of communication traffic . A compression mechanism is typically formalized as an operator 𝒞 : R𝑑 ↦→ R𝑑 mapping hard-tocommunicate ( e.g. , dense ) input messages into easy-to-communicate ( e.g. , sparse ) output messages . The operator is allowed to be randomized , and typically operates on models Khaled & Richtárik ( 2019 ) or on gradients Alistarh et al . ( 2017 ) ; Beznosikov et al . ( 2020 ) , both of which can be described as vectors in R𝑑 . Besides sparsification ( Alistarh et al. , 2018 ) , typical examples of useful compression mechanisms include quantization ( Alistarh et al. , 2017 ; Horváth et al. , 2019a ) and low-rank approximation ( Vogels et al. , 2019 ; Safaryan et al. , 2021 ) . There are two large classes of compression operators often studied in the literature : i ) unbiased compression operators 𝒞 , meaning that there exists 𝜔 ≥ 0 such that E [ 𝒞 ( 𝑥 ) ] = 𝑥 , E [ ︀ ‖𝒞 ( 𝑥 ) − 𝑥‖2 ] ︀ ≤ 𝜔‖𝑥‖2 , ∀𝑥 ∈ R𝑑 ; ( 4 ) and ii ) biased compression operators 𝒞 , meaning that there exists 0 < 𝛼 ≤ 1 such that E [ ︀ ‖𝒞 ( 𝑥 ) − 𝑥‖2 ] ︀ ≤ ( 1 − 𝛼 ) ‖𝑥‖2 , ∀𝑥 ∈ R𝑑 . ( 5 ) Note that the latter “ biased ” class contains the former one , i.e. , if 𝒞 satisfies ( 4 ) with 𝜔 , then a scaled version ( 1 + 𝜔 ) −1𝒞 satisfies ( 5 ) with 𝛼 = 1/ ( 1+𝜔 ) . While distributed optimization methods with unbiased compressors ( 4 ) are well understood ( Alistarh et al. , 2017 ; Khirirat et al. , 2018 ; Mishchenko et al. , 2019 ; Horváth et al. , 2019b ; Li et al. , 2020 ; Li & Richtárik , 2021a ; Li & Richtárik , 2020 ; Islamov et al. , 2021 ; Gorbunov et al. , 2021 ) , biased compressors ( 5 ) are significantly harder to analyze . One of the main reasons behind this is rooted in the observation that when deployed within distributed gradient descent in a naive way , biased compresors may lead to ( even exponential ) divergence ( Karimireddy et al. , 2019 ; Beznosikov et al. , 2020 ) . Error Feedback ( EF ) ( or Error Compensation ( EC ) ) —a technique originally proposed by Seide et al . ( 2014 ) —emerged as an empirical fix of this problem . However , this technique remained poorly understood until very recently . Although several theoretical results were obtained supporting the EF framework in recent years ( Stich et al. , 2018 ; Alistarh et al. , 2018 ; Beznosikov et al. , 2020 ; Gorbunov et al. , 2020 ; Qian et al. , 2020 ; Tang et al. , 2020 ; Koloskova et al. , 2020 ) , they use strong assumptions ( e.g. , convexity , bounded gradients , bounded dissimilarity ) , and do not get 𝒪 ( 1/𝛼𝑇 ) convergence rates in the smooth nonconvex regime . Very recently , Richtárik et al . ( 2021 ) proposed a new EF mechanism called EF21 , which uses standard smoothness assumptions only , and also enjoys the desirable 𝑂 ( 1/𝛼𝑇 ) convergence rate for the nonconvex case ( in terms of number of communication rounds 𝑇 this matches the best-known rate 𝒪 ( ( 1+𝜔/√𝑛 ) /𝑇 ) obtained by Gorbunov et al . ( 2021 ) using unbiased compressors ) , improving the previous 𝑂 ( 1/ ( 𝛼𝑇 ) 2/3 ) rate of the standard EF mechanism ( Koloskova et al. , 2020 ) . 2 OUR CONTRIBUTIONS While Richtárik et al . ( 2021 ) provide a new theoretical SOTA for error feedback based methods , the authors only study their EF21 mechanism in a pure form , without any additional “ bells and whistles ” which are of importance in practice . In this paper , we aim to push the EF21 framework beyond its pure form by extending it in several directions of high theoretical and practical importance . In particular , we further enhance the EF21 mechanism with the following six useful and practical algorithmic extensions : stochastic approximation , variance reduction , partial participation , bidirectional compression , momentum , and proximal ( regularization ) . We do not stop at merely proposing these algorithmic enhancements : we derive strong convergence results for all of these extensions . Several of these techniques were never analyzed in conjunction with the original EF mechanism before , and in cases where they were , our new results with EF21 are vastly superior . See Table 1 for an overview of our results . In summary , our results constitute the new algorithmic and theoretical state-of-the-art in the area of error feedback . We now briefly comment on each extension proposed in this paper : ◇ Stochastic approximation . The vanilla EF21 method requires all clients to compute the exact/full gradient in each round . While Richtárik et al . ( 2021 ) do consider a stochastic extension of EF21 , they do not formalize their result , and only consider the simplistic scenario of uniformly bounded variance , which does not in general hold for stochasticity coming from subsampling ( Khaled & Richtárik , 2020 ) . However , exact gradients are not available in the stochastic/online setting ( 2 ) , and in the finite-sum setting ( 3 ) it is more efficient in practice to use subsampling and work with stochastic gradients instead . In our paper , we extend EF21 to a more general stochastic approximation framework than the simplistic framework considered in the original paper . Our method is called EF21-SGD ( Algorithm 2 ) ; see Appendix D for more details . ◇ Variance reduction . As mentioned above , EF21 relies on full gradient computations at all clients . This incurs a high or unaffordable computation cost , especially when local clients hold large training sets , i.e. , if 𝑚 is very large in ( 3 ) . In the finite-sum setting ( 3 ) , we enhance EF21 with a variance reduction technique to reduce the computational complexity . In particular , we adopt the simple and efficient variance-reduced method PAGE ( Li et al. , 2021 ; Li , 2021b ) ( which is optimal for solving problems ( 3 ) ) into EF21 , and call the resulting method EF21-PAGE ( Algorithm 3 ) . See Appendix E for more details . ◇ Partial participation . The EF21 method proposed by Richtárik et al . ( 2021 ) requires full participation of clients for solving problem ( 1 ) , i.e. , in each round , the server needs to communicate with all 𝑛 clients . However , full participation is usually impractical or very hard to achieve in massively distributed ( e.g. , federated ) learning problems ( Konečný et al. , 2016 ; Cho et al. , 2020 ; Kairouz , 2019 ; Li & Richtárik , 2021b ; Zhao et al. , 2021 ) . To remedy this situation , we propose a partial participation ( PP ) variant of EF21 , which we call EF21-PP ( Algorithm 4 ) . See Appendix F for more details . ◇ Bidirectional compression . The vanilla EF21 method only considers upstream compression of the messages sent by the clients to the server . However , in some situations , downstream communication is also costly ( Horváth et al. , 2019a ; Tang et al. , 2020 ; Philippenko & Dieuleveut , 2020 ) . In order to cater to these situations , we modify EF21 so that the server can also optionally compresses messages before communication . Our master compression is intelligent in that it employs the Markov compressor proposed in EF21 to be used at the devices . The proposed method , based on bidirectional compression , is EF21-BC ( Algorithm 5 ) . See Appendix G for more details . ◇ Momentum . A very successful and popular technique for enhancing both optimization and generalization is momentum/acceleration ( Polyak , 1964 ; Nesterov , 1983 ; Lan & Zhou , 2015 ; AllenZhu , 2017 ; Lan et al. , 2019 ; Li , 2021a ) . For instance , momentum is a key building block behind the widely-used Adam method ( Kingma & Ba , 2014 ) . In this paper , we add the well-known ( Polyak ) heavy ball momentum ( Polyak , 1964 ; Loizou & Richtárik , 2020 ) to EF21 , and call the resulting method EF21-HB ( Algorithm 6 ) . See Appendix H for more details . ◇ Proximal setting . It is common practice to solve regularized versions of empirical risk minimization problems instead of their vanilla variants ( Shalev-Shwartz & Ben-David , 2014 ) . We thus consider the composite/regularized/proximal problem min 𝑥∈R𝑑 { ︂ Φ ( 𝑥 ) def = 1𝑛 𝑛∑︀ 𝑖=1 𝑓𝑖 ( 𝑥 ) + 𝑟 ( 𝑥 ) } ︂ , ( 6 ) where 𝑟 ( 𝑥 ) : R𝑑 → R ∪ { +∞ } is a regularizer , e.g. , ℓ1 regularizer ‖𝑥‖1 or ℓ2 regularizer ‖𝑥‖22 . To broaden the applicability of EF21 to such problems , we propose a proximal variant of EF21 to solve the more general composite problems ( 6 ) . We call this new method EF21-Prox ( Algorithm 7 ) . See Appendix I for more details . Our theoretical complexity results are summarized in Table 1 . In addition , we also analyze EF21SGD , EF21-PAGE , EF21-PP , EF21-BC under Polyak-Łojasiewicz ( PŁ ) condition ( Polyak , 1963 ; Lojasiewicz , 1963 ) and EF21-Prox under the generalized PŁ-condition ( Li & Li , 2018 ) for composite optimization problems . Due to space limitations , we defer all the details about the analysis under the PŁ-condition to the appendix and provide only simplified rates in Table 1 . We comment on some preliminary experimental results in Section 5 . More experiments including deep learning experiments are presented in Appendix A . 3 METHODS Since our methods are modifications of EF21 , they share many features , and are presented in a unified way in Table 2 . At each iteration of the proposed methods , worker 𝑖 computes the compressed vector 𝑐𝑡𝑖 and sends it to the master . The methods differ in the way of computing 𝑐 𝑡 𝑖 but have similar ( in case of EF21-SGD , EF21-PAGE , EF21-PP – exactly the same ) update rules to the one of EF21 : 𝑥𝑡+1 = 𝑥𝑡 − 𝛾𝑔𝑡 , 𝑔𝑡+1𝑖 = 𝑔𝑡𝑖 + 𝑐𝑡𝑖 , 𝑔𝑡+1 = 1𝑛 𝑛∑︀ 𝑖=1 𝑔𝑡+1𝑖 = 𝑔 𝑡 + 1𝑛 𝑛∑︀ 𝑖=1 𝑐𝑡𝑖 . ( 7 ) The pseudocodes of the methods are given in the appendix . Below we briefly describe each method . ◇ EF21-SGD : Error feedback and SGD . EF21-SGD is essentially EF21 but instead of the full gradients ∇𝑓𝑖 ( 𝑥𝑡+1 ) , workers compute the stochastic gradients 𝑔𝑖 ( 𝑥𝑡+1 ) , and use them to compute 𝑐𝑡𝑖 = 𝒞 ( 𝑔𝑖 ( 𝑥𝑡+1 ) − 𝑔𝑡𝑖 ) . Despite the seeming simplicity of this extension , it is highly important for various applications of machine learning and statistics where exact gradients are either unavailable or prohibitively expensive to compute . ◇ EF21-PAGE : Error feedback and variance reduction . In the finite-sum regime ( 3 ) , variance reduced methods usually perform better than vanilla SGD in many situations ( Gower et al. , 2020 ) . Therefore , for this setup we modify EF21 and combine it with variance reduction . In particular , this time we replace ∇𝑓𝑖 ( 𝑥𝑡+1 ) in the formula for 𝑐𝑡𝑖 with the PAGE estimator ( Li et al. , 2021 ) 𝑣 𝑡+1 𝑖 . With ( typically small ) probability 𝑝 this estimator equals the full gradient 𝑣𝑡+1𝑖 = ∇𝑓𝑖 ( 𝑥𝑡+1 ) , and with probability 1 − 𝑝 it is set to 𝑣𝑡+1𝑖 = 𝑣 𝑡 𝑖 + 1 𝜏𝑖 ∑︀ 𝑗∈𝐼𝑡𝑖 ( ︀ ∇𝑓𝑖𝑗 ( 𝑥𝑡+1 ) −∇𝑓𝑖𝑗 ( 𝑥𝑡 ) ) ︀ , where 𝐼𝑡𝑖 is a minibatch of size 𝜏𝑖 . Typically , the number of data points 𝑚 owned by each client is large , and 𝑝 ≤ 1/𝑚 when 𝜏𝑖 ≡ 1 . As a result , computation of full gradients rarely happens during the optimization procedure : on average , once in every 𝑚 iterations only . Although it is possible to use other variance-reduced estimators like in SVRG or SAGA , we use the PAGE-estimator : unlike SVRG or SAGA , PAGE is optimal for smooth nonconvex optimization , and therefore gives the best theoretical guarantees ( we have obtained results for both SVRG and SAGA and indeed , they are worse , and hence we do not include them ) . Notice that unlike VR-MARINA ( Gorbunov et al. , 2021 ) , which is a state-of-the-art distributed optimization method designed specifically for unbiased compressors and which also uses the PAGEestimator , EF21-PAGE does not require the communication of full ( non-compressed ) vectors at all . This is an important property of the algorithm since , in some distributed networks , and especially when 𝑑 is very large , as is the case in modern over-parameterized deep learning , full vector communication is prohibitive . However , unlike the rate of VR-MARINA , the rate of EF21-PAGE does not improve with increasing 𝑛 . This is not a flaw of our method , but rather an inevitable drawback of distributed methods that rely on biased compressors such as Top-𝑘 . ◇ EF21-PP : Error feedback and partial participation . The extension of EF21 to the case of partial participation of the clients is mathematically identical to EF21 up to the following change : 𝑐𝑡𝑖 = 0 for all clients 𝑖 ̸∈ 𝑆𝑡 ⊆ { 1 , . . . , 𝑛 } that are not selected for communication at iteration 𝑡 . In practice , 𝑐𝑡𝑖 = 0 means that client 𝑖 does not take part in the 𝑡-th communication round . Here the set 𝑆𝑡 ⊆ { 1 , . . . , 𝑛 } is formed randomly such that Prob ( 𝑖 ∈ 𝑆𝑡 ) = 𝑝𝑖 > 0 for all 𝑖 = 1 , . . . , 𝑛 . ◇ EF21-BC : Error feedback and bidirectional compression . The simplicity of the EF21 mechanism allows us to naturally extend it to the case when it is desirable to have efficient/compressed communication between the clients and the server in both directions . At each iteration of EF21-BC , clients compute and send to the master node 𝑐𝑡𝑖 = 𝒞𝑤 ( ∇𝑓𝑖 ( 𝑥𝑡+1 ) − ̃︀𝑔𝑡𝑖 ) and update ̃︀𝑔𝑡+1𝑖 = ̃︀𝑔𝑡𝑖 + 𝑐𝑡𝑖 in the usual way , i.e. , workers apply the EF21 mechanism . The key difference between EF21 and EF21-BC is that the master node in EF21-BC also uses this mechanism : it computes and broadcasts to the workers the compressed vector 𝑏𝑡+1 = 𝒞𝑀 ( ̃︀𝑔𝑡+1 − 𝑔𝑡 ) and updates 𝑔𝑡+1 = 𝑔𝑡 + 𝑏𝑡+1 , wherẽ︀𝑔𝑡+1 = 1𝑛 ∑︀𝑛𝑖=1 ̃︀𝑔𝑡+1𝑖 . Vector 𝑔𝑡 is maintained by the master and workers . Therefore , the clients are able to update it via using 𝑔𝑡+1 = 𝑔𝑡 + 𝑏𝑡+1 and compute 𝑥𝑡+1 = 𝑥𝑡 − 𝛾𝑔𝑡 once they receive 𝑏𝑡+1 . ◇ EF21-HB : Error feedback with momentum . We consider classical Heavy-ball method ( Polyak , 1964 ) with EF21 estimator 𝑔𝑡 : 𝑥𝑡+1 = 𝑥𝑡 − 𝛾𝑣𝑡 , 𝑣𝑡+1 = 𝜂𝑣𝑡 + 𝑔𝑡+1 , 𝑔𝑡+1𝑖 = 𝑔𝑡𝑖 + 𝑐𝑡𝑖 , 𝑔𝑡+1 = 1𝑛 𝑛∑︀ 𝑖=1 𝑔𝑡+1𝑖 = 𝑔 𝑡 + 1𝑛 𝑛∑︀ 𝑖=1 𝑐𝑡𝑖 . The resulting method is not better than EF21 in terms of the complexity of finding 𝜀-stationary point , i.e. , momentum does not improve the theoretical convergence rate . Unfortunately , this is common issue for a wide range of results for momentum methods Loizou & Richtárik ( 2020 ) . However , it is important to theoretically analyze momentum-extensions such as EF21-HB due to their importance in practice and generalization behaviour . ◇ EF21-Prox : Error feedback for composite problems . Finally , we make EF21 applicable to the composite optimization problems ( 6 ) by simply taking the prox-operator from the right-hand side of the 𝑥𝑡+1 update rule ( 7 ) : 𝑥𝑡+1 = prox𝛾𝑟 ( 𝑥 𝑡 − 𝛾𝑔𝑡 ) = arg min𝑥∈R𝑑 { 𝛾𝑟 ( 𝑥 ) + ‖𝑥−𝑥 𝑡+𝛾𝑔𝑡‖2/2 } . This trick is simple , but , surprisingly , EF21-Prox is the first distributed method with error-feedback that provably converges for composite problems ( 6 ) . 4 THEORETICAL CONVERGENCE RESULTS In this section , we formulate a single corollary derived from the main convergence theorems for our six enhancements of EF21 , and formulate the assumptions that we use in the analysis . The complete statements of the theorems and their proofs are provided in the appendices . In Table 1 we compare our new results with existing results . 4.1 ASSUMPTIONS In this subsection , we list and discuss the assumptions that we use in the analysis . 4.1.1 GENERAL ASSUMPTIONS To derive our convergence results , we invoke the following standard smoothness assumption . Assumption 1 ( Smoothness and lower boundedness ) . Every 𝑓𝑖 has 𝐿𝑖-Lipschitz gradient , i.e. , ‖∇𝑓𝑖 ( 𝑥 ) −∇𝑓𝑖 ( 𝑦 ) ‖ ≤ 𝐿𝑖 ‖𝑥− 𝑦‖ for all 𝑖 ∈ [ 𝑛 ] , 𝑥 , 𝑦 ∈ R𝑑 , and 𝑓 inf def = inf𝑥∈R𝑑 𝑓 ( 𝑥 ) > −∞ . We also assume that the compression operators used by all algorithms satisfy the following property . Definition 1 ( Contractive compressors ) . We say that a ( possibly randomized ) map 𝒞 : R𝑑 → R𝑑 is a contractive compression operator , or simply contractive compressor , if there exists a constant 0 < 𝛼 ≤ 1 such that E [ ︀ ‖𝒞 ( 𝑥 ) − 𝑥‖2 ] ︀ ≤ ( 1 − 𝛼 ) ‖𝑥‖2 , ∀𝑥 ∈ R𝑑 . ( 8 ) We emphasize that we do not assume 𝒞 to be unbiased . Hence , our theory works with the Top-𝑘 ( Alistarh et al. , 2018 ) and the Rank-𝑟 ( Safaryan et al. , 2021 ) compressors , for example . 4.1.2 ADDTIONAL ASSUMPTIONS FOR EF21-SGD We analyze EF21-SGD under the assumption that local stochastic gradients ∇𝑓𝜉𝑡𝑖𝑗 ( 𝑥 𝑡 ) satisfy the following inequality ( see Assumption 2 of Khaled & Richtárik ( 2020 ) ) . Assumption 2 ( General assumption for stochastic gradients ) . We assume that for all 𝑖 = 1 , . . . , 𝑛 there exist parameters 𝐴𝑖 , 𝐶𝑖 ≥ 0 , 𝐵𝑖 ≥ 1 such that E [ ︁ ‖∇𝑓𝜉𝑡𝑖𝑗 ( 𝑥 𝑡 ) ‖2 | 𝑥𝑡 ] ︁ ≤ 2𝐴𝑖 ( ︀ 𝑓𝑖 ( 𝑥 𝑡 ) − 𝑓 inf𝑖 ) ︀ + 𝐵𝑖‖∇𝑓𝑖 ( 𝑥𝑡 ) ‖2 + 𝐶𝑖 , ( 9 ) where1 𝑓 inf𝑖 = inf𝑥∈R𝑑 𝑓𝑖 ( 𝑥 ) > −∞ . Below we provide two examples of stochastic gradients fitting this assumption ( for more detail , see ( Khaled & Richtárik , 2020 ) ) . Example 1 . Consider ∇𝑓𝜉𝑡𝑖𝑗 ( 𝑥 𝑡 ) such that E [ ︁ ∇𝑓𝜉𝑡𝑖𝑗 ( 𝑥 𝑡 ) | 𝑥𝑡 ] ︁ = ∇𝑓𝑖 ( 𝑥𝑡 ) and E [ ︂⃦⃦⃦ ∇𝑓𝜉𝑡𝑖𝑗 ( 𝑥 𝑡 ) −∇𝑓𝑖 ( 𝑥𝑡 ) ⃦⃦⃦2 | 𝑥𝑡 ] ︂ ≤ 𝜎2𝑖 for some 𝜎𝑖 ≥ 0 . Then , due to variance decomposition , ( 9 ) holds with 𝐴𝑖 = 0 , 𝐵𝑖 = 0 , 𝐶𝑖 = 𝜎2𝑖 . 1When 𝐴𝑖 = 0 one can ignore the first term in the right-hand side of ( 9 ) , i.e. , assumption inf𝑥∈R𝑑 𝑓𝑖 ( 𝑥 ) > −∞ is not required in this case . Example 2 . Let 𝑓𝑖 ( 𝑥 ) = 1𝑚𝑖 ∑︀𝑚𝑖 𝑗=1 𝑓𝑖𝑗 ( 𝑥 ) , 𝑓𝑖𝑗 be 𝐿𝑖𝑗-smooth and 𝑓 inf 𝑖𝑗 = inf𝑥∈R𝑑 𝑓𝑖𝑗 ( 𝑥 ) > −∞ . Following Gower et al . ( 2019 ) , we consider a stochastic reformulation 𝑓𝑖 ( 𝑥 ) = E𝑣𝑖∼𝒟𝑖 [ 𝑓𝑣𝑖 ( 𝑥 ) ] = E𝑣𝑖∼𝒟𝑖 [ ︃ 1 𝑚𝑖 𝑚𝑖∑︀ 𝑗=1 𝑓𝑣𝑖𝑗 ( 𝑥 ) ] ︃ , ( 10 ) where E𝑣𝑖∼𝒟𝑖 [ 𝑣𝑖𝑗 ] = 1 . One can show ( see Proposition 2 of Khaled & Richtárik ( 2020 ) ) that under the assumption that E𝑣𝑖∼𝒟𝑖 [ ︀ 𝑣2𝑖𝑗 ] ︀ is finite for all 𝑗 stochastic gradient ∇𝑓𝜉𝑡𝑖𝑗 ( 𝑥 𝑡 ) = ∇𝑓𝑣𝑡𝑖 ( 𝑥 𝑡 ) with 𝑣𝑡𝑖 sampled from 𝒟𝑖 satisfies ( 9 ) with 𝐴𝑖 = max𝑗 𝐿𝑖𝑗E𝑣𝑖∼𝒟𝑖 [ ︀ 𝑣2𝑖𝑗 ] ︀ , 𝐵𝑖 = 1 , 𝐶𝑖 = 2𝐴𝑖∆inf𝑖 , where ∆inf𝑖 = 1 𝑚𝑖 ∑︀𝑚𝑖 𝑗=1 ( 𝑓 inf 𝑖 − 𝑓 inf𝑖𝑗 ) . In particular , if Prob ( ∇𝑓𝜉𝑡𝑖𝑗 ( 𝑥 𝑡 ) = ∇𝑓𝑖𝑗 ( 𝑥𝑡 ) ) = 𝐿𝑖𝑗∑︀𝑚𝑖 𝑙=1 𝐿𝑖𝑙 , then 𝐴𝑖 = 𝐿𝑖 = 1 𝑚𝑖 ∑︀𝑚𝑖 𝑗=1 𝐿𝑖𝑗 , 𝐵𝑖 = 1 , and 𝐶𝑖 = 2𝐴𝑖∆ inf 𝑖 . Stochastic gradient 𝑔𝑖 ( 𝑥𝑡 ) is computed using a mini-batch of 𝜏𝑖 independent samples satisfying ( 9 ) : 𝑔𝑖 ( 𝑥 𝑡 ) def = 1𝜏𝑖 𝜏𝑖∑︀ 𝑗=1 ∇𝑓𝜉𝑡𝑖𝑗 ( 𝑥 𝑡 ) . 4.1.3 ADDITIONAL ASSUMPTIONS FOR EF21-PAGE In the analysis of EF21-PAGE , we rely on the following assumption . Assumption 3 ( Average ℒ-smoothness ) . Let every 𝑓𝑖 have the form ( 3 ) . Assume that for all 𝑡 ≥ 0 , 𝑖 = 1 , . . . , 𝑛 , and batch 𝐼𝑡𝑖 ( of size 𝜏𝑖 ) , the minibatch stochastic gradients difference ̃︀∆𝑡𝑖 def= 1 𝜏𝑖 ∑︀ 𝑗∈𝐼𝑡𝑖 ( ∇𝑓𝑖𝑗 ( 𝑥𝑡+1 ) −∇𝑓𝑖𝑗 ( 𝑥𝑡 ) ) computed on the node 𝑖 , satisfies E [ ︁̃︀∆𝑡𝑖 | 𝑥𝑡 , 𝑥𝑡+1 ] ︁ = ∆𝑡𝑖 and E [ ︂⃦⃦⃦ ̃︀∆𝑡𝑖 − ∆𝑡𝑖 ⃦⃦⃦2 | 𝑥𝑡 , 𝑥𝑡+1 ] ︂ ≤ ℒ2𝑖𝜏𝑖 ‖𝑥𝑡+1 − 𝑥𝑡‖2 ( 11 ) with some ℒ𝑖 ≥ 0 , where ∆𝑡𝑖 def = ∇𝑓𝑖 ( 𝑥𝑡+1 ) −∇𝑓𝑖 ( 𝑥𝑡 ) . We also define ̃︀ℒ def= 1𝑛 ∑︀𝑛𝑖=1 ( 1−𝑝𝑖 ) ℒ2𝑖𝜏𝑖 . This assumption is satisfied for many standard/popular sampling strategies . For example , if 𝐼𝑡𝑖 is a full batch , then ℒ𝑖 = 0 . Another example is uniform sampling on { 1 , . . . , 𝑚 } , and each 𝑓𝑖𝑗 is 𝐿𝑖𝑗-smooth . In this regime , one may verify that ℒ𝑖 ≤ max1≤𝑗≤𝑚 𝐿𝑖𝑗 . 4.2 MAIN RESULTS Below we formulate the corollary establishing the complexities for each method . The complete version of this result is formulated and rigorously derived for each method in the appendix . Corollary 1 . Suppose that Assumption 1 holds . Then , there exist appropriate choices of parameters for EF21-PP , EF21-BC , EF21-HB , EF21-Prox such that the number of communication rounds 𝑇 and the ( expected ) number of gradient computations at each node # grad for these methods to find an 𝜀-stationary point , i.e. , a point ? ̂ ? 𝑇 such that E [ ‖∇𝑓 ( ? ̂ ? 𝑇 ) ‖2 ] ≤ 𝜀2 for EF21-PP , EF21-BC , EF21-HB and E [ ‖𝒢𝛾 ( ? ̂ ? 𝑇 ) ‖2 ] ≤ 𝜀2 for EF21-Prox , where 𝒢𝛾 ( 𝑥 ) = 1/𝛾 ( ︀ 𝑥− prox𝛾𝑟 ( 𝑥− 𝛾∇𝑓 ( 𝑥 ) ) ) ︀ , are EF21-PP : 𝑇 = 𝒪 ( ︁ ̃︀𝐿𝛿0 𝑝𝛼𝜀2 ) ︁ , # grad = 𝒪 ( ︁ ̃︀𝐿𝛿0 𝛼𝜀2 ) ︁ EF21-BC : 𝑇 = # grad = 𝒪 ( ︁ ̃︀𝐿𝛿0 𝛼𝑤𝛼𝑀𝜀2 ) ︁ EF21-HB : 𝑇 = # grad = 𝒪 ( ︁ ̃︀𝐿𝛿0 𝜀2 ( ︁ 1 𝛼 + 1 1−𝜂 ) ︁ ) ︁ EF21-Prox : 𝑇 = # grad = 𝒪 ( ︁ ̃︀𝐿𝛿0 𝛼𝜀2 ) ︁ , where ̃︀𝐿 def= √︁ 1𝑛 ∑︀𝑛𝑖=1 𝐿2𝑖 , 𝛿0 def= 𝑓 ( 𝑥0 ) −𝑓 inf ( for EF21-Prox 𝛿0 = Φ ( 𝑥0 ) −Φ𝑖𝑛𝑓 ) , 𝑝 is the probability of sampling the client in EF21-PP , 𝛼𝑤 and 𝛼𝑀 are contraction factors for compressors applied on the workers ’ and the master ’ s sides respectively in EF21-BC , and 𝜂 ∈ [ 0,1 ) is the momentum parameter in EF21-HB . If Assumptions 1 and 2 in the setup from Example 1 hold , then there exist appropriate choices of parameters for EF21-SGD such that the corresponding 𝑇 and the averaged number of gradient computations at each node # grad are EF21-SGD : 𝑇 = 𝒪 ( ︁ ̃︀𝐿𝛿0 𝛼𝜀2 ) ︁ , # grad = 𝒪 ( ︁ ̃︀𝐿𝛿0 𝛼𝜀2 + ̃︀𝐿𝛿0𝜎2 𝛼3𝜀4 ) ︁ , where 𝜎 = 1𝑛 ∑︀𝑛 𝑖=1 𝜎 2 𝑖 . If Assumptions 1 and 3 hold , then there exist appropriate choices of parameters for EF21-PAGE such that the corresponding 𝑇 and # grad are EF21-PAGE : 𝑇 = 𝒪 ( ︁ ( ̃︀𝐿+ ̃︀ℒ ) 𝛿0 𝛼𝜀2 + √ 𝑚 ̃︀ℒ𝛿0 𝜀2 ) ︁ , # grad = 𝒪 ( ︁ 𝑚 + ( ̃︀𝐿+ ̃︀ℒ ) 𝛿0 𝛼𝜀2 + √ 𝑚 ̃︀ℒ𝛿0 𝜀2 ) ︁ , where ̃︀ℒ = √︁ 1−𝑝𝑛 ∑︀𝑛𝑖=1 ℒ2𝑖 , 𝜏𝑖 ≡ 𝜏 = 1 . Remark : We highlight some points for our results in Corollary 1 as follows : ∙ For EF21-PP and EF21-Prox , none of previous error feedback methods work on these two settings ( partial participation and proximal/composite case ) . Thus , we provide the first convergence results for them . Moreover , we show that the gradient ( computation ) complexity for both EF21-PP and EF21-Prox is 𝒪 ( 1/𝛼𝜀 ) , matching the original vanilla EF21 . It means that we extend EF21 to both settings for free . ∙ For EF21-BC , we show 𝒪 ( 1/𝛼𝑤𝛼𝑀𝜀2 ) complexity result . In particular , if one uses constant ratio of compression ( e.g. , 10 % ) , then 𝛼 ≈ 0.1 . Then the result will be 𝒪 ( 1/𝜀2 ) . However , previous result of DoubleSqueeze is 𝒪 ( Δ/𝜀3 ) and it also uses more strict assumption for the compressors ( E [ ‖𝒞 ( 𝑥 ) − 𝑥‖ ] ≤ ∆ ) . Even if we ignore this , our results for EF21-BC is better than the one for DoubleSqueeze by a large factor 1/𝜀 . ∙ Similarly , our result for EF21-HB is roughly 𝒪 ( 1/𝜀2 ) ( note that the momentum parameter 𝜂 is usually constant such as 0.2 , 0.4 , 0.9 used in our experiments ) . However , previous result of M-CSER is roughly 𝒪 ( 𝐺/𝜀3 ) and it is proven under an additional bounded gradient assumption . Similarly , our EF21-HB is better by a large factor 1/𝜀 . ∙ For EF21-SGD and EF21-PAGE , we want to reduce the gradient complexity by using ( variancereduced ) stochastic gradients instead of full gradient in the vanilla EF21 . Note that 𝜎2 and ∆inf in EF21-SGD could be much smaller than 𝐺 in Choco-SGD since 𝐺 always depends on the dimension ( and can be even infinite ) , while 𝜎2 and ∆inf are mostly dimension-free parameters ( particularly , they are very small if the functions/data samples are similar/close ) . Thus , for high dimensional problems ( e.g. , deep neural networks ) , EF21-SGD can be better than Choco-SGD . Besides , in the finite-sum case ( 3 ) , especially if the number of data samples 𝑚 on each client is not very large , then EF21-PAGE is much better since its complexity is roughly 𝒪 ( √ 𝑚/𝜀2 ) while EF21-SGD ones is roughly 𝒪 ( 𝜎2/𝜀4 ) . 5 EXPERIMENTS In this section , we consider a logistic regression problem with a non-convex regularizer min 𝑥∈R𝑑 { ︃ 𝑓 ( 𝑥 ) = 1𝑁 𝑁∑︀ 𝑖=1 log ( ︀ 1 + exp ( ︀ −𝑏𝑖𝑎⊤𝑖 𝑥 ) ︀ ) ︀ + 𝜆 𝑑∑︀ 𝑗=1 𝑥2𝑗 1+𝑥2𝑗 } ︃ , ( 12 ) where 𝑎𝑖 ∈ R𝑑 , 𝑏𝑖 ∈ { −1,1 } are the training data , and 𝜆 > 0 is the regularization parameter , which is set to 𝜆 = 0.1 in all experiments . For all methods the stepsizes are initially chosen as the largest stepsize predicted by theory for EF21 ( see Theorem 1 ) , then they are tuned individually for each parameter setting . We provide more details on the datasets , hardware , experimental setups , and additional experiments , including deep learning experiments in Appendix A . Experiment 1 : Fast convergence with variance reduction . In our first experiment , we showcase the computation and communication superiority of EF21-PAGE ( Alg . 3 ) over EF21-SGD . Figure 8 illustrates that , in all cases , EF21-PAGE perfectly reduces the accumulated variance and converges to the desired tolerance , whereas EF21-SGD is stuck at some accuracy level . Moreover , EF21-PAGE turns out to be surprisingly efficient with small bathsizes ( eg , 1.5 % of the local data ) both in terms of the number of epochs and the # bits sent to the server per client . Interestingly , for most datasets , a further increase of bathsize does not considerably improve the convergence . Experiment 2 : On the effect of partial participation of clients . This experiment shows that EF21-PP ( Alg . 4 ) can reduce communication costs and can be more practical than EF21 . For this comparison , we consider 𝑛 = 100 and , therefore , apply a different data partitioning , see Table 5 from Appendix A for more details . It is predicted by our theory ( Corollary 1 ) that , in terms of the number of iterations/communication rounds , partial participation slows down the convergence of EF21 by a fraction of participating clients . We observe this behavior in practice as well ( see Figure 2a ) . However , since for EF21-PP the communications are considerably cheaper it outperforms EF21 in terms of # number of bits sent to the server per client on average ( see Figure 2 ) . REFERENCES Dan Alistarh , Demjan Grubic , Jerry Li , Ryota Tomioka , and Milan Vojnovic . QSGD : Communicationefficient SGD via gradient quantization and encoding . In Advances in Neural Information Processing Systems ( NIPS ) , pp . 1709–1720 , 2017 . Dan Alistarh , Torsten Hoefler , Mikael Johansson , Sarit Khirirat , Nikola Konstantinov , and Cédric Renggli . The convergence of sparsified gradient methods . In Advances in Neural Information Processing Systems ( NeurIPS ) , 2018 . Zeyuan Allen-Zhu . Katyusha : The first direct acceleration of stochastic gradient methods . In Proceedings of the 49th Annual ACM SIGACT Symposium on Theory of Computing , pp . 1200– 1205 . ACM , 2017 . Yossi Arjevani , Yair Carmon , John C Duchi , Dylan J Foster , Nathan Srebro , and Blake Woodworth . Lower bounds for non-convex stochastic optimization . arXiv preprint arXiv:1912.02365 , 2019 . Sanjeev Arora , Nadav Cohen , and Elad Hazan . On the optimization of deep networks : Implicit acceleration by overparameterization . In Proceedings of the 35th International Conference on Machine Learning ( ICML ) , 2018 . Amir Beck . First-Order Methods in Optimization . Society for Industrial and Applied Mathematics , 2017 . Aleksandr Beznosikov , Samuel Horváth , Peter Richtárik , and Mher Safaryan . On biased compression for distributed learning . arXiv preprint arXiv:2002.12410 , 2020 . Léon Bottou . Curiously fast convergence of some stochastic gradient descent algorithms . In Proceedings of the symposium on learning and data science , Paris , volume 8 , pp . 2624–2633 , 2009 . Léon Bottou . Stochastic gradient descent tricks . In Neural networks : Tricks of the trade , pp . 421–436 . Springer , 2012 . Chih-Chung Chang and Chih-Jen Lin . LIBSVM : a library for support vector machines . ACM Transactions on Intelligent Systems and Technology ( TIST ) , 2 ( 3 ) :1–27 , 2011 . Yae Jee Cho , Jianyu Wang , and Gauri Joshi . Client selection in federated learning : Convergence analysis and power-of-choice selection strategies . arXiv preprint arXiv:2010.01243v1 , 2020 . Eduard Gorbunov , Dmitry Kovalev , Dmitry Makarenko , and Peter Richtárik . Linearly converging error compensated SGD . In 34th Conference on Neural Information Processing Systems ( NeurIPS ) , 2020 . Eduard Gorbunov , Konstantin Burlachenko , Zhize Li , and Peter Richtárik . MARINA : Faster nonconvex distributed learning with compression . In International Conference on Machine Learning , pp . 3788–3798 . PMLR , 2021. arXiv:2102.07845 . Robert M Gower , Mark Schmidt , Francis Bach , and Peter Richtárik . Variance-reduced methods for machine learning . Proceedings of the IEEE , 108 ( 11 ) :1968–1983 , 2020 . Robert Mansel Gower , Nicolas Loizou , Xun Qian , Alibek Sailanbayev , Egor Shulgin , and Peter Richtárik . SGD : General analysis and improved rates . In International Conference on Machine Learning , pp . 5200–5209 . PMLR , 2019 . Priya Goyal , Piotr Dollár , Ross Girshick , Pieter Noordhuis , Lukasz Wesolowski , Aapo Kyrola , Andrew Tulloch , Yangqing Jia , and Kaiming He . Accurate , large minibatch sgd : Training imagenet in 1 hour . arXiv preprint arXiv:1706.02677 , 2017 . Kaiming He , Xiangyu Zhang , Shaoqing Ren , and Jian Sun . Deep residual learning for image recognition . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition ( CVPR ) , pp . 770–778 , 2016 . Samuel Horváth and Peter Richtárik . A better alternative to error feedback for communicationefficient distributed learning . In 9th International Conference on Learning Representations ( ICLR ) , 2021 . Samuel Horváth , Chen-Yu Ho , L ’ udovít Horváth , Atal Narayan Sahu , Marco Canini , and Peter Richtárik . Natural compression for distributed deep learning . arXiv preprint arXiv:1905.10988 , 2019a . Samuel Horváth , Dmitry Kovalev , Konstantin Mishchenko , Sebastian Stich , and Peter Richtárik . Stochastic distributed learning with gradient quantization and variance reduction . arXiv preprint arXiv:1904.05115 , 2019b . Rustem Islamov , Xun Qian , and Peter Richtárik . Distributed second order methods with fast rates and compressed communication . arXiv preprint arXiv:2102.07158 , 2021 . Peter et al Kairouz . Advances and open problems in federated learning . arXiv preprint arXiv:1912.04977 , 2019 . Sai Praneeth Karimireddy , Quentin Rebjock , Sebastian Stich , and Martin Jaggi . Error feedback fixes SignSGD and other gradient compression schemes . In 36th International Conference on Machine Learning ( ICML ) , 2019 . Sai Praneeth Karimireddy , Satyen Kale , Mehryar Mohri , Sashank Reddi , Sebastian Stich , and Ananda Theertha Suresh . SCAFFOLD : Stochastic controlled averaging for federated learning . In Proceedings of the 37th International Conference on Machine Learning , 2020 . Ahmed Khaled and Peter Richtárik . Gradient descent with compressed iterates . In NeurIPS Workshop on Federated Learning for Data Privacy and Confidentiality , 2019 . Ahmed Khaled and Peter Richtárik . Better theory for SGD in the nonconvex world . arXiv preprint arXiv:2002.03329 , 2020 . Sarit Khirirat , Hamid Reza Feyzmahdavian , and Mikael Johansson . Distributed learning with compressed gradients . arXiv preprint arXiv:1806.06573 , 2018 . Diederik P Kingma and Jimmy Ba . Adam : A method for stochastic optimization . arXiv preprint arXiv:1412.6980 , 2014 . Anastasia Koloskova , Tao Lin , S. Stich , and Martin Jaggi . Decentralized deep learning with arbitrary communication compression . In International Conference on Learning Representations ( ICLR ) , 2020 . Jakub Konečný , H. Brendan McMahan , Felix Yu , Peter Richtárik , Ananda Theertha Suresh , and Dave Bacon . Federated learning : strategies for improving communication efficiency . In NIPS Private Multi-Party Machine Learning Workshop , 2016 . Alex Krizhevsky , Geoffrey Hinton , et al . Learning multiple layers of features from tiny images . Technical report , University of Toronto , Toronto , 2009 . Guanghui Lan and Yi Zhou . An optimal randomized incremental gradient method . arXiv preprint arXiv:1507.02000 , 2015 . Guanghui Lan , Zhize Li , and Yi Zhou . A unified variance-reduced accelerated gradient method for convex optimization . In Advances in Neural Information Processing Systems , pp . 10462–10472 , 2019 . Zhize Li . ANITA : An optimal loopless accelerated variance-reduced gradient method . arXiv preprint arXiv:2103.11333 , 2021a . Zhize Li . A short note of page : Optimal convergence rates for nonconvex optimization . arXiv preprint arXiv:2106.09663 , 2021b . Zhize Li and Jian Li . A simple proximal stochastic gradient method for nonsmooth nonconvex optimization . In Advances in Neural Information Processing Systems ( NeurIPS ) , pp . 5569–5579 , 2018 . Zhize Li and Peter Richtárik . A unified analysis of stochastic gradient methods for nonconvex federated optimization . arXiv preprint arXiv:2006.07013 , 2020 . Zhize Li and Peter Richtárik . CANITA : Faster rates for distributed convex optimization with communication compression . arXiv preprint arXiv:2107.09461 , 2021a . Zhize Li and Peter Richtárik . ZeroSARAH : Efficient nonconvex finite-sum optimization with zero full gradient computation . arXiv preprint arXiv:2103.01447 , 2021b . Zhize Li , Dmitry Kovalev , Xun Qian , and Peter Richtárik . Acceleration for compressed gradient descent in distributed and federated optimization . In International Conference on Machine Learning ( ICML ) , pp . 5895–5904 . PMLR , 2020 . Zhize Li , Hongyan Bao , Xiangliang Zhang , and Peter Richtárik . PAGE : A simple and optimal probabilistic gradient estimator for nonconvex optimization . In International Conference on Machine Learning ( ICML ) , pp . 6286–6295 . PMLR , 2021. arXiv:2008.10898 . Nicolas Loizou and Peter Richtárik . Momentum and stochastic momentum for stochastic gradient , Newton , proximal point and subspace descent methods . Computational Optimization and Applications , 77:653–710 , 2020 . Stanislaw Lojasiewicz . A topological property of real analytic subsets . Coll . du CNRS , Les équations aux dérivées partielles , 117 ( 87-89 ) :2 , 1963 . Konstantin Mishchenko , Eduard Gorbunov , Martin Takáč , and Peter Richtárik . Distributed learning with compressed gradient differences . arXiv preprint arXiv:1901.09269 , 2019 . Konstantin Mishchenko , Ahmed Khaled , and Peter Richtarik . Random reshuffling : Simple analysis with vast improvements . In H. Larochelle , M. Ranzato , R. Hadsell , M. F. Balcan , and H. Lin ( eds . ) , Advances in Neural Information Processing Systems , volume 33 , pp . 17309–17320 . Curran Associates , Inc. , 2020 . URL https : //proceedings.neurips.cc/paper/2020/file/ c8cc6e90ccbff44c9cee23611711cdc4- [ ] Paper.pdf . Yurii Nesterov . A method for unconstrained convex minimization problem with the rate of convergence o ( 1/kˆ 2 ) . In Doklady AN USSR , volume 269 , pp . 543–547 , 1983 . Adam Paszke , Sam Gross , Francisco Massa , Adam Lerer , James Bradbury , Gregory Chanan , Trevor Killeen , Zeming Lin , Natalia Gimelshein , Luca Antiga , et al . Pytorch : An imperative style , high-performance deep learning library . In Advances in Neural Information Processing Systems ( NeurIPS ) , 2019 . Constantin Philippenko and Aymeric Dieuleveut . Bidirectional compression in heterogeneous settings for distributed or federated learning with partial participation : tight convergence guarantees . arXiv preprint arXiv:2006.14591 , 2020 . Boris T Polyak . Gradient methods for the minimisation of functionals . USSR Computational Mathematics and Mathematical Physics , 3 ( 4 ) :864–878 , 1963 . Boris T Polyak . Some methods of speeding up the convergence of iteration methods . Ussr computational mathematics and mathematical physics , 4 ( 5 ) :1–17 , 1964 . Xun Qian , Peter Richtárik , and Tong Zhang . Error compensated distributed SGD can be accelerated . arXiv preprint arXiv:2010.00091 , 2020 . Zheng Qu and Peter Richtárik . Coordinate descent with arbitrary sampling ii : Expected separable overapproximation . arXiv preprint arXiv:1412.8063 , 2014 . Peter Richtárik , Igor Sokolov , and Ilyas Fatkhullin . EF21 : A new , simpler , theoretically better , and practically faster error feedback . arXiv preprint arXiv:2106.05203 , 2021 . Mher Safaryan , Rustem Islamov , Xun Qian , and Peter Richtárik . FedNL : Making Newton-type methods applicable to federated learning . arXiv preprint arXiv:2106.02969 , 2021 . Frank Seide , Hao Fu , Jasha Droppo , Gang Li , and Dong Yu . 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs . In Fifteenth Annual Conference of the International Speech Communication Association , 2014 . Shai Shalev-Shwartz and Shai Ben-David . Understanding machine learning : from theory to algorithms . Cambridge University Press , 2014 . Sebastian U. Stich , J.-B . Cordonnier , and Martin Jaggi . Sparsified SGD with memory . In Advances in Neural Information Processing Systems ( NeurIPS ) , 2018 . Hanlin Tang , Xiangru Lian , Chen Yu , Tong Zhang , and Ji Liu . DoubleSqueeze : Parallel stochastic gradient descent with double-pass error-compensated compression . In Proceedings of the 36th International Conference on Machine Learning ( ICML ) , 2020 . Thijs Vogels , Sai Praneeth Karimireddy , and Martin Jaggi . PowerSGD : Practical low-rank gradient compression for distributed optimization . In Neural Information Processing Systems , 2019 . Zhe Wang , Kaiyi Ji , Yi Zhou , Yingbin Liang , and Vahid Tarokh . Spiderboost and momentum : Faster stochastic variance reduction algorithms . arXiv preprint arXiv:1810.10690 , 2018 . Cong Xie , Shuai Zheng , Oluwasanmi Koyejo , Indranil Gupta , Mu Li , and Haibin Lin . CSER : Communication-efficient SGD with error reset . In Advances in Neural Information Processing Systems ( NeurIPS ) , pp . 12593–12603 , 2020 . Haibo Yang , Minghong Fang , and Jia Liu . Achieving linear speedup with partial worker participation in non-iid federated learning . arXiv preprint arXiv:2101.11203v3 , 2021 . Tianbao Yang , Qihang Lin , and Zhe Li . Unified convergence analysis of stochastic momentum methods for convex and non-convex optimization . arXiv preprint arXiv:1604.03257 , 2016 . Yang You , Jing Li , Sashank Reddi , Jonathan Hseu , Sanjiv Kumar , Srinadh Bhojanapalli , Xiaodan Song , James Demmel , Kurt Keutzer , and Cho-Jui Hsieh . Large batch optimization for deep learning : Training bert in 76 minutes . In International Conference on Learning Representations , 2020 . URL https : //openreview.net/forum ? id=Syx4wnEtvH . Haoyu Zhao , Zhize Li , and Peter Richtárik . FedPAGE : A fast local stochastic gradient method for communication-efficient federated learning . arXiv preprint arXiv:2108.04755 , 2021 . TABLE OF CONTENTS 1 Introduction 1 2 Our Contributions 2 3 Methods 4 4 Theoretical Convergence Results 6 4.1 Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 4.1.1 General Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 4.1.2 Addtional Assumptions for EF21-SGD . . . . . . . . . . . . . . . . . . . 6 4.1.3 Additional Assumptions for EF21-PAGE . . . . . . . . . . . . . . . . . . . 7 4.2 Main Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 5 Experiments 8 A Extra Experiments 15 A.1 Non-Convex Logistic Regression : Additional Experiments and Details . . . . . . . 15 A.2 Experiments with Least Squares . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 A.3 Deep Learning Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 B Notations and Assumptions 22 C EF21 23 C.1 Convergence for General Non-Convex Functions . . . . . . . . . . . . . . . . . . 23 C.2 Convergence under Polyak-Łojasiewicz Condition . . . . . . . . . . . . . . . . . . 25 D Stochastic Gradients 27 D.1 Convergence for General Non-Convex Functions . . . . . . . . . . . . . . . . . . 28 D.2 Convergence under Polyak-Łojasiewicz Condition . . . . . . . . . . . . . . . . . . 32 E Variance Reduction 35 E.1 Convergence for General Non-Convex Functions . . . . . . . . . . . . . . . . . . 37 E.2 Convergence under Polyak-Łojasiewicz Condition . . . . . . . . . . . . . . . . . . 41 F Partial Participation 44 F.1 Convergence for General Non-Convex Functions . . . . . . . . . . . . . . . . . . 47 F.2 Convergence under Polyak-Łojasiewicz Condition . . . . . . . . . . . . . . . . . . 48 G Bidirectional Compression 49 G.1 Convergence for General Non-Convex Functions . . . . . . . . . . . . . . . . . . 51 G.2 Convergence under Polyak-Łojasiewicz Condition . . . . . . . . . . . . . . . . . . 53 H Heavy Ball Momentum 55 H.1 Convergence for General Non-Convex Functions . . . . . . . . . . . . . . . . . . 57 I Composite Case 63 I.1 Convergence for General Non-Convex Functions . . . . . . . . . . . . . . . . . . 65 I.2 Convergence under Polyak-Łojasiewicz Condition . . . . . . . . . . . . . . . . . . 67 J Useful Auxiliary Results 70 J.1 Basic Facts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 J.2 Useful Lemmas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 A EXTRA EXPERIMENTS In this section , we give missing details on the experiments from Section 5 , and provide additional experiments . A.1 NON-CONVEX LOGISTIC REGRESSION : ADDITIONAL EXPERIMENTS AND DETAILS Datasets , hardware and implementation . We use standard LibSVM datasets ( Chang & Lin , 2011 ) , and split each dataset among 𝑛 clients . For experiments 1 , 3 , 4 and 5 , we chose 𝑛 = 20 whereas for the experiment 2 we consider 𝑛 = 100 . The first 𝑛−1 clients own equal parts , and the remaining part , of size 𝑁 − 𝑛 · ⌊𝑁/𝑛⌋ , is assigned to the last client . We consider the heterogeneous data distribution regime ( i.e . we do not make any additional assumptions on data similarity between workers ) . A summary of datasets and details of splitting data among workers can be found in Tables 3 and 5 . The algorithms are implemented in Python 3.8 ; we use 3 different CPU cluster node types in all experiments : 1 ) AMD EPYC 7702 64-Core ; 2 ) Intel ( R ) Xeon ( R ) Gold 6148 CPU @ 2.40GHz ; 3 ) Intel ( R ) Xeon ( R ) Gold 6248 CPU @ 2.50GHz . In all algorithms involving compression , we use Top-𝑘 ( Alistarh et al. , 2017 ) as a canonical example of contractive compressor 𝒞 , and fix the compression ratio 𝑘/𝑑 ≈ 0.01 , where 𝑑 is the number of features in the dataset . For all algorithms , at each iteration we compute the squared norm of the exact/full gradient for comparison of the methods performance . We terminate our algorithms either if they reach the certain number of iterations or the following stopping criterion is satisfied : ‖∇𝑓 ( 𝑥𝑡 ) ‖2 ≤ 10−7 . In all experiments , the stepsize is set to the largest stepsize predicted by theory for EF21 multiplied by some constant multiplier which was individually tuned in all cases . Experiment 1 : Fast convergence with variance reductions ( extra details ) . The parameters 𝑝𝑖 of the PAGE estimator are set to 𝑝𝑖 = 𝑝 def = 1𝑛 ∑︀𝑛 𝑖=1 𝜏𝑖 𝜏𝑖+𝑁𝑖 , where 𝜏𝑖 is the batchsize for clients 𝑖 = 1 , . . . , 𝑛 ( see Table 4 for details ) . In our experiments , we assume that the sampling of Bernoulli random variable is performed on server side ( which means that at each iteration for all clients 𝑏𝑡𝑖 = 1 or 𝑏𝑡𝑖 = 0 ) . And if 𝑏 𝑡 𝑖 = 0 , then in line 5 of Algorithm 3 𝐼 𝑡 𝑖 is sampled without replacement uniformly at random . Table 4 shows the selection of parameter 𝑝 for each experiment . For each batchsize from the set2 { 95 % , 50 % , 25 % ,12.5 % , 6.5 % ,3 % } , we tune the stepsize multiplier for EF21-PAGE within the set { 0.25 , 0.5 , 1 , 2 , 4 , 8 , 16 , 32 , 64 , 128 , 256 , 512 , 1024 , 2048 } . The best pair ( batchsize , stepsize multiplier ) is chosen in such a way that it gives the best convergence in terms of # bits/𝑛 ( 𝐶 → 𝑆 ) . In the rest of the experiments , fine tuning is performed in a similar fashion . 2By 50 % , 25 % ( and so on ) we refer to a batchsize , which is equals to ⌊0.5𝑁𝑖⌋ , ⌊0.25𝑁𝑖⌋ ( and so on ) for all clients 𝑖 = 1 , . . . , 𝑛 . We tune the stepsize multiplier for EF21-PP within the following set : { 0.125,0.25 , 0.5 , 1 , 2 , 4 , 8 , 16 , 32 , 64 , 128 , 256 , 512 , 1024 , 2048 , 4096 } . Experiment 3 : On the advantages of bidirectional biased compression . Our next experiment demonstrates that the application of the Server → Clients compression in EF21-BC ( Alg . 5 ) does not significantly slow down the convergence in terms of the communication rounds but requires much less bits to be transmitted . Indeed , Figure 3a illustrates that that it is sufficient to communicate only 5 % − 15 % of data to perform similarly to EF21 ( Alg . 1 ) .3 Note that EF21 communicates full vectors from the Server → Clients , and , therefore , may have slower communication at each round . In Figure 3b we take into account only the number of bits sent from clients to the server , and therefore we observe the same behavior as in Figure 3a . However , if we care about the total number of bits ( see Figure 3c ) , then EF21-BC considerably outperforms EF21 in all cases . 3The range 5 % − 15 % comes from the fractions 𝑘/𝑑 for each dataset . For each parameter 𝑘 in Server-Clients compression , we tune the stepsize multiplier for EF21-BC within the following set : { 0.125,0.25 , 0.5 , 1 , 2 , 4 , 8 , 16 , 32 , 64 , 128 , 256 , 512 , 1024 , 2048 } . Experiment 4 : On the cheaper computations via EF21-SGD . The fourth experiment ( see Figure 4a ) illustrates that EF21-SGD ( Alg . 2 ) is the more preferable choice than EF21 for the cases when full gradient computations are costly . For each batchsize from the set4 { 95 % , 50 % , 25 % ,12.5 % , 6.5 % ,3 % } , we tune the stepsize multiplier for EF21-SGD within the following set : { 0.25 , 0.5 , 1 , 2 , 4 , 8 , 16 , 32 , 64 , 128 , 256 , 512 , 1024 , 2048 } . Figure 4a illustrates that EF21-SGD is able to reach a moderate tolerance in 5 − 10 epochs . 4By 50 % , 25 % ( and so on ) we refer to a batchsize , which is equals to ⌊0.5𝑁𝑖⌋ , ⌊0.25𝑁𝑖⌋ ( and so on ) for all clients 𝑖 = 1 , . . . , 𝑛 . However , due to the accumulated variance introduced by SGD , estimator EF21-SGD is stuck at some accuracy level ( see Figure 4b ) , showing the usual behavior of the SGD observed in practice . Experiment 5 : On the effect of heavy ball momentum . In this experiment ( see Figure 5 ) , we show that for the majority of the considered datasets heavy ball acceleration used in EF21-HB ( Alg . 6 ) improves the convergence of EF21 method . For every dataset ( and correspondingly chosen parameter 𝑘 ) we tune momentum parameter 𝜂 in EF21-HB by making a grid search over all possible parameter values from 0.05 to 0.99 with the step 0.05 . Finally , for our plots we pick 𝜂 ∈ { 0.05 , 0.2 , 0.25 , 0.4 , 0.9 } since the first four values shows the best performance and 𝜂 = 0.9 is a popular choice in practice . For each parameter 𝜂 from the set { 0.05,0.1,0.15,0.2,0.25,0.3,0.35,0.4,0.45,0.5,0.55,0.6,0.65,0.7,0.75,0.8,0.85,0.9,0.95,0.99 } . we perform a grid search of stepsize multiplier within the powers of 2 : { 0.125 , 0.25 , 0.5 , 1 , 2 , 4 , 8 , 16 , 32 , 64 , 128 , 256 , 512 , 1024 , 2048 } . Experiments on a larger dataset . In these additional experiments , we test our methods on larger problem and dataset . The dimension of the dataset used in these experiments is 𝑑 = 20958 . Each method is run for 500 epochs . In this case , we observe a similar behavior as in our previous experiments . Comparison to non-compressed methods . In addition , we compare EF21-PAGE and EF21-SGD to the baseline methods without compression : PAGE ( Figure 8a ) and SGD ( Figure 9a ) . In these experiments , we observe that EF21-PAGE and EF21-SGD require much less information to transmit in order to achieve the same accuracy of the solution as the methods without compression ( PAGE , SGD ) . A.2 EXPERIMENTS WITH LEAST SQUARES In this section , we conduct the experiments on a function satisfying the PŁ-condition ( see Assumption 4 ) . In particular , we consider the least squares problem : min 𝑥∈R𝑑 { ︃ 𝑓 ( 𝑥 ) = 1 𝑛 𝑛∑︁ 𝑖=1 ( 𝑎⊤𝑖 𝑥− 𝑏𝑖 ) 2 } ︃ , where 𝑎𝑖 ∈ R𝑑 , 𝑏𝑖 ∈ { −1,1 } are the training data . We use the same datasets as for the logistic regression problem . Experiment : On the effect of heavy ball momentum in PŁ-setting . For PŁ-setting , EF21-HB also improves the convergence over EF21 for the majority of the datasets ( see Figure 10 ) . Stepsize and momentum parameter 𝜂 are chosen using the same strategy as for the logistic regression experiments ( see section A.1 ) . A.3 DEEP LEARNING EXPERIMENTS In this experiment , the exact/full gradient ∇𝑓𝑖 ( 𝑥𝑘+1 ) in the algorithm EF21-HB is replaced by its stochastic estimator ( we later refer to this method as EF21-SGD-HB ) . We compare the resulting method with some existing baselines on a deep learning multi-class image classification task . In particular , we compare our EF21-SGD-HB method to EF21+-SGD-HB5 , EF-SGD-HB6 , EF21-SGD 5EF21+-SGD-HB is the method obtained from EF21-SGD-HB via replacing EF21 by EF21+ compressor 6EF-SGD-HB is the method obtained from EF21-SGD-HB via replacing EF21 by EF compressor and EF-SGD on the problem of training ResNet18 ( He et al. , 2016 ) model on CIFAR-10 ( Krizhevsky et al. , 2009 ) dataset . For more details about the EF21+ and EF type methods and their applications in deep learning we refer reader to ( Richtárik et al. , 2021 ) . We implement the algorithms in PyTorch ( Paszke et al. , 2019 ) and run the experiments on a single GPU NVIDIA GeForce RTX 2080 Ti . The dataset is split into 𝑛 = 8 equal parts . Total train set size for CIFAR-10 is 50,000 . The test set for evaluation has 10,000 data points . The train set is split into batches of size 𝜏 = 32 . The first seven workers own an equal number of batches of data , while the last worker gets the rest . In our experiments , we fix 𝑘 ≈ 0.05𝑑 , 𝜏 = 32 and momentum parameter 𝜂 = 0.9.7 As it is usually done in deep learning applications , stochastic gradients are generated via so-called “ shuffle once ” strategy , i.e. , workers randomly shuffle their datasets and then select minibatches using the obtained order ( Bottou , 2009 ; 2012 ; Mishchenko et al. , 2020 ) . We tune the stepsize 𝛾 within the range { 0.0625 , 0.125 , 0.25 , 0.5 , 1 } and for each method we individually chose the one 𝛾 giving the highest accuracy score on test . For momentum methods , the best stepsize was 0.5 , whereas for the non-momentum ones it was 0.125 . The experiments show ( see Figure 11 ) that the train loss for momentum methods decreases slower than for the non-momentum ones , whereas for the test loss situation is the opposite . Finally , momentum methods show a considerable improvement in the accuracy score on the test set over the existing EF21-SGD and EF-SGD . 7Here , 𝑑 is the number of model parameters . For ResNet18 , 𝑑 = 11,511,784 . B NOTATIONS AND ASSUMPTIONS We now introduce an additional assumption , which enables us to obtain a faster linear convergence result in different settings . Assumption 4 ( Polyak-Łojasiewicz ) . There exists 𝜇 > 0 such that 𝑓 ( 𝑥 ) − 𝑓 ( 𝑥⋆ ) ≤ 12𝜇 ‖∇𝑓 ( 𝑥 ) ‖ 2 for all 𝑥 ∈ R𝑑 , where 𝑥⋆ = arg min𝑥∈R𝑑 𝑓 . Table 6 summarizes the most frequently used notations in our analysis . Additionally , we comment on the main quantities here . We define 𝛿𝑡 def= 𝑓 ( 𝑥𝑡 ) −𝑓 inf 8 , 𝑅𝑡 def= ⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 . In the analysis of EF21- HB , it is useful to adapt this notation to 𝑅𝑡 def= ( 1 − 𝜂 ) 2 ⃦⃦ 𝑧𝑡+1 − 𝑧𝑡 ⃦⃦2 , where { 𝑧𝑡 } 𝑡≥0 is the sequence of virtual iterates introduced in Section H. We denote 𝐺𝑡𝑖 def = ‖∇𝑓𝑖 ( 𝑥𝑡 ) − 𝑔𝑡𝑖‖ 2 , 𝐺𝑡 def= 1𝑛 ∑︀𝑛 𝑖=1 𝐺 𝑡 𝑖 following Richtárik et al . ( 2021 ) , where 𝑔𝑡𝑖 is an EF21 estimator at a node 𝑖 . Throughout the paper̃︀𝐿2 def= 1𝑛 ∑︀𝑛𝑖=1 𝐿2𝑖 , where 𝐿𝑖 is a smoothness constant for 𝑓𝑖 ( · ) , 𝑖 = 1 , . . . , 𝑛 ( see Assumption 1 ) . 8If , additionally , Assumption 4 holds , then 𝑓 inf can be replaced by 𝑓 ( 𝑥⋆ ) for 𝑥⋆ = argmin𝑥∈R𝑑 𝑓 ( 𝑥 ) . C EF21 For completeness , we provide here the detailed proofs for EF21 ( Richtárik et al. , 2021 ) . Algorithm 1 EF21 1 : Input : starting point 𝑥0 ∈ R𝑑 ; 𝑔0𝑖 ∈ R𝑑 for 𝑖 = 1 , . . . , 𝑛 ( known by nodes ) ; 𝑔0 = 1𝑛 ∑︀𝑛 𝑖=1 𝑔 0 𝑖 ( known by master ) ; learning rate 𝛾 > 0 2 : for 𝑡 = 0,1 , 2 , . . . , 𝑇 − 1 do 3 : Master computes 𝑥𝑡+1 = 𝑥𝑡 − 𝛾𝑔𝑡 and broadcasts 𝑥𝑡+1 to all nodes 4 : for all nodes 𝑖 = 1 , . . . , 𝑛 in parallel do 5 : Compress 𝑐𝑡𝑖 = 𝒞 ( ∇𝑓𝑖 ( 𝑥𝑡+1 ) − 𝑔𝑡𝑖 ) and send 𝑐𝑡𝑖 to the master 6 : Update local state 𝑔𝑡+1𝑖 = 𝑔 𝑡 𝑖 + 𝑐 𝑡 𝑖 7 : end for 8 : Master computes 𝑔𝑡+1 = 1𝑛 ∑︀𝑛 𝑖=1 𝑔 𝑡+1 𝑖 via 𝑔 𝑡+1 = 𝑔𝑡 + 1𝑛 ∑︀𝑛 𝑖=1 𝑐 𝑡 𝑖 9 : end for Lemma 1 . Let 𝒞 be a contractive compressor , then for all 𝑖 = 1 , . . . , 𝑛 E [ ︀ 𝐺𝑡+1𝑖 ] ︀ ≤ ( 1 − 𝜃 ) E [ ︀ 𝐺𝑡𝑖 ] ︀ + 𝛽𝐿2𝑖E [ ︁⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 ] ︁ , and ( 13 ) E [ ︀ 𝐺𝑡+1 ] ︀ ≤ ( 1 − 𝜃 ) E [ ︀ 𝐺𝑡 ] ︀ + 𝛽̃︀𝐿2E [ ︁⃦⃦𝑥𝑡+1 − 𝑥𝑡⃦⃦2 ] ︁ , ( 14 ) where 𝜃 def = 1 − ( 1 − 𝛼 ) ( 1 + 𝑠 ) , 𝛽 def= ( 1 − 𝛼 ) ( ︀ 1 + 𝑠−1 ) ︀ for any 𝑠 > 0 . Proof . Define 𝑊 𝑡 def= { 𝑔𝑡1 , . . . , 𝑔𝑡𝑛 , 𝑥𝑡 , 𝑥𝑡+1 } , then E [ ︀ 𝐺𝑡+1𝑖 ] ︀ = E [ ︀ E [ ︀ 𝐺𝑡+1𝑖 | 𝑊 𝑡 ] ︀ ] ︀ = E [ ︁ E [ ︁⃦⃦ 𝑔𝑡+1𝑖 −∇𝑓𝑖 ( 𝑥 𝑡+1 ) ⃦⃦2 | 𝑊 𝑡 ] ︁ ] ︁ = E [ ︁ E [ ︁⃦⃦ 𝑔𝑡𝑖 + 𝒞 ( ∇𝑓𝑖 ( 𝑥𝑡+1 ) − 𝑔𝑡𝑖 ) −∇𝑓𝑖 ( 𝑥𝑡+1 ) ⃦⃦2 | 𝑊 𝑡 ] ︁ ] ︁ ( 8 ) ≤ ( 1 − 𝛼 ) E [ ︁⃦⃦ ∇𝑓𝑖 ( 𝑥𝑡+1 ) − 𝑔𝑡𝑖 ⃦⃦2 ] ︁ ( 𝑖 ) ≤ ( 1 − 𝛼 ) ( 1 + 𝑠 ) E [ ︁⃦⃦ ∇𝑓𝑖 ( 𝑥𝑡 ) − 𝑔𝑡𝑖 ⃦⃦2 ] ︁ + ( 1 − 𝛼 ) ( ︀ 1 + 𝑠−1 ) ︀ ⃦⃦ ∇𝑓𝑖 ( 𝑥𝑡+1 ) −∇𝑓𝑖 ( 𝑥𝑡 ) ⃦⃦2 ( 15 ) ( 𝑖𝑖 ) ≤ ( 1 − 𝛼 ) ( 1 + 𝑠 ) E [ ︁⃦⃦ ∇𝑓𝑖 ( 𝑥𝑡 ) − 𝑔𝑡𝑖 ⃦⃦2 ] ︁ + ( 1 − 𝛼 ) ( ︀ 1 + 𝑠−1 ) ︀ 𝐿2𝑖E [ ︁⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 ] ︁ ( 𝑖𝑖𝑖 ) ≤ ( 1 − 𝜃 ) E [ ︁⃦⃦ ∇𝑓𝑖 ( 𝑥𝑡 ) − 𝑔𝑡𝑖 ⃦⃦2 ] ︁ + 𝛽𝐿2𝑖E [ ︁⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 ] ︁ , where ( 𝑖 ) follows by Young ’ s inequality ( 118 ) , ( 𝑖𝑖 ) holds by Assumption 1 , and in ( 𝑖𝑖𝑖 ) we apply the definition of 𝜃 and 𝛽 . Averaging the above inequalities over 𝑖 = 1 , . . . , 𝑛 , we obtain ( 14 ) . C.1 CONVERGENCE FOR GENERAL NON-CONVEX FUNCTIONS Theorem 1 . Let Assumption 1 hold , and let the stepsize in Algorithm 1 be set as 0 < 𝛾 ≤ ( ︃ 𝐿 + ̃︀𝐿√︂𝛽 𝜃 ) ︃−1 . ( 16 ) Fix 𝑇 ≥ 1 and let ? ̂ ? 𝑇 be chosen from the iterates 𝑥0 , 𝑥1 , . . . , 𝑥𝑇−1 uniformly at random . Then E [ ︁⃦⃦ ∇𝑓 ( ? ̂ ? 𝑇 ) ⃦⃦2 ] ︁ ≤ 2 ( ︀𝑓 ( 𝑥0 ) − 𝑓 inf ) ︀ 𝛾𝑇 + E [ ︀ 𝐺0 ] ︀ 𝜃𝑇 , ( 17 ) where ̃︀𝐿 = √︁ 1𝑛 ∑︀𝑛𝑖=1 𝐿2𝑖 , 𝜃 = 1 − ( 1 − 𝛼 ) ( 1 + 𝑠 ) , 𝛽 = ( 1 − 𝛼 ) ( ︀1 + 𝑠−1 ) ︀ for any 𝑠 > 0 . Proof . According to our notation , for Algorithm 1 𝑅𝑡 = ⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 . By Lemma 1 , we have E [ ︀ 𝐺𝑡+1 ] ︀ ≤ ( 1 − 𝜃 ) E [ ︀ 𝐺𝑡 ] ︀ + 𝛽̃︀𝐿2E [ ︀𝑅𝑡 ] ︀ . ( 18 ) Next , using Lemma 16 and Jensen ’ s inequality ( 119 ) , we obtain the bound 𝑓 ( 𝑥𝑡+1 ) ≤ 𝑓 ( 𝑥𝑡 ) − 𝛾 2 ⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 − ( ︂ 1 2𝛾 − 𝐿 2 ) ︂ 𝑅𝑡 + 𝛾 2 ⃦⃦⃦⃦ ⃦ 1𝑛 𝑛∑︁ 𝑖=1 ( ︀ 𝑔𝑡𝑖 −∇𝑓𝑖 ( 𝑥𝑡 ) ) ︀⃦⃦⃦⃦⃦ 2 ≤ 𝑓 ( 𝑥𝑡 ) − 𝛾 2 ⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 − ( ︂ 1 2𝛾 − 𝐿 2 ) ︂ 𝑅𝑡 + 𝛾 2 1 𝑛 𝑛∑︁ 𝑖=1 ⃦⃦ 𝑔𝑡𝑖 −∇𝑓𝑖 ( 𝑥𝑡 ) ⃦⃦2 = 𝑓 ( 𝑥𝑡 ) − 𝛾 2 ⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 − ( ︂ 1 2𝛾 − 𝐿 2 ) ︂ 𝑅𝑡 + 𝛾 2 𝐺𝑡 . ( 19 ) Subtracting 𝑓 inf from both sides of the above inequality , taking expectation and using the notation 𝛿𝑡 = 𝑓 ( 𝑥𝑡 ) − 𝑓 inf , we get E [ ︀ 𝛿𝑡+1 ] ︀ ≤ E [ ︀ 𝛿𝑡 ] ︀ − 𝛾 2 E [ ︁⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 ] ︁− ( ︂ 1 2𝛾 − 𝐿 2 ) ︂ E [ ︀ 𝑅𝑡 ] ︀ + 𝛾 2 E [ ︀ 𝐺𝑡 ] ︀ . ( 20 ) Then by adding ( 20 ) with a 𝛾2𝜃 multiple of ( 18 ) we obtain E [ ︀ 𝛿𝑡+1 ] ︀ + 𝛾 2𝜃 E [ ︀ 𝐺𝑡+1 ] ︀ ≤ E [ ︀ 𝛿𝑡 ] ︀ − 𝛾 2 E [ ︁⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 ] ︁− ( ︂ 1 2𝛾 − 𝐿 2 ) ︂ E [ ︀ 𝑅𝑡 ] ︀ + 𝛾 2 E [ ︀ 𝐺𝑡 ] ︀ + 𝛾 2𝜃 ( ︁ 𝛽̃︀𝐿2E [ ︀𝑅𝑡 ] ︀+ ( 1 − 𝜃 ) E [ ︀𝐺𝑡 ] ︀ ) ︁ = E [ ︀ 𝛿𝑡 ] ︀ + 𝛾 2𝜃 E [ ︀ 𝐺𝑡 ] ︀ − 𝛾 2 E [ ︁⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 ] ︁ − ( ︂ 1 2𝛾 − 𝐿 2 − 𝛾 2𝜃 𝛽̃︀𝐿2 ) ︂E [ ︀𝑅𝑡 ] ︀ ≤ E [ ︀ 𝛿𝑡 ] ︀ + 𝛾 2𝜃 E [ ︀ 𝐺𝑡 ] ︀ − 𝛾 2 E [ ︁⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 ] ︁ . The last inequality follows from the bound 𝛾2 𝛽 ̃︀𝐿2 𝜃 + 𝐿𝛾 ≤ 1 , which holds because of Lemma 15 and our assumption on the stepsize . By summing up inequalities for 𝑡 = 0 , . . . , 𝑇 − 1 , we get 0 ≤ E [ ︁ 𝛿𝑇 + 𝛾 2𝜃 𝐺𝑇 ] ︁ ≤ 𝛿0 + 𝛾 2𝜃 E [ ︀ 𝐺0 ] ︀ − 𝛾 2 𝑇−1∑︁ 𝑡=0 E [ ︁⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 ] ︁ . Multiplying both sides by 2𝛾𝑇 , after rearranging we get 𝑇−1∑︁ 𝑡=0 1 𝑇 E [ ︁⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 ] ︁ ≤ 2𝛿0 𝛾𝑇 + E [ ︀ 𝐺0 ] ︀ 𝜃𝑇 . It remains to notice that the left hand side can be interpreted as E [ ︁⃦⃦ ∇𝑓 ( ? ̂ ? 𝑇 ) ⃦⃦2 ] ︁ , where ? ̂ ? 𝑇 is chosen from 𝑥0 , 𝑥1 , . . . , 𝑥𝑇−1 uniformly at random . Corollary 2 . Let assumptions of Theorem 1 hold , 𝑔0𝑖 = ∇𝑓𝑖 ( 𝑥0 ) , 𝑖 = 1 , . . . , 𝑛 , 𝛾 = ( ︁ 𝐿 + ̃︀𝐿√︀𝛽/𝜃 ) ︁−1 . Then , after 𝑇 iterations/communication rounds of EF21 we have E [ ︁⃦⃦ ∇𝑓 ( ? ̂ ? 𝑇 ) ⃦⃦2 ] ︁ ≤ 𝜀2 . It requires 𝑇 = # grad = 𝒪 ( ︃ ̃︀𝐿𝛿0 𝛼𝜀2 ) ︃ iterations/communications rounds/gradint computations at each node , where ̃︀𝐿 = √︁ 1𝑛 ∑︀𝑛𝑖=1 𝐿2𝑖 , 𝛿0 = 𝑓 ( 𝑥0 ) − 𝑓 𝑖𝑛𝑓 . Proof . Since 𝑔0𝑖 = ∇𝑓𝑖 ( 𝑥0 ) , 𝑖 = 1 , . . . , 𝑛 , we have 𝐺0 = 0 and by Theorem 1 # grad = 𝑇 ( 𝑖 ) ≤ 2𝛿 0 𝛾𝜀2 ( 𝑖𝑖 ) ≤ 2𝛿 0 𝜀2 ( ︃ 𝐿 + ̃︀𝐿√︂𝛽 𝜃 ) ︃ ( 𝑖𝑖𝑖 ) ≤ 2𝛿 0 𝜀2 ( ︂ 𝐿 + ̃︀𝐿 ( ︂ 2 𝛼 − 1 ) ︂ ) ︂ ≤ 2𝛿 0 𝜀2 ( ︃ 𝐿 + 2̃︀𝐿 𝛼 ) ︃ ( 𝑖𝑣 ) ≤ 2𝛿 0 𝜀2 ( ︃ ̃︀𝐿 𝛼 + 2̃︀𝐿 𝛼 ) ︃ = 6̃︀𝐿𝛿0 𝛼𝜀2 , where in ( 𝑖 ) is due to the rate ( 17 ) given by Theorem 1 . In two ( 𝑖𝑖 ) we plug in the stepsize , in ( 𝑖𝑖𝑖 ) we use Lemma 17 , and ( 𝑖𝑣 ) follows by the inequalities 𝛼 ≤ 1 , and 𝐿 ≤ ̃︀𝐿 . C.2 CONVERGENCE UNDER POLYAK-ŁOJASIEWICZ CONDITION Theorem 2 . Let Assumptions 1 and 4 hold , and let the stepsize in Algorithm 1 be set as 0 < 𝛾 ≤ min ⎧⎨⎩ ( ︃ 𝐿 + ̃︀𝐿√︂2𝛽 𝜃 ) ︃−1 , 𝜃 2𝜇 ⎫⎬⎭ . ( 21 ) Let Ψ𝑡 def = 𝑓 ( 𝑥𝑡 ) − 𝑓 ( 𝑥⋆ ) + 𝛾𝜃𝐺 𝑡 . Then for any 𝑇 ≥ 0 , we have E [ ︀ Ψ𝑇 ] ︀ ≤ ( 1 − 𝛾𝜇 ) 𝑇E [ ︀ Ψ0 ] ︀ , ( 22 ) where ̃︀𝐿 = √︁ 1𝑛 ∑︀𝑛𝑖=1 𝐿2𝑖 , 𝜃 = 1 − ( 1 − 𝛼 ) ( 1 + 𝑠 ) , 𝛽 = ( 1 − 𝛼 ) ( ︀1 + 𝑠−1 ) ︀ for any 𝑠 > 0 . Proof . We proceed as in the previous proof , but use the PL inequality , subtract 𝑓 ( 𝑥⋆ ) from both sides of ( 19 ) and utilize the notation 𝛿𝑡 = 𝑓 ( 𝑥𝑡 ) − 𝑓 ( 𝑥⋆ ) 𝛿𝑡+1 ≤ 𝛿𝑡 − 𝛾 2 ⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 − ( ︂ 1 2𝛾 − 𝐿 2 ) ︂ 𝑅𝑡 + 𝛾 2 𝐺𝑡 ≤ 𝛿𝑡 − 𝛾𝜇 ( ︀ 𝑓 ( 𝑥𝑡 ) − 𝑓 ( 𝑥⋆ ) ) ︀ − ( ︂ 1 2𝛾 − 𝐿 2 ) ︂ 𝑅𝑡 + 𝛾 2 𝐺𝑡 . = ( 1 − 𝛾𝜇 ) 𝛿𝑡 − ( ︂ 1 2𝛾 − 𝐿 2 ) ︂ 𝑅𝑡 + 𝛾 2 𝐺𝑡 . Take expectation on both sides of the above inequality and add it with a 𝛾𝜃 multiple of ( 18 ) , then E [ ︀ 𝛿𝑡+1 ] ︀ + E [ ︁𝛾 𝜃 𝐺𝑡+1 ] ︁ ≤ ( 1 − 𝛾𝜇 ) E [ ︀ 𝛿𝑡 ] ︀ − ( ︂ 1 2𝛾 − 𝐿 2 ) ︂ E [ ︀ 𝑅𝑡 ] ︀ + 𝛾 2 E [ ︀ 𝐺𝑡 ] ︀ + 𝛾 𝜃 ( ︁ ( 1 − 𝜃 ) E [ ︀ 𝐺𝑡 ] ︀ + 𝛽̃︀𝐿2E [ ︀𝑅𝑡 ] ︀ ) ︁ = ( 1 − 𝛾𝜇 ) E [ ︀ 𝛿𝑡 ] ︀ + 𝛾 𝜃 ( ︂ 1 − 𝜃 2 ) ︂ E [ ︀ 𝐺𝑡 ] ︀ − ( ︃ 1 2𝛾 − 𝐿 2 − 𝛽 ̃︀𝐿2𝛾 𝜃 ) ︃ E [ ︀ 𝑅𝑡 ] ︀ . Note that our assumption on the stepsize implies that 1 − 𝜃2 ≤ 1 − 𝛾𝜇 and 1 2𝛾 − 𝐿 2 − 𝛽̃︀𝐿2𝛾 𝜃 ≥ 0 . The last inequality follows from the bound 𝛾2 2𝛽 ̃︀𝐿2 𝜃 + 𝛾𝐿 ≤ 1 , which holds because of Lemma 15 and our assumption on the stepsize . Thus , E [ ︁ 𝛿𝑡+1 + 𝛾 𝜃 𝐺𝑡+1 ] ︁ ≤ ( 1 − 𝛾𝜇 ) E [ ︁ 𝛿𝑡 + 𝛾 𝜃 𝐺𝑡 ] ︁ . It remains to unroll the recurrence . Corollary 3 . Let assumptions of Theorem 2 hold , 𝑔0𝑖 = ∇𝑓𝑖 ( 𝑥0 ) , 𝑖 = 1 , . . . , 𝑛 , 𝛾 = min ⎧⎨⎩ ( ︃ 𝐿 + ̃︀𝐿√︂2𝛽 𝜃 ) ︃−1 , 𝜃 2𝜇 ⎫⎬⎭ . Then , after 𝑇 iterations/communication rounds of EF21 we have E [ ︀ 𝑓 ( 𝑥𝑇 ) − 𝑓 ( 𝑥⋆ ) ] ︀ ≤ 𝜀 . It requires 𝑇 = # grad = 𝒪 ( ︃ ̃︀𝐿 𝛼𝜇 log ( ︂ 𝛿0 𝜀 ) ︂ ) ︃ ( 23 ) iterations/communications rounds/gradint computations at each node , where ̃︀𝐿 = √︁ 1𝑛 ∑︀𝑛𝑖=1 𝐿2𝑖 , 𝛿0 = 𝑓 ( 𝑥0 ) − 𝑓 𝑖𝑛𝑓 . Proof . Notice that min ⎧⎨⎩ ( ︃ 𝐿 + ̃︀𝐿√︂2𝛽 𝜃 ) ︃−1 , 𝜃 2𝜇 ⎫⎬⎭𝜇 ( 𝑖 ) ≥ min { ︃ 𝜇 ( ︂ 𝐿 + ̃︀𝐿√2 ( ︂ 2 𝛼 − 1 ) ︂ ) ︂−1 , 1 − √ 1 − 𝛼 2 } ︃ ( 𝑖𝑖 ) ≥ min ⎧⎨⎩𝜇 ( ︃ 𝐿 + 2 √ 2̃︀𝐿 𝛼 ) ︃−1 , 𝛼 4 ⎫⎬⎭ ( 𝑖𝑖𝑖 ) ≥ min ⎧⎨⎩𝜇 ( ︃ ( 1 + 2 √ 2 ) ̃︀𝐿 𝛼 ) ︃−1 , 𝛼 4 ⎫⎬⎭ = min { ︃ 𝛼𝜇 ( 1 + 2 √ 2 ) ̃︀𝐿 , 𝛼4 } ︃ ≥ min { ︂ 𝛼𝜇 4̃︀𝐿 , 𝛼4 } ︂ = 𝛼𝜇 4̃︀𝐿 , where in ( 𝑖 ) we apply Lemma 17 , and plug in 𝜃 = 1 − √ 1 − 𝛼 according to Lemma 17 , ( 𝑖𝑖 ) follows by √ 1 − 𝛼 ≤ 1 − 𝛼/2 , ( 𝑖𝑖𝑖 ) follows by the inequalities 𝛼 ≤ 1 , and 𝐿 ≤ ̃︀𝐿 . Let 𝑔0𝑖 = ∇𝑓𝑖 ( 𝑥0 ) , 𝑖 = 1 , . . . , 𝑛 , then 𝐺0 = 0 . Thus using ( 22 ) and the above computations , we arrive at # grad = 𝑇≤ log ( ︁ 𝛿0 𝜀 ) ︁ log ( 1 − 𝛾𝜇 ) −1 ( 𝑖 ) ≤ 1 𝛾𝜇 log ( ︂ 𝛿0 𝜀 ) ︂ ≤ 4 ̃︀𝐿 𝛼𝜇 log ( ︂ 𝛿0 𝜀 ) ︂ , where ( 𝑖 ) is due to ( 122 ) . D STOCHASTIC GRADIENTS In this section , we study the extension of EF21 to the case when stochastic gradients are used instead of full gradients . Algorithm 2 EF21-SGD 1 : Input : starting point 𝑥0 ∈ R𝑑 ; 𝑔0𝑖 ∈ R𝑑 ( known by nodes ) ; 𝑔0 = 1𝑛 ∑︀𝑛 𝑖=1 𝑔 0 𝑖 ( known by master ) ; learning rate 𝛾 > 0 2 : for 𝑡 = 0,1 , 2 , . . . , 𝑇 − 1 do 3 : Master computes 𝑥𝑡+1 = 𝑥𝑡 − 𝛾𝑔𝑡 and broadcasts 𝑥𝑡+1 to all nodes 4 : for all nodes 𝑖 = 1 , . . . , 𝑛 in parallel do 5 : Compute a stochastic gradient 𝑔𝑖 ( 𝑥𝑡+1 ) = 1𝜏 ∑︀𝜏 𝑗=1 ∇𝑓𝜉𝑡𝑖𝑗 ( 𝑥 𝑡+1 ) 6 : Compress 𝑐𝑡𝑖 = 𝒞 ( 𝑔𝑖 ( 𝑥𝑡+1 ) − 𝑔𝑡𝑖 ) and send 𝑐𝑡𝑖 to the master 7 : Update local state 𝑔𝑡+1𝑖 = 𝑔 𝑡 𝑖 + 𝑐 𝑡 𝑖 8 : end for 9 : Master computes 𝑔𝑡+1 = 1𝑛 ∑︀𝑛 𝑖=1 𝑔 𝑡+1 𝑖 via 𝑔 𝑡+1 = 𝑔𝑡 + 1𝑛 ∑︀𝑛 𝑖=1 𝑐 𝑡 𝑖 10 : end for Lemma 2 . Let Assumptions 1 and 2 hold . Then for all 𝑡 ≥ 0 and all constants 𝜌 , 𝜈 > 0 EF21-SGD satisfies E [ ︀ 𝐺𝑡+1 ] ︀ ≤ ( 1 − 𝜃 ) E [ ︀ 𝐺𝑡 ] ︀ + 𝛽1̃︀𝐿2E [ ︁⃦⃦𝑥𝑡+1 − 𝑥𝑡⃦⃦2 ] ︁ + ̃︀𝐴𝛽2E [ ︀𝑓 ( 𝑥𝑡+1 ) − 𝑓 inf ] ︀+ ̃︀𝐶𝛽2 , ( 24 ) where 𝜃 def = 1 − ( 1 − 𝛼 ) ( 1 + 𝜌 ) ( 1 + 𝜈 ) , 𝛽1 def = 2 ( 1 − 𝛼 ) ( 1 + 𝜌 ) ( ︀ 1 + 1𝜈 ) ︀ , 𝛽2 def = 2 ( 1 − 𝛼 ) ( 1 + 𝜌 ) ( ︀ 1 + 1𝜈 ) ︀ + ( ︁ 1 + 1𝜌 ) ︁ , ̃︀𝐴 = max𝑖=1 , ... , 𝑛 2 ( 𝐴𝑖+𝐿𝑖 ( 𝐵𝑖−1 ) ) 𝜏𝑖 , ̃︀𝐶 = 1𝑛 𝑛∑︀ 𝑖=1 ( ︁ 2 ( 𝐴𝑖+𝐿𝑖 ( 𝐵𝑖−1 ) ) 𝜏𝑖 ( ︀ 𝑓 inf − 𝑓 inf𝑖 ) ︀ + 𝐶𝑖𝜏𝑖 ) ︁ . Proof . For all 𝜌 , 𝜈 > 0 we have E [ ︀ 𝐺𝑡+1𝑖 ] ︀ = E [ ︁⃦⃦ 𝑔𝑡+1𝑖 −∇𝑓𝑖 ( 𝑥 𝑡+1 ) ⃦⃦2 ] ︁ ≤ ( 1 + 𝜌 ) E [ ︁⃦⃦ 𝒞 ( ︀ 𝑔𝑖 ( 𝑥 𝑡+1 ) − 𝑔𝑡𝑖 ) ︀ − ( ︀ 𝑔𝑖 ( 𝑥 𝑡+1 ) − 𝑔𝑡𝑖 ) ︀⃦⃦2 ] ︁ + ( ︂ 1 + 1 𝜌 ) ︂ E [ ︁⃦⃦ 𝑔𝑖 ( 𝑥 𝑡+1 ) −∇𝑓𝑖 ( 𝑥𝑡+1 ) ⃦⃦2 ] ︁ ≤ ( 1 − 𝛼 ) ( 1 + 𝜌 ) E [ ︁⃦⃦ 𝑔𝑡𝑖 − 𝑔𝑖 ( 𝑥𝑡+1 ) ⃦⃦2 ] ︁ + ( ︂ 1 + 1 𝜌 ) ︂ E [ ︁⃦⃦ 𝑔𝑖 ( 𝑥 𝑡+1 ) −∇𝑓𝑖 ( 𝑥𝑡+1 ) ⃦⃦2 ] ︁ ≤ ( 1 − 𝛼 ) ( 1 + 𝜌 ) ( 1 + 𝜈 ) E [ ︁⃦⃦ 𝑔𝑡𝑖 −∇𝑓𝑖 ( 𝑥𝑡 ) ⃦⃦2 ] ︁ +2 ( 1 − 𝛼 ) ( 1 + 𝜌 ) ( ︂ 1 + 1 𝜈 ) ︂ E [ ︁⃦⃦ ∇𝑓𝑖 ( 𝑥𝑡+1 ) − 𝑔𝑖 ( 𝑥𝑡+1 ) ⃦⃦2 ] ︁ +2 ( 1 − 𝛼 ) ( 1 + 𝜌 ) ( ︂ 1 + 1 𝜈 ) ︂ E [ ︁⃦⃦ ∇𝑓𝑖 ( 𝑥𝑡+1 ) −∇𝑓𝑖 ( 𝑥𝑡 ) ⃦⃦2 ] ︁ + ( ︂ 1 + 1 𝜌 ) ︂ E [ ︁⃦⃦ 𝑔𝑖 ( 𝑥 𝑡+1 ) −∇𝑓𝑖 ( 𝑥𝑡+1 ) ⃦⃦2 ] ︁ ≤ ( 1 − 𝜃 ) E [ ︀ 𝐺𝑡𝑖 ] ︀ + 𝛽1𝐿 2 𝑖E [ ︁⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 ] ︁ +𝛽2E [ ︁⃦⃦ 𝑔𝑖 ( 𝑥 𝑡+1 ) −∇𝑓𝑖 ( 𝑥𝑡+1 ) ⃦⃦2 ] ︁ , where we introduced 𝜃 def= 1 − ( 1 − 𝛼 ) ( 1 + 𝜌 ) ( 1 + 𝜈 ) , 𝛽1 def = 2 ( 1 − 𝛼 ) ( 1 + 𝜌 ) ( ︀ 1 + 1𝜈 ) ︀ , 𝛽2 def = 2 ( 1 − 𝛼 ) ( 1 + 𝜌 ) ( ︀ 1 + 1𝜈 ) ︀ + ( ︁ 1 + 1𝜌 ) ︁ . Next we use independence of ∇𝑓𝜉𝑡𝑖𝑗 ( 𝑥 𝑡 ) , variance decomposition , and ( 9 ) to estimate the last term : E [ ︀ 𝐺𝑡+1𝑖 ] ︀ ≤ ( 1 − 𝜃 ) E [ ︀ 𝐺𝑡𝑖 ] ︀ + 𝛽1𝐿 2 𝑖E [ ︁⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 ] ︁ + 𝛽2 𝜏2𝑖 𝜏𝑖∑︁ 𝑗=1 E [ ︂⃦⃦⃦ ∇𝑓𝜉𝑡𝑖𝑗 ( 𝑥 𝑡+1 ) −∇𝑓𝑖 ( 𝑥𝑡+1 ) ⃦⃦⃦2 ] ︂ = ( 1 − 𝜃 ) E [ ︀ 𝐺𝑡𝑖 ] ︀ + 𝛽1𝐿 2 𝑖E [ ︁⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 ] ︁ + 𝛽2 𝜏2𝑖 𝜏𝑖∑︁ 𝑗=1 ( ︂ E [ ︂⃦⃦⃦ ∇𝑓𝜉𝑡𝑖𝑗 ( 𝑥 𝑡+1 ) ⃦⃦⃦2 ] ︂ − E [ ︁⃦⃦ ∇𝑓𝑖 ( 𝑥𝑡+1 ) ⃦⃦2 ] ︁ ) ︂ ( 9 ) ≤ ( 1 − 𝜃 ) E [ ︀ 𝐺𝑡𝑖 ] ︀ + 𝛽1𝐿 2 𝑖E [ ︁⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 ] ︁ + 2𝐴𝑖𝛽2 𝜏𝑖 E [ ︀ 𝑓𝑖 ( 𝑥 𝑡+1 ) − 𝑓 inf𝑖 ] ︀ + 𝛽2 ( 𝐵𝑖 − 1 ) 𝜏𝑖 E [ ︁⃦⃦ ∇𝑓𝑖 ( 𝑥𝑡+1 ) ⃦⃦2 ] ︁ + 𝐶𝑖𝛽2 𝜏𝑖 ≤ ( 1 − 𝜃 ) E [ ︀ 𝐺𝑡𝑖 ] ︀ + 𝛽1𝐿 2 𝑖E [ ︁⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 ] ︁ + 2 ( 𝐴𝑖 + 𝐿𝑖 ( 𝐵𝑖 − 1 ) ) 𝛽2 𝜏𝑖 E [ ︀ 𝑓𝑖 ( 𝑥 𝑡+1 ) − 𝑓 inf𝑖 ] ︀ + 𝐶𝑖𝛽2 𝜏𝑖 Averaging the obtained inequality for 𝑖 = 1 , . . . , 𝑛 we get E [ ︀ 𝐺𝑡+1 ] ︀ ≤ ( 1 − 𝜃 ) E [ ︀ 𝐺𝑡 ] ︀ + 𝛽1̃︀𝐿2E [ ︁⃦⃦𝑥𝑡+1 − 𝑥𝑡⃦⃦2 ] ︁ + 1 𝑛 𝑛∑︁ 𝑖=1 ( ︃ 2 ( 𝐴𝑖 + 𝐿𝑖 ( 𝐵𝑖 − 1 ) ) 𝛽2 𝜏𝑖 E [ ︀ 𝑓𝑖 ( 𝑥 𝑡+1 ) − 𝑓 inf𝑖 ] ︀ + 𝐶𝑖𝛽2 𝜏𝑖 ) ︃ ≤ ( 1 − 𝜃 ) E [ ︀ 𝐺𝑡 ] ︀ + 𝛽1̃︀𝐿2E [ ︁⃦⃦𝑥𝑡+1 − 𝑥𝑡⃦⃦2 ] ︁ + 1 𝑛 𝑛∑︁ 𝑖=1 ( ︃ 2 ( 𝐴𝑖 + 𝐿𝑖 ( 𝐵𝑖 − 1 ) ) 𝛽2 𝜏𝑖 E [ ︀ 𝑓𝑖 ( 𝑥 𝑡+1 ) − 𝑓 inf ] ︀ ) ︃ + 𝛽2 𝑛 𝑛∑︁ 𝑖=1 ( ︂ 2 ( 𝐴𝑖 + 𝐿𝑖 ( 𝐵𝑖 − 1 ) ) 𝜏𝑖 ( ︀ 𝑓 inf − 𝑓 inf𝑖 ) ︀ + 𝐶𝑖 𝜏𝑖 ) ︂ ≤ ( 1 − 𝜃 ) E [ ︀ 𝐺𝑡 ] ︀ + 𝛽1̃︀𝐿2E [ ︁⃦⃦𝑥𝑡+1 − 𝑥𝑡⃦⃦2 ] ︁+ ̃︀𝐴𝛽2E [ ︀𝑓 ( 𝑥𝑡+1 ) − 𝑓 inf ] ︀+ ̃︀𝐶𝛽2 D.1 CONVERGENCE FOR GENERAL NON-CONVEX FUNCTIONS Theorem 3 . Let Assumptions 1 and 2 hold , and let the stepsize in Algorithm 2 be set as 0 < 𝛾 ≤ ⎛⎝𝐿 + ̃︀𝐿 √︃ 𝛽1 𝜃 ⎞⎠−1 , ( 25 ) where ̃︀𝐿 = √︁ 1𝑛 ∑︀𝑛𝑖=1 𝐿2𝑖 , 𝜃 def= 1 − ( 1 − 𝛼 ) ( 1 + 𝜌 ) ( 1 + 𝜈 ) , 𝛽1 def= 2 ( 1 − 𝛼 ) ( 1 + 𝜌 ) ( ︀1 + 1𝜈 ) ︀ , and 𝜌 , 𝜈 > 0 are some positive numbers . Assume that batchsizes 𝜏1 , . . . , 𝜏𝑖 are such that 𝛾 ̃︀𝐴𝛽2 2𝜃 < 1 , wherẽ︀𝐴 = max𝑖=1 , ... , 𝑛 2 ( 𝐴𝑖+𝐿𝑖 ( 𝐵𝑖−1 ) ) 𝜏𝑖 and 𝛽2 def= 2 ( 1 − 𝛼 ) ( 1 + 𝜌 ) ( ︀1 + 1𝜈 ) ︀+ ( ︁1 + 1𝜌 ) ︁ . Fix 𝑇 ≥ 1 and let ? ̂ ? 𝑇 be chosen from the iterates 𝑥0 , 𝑥1 , . . . , 𝑥𝑇−1 with following probabilities : Prob { ︀ ? ̂ ? 𝑇 = 𝑥𝑡 } ︀ = 𝑤𝑡 𝑊𝑇 , 𝑤𝑡 = ( ︃ 1 − 𝛾 ̃︀𝐴𝛽2 2𝜃 ) ︃𝑡 , 𝑊𝑇 = 𝑇∑︁ 𝑡=0 𝑤𝑡 . Then E [ ︁⃦⃦ ∇𝑓 ( ? ̂ ? 𝑇 ) ⃦⃦2 ] ︁ ≤ 2 ( 𝑓 ( 𝑥0 ) − 𝑓 inf ) 𝛾𝑇 ( ︁ 1 − 𝛾 ̃︀𝐴𝛽2 2𝜃 ) ︁𝑇 + E [ ︀ 𝐺0 ] ︀ 𝜃𝑇 ( ︁ 1 − 𝛾 ̃︀𝐴𝛽2 2𝜃 ) ︁𝑇 + ̃︀𝐶𝛽2𝜃 , ( 26 ) where ̃︀𝐶 = 1𝑛 𝑛∑︀ 𝑖=1 ( ︁ 2 ( 𝐴𝑖+𝐿𝑖 ( 𝐵𝑖−1 ) ) 𝜏𝑖 ( ︀ 𝑓 inf − 𝑓 inf𝑖 ) ︀ + 𝐶𝑖𝜏𝑖 ) ︁ . Proof . We notice that inequality ( 20 ) holds for EF21-SGD as well , i.e. , we have E [ ︀ 𝛿𝑡+1 ] ︀ ≤ E [ ︀ 𝛿𝑡 ] ︀ − 𝛾 2 E [ ︁⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 ] ︁− ( ︂ 1 2𝛾 − 𝐿 2 ) ︂ E [ ︀ 𝑅𝑡 ] ︀ + 𝛾 2 E [ ︀ 𝐺𝑡 ] ︀ . Summing up the above inequality with a 𝛾 2𝜃 multiple of ( 24 ) , we derive E [ ︂ 𝛿𝑡+1 + 𝛾 2𝜃 𝐺𝑡+1 ] ︂ ≤ E [ ︀ 𝛿𝑡 ] ︀ − 𝛾 2 E [ ︁⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 ] ︁− ( ︂ 1 2𝛾 − 𝐿 2 ) ︂ E [ ︀ 𝑅𝑡 ] ︀ + 𝛾 2 E [ ︀ 𝐺𝑡 ] ︀ + 𝛾 2𝜃 ( 1 − 𝜃 ) E [ ︀ 𝐺𝑡 ] ︀ + 𝛾 2𝜃 𝛽1̃︀𝐿2E [ ︀𝑅𝑡 ] ︀ + 𝛾 2𝜃 ̃︀𝐴𝛽2E [ ︀𝛿𝑡+1 ] ︀+ 𝛾 2𝜃 ̃︀𝐶𝛽2 ≤ 𝛾 ̃︀𝐴𝛽2 2𝜃 E [ ︀ 𝛿𝑡+1 ] ︀ + E [ ︂ 𝛿𝑡 + 𝛾 2𝜃 𝐺𝑡 ] ︂ − 𝛾 2 E [ ︁⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 ] ︁ + 𝛾 2𝜃 ̃︀𝐶𝛽2 − ( ︃ 1 2𝛾 − 𝐿 2 − 𝛾𝛽1 ̃︀𝐿2 2𝜃 ) ︃ E [ ︀ 𝑅𝑡 ] ︀ ( 25 ) ≤ 𝛾 ̃︀𝐴𝛽2 2𝜃 E [ ︀ 𝛿𝑡+1 ] ︀ + E [ ︂ 𝛿𝑡 + 𝛾 2𝜃 𝐺𝑡 ] ︂ − 𝛾 2 E [ ︁⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 ] ︁ + 𝛾 2𝜃 ̃︀𝐶𝛽2 , where 𝜃 def= 1 − ( 1 − 𝛼 ) ( 1 + 𝜌 ) ( 1 + 𝜈 ) , 𝛽1 def = 2 ( 1 − 𝛼 ) ( 1 + 𝜌 ) ( ︀ 1 + 1𝜈 ) ︀ , 𝛽2 def = 2 ( 1 − 𝛼 ) ( 1 + 𝜌 ) ( ︀ 1 + 1𝜈 ) ︀ + ( ︁ 1 + 1𝜌 ) ︁ , and 𝜌 , 𝜈 > 0 are some positive numbers . Next , we rearrange the terms E [ ︁⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 ] ︁ ≤ 2 𝛾 ( ︃ E [ ︂ 𝛿𝑡 + 𝛾 2𝜃 𝐺𝑡 ] ︂ − ( ︃ 1 − 𝛾 ̃︀𝐴𝛽2 2𝜃 ) ︃ E [ ︀ 𝛿𝑡+1 ] ︀ − 𝛾 2𝜃 E [ ︀ 𝐺𝑡+1 ] ︀ ) ︃ + ̃︀𝐶𝛽2 𝜃 ≤ 2 𝛾 ( ︃ E [ ︂ 𝛿𝑡 + 𝛾 2𝜃 𝐺𝑡 ] ︂ − ( ︃ 1 − 𝛾 ̃︀𝐴𝛽2 2𝜃 ) ︃ E [ ︂ 𝛿𝑡+1 + 𝛾 2𝜃 E [ ︀ 𝐺𝑡+1 ] ︀ ] ︂ ) ︃ + ̃︀𝐶𝛽2 𝜃 , sum up the obtained inequalities for 𝑡 = 0,1 , . . . , 𝑇 with weights 𝑤𝑡/𝑊𝑇 , and use the definition of ? ̂ ? 𝑇 E [ ︁⃦⃦ ∇𝑓 ( ? ̂ ? 𝑇 ) ⃦⃦2 ] ︁ = 1 𝑊𝐾 𝑇∑︁ 𝑡=0 𝑤𝑡E [ ︁⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 ] ︁ ≤ 2 𝛾𝑊𝑇 𝑇∑︁ 𝑡=0 ( ︂ 𝑤𝑡E [ ︂ 𝛿𝑡 + 𝛾 2𝜃 𝐺𝑡 ] ︂ − 𝑤𝑡+1E [ ︂ 𝛿𝑡+1 + 𝛾 2𝜃 E [ ︀ 𝐺𝑡+1 ] ︀ ] ︂ ) ︂ + ̃︀𝐶𝛽2 𝜃 ≤ 2𝛿 0 𝛾𝑊𝑇 + E [ ︀ 𝐺0 ] ︀ 𝜃𝑊𝑇 + ̃︀𝐶𝛽2 𝜃 . Finally , we notice 𝑊𝑇 = 𝑇∑︁ 𝑡=0 𝑤𝑡 ≥ ( 𝑇 + 1 ) min 𝑡=0,1 , ... , 𝑇 𝑤𝑡 > 𝑇 ( ︃ 1 − 𝛾 ̃︀𝐴𝛽2 2𝜃 ) ︃𝑇 that finishes the proof . Corollary 4 . Let assumptions of Theorem 3 hold , 𝜌 = 𝛼/2 , 𝜈 = 𝛼/4 , 𝛾 = 1 𝐿 + ̃︀𝐿√︁𝛽1 𝜃 , 𝜏𝑖 = ⌈︃ max { ︃ 1 , 2𝑇𝛾 ( 𝐴𝑖 + 𝐿𝑖 ( 𝐵𝑖 − 1 ) ) 𝛽2 𝜃 , 8 ( 𝐴𝑖 + 𝐿𝑖 ( 𝐵𝑖 − 1 ) ) 𝛽2 𝜃𝜀2 𝛿inf𝑖 , 4𝐶𝑖𝛽2 𝜃𝜀2 } ︃⌉︃ , 𝑇 = ⌈︃ max { ︃ 16𝛿0 𝛾𝜀2 , 8E [ ︀ 𝐺0 ] ︀ 𝜃𝜀2 } ︃⌉︃ , where 𝛿inf𝑖 = 𝑓 inf − 𝑓 inf𝑖 , 𝛿0 = 𝑓 ( 𝑥0 ) − 𝑓 inf . Then , after 𝑇 iterations of EF21-SGD we have E [ ︁⃦⃦ ∇𝑓 ( ? ̂ ? 𝑇 ) ⃦⃦2 ] ︁ ≤ 𝜀2 . It requires 𝑇 = 𝒪 ( ︃ ̃︀𝐿𝛿0 + E [ ︀𝐺0 ] ︀ 𝛼𝜀2 ) ︃ iterations/communications rounds , # grad𝑖 = 𝜏𝑖𝑇 = 𝒪 ( ︃ ̃︀𝐿𝛿0 + E [ ︀𝐺0 ] ︀ 𝛼𝜀2 + ( ︁̃︀𝐿𝛿0 + E [ ︀𝐺0 ] ︀ ) ︁ ( ︁𝐴𝑖 ( 𝛿0 + 𝛿inf𝑖 ) + 𝐶𝑖 ) ︁ 𝛼3𝜀4 + ( ̃︀𝐿𝛿0 + E [ ︀𝐺0 ] ︀ ) 𝐴𝑖E [ ︀𝐺0 ] ︀ 𝛼2 ( 𝛼𝐿 + ̃︀𝐿 ) 𝜀4 ) ︃ stochastic oracle calls for worker 𝑖 , and # grad = 1 𝑛 𝑛∑︁ 𝑖=1 𝜏𝑖𝑇 = 𝒪 ( ︃ ̃︀𝐿𝛿0 + E [ ︀𝐺0 ] ︀ 𝛼𝜀2 + 1 𝑛 𝑛∑︁ 𝑖=1 ( ︁̃︀𝐿𝛿0 + E [ ︀𝐺0 ] ︀ ) ︁ ( ︁𝐴𝑖 ( 𝛿0 + 𝛿inf𝑖 ) + 𝐶𝑖 ) ︁ 𝛼3𝜀4 + 1 𝑛 𝑛∑︁ 𝑖=1 ( ̃︀𝐿𝛿0 + E [ ︀𝐺0 ] ︀ ) 𝐴𝑖E [ ︀𝐺0 ] ︀ 𝛼2 ( 𝛼𝐿 + ̃︀𝐿 ) 𝜀4 ) ︃ stochastic oracle calls per worker on average , where 𝐴𝑖 = 𝐴𝑖 + 𝐿𝑖 ( 𝐵𝑖 − 1 ) . Proof . The given choice of 𝜏𝑖 ensures that ( ︁ 1 − 𝛾 ̃︀𝐴𝛽2 2𝜃 ) ︁𝑇 = 𝒪 ( 1 ) and ̃︀𝐶𝛽2/𝜃 ≤ 𝜀/2 . Next , the choice of 𝑇 ensures that the right-hand side of ( 26 ) is smaller than 𝜀 . Finally , after simple computation we get the expression for 𝜏𝑖𝑇 . Corollary 5 . Consider the setting described in Example 1 . Let assumptions of Theorem 3 hold , 𝜌 = 𝛼/2 , 𝜈 = 𝛼/4 , 𝛾 = 1 𝐿 + ̃︀𝐿√︁𝛽1 𝜃 , 𝜏𝑖 = ⌈︃ max { ︃ 1 , 4𝜎2𝑖 𝛽2 𝜃𝜀2 } ︃⌉︃ , 𝑇 = ⌈︃ max { ︃ 16𝛿0 𝛾𝜀2 , 8E [ ︀ 𝐺0 ] ︀ 𝜃𝜀2 } ︃⌉︃ , where 𝛿0 = 𝑓 ( 𝑥0 ) − 𝑓 inf . Then , after 𝑇 iterations of EF21-SGD we have E [ ︁⃦⃦ ∇𝑓 ( ? ̂ ? 𝑇 ) ⃦⃦2 ] ︁ ≤ 𝜀2 . It requires 𝑇 = 𝒪 ( ︃ ̃︀𝐿𝛿0 + E [ ︀𝐺0 ] ︀ 𝛼𝜀2 ) ︃ iterations/communications rounds , # grad𝑖 = 𝜏𝑖𝑇 = 𝒪 ⎛⎝̃︀𝐿𝛿0 + E [ ︀𝐺0 ] ︀ 𝛼𝜀2 + ( ︁̃︀𝐿𝛿0 + E [ ︀𝐺0 ] ︀ ) ︁𝜎2𝑖 𝛼3𝜀4 ⎞⎠ stochastic oracle calls for worker 𝑖 , and # grad = 1 𝑛 𝑛∑︁ 𝑖=1 𝜏𝑖𝑇 = 𝒪 ⎛⎝̃︀𝐿𝛿0 + E [ ︀𝐺0 ] ︀ 𝛼𝜀2 + ( ︁̃︀𝐿𝛿0 + E [ ︀𝐺0 ] ︀ ) ︁𝜎2 𝛼3𝜀4 ⎞⎠ stochastic oracle calls per worker on average , where 𝜎2 = 1𝑛 ∑︀𝑛 𝑖=1 𝜎 2 𝑖 . Corollary 6 . Consider the setting described in Example 2 . Let assumptions of Theorem 3 hold , 𝜌 = 𝛼/2 , 𝜈 = 𝛼/4 , 𝛾 = 1 𝐿 + ̃︀𝐿√︁𝛽1 𝜃 , 𝜏𝑖 = ⌈︃ max { ︃ 1 , 2𝑇𝛾𝐿𝑖𝛽2 𝜃 , 8𝐿𝑖𝛽2 𝜃𝜀2 𝛿inf𝑖 , 8𝐿𝑖∆ inf 𝑖 𝛽2 𝜃𝜀2 } ︃⌉︃ , 𝑇 = ⌈︃ max { ︃ 16𝛿0 𝛾𝜀2 , 8E [ ︀ 𝐺0 ] ︀ 𝜃𝜀2 } ︃⌉︃ , where 𝛿inf𝑖 = 𝑓 inf − 𝑓 inf𝑖 , 𝛿0 = 𝑓 ( 𝑥0 ) − 𝑓 inf , 𝐿𝑖 = 1𝑚𝑖 ∑︀𝑚𝑖 𝑗=1 𝐿𝑖𝑗 , ∆ inf 𝑖 = 1 𝑚𝑖 ∑︀𝑚𝑖 𝑗=1 ( 𝑓 inf 𝑖 − 𝑓 inf𝑖𝑗 ) . Then , after 𝑇 iterations of EF21-SGD we have E [ ︁⃦⃦ ∇𝑓 ( ? ̂ ? 𝑇 ) ⃦⃦2 ] ︁ ≤ 𝜀2 . It requires 𝑇 = 𝒪 ( ︃ ̃︀𝐿𝛿0 + E [ ︀𝐺0 ] ︀ 𝛼𝜀2 ) ︃ iterations/communications rounds , # grad𝑖 = 𝜏𝑖𝑇 = 𝒪 ( ︃ ̃︀𝐿𝛿0 + E [ ︀𝐺0 ] ︀ 𝛼𝜀2 + ( ︁̃︀𝐿𝛿0 + E [ ︀𝐺0 ] ︀ ) ︁ ( ︀𝐿𝑖 ( 𝛿0 + 𝛿inf𝑖 ) + 𝐿𝑖∆inf𝑖 ) ︀ 𝛼3𝜀4 + ( ̃︀𝐿𝛿0 + E [ ︀𝐺0 ] ︀ ) 𝐿𝑖E [ ︀𝐺0 ] ︀ 𝛼2 ( 𝛼𝐿 + ̃︀𝐿 ) 𝜀4 ) ︃ stochastic oracle calls for worker 𝑖 , and # grad = 1 𝑛 𝑛∑︁ 𝑖=1 𝜏𝑖𝑇 = 𝒪 ( ︃ ̃︀𝐿𝛿0 + E [ ︀𝐺0 ] ︀ 𝛼𝜀2 + 1 𝑛 𝑛∑︁ 𝑖=1 ( ︁̃︀𝐿𝛿0 + E [ ︀𝐺0 ] ︀ ) ︁ ( ︀𝐿𝑖 ( 𝛿0 + 𝛿inf𝑖 ) + 𝐿𝑖∆inf𝑖 ) ︀ 𝛼3𝜀4 + 1 𝑛 𝑛∑︁ 𝑖=1 ( ̃︀𝐿𝛿0 + E [ ︀𝐺0 ] ︀ ) 𝐿𝑖E [ ︀𝐺0 ] ︀ 𝛼2 ( 𝛼𝐿 + ̃︀𝐿 ) 𝜀4 ) ︃ stochastic oracle calls per worker on average . D.2 CONVERGENCE UNDER POLYAK-ŁOJASIEWICZ CONDITION Theorem 4 . Let Assumptions 1 , 2 , and 4 hold , and let the stepsize in Algorithm 2 be set as 0 < 𝛾 ≤ min ⎧⎪⎨⎪⎩ ⎛⎝𝐿 + ̃︀𝐿 √︃ 2𝛽1 𝜃 ⎞⎠−1 , 𝜃 2𝜇 ⎫⎪⎬⎪⎭ , ( 27 ) where ̃︀𝐿 = √︁ 1𝑛 ∑︀𝑛𝑖=1 𝐿2𝑖 , 𝜃 def= 1 − ( 1 − 𝛼 ) ( 1 + 𝜌 ) ( 1 + 𝜈 ) , 𝛽1 def= 2 ( 1 − 𝛼 ) ( 1 + 𝜌 ) ( ︀1 + 1𝜈 ) ︀ , and 𝜌 , 𝜈 > 0 are some positive numbers . Assume that batchsizes 𝜏1 , . . . , 𝜏𝑖 are such that 2 ̃︀𝐴𝛽2 𝜃 ≤ 𝜇2 , where ̃︀𝐴 = max𝑖=1 , ... , 𝑛 2 ( 𝐴𝑖+𝐿𝑖 ( 𝐵𝑖−1 ) ) 𝜏𝑖 and 𝛽2 def= 2 ( 1 − 𝛼 ) ( 1 + 𝜌 ) ( ︀1 + 1𝜈 ) ︀+ ( ︁1 + 1𝜌 ) ︁ . Then for all 𝑇 ≥ 1 E [ ︂ 𝛿𝑇 + 𝛾 𝜃 𝐺𝑇 ] ︂ ≤ ( ︁ 1 − 𝛾𝜇 2 ) ︁𝑇 E [ ︂ 𝛿0 + 𝛾 𝜃 𝐺0 ] ︂ + 4 𝜇𝜃 ̃︀𝐶𝛽2 , ( 28 ) where ̃︀𝐶 = 1𝑛 𝑛∑︀ 𝑖=1 ( ︁ 2 ( 𝐴𝑖+𝐿𝑖 ( 𝐵𝑖−1 ) ) 𝜏𝑖 ( ︀ 𝑓 inf − 𝑓 inf𝑖 ) ︀ + 𝐶𝑖𝜏𝑖 ) ︁ . Proof . We notice that inequality ( 20 ) holds for EF21-SGD as well , i.e. , we have E [ ︀ 𝛿𝑡+1 ] ︀ ≤ E [ ︀ 𝛿𝑡 ] ︀ − 𝛾 2 E [ ︁⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 ] ︁− ( ︂ 1 2𝛾 − 𝐿 2 ) ︂ E [ ︀ 𝑅𝑡 ] ︀ + 𝛾 2 E [ ︀ 𝐺𝑡 ] ︀ PŁ ≤ ( 1 − 𝛾𝜇 ) E [ ︀ 𝛿𝑡 ] ︀ − ( ︂ 1 2𝛾 − 𝐿 2 ) ︂ E [ ︀ 𝑅𝑡 ] ︀ + 𝛾 2 E [ ︀ 𝐺𝑡 ] ︀ . Summing up the above inequality with a 𝛾 𝜃 multiple of ( 24 ) , we derive E [ ︂ 𝛿𝑡+1 + 𝛾 𝜃 𝐺𝑡+1 ] ︂ ≤ ( 1 − 𝛾𝜇 ) E [ ︀ 𝛿𝑡 ] ︀ − ( ︂ 1 2𝛾 − 𝐿 2 ) ︂ E [ ︀ 𝑅𝑡 ] ︀ + 𝛾 2 E [ ︀ 𝐺𝑡 ] ︀ + 𝛾 𝜃 ( 1 − 𝜃 ) E [ ︀ 𝐺𝑡 ] ︀ + 𝛾 𝜃 𝛽1̃︀𝐿2E [ ︀𝑅𝑡 ] ︀ + 𝛾 𝜃 ̃︀𝐴𝛽2E [ ︀𝛿𝑡+1 ] ︀+ 𝛾 𝜃 ̃︀𝐶𝛽2 ≤ 𝛾 ̃︀𝐴𝛽2 𝜃 E [ ︀ 𝛿𝑡+1 ] ︀ + ( 1 − 𝛾𝜇 ) E [ ︀ 𝛿𝑡 ] ︀ + ( ︃ 1 − 𝜃 2 ) ︃ E [ ︂ 𝛾 𝜃 𝐺𝑡 ] ︂ + 𝛾 𝜃 ̃︀𝐶𝛽2 − ( ︃ 1 2𝛾 − 𝐿 2 − 𝛾𝛽1 ̃︀𝐿2 𝜃 ) ︃ E [ ︀ 𝑅𝑡 ] ︀ ( 27 ) ≤ 𝛾 ̃︀𝐴𝛽2 𝜃 E [ ︀ 𝛿𝑡+1 ] ︀ + ( 1 − 𝛾𝜇 ) E [ ︂ 𝛿𝑡 + 𝛾 𝜃 𝐺𝑡 ] ︂ + 𝛾 𝜃 ̃︀𝐶𝛽2 , where 𝜃 def= 1 − ( 1 − 𝛼 ) ( 1 + 𝜌 ) ( 1 + 𝜈 ) , 𝛽1 def = 2 ( 1 − 𝛼 ) ( 1 + 𝜌 ) ( ︀ 1 + 1𝜈 ) ︀ , 𝛽2 def = 2 ( 1 − 𝛼 ) ( 1 + 𝜌 ) ( ︀ 1 + 1𝜈 ) ︀ + ( ︁ 1 + 1𝜌 ) ︁ , and 𝜌 , 𝜈 > 0 are some positive numbers . Next , we rearrange the terms ( ︃ 1 − 𝛾 ̃︀𝐴𝛽2 𝜃 ) ︃ E [ ︂ 𝛿𝑡+1 + 𝛾 𝜃 𝐺𝑡+1 ] ︂ ≤ E [ ︃ ( ︃ 1 − 𝛾 ̃︀𝐴𝛽2 𝜃 ) ︃ 𝛿𝑡+1 + 𝛾 𝜃 𝐺𝑡+1 ] ︃ ≤ ( 1 − 𝛾𝜇 ) E [ ︂ 𝛿𝑡 + 𝛾 𝜃 𝐺𝑡 ] ︂ + 𝛾 𝜃 ̃︀𝐶𝛽2 and divide both sides of the inequality by ( ︁ 1 − 𝛾 ̃︀𝐴𝛽2 𝜃 ) ︁ : E [ ︂ 𝛿𝑡+1 + 𝛾 𝜃 𝐺𝑡+1 ] ︂ ≤ 1 − 𝛾𝜇 1 − 𝛾 ̃︀𝐴𝛽2 𝜃 E [ ︂ 𝛿𝑡 + 𝛾 𝜃 𝐺𝑡 ] ︂ + 𝛾 𝜃 ( ︁ 1 − 𝛾 ̃︀𝐴𝛽2 𝜃 ) ︁ ̃︀𝐶𝛽2 ( 120 ) ≤ ( 1 − 𝛾𝜇 ) ( ︃ 1 + 2𝛾 ̃︀𝐴𝛽2 𝜃 ) ︃ E [ ︂ 𝛿𝑡 + 𝛾 𝜃 𝐺𝑡 ] ︂ + ( ︃ 1 + 2𝛾 ̃︀𝐴𝛽2 𝜃 ) ︃ 𝛾 𝜃 ̃︀𝐶𝛽2 . Since 2 ̃︀𝐴𝛽2 𝜃 ≤ 𝜇2 and 𝛾 ≤ 2 𝜇 , we have E [ ︂ 𝛿𝑡+1 + 𝛾 𝜃 𝐺𝑡+1 ] ︂ ≤ ( 1 − 𝛾𝜇 ) ( ︁ 1 + 𝛾𝜇 2 ) ︁ E [ ︂ 𝛿𝑡 + 𝛾 𝜃 𝐺𝑡 ] ︂ + 2𝛾 𝜃 ̃︀𝐶𝛽2 ( 121 ) ≤ ( ︁ 1 − 𝛾𝜇 2 ) ︁ E [ ︂ 𝛿𝑡 + 𝛾 𝜃 𝐺𝑡 ] ︂ + 2𝛾 𝜃 ̃︀𝐶𝛽2 . Unrolling the recurrence , we get E [ ︂ 𝛿𝑇 + 𝛾 𝜃 𝐺𝑇 ] ︂ ≤ ( ︁ 1 − 𝛾𝜇 2 ) ︁𝑇 E [ ︂ 𝛿0 + 𝛾 𝜃 𝐺0 ] ︂ + 2𝛾 𝜃 ̃︀𝐶𝛽2 𝑇−1∑︁ 𝑡=0 ( ︁ 1 − 𝛾𝜇 2 ) ︁𝑡 ≤ ( ︁ 1 − 𝛾𝜇 2 ) ︁𝑇 E [ ︂ 𝛿0 + 𝛾 𝜃 𝐺0 ] ︂ + 2𝛾 𝜃 ̃︀𝐶𝛽2 ∞∑︁ 𝑡=0 ( ︁ 1 − 𝛾𝜇 2 ) ︁𝑡 = ( ︁ 1 − 𝛾𝜇 2 ) ︁𝑇 E [ ︂ 𝛿0 + 𝛾 𝜃 𝐺0 ] ︂ + 4 𝜇𝜃 ̃︀𝐶𝛽2 that finishes the proof . Corollary 7 . Let assumptions of Theorem 4 hold , 𝜌 = 𝛼/2 , 𝜈 = 𝛼/4 , 𝛾 = min ⎧⎨⎩ 1 𝐿 + ̃︀𝐿√︁𝛽1 𝜃 , 𝜃 2𝜇 ⎫⎬⎭ , 𝜏𝑖 = ⌈︃ max { ︃ 1 , 8 ( 𝐴𝑖 + 𝐿𝑖 ( 𝐵𝑖 − 1 ) ) 𝛽2 𝜇𝜃 , 64 ( 𝐴𝑖 + 𝐿𝑖 ( 𝐵𝑖 − 1 ) ) 𝛽2 𝜃𝜀𝜇 𝛿inf𝑖 , 32𝐶𝑖𝛽2 𝜃𝜀𝜇 } ︃⌉︃ , 𝑇 = ⌈︂ 2 𝛾𝜇 ln ( ︂ 2𝛿0 𝜀 + E [ ︂ 2𝛾𝐺0 𝜃𝜀 ] ︂ ) ︂⌉︂ , where 𝛿inf𝑖 = 𝑓 inf − 𝑓 inf𝑖 , 𝛿0 = 𝑓 ( 𝑥0 ) − 𝑓 inf . Then , after 𝑇 iterations of EF21-SGD we have E [ ︀ 𝑓 ( 𝑥𝑇 ) − 𝑓 inf ] ︀ ≤ 𝜀 . It requires 𝑇 = 𝒪 ( ︃ ̃︀𝐿 𝜇𝛼 ln ( ︂ 𝛿0 𝜀 + E [ ︂ 2𝐺0̃︀𝐿𝜀 ] ︂ ) ︂ ) ︃ iterations/communications rounds , # grad𝑖 = 𝜏𝑖𝑇 = 𝒪 ⎛⎝⎛⎝ ̃︀𝐿 𝜇𝛼 + ̃︀𝐿 ( ︁𝐴𝑖 ( 𝜀 + 𝛿inf𝑖 ) + 𝐶𝑖 ) ︁ 𝜇2𝛼3𝜀 ⎞⎠ ln ( ︂𝛿0 𝜀 + E [ ︂ 2𝐺0̃︀𝐿𝜀 ] ︂ ) ︂⎞⎠ stochastic oracle calls for worker 𝑖 , and # grad = 1 𝑛 𝑛∑︁ 𝑖=1 𝜏𝑖𝑇 = 𝒪 ⎛⎝⎛⎝ ̃︀𝐿 𝜇𝛼 + 1 𝑛 𝑛∑︁ 𝑖=1 ̃︀𝐿 ( ︁𝐴𝑖 ( 𝜀 + 𝛿inf𝑖 ) + 𝐶𝑖 ) ︁ 𝜇2𝛼3𝜀 ⎞⎠ ln ( ︂𝛿0 𝜀 + E [ ︂ 2𝐺0̃︀𝐿𝜀 ] ︂ ) ︂⎞⎠ stochastic oracle calls per worker on average , where 𝐴𝑖 = 𝐴𝑖 + 𝐿𝑖 ( 𝐵𝑖 − 1 ) . Proof . The given choice of 𝜏𝑖 ensures that 2 ̃︀𝐴𝛽2 𝜃 ≤ 𝜇2 and 4 ̃︀𝐶𝛽2/𝜇𝜃 ≤ 𝜀/2 . Next , the choice of 𝑇 ensures that the right-hand side of ( 28 ) is smaller than 𝜀 . Finally , after simple computation we get the expression for 𝜏𝑖𝑇 . Corollary 8 . Consider the setting described in Example 1 . Let assumptions of Theorem 4 hold , 𝜌 = 𝛼/2 , 𝜈 = 𝛼/4 , 𝛾 = min ⎧⎨⎩ 1 𝐿 + ̃︀𝐿√︁𝛽1 𝜃 , 𝜃 2𝜇 ⎫⎬⎭ , 𝜏𝑖 = ⌈︃ max { ︃ 1 , 32𝐶𝑖𝛽2 𝜃𝜀𝜇 } ︃⌉︃ , 𝑇 = ⌈︂ 2 𝛾𝜇 ln ( ︂ 2𝛿0 𝜀 + E [ ︂ 2𝛾𝐺0 𝜃𝜀 ] ︂ ) ︂⌉︂ , where 𝛿0 = 𝑓 ( 𝑥0 ) − 𝑓 inf . Then , after 𝑇 iterations of EF21-SGD we have E [ ︀ 𝑓 ( 𝑥𝑇 ) − 𝑓 inf ] ︀ ≤ 𝜀 . It requires 𝑇 = 𝒪 ( ︃ ̃︀𝐿 𝜇𝛼 ln ( ︂ 𝛿0 𝜀 + E [ ︂ 2𝐺0̃︀𝐿𝜀 ] ︂ ) ︂ ) ︃ iterations/communications rounds , # grad𝑖 = 𝜏𝑖𝑇 = 𝒪 ( ︃ ( ︃ ̃︀𝐿 𝜇𝛼 + ̃︀𝐿𝜎2𝑖 𝜇2𝛼3𝜀 ) ︃ ln ( ︂ 𝛿0 𝜀 + E [ ︂ 2𝐺0̃︀𝐿𝜀 ] ︂ ) ︂ ) ︃ stochastic oracle calls for worker 𝑖 , and # grad = 1 𝑛 𝑛∑︁ 𝑖=1 𝜏𝑖𝑇 = 𝒪 ( ︃ ( ︃ ̃︀𝐿 𝜇𝛼 + ̃︀𝐿𝜎2 𝜇2𝛼3𝜀 ) ︃ ln ( ︂ 𝛿0 𝜀 + E [ ︂ 2𝐺0̃︀𝐿𝜀 ] ︂ ) ︂ ) ︃ stochastic oracle calls per worker on average , where 𝜎2 = 1𝑛 ∑︀𝑛 𝑖=1 𝜎 2 𝑖 . Corollary 9 . Consider the setting described in Example 2 . Let assumptions of Theorem 4 hold , 𝜌 = 𝛼/2 , 𝜈 = 𝛼/4 , 𝛾 = min ⎧⎨⎩ 1 𝐿 + ̃︀𝐿√︁𝛽1 𝜃 , 𝜃 2𝜇 ⎫⎬⎭ , 𝜏𝑖 = ⌈︃ max { ︃ 1 , 8𝐿𝑖𝛽2 𝜇𝜃 , 64𝐿𝑖𝛽2 𝜃𝜀𝜇 𝛿inf𝑖 , 64𝐿𝑖∆ inf 𝑖 𝛽2 𝜃𝜀𝜇 } ︃⌉︃ , 𝑇 = ⌈︂ 2 𝛾𝜇 ln ( ︂ 2𝛿0 𝜀 + E [ ︂ 2𝛾𝐺0 𝜃𝜀 ] ︂ ) ︂⌉︂ , where 𝛿inf𝑖 = 𝑓 inf − 𝑓 inf𝑖 , 𝛿0 = 𝑓 ( 𝑥0 ) − 𝑓 inf , 𝐿𝑖 = 1𝑚𝑖 ∑︀𝑚𝑖 𝑗=1 𝐿𝑖𝑗 , ∆ inf 𝑖 = 1 𝑚𝑖 ∑︀𝑚𝑖 𝑗=1 ( 𝑓 inf 𝑖 − 𝑓 inf𝑖𝑗 ) . Then , after 𝑇 iterations of EF21-SGD we have E [ ︀ 𝑓 ( 𝑥𝑇 ) − 𝑓 inf ] ︀ ≤ 𝜀 . It requires 𝑇 = 𝒪 ( ︃ ̃︀𝐿 𝜇𝛼 ln ( ︂ 𝛿0 𝜀 + E [ ︂ 2𝐺0̃︀𝐿𝜀 ] ︂ ) ︂ ) ︃ iterations/communications rounds , # grad𝑖 = 𝜏𝑖𝑇 = 𝒪 ( ︃ ( ︃ ̃︀𝐿 𝜇𝛼 + ̃︀𝐿𝐿𝑖 ( ︀𝜀 + 𝛿inf𝑖 + ∆inf𝑖 ) ︀ 𝜇2𝛼3𝜀 ) ︃ ln ( ︂ 𝛿0 𝜀 + E [ ︂ 2𝐺0̃︀𝐿𝜀 ] ︂ ) ︂ ) ︃ stochastic oracle calls for worker 𝑖 , and # grad = 1 𝑛 𝑛∑︁ 𝑖=1 𝜏𝑖𝑇 = 𝒪 ( ︃ ( ︃ ̃︀𝐿 𝜇𝛼 + 1 𝑛 𝑛∑︁ 𝑖=1 ̃︀𝐿𝐿𝑖 ( ︀𝜀 + 𝛿inf𝑖 + ∆inf𝑖 ) ︀ 𝜇2𝛼3𝜀 ) ︃ ln ( ︂ 𝛿0 𝜀 + E [ ︂ 2𝐺0̃︀𝐿𝜀 ] ︂ ) ︂ ) ︃ stochastic oracle calls per worker on average . E VARIANCE REDUCTION In this part , we modify the EF21 framework to better handle finite-sum problems with smooth summands . Unlike the online/streaming case where SGD has the optimal complexity ( without additional assumption on the smoothness of stochastic trajectories ) ( Arjevani et al. , 2019 ) , in the finite sum regime , it is well-known that one can hope for convergence to the exact stationary point rather than its neighborhood . To achieve this , variance reduction techniques are instrumental . One approach is to apply a PAGE-estimator ( Li et al. , 2021 ) instead of a random minibatch applied in SGD . Note that PAGE has optimal complexity for nonconvex problems of the form ( 3 ) . With Corollary 10 , we illustrate that this 𝑂 ( 𝑚 + √ 𝑚/𝜀2 ) complexity is recovered for our Algorithm 3 when no compression is applied and 𝑛 = 1 . We show how to combine PAGE estimator with EF21 mechanism and call the new method EF21PAGE . At each step of EF21-PAGE , clients ( nodes ) either compute ( with probability 𝑝 ) full gradients or use a recursive estimator 𝑣𝑡𝑖 + 1 𝜏𝑖 ∑︀ 𝑗∈𝐼𝑡𝑖 ( ︀ ∇𝑓𝑖𝑗 ( 𝑥𝑡+1 ) −∇𝑓𝑖𝑗 ( 𝑥𝑡 ) ) ︀ ( with probability 1− 𝑝 ) . Then each client applies a Markov compressor/EF21-estimator and sends the result to the master node . Typically the number of data points 𝑚 is large , and 𝑝 < 1/𝑚 . As a result , computation of full gradients rarely happens during optimization procedure , on average , only once in every 𝑚 iterations . Notice that unlike VR-MARINA ( Gorbunov et al. , 2021 ) , which is a state-of-the-art distributed optimization method designed specifically for unbiased compressors and which also uses PAGEestimator , EF21-PAGE does not require the communication of full ( not compressed ) vectors at all . This is an important property of the algorithm since , in some distributed networks , and especially when 𝑑 is very large , as is the case in modern over-parameterized deep learning , full vector communication is prohibitive . However , unlike the rate of VR-MARINA , the rate of EF21-PAGE does not improve with the growth of 𝑛 . This is not a flaw of our method , but rather an inevitable drawback of the distributed methods that use biased compressions only . Notations for this section . In this section , we use the following additional notations 𝑃 𝑡𝑖 def = ‖∇𝑓𝑖 ( 𝑥𝑡 ) − 𝑣𝑡𝑖‖ 2 , 𝑃 𝑡 def= 1𝑛 ∑︀𝑛 𝑖=1 𝑃 𝑡 𝑖 , 𝑉 𝑡 𝑖 def = ‖𝑣𝑡𝑖 − 𝑔𝑡𝑖‖ 2 , 𝑉 𝑡 def= 1𝑛 ∑︀𝑛 𝑖=1 𝑉 𝑡 𝑖 , where 𝑣 𝑡 𝑖 is a PAGE estimator . Recall that 𝐺𝑡 def= 1𝑛 ∑︀𝑛 𝑖=1 𝐺 𝑡 𝑖 , 𝐺 𝑡 𝑖 def = ‖∇𝑓𝑖 ( 𝑥𝑡 ) − 𝑔𝑡𝑖‖ 2 . The main idea of the analysis in this section is to split the error in two parts 𝐺𝑡𝑖 ≤ 2𝑃 𝑡𝑖 + 2𝑉 𝑡𝑖 , and bound them separetely . Algorithm 3 EF21-PAGE 1 : Input : starting point 𝑥0 ∈ R𝑑 ; 𝑔0𝑖 , 𝑣0𝑖 ∈ R𝑑 for 𝑖 = 1 , . . . , 𝑛 ( known by nodes ) ; 𝑔0 = 1𝑛 ∑︀𝑛 𝑖=1 𝑔 0 𝑖 ( known by master ) ; learning rate 𝛾 > 0 ; probabilities 𝑝𝑖 ∈ ( 0,1 ] ; batch-sizes 1 ≤ 𝜏𝑖 ≤ 𝑚𝑖 2 : for 𝑡 = 0,1 , 2 , . . . , 𝑇 − 1 do 3 : Master computes 𝑥𝑡+1 = 𝑥𝑡 − 𝛾𝑔𝑡 4 : for all nodes 𝑖 = 1 , . . . , 𝑛 in parallel do 5 : Sample 𝑏𝑡𝑖 ∼ Be ( 𝑝𝑖 ) 6 : If 𝑏𝑡𝑖 = 0 , sample a minibatch of data samples 𝐼 𝑡 𝑖 with |𝐼𝑡𝑖 | = 𝜏𝑖 7 : 𝑣𝑡+1𝑖 = ⎧⎨⎩∇𝑓𝑖 ( 𝑥 𝑡+1 ) if 𝑏𝑡𝑖 = 1 , 𝑣𝑡𝑖 + 1 𝜏𝑖 ∑︀ 𝑗∈𝐼𝑡𝑖 ( ︀ ∇𝑓𝑖𝑗 ( 𝑥𝑡+1 ) −∇𝑓𝑖𝑗 ( 𝑥𝑡 ) ) ︀ if 𝑏𝑡𝑖 = 0 8 : Compress 𝑐𝑡𝑖 = 𝒞 ( 𝑣 𝑡+1 𝑖 − 𝑔𝑡𝑖 ) and send 𝑐𝑡𝑖 to the master 9 : Update local state 𝑔𝑡+1𝑖 = 𝑔 𝑡 𝑖 + 𝑐 𝑡 𝑖 10 : end for 11 : Master computes 𝑔𝑡+1 = 1𝑛 ∑︀𝑛 𝑖=1 𝑔 𝑡+1 𝑖 via 𝑔 𝑡+1 = 𝑔𝑡 + 1𝑛 ∑︀𝑛 𝑖=1 𝑐 𝑡 𝑖 12 : end for 13 : Output : ? ̂ ? 𝑇 chosen uniformly from { 𝑥𝑡 } 𝑡∈ [ 𝑇 ] Lemma 3 . Let Assumption 3 hold , and let 𝑣𝑡+1𝑖 be a PAGE estimator , i. e. for 𝑏𝑡𝑖 ∼ Be ( 𝑝𝑖 ) 𝑣𝑡+1𝑖 = ⎧⎨⎩∇𝑓𝑖 ( 𝑥 𝑡+1 ) if 𝑏𝑡𝑖 = 1 , 𝑣𝑡𝑖 + 1 𝜏𝑖 ∑︀ 𝑗∈𝐼𝑡𝑖 ( ︀ ∇𝑓𝑖𝑗 ( 𝑥𝑡+1 ) −∇𝑓𝑖𝑗 ( 𝑥𝑡 ) ) ︀ if 𝑏𝑡𝑖 = 0 , ( 29 ) for all 𝑖 = 1 , . . . , 𝑛 , 𝑡 ≥ 0 . Then E [ ︀ 𝑃 𝑡+1 ] ︀ ≤ ( 1 − 𝑝min ) E [ ︀ 𝑃 𝑡 ] ︀ + ̃︀ℒ2E [ ︁⃦⃦𝑥𝑡+1 − 𝑥𝑡⃦⃦2 ] ︁ , ( 30 ) where ̃︀ℒ = 1𝑛 ∑︀𝑛𝑖=1 ( 1−𝑝𝑖 ) ℒ2𝑖𝜏𝑖 , 𝑝min = min𝑖=1 , ... , 𝑛 𝑝𝑖 . Proof . E [ ︀ 𝑃 𝑡+1𝑖 ] ︀ = E [ ︁⃦⃦ 𝑣𝑡+1𝑖 −∇𝑓𝑖 ( 𝑥 𝑡+1 ) ⃦⃦2 ] ︁ = ( 1 − 𝑝𝑖 ) E ⎡⎢⎣ ⃦⃦⃦⃦ ⃦⃦𝑣𝑡𝑖 + 1𝜏𝑖 ∑︁ 𝑗∈𝐼𝑡𝑖 ( ∇𝑓𝑖𝑗 ( 𝑥𝑡+1 ) −∇𝑓𝑖𝑗 ( 𝑥𝑡 ) ) −∇𝑓𝑖 ( 𝑥𝑡+1 ) ⃦⃦⃦⃦ ⃦⃦ 2 ⎤⎥⎦ = ( 1 − 𝑝𝑖 ) E [ ︂⃦⃦⃦ 𝑣𝑡𝑖 −∇𝑓𝑖 ( 𝑥𝑡 ) + ̃︀∆𝑡𝑖 −∇𝑓𝑖 ( 𝑥𝑡+1 ) + ∇𝑓𝑖 ( 𝑥𝑡 ) ⃦⃦⃦2 ] ︂ = ( 1 − 𝑝𝑖 ) E [ ︂⃦⃦⃦ 𝑣𝑡𝑖 −∇𝑓𝑖 ( 𝑥𝑡 ) + ̃︀∆𝑡𝑖 − ∆𝑡𝑖 ⃦⃦⃦2 ] ︂ ( 𝑖 ) = ( 1 − 𝑝𝑖 ) E [ ︁⃦⃦ 𝑣𝑡𝑖 −∇𝑓𝑖 ( 𝑥𝑡 ) ⃦⃦2 ] ︁ + ( 1 − 𝑝𝑖 ) E [ ︂⃦⃦⃦ ̃︀∆𝑡𝑖 − ∆𝑡𝑖 ⃦⃦⃦2 ] ︂ ( 𝑖𝑖 ) ≤ ( 1 − 𝑝𝑖 ) E [ ︀ 𝑃 𝑡𝑖 ] ︀ + ( 1 − 𝑝𝑖 ) ℒ2𝑖 𝜏𝑖 E [ ︁⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 ] ︁ ≤ ( 1 − 𝑝min ) E [ ︀ 𝑃 𝑡𝑖 ] ︀ + ( 1 − 𝑝𝑖 ) ℒ2𝑖 𝜏𝑖 E [ ︁⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 ] ︁ , ( 31 ) where equality ( 𝑖 ) holds because E [ ︁̃︀∆𝑡𝑖 − ∆𝑡𝑖 | 𝑥𝑡 , 𝑥𝑡+1 , 𝑣𝑡𝑖 ] ︁ = 0 , and ( 𝑖𝑖 ) holds by Assumption 3 . It remains to average the above inequality over 𝑖 = 1 , . . . , 𝑛. Lemma 4 . Let Assumptions 1 and 3 hold , let 𝑣𝑡+1𝑖 be a PAGE estimator , i. e. for 𝑏𝑡𝑖 ∼ Be ( 𝑝𝑖 ) and for all 𝑖 = 1 , . . . , 𝑛 , 𝑡 ≥ 0 𝑣𝑡+1𝑖 = ⎧⎨⎩∇𝑓𝑖 ( 𝑥 𝑡+1 ) if 𝑏𝑡𝑖 = 1 , 𝑣𝑡𝑖 + 1 𝜏𝑖 ∑︀ 𝑗∈𝐼𝑡𝑖 ( ︀ ∇𝑓𝑖𝑗 ( 𝑥𝑡+1 ) −∇𝑓𝑖𝑗 ( 𝑥𝑡 ) ) ︀ if 𝑏𝑡𝑖 = 0 , ( 32 ) and let 𝑔𝑡+1𝑖 be an EF21 estimator , i. e. 𝑔𝑡+1𝑖 = 𝑔 𝑡 𝑖 + 𝒞 ( 𝑣𝑡+1𝑖 − 𝑔 𝑡 𝑖 ) , 𝑔 0 𝑖 = 𝒞 ( ︀ 𝑣0𝑖 ) ︀ ( 33 ) for all 𝑖 = 1 , . . . , 𝑛 , 𝑡 ≥ 0 . Then E [ ︀ 𝑉 𝑡+1 ] ︀ ≤ ( 1 − 𝜃 ) E [ ︀ 𝑉 𝑡 ] ︀ + 2𝛽𝑝maxE [ ︀ 𝑃 𝑡 ] ︀ + 𝛽 ( ︁ 2̃︀𝐿2 + ̃︀ℒ2 ) ︁E [ ︁⃦⃦𝑥𝑡+1 − 𝑥𝑡⃦⃦2 ] ︁ , ( 34 ) where ̃︀ℒ = 1𝑛 ∑︀𝑛𝑖=1 ( 1−𝑝𝑖 ) ℒ2𝑖𝜏𝑖 , 𝑝max = max𝑖=1 , ... , 𝑛 𝑝𝑖 , 𝜃 = 1− ( 1−𝛼 ) ( 1+𝑠 ) , 𝛽 = ( 1−𝛼 ) ( ︀1 + 𝑠−1 ) ︀ for any 𝑠 > 0 . Proof . Following the steps in proof of Lemma 1 , but with ∇𝑓𝑖 ( 𝑥𝑡+1 ) and ∇𝑓𝑖 ( 𝑥𝑡 ) being substituted by their estimators 𝑣𝑡+1𝑖 and 𝑣 𝑡 𝑖 , we end up with an analogue of ( 15 ) E [ ︁⃦⃦ 𝑔𝑡+1𝑖 − 𝑣 𝑡+1 𝑖 ⃦⃦2 ] ︁ ≤ ( 1 − 𝜃 ) E [ ︁⃦⃦𝑔𝑡𝑖 − 𝑣𝑡𝑖 ⃦⃦2 ] ︁+ 𝛽E [ ︁⃦⃦𝑣𝑡+1𝑖 − 𝑣𝑡𝑖 ⃦⃦2 ] ︁ , ( 35 ) where 𝜃 = 1 − ( 1 − 𝛼 ) ( 1 + 𝑠 ) , 𝛽 = ( 1 − 𝛼 ) ( ︀ 1 + 𝑠−1 ) ︀ for any 𝑠 > 0 . Then E [ ︀ 𝑉 𝑡𝑖 ] ︀ = E [ ︁⃦⃦ 𝑔𝑡+1𝑖 − 𝑣 𝑡+1 𝑖 ⃦⃦2 ] ︁ ( 35 ) ≤ ( 1 − 𝜃 ) E [ ︁⃦⃦ 𝑔𝑡𝑖 − 𝑣𝑡𝑖 ⃦⃦2 ] ︁ + 𝛽E [ ︁⃦⃦ 𝑣𝑡+1𝑖 − 𝑣 𝑡 𝑖 ⃦⃦2 ] ︁ = ( 1 − 𝜃 ) E [ ︁⃦⃦ 𝑔𝑡𝑖 − 𝑣𝑡𝑖 ⃦⃦2 ] ︁ + 𝛽E [ ︁ E [ ︁⃦⃦ 𝑣𝑡+1𝑖 − 𝑣 𝑡 𝑖 ⃦⃦2 | 𝑣𝑡𝑖 , 𝑥𝑡 , 𝑥𝑡+1 ] ︁ ] ︁ ( 𝑖 ) = ( 1 − 𝜃 ) E [ ︀ 𝑉 𝑡𝑖 ] ︀ + 𝛽𝑝𝑖E [ ︁⃦⃦ 𝑣𝑡𝑖 −∇𝑓𝑖 ( 𝑥𝑡+1 ) ⃦⃦2 ] ︁ +𝛽 ( 1 − 𝑝𝑖 ) E ⎡⎢⎣ ⃦⃦⃦⃦ ⃦⃦ 1𝜏𝑖 ∑︁ 𝑗∈𝐼𝑡𝑖 ( ︀ ∇𝑓𝑖𝑗 ( 𝑥𝑡+1 ) −∇𝑓𝑖𝑗 ( 𝑥𝑡 ) ) ︀⃦⃦⃦⃦⃦⃦ 2 ⎤⎥⎦ = ( 1 − 𝜃 ) E [ ︀ 𝑉 𝑡𝑖 ] ︀ + 𝛽𝑝𝑖E [ ︁⃦⃦ 𝑣𝑡𝑖 −∇𝑓𝑖 ( 𝑥𝑡+1 ) ⃦⃦2 ] ︁ + 𝛽 ( 1 − 𝑝𝑖 ) E [ ︂⃦⃦⃦ ̃︀∆𝑡𝑖 ⃦⃦⃦2 ] ︂ ( 𝑖𝑖 ) = ( 1 − 𝜃 ) E [ ︀ 𝑉 𝑡𝑖 ] ︀ + 2𝛽𝑝𝑖E [ ︁⃦⃦ 𝑣𝑡𝑖 −∇𝑓𝑖 ( 𝑥𝑡 ) ⃦⃦2 ] ︁ +2𝛽𝑝𝑖E [ ︁⃦⃦ ∇𝑓𝑖 ( 𝑥𝑡+1 ) −∇𝑓𝑖 ( 𝑥𝑡 ) ⃦⃦2 ] ︁ + 𝛽 ( 1 − 𝑝𝑖 ) E [ ︂⃦⃦⃦ ̃︀∆𝑡𝑖 ⃦⃦⃦2 ] ︂ = ( 1 − 𝜃 ) E [ ︀ 𝑉 𝑡𝑖 ] ︀ + 2𝛽𝑝𝑖E [ ︀ 𝑃 𝑡𝑖 ] ︀ + 2𝛽𝑝𝑖E [ ︁⃦⃦ ∆𝑡𝑖 ⃦⃦2 ] ︁ + 𝛽 ( 1 − 𝑝𝑖 ) E [ ︂⃦⃦⃦ ̃︀∆𝑡𝑖 ⃦⃦⃦2 ] ︂ ( 𝑖𝑖𝑖 ) = ( 1 − 𝜃 ) E [ ︀ 𝑉 𝑡𝑖 ] ︀ + 2𝛽𝑝𝑖E [ ︀ 𝑃 𝑡𝑖 ] ︀ + 𝛽 ( 2𝑝𝑖 + 1 − 𝑝𝑖 ) E [ ︁⃦⃦ ∆𝑡𝑖 ⃦⃦2 ] ︁ +𝛽 ( 1 − 𝑝𝑖 ) E [ ︂⃦⃦⃦ ̃︀∆𝑡𝑖 − ∆𝑡𝑖 ⃦⃦⃦2 ] ︂ ( 𝑖𝑣 ) ≤ ( 1 − 𝜃 ) E [ ︀ 𝑉 𝑡𝑖 ] ︀ + 2𝛽𝑝𝑖E [ ︀ 𝑃 𝑡𝑖 ] ︀ + 𝛽 ( 1 + 𝑝𝑖 ) 𝐿 2 𝑖E [ ︁⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 ] ︁ +𝛽 ( 1 − 𝑝𝑖 ) ℒ2𝑖 𝜏𝑖 E [ ︁⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 ] ︁ ≤ ( 1 − 𝜃 ) E [ ︀ 𝑉 𝑡𝑖 ] ︀ + 2𝛽𝑝maxE [ ︀ 𝑃 𝑡𝑖 ] ︀ + 𝛽 ( ︂ 2𝐿2𝑖 + ( 1 − 𝑝𝑖 ) ℒ2𝑖 𝜏𝑖 ) ︂ E [ ︁⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 ] ︁ , where in ( 𝑖 ) we use the definition of PAGE estimator ( 32 ) , ( 𝑖𝑖 ) applies ( 119 ) with 𝑠 = 1 , ( 𝑖𝑖𝑖 ) is due to bias-variance decomposition ( 123 ) , ( 𝑖𝑣 ) makes use of Assumptions 1 and 3 , and the last step is due to 𝑝𝑖 ≤ 1 , 𝑝𝑖 ≤ 𝑝max . It remains to average the above inequality over 𝑖 = 1 , . . . , 𝑛. E.1 CONVERGENCE FOR GENERAL NON-CONVEX FUNCTIONS Theorem 5 . Let Assumptions 1 and 3 hold , and let the stepsize in Algorithm 3 be set as 0 < 𝛾 ≤ ( ︃ 𝐿 + √︃ 4𝛽 𝜃 ̃︀𝐿2 + 2 ( ︂3𝛽 𝜃 𝑝max 𝑝min + 1 𝑝min ) ︂ ̃︀ℒ2 ) ︃−1 . ( 36 ) Fix 𝑇 ≥ 1 and let ? ̂ ? 𝑇 be chosen from the iterates 𝑥0 , 𝑥1 , . . . , 𝑥𝑇−1 uniformly at random . Then E [ ︁⃦⃦ ∇𝑓 ( ? ̂ ? 𝑇 ) ⃦⃦2 ] ︁ ≤ 2Ψ0 𝛾𝑇 , ( 37 ) where Ψ𝑡 def = 𝑓 ( 𝑥𝑡 ) − 𝑓 inf + 𝛾𝜃𝑉 𝑡 + 𝛾𝑝min ( ︁ 1 + 2𝛽𝑝min𝜃 ) ︁ 𝑃 𝑡 , 𝑝max = max𝑖=1 , ... , 𝑛 𝑝𝑖 , 𝑝min = min𝑖=1 , ... , 𝑛 𝑝𝑖 , ̃︀𝐿 = √︁ 1𝑛 ∑︀𝑛𝑖=1 𝐿2𝑖 , 𝜃 = 1 − ( 1 − 𝛼 ) ( 1 + 𝑠 ) , 𝛽 = ( 1 − 𝛼 ) ( ︀1 + 𝑠−1 ) ︀ for any 𝑠 > 0 . Proof . We apply Lemma 16 and split the error ‖𝑔𝑡𝑖 −∇𝑓𝑖 ( 𝑥𝑡 ) ‖ 2 in two parts 𝑓 ( 𝑥𝑡+1 ) ≤ 𝑓 ( 𝑥𝑡 ) − 𝛾 2 ⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 − ( ︂ 1 2𝛾 − 𝐿 2 ) ︂ 𝑅𝑡 + 𝛾 2 ⃦⃦ 𝑔𝑡 −∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 ≤ 𝑓 ( 𝑥𝑡 ) − 𝛾 2 ⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 − ( ︂ 1 2𝛾 − 𝐿 2 ) ︂ 𝑅𝑡 +𝛾 ⃦⃦ 𝑔𝑡 − 𝑣𝑡 ⃦⃦2 + 𝛾E [ ︁⃦⃦ 𝑣𝑡 −∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 ] ︁ ≤ 𝑓 ( 𝑥𝑡 ) − 𝛾 2 ⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 − ( ︂ 1 2𝛾 − 𝐿 2 ) ︂ 𝑅𝑡 +𝛾 1 𝑛 𝑛∑︁ 𝑖=1 ⃦⃦ 𝑔𝑡𝑖 − 𝑣𝑡𝑖 ⃦⃦2 + 𝛾 1 𝑛 𝑛∑︁ 𝑖=1 ⃦⃦ 𝑣𝑡𝑖 −∇𝑓𝑖 ( 𝑥𝑡 ) ⃦⃦2 = 𝑓 ( 𝑥𝑡 ) − 𝛾 2 ⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 − ( ︂ 1 2𝛾 − 𝐿 2 ) ︂ 𝑅𝑡 + 𝛾𝑉 𝑡 + 𝛾𝑃 𝑡 , ( 38 ) where we used notation 𝑅𝑡 = ‖𝛾𝑔𝑡‖2 = ⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 , and applied ( 118 ) and ( 119 ) . Subtracting 𝑓 inf from both sides of the above inequality , taking expectation and using the notation 𝛿𝑡 = 𝑓 ( 𝑥𝑡+1 ) − 𝑓 inf , we get E [ ︀ 𝛿𝑡+1 ] ︀ ≤ E [ ︀ 𝛿𝑡 ] ︀ − 𝛾 2 E [ ︁⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 ] ︁− ( ︂ 1 2𝛾 − 𝐿 2 ) ︂ E [ ︀ 𝑅𝑡 ] ︀ + 𝛾E [ ︀ 𝑉 𝑡 ] ︀ + 𝛾E [ ︀ 𝑃 𝑡 ] ︀ . ( 39 ) Further , Lemma 3 and 4 provide the recursive bounds for the last two terms of ( 39 ) E [ ︀ 𝑃 𝑡+1 ] ︀ ≤ ( 1 − 𝑝min ) E [ ︀ 𝑃 𝑡 ] ︀ + ̃︀ℒ2E [ 𝑅𝑡 ] , ( 40 ) E [ ︀ 𝑉 𝑡+1 ] ︀ ≤ ( 1 − 𝜃 ) E [ ︀ 𝑉 𝑡 ] ︀ + 𝛽 ( ︁ 2̃︀𝐿2 + ̃︀ℒ2 ) ︁E [ 𝑅𝑡 ] + 2𝛽𝑝maxE [ ︀𝑃 𝑡 ] ︀ . ( 41 ) Adding ( 39 ) with a 𝛾𝜃 multiple of ( 41 ) we obtain E [ ︀ 𝛿𝑡+1 ] ︀ + 𝛾 𝜃 E [ ︀ 𝑉 𝑡+1 ] ︀ ≤ E [ ︀ 𝛿𝑡 ] ︀ − 𝛾 2 E [ ︁⃦⃦ ∇𝑓 ( ︀ 𝑥𝑡 ) ︀⃦⃦2 ] ︁− ( ︂ 1 2𝛾 − 𝐿 2 ) ︂ E [ ︀ 𝑅𝑡 ] ︀ + 𝛾E [ ︀ 𝑉 𝑡 ] ︀ +𝛾E [ ︀ 𝑃 𝑡 ] ︀ + 𝛾 𝜃 ( ︀ ( 1 − 𝜃 ) E [ ︀ 𝑉 𝑡 ] ︀ + 𝐴𝑟𝑡 + 𝐶E [ ︀ 𝑃 𝑡 ] ︀ ) ︀ ≤ 𝛿𝑡 + 𝛾 𝜃 E [ ︀ 𝑉 𝑡 ] ︀ − 𝛾 2 E [ ︁⃦⃦ ∇𝑓 ( ︀ 𝑥𝑡 ) ︀⃦⃦2 ] ︁− ( ︂ 1 2𝛾 − 𝐿 2 − 𝛾𝐴 𝜃 ) ︂ E [ ︀ 𝑅𝑡 ] ︀ +𝛾 ( ︂ 1 + 𝐶 𝜃 ) ︂ E [ ︀ 𝑃 𝑡 ] ︀ , where we denote 𝐴 def= 𝛽 ( ︁ 2̃︀𝐿2 + ̃︀ℒ2 ) ︁ , 𝐶 def= 2𝛽𝑝max . Then adding the above inequality with a 𝛾𝑝min ( ︀ 1 + 𝐶𝜃 ) ︀ multiple of ( 40 ) , we get E [ ︀ Φ𝑡+1 ] ︀ = E [ ︀ 𝛿𝑡+1 ] ︀ + 𝛾 𝜃 E [ ︀ 𝑉 𝑡+1 ] ︀ + 𝛾 𝑝min ( ︂ 1 + 𝐶 𝜃 ) ︂ E [ ︀ 𝑃 𝑡+1 ] ︀ ≤ 𝛿𝑡 + 𝛾 𝜃 E [ ︀ 𝑉 𝑡 ] ︀ − 𝛾 2 E [ ︁⃦⃦ ∇𝑓 ( ︀ 𝑥𝑡 ) ︀⃦⃦2 ] ︁− ( ︂ 1 2𝛾 − 𝐿 2 − 𝛾𝐴 𝜃 ) ︂ E [ ︀ 𝑅𝑡 ] ︀ +𝛾 ( ︂ 1 + 𝐶 𝜃 ) ︂ E [ ︀ 𝑃 𝑡 ] ︀ + 𝛾 𝑝min ( ︂ 1 + 𝐶 𝜃 ) ︂ ( ︁ ( 1 − 𝑝min ) E [ ︀ 𝑃 𝑡 ] ︀ + ̃︀ℒ2E [ ︀𝑅𝑡 ] ︀ ) ︁ ≤ E [ ︀ 𝛿𝑡 ] ︀ + 𝛾 𝜃 E [ ︀ 𝑉 𝑡 ] ︀ + 𝛾 𝑝min ( ︂ 1 + 𝐶 𝜃 ) ︂ E [ ︀ 𝑃 𝑡 ] ︀ − 𝛾 2 E [ ︁⃦⃦ ∇𝑓 ( ︀ 𝑥𝑡 ) ︀⃦⃦2 ] ︁ − ( ︂ 1 2𝛾 − 𝐿 2 − 𝛾𝐴 𝜃 − 𝛾 𝑝min ( ︂ 1 + 𝐶 𝜃 ) ︂ ̃︀ℒ2 ) ︂E [ ︀𝑅𝑡 ] ︀ = E [ ︀ Φ𝑡 ] ︀ − 𝛾 2 E [ ︁⃦⃦ ∇𝑓 ( ︀ 𝑥𝑡 ) ︀⃦⃦2 ] ︁ − ( ︂ 1 2𝛾 − 𝐿 2 − 𝛾𝐴 𝜃 − 𝛾 𝑝min ( ︂ 1 + 𝐶 𝜃 ) ︂ ̃︀ℒ2 ) ︂E [ ︀𝑅𝑡 ] ︀ . ( 42 ) The coefficient in front of E [ 𝑅𝑡 ] simplifies after substitution by 𝐴 and 𝐶 𝛾𝐴 𝜃 + 𝛾 𝑝min ( ︂ 1 + 𝐶 𝜃 ) ︂ ̃︀ℒ2 ≤ 2𝛽 𝜃 ̃︀𝐿2 + ( ︂3𝛽 𝜃 𝑝max 𝑝min + 1 𝑝min ) ︂ ̃︀ℒ2 . Thus by Lemma 15 and the stepsize choice 0 < 𝛾 ≤ ( ︃ 𝐿 + √︃ 4𝛽 𝜃 ̃︀𝐿2 + 2 ( ︂3𝛽 𝜃 𝑝max 𝑝min + 1 𝑝min ) ︂ ̃︀ℒ2 ) ︃−1 ( 43 ) the last term in ( 42 ) is not positive . By summing up inequalities for 𝑡 = 0 , . . . , 𝑇 − 1 , we get 0 ≤ E [ ︀ Φ𝑇 ] ︀ ≤ E [ ︀ Φ0 ] ︀ − 𝛾 2 𝑇−1∑︁ 𝑡=0 E [ ︁⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 ] ︁ . Multiplying both sides by 2𝛾𝑇 and rearranging we get 𝑇−1∑︁ 𝑡=0 1 𝑇 E [ ︁⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 ] ︁ ≤ 2E [ ︀Φ0 ] ︀ 𝛾𝑇 . It remains to notice that the left hand side can be interpreted as E [ ︁⃦⃦ ∇𝑓 ( ? ̂ ? 𝑇 ) ⃦⃦2 ] ︁ , where ? ̂ ? 𝑇 is chosen from 𝑥0 , 𝑥1 , . . . , 𝑥𝑇−1 uniformly at random . Corollary 10 . Let assumptions of Theorem 5 hold , 𝑣0𝑖 = 𝑔 0 𝑖 = ∇𝑓𝑖 ( 𝑥0 ) , 𝑖 = 1 , . . . , 𝑛 , 𝛾 = ( ︃ 𝐿 + √︃ 4𝛽 𝜃 ̃︀𝐿2 + 2 ( ︂3𝛽 𝜃 𝑝max 𝑝min + 1 𝑝min ) ︂ ̃︀ℒ2 ) ︃−1 , 𝑝𝑖 = 𝜏𝑖 𝜏𝑖 + 𝑚𝑖 , 𝑖 = 1 , . . . , 𝑛 . Then , after 𝑇 iterations/communication rounds of EF21-PAGE we have E [ ︁⃦⃦ ∇𝑓 ( ? ̂ ? 𝑇 ) ⃦⃦2 ] ︁ ≤ 𝜀2 . It requires 𝑇 = 𝒪 ( ︃ ( ̃︀𝐿 + ̃︀ℒ ) 𝛿0 𝛼𝜀2 √︂ 𝑝max 𝑝min + √ 𝑚max ̃︀ℒ𝛿0 𝜀2 ) ︃ iterations/communications rounds , # grad𝑖 = 𝒪 ( ︃ 𝑚𝑖 + 𝜏𝑖 ( ̃︀𝐿 + ̃︀ℒ ) 𝛿0 𝛼𝜀2 √︂ 𝑝max 𝑝min + 𝜏𝑖 √ 𝑚max ̃︀ℒ𝛿0 𝜀2 ) ︃ stochastic oracle calls for worker 𝑖 , and # grad = 𝒪 ( ︃ 𝑚 + 𝜏 ( ̃︀𝐿 + ̃︀ℒ ) 𝛿0 𝛼𝜀2 √︂ 𝑝max 𝑝min + 𝜏 √ 𝑚max ̃︀ℒ𝛿0 𝜀2 ) ︃ stochastic oracle calls per worker on average , where 𝜏 = 1𝑛 ∑︀𝑛 𝑖=1 𝜏𝑖 , 𝑚 = 1 𝑛 ∑︀𝑛 𝑖=1 𝑚𝑖 , 𝑚max = max𝑖=1 , ... , 𝑛 𝑚𝑖 , 𝑝max = max𝑖=1 , ... , 𝑛 𝑝𝑖 , 𝑝min = min𝑖=1 , ... , 𝑛 𝑝𝑖 . Proof . Notice that by Lemma 17 we have 𝐿 + √︃ 4𝛽 𝜃 ̃︀𝐿2 + 2 ( ︂3𝛽 𝜃 𝑝max 𝑝min + 1 𝑝min ) ︂ ̃︀ℒ2 ≤ 𝐿 +√︃ 16 𝛼2 ̃︀𝐿2 + 2 ( ︂ 12 𝛼2 𝑝max 𝑝min + 1 𝑝min ) ︂ ̃︀ℒ2 ≤ 𝐿 + 4 𝛼 ̃︀𝐿 +√︂ 24 𝛼2 𝑝max 𝑝min + 2 𝑝min ̃︀ℒ ≤ 𝐿 + 4 𝛼 ̃︀𝐿 + √24 𝛼 √︂ 𝑝max 𝑝min ̃︀ℒ + √2√ 𝑝min ̃︀ℒ ≤ 5 𝛼 ̃︀𝐿 + √24 𝛼 √︂ 𝑝max 𝑝min ̃︀ℒ + √2√ 𝑝min ̃︀ℒ ≤ 5 𝛼 √︂ 𝑝max 𝑝min ( ︁̃︀𝐿 + ̃︀ℒ ) ︁+ √2√ 𝑝min ̃︀ℒ ≤ 5 𝛼 √︂ 𝑝max 𝑝min ( ︁̃︀𝐿 + ̃︀ℒ ) ︁+ 2√𝑚max ̃︀ℒ , where we used 𝐿 ≤ ̃︀𝐿 , 𝑝min ≤ 𝑝max , and the fact that √𝑎 + 𝑏 ≤ √𝑎 + √𝑏 for 𝑎 , 𝑏 ≥ 0 . Then the number of communication rounds 𝑇 ≤ 2𝛿 0 𝛾𝜀2 ≤ 2𝛿 0 𝜀2 ( ︂ 5 𝛼 √︂ 𝑝max 𝑝min ( ︁̃︀𝐿 + ̃︀ℒ ) ︁+ 2√𝑚max ̃︀ℒ ) ︂ = 𝒪 ( ︃ ( ̃︀𝐿 + ̃︀ℒ ) 𝛿0 𝛼𝜀2 √︂ 𝑝max 𝑝min + √ 𝑚max ̃︀ℒ𝛿0 𝜀2 ) ︃ . At each worker , we have # grad𝑖 = 𝑚𝑖 + 𝑇 ( 𝑝𝑖𝑚𝑖 + ( 1 − 𝑝𝑖 ) 𝜏𝑖 ) = 𝑚𝑖 + 2𝑚𝑖𝜏𝑖 𝜏𝑖 + 𝑚𝑖 𝑇 ≤ 𝑚𝑖 + 2𝜏𝑖𝑇 . Averaging over 𝑖 = 1 , . . . , 𝑛 , we get # grad ≤ 𝑚 + 2𝜏𝑇 = 𝒪 ( ︃ 𝑚 + 𝜏 ( ̃︀𝐿 + ̃︀ℒ ) 𝛿0 𝛼𝜀2 √︂ 𝑝max 𝑝min + 𝜏 √ 𝑚max ̃︀ℒ𝛿0 𝜀2 ) ︃ . E.2 CONVERGENCE UNDER POLYAK-ŁOJASIEWICZ CONDITION Theorem 6 . Let Assumptions 1 and 4 hold , and let the stepsize in Algorithm 3 be set as 0 < 𝛾 ≤ min { ︂ 𝛾0 , 𝜃 2𝜇 , 𝑝min 2𝜇 } ︂ , ( 44 ) where 𝛾0 def = 0 < 𝛾 ≤ ( ︂ 𝐿 + √︂ 8𝛽 𝜃 ̃︀𝐿2 + 4 ( ︁ 5𝛽𝜃 𝑝max𝑝min + 1𝑝min ) ︁ ̃︀ℒ2 ) ︂−1 , ̃︀𝐿 = √︁ 1𝑛 ∑︀𝑛𝑖=1 𝐿2𝑖 , 𝜃 = 1 − ( 1 − 𝛼 ) ( 1 + 𝑠 ) , 𝛽 = ( 1 − 𝛼 ) ( ︀ 1 + 𝑠−1 ) ︀ for any 𝑠 > 0 . Let Ψ𝑡 def = 𝑓 ( 𝑥𝑡 ) − 𝑓 ( 𝑥⋆ ) + 2𝛾𝜃 𝑉 𝑡 + 2𝛾𝑝min ( ︁ 1 + 4𝛽𝑝max𝜃 ) ︁ 𝑃 𝑡 . Then for any 𝑇 ≥ 0 , we have E [ ︀ Ψ𝑇 ] ︀ ≤ ( 1 − 𝛾𝜇 ) 𝑇E [ ︀ Ψ0 ] ︀ . ( 45 ) Proof . Similarly to the proof of Theorem 5 the inequalities ( 39 ) , ( 40 ) , ( 41 ) hold with 𝛿𝑡 = 𝑓 ( 𝑥𝑡 ) − 𝑓 ( 𝑥⋆ ) . Adding ( 39 ) with a 2𝛾𝜃 multiple of ( 41 ) we obtain E [ ︀ 𝛿𝑡+1 ] ︀ + 2𝛾 𝜃 E [ ︀ 𝑉 𝑡+1 ] ︀ ≤ E [ ︀ 𝛿𝑡 ] ︀ − 𝛾 2 E [ ︁⃦⃦ ∇𝑓 ( ︀ 𝑥𝑡 ) ︀⃦⃦2 ] ︁− ( ︂ 1 2𝛾 − 𝐿 2 ) ︂ E [ ︀ 𝑅𝑡 ] ︀ + 𝛾E [ ︀ 𝑉 𝑡 ] ︀ +𝛾E [ ︀ 𝑃 𝑡 ] ︀ + 2𝛾 𝜃 ( ︀ ( 1 − 𝜃 ) E [ ︀ 𝑉 𝑡 ] ︀ + 𝐴𝑟𝑡 + 𝐶E [ ︀ 𝑃 𝑡 ] ︀ ) ︀ ≤ 𝛿𝑡 + 2𝛾 𝜃 E [ ︀ 𝑉 𝑡 ] ︀ ( ︂ 1 − 𝜃 2 ) ︂ −𝛾 2 E [ ︁⃦⃦ ∇𝑓 ( ︀ 𝑥𝑡 ) ︀⃦⃦2 ] ︁− ( ︂ 1 2𝛾 − 𝐿 2 − 2𝛾𝐴 𝜃 ) ︂ E [ ︀ 𝑅𝑡 ] ︀ +𝛾 ( ︂ 1 + 2𝐶 𝜃 ) ︂ E [ ︀ 𝑃 𝑡 ] ︀ , where 𝐴 def= 𝛽 ( ︁ 2̃︀𝐿2 + ̃︀ℒ2 ) ︁ , 𝐶 def= 2𝛽𝑝max . Then adding the above inequality with a 2𝛾𝑝min ( ︀ 1 + 2𝐶𝜃 ) ︀ multiple of ( 40 ) , we get E [ ︀ Ψ𝑡+1 ] ︀ = E [ ︀ 𝛿𝑡+1 ] ︀ + 2𝛾 𝜃 E [ ︀ 𝑉 𝑡+1 ] ︀ + 2𝛾 𝑝min ( ︂ 1 + 2𝐶 𝜃 ) ︂ E [ ︀ 𝑃 𝑡+1 ] ︀ ≤ 𝛿𝑡 + 𝛾 𝜃 E [ ︀ 𝑉 𝑡 ] ︀ ( ︂ 1 − 𝜃 2 ) ︂ − 𝛾 2 E [ ︁⃦⃦ ∇𝑓 ( ︀ 𝑥𝑡 ) ︀⃦⃦2 ] ︁− ( ︂ 1 2𝛾 − 𝐿 2 − 2𝛾𝐴 𝜃 ) ︂ E [ ︀ 𝑅𝑡 ] ︀ +𝛾 ( ︂ 1 + 2𝐶 𝜃 ) ︂ E [ ︀ 𝑃 𝑡 ] ︀ + 2𝛾 𝑝min ( ︂ 1 + 2𝐶 𝜃 ) ︂ ( ︁ ( 1 − 𝑝min ) E [ ︀ 𝑃 𝑡 ] ︀ + ̃︀ℒ2E [ ︀𝑅𝑡 ] ︀ ) ︁ ≤ E [ ︀ 𝛿𝑡 ] ︀ + 2𝛾 𝜃 E [ ︀ 𝑉 𝑡 ] ︀ ( ︂ 1 − 𝜃 2 ) ︂ + 2𝛾 𝑝min ( ︂ 1 + 2𝐶 𝜃 ) ︂ E [ ︀ 𝑃 𝑡 ] ︀ ( ︁ 1 − 𝑝min 2 ) ︁ −𝛾 2 E [ ︁⃦⃦ ∇𝑓 ( ︀ 𝑥𝑡 ) ︀⃦⃦2 ] ︁− ( ︂ 1 2𝛾 − 𝐿 2 − 2𝛾𝐴 𝜃 − 2𝛾 𝑝min ( ︂ 1 + 2𝐶 𝜃 ) ︂ ̃︀ℒ2 ) ︂E [ ︀𝑅𝑡 ] ︀ . ( 46 ) PL inequality implies that 𝛿𝑡 − 𝛾2 ‖∇𝑓 ( 𝑥 𝑡 ) ‖2 ≤ ( 1 − 𝛾𝜇 ) 𝛿𝑡 . In view of the above inequality and our assumption on the stepsize ( 𝛾 ≤ 𝜃2𝜇 , 𝛾 ≤ 𝑝min 2𝜇 ) , we get E [ ︀ Ψ𝑡+1 ] ︀ ≤ ( 1 − 𝛾𝜇 ) E [ ︀ Ψ𝑡 ] ︀ − ( ︂ 1 2𝛾 − 𝐿 2 − 2𝛾𝐴 𝜃 − 2𝛾 𝑝min ( ︂ 1 + 2𝐶 𝜃 ) ︂ ̃︀ℒ2 ) ︂E [ ︀𝑅𝑡 ] ︀ . The coefficient in front of E [ 𝑅𝑡 ] simplifies after substitution by 𝐴 and 𝐶 2𝛾𝐴 𝜃 + 2𝛾 𝑝min ( ︂ 1 + 2𝐶 𝜃 ) ︂ ̃︀ℒ2 = 4𝛽 𝜃 ̃︀𝐿2 + ( ︂2𝛽 𝜃 + 2 𝑝min + 8𝛽 𝜃 𝑝max 𝑝min ) ︂ ̃︀ℒ2 ≤ 4𝛽 𝜃 ̃︀𝐿2 + 2 ( ︂5𝛽 𝜃 𝑝max 𝑝min + 1 𝑝min ) ︂ ̃︀ℒ2 . Thus by Lemma 15 and the stepsize choice 0 < 𝛾 ≤ ( ︃ 𝐿 + √︃ 8𝛽 𝜃 ̃︀𝐿2 + 4 ( ︂5𝛽 𝜃 𝑝max 𝑝min + 1 𝑝min ) ︂ ̃︀ℒ2 ) ︃−1 ( 47 ) the last term in ( 46 ) is not positive . E [ ︀ Ψ𝑡+1 ] ︀ ≤ ( 1 − 𝛾𝜇 ) E [ ︀ Ψ𝑡 ] ︀ . It remains to unroll the recurrence . Corollary 11 . Let assumptions of Theorem 6 hold , 𝑣0𝑖 = 𝑔 0 𝑖 = ∇𝑓𝑖 ( 𝑥0 ) , 𝑖 = 1 , . . . , 𝑛 , 𝛾 = min { ︂ 𝛾0 , 𝜃 2𝜇 , 𝑝min 2𝜇 } ︂ , 𝛾0 = ( ︃ 𝐿 + √︃ 8𝛽 𝜃 ̃︀𝐿2 + 4 ( ︂5𝛽 𝜃 𝑝max 𝑝min + 1 𝑝min ) ︂ ̃︀ℒ2 ) ︃−1 , 𝑝𝑖 = 𝜏𝑖 𝜏𝑖 + 𝑚𝑖 , 𝑖 = 1 , . . . , 𝑛 . Then , after 𝑇 iterations of EF21-PAGE we have E [ ︀ 𝑓 ( 𝑥𝑇 ) − 𝑓 inf ] ︀ ≤ 𝜀 . It requires 𝑇 = 𝒪 ( ︃ 1 𝜇 ( ︃ ̃︀𝐿 + ̃︀ℒ 𝛼 √︂ 𝑝max 𝑝min + √ 𝑚max ̃︀ℒ ) ︃ ln ( ︂𝛿0 𝜀 ) ︂ ) ︃ iterations/communications rounds , # grad𝑖 = 𝒪 ( ︃ 𝑚𝑖 + 𝜏𝑖 𝜇 ( ︃ ̃︀𝐿 + ̃︀ℒ 𝛼 √︂ 𝑝max 𝑝min + √ 𝑚max ̃︀ℒ ) ︃ ln ( ︂𝛿0 𝜀 ) ︂ ) ︃ stochastic oracle calls for worker 𝑖 , and # grad = 𝒪 ( ︃ 𝑚 + 𝜏 𝜇 ( ︃ ̃︀𝐿 + ̃︀ℒ 𝛼 √︂ 𝑝max 𝑝min + √ 𝑚max ̃︀ℒ ) ︃ ln ( ︂𝛿0 𝜀 ) ︂ ) ︃ stochastic oracle calls per worker on average , where 𝜏 = 1𝑛 ∑︀𝑛 𝑖=1 𝜏𝑖 , 𝑚 = 1 𝑛 ∑︀𝑛 𝑖=1 𝑚𝑖 , 𝑚max = max𝑖=1 , ... , 𝑛 𝑚𝑖 , 𝑝max = max𝑖=1 , ... , 𝑛 𝑝𝑖 , 𝑝min = min𝑖=1 , ... , 𝑛 𝑝𝑖 . F PARTIAL PARTICIPATION In this section , we provide an option for partial participation of the clients – a feature important in federated learning . Most of the works in compressed distributed optimization deal with full worker participation , i.e. , the case when all clients are involved in computation and communication at every iteration . However , in the practice of federated learning , only a subset of clients are allowed to participate at each training round . This limitation comes mainly due to the following two reasons . First , clients ( e.g. , mobile devices ) may wish to join or leave the network randomly . Second , it is often prohibitive to wait for all available clients since stragglers can significantly slow down the training process . Although many existing works ( Gorbunov et al. , 2021 ; Horváth & Richtárik , 2021 ; Philippenko & Dieuleveut , 2020 ; Karimireddy et al. , 2020 ; Yang et al. , 2021 ; Cho et al. , 2020 ) allow for partial participation , they assume either unbiased compressors or no compression at all . We provide a simple analysis of partial participation , which works with biased compressors and builds upon the EF21 mechanism . The modified method ( Algorithm 4 ) is called EF21-PP . At each iteration of EF21-PP , the master samples a subset 𝑆𝑡 of clients ( nodes ) , which are required to perform computation . Note , that all other clients ( nodes ) 𝑖 /∈ 𝑆𝑡 participate neither in the computation nor in communication at iteration 𝑡 . We allow for an arbitrary sampling strategy of a subset 𝑆𝑡 at the master node . The only requirement is that Prob ( 𝑖 ∈ 𝑆𝑡 ) = 𝑝𝑖 > 0 for all 𝑖 = 1 , . . . , 𝑛 , which is often referred to as a proper arbitrary sampling.9 Clearly , many poplular sampling procedures fell into this setting , for instance , independent sampling with/without replacement , 𝜏 -nice sampling . We do not discuss particular sampling strategies here , more on samplings can be found in ( Qu & Richtárik , 2014 ) . Algorithm 4 EF21-PP ( EF21 with partial participation ) 1 : Input : starting point 𝑥0 ∈ R𝑑 ; 𝑔0𝑖 ∈ R𝑑 for 𝑖 = 1 , . . . , 𝑛 ( known by nodes ) ; 𝑔0 = 1𝑛 ∑︀𝑛 𝑖=1 𝑔 0 𝑖 ( known by master ) ; learning rate 𝛾 > 0 2 : for 𝑡 = 0,1 , 2 , . . . , 𝑇 − 1 do 3 : Master computes 𝑥𝑡+1 = 𝑥𝑡 − 𝛾𝑔𝑡 4 : Master samples a subset 𝑆𝑡 of nodes ( |𝑆𝑡| ≤ 𝑛 ) such that Prob ( 𝑖 ∈ 𝑆𝑡 ) = 𝑝𝑖 5 : Master broadcasts 𝑥𝑡+1 to the nodes with 𝑖 ∈ 𝑆𝑡 6 : for all nodes 𝑖 = 1 , . . . , 𝑛 in parallel do 7 : if 𝑖 ∈ 𝑆𝑡 then 8 : Compress 𝑐𝑡𝑖 = 𝒞 ( ∇𝑓𝑖 ( 𝑥𝑡+1 ) − 𝑔𝑡𝑖 ) and send 𝑐𝑡𝑖 to the master 9 : Update local state 𝑔𝑡+1𝑖 = 𝑔 𝑡 𝑖 + 𝑐 𝑡 𝑖 10 : end if 11 : if 𝑖 /∈ 𝑆𝑡 then 12 : Do not change local state 𝑔𝑡+1𝑖 = 𝑔 𝑡 𝑖 13 : end if 14 : end for 15 : Master updates 𝑔𝑡+1𝑖 = 𝑔 𝑡 𝑖 , 𝑐 𝑡 𝑖 = 0 for 𝑖 /∈ 𝑆𝑡 16 : Master computes 𝑔𝑡+1 = 1𝑛 ∑︀𝑛 𝑖=1 𝑔 𝑡+1 𝑖 via 𝑔 𝑡+1 = 𝑔𝑡 + 1𝑛 ∑︀𝑛 𝑖=1 𝑐 𝑡 𝑖 17 : end for Lemma 5 . Then for Algorithm 4 holds E [ ︀ 𝐺𝑡+1 ] ︀ ≤ ( 1 − 𝜃𝑝 ) E [ ︀ 𝐺𝑡 ] ︀ + 𝐵E [ ︁⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 ] ︁ ( 48 ) with 𝜃𝑝 def = 𝜌𝑝𝑚𝑖𝑛 + 𝜃𝑝𝑚𝑎𝑥 − 𝜌− ( 𝑝𝑚𝑎𝑥 − 𝑝𝑚𝑖𝑛 ) , 𝐵 def = 1𝑛 ∑︀𝑛 𝑖=1 ( ︀ 𝛽𝑝𝑖 + ( ︀ 1 + 𝜌−1 ) ︀ ( 1 − 𝑝𝑖 ) ) ︀ 𝐿2𝑖 , 𝑝𝑚𝑎𝑥 def = max1≤𝑖≤𝑛 𝑝𝑖 , 𝑝𝑚𝑖𝑛 def = min1≤𝑖≤𝑛 𝑝𝑖 , 𝜃 = 1 − ( 1 + 𝑠 ) ( 1 − 𝛼 ) , 𝛽 = ( ︀ 1 + 1𝑠 ) ︀ ( 1 − 𝛼 ) and small enough 𝜌 , 𝑠 > 0 . Proof . By ( 13 ) in Lemma 1 , we have for all 𝑖 ∈ 𝑆𝑡 E [ ︀ 𝐺𝑡+1𝑖 | 𝑖 ∈ 𝑆𝑡 ] ︀ ≤ ( 1 − 𝜃 ) E [ ︀ 𝐺𝑡𝑖 ] ︀ + 𝛽𝐿2𝑖E [ ︁⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 | 𝑖 ∈ 𝑆𝑡 ] ︁ ( 49 ) 9It is natural to focus on proper samplings only since otherwise there is a node 𝑖 , which never communicaties . This would be a critical issue when trying to minimize ( 1 ) as we do not assume any similarity between 𝑓𝑖 ( · ) . with 𝜃 = 1 − ( 1 + 𝑠 ) ( 1 − 𝛼 ) , 𝛽 = ( ︀ 1 + 1𝑠 ) ︀ ( 1 − 𝛼 ) and arbitrary 𝑠 > 0 . Define 𝑊 𝑡 def= { 𝑔𝑡1 , . . . , 𝑔𝑡𝑛 , 𝑥𝑡 , 𝑥𝑡+1 } and let 𝑖 /∈ 𝑆𝑡 , then E [ ︀ 𝐺𝑡+1𝑖 | 𝑖 /∈ 𝑆𝑡 ] ︀ = E [ ︀ E [ ︀ 𝐺𝑡+1𝑖 | 𝑊 𝑡 ] ︀ | 𝑖 /∈ 𝑆𝑡 ] ︀ = E [ ︁ E [ ︁⃦⃦ 𝑔𝑡+1𝑖 −∇𝑓𝑖 ( 𝑥 𝑡+1 ) ⃦⃦2 | 𝑊 𝑡 ] ︁ | 𝑖 /∈ 𝑆𝑡 ] ︁ ≤ ( 1 + 𝜌 ) E [ ︁ E [ ︁⃦⃦ 𝑔𝑡𝑖 −∇𝑓𝑖 ( 𝑥𝑡 ) ⃦⃦2 | 𝑊 𝑡 ] ︁ | 𝑖 /∈ 𝑆𝑡 ] ︁ + ( ︀ 1 + 𝜌−1 ) ︀ E [ ︁ E [ ︁⃦⃦ ∇𝑓𝑖 ( 𝑥𝑡+1 ) −∇𝑓𝑖 ( 𝑥𝑡 ) ⃦⃦2 | 𝑊 𝑡 ] ︁ | 𝑖 /∈ 𝑆𝑡 ] ︁ ≤ ( 1 + 𝜌 ) E [ ︀ 𝐺𝑡𝑖 ] ︀ + ( ︀ 1 + 𝜌−1 ) ︀ E [ ︁⃦⃦ ∇𝑓𝑖 ( 𝑥𝑡+1 ) −∇𝑓𝑖 ( 𝑥𝑡 ) ⃦⃦2 | 𝑖 /∈ 𝑆𝑡 ] ︁ ≤ ( 1 + 𝜌 ) E [ ︀ 𝐺𝑡𝑖 ] ︀ + ( ︀ 1 + 𝜌−1 ) ︀ 𝐿2𝑖E [ ︁⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 ] ︁ . ( 50 ) Combining ( 49 ) and ( 50 ) , we get E [ ︀ 𝐺𝑡+1 ] ︀ = 1 𝑛 𝑛∑︁ 𝑖=1 E [ ︀ 𝐺𝑡+1𝑖 ] ︀ = 1 𝑛 𝑛∑︁ 𝑖=1 𝑝𝑖E [ ︀ 𝐺𝑡+1𝑖 | 𝑖 ∈ 𝑆𝑡 ] ︀ + 1 𝑛 𝑛∑︁ 𝑖=1 ( 1 − 𝑝𝑖 ) E [ ︀ 𝐺𝑡+1𝑖 | 𝑖 /∈ 𝑆𝑡 ] ︀ ( 49 ) , ( 50 ) ≤ ( 1 − 𝜃 ) 1 𝑛 𝑛∑︁ 𝑖=1 𝑝𝑖E [ ︀ 𝐺𝑡𝑖 ] ︀ + 𝛽 ( ︃ 1 𝑛 𝑛∑︁ 𝑖=1 𝑝𝑖𝐿 2 𝑖 ) ︃ E [ ︁⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 ] ︁ + ( 1 + 𝜌 ) 1 𝑛 𝑛∑︁ 𝑖=1 ( 1 − 𝑝𝑖 ) E [ ︀ 𝐺𝑡𝑖 ] ︀ + ( ︀ 1 + 𝜌−1 ) ︀ ( ︃ 1 𝑛 𝑛∑︁ 𝑖=1 ( 1 − 𝑝𝑖 ) 𝐿2𝑖 ) ︃ E [ ︁⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 ] ︁ ( 𝑖 ) ≤ ( 1 − 𝜃 ) 𝑝𝑚𝑎𝑥 1 𝑛 𝑛∑︁ 𝑖=1 E [ ︀ 𝐺𝑡𝑖 ] ︀ + 𝛽 ( ︃ 1 𝑛 𝑛∑︁ 𝑖=1 𝑝𝑖𝐿 2 𝑖 ) ︃ E [ ︁⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 ] ︁ + ( 1 + 𝜌 ) ( 1 − 𝑝𝑚𝑖𝑛 ) 1 𝑛 𝑛∑︁ 𝑖=1 E [ ︀ 𝐺𝑡𝑖 ] ︀ + ( ︀ 1 + 𝜌−1 ) ︀ ( ︃ 1 𝑛 𝑛∑︁ 𝑖=1 ( 1 − 𝑝𝑖 ) 𝐿2𝑖 ) ︃ E [ ︁⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 ] ︁ = ( ︂ ( 1 − 𝜃 ) 𝑝𝑚𝑎𝑥 + ( 1 + 𝜌 ) ( 1 − 𝑝𝑚𝑖𝑛 ) ) ︂ E [ ︀ 𝐺𝑡 ] ︀ + ( ︃ 1 𝑛 𝑛∑︁ 𝑖=1 ( ︀ 𝛽𝑝𝑖 + ( ︀ 1 + 𝜌−1 ) ︀ ( 1 − 𝑝𝑖 ) ) ︀ 𝐿2𝑖 ) ︃ E [ ︁⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 ] ︁ = ( ︂ 1 − ( 𝜌𝑝𝑚𝑖𝑛 + 𝜃𝑝𝑚𝑎𝑥 − 𝜌− ( 𝑝𝑚𝑎𝑥 − 𝑝𝑚𝑖𝑛 ) ) ) ︂ E [ ︀ 𝐺𝑡 ] ︀ + ( ︃ 1 𝑛 𝑛∑︁ 𝑖=1 ( ︀ 𝛽𝑝𝑖 + ( ︀ 1 + 𝜌−1 ) ︀ ( 1 − 𝑝𝑖 ) ) ︀ 𝐿2𝑖 ) ︃ E [ ︁⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 ] ︁ . = ( 1 − 𝜃𝑝 ) E [ ︀ 𝐺𝑡 ] ︀ + 𝐵E [ ︁⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 ] ︁ , Lemma 6 . [ To simplify the rates for partial participation ] Let 𝐵 and 𝜃𝑝 be defined as in Theorems 7 and Theorems 8 , and let 𝑝𝑖 = 𝑝 > 0 for all 𝑖 = 1 , . . . , 𝑛 . Then there exist 𝜌 , 𝑠 > 0 such that 𝜃𝑝 ≥ 𝑝𝛼 2 , ( 51 ) 0 < 𝐵 𝜃𝑝 ≤ ( ︃ 4̃︀𝐿 𝑝𝛼 ) ︃2 . ( 52 ) Proof . Under the assumption that 𝑝𝑖 = 𝑝 for all 𝑖 = 1 , . . . , 𝑛 , the constants simplify to 𝜃𝑝 = 𝜌𝑝 + 𝜃𝑝− 𝜌 , 𝐵 = ( ︀ 𝛽𝑝 + ( ︀ 1 + 𝜌−1 ) ︀ ( 1 − 𝑝 ) ) ︀ ̃︀𝐿2 , 𝑝𝑚𝑎𝑥 = 𝑝𝑚𝑖𝑛 = 𝑝 . Case I : let 𝛼 = 1 , 𝑝 = 1 , then the result holds trivially . Case II : let 0 < 𝛼 < 1 , 𝑝 = 1 , then 𝐵 = 𝛽̃︀𝐿2 , 𝜃𝑝 = 𝜃 = 1 − √1 − 𝛼 ≥ 𝛼2 and ( 52 ) follows by Lemma 17 . Case III : let 𝛼 = 1 , and 0 < 𝑝 < 1 , then 𝜃 = 1 , 𝛽 = 0 , 𝐵 = ( ︀ 1 + 𝜌−1 ) ︀ ( 1−𝑝 ) ̃︀𝐿2 , 𝜃𝑝 = 𝑝−𝜌 ( 1−𝑝 ) . Then the choice 𝜌 = 𝑝𝛼2 ( 1−𝑝 ) simplifies 𝜃𝑝 = 𝑝 2 , 𝐵 𝜃𝑝 = ( ︀ 1 + 𝜌−1 ) ︀ ( 1 − 𝑝 ) ̃︀𝐿2 𝑝− 𝜌 ( 1 − 𝑝 ) = 2 ( 1 − 𝑝 ) ̃︀𝐿2 𝑝 ( ︂ 2 𝑝 − 1 ) ︂ ≤ 4 ̃︀𝐿2 𝑝2 . Case IV : let 0 < 𝛼 < 1 , and 0 < 𝑝 < 1.Then the choice of constants 𝜃 = 1 − ( 1 − 𝛼 ) ( 1 + 𝑠 ) , 𝛽 = ( 1 − 𝛼 ) ( ︀ 1 + 1𝑠 ) ︀ , 𝜌 = 𝑝𝛼4 ( 1−𝑝 ) , 𝑠 = 𝛼 4 ( 1−𝛼 ) yields 𝑝𝜌 + 𝜃𝑝− 𝜌 = 𝑝 ( 𝜌 + 1 − ( 1 − 𝛼 ) ( 1 + 𝑠 ) ) − 𝜌 = 𝑝𝛼− 𝑝 ( 1 − 𝛼 ) 𝑠− ( 1 − 𝑝 ) 𝜌 = 1 2 𝑝𝛼 . ( 53 ) Also 1 + 1 𝑠 = 4 − 3𝛼 𝛼 ≤ 4 𝛼 , 1 + 1 𝜌 = 4 ( 1 − 𝑝 ) + 𝛼𝑝 𝑝𝛼 = 4 − 𝑝 ( 4 − 𝛼 ) 𝑝𝛼 ≤ 4 𝑝𝛼 . Thus 𝐵 𝜃𝑝 = 𝑝𝛽 + ( 1 − 𝑝 ) ( ︁ 1 + 1𝜌 ) ︁ 𝑝 ( 𝜌 + 𝜃 ) − 𝜌 ̃︀𝐿2 = 𝑝 ( 1 − 𝛼 ) ( ︀1 + 1𝑠 ) ︀+ ( 1 − 𝑝 ) ( ︁ 1 + 1𝜌 ) ︁ 1 2𝑝𝛼 ̃︀𝐿2 ≤ 𝑝 ( 1 − 𝛼 ) 4𝛼 + ( 1 − 𝑝 ) 4 𝑝𝛼 1 2𝑝𝛼 ̃︀𝐿2 ≤ 4 𝛼 + 4 𝑝𝛼 1 2𝑝𝛼 ̃︀𝐿2 ≤ 8 𝑝𝛼 1 2𝑝𝛼 ̃︀𝐿2 ≤ 16 ̃︀𝐿2 𝑝2𝛼2 . ( 54 ) F.1 CONVERGENCE FOR GENERAL NON-CONVEX FUNCTIONS Theorem 7 . Let Assumption 1 hold , and let the stepsize in Algorithm 4 be set as 0 < 𝛾 ≤ ( ︃ 𝐿 + √︃ 𝐵 𝜃𝑝 ) ︃−1 . ( 55 ) Fix 𝑇 ≥ 1 and let ? ̂ ? 𝑇 be chosen from the iterates 𝑥0 , 𝑥1 , . . . , 𝑥𝑇−1 uniformly at random . Then E [ ︁⃦⃦ ∇𝑓 ( ? ̂ ? 𝑇 ) ⃦⃦2 ] ︁ ≤ 2 ( ︀𝑓 ( 𝑥0 ) − 𝑓 inf ) ︀ 𝛾𝑇 + E [ ︀ 𝐺0 ] ︀ 𝜃𝑝𝑇 ( 56 ) with 𝜃𝑝 = 𝜌𝑝𝑚𝑖𝑛 + 𝜃𝑝𝑚𝑎𝑥 − 𝜌− ( 𝑝𝑚𝑎𝑥 − 𝑝𝑚𝑖𝑛 ) , 𝐵 = 1𝑛 ∑︀𝑛 𝑖=1 ( ︀ 𝛽𝑝𝑖 + ( ︀ 1 + 𝜌−1 ) ︀ ( 1 − 𝑝𝑖 ) ) ︀ 𝐿2𝑖 , 𝑝𝑚𝑎𝑥 = max1≤𝑖≤𝑛 𝑝𝑖 , 𝑝𝑚𝑖𝑛 = min1≤𝑖≤𝑛 𝑝𝑖 , 𝜃 = 1 − ( 1 + 𝑠 ) ( 1 − 𝛼 ) , 𝛽 = ( ︀ 1 + 1𝑠 ) ︀ ( 1 − 𝛼 ) and 𝜌 , 𝑠 > 0 . Proof . By ( 20 ) , we have E [ ︀ 𝛿𝑡+1 ] ︀ ≤ E [ ︀ 𝛿𝑡 ] ︀ − 𝛾 2 E [ ︁⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 ] ︁− ( ︂ 1 2𝛾 − 𝐿 2 ) ︂ E [ ︀ 𝑅𝑡 ] ︀ + 𝛾 2 E [ ︀ 𝐺𝑡 ] ︀ . ( 57 ) Lemma 5 states that E [ ︀ 𝐺𝑡+1 ] ︀ ≤ ( 1 − 𝜃𝑝 ) E [ ︀ 𝐺𝑡 ] ︀ + 𝐵E [ ︀ 𝑅𝑡 ] ︀ ( 58 ) with 𝜃𝑝 = 𝜌𝑝𝑚𝑖𝑛 + 𝜃𝑝𝑚𝑎𝑥 − 𝜌− ( 𝑝𝑚𝑎𝑥 − 𝑝𝑚𝑖𝑛 ) , 𝐵 = 1𝑛 ∑︀𝑛 𝑖=1 ( ︀ 𝛽𝑝𝑖 + ( ︀ 1 + 𝜌−1 ) ︀ ( 1 − 𝑝𝑖 ) ) ︀ 𝐿2𝑖 , 𝑝𝑚𝑎𝑥 = max1≤𝑖≤𝑛 𝑝𝑖 , 𝑝𝑚𝑖𝑛 = min1≤𝑖≤𝑛 𝑝𝑖 , 𝜃 = 1 − ( 1 + 𝑠 ) ( 1 − 𝛼 ) , 𝛽 = ( ︀ 1 + 1𝑠 ) ︀ ( 1 − 𝛼 ) and small enough 𝜌 , 𝑠 > 0 . Adding ( 57 ) with a 𝛾2𝜃2 multiple of ( 58 ) and rearranging terms in the right hand side , we have E [ ︀ 𝛿𝑡+1 ] ︀ + 𝛾 2𝜃𝑝 E [ ︀ 𝐺𝑡+1 ] ︀ ≤ E [ ︀ 𝛿𝑡 ] ︀ + 𝛾 2𝜃𝑝 E [ ︀ 𝐺𝑡 ] ︀ −𝛾 2 E [ ︁⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 ] ︁− ( ︂ 1 2𝛾 − 𝐿 2 − 𝛾𝐵 2𝜃 ) ︂ E [ ︀ 𝑅𝑡 ] ︀ ≤ E [ ︀ 𝛿𝑡 ] ︀ + 𝛾 2𝜃𝑝 E [ ︀ 𝐺𝑡 ] ︀ − 𝛾 2 E [ ︁⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 ] ︁ . The last inequality follows from the bound 𝛾2 𝐵𝜃𝑝 + 𝐿𝛾 ≤ 1 , which holds because of Lemma 15 and our assumption on the stepsize . By summing up inequalities for 𝑡 = 0 , . . . , 𝑇 − 1 , we get 0 ≤ E [ ︁ 𝛿𝑇 + 𝛾 2𝜃 𝐺𝑇 ] ︁ ≤ 𝛿0 + 𝛾 2𝜃 E [ ︀ 𝐺0 ] ︀ − 𝛾 2 𝑇−1∑︁ 𝑡=0 E [ ︁⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 ] ︁ . Multiplying both sides by 2𝛾𝑇 , after rearranging we get 𝑇−1∑︁ 𝑡=0 1 𝑇 E [ ︁⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 ] ︁ ≤ 2𝛿0 𝛾𝑇 + E [ ︀ 𝐺0 ] ︀ 𝜃𝑇 . It remains to notice that the left hand side can be interpreted as E [ ︁⃦⃦ ∇𝑓 ( ? ̂ ? 𝑇 ) ⃦⃦2 ] ︁ , where ? ̂ ? 𝑇 is chosen from 𝑥0 , 𝑥1 , . . . , 𝑥𝑇−1 uniformly at random . Corollary 12 . Let assumptions of Theorem 7 hold , 𝑔0𝑖 = ∇𝑓𝑖 ( 𝑥0 ) , 𝑖 = 1 , . . . , 𝑛 , 𝛾 = ( ︃ 𝐿 + √︃ 𝐵 𝜃𝑝 ) ︃−1 , 𝑝𝑖 = 𝑝 , 𝑖 = 1 , . . . , 𝑛 , where 𝐵 and 𝜃𝑝 are given in Theorem 7 . Then , after 𝑇 iterations/communication rounds of EF21-PP we have E [ ︁⃦⃦ ∇𝑓 ( ? ̂ ? 𝑇 ) ⃦⃦2 ] ︁ ≤ 𝜀2 . It requires 𝑇 = # grad = 𝒪 ( ︃ ̃︀𝐿𝛿0 𝑝𝛼𝜀2 ) ︃ ( 59 ) iterations/communications rounds/gradint computations at each node . Proof . Let 𝑔0𝑖 = ∇𝑓𝑖 ( 𝑥0 ) , 𝑖 = 1 , . . . , 𝑛 , then 𝐺0 = 0 and by Theorem 7 # grad = 𝑇 ( 𝑖 ) ≤ 2𝛿 0 𝛾𝜀2 ( 𝑖𝑖 ) ≤ 2𝛿 0 𝜀2 ( ︃ 𝐿 + ̃︀𝐿√︃𝐵 𝜃𝑝 ) ︃ ( 𝑖𝑖𝑖 ) ≤ 2𝛿 0 𝜀2 ( ︃ 𝐿 + 4̃︀𝐿 𝑝𝛼 ) ︃ ≤ 2𝛿 0 𝜀2 ( ︃ 𝐿 + 4̃︀𝐿 𝑝𝛼 ) ︃ ( 𝑖𝑣 ) ≤ 2𝛿 0 𝜀2 ( ︃ ̃︀𝐿 𝑝𝛼 + 4̃︀𝐿 𝑝𝛼 ) ︃ = 5̃︀𝐿𝛿0 𝑝𝛼𝜀2 , where ( 𝑖 ) is due to the rate ( 56 ) given by Theorem 7 . In two ( 𝑖𝑖 ) we use the largest possible stepsize ( 55 ) , in ( 𝑖𝑖𝑖 ) we utilize Lemma 6 , and ( 𝑖𝑣 ) follows by the inequalities 𝛼 ≤ 1 , 𝑝 ≤ 1 and 𝐿 ≤ ̃︀𝐿 . F.2 CONVERGENCE UNDER POLYAK-ŁOJASIEWICZ CONDITION Theorem 8 . Let Assumptions 1 and 4 hold , and let the stepsize in Algorithm 4 be set as 0 < 𝛾 ≤ min ⎧⎨⎩ ( ︃ 𝐿 + √︃ 2𝐵 𝜃𝑝 ) ︃−1 , 𝜃𝑝 2𝜇 ⎫⎬⎭ . ( 60 ) Let Ψ𝑡 def = 𝑓 ( 𝑥𝑡 ) − 𝑓 ( 𝑥⋆ ) + 𝛾𝜃𝑝𝐺 𝑡 . Then for any 𝑇 ≥ 0 , we have E [ ︀ Ψ𝑇 ] ︀ ≤ ( 1 − 𝛾𝜇 ) 𝑇E [ ︀ Ψ0 ] ︀ ( 61 ) with 𝜃𝑝 = 𝜌𝑝𝑚𝑖𝑛 + 𝜃𝑝𝑚𝑎𝑥 − 𝜌− ( 𝑝𝑚𝑎𝑥 − 𝑝𝑚𝑖𝑛 ) , 𝐵 = 1𝑛 ∑︀𝑛 𝑖=1 ( ︀ 𝛽𝑝𝑖 + ( ︀ 1 + 𝜌−1 ) ︀ ( 1 − 𝑝𝑖 ) ) ︀ 𝐿2𝑖 , 𝑝𝑚𝑎𝑥 = max1≤𝑖≤𝑛 𝑝𝑖 , 𝑝𝑚𝑖𝑛 = min1≤𝑖≤𝑛 𝑝𝑖 , 𝜃 = 1 − ( 1 + 𝑠 ) ( 1 − 𝛼 ) , 𝛽 = ( ︀ 1 + 1𝑠 ) ︀ ( 1 − 𝛼 ) and 𝜌 , 𝑠 > 0 . Proof . Following the same steps as in the proof of Theorem 2 , but using ( 58 ) , and assumption on the stepsize ( 60 ) , we obtain the result . Corollary 13 . Let assumptions of Theorem 8 hold , 𝑔0𝑖 = ∇𝑓𝑖 ( 𝑥0 ) , 𝑖 = 1 , . . . , 𝑛 , 𝛾 = min ⎧⎨⎩ ( ︃ 𝐿 + √︃ 2𝐵 𝜃𝑝 ) ︃−1 , 𝜃𝑝 2𝜇 ⎫⎬⎭ , 𝑝𝑖 = 𝑝 , 𝑖 = 1 , . . . , 𝑛 , where 𝐵 and 𝜃𝑝 are given in Theorem 8 . Then , after 𝑇 iterations/communication rounds of EF21-PP we have E [ ︀ 𝑓 ( 𝑥𝑇 ) − 𝑓 ( 𝑥⋆ ) ] ︀ ≤ 𝜀 . It requires 𝑇 = # grad = 𝒪 ( ︃ ̃︀𝐿 𝑝𝛼𝜇 log ( ︂ 𝛿0 𝜀 ) ︂ ) ︃ ( 62 ) iterations/communications rounds/gradint computations at each node . Proof . The proof is the same as for Corollary 3 . The only difference is that Lemma 6 is needed to upper bound the quantities 1/𝜃𝑝 and 𝐵/𝜃𝑝 , which appear in Theorem 8 . G BIDIRECTIONAL COMPRESSION In the majority of applications , the uplink ( Client → Server ) communication is the bottleneck . However , in some settings the downlink ( Server → Client ) communication can also slowdown training . Tang et al . ( 2020 ) construct a mechanism which allows bidirectional biased compression . Their method builds upon the original EF meachanism and they prove 𝒪 ( ︁ 1 𝑇 2/3 ) ︁ rate for general nonconvex objectives . However , the main defficiency of this approach is that it requires an additional assumption of bounded magnitude of error ( there exists ∆ > 0 such that E [ ︁ ‖𝒞 ( 𝑥 ) − 𝑥‖2 ] ︁ ≤ ∆ for all 𝑥 ) . In this section , we lift this limitation and propose a new method EF21-BC ( Algorithm 5 ) , which enjoys the desirable 𝒪 ( ︀ 1 𝑇 ) ︀ , and does not rely on additional assumptions . Algorithm 5 EF21-BC ( EF21 with bidirectional biased compression ) 1 : Input : starting point 𝑥0 ∈ R𝑑 ; 𝑔0 , 𝑏0 , ̃︀𝑔0𝑖 ∈ R𝑑 for 𝑖 = 1 , . . . , 𝑛 ( known by nodes ) ; ̃︀𝑔0 = 1 𝑛 ∑︀𝑛 𝑖=1 ̃︀𝑔0𝑖 ( known by master ) ; learning rate 𝛾 > 0 2 : for 𝑡 = 0,1 , 2 , . . . , 𝑇 − 1 do 3 : Master updates 𝑥𝑡+1 = 𝑥𝑡 − 𝛾𝑔𝑡 4 : for all nodes 𝑖 = 1 , . . . , 𝑛 in parallel do 5 : Update 𝑥𝑡+1 = 𝑥𝑡 − 𝛾𝑔𝑡 , 𝑔𝑡+1 = 𝑔𝑡 + 𝑏𝑡 , 6 : compress 𝑐𝑡𝑖 = 𝒞𝑤 ( ∇𝑓𝑖 ( 𝑥𝑡+1 ) − ̃︀𝑔𝑡𝑖 ) , send 𝑐𝑡𝑖 to the master , and 7 : update local state ̃︀𝑔𝑡+1𝑖 = ̃︀𝑔𝑡𝑖 + 𝑐𝑡𝑖 8 : end for 9 : Master computes ̃︀𝑔𝑡+1 = 1𝑛 ∑︀𝑛𝑖=1 ̃︀𝑔𝑡+1𝑖 via ̃︀𝑔𝑡+1 = ̃︀𝑔𝑡 + 1𝑛 ∑︀𝑛𝑖=1 𝑐𝑡𝑖 , 10 : compreses 𝑏𝑡+1 = 𝒞𝑀 ( ̃︀𝑔𝑡+1 − 𝑔𝑡 ) , broadcast 𝑏𝑡+1 to workers , 11 : and updates 𝑔𝑡+1 = 𝑔𝑡 + 𝑏𝑡+1 12 : end for Note that 𝒞𝑀 and 𝒞𝑤 stand for contractive compressors of the type 1 of master and workers respectively . In general , different 𝛼𝑀 and 𝛼𝑤 are accepted . Notations for this section : 𝑃 𝑡𝑖 def = ‖̃︀𝑔𝑡𝑖 −∇𝑓𝑖 ( 𝑥𝑡 ) ‖2 , 𝑃 𝑡 def= 1𝑛 ∑︀𝑛𝑖=1 𝑃 𝑡𝑖 . Lemma 7 . Let Assumption 1 hold , 𝒞𝑤 be a contractive compressor , and ̃︀𝑔𝑡+1𝑖 be an EF21 estimator of ∇𝑓𝑖 ( 𝑥𝑡+1 ) , i. e. ̃︀𝑔𝑡+1𝑖 = ̃︀𝑔𝑡𝑖 + 𝒞𝑤 ( ∇𝑓𝑖 ( 𝑥𝑡+1 ) − ̃︀𝑔𝑡𝑖 ) ( 63 ) for arbitrary ̃︀𝑔0𝑖 and all all 𝑖 = 1 , . . . , 𝑛 , 𝑡 ≥ 0 . Then E [ ︀ 𝑃 𝑡+1 ] ︀ ≤ ( 1 − 𝜃𝑤 ) E [ ︀ 𝑃 𝑡 ] ︀ + 𝛽𝑤̃︀𝐿2E [ ︀𝑅𝑡 ] ︀ , ( 64 ) where 𝜃𝑤 def = 1 − ( 1 − 𝛼𝑤 ) ( 1 + 𝑠 ) , 𝛽𝑤 def = ( 1 − 𝛼𝑤 ) ( ︀ 1 + 𝑠−1 ) ︀ for any 𝑠 > 0 . Proof . The proof is the same as for Lemma 1 . Lemma 8 . Let Assumption 1 hold , 𝒞𝑀 , 𝒞𝑤 be contractive compressors . Let ̃︀𝑔𝑡+1𝑖 be an EF21 estimator of ∇𝑓𝑖 ( 𝑥𝑡+1 ) , i. e. ̃︀𝑔𝑡+1𝑖 = ̃︀𝑔𝑡𝑖 + 𝒞𝑤 ( ∇𝑓𝑖 ( 𝑥𝑡+1 ) − ̃︀𝑔𝑡𝑖 ) , ( 65 ) and let 𝑔𝑡+1 be an EF21 estimator of ̃︀𝑔𝑡+1 = 1𝑛 ∑︀𝑛𝑖=1 ̃︀𝑔𝑡+1𝑖 , i. e. 𝑔𝑡+1 = 𝑔𝑡 + 𝒞𝑀 ( ̃︀𝑔𝑡+1 − 𝑔𝑡 ) ( 66 ) for arbitrary 𝑔0 , ̃︀𝑔0𝑖 and all 𝑖 = 1 , . . . , 𝑛 , 𝑡 ≥ 0 . Then E [ ︁⃦⃦ 𝑔𝑡+1 − ̃︀𝑔𝑡+1⃦⃦2 ] ︁ ≤ ( 1 − 𝜃𝑀 ) E [ ︁⃦⃦𝑔𝑡 − ̃︀𝑔𝑡⃦⃦2 ] ︁+ 8𝛽𝑀E [ ︀𝑃 𝑡 ] ︀+ 8𝛽𝑀 ̃︀𝐿2E [ ︀𝑅𝑡 ] ︀ , ( 67 ) where 𝑔𝑡 = 1𝑛 ∑︀𝑛 𝑖=1 𝑔 𝑡 𝑖 , ̃︀𝑔𝑡 = 1𝑛 ∑︀𝑛𝑖=1 ̃︀𝑔𝑡𝑖 , 𝜃𝑀 = 1 − ( 1 − 𝛼𝑀 ) ( 1 + 𝜌 ) , 𝛽𝑀 = ( 1 − 𝛼𝑀 ) ( ︀1 + 𝜌−1 ) ︀ for any 𝜌 > 0 . Proof . Similarly to the proof of Lemma 1 , define 𝑊 𝑡 def= { 𝑔𝑡1 , . . . , 𝑔𝑡𝑛 , 𝑥𝑡 , 𝑥𝑡+1 } and E [ ︁⃦⃦ 𝑔𝑡+1 − ̃︀𝑔𝑡+1⃦⃦2 ] ︁ = E [ ︁E [ ︁⃦⃦𝑔𝑡+1 − ̃︀𝑔𝑡+1⃦⃦2 | 𝑊 𝑡 ] ︁ ] ︁ = E [ ︁ E [ ︁⃦⃦ 𝑔𝑡 + 𝒞𝑀 ( ̃︀𝑔𝑡+1 − 𝑔𝑡 ) − ̃︀𝑔𝑡+1⃦⃦2 | 𝑊 𝑡 ] ︁ ] ︁ ( 8 ) ≤ ( 1 − 𝛼𝑀 ) E [ ︁⃦⃦̃︀𝑔𝑡+1 − 𝑔𝑡⃦⃦2 ] ︁ ( 𝑖 ) ≤ ( 1 − 𝛼𝑀 ) ( 1 + 𝜌 ) E [ ︁⃦⃦̃︀𝑔𝑡 − 𝑔𝑡⃦⃦2 ] ︁ + ( 1 − 𝛼𝑀 ) ( ︀ 1 + 𝜌−1 ) ︀ ⃦⃦̃︀𝑔𝑡+1 − ̃︀𝑔𝑡⃦⃦2 = ( 1 − 𝜃𝑀 ) E [ ︁⃦⃦ 𝑔𝑡 − ̃︀𝑔𝑡⃦⃦2 ] ︁+ 𝛽𝑀 ⃦⃦̃︀𝑔𝑡+1 − ̃︀𝑔𝑡⃦⃦2 , ( 68 ) where ( 𝑖 ) follows by Young ’ s inequality ( 118 ) , and in ( 𝑖𝑖 ) we use the definition of 𝜃𝑀 and 𝛽𝑀 . Further we bound the last term in ( 68 ) . Recall that ̃︀𝑔𝑡+1 = ̃︀𝑔𝑡 + 1 𝑛 𝑛∑︁ 𝑖=1 𝑐𝑡𝑖 . ( 69 ) where 𝑐𝑡𝑖 = 𝒞𝑤 ( ∇𝑓𝑖 ( 𝑥𝑡+1 ) − ̃︀𝑔𝑡𝑖 ) and ̃︀𝑔𝑡 = 1𝑛 ∑︀𝑛𝑖=1 ̃︀𝑔𝑡𝑖 . Then E [ ︁⃦⃦̃︀𝑔𝑡+1 − ̃︀𝑔𝑡⃦⃦2 ] ︁ ( 69 ) = E ⎡⎣⃦⃦⃦⃦⃦̃︀𝑔𝑡 + 1𝑛 𝑛∑︁ 𝑖=1 𝑐𝑡𝑖 − ̃︀𝑔𝑡 ⃦⃦⃦⃦ ⃦ 2 ⎤⎦ = E ⎡⎣⃦⃦⃦⃦⃦ 1𝑛 𝑛∑︁ 𝑖=1 𝑐𝑡𝑖 ⃦⃦⃦⃦ ⃦ 2 ⎤⎦ ( 𝑖 ) ≤ 1 𝑛 𝑛∑︁ 𝑖=1 E [ ︁⃦⃦ 𝑐𝑡𝑖 ⃦⃦2 ] ︁ = 1 𝑛 𝑛∑︁ 𝑖=1 E [ ︁⃦⃦ 𝑐𝑡𝑖 − ( ︀ ∇𝑓𝑖 ( 𝑥𝑡+1 ) − ̃︀𝑔𝑡𝑖 ) ︀+ ( ︀∇𝑓𝑖 ( 𝑥𝑡+1 ) − ̃︀𝑔𝑡𝑖 ) ︀⃦⃦2 ] ︁ ( 118 ) ≤ 2 1 𝑛 𝑛∑︁ 𝑖=1 E [ ︁ E [ ︁⃦⃦ 𝒞𝑤 ( ︀ ∇𝑓𝑖 ( 𝑥𝑡+1 ) − ̃︀𝑔𝑡𝑖 ) ︀− ( ︀∇𝑓𝑖 ( 𝑥𝑡+1 ) − ̃︀𝑔𝑡𝑖 ) ︀⃦⃦2 | 𝑊 𝑡 ] ︁ ] ︁ +2 1 𝑛 𝑛∑︁ 𝑖=1 E [ ︁⃦⃦ ∇𝑓𝑖 ( 𝑥𝑡+1 ) − ̃︀𝑔𝑡𝑖 ⃦⃦2 ] ︁ ( 8 ) ≤ 2 ( 1 − 𝛼𝑤 ) 1 𝑛 𝑛∑︁ 𝑖=1 E [ ︁⃦⃦ ∇𝑓𝑖 ( 𝑥𝑡+1 ) − ̃︀𝑔𝑡𝑖 ⃦⃦2 ] ︁+ 2 1𝑛 𝑛∑︁ 𝑖=1 E [ ︁⃦⃦ ∇𝑓𝑖 ( 𝑥𝑡+1 ) − ̃︀𝑔𝑡𝑖 ⃦⃦2 ] ︁ = 2 ( 2 − 𝛼𝑤 ) 1 𝑛 𝑛∑︁ 𝑖=1 E [ ︁⃦⃦ ∇𝑓𝑖 ( 𝑥𝑡+1 ) − ̃︀𝑔𝑡𝑖 ⃦⃦2 ] ︁ ( 𝑖𝑖 ) < 4 1 𝑛 𝑛∑︁ 𝑖=1 E [ ︁⃦⃦ ∇𝑓𝑖 ( 𝑥𝑡+1 ) − ̃︀𝑔𝑡𝑖 ⃦⃦2 ] ︁ = 4 1 𝑛 𝑛∑︁ 𝑖=1 E [ ︁⃦⃦ ∇𝑓𝑖 ( 𝑥𝑡+1 ) −∇𝑓𝑖 ( 𝑥𝑡 ) − ( ︀̃︀𝑔𝑡𝑖 −∇𝑓𝑖 ( 𝑥𝑡 ) ) ︀⃦⃦2 ] ︁ ( 118 ) ≤ 8 1 𝑛 𝑛∑︁ 𝑖=1 ⃦⃦̃︀𝑔𝑡𝑖 −∇𝑓𝑖 ( 𝑥𝑡 ) ⃦⃦2 + 8 1𝑛 𝑛∑︁ 𝑖=1 E [ ︁⃦⃦ ∇𝑓𝑖 ( 𝑥𝑡+1 ) −∇𝑓𝑖 ( 𝑥𝑡 ) ⃦⃦2 ] ︁ ( 𝑖𝑖𝑖 ) ≤ 8 1 𝑛 𝑛∑︁ 𝑖=1 E [ ︁⃦⃦̃︀𝑔𝑡𝑖 −∇𝑓𝑖 ( 𝑥𝑡 ) ⃦⃦2 ] ︁+ 8̃︀𝐿2E [ ︁⃦⃦𝑥𝑡+1 − 𝑥𝑡⃦⃦2 ] ︁ = 8E [ ︀ 𝑃 𝑡 ] ︀ + 8̃︀𝐿2E [ ︀𝑅𝑡 ] ︀ , ( 70 ) where in ( 𝑖 ) we use ( 119 ) , ( 𝑖𝑖 ) is due to 𝛼𝑤 > 0 , ( 𝑖𝑖𝑖 ) holds by Assumption 1 . In the last step we apply the definition of 𝑃 𝑡 = 1𝑛 ∑︀𝑛 𝑖=1 ‖̃︀𝑔𝑡𝑖 −∇𝑓𝑖 ( 𝑥𝑡 ) ‖2 , and 𝑅𝑡 = ⃦⃦𝑥𝑡+1 − 𝑥𝑡⃦⃦2 Finally , plugging ( 70 ) into ( 68 ) , we conclude the proof . G.1 CONVERGENCE FOR GENERAL NON-CONVEX FUNCTIONS Theorem 9 . Let Assumption 1 hold , and let the stepsize in Algorithm 5 be set as 0 < 𝛾 ≤ ( ︃ 𝐿 + ̃︀𝐿√︃16𝛽𝑀 𝜃𝑀 + 2𝛽𝑤 𝜃𝑤 ( ︂ 1 + 8𝛽𝑀 𝜃𝑀 ) ︂ ) ︃−1 ( 71 ) Fix 𝑇 ≥ 1 and let ? ̂ ? 𝑇 be chosen from the iterates 𝑥0 , 𝑥1 , . . . , 𝑥𝑇−1 uniformly at random . Then E [ ︁⃦⃦ ∇𝑓 ( ? ̂ ? 𝑇 ) ⃦⃦2 ] ︁ ≤ 2E [ ︀Ψ0 ] ︀ 𝛾𝑇 , ( 72 ) where Ψ𝑡 def = 𝑓 ( 𝑥𝑡 ) −𝑓 inf + 𝛾𝜃𝑀 ‖𝑔 𝑡 − ̃︀𝑔𝑡‖2 + 𝛾𝜃𝑤 ( ︁1 + 8𝛽𝑀𝜃𝑀 ) ︁𝑃 𝑡 , ̃︀𝐿 = √︁ 1𝑛 ∑︀𝑛𝑖=1 𝐿2𝑖 , 𝜃𝑤 def= 1− ( 1− 𝛼𝑤 ) ( 1 + 𝑠 ) , 𝛽𝑤 def = ( 1−𝛼𝑤 ) ( ︀ 1 + 𝑠−1 ) ︀ , 𝜃𝑀 def = 1− ( 1−𝛼𝑀 ) ( 1 +𝜌 ) , 𝛽𝑀 def = ( 1−𝛼𝑀 ) ( ︀ 1 + 𝜌−1 ) ︀ for any 𝜌 , 𝑠 > 0 . Proof . We apply Lemma 16 and split the error ‖𝑔𝑡 −∇𝑓 ( 𝑥𝑡 ) ‖2 in two parts 𝑓 ( 𝑥𝑡+1 ) ≤ 𝑓 ( 𝑥𝑡 ) − 𝛾 2 ⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 − ( ︂ 1 2𝛾 − 𝐿 2 ) ︂ 𝑅𝑡 + 𝛾 2 ⃦⃦ 𝑔𝑡 −∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 ≤ 𝑓 ( 𝑥𝑡 ) − 𝛾 2 ⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 − ( ︂ 1 2𝛾 − 𝐿 2 ) ︂ 𝑅𝑡 +𝛾 ⃦⃦ 𝑔𝑡 − ̃︀𝑔𝑡⃦⃦2 + 𝛾 ⃦⃦̃︀𝑔𝑡 −∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 ≤ 𝑓 ( 𝑥𝑡 ) − 𝛾 2 ⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 − ( ︂ 1 2𝛾 − 𝐿 2 ) ︂ 𝑅𝑡 +𝛾 1 𝑛 𝑛∑︁ 𝑖=1 ⃦⃦ 𝑔𝑡 − ̃︀𝑔𝑡⃦⃦2 + 𝛾 1 𝑛 𝑛∑︁ 𝑖=1 ⃦⃦̃︀𝑔𝑡𝑖 −∇𝑓𝑖 ( 𝑥𝑡 ) ⃦⃦2 = 𝑓 ( 𝑥𝑡 ) − 𝛾 2 ⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 − ( ︂ 1 2𝛾 − 𝐿 2 ) ︂ 𝑅𝑡 + 𝛾 ⃦⃦ 𝑔𝑡 − ̃︀𝑔𝑡⃦⃦2 + 𝛾𝑃 𝑡 , ( 73 ) where we used notation 𝑅𝑡 = ‖𝛾𝑔𝑡‖2 = ⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 , 𝑃 𝑡 = 1𝑛 ∑︀𝑛 𝑖=1 ‖̃︀𝑔𝑡𝑖 −∇𝑓𝑖 ( 𝑥𝑡 ) ‖2 and applied ( 118 ) and ( 119 ) . Subtracting 𝑓 inf from both sides of the above inequality , taking expectation and using the notation 𝛿𝑡 = 𝑓 ( 𝑥𝑡+1 ) − 𝑓 inf , we get E [ ︀ 𝛿𝑡+1 ] ︀ ≤ E [ ︀ 𝛿𝑡 ] ︀ − 𝛾 2 E [ ︁⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 ] ︁− ( ︂ 1 2𝛾 − 𝐿 2 ) ︂ E [ ︀ 𝑅𝑡 ] ︀ + 𝛾E [ ︁⃦⃦ 𝑔𝑡 − ̃︀𝑔𝑡⃦⃦2 ] ︁+ 𝛾E [ ︀𝑃 𝑡 ] ︀ . ( 74 ) Further , Lemma 7 and 8 provide the recursive bounds for the last two terms of ( 74 ) E [ ︀ 𝑃 𝑡+1 ] ︀ ≤ ( 1 − 𝜃𝑤 ) E [ ︀ 𝑃 𝑡 ] ︀ + 𝛽𝑤̃︀𝐿2E [ 𝑅𝑡 ] , ( 75 ) E [ ︁⃦⃦ 𝑔𝑡+1 − ̃︀𝑔𝑡+1⃦⃦2 ] ︁ ≤ ( 1 − 𝜃𝑀 ) E [ ︁⃦⃦𝑔𝑡 − ̃︀𝑔𝑡⃦⃦2 ] ︁+ 8𝛽𝑀 ̃︀𝐿2E [ 𝑅𝑡 ] + 8𝛽𝑀E [ ︀𝑃 𝑡 ] ︀ . ( 76 ) Summing up ( 74 ) with a 𝛾𝜃𝑀 multiple of ( 76 ) we obtain E [ ︀ 𝛿𝑡+1 ] ︀ + 𝛾 𝜃𝑀 E [ ︁⃦⃦ 𝑔𝑡+1 − ̃︀𝑔𝑡+1⃦⃦2 ] ︁ ≤ E [ ︀𝛿𝑡 ] ︀− 𝛾 2 E [ ︁⃦⃦ ∇𝑓 ( ︀ 𝑥𝑡 ) ︀⃦⃦2 ] ︁− ( ︂ 1 2𝛾 − 𝐿 2 ) ︂ E [ ︀ 𝑅𝑡 ] ︀ +𝛾E [ ︁⃦⃦ 𝑔𝑡 − ̃︀𝑔𝑡⃦⃦2 ] ︁+ 𝛾E [ ︀𝑃 𝑡 ] ︀ + 𝛾 𝜃𝑀 ( ︁ ( 1 − 𝜃𝑀 ) E [ ︁⃦⃦ 𝑔𝑡 − ̃︀𝑔𝑡⃦⃦2 ] ︁ ) ︁ + 𝛾 𝜃𝑀 ( ︁ 8𝛽𝑀 ̃︀𝐿2E [ ︀𝑅𝑡 ] ︀+ 8𝛽𝑀E [ ︀𝑃 𝑡 ] ︀ ) ︁ ≤ E [ ︀ 𝛿𝑡 ] ︀ + 𝛾 𝜃𝑀 E [ ︁⃦⃦ 𝑔𝑡 − ̃︀𝑔𝑡⃦⃦2 ] ︁− 𝛾 2 E [ ︁⃦⃦ ∇𝑓 ( ︀ 𝑥𝑡 ) ︀⃦⃦2 ] ︁ − ( ︃ 1 2𝛾 − 𝐿 2 − 8𝛾𝛽𝑀 ̃︀𝐿2 𝜃𝑀 ) ︃ E [ ︀ 𝑅𝑡 ] ︀ +𝛾 ( ︂ 1 + 8𝛽𝑀 𝜃𝑀 ) ︂ E [ ︀ 𝑃 𝑡 ] ︀ . Then adding the above inequality with a 𝛾𝜃𝑤 ( ︁ 1 + 8𝛽𝑀𝜃𝑀 ) ︁ multiple of ( 75 ) , we get E [ ︀ Ψ𝑡+1 ] ︀ = E [ ︀ 𝛿𝑡+1 ] ︀ + 𝛾 𝜃𝑀 E [ ︁⃦⃦ 𝑔𝑡+1 − ̃︀𝑔𝑡+1⃦⃦2 ] ︁+ 𝛾 𝜃𝑤 ( ︂ 1 + 8𝛽𝑀 𝜃𝑀 ) ︂ E [ ︀ 𝑃 𝑡+1 ] ︀ ≤ E [ ︀ 𝛿𝑡 ] ︀ + 𝛾 𝜃𝑀 E [ ︁⃦⃦ 𝑔𝑡 − ̃︀𝑔𝑡⃦⃦2 ] ︁− 𝛾 2 E [ ︁⃦⃦ ∇𝑓 ( ︀ 𝑥𝑡 ) ︀⃦⃦2 ] ︁− ( ︃ 1 2𝛾 − 𝐿 2 − 8𝛾𝛽𝑀 ̃︀𝐿2 𝜃𝑀 ) ︃ E [ ︀ 𝑅𝑡 ] ︀ +𝛾 ( ︂ 1 + 8𝛽𝑀 𝜃𝑀 ) ︂ E [ ︀ 𝑃 𝑡 ] ︀ + 𝛾 𝜃𝑤 ( ︂ 1 + 8𝛽𝑀 𝜃𝑀 ) ︂ ( ︁ ( 1 − 𝜃𝑤 ) E [ ︀ 𝑃 𝑡 ] ︀ + 𝛽𝑤̃︀𝐿2E [ ︀𝑅𝑡 ] ︀ ) ︁ ≤ E [ ︀ 𝛿𝑡 ] ︀ + 𝛾 𝜃𝑀 E [ ︁⃦⃦ 𝑔𝑡 − ̃︀𝑔𝑡⃦⃦2 ] ︁+ 𝛾 𝜃𝑤 ( ︂ 1 + 8𝛽𝑀 𝜃𝑀 ) ︂ E [ ︀ 𝑃 𝑡 ] ︀ − 𝛾 2 E [ ︁⃦⃦ ∇𝑓 ( ︀ 𝑥𝑡 ) ︀⃦⃦2 ] ︁ − ( ︃ 1 2𝛾 − 𝐿 2 − 8𝛾𝛽𝑀 ̃︀𝐿2 𝜃𝑀 − 𝛾 𝜃𝑤 ( ︂ 1 + 8𝛽𝑀 𝜃𝑀 ) ︂ 𝛽𝑤̃︀𝐿2 ) ︃E [ ︀𝑅𝑡 ] ︀ = E [ ︀ Ψ𝑡 ] ︀ − 𝛾 2 E [ ︁⃦⃦ ∇𝑓 ( ︀ 𝑥𝑡 ) ︀⃦⃦2 ] ︁ − ( ︃ 1 2𝛾 − 𝐿 2 − 8𝛾𝛽𝑀 ̃︀𝐿2 𝜃𝑀 − 𝛾𝛽𝑤 ̃︀𝐿2 𝜃𝑤 ( ︂ 1 + 8𝛽𝑀 𝜃𝑀 ) ︂ ) ︃ E [ ︀ 𝑅𝑡 ] ︀ . ( 77 ) Thus by Lemma 15 and the stepsize choice 0 < 𝛾 ≤ ( ︃ 𝐿 + ̃︀𝐿√︃16𝛽𝑀 𝜃𝑀 + 2𝛽𝑤 𝜃𝑤 ( ︂ 1 + 8𝛽𝑀 𝜃𝑀 ) ︂ ) ︃−1 ( 78 ) the last term in ( 77 ) is not positive . By summing up inequalities for 𝑡 = 0 , . . . , 𝑇 − 1 , we get 0 ≤ E [ ︀ Ψ𝑇 ] ︀ ≤ E [ ︀ Ψ0 ] ︀ − 𝛾 2 𝑇−1∑︁ 𝑡=0 E [ ︁⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 ] ︁ . Multiplying both sides by 2𝛾𝑇 and rearranging we get 𝑇−1∑︁ 𝑡=0 1 𝑇 E [ ︁⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 ] ︁ ≤ 2E [ ︀Ψ0 ] ︀ 𝛾𝑇 . It remains to notice that the left hand side can be interpreted as E [ ︁⃦⃦ ∇𝑓 ( ? ̂ ? 𝑇 ) ⃦⃦2 ] ︁ , where ? ̂ ? 𝑇 is chosen from 𝑥0 , 𝑥1 , . . . , 𝑥𝑇−1 uniformly at random . Corollary 14 . Let assumption of Theorem 9 hold , 𝑔0 = ∇𝑓 ( 𝑥0 ) , ̃︀𝑔0𝑖 = ∇𝑓𝑖 ( 𝑥0 ) , 𝑖 = 1 , . . . , 𝑛 , 𝛾 = ( ︃ 𝐿 + ̃︀𝐿√︃16𝛽𝑀 𝜃𝑀 + 2𝛽𝑤 𝜃𝑤 ( ︂ 1 + 8𝛽𝑀 𝜃𝑀 ) ︂ ) ︃−1 , Then , after 𝑇 iterations/communication rounds of EF21-BC we have E [ ︁⃦⃦ ∇𝑓 ( ? ̂ ? 𝑇 ) ⃦⃦2 ] ︁ ≤ 𝜀2 . It requires 𝑇 = # grad = 𝒪 ( ︃ ̃︀𝐿𝛿0 𝛼𝑤𝛼𝑀𝜀2 ) ︃ ( 79 ) iterations/communications rounds/gradint computations at each node . Proof . Note that by Lemma 17 and 𝛼𝑀 , 𝛼𝑤 ≤ 1 , we have 16𝛽𝑀 𝜃𝑀 + 2𝛽𝑤 𝜃𝑤 ( ︂ 1 + 8𝛽𝑀 𝜃𝑀 ) ︂ ≤ 16 4 𝛼2𝑀 + 2 4 𝛼2𝑤 ( ︂ 1 + 8 4 𝛼2𝑀 ) ︂ ≤ 64 𝛼2𝑀 + 8 𝛼2𝑤 33 𝛼2𝑀 ≤ 64 + 8 · 33 𝛼2𝑤𝛼 2 𝑀 . It remains to apply the steps similar to those in the proof of Corollary 2 . G.2 CONVERGENCE UNDER POLYAK-ŁOJASIEWICZ CONDITION Theorem 10 . Let Assumptions 1 and 4 hold , and let the stepsize in Algorithm 3 be set as 0 < 𝛾 ≤ min { ︂ 𝛾0 , 𝜃𝑀 2𝜇 , 𝜃𝑤 2𝜇 } ︂ , ( 80 ) where 𝛾0 def = ( ︂ 𝐿 + ̃︀𝐿√︂ 32𝛽𝑀𝜃𝑀 + 4𝛽𝑤̃︀𝐿2𝜃𝑤 ( ︁1 + 16𝛽𝑀𝜃𝑀 ) ︁ ) ︂−1 , ̃︀𝐿 = √︁ 1𝑛 ∑︀𝑛𝑖=1 𝐿2𝑖 , 𝜃𝑤 def= 1− ( 1−𝛼𝑤 ) ( 1+ 𝑠 ) , 𝛽𝑤 def = ( 1 − 𝛼𝑤 ) ( ︀ 1 + 𝑠−1 ) ︀ , 𝜃𝑀 def = 1 − ( 1 − 𝛼𝑀 ) ( 1 + 𝜌 ) , 𝛽𝑀 def = ( 1 − 𝛼𝑀 ) ( ︀ 1 + 𝜌−1 ) ︀ for any 𝜌 , 𝑠 > 0 . Let Ψ𝑡 def = 𝑓 ( 𝑥𝑡 ) − 𝑓 inf + 𝛾𝜃𝑀 ‖𝑔 𝑡 − ̃︀𝑔𝑡‖2 + 𝛾𝜃𝑤 ( ︁1 + 8𝛽𝑀𝜃𝑀 ) ︁𝑃 𝑡 . Then for any 𝑇 ≥ 0 , we have E [ ︀ Ψ𝑇 ] ︀ ≤ ( 1 − 𝛾𝜇 ) 𝑇E [ ︀ Ψ0 ] ︀ . ( 81 ) Proof . Similarly to the proof of Theorem 9 the inequalities ( 74 ) , ( 75 ) , ( 76 ) hold with 𝛿𝑡 = 𝑓 ( 𝑥𝑡 ) − 𝑓 ( 𝑥⋆ ) . It remains to apply the steps similar to those in the proof of Theorem 6 . Corollary 15 . Let assumption of Theorem 10 hold , 𝑔0 = ∇𝑓 ( 𝑥0 ) , ̃︀𝑔0𝑖 = ∇𝑓𝑖 ( 𝑥0 ) , 𝑖 = 1 , . . . , 𝑛 , 𝛾 = min { ︂ 𝛾0 , 𝜃𝑀 2𝜇 , 𝜃𝑤 2𝜇 } ︂ , 𝛾0 = ⎛⎝𝐿 + ̃︀𝐿 √︃ 32𝛽𝑀 𝜃𝑀 + 4𝛽𝑤̃︀𝐿2 𝜃𝑤 ( ︂ 1 + 16𝛽𝑀 𝜃𝑀 ) ︂⎞⎠−1 , Then , after 𝑇 iterations of EF21-PAGE we have E [ ︀ 𝑓 ( 𝑥𝑇 ) − 𝑓 inf ] ︀ ≤ 𝜀 . It requires 𝑇 = # grad = 𝒪 ( ︃ ̃︀𝐿 𝜇𝛼𝑤𝛼𝑀 ln ( ︂ 𝛿0 𝜀 ) ︂ ) ︃ iterations/communications rounds/gradint computations at each node . H HEAVY BALL MOMENTUM Notations for this section : 𝑅𝑡 = ‖𝛾𝑔𝑡‖2 = ( 1 − 𝜂 ) 2 ⃦⃦ 𝑧𝑡+1 − 𝑧𝑡 ⃦⃦2 . In this section , we study the momentum version of EF21 . In particular , we focus on Polyak style momentum ( Polyak , 1964 ; Yang et al. , 2016 ) . Let 𝑔𝑡 be a gradient estimator at iteration 𝑡 , then the update rule of heavy ball ( HB ) is given by 𝑥𝑡+1 = 𝑥𝑡 − 𝛾𝑔𝑡 + 𝜂 ( ︀ 𝑥𝑡 − 𝑥𝑡−1 ) ︀ , where 𝑥−1 = 𝑥0 , 𝜂 ∈ [ 0 , 1 ) is called the momentum parameter , and 𝛾 > 0 is the stepsize . The above update rule can be viewed as a combination of the classical gradient step 𝑦𝑡 = 𝑥𝑡 − 𝛾𝑔𝑡 followed by additional momentum step 𝑥𝑡+1 = 𝑦𝑡 + 𝜂 ( ︀ 𝑥𝑡 − 𝑥𝑡−1 ) ︀ . Here the momentum term is added to accelerate the convergence and make the trajectory look like a smooth descent to the bottom of the ravine , rather than zigzag . Equivalently , the update of HB can be implemented by the following two steps ( Yang et al. , 2016 ) : { ︂ 𝑥𝑡+1 = 𝑥𝑡 − 𝛾𝑣𝑡 𝑣𝑡+1 = 𝜂𝑣𝑡 + 𝑔𝑡+1 . We are now ready to present the distributed variant of heavy ball method enhanced with a contractive compressor 𝒞 , and EF21 mechanism , which we call EF21-HB ( Algorithm 6 ) . We present the complexity results in Theorem 11 and Corollary 16 . Algorithm 6 EF21-HB 1 : Input : starting point 𝑥0 ∈ R𝑑 ; 𝑔0𝑖 ∈ R𝑑 for 𝑖 = 1 , . . . , 𝑛 ( known by nodes ) ; 𝑣0 = 𝑔0 = 1 𝑛 ∑︀𝑛 𝑖=1 𝑔 0 𝑖 ( known by master ) ; learning rate 𝛾 > 0 ; momentum parameter 0 ≤ 𝜂 < 1 2 : for 𝑡 = 0,1 , 2 , . . . , 𝑇 − 1 do 3 : Master computes 𝑥𝑡+1 = 𝑥𝑡 − 𝛾𝑣𝑡 and broadcasts 𝑥𝑡+1 to all nodes 4 : for all nodes 𝑖 = 1 , . . . , 𝑛 in parallel do 5 : Compress 𝑐𝑡𝑖 = 𝒞 ( ∇𝑓𝑖 ( 𝑥𝑡+1 ) − 𝑔𝑡𝑖 ) and send 𝑐𝑡𝑖 to the master 6 : Update local state 𝑔𝑡+1𝑖 = 𝑔 𝑡 𝑖 + 𝑐 𝑡 𝑖 7 : end for 8 : Master computes 𝑔𝑡+1 = 1𝑛 ∑︀𝑛 𝑖=1 𝑔 𝑡+1 𝑖 via 𝑔 𝑡+1 = 𝑔𝑡 + 1𝑛 ∑︀𝑛 𝑖=1 𝑐 𝑡 𝑖 , and 𝑣 𝑡+1 = 𝜂𝑣𝑡 + 𝑔𝑡+1 9 : end for In the analysis of EF21-HB , we assume by default that 𝑣−1 = 0 . Lemma 9 . Let sequences { 𝑥𝑡 } 𝑡≥0 , and { 𝑣𝑡 } 𝑡≥0 be generated by Algorithm 6 and let the sequence { 𝑧𝑡 } 𝑡≥0 be defined as 𝑧𝑡+1 def = 𝑥𝑡+1 − 𝛾𝜂1−𝜂𝑣 𝑡 with 0 ≤ 𝜂 < 1 . Then for all 𝑡 ≥ 0 𝑧𝑡+1 = 𝑧𝑡 − 𝛾 1 − 𝜂 𝑔𝑡 . Proof . 𝑧𝑡+1 ( 𝑖 ) = 𝑥𝑡+1 − 𝛾𝜂 1 − 𝜂 𝑣𝑡 ( 𝑖𝑖 ) = 𝑥𝑡 − 𝛾𝑣𝑡 − 𝛾𝜂 1 − 𝜂 𝑣𝑡 ( 𝑖𝑖𝑖 ) = 𝑧𝑡 + 𝛾𝜂 1 − 𝜂 𝑣𝑡−1 − 𝛾 1 − 𝜂 𝑣𝑡 = 𝑧𝑡 − 𝛾 1 − 𝜂 ( ︀ 𝑣𝑡 − 𝜂𝑣𝑡−1 ) ︀ = 𝑧𝑡 − 𝛾 1 − 𝜂 𝑔𝑡 , where in ( 𝑖 ) and ( 𝑖𝑖𝑖 ) we use the definition of 𝑧𝑡+1 and 𝑧𝑡 , in ( 𝑖𝑖 ) we use the step 𝑥𝑡+1 = 𝑥𝑡 − 𝛾𝑣𝑡 ( line 3 of Algorithm 6 ) . Finally , the last equality follows by the update 𝑣𝑡+1 = 𝜂𝑣𝑡 + 𝑔𝑡+1 ( line 8 of Algorithm 6 ) . Lemma 10 . Let the sequence { 𝑣𝑡 } 𝑡≥0 be defined as 𝑣𝑡+1 = 𝜂𝑣𝑡 + 𝑔𝑡+1 with 0 ≤ 𝜂 < 1 . Then 𝑇−1∑︁ 𝑡=0 ⃦⃦ 𝑣𝑡 ⃦⃦2 ≤ 1 ( 1 − 𝜂 ) 2 𝑇−1∑︁ 𝑡=0 ⃦⃦ 𝑔𝑡 ⃦⃦2 . Proof . Unrolling the given recurrence and noticing that 𝑣−1 = 0 , we have 𝑣𝑡 = ∑︀𝑡 𝑙=0 𝜂 𝑡−𝑙𝑔𝑙 . Define 𝐻 def = ∑︀𝑡 𝑙=0 𝜂 𝑙 ≤ 11−𝜂 . Then by Jensen ’ s inequality 𝑇−1∑︁ 𝑡=0 ⃦⃦ 𝑣𝑡 ⃦⃦2 = 𝐻2 𝑇−1∑︁ 𝑡=0 ⃦⃦⃦⃦ ⃦ 𝑡∑︁ 𝑙=0 𝜂𝑡−𝑙 𝐻 𝑔𝑙 ⃦⃦⃦⃦ ⃦ 2 ≤ 𝐻2 𝑇−1∑︁ 𝑡=0 𝑡∑︁ 𝑙=0 𝜂𝑡−𝑙 𝐻 ⃦⃦ 𝑔𝑙 ⃦⃦2 = 𝐻 𝑇−1∑︁ 𝑡=0 𝑡∑︁ 𝑙=0 𝜂𝑡−𝑙 ⃦⃦ 𝑔𝑙 ⃦⃦2 ≤ 1 1 − 𝜂 𝑇−1∑︁ 𝑡=0 𝑡∑︁ 𝑙=0 𝜂𝑡−𝑙 ⃦⃦ 𝑔𝑙 ⃦⃦2 = 1 1 − 𝜂 𝑇−1∑︁ 𝑙=0 ⃦⃦ 𝑔𝑙 ⃦⃦2 𝑇−1∑︁ 𝑡=𝑙 𝜂𝑡−𝑙 ≤ 1 ( 1 − 𝜂 ) 2 𝑇−1∑︁ 𝑡=0 ⃦⃦ 𝑔𝑡 ⃦⃦2 . Lemma 11 . Let the sequence { 𝑧𝑡 } 𝑡≥0 be defined as 𝑧𝑡+1 def = 𝑥𝑡+1 − 𝛾𝜂1−𝜂𝑣 𝑡 with 0 ≤ 𝜂 < 1 . Then 𝑇−1∑︁ 𝑡=0 E [ ︀ 𝐺𝑡+1 ] ︀ ≤ ( 1 − 𝜃 ) 𝑇−1∑︁ 𝑡=0 E [ ︀ 𝐺𝑡 ] ︀ + 2𝛽̃︀𝐿2 ( 1 + 4𝜂2 ) 𝑇−1∑︁ 𝑡=0 E [ ︁⃦⃦ 𝑧𝑡+1 − 𝑧𝑡 ⃦⃦2 ] ︁ , where 𝜃 = 1 − ( 1 − 𝛼 ) ( 1 + 𝑠 ) , 𝛽 = ( 1 − 𝛼 ) ( ︀ 1 + 𝑠−1 ) ︀ for any 𝑠 > 0 . Proof . Summing up the inequality in Lemma 1 ( for EF21 estimator ) for 𝑡 = 0 , . . . , 𝑇 − 1 , we have 𝑇−1∑︁ 𝑡=0 E [ ︀ 𝐺𝑡+1 ] ︀ ≤ ( 1 − 𝜃 ) 𝑇−1∑︁ 𝑡=0 E [ ︀ 𝐺𝑡 ] ︀ + 𝛽̃︀𝐿2 𝑇−1∑︁ 𝑡=0 E [ ︁⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 ] ︁ . ( 82 ) It remains to bound ∑︀𝑇−1 𝑡=0 E [ ︁⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 ] ︁ . Notice that by definition of { 𝑧𝑡 } 𝑡≥0 , we have 𝑥𝑡+1 − 𝑥𝑡 = 𝑧𝑡+1 − 𝑧𝑡 + 𝛾𝜂 1 − 𝜂 ( ︀ 𝑣𝑡 − 𝑣𝑡−1 ) ︀ . Thus 𝑇−1∑︁ 𝑡=0 E [ ︁⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 ] ︁ ≤ 2 𝑇−1∑︁ 𝑡=0 E [ ︁⃦⃦ 𝑧𝑡+1 − 𝑧𝑡 ⃦⃦2 ] ︁ + 2𝛾2𝜂2 ( 1 − 𝜂 ) 2 𝑇−1∑︁ 𝑡=0 E [ ︁⃦⃦ 𝑣𝑡 − 𝑣𝑡−1 ⃦⃦2 ] ︁ = 2 𝑇−1∑︁ 𝑡=0 E [ ︁⃦⃦ 𝑧𝑡+1 − 𝑧𝑡 ⃦⃦2 ] ︁ + 2𝛾2𝜂2 ( 1 − 𝜂 ) 2 𝑇−1∑︁ 𝑡=0 E [ ︁⃦⃦ 𝑔𝑡 − ( 1 − 𝜂 ) 𝑣𝑡−1 ⃦⃦2 ] ︁ ≤ 2 𝑇−1∑︁ 𝑡=0 E [ ︁⃦⃦ 𝑧𝑡+1 − 𝑧𝑡 ⃦⃦2 ] ︁ + 4𝛾2𝜂2 ( 1 − 𝜂 ) 2 𝑇−1∑︁ 𝑡=0 E [ ︁⃦⃦ 𝑔𝑡 ⃦⃦2 ] ︁ + 4𝛾2𝜂2 ( 1 − 𝜂 ) 2 𝑇−1∑︁ 𝑡=0 ( 1 − 𝜂 ) 2E [ ︀ ‖𝑣𝑡−1‖2 ] ︀ ( 𝑖 ) ≤ 2 𝑇−1∑︁ 𝑡=0 E [ ︁⃦⃦ 𝑧𝑡+1 − 𝑧𝑡 ⃦⃦2 ] ︁ + 4𝛾2𝜂2 ( 1 − 𝜂 ) 2 𝑇−1∑︁ 𝑡=0 E [ ︁⃦⃦ 𝑔𝑡 ⃦⃦2 ] ︁ + 4𝛾2𝜂2 ( 1 − 𝜂 ) 2 𝑇−1∑︁ 𝑡=0 E [ ︁⃦⃦ 𝑔𝑡 ⃦⃦2 ] ︁ = 2 𝑇−1∑︁ 𝑡=0 E [ ︁⃦⃦ 𝑧𝑡+1 − 𝑧𝑡 ⃦⃦2 ] ︁ + 8𝛾2𝜂2 ( 1 − 𝜂 ) 2 𝑇−1∑︁ 𝑡=0 E [ ︁⃦⃦ 𝑔𝑡 ⃦⃦2 ] ︁ ( 𝑖𝑖 ) = 2 𝑇−1∑︁ 𝑡=0 E [ ︁⃦⃦ 𝑧𝑡+1 − 𝑧𝑡 ⃦⃦2 ] ︁ + 8𝜂2 𝑇−1∑︁ 𝑡=0 E [ ︁⃦⃦ 𝑧𝑡+1 − 𝑧𝑡 ⃦⃦2 ] ︁ = 2 ( 1 + 4𝜂2 ) 𝑇−1∑︁ 𝑡=0 E [ ︁⃦⃦ 𝑧𝑡+1 − 𝑧𝑡 ⃦⃦2 ] ︁ , where in ( 𝑖 ) we apply Lemma 10 , and in ( 𝑖𝑖 ) Lemma 9 is utilized . It remains to plug in the above inequality into ( 82 ) Lemma 12 . Let the sequence { 𝑧𝑡 } 𝑡≥0 be generated as in Lemma 9 , i.e. , 𝑧𝑡+1 = 𝑧𝑡 − 𝛾 1−𝜂 𝑔 𝑡 , then for all 𝑡 ≥ 0 ⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 ≤ 2𝐺𝑡 + 2 ( 1 − 𝜂 ) 2 𝛾2 ⃦⃦ 𝑧𝑡+1 − 𝑧𝑡 ⃦⃦2 with 𝐺𝑡 = 1𝑛 ∑︀𝑛 𝑖=1 ‖∇𝑓𝑖 ( 𝑥𝑡 ) − 𝑔𝑡𝑖‖ 2 . Proof . Notice that for 𝛾 > 0 we have ∇𝑓 ( 𝑥𝑡 ) = ∇𝑓 ( 𝑥𝑡 ) − 𝑔𝑡 − 1−𝜂𝛾 ( 𝑧 𝑡+1 − 𝑧𝑡 ) . Then ⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 ≤ 2 ⃦⃦∇𝑓 ( 𝑥𝑡 ) − 𝑔𝑡⃦⃦2 + 2 ( 1 − 𝜂 ) 2 𝛾2 ⃦⃦ 𝑧𝑡+1 − 𝑧𝑡 ⃦⃦2 ≤ 2 𝑛 𝑛∑︁ 𝑖=1 ⃦⃦ ∇𝑓𝑖 ( 𝑥𝑡 ) − 𝑔𝑡𝑖 ⃦⃦2 + 2 ( 1 − 𝜂 ) 2 𝛾2 ⃦⃦ 𝑧𝑡+1 − 𝑧𝑡 ⃦⃦2 , where the inequalities hold due to ( 118 ) with 𝑠 = 1 , and ( 119 ) . H.1 CONVERGENCE FOR GENERAL NON-CONVEX FUNCTIONS Theorem 11 . Let Assumption 1 hold , and let the stepsize in Algorithm 6 be set as 0 < 𝛾 < ( ︃ ( 1 + 𝜂 ) 𝐿 2 ( 1 − 𝜂 ) 2 + ̃︀𝐿 1 − 𝜂 √︂ 2𝛽 𝜃 ( 1 + 4𝜂2 ) ) ︃−1 def = 𝛾0 , ( 83 ) where 0 ≤ 𝜂 < 1 , 𝜃 = 1 − ( 1 − 𝛼 ) ( 1 + 𝑠 ) , 𝛽 = ( 1 − 𝛼 ) ( ︀ 1 + 𝑠−1 ) ︀ , and 𝑠 > 0 . Fix 𝑇 ≥ 1 and let ? ̂ ? 𝑇 be chosen from the iterates 𝑥0 , 𝑥1 , . . . , 𝑥𝑇−1 uniformly at random . Then 𝑇−1∑︁ 𝑡=0 E [ ︁⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 ] ︁ ≤ 3𝛿0 ( 1 − 𝜂 ) 𝑇𝛾 ( ︁ 1 − 𝛾𝛾0 ) ︁ + E [ ︀𝐺0 ] ︀ 𝜃𝑇 ⎛⎝2 + 1 2𝜆1 3 ( 1 − 𝜂 ) 𝛾 ( ︁ 1 − 𝛾𝛾0 ) ︁ ⎞⎠ , ( 84 ) where 𝜆1 def = ̃︀𝐿√︁ 2𝛽𝜃 ( 1 + 4𝜂2 ) . If the stepsize is set to 0 < 𝛾 ≤ 𝛾0/2 , then 𝑇−1∑︁ 𝑡=0 E [ ︁⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 ] ︁ ≤ 6𝛿0 ( 1 − 𝜂 ) 𝛾𝑇 + E [ ︀ 𝐺0 ] ︀ 𝑇𝜃 ⎛⎝2 + 3 ( 1 − 𝜂 ) 𝛾̃︀𝐿√︁ 2𝛽𝜃 ( 1 + 4𝜂2 ) ⎞⎠ . ( 85 ) Proof . Consider the sequence 𝑧𝑡+1 def= 𝑥𝑡+1 − 𝛾𝜂1−𝜂𝑣 𝑡 with 0 ≤ 𝜂 < 1 . Then Lemma 9 states that 𝑧𝑡+1 = 𝑧𝑡 − 𝛾1−𝜂 𝑔 𝑡 . By 𝐿-smoothness of 𝑓 ( · ) 𝑓 ( 𝑧𝑡+1 ) − 𝑓 ( 𝑧𝑡 ) ≤ ⟨∇𝑓 ( 𝑧𝑡 ) , 𝑧𝑡+1 − 𝑧𝑡⟩ + 𝐿 2 ⃦⃦ 𝑧𝑡+1 − 𝑧𝑡 ⃦⃦2 = ⟨∇𝑓 ( 𝑧𝑡 ) − 𝑔𝑡 , 𝑧𝑡+1 − 𝑧𝑡⟩ + ⟨𝑔𝑡 , 𝑧𝑡+1 − 𝑧𝑡⟩ + 𝐿 2 ⃦⃦ 𝑧𝑡+1 − 𝑧𝑡 ⃦⃦2 ( 𝑖 ) = ⟨∇𝑓 ( 𝑧𝑡 ) − 𝑔𝑡 , 𝑧𝑡+1 − 𝑧𝑡⟩ − 1 − 𝜂 𝛾 ⃦⃦ 𝑧𝑡+1 − 𝑧𝑡 ⃦⃦2 + 𝐿 2 ⃦⃦ 𝑧𝑡+1 − 𝑧𝑡 ⃦⃦2 = ⟨∇𝑓 ( 𝑧𝑡 ) − 𝑔𝑡 , 𝑧𝑡+1 − 𝑧𝑡⟩ − ( ︂ 1 − 𝜂 𝛾 − 𝐿 2 ) ︂ ⃦⃦ 𝑧𝑡+1 − 𝑧𝑡 ⃦⃦2 = ⟨∇𝑓 ( 𝑥𝑡 ) − 𝑔𝑡 , 𝑧𝑡+1 − 𝑧𝑡⟩ + ⟨∇𝑓 ( 𝑧𝑡 ) −∇𝑓 ( 𝑥𝑡 ) , 𝑧𝑡+1 − 𝑧𝑡⟩ − ( ︂ 1 − 𝜂 𝛾 − 𝐿 2 ) ︂ ⃦⃦ 𝑧𝑡+1 − 𝑧𝑡 ⃦⃦2 ( 𝑖𝑖 ) ≤ 1 2𝜆1 ⃦⃦ ∇𝑓 ( 𝑥𝑡 ) − 𝑔𝑡 ⃦⃦2 + 𝜆1 2 ⃦⃦ 𝑧𝑡+1 − 𝑧𝑡 ⃦⃦2 + 1 2𝜆2 ⃦⃦ ∇𝑓 ( 𝑧𝑡 ) −∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 + 𝜆2 2 ⃦⃦ 𝑧𝑡+1 − 𝑧𝑡 ⃦⃦2 − ( ︂1 − 𝜂 𝛾 − 𝐿 2 ) ︂ ⃦⃦ 𝑧𝑡+1 − 𝑧𝑡 ⃦⃦2 = 1 2𝜆1 ⃦⃦ ∇𝑓 ( 𝑥𝑡 ) − 𝑔𝑡 ⃦⃦2 + 1 2𝜆2 ⃦⃦ ∇𝑓 ( 𝑧𝑡 ) −∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 − ( ︂ 1 − 𝜂 𝛾 − 𝐿 2 − 𝜆1 2 − 𝜆2 2 ) ︂ ⃦⃦ 𝑧𝑡+1 − 𝑧𝑡 ⃦⃦2 ( 𝑖𝑖𝑖 ) ≤ 1 2𝜆1 ⃦⃦ ∇𝑓 ( 𝑥𝑡 ) − 𝑔𝑡 ⃦⃦2 + 𝐿2 2𝜆2 ⃦⃦ 𝑧𝑡 − 𝑥𝑡 ⃦⃦2 − ( ︂ 1 − 𝜂 𝛾 − 𝐿 2 − 𝜆1 2 − 𝜆2 2 ) ︂ ⃦⃦ 𝑧𝑡+1 − 𝑧𝑡 ⃦⃦2 ( 𝑖𝑣 ) ≤ 1 2𝜆1 ⃦⃦ ∇𝑓 ( 𝑥𝑡 ) − 𝑔𝑡 ⃦⃦2 + 𝛾2𝜂2𝐿2 2𝜆2 ( 1 − 𝜂 ) 2 ⃦⃦ 𝑣𝑡−1 ⃦⃦2 − ( ︂ 1 − 𝜂 𝛾 − 𝐿 2 − 𝜆1 2 − 𝜆2 2 ) ︂ ⃦⃦ 𝑧𝑡+1 − 𝑧𝑡 ⃦⃦2 , where in ( 𝑖 ) Lemma 9 is applied , in ( 𝑖𝑖 ) the inequality ( 115 ) is applied twice for 𝜆1 , 𝜆2 > 0 , ( 𝑖𝑖𝑖 ) holds due to Assumption 1 , and ( 𝑖𝑣 ) holds by definition of 𝑧𝑡 = 𝑥𝑡 − 𝛾𝜂1−𝜂𝑣 𝑡−1 . Summing up the above inequalities for 𝑡 = 0 , . . . , 𝑇 − 1 ( assuming 𝑣−1 = 0 ) , we have 𝑓 ( 𝑧𝑇 ) ≤ 𝑓 ( 𝑧0 ) + 1 2𝜆1 𝑇−1∑︁ 𝑡=0 ⃦⃦ ∇𝑓 ( 𝑥𝑡 ) − 𝑔𝑡 ⃦⃦2 + 𝛾2𝜂2𝐿2 2𝜆2 ( 1 − 𝜂 ) 2 𝑇−1∑︁ 𝑡=0 ⃦⃦ 𝑣𝑡 ⃦⃦2 − ( ︂ 1 − 𝜂 𝛾 − 𝐿 2 − 𝜆1 2 − 𝜆2 2 ) ︂ 𝑇−1∑︁ 𝑡=0 ⃦⃦ 𝑧𝑡+1 − 𝑧𝑡 ⃦⃦2 ( 𝑖 ) ≤ 𝑓 ( 𝑧0 ) + 1 2𝜆1 𝑇−1∑︁ 𝑡=0 ⃦⃦ ∇𝑓 ( 𝑥𝑡 ) − 𝑔𝑡 ⃦⃦2 + 𝛾2𝜂2𝐿2 2𝜆2 ( 1 − 𝜂 ) 4 𝑇−1∑︁ 𝑡=0 ⃦⃦ 𝑔𝑡 ⃦⃦2 − ( ︂ 1 − 𝜂 𝛾 − 𝐿 2 − 𝜆1 2 − 𝜆2 2 ) ︂ 𝑇−1∑︁ 𝑡=0 ⃦⃦ 𝑧𝑡+1 − 𝑧𝑡 ⃦⃦2 ( 𝑖𝑖 ) = 𝑓 ( 𝑧0 ) + 1 2𝜆1 𝑇−1∑︁ 𝑡=0 ⃦⃦ ∇𝑓 ( 𝑥𝑡 ) − 𝑔𝑡 ⃦⃦2 + 𝛾2𝜂2𝐿2 2𝜆2 ( 1 − 𝜂 ) 4 𝑇−1∑︁ 𝑡=0 ( 1 − 𝜂 ) 2 𝛾2 ⃦⃦ 𝑧𝑡+1 − 𝑧𝑡 ⃦⃦2 − ( ︂ 1 − 𝜂 𝛾 − 𝐿 2 − 𝜆1 2 − 𝜆2 2 ) ︂ 𝑇−1∑︁ 𝑡=0 ⃦⃦ 𝑧𝑡+1 − 𝑧𝑡 ⃦⃦2 = 𝑓 ( 𝑧0 ) + 1 2𝜆1 𝑇−1∑︁ 𝑡=0 ⃦⃦ ∇𝑓 ( 𝑥𝑡 ) − 𝑔𝑡 ⃦⃦2 − ( ︂ 1 − 𝜂 𝛾 − 𝐿 2 − 𝜆1 2 − 𝜆2 2 − 𝜂 2𝐿2 2𝜆2 ( 1 − 𝜂 ) 2 ) ︂ 𝑇−1∑︁ 𝑡=0 ⃦⃦ 𝑧𝑡+1 − 𝑧𝑡 ⃦⃦2 ( 𝑖𝑖𝑖 ) ≤ 𝑓 ( 𝑧0 ) + 1 2𝜆1 𝑇−1∑︁ 𝑡=0 𝐺𝑡 − ( ︂ 1 − 𝜂 𝛾 − 𝐿 2 − 𝜆1 2 − 𝜆2 2 − 𝜂 2𝐿2 2𝜆2 ( 1 − 𝜂 ) 2 ) ︂ 𝑇−1∑︁ 𝑡=0 ⃦⃦ 𝑧𝑡+1 − 𝑧𝑡 ⃦⃦2 = 𝑓 ( 𝑧0 ) + 1 2𝜆1 𝑇−1∑︁ 𝑡=0 𝐺𝑡 − ( ︂ 1 − 𝜂 𝛾 − 𝐿 2 − 𝜆1 2 − 𝜂𝐿 ( 1 − 𝜂 ) ) ︂ 𝑇−1∑︁ 𝑡=0 ⃦⃦ 𝑧𝑡+1 − 𝑧𝑡 ⃦⃦2 = 𝑓 ( 𝑧0 ) + 1 2𝜆1 𝑇−1∑︁ 𝑡=0 𝐺𝑡 − ( ︂ 1 − 𝜂 𝛾 − 𝐿 2 − 𝜆1 2 − 𝜂𝐿 ( 1 − 𝜂 ) ) ︂ 1 ( 1 − 𝜂 ) 2 𝑇−1∑︁ 𝑡=0 𝑅𝑡 , where ( 𝑖 ) holds due to Lemma 10 , in ( 𝑖𝑖 ) Lemma 9 is applied , in ( 𝑖𝑖𝑖 ) we apply ‖∇𝑓 ( 𝑥𝑡 ) − 𝑔𝑡‖2 ≤ 𝐺𝑡 . Finally , in the last two steps we choose 𝜆2 = 𝜂𝐿1−𝜂 , and recall the definition 𝑅 𝑡 = ‖𝛾𝑔𝑡‖2 = ( 1 − 𝜂 ) 2 ⃦⃦ 𝑧𝑡+1 − 𝑧𝑡 ⃦⃦2 . Subtracting 𝑓 inf from both sides of the above inequality , taking expectation and using the notation 𝛿𝑡 = 𝑓 ( 𝑧𝑡 ) − 𝑓 inf , we get E [ ︀ 𝛿𝑇 ] ︀ ≤ E [ ︀ 𝛿0 ] ︀ + 1 2𝜆1 𝑇−1∑︁ 𝑡=0 E [ ︀ 𝐺𝑡 ] ︀ − ( ︂ 1 − 𝜂 𝛾 − 𝐿 2 − 𝜆1 2 − 𝜂𝐿 ( 1 − 𝜂 ) ) ︂ 1 ( 1 − 𝜂 ) 2 𝑇−1∑︁ 𝑡=0 E [ ︀ 𝑅𝑡 ] ︀ . ( 86 ) By Lemma 11 , we have 𝑇−1∑︁ 𝑡=0 E [ ︀ 𝐺𝑡+1 ] ︀ ≤ ( 1 − 𝜃 ) 𝑇−1∑︁ 𝑡=0 E [ ︀ 𝐺𝑡 ] ︀ + 2𝛽̃︀𝐿2 ( 1 + 4𝜂2 ) ( 1 − 𝜂 ) 2 𝑇−1∑︁ 𝑡=0 E [ ︀ 𝑅𝑡 ] ︀ . ( 87 ) Next , we are going to add ( 86 ) with a 12𝜃𝜆1 multiple of ( 87 ) . First , let us `` forget '' , for a moment , about all the terms involving 𝑅𝑡 and denote their sum appearing on the right hand side by ℛ , then E [ ︀ 𝛿𝑇 ] ︀ + 1 2𝜃𝜆1 𝑇−1∑︁ 𝑡=0 E [ ︀ 𝐺𝑡+1 ] ︀ ≤ E [ ︀ 𝛿0 ] ︀ + 1 2𝜆1 𝑇−1∑︁ 𝑡=0 E [ ︀ 𝐺𝑡 ] ︀ + ( 1 − 𝜃 ) 1 2𝜆1 𝑇−1∑︁ 𝑡=0 E [ ︀ 𝐺𝑡 ] ︀ + ℛ = E [ ︀ 𝛿0 ] ︀ + 1 2𝜃𝜆1 𝑇−1∑︁ 𝑡=0 E [ ︀ 𝐺𝑡 ] ︀ + ℛ. Canceling out the same terms in both sides of the above inequality , we get E [ ︀ 𝛿𝑇 ] ︀ + 1 2𝜃𝜆1 E [ ︀ 𝐺𝑇 ] ︀ ≤ E [ ︀ 𝛿0 ] ︀ + 1 2𝜃𝜆1 E [ ︀ 𝐺0 ] ︀ + ℛ , where ℛ def= − ( ︁ 1−𝜂 𝛾 − 𝐿 2 ( ︁ 1 + 2𝜂1−𝜂 ) ︁ − 𝜆12 − 𝛽̃︀𝐿2 ( 1+4𝜂2 ) 𝜃𝜆1 ) ︁ 1 ( 1−𝜂 ) 2 ∑︀𝑇−1 𝑡=0 E [ 𝑅𝑡 ] . Now choosing 𝜆1 = ̃︀𝐿√︁ 2𝛽𝜃 ( 1 + 4𝜂2 ) and using the definition of 𝛾0 given by ( 83 ) , i.e. , 𝛾0 def= ( ︂ ( 1+𝜂 ) 𝐿 2 ( 1−𝜂 ) 2 + ̃︀𝐿 1−𝜂 √︁ 2𝛽 𝜃 ( 1 + 4𝜂 2 ) ) ︂−1 , we have ( ︃ 1 − 𝜂 𝛾 − 𝐿 2 ( ︂ 1 + 2𝜂 1 − 𝜂 ) ︂ − 𝜆1 2 − 𝛽 ̃︀𝐿2 ( 1 + 4𝜂2 ) 𝜃𝜆1 ) ︃ 1 ( 1 − 𝜂 ) 2 = ( ︃ 1 − 𝜂 𝛾 − 𝐿 2 ( ︂ 1 + 2𝜂 1 − 𝜂 ) ︂ − ̃︀𝐿√︂2𝛽 𝜃 ( 1 + 4𝜂2 ) ) ︃ 1 ( 1 − 𝜂 ) 2 = ( ︃ 1 𝛾 − 𝐿 2 1 + 𝜂 ( 1 − 𝜂 ) 2 − ̃︀𝐿 1 − 𝜂 √︂ 2𝛽 𝜃 ( 1 + 4𝜂2 ) ) ︃ 1 1 − 𝜂 = ( ︂ 1 𝛾 − 1 𝛾0 ) ︂ 1 1 − 𝜂 . Then 0 ≤ E [ ︀ Φ𝑇 ] ︀ def = E [ ︂ 𝛿𝑇 + 1 2𝜃𝜆1 𝐺𝑇 ] ︂ ≤ E [ ︂ 𝛿0 + 1 2𝜃𝜆1 𝐺0 ] ︂ − ( ︂ 1 𝛾 − 1 𝛾0 ) ︂ 1 1 − 𝜂 𝑇−1∑︁ 𝑡=0 E [ ︀ 𝑅𝑡 ] ︀ = E [ ︀ Φ0 ] ︀ − ( ︂ 1 𝛾 − 1 𝛾0 ) ︂ 1 1 − 𝜂 𝑇−1∑︁ 𝑡=0 E [ ︀ 𝑅𝑡 ] ︀ . After rearranging , we get 1 𝛾2 𝑇−1∑︁ 𝑡=0 E [ ︀ 𝑅𝑡 ] ︀ ≤ E [ ︀ Φ0 ] ︀ ( 1 − 𝜂 ) 𝛾 ( ︁ 1 − 𝛾𝛾0 ) ︁ . Summing the result of Lemma 12 over 𝑡 = 0 , . . . , 𝑇 − 1 and applying expectation , we get 𝑇−1∑︁ 𝑡=0 E [ ︁⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 ] ︁ ≤ 2 𝑇−1∑︁ 𝑡=0 E [ ︀ 𝐺𝑡 ] ︀ + 2 𝛾2 𝑇−1∑︁ 𝑡=0 E [ ︀ 𝑅𝑡 ] ︀ . Due to Lemma 11 , the conditions of Lemma 18 hold with 𝐶 def= 2𝛽̃︀𝐿2 1+4𝜂2 ( 1−𝜂 ) 2 , 𝑠𝑡 = E [ 𝐺𝑡 ] , 𝑟𝑡 = E [ 𝑅𝑡 ] , thus 𝑇−1∑︁ 𝑡=0 E [ ︀ 𝐺𝑡 ] ︀ ≤ E [ ︀ 𝐺0 ] ︀ 𝜃 + 𝐶 𝜃 𝑇−1∑︁ 𝑡=0 E [ ︀ 𝑅𝑡 ] ︀ . Combining the above inequalities , we can continue with 𝑇−1∑︁ 𝑡=0 E [ ︁⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 ] ︁ ≤ 2 𝑇−1∑︁ 𝑡=0 E [ ︀ 𝐺𝑡 ] ︀ + 2 𝛾2 𝑇−1∑︁ 𝑡=0 E [ ︀ 𝑅𝑡 ] ︀ ≤ 2E [ ︀ 𝐺0 ] ︀ 𝜃 + ( ︂ 2 + 𝛾2𝐶 𝜃 ) ︂ 1 𝛾2 𝑇−1∑︁ 𝑡=0 E [ ︀ 𝑅𝑡 ] ︀ ≤ 2E [ ︀ 𝐺0 ] ︀ 𝜃 + ( ︂ 2 + 𝛾2𝐶 𝜃 ) ︂ E [ ︀ Φ0 ] ︀ ( 1 − 𝜂 ) 𝛾 ( ︁ 1 − 𝛾𝛾0 ) ︁ . Note that for 𝛾 < 𝛾0 = ( ︁ ( 1+𝜂 ) 𝐿 2 ( 1−𝜂 ) 2 + √︁ 𝐶 𝜃 ) ︁−1 , we have 𝛾2𝐶 𝜃 < 𝐶 𝜃 ( ︁ ( 1+𝜂 ) 𝐿 2 ( 1−𝜂 ) 2 + √︁ 𝐶 𝜃 ) ︁2 ≤ 1 . ( 88 ) Thus 𝑇−1∑︁ 𝑡=0 E [ ︁⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 ] ︁ ≤ 2E [ ︀𝐺0 ] ︀ 𝜃 + 3E [ ︀ Φ0 ] ︀ ( 1 − 𝜂 ) 𝛾 ( ︁ 1 − 𝛾𝛾0 ) ︁ = 3𝛿0 ( 1 − 𝜂 ) 𝛾 ( ︁ 1 − 𝛾𝛾0 ) ︁ + E [ ︀𝐺0 ] ︀ 𝜃 ⎛⎝2 + 1 2𝜆1 3 ( 1 − 𝜂 ) 𝛾 ( ︁ 1 − 𝛾𝛾0 ) ︁ ⎞⎠ , where 𝜆1 = ̃︀𝐿√︁ 2𝛽𝜃 ( 1 + 4𝜂2 ) . Corollary 16 . Let assumptions of Theorem 11 hold , 𝑔0𝑖 = ∇𝑓𝑖 ( 𝑥0 ) , 𝑖 = 1 , . . . , 𝑛 , 𝛾 = ( ︃ ( 1 + 𝜂 ) 𝐿 2 ( 1 − 𝜂 ) 2 + ̃︀𝐿 1 − 𝜂 √︂ 2𝛽 𝜃 ( 1 + 4𝜂2 ) ) ︃−1 . Then , after 𝑇 iterations/communication rounds of EF21-HB we have E [ ︁⃦⃦ ∇𝑓 ( ? ̂ ? 𝑇 ) ⃦⃦2 ] ︁ ≤ 𝜀2 . It requires 𝑇 = # grad = 𝒪 ( ︃ ̃︀𝐿𝛿0 𝜀2 ( ︂ 1 𝛼 + 1 1 − 𝜂 ) ︂ ) ︃ ( 89 ) iterations/communications rounds/gradint computations at each node . Proof . Notice that by using 𝐿 ≤ ̃︀𝐿 , 𝜂 < 1 and Lemma 17 , we have ( 1 + 𝜂 ) 𝐿 2 ( 1 − 𝜂 ) 2 + ̃︀𝐿 1 − 𝜂 √︂ 2𝛽 𝜃 ( 1 + 4𝜂2 ) ≤ ̃︀𝐿 ( 1 − 𝜂 ) 2 + ̃︀𝐿 1 − 𝜂 √︂ 10𝛽 𝜃 ≤ ̃︀𝐿 1 − 𝜂 ( ︃ 1 1 − 𝜂 + 2 √ 10 𝛼 ) ︃ . Using the above inequality , ( 85 ) , and ( 83 ) , we get # grad = 𝑇≤6𝛿 0 ( 1 − 𝜂 ) 𝛾𝜀2 ≤6𝛿 0 ( 1 − 𝜂 ) 𝜀2 ̃︀𝐿 1 − 𝜂 ( ︃ 1 1 − 𝜂 + 2 √ 10 𝛼 ) ︃ ≤ 6 ̃︀𝐿𝛿0 𝜀2 ( ︃ 1 1 − 𝜂 + 2 √ 10 𝛼 ) ︃ . I COMPOSITE CASE Now we focus on solving a composite optimization problem min 𝑥∈R𝑑 Φ ( 𝑥 ) def = 1 𝑛 𝑛∑︁ 𝑖=1 𝑓𝑖 ( 𝑥 ) + 𝑟 ( 𝑥 ) , ( 90 ) where each 𝑓𝑖 ( · ) is 𝐿𝑖-smooth ( possibly non-convex ) , 𝑟 ( · ) is convex , and Φinf = inf𝑥∈R𝑑 Φ ( 𝑥 ) > −∞ . This is a standard and important generalization of setting ( 1 ) . Namely , it includes three special cases . • Smooth unconstrained optimization . Set 𝑟 ≡ 0 , then we recover the initially stated problem formulation ( 1 ) . • Smooth optimization over convex set . Let 𝑟 = 𝛿𝑄 ( indicator function of the set 𝑄 ) , where 𝑄 is a nonempty closed convex set . Then ( 90 ) reduces to the problem of minimizing finite a sum of smooth ( possibly non-convex ) functions over a nonempty closed convex set min 𝑥∈𝑄 { ︃ 1 𝑛 𝑛∑︁ 𝑖=1 𝑓𝑖 ( 𝑥 ) } ︃ . • 𝑙1-regularized optimization . Choose 𝑟 ( 𝑥 ) = 𝜆‖𝑥‖1 with 𝜆 > 0 , then ( 90 ) amounts to the 𝑙1-regularized ( also known as LASSO ) problem min 𝑥∈R𝑑 { ︃ 1 𝑛 𝑛∑︁ 𝑖=1 𝑓𝑖 ( 𝑥 ) + 𝜆‖𝑥‖1 } ︃ . For any 𝛾 > 0 , 𝑥 ∈ R𝑑 , define a proximal mapping of function 𝑟 ( · ) ( prox-operator ) as prox𝛾𝑟 ( 𝑥 ) def = arg min 𝑦∈R𝑑 { ︂ 𝑟 ( 𝑦 ) + 1 2𝛾 ‖𝑦 − 𝑥‖2 } ︂ . ( 91 ) Throughout this section , we assume that the master node can efficiently compute prox-operator at every iteration . This is a reasonable assumption , and in many cases ( choices of 𝑟 ( · ) ) appearing in applications , there exists an analytical solution of ( 91 ) , or its computation is cheap compared to the aggregation step . To evaluate convergence in composite case , we define the generalized gradient mapping at a point 𝑥 ∈ R𝑑 with a parameter 𝛾 𝒢𝛾 ( 𝑥 ) def = 1 𝛾 ( ︀ 𝑥− prox𝛾𝑟 ( 𝑥− 𝛾∇𝑓 ( 𝑥 ) ) ) ︀ . ( 92 ) One can verify that the above quantity is a well-defined evaluation metric ( Beck , 2017 ) . Namely , for any 𝑥 * ∈ R𝑑 , it holds that 𝒢𝛾 ( 𝑥 ) = 0 if and only if 𝑥 * is a stationary point of ( 90 ) , and in a special case when 𝑟 ≡ 0 , we have 𝒢𝛾 ( 𝑥 ) = ∇𝑓 ( 𝑥 ) . Notations for this section : in this section we re-define 𝛿𝑡 def= Φ ( 𝑥𝑡 ) − Φinf Lemma 13 ( Gradient mapping bound ) . Let 𝑥𝑡+1 def= prox𝛾𝑟 ( 𝑥𝑡 − 𝛾𝑣𝑡 ) , then E [ ︁⃦⃦ 𝒢𝛾 ( ︀ 𝑥𝑡 ) ︀⃦⃦2 ] ︁ ≤ 2 𝛾2 E [ ︁⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 ] ︁ + 2E [ ︁⃦⃦ 𝑣𝑡 −∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 ] ︁ . ( 93 ) Proof . E [ ︁⃦⃦ 𝒢𝛾 ( ︀ 𝑥𝑡 ) ︀⃦⃦2 ] ︁ = 1 𝛾2 E [ ︁⃦⃦ 𝑥𝑡 − prox𝛾𝑟 ( 𝑥𝑡 − 𝛾∇𝑓 ( 𝑥𝑡 ) ) ⃦⃦2 ] ︁ ≤ 2 𝛾2 E [ ︁⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 ] ︁ + 2 𝛾2 E [ ︁⃦⃦ 𝑥𝑡+1 − prox𝛾𝑟 ( 𝑥𝑡 − 𝛾∇𝑓 ( 𝑥𝑡 ) ) ⃦⃦2 ] ︁ = 2 𝛾2 E [ ︁⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 ] ︁ + 2 𝛾2 E [ ︁⃦⃦ prox𝛾𝑟 ( 𝑥 𝑡 − 𝛾𝑣𝑡 ) − prox𝛾𝑟 ( 𝑥𝑡 − 𝛾∇𝑓 ( 𝑥𝑡 ) ) ⃦⃦2 ] ︁ ≤ 2 𝛾2 E [ ︁⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 ] ︁ + 2 𝛾2 E [ ︁⃦⃦ ( 𝑥𝑡 − 𝛾𝑣𝑡 ) − ( 𝑥𝑡 − 𝛾∇𝑓 ( 𝑥𝑡 ) ) ⃦⃦2 ] ︁ = 2 𝛾2 E [ ︁⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 ] ︁ + 2E [ ︁⃦⃦ 𝑣𝑡 −∇𝑓 ( 𝑥𝑡 ) ) ⃦⃦2 ] ︁ , ( 94 ) where in the last inequality we apply non-expansiveness of prox-operator . Lemma 14 . Let 𝑥𝑡+1 def= prox𝛾𝑟 ( 𝑥𝑡 − 𝛾𝑣𝑡 ) , then for any 𝜆 > 0 , Φ ( ︀ 𝑥𝑡+1 ) ︀ ≤ Φ ( ︀ 𝑥𝑡 ) ︀ + 1 2𝜆 ⃦⃦ 𝑣𝑡 −∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 − ( ︂ 1 𝛾 − 𝐿 2 − 𝜆 2 ) ︂ ⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 . ( 95 ) Proof . Define 𝑟 ( 𝑥 ) def= 𝑟 ( 𝑥 ) + 12𝛾 ‖𝑥− 𝑥 𝑡 + 𝛾𝑣𝑡‖2 , and note that 𝑥𝑡+1 = arg min𝑥∈R𝑑 { 𝑟 ( 𝑥 ) } . Since 𝑟 ( · ) is 1/𝛾 - strongly convex , we have 𝑟 ( 𝑥𝑡 ) ≥ 𝑟 ( 𝑥𝑡+1 ) + 1 2𝛾 ⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 , 𝑟 ( 𝑥𝑡 ) + 1 2𝛾 ⃦⃦ 𝛾𝑣𝑡 ⃦⃦2 ≥ 𝑟 ( 𝑥𝑡+1 ) + 1 2𝛾 ⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 + 𝛾𝑣𝑡 ⃦⃦2 + 1 2𝛾 ⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 . Thus 𝑟 ( 𝑥𝑡+1 ) − 𝑟 ( 𝑥𝑡 ) ≤ − 1 𝛾 ⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 − ⟨𝑣𝑡 , 𝑥𝑡+1 − 𝑥𝑡⟩ . ( 96 ) By 𝐿 smoothness of 𝑓 ( · ) , 𝑓 ( ︀ 𝑥𝑡+1 ) ︀ − 𝑓 ( ︀ 𝑥𝑡 ) ︀ ≤ ⟨︀ ∇𝑓 ( ︀ 𝑥𝑡 ) ︀ , 𝑥𝑡+1 − 𝑥𝑡 ⟩︀ + 𝐿 2 ⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 . ( 97 ) Summing up ( 97 ) with ( 96 ) we obtain Φ ( ︀ 𝑥𝑡+1 ) ︀ − Φ ( ︀ 𝑥𝑡 ) ︀ ≤ ⟨∇𝑓 ( 𝑥𝑡 ) − 𝑣𝑡 , 𝑥𝑡+1 − 𝑥𝑡⟩ − ( ︂ 1 𝛾 − 𝐿 2 ) ︂ ⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 ≤ 1 2𝜆 ⃦⃦ ∇𝑓 ( 𝑥𝑡 ) − 𝑣𝑡 ⃦⃦2 − ( ︂ 1 𝛾 − 𝐿 2 − 𝜆 2 ) ︂ ⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 . We are now ready to present EF21-Prox and provide its convergence guarantees in general non-convex case . I.1 CONVERGENCE FOR GENERAL NON-CONVEX FUNCTIONS Algorithm 7 EF21-Prox 1 : Input : starting point 𝑥0 ∈ R𝑑 ; 𝑔0𝑖 ∈ R𝑑 for 𝑖 = 1 , . . . , 𝑛 ( known by nodes ) ; 𝑔0 = 1𝑛 ∑︀𝑛 𝑖=1 𝑔 0 𝑖 ( known by master ) ; learning rate 𝛾 > 0 2 : for 𝑡 = 0,1 , 2 , . . . , 𝑇 − 1 do 3 : Master computes 𝑥𝑡+1 = prox𝛾𝑟 ( 𝑥 𝑡 − 𝛾𝑔𝑡 ) 4 : for all nodes 𝑖 = 1 , . . . , 𝑛 in parallel do 5 : Compress 𝑐𝑡𝑖 = 𝒞 ( ∇𝑓𝑖 ( 𝑥𝑡+1 ) − 𝑔𝑡𝑖 ) and send 𝑐𝑡𝑖 to the master 6 : Update local state 𝑔𝑡+1𝑖 = 𝑔 𝑡 𝑖 + 𝑐 𝑡 𝑖 7 : end for 8 : Master computes 𝑔𝑡+1 = 1𝑛 ∑︀𝑛 𝑖=1 𝑔 𝑡+1 𝑖 via 𝑔 𝑡+1 = 𝑔𝑡 + 1𝑛 ∑︀𝑛 𝑖=1 𝑐 𝑡 𝑖 9 : end for 10 : Output : ? ̂ ? 𝑇 chosen uniformly from { 𝑥𝑡 } 𝑡∈ [ 𝑇 ] Theorem 12 . Let Assumption 1 hold , 𝑟 ( · ) be convex and Φinf = inf𝑥∈R𝑑 Φ ( 𝑥 ) > −∞ . Set the stepsize in Algorithm 7 as 0 < 𝛾 < ( ︃ 𝐿 2 + ̃︀𝐿√︂𝛽 𝜃 ) ︃−1 def = 𝛾0 , ( 98 ) where ̃︀𝐿 = √︁ 1𝑛 ∑︀𝑛𝑖=1 𝐿2𝑖 , 𝜃 = 1 − ( 1 − 𝛼 ) ( 1 + 𝑠 ) , 𝛽 = ( 1 − 𝛼 ) ( ︀1 + 𝑠−1 ) ︀ for any 𝑠 > 0 . Fix 𝑇 ≥ 1 and let ? ̂ ? 𝑇 be chosen from the iterates 𝑥0 , 𝑥1 , . . . , 𝑥𝑇−1 uniformly at random . Then E [ ︁⃦⃦ 𝒢𝛾 ( ? ̂ ? 𝑇 ) ⃦⃦2 ] ︁ ≤ 4 ( ︀Φ0 − Φinf ) ︀ 𝑇𝛾 ( ︁ 1 − 𝛾𝛾0 ) ︁ + 2E [ ︀𝐺0 ] ︀ 𝜃𝑇 ⎛⎝1 + 1 𝛾 ( ︁ 1 − 𝛾𝛾0 ) ︁ 1̃︀𝐿 √︃ 𝜃 𝛽 ⎞⎠ . ( 99 ) If the stepsize is set to 0 < 𝛾 ≤ 𝛾0/2 , then E [ ︁⃦⃦ 𝒢𝛾 ( ? ̂ ? 𝑇 ) ⃦⃦2 ] ︁ ≤ 8 ( ︀Φ0 − Φinf ) ︀ 𝛾𝑇 + 2E [ ︀ 𝐺0 ] ︀ 𝜃𝑇 ( ︃ 1 + 2 𝛾̃︀𝐿 √︃ 𝜃 𝛽 ) ︃ . ( 100 ) Proof . First , let us apply Lemma 14 with 𝑣𝑡 = 𝑔𝑡 , 𝜆 > 0 Φ ( ︀ 𝑥𝑡+1 ) ︀ ≤ Φ ( ︀ 𝑥𝑡 ) ︀ + 1 2𝜆 ⃦⃦ 𝑔𝑡 −∇𝑓 ( ︀ 𝑥𝑡 ) ︀⃦⃦2 − ( ︂ 1 𝛾 − 𝐿 2 − 𝜆 2 ) ︂ ⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 . ( 101 ) Subtract Φinf from both sides , take expectation , and define 𝛿𝑡 = Φ ( 𝑥𝑡 ) − Φinf , 𝐺𝑡 = 1 𝑛 ∑︀𝑛 𝑖=1 ‖𝑔𝑡𝑖 −∇𝑓𝑖 ( 𝑥𝑡 ) ‖ 2 , 𝑅𝑡 = ⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 , then E [ ︀ 𝛿𝑡+1 ] ︀ ≤ E [ ︀ 𝛿𝑡 ] ︀ − ( ︂ 1 𝛾 − 𝐿 2 − 𝜆 2 ) ︂ E [ ︀ 𝑅𝑡 ] ︀ + 1 2𝜆 E [ ︀ 𝐺𝑡 ] ︀ . ( 102 ) Note that the proof of Lemma 1 does not rely on the update rule for 𝑥𝑡+1 , but only on the way the estimator 𝑔𝑡+1𝑖 is constructed . Therefore , ( 14 ) also holds for the composite case E [ ︀ 𝐺𝑡+1 ] ︀ ≤ ( 1 − 𝜃 ) E [ ︀ 𝐺𝑡 ] ︀ + 𝛽̃︀𝐿2E [ ︀𝑅𝑡 ] ︀ . ( 103 ) Adding ( 102 ) with a 12𝜃𝜆 multiple of ( 103 ) , we obtain E [ ︀ 𝛿𝑡+1 ] ︀ + 1 2𝜃𝜆 E [ ︀ 𝐺𝑡+1 ] ︀ ≤ E [ ︀ 𝛿𝑡 ] ︀ + 1 2𝜆 E [ ︀ 𝐺𝑡 ] ︀ + 1 − 𝜃 2𝜃𝜆 E [ ︀ 𝐺𝑡 ] ︀ − ( ︂ 1 𝛾 − 𝐿 2 − 𝜆 2 ) ︂ E [ ︀ 𝑅𝑡 ] ︀ + 1 2𝜃𝜆 𝛽̃︀𝐿2E [ ︀𝑅𝑡 ] ︀ = E [ ︀ 𝛿𝑡 ] ︀ + 1 2𝜃𝜆 E [ ︀ 𝐺𝑡 ] ︀ − ( ︂ 1 𝛾 − 𝐿 2 − 𝜆 2 − 𝛽 2𝜃𝜆 ̃︀𝐿2 ) ︂E [ ︀𝑅𝑡 ] ︀ . By summing up inequalities for 𝑡 = 0 , . . . , 𝑇 − 1 , we arrive at 0 ≤ E [ ︀ 𝛿𝑇 ] ︀ + 1 2𝜃𝜆 E [ ︀ 𝐺𝑇 ] ︀ ≤ 𝛿0 + 1 2𝜃𝜆 E [ ︀ 𝐺0 ] ︀ − ( ︂ 1 𝛾 − 𝐿 2 − 𝜆 2 − 𝛽 2𝜃𝜆 ̃︀𝐿2 ) ︂ 𝑇−1∑︁ 𝑡=0 E [ ︀ 𝑅𝑡 ] ︀ . Thus 𝑇−1∑︁ 𝑡=0 E [ ︀ 𝑅𝑡 ] ︀ ≤ ( ︂ 𝛿0 + 1 2𝜃𝜆 E [ ︀ 𝐺0 ] ︀ ) ︂ ( ︂ 1 𝛾 − 𝐿 2 − 𝜆 2 − 𝛽 2𝜃𝜆 ̃︀𝐿2 ) ︂−1 = ( ︃ 𝛿0 + 1 2𝜃 √︃ 𝜃 𝛽̃︀𝐿2E [ ︀𝐺0 ] ︀ ) ︃ ( ︃ 1 𝛾 − 𝐿 2 − √︂ 𝛽 𝜃 ̃︀𝐿2 ) ︃−1 = 𝛾2𝐹 0𝐵 . ( 104 ) where in the first equality we choose 𝜆 = √︁ 𝛽 𝜃 ̃︀𝐿2 , and in the second we define 𝐹 0 def= 𝛿0 + 1 2𝜃 √︁ 𝜃 𝛽̃︀𝐿2E [ ︀ 𝐺0 ] ︀ , 𝐵 def= ( ︂ 𝛾 − 𝐿𝛾 2 2 − √︁ 𝛽 𝜃 ̃︀𝐿2𝛾2 ) ︂−1 = ( ︁𝛾 − 𝛾2𝛾0 ) ︁−1 . By Lemma 13 with 𝑣𝑡 = 𝑔𝑡 we have E [ ︁⃦⃦ 𝒢𝛾 ( ︀ ? ̂ ? 𝑇 ) ︀⃦⃦2 ] ︁ = 1 𝑇 𝑇−1∑︁ 𝑡=0 E [ ︁⃦⃦ 𝒢𝛾 ( ︀ 𝑥𝑡 ) ︀⃦⃦2 ] ︁ ≤ 2 𝛾2𝑇 𝑇−1∑︁ 𝑡=0 E [ ︀ 𝑅𝑡 ] ︀ + 2 𝑇 𝑇−1∑︁ 𝑡=0 E [ ︀ 𝐺𝑡 ] ︀ ( 𝑖 ) ≤ 2 𝛾2𝑇 𝑇−1∑︁ 𝑡=0 E [ ︀ 𝑅𝑡 ] ︀ + 2 𝑇 E [ ︀ 𝐺0 ] ︀ 𝜃 + 2 𝑇 𝛽̃︀𝐿2 𝜃 𝑇−1∑︁ 𝑡=0 E [ ︀ 𝑅𝑡 ] ︀ ( 𝑖𝑖 ) ≤ 2𝐹 0𝐵 𝑇 + 2 𝑇 E [ ︀ 𝐺0 ] ︀ 𝜃 + 2 𝑇 𝛽̃︀𝐿2 𝜃 𝛾2𝐹 0𝐵 = 2𝐹 0𝐵 𝑇 ( ︃ 1 + 𝛾2𝛽̃︀𝐿2 𝜃 ) ︃ + 2 𝑇 E [ ︀ 𝐺0 ] ︀ 𝜃 = 2𝐹 0 𝑇𝛾 ( ︁ 1 − 𝛾𝛾0 ) ︁ ( ︃1 + 𝛾2𝛽̃︀𝐿2 𝜃 ) ︃ + 2 𝑇 E [ ︀ 𝐺0 ] ︀ 𝜃 , where in ( 𝑖 ) we apply Lemma 18 with 𝐶 def= 𝛽̃︀𝐿2 , 𝑠𝑡 def= E [ 𝐺𝑡 ] , 𝑟𝑡 def= E [ 𝑅𝑡 ] . ( 𝑖𝑖 ) is due to ( 104 ) . Note that for 𝛾 < ( ︂ 𝐿 2 + √︁ 𝛽 𝜃 ̃︀𝐿 ) ︂−1 , we have 𝛾2𝛽̃︀𝐿2 𝜃 < 𝛽 𝜃 ̃︀𝐿2 ( ︂ 𝐿 2 + √︁ 𝛽 𝜃 ̃︀𝐿 ) ︂2 ≤ 1 . ( 105 ) Thus E [ ︁⃦⃦ 𝒢𝛾 ( ? ̂ ? 𝑇 ) ⃦⃦2 ] ︁ ≤ 4𝐹 0 𝑇𝛾 ( ︁ 1 − 𝛾𝛾0 ) ︁ + 2 𝑇 E [ ︀ 𝐺0 ] ︀ 𝜃 = 4𝛿0 𝑇𝛾 ( ︁ 1 − 𝛾𝛾0 ) ︁ + 2E [ ︀𝐺0 ] ︀ 𝜃𝑇 + 2E [ ︀ 𝐺0 ] ︀ 𝑇𝛾 ( ︁ 1 − 𝛾𝛾0 ) ︁ 1 𝜃 √︃ 𝜃 𝛽̃︀𝐿2 . ( 106 ) Set 𝛾 ≤ 𝛾0/2 , then the bound simplifies to E [ ︁⃦⃦ 𝒢𝛾 ( ? ̂ ? 𝑇 ) ⃦⃦2 ] ︁ ≤ 8𝛿0 𝛾𝑇 + 2E [ ︀ 𝐺0 ] ︀ 𝜃𝑇 ( ︃ 1 + 2 𝛾 √︃ 𝜃 𝛽̃︀𝐿2 ) ︃ . Corollary 17 . Let assumptions of Theorem 12 hold , 𝑔0𝑖 = ∇𝑓𝑖 ( 𝑥0 ) , 𝑖 = 1 , . . . , 𝑛 , 𝛾 = ( ︁ 𝐿 + 2̃︀𝐿√︀𝛽/𝜃 ) ︁−1 . Then , after 𝑇 iterations/communication rounds of EF21-Prox we have E [ ︁⃦⃦ ∇𝑓 ( ? ̂ ? 𝑇 ) ⃦⃦2 ] ︁ ≤ 𝜀2 . It requires # grad = 𝒪 ( ︃ ̃︀𝐿𝛿0 𝛼𝜀2 ) ︃ , where ̃︀𝐿 = √︁ 1𝑛 ∑︀𝑛𝑖=1 𝐿2𝑖 , 𝛿0 = Φ ( 𝑥0 ) − Φ𝑖𝑛𝑓 . Proof . The proof is the same as for Corollary 2 . I.2 CONVERGENCE UNDER POLYAK-ŁOJASIEWICZ CONDITION In order to extend the analysis of Polyak-Łojasiewicz functions to composite optimization , we use the following Assumption 5 from ( Li & Li , 2018 ; Wang et al. , 2018 ) . Assumption 5 ( Polyak-Łojasiewicz ) . There exists 𝜇 > 0 such that ‖𝒢𝛾 ( 𝑥 ) ‖2 ≥ 2𝜇 ( Φ ( 𝑥 ) − Φ ( 𝑥⋆ ) ) for all 𝑥 ∈ R𝑑 , where 𝑥⋆ = arg min𝑥 Φ ( 𝑥 ) . Theorem 13 . Let Assumptions 1 and 5 hold , 𝑟 ( · ) be convex and Φinf = inf𝑥∈R𝑑 Φ ( 𝑥 ) > −∞ . Set the stepsize in Algorithm 7 as 𝛾 ≤ min ⎧⎨⎩ ( ︃ 𝐿 + 2̃︀𝐿√︂2𝛽 𝜃 ) ︃−1 , 𝜃 𝜇 + 𝜃̃︀𝐿√︁ 2𝛽𝜃 ⎫⎬⎭ . ( 107 ) Let Ψ𝑡 def = Φ ( 𝑥𝑡 ) − Φ ( 𝑥⋆ ) + 1𝜃𝜆𝐺 𝑡 with 𝜆 = √︁ 2𝛽 𝜃 ̃︀𝐿 . Then for any 𝑇 ≥ 0 , we have E [ ︀ Ψ𝑇 ] ︀ ≤ ( ︁ 1 − 𝛾𝜇 2 ) ︁𝑇 E [ ︀ Ψ0 ] ︀ , ( 108 ) where ̃︀𝐿 = √︁ 1𝑛 ∑︀𝑛𝑖=1 𝐿2𝑖 , 𝜃 = 1 − ( 1 − 𝛼 ) ( 1 + 𝑠 ) , 𝛽 = ( 1 − 𝛼 ) ( ︀1 + 𝑠−1 ) ︀ for any 𝑠 > 0 . Proof . We start as in the previous proof , but subtract Φ ( 𝑥⋆ ) from both sides of ( 101 ) and define 𝛿𝑡 def = Φ ( 𝑥𝑡 ) − Φ ( 𝑥⋆ ) . Recall that 𝐺𝑡 = 1𝑛 ∑︀𝑛 𝑖=1 ‖𝑔𝑡𝑖 −∇𝑓𝑖 ( 𝑥𝑡 ) ‖ 2 , 𝑅𝑡 = ⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 . Then E [ ︀ 𝛿𝑡+1 ] ︀ ≤ E [ ︀ 𝛿𝑡 ] ︀ − ( ︂ 1 𝛾 − 𝐿 2 − 𝜆 2 ) ︂ E [ ︀ 𝑅𝑡 ] ︀ + 1 2𝜆 E [ ︀ 𝐺𝑡 ] ︀ . ( 109 ) By Lemma 1 , we have E [ ︀ 𝐺𝑡+1 ] ︀ ≤ ( 1 − 𝜃 ) E [ ︀ 𝐺𝑡 ] ︀ + 𝛽̃︀𝐿2E [ ︀𝑅𝑡 ] ︀ . ( 110 ) Then by adding ( 109 ) with a 1𝜃𝜆 multiple of ( 110 ) we obtain E [ ︀ 𝛿𝑡+1 ] ︀ + 1 𝜃𝜆 E [ ︀ 𝐺𝑡+1 ] ︀ ≤ E [ ︀ 𝛿𝑡 ] ︀ + 1 𝜃𝜆 ( ︂ 1 − 𝜃 + 𝜃 2 ) ︂ E [ ︀ 𝐺𝑡 ] ︀ − ( ︂ 1 𝛾 − 𝐿 2 − 𝜆 2 ) ︂ E [ ︀ 𝑅𝑡 ] ︀ + 1 𝜃𝜆 𝛽̃︀𝐿2E [ ︀𝑅𝑡 ] ︀ = E [ ︀ 𝛿𝑡 ] ︀ + 1 𝜃𝜆 ( ︂ 1 − 𝜃 2 ) ︂ E [ ︀ 𝐺𝑡 ] ︀ − ( ︂ 1 𝛾 − 𝐿 2 − 𝜆 2 − 𝛽 𝜃𝜆 ̃︀𝐿2 ) ︂E [ ︀𝑅𝑡 ] ︀ ( 𝑖 ) = E [ ︀ 𝛿𝑡 ] ︀ + 1 𝜃𝜆 ( ︂ 1 − 𝜃 2 ) ︂ E [ ︀ 𝐺𝑡 ] ︀ − ( ︃ 1 𝛾 − 𝐿 2 − √︂ 2𝛽 𝜃 ̃︀𝐿 ) ︃E [ ︀𝑅𝑡 ] ︀ ( 𝑖𝑖 ) ≤ E [ ︀ 𝛿𝑡 ] ︀ + 1 𝜃𝜆 ( ︂ 1 − 𝜃 2 ) ︂ E [ ︀ 𝐺𝑡 ] ︀ − 1 2𝛾 E [ ︀ 𝑅𝑡 ] ︀ , ( 111 ) where in ( 𝑖 ) we choose 𝜆 = √︁ 2𝛽 𝜃 ̃︀𝐿2 , ( 𝑖𝑖 ) is due to the stepsize choice ( the first term in minimum ) . Next , combining Assumption 5 with Lemma 13 , we have 2𝜇𝛿𝑡 = 2𝜇 ( ︀ Φ ( 𝑥𝑡 ) − Φ ( 𝑥⋆ ) ) ︀ ≤ ⃦⃦ 𝒢𝛾 ( 𝑥𝑡 ) ⃦⃦2 ≤ 2 𝛾2 𝑅𝑡 + 2𝐺𝑡 , and −𝑅𝑡 ≤ −𝜇𝛾2𝛿𝑡 + 𝛾2𝐺𝑡 . ( 112 ) Thus ( 111 ) can be further bounded as E [ Ψ ] = E [ ︂ 𝛿𝑡+1 + 1 𝜃𝜆 𝐺𝑡+1 ] ︂ ≤ E [ ︀ 𝛿𝑡 ] ︀ + 1 𝜃𝜆 ( ︂ 1 − 𝜃 2 ) ︂ E [ ︀ 𝐺𝑡 ] ︀ − 1 2𝛾 E [ ︀ 𝑅𝑡 ] ︀ ( 112 ) ≤ E [ ︀ 𝛿𝑡 ] ︀ + 1 𝜃𝜆 ( ︂ 1 − 𝜃 2 ) ︂ E [ ︀ 𝐺𝑡 ] ︀ − 𝛾𝜇 2 E [ ︀ 𝛿𝑡 ] ︀ + 𝛾 2 E [ ︀ 𝐺𝑡 ] ︀ = ( ︁ 1 − 𝛾𝜇 2 ) ︁ E [ ︀ 𝛿𝑡 ] ︀ + 1 𝜃𝜆 ( ︂ 1 − 𝜃 2 + 𝛾𝜃𝜆 2 ) ︂ E [ ︀ 𝐺𝑡 ] ︀ ≤ ( ︁ 1 − 𝛾𝜇 2 ) ︁ E [ ︂ 𝛿𝑡 + 1 𝜃𝜆 𝐺𝑡 ] ︂ , ( 113 ) where the last inequality follows by our assumption on the stepsize ( the second term in minimum ) . It remains to unroll the recurrence . Corollary 18 . Let assumptions of Theorem 13 hold , 𝑔0𝑖 = ∇𝑓𝑖 ( 𝑥0 ) , 𝑖 = 1 , . . . , 𝑛 , 𝛾 = min ⎧⎨⎩ ( ︃ 𝐿 + 2̃︀𝐿√︂2𝛽 𝜃 ) ︃−1 , 𝜃 𝜇 + 𝜃̃︀𝐿√︁ 2𝛽𝜃 ⎫⎬⎭ . Then , after 𝑇 iterations/communication rounds of EF21-Prox we have E [ ︀ 𝑓 ( 𝑥𝑇 ) − 𝑓 ( 𝑥⋆ ) ] ︀ ≤ 𝜀 . It requires 𝑇 = # grad = 𝒪 ( ︃ 𝜇 + ̃︀𝐿 𝛼𝜇 log ( ︂ 𝛿0 𝜀 ) ︂ ) ︃ ( 114 ) iterations/communications rounds/gradint computations at each node , where ̃︀𝐿 = √︁ 1𝑛 ∑︀𝑛𝑖=1 𝐿2𝑖 , 𝛿0 = Φ ( 𝑥0 ) − Φ𝑖𝑛𝑓 . Proof . Note that by Lemma 17 we have 𝜇 𝜃 + ̃︀𝐿√︂2𝛽 𝜃 ≤ 4𝜇 𝛼 + ̃︀𝐿2√2 𝛼 ≤ 4 ( ︁ 𝜇 + ̃︀𝐿 ) ︁ 𝛼 . The remainder of the proof is the same as for Corollary 3 . J USEFUL AUXILIARY RESULTS J.1 BASIC FACTS For all 𝑎 , 𝑏 , 𝑥1 , . . . , 𝑥𝑛 ∈ R𝑑 , 𝑠 > 0 and 𝑝 ∈ ( 0,1 ] the following inequalities hold ⟨𝑎 , 𝑏⟩ ≤ ‖𝑎‖ 2 2𝑠 + 𝑠‖𝑏‖2 2 , ( 115 ) ⟨𝑎− 𝑏 , 𝑎 + 𝑏⟩ = ‖𝑎‖2 − ‖𝑏‖2 , ( 116 ) 1 2 ‖𝑎‖2 − ‖𝑏‖2 ≤ ‖𝑎 + 𝑏‖2 , ( 117 ) ‖𝑎 + 𝑏‖2 ≤ ( 1 + 𝑠 ) ‖𝑎‖2 + ( 1 + 1/𝑠 ) ‖𝑏‖2 , ( 118 ) ⃦⃦⃦⃦ ⃦ 1𝑛 𝑛∑︁ 𝑖=1 𝑥𝑖 ⃦⃦⃦⃦ ⃦ 2 ≤ 1 𝑛 𝑛∑︁ 𝑖=1 ‖𝑥𝑖‖2 , ( 119 ) ( ︁ 1 − 𝑝 2 ) ︁−1 ≤ 1 + 𝑝 , ( 120 ) ( ︁ 1 + 𝑝 2 ) ︁ ( 1 − 𝑝 ) ≤ 1 − 𝑝 2 , ( 121 ) log ( 1 − 𝑝 ) ≤ −𝑝 . ( 122 ) Bias-variance decomposition For a random vector 𝜉 ∈ R𝑑 and any deterministic vector 𝑥 ∈ R𝑑 , the variance of 𝜉 can be decomposed as E [ ︀ ‖𝜉 − E [ 𝜉 ] ‖2 ] ︀ = E [ ︀ ‖𝜉‖2 ] ︀ − ‖E [ 𝜉 ] ‖2 ( 123 ) Tower property of mathematical expectation . For random variables 𝜉 , 𝜂 ∈ R𝑑 we have E [ 𝜉 ] = E [ E [ 𝜉 | 𝜂 ] ] ( 124 ) under assumption that all expectations in the expression above are well-defined . J.2 USEFUL LEMMAS Lemma 15 ( Lemma 5 of ( Richtárik et al. , 2021 ) ) . If 0 ≤ 𝛾 ≤ 1√ 𝑎+𝑏 , then 𝑎𝛾2 + 𝑏𝛾 ≤ 1 . Moreover , the bound is tight up to the factor of 2 since 1√ 𝑎+𝑏 ≤ min { ︁ 1√ 𝑎 , 1𝑏 } ︁ ≤ 2√ 𝑎+𝑏 . Lemma 16 ( Lemma 2 of ( Li et al. , 2021 ) ) . Suppose that function 𝑓 is 𝐿-smooth and let 𝑥𝑡+1 def= 𝑥𝑡 − 𝛾𝑔𝑡 , where 𝑔𝑡 ∈ R𝑑 is any vector , and 𝛾 > 0 any scalar . Then we have 𝑓 ( 𝑥𝑡+1 ) ≤ 𝑓 ( 𝑥𝑡 ) − 𝛾 2 ⃦⃦ ∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 − ( ︂ 1 2𝛾 − 𝐿 2 ) ︂ ⃦⃦ 𝑥𝑡+1 − 𝑥𝑡 ⃦⃦2 + 𝛾 2 ⃦⃦ 𝑔𝑡 −∇𝑓 ( 𝑥𝑡 ) ⃦⃦2 . ( 125 ) Lemma 17 ( Lemma 3 of ( Richtárik et al. , 2021 ) ) . Let 0 < 𝛼 < 1 and for 𝑠 > 0 let 𝜃 ( 𝑠 ) and 𝛽 ( 𝑠 ) be defined as 𝜃 ( 𝑠 ) def = 1 − ( 1 − 𝛼 ) ( 1 + 𝑠 ) , 𝛽 ( 𝑠 ) def= ( 1 − 𝛼 ) ( 1 + 𝑠−1 ) . Then the solution of the optimization problem min 𝑠 { ︂ 𝛽 ( 𝑠 ) 𝜃 ( 𝑠 ) : 0 < 𝑠 < 𝛼 1 − 𝛼 } ︂ ( 126 ) is given by 𝑠 * = 1√ 1−𝛼 − 1 . Furthermore , 𝜃 ( 𝑠 * ) = 1 − √ 1 − 𝛼 , 𝛽 ( 𝑠 * ) = 1−𝛼 1− √ 1−𝛼 and√︃ 𝛽 ( 𝑠 * ) 𝜃 ( 𝑠 * ) = 1√ 1 − 𝛼 − 1 = 1 𝛼 + √ 1 − 𝛼 𝛼 − 1 ≤ 2 𝛼 − 1 . ( 127 ) In the trivial case 𝛼 = 1 , we have 𝛽 ( 𝑠 ) 𝜃 ( 𝑠 ) = 0 for any 𝑠 > 0 , and ( 127 ) is satisfied . Lemma 18 . Let ( arbitrary scalar ) non-negative sequences { 𝑠𝑡 } 𝑡≥0 , and { 𝑟𝑡 } 𝑡≥0 satisfy 𝑇−1∑︁ 𝑡=0 𝑠𝑡+1 ≤ ( 1 − 𝜃 ) 𝑇−1∑︁ 𝑡=0 𝑠𝑡 + 𝐶 𝑇−1∑︁ 𝑡=0 𝑟𝑡 for some parameters 𝜃 ∈ ( 0 , 1 ] , 𝐶 > 0 . Then for all 𝑇 ≥ 0 𝑇−1∑︁ 𝑡=0 𝑠𝑡 ≤ 𝑠 0 𝜃 + 𝐶 𝜃 𝑇−1∑︁ 𝑡=0 𝑟𝑡 . ( 128 ) Proof . We have 𝑇−1∑︁ 𝑡=0 𝑠𝑡 − 𝑠0 ≤ 𝑇−1∑︁ 𝑡=0 𝑠𝑡 + 𝑠𝑇 − 𝑠0 = 𝑇−1∑︁ 𝑡=0 𝑠𝑡+1 ≤ ( 1 − 𝜃 ) 𝑇−1∑︁ 𝑡=0 𝑠𝑡 + 𝐶 𝑇−1∑︁ 𝑡=0 𝑟𝑡 = 𝑇−1∑︁ 𝑡=0 𝑠𝑡 − 𝜃 𝑇−1∑︁ 𝑡=0 𝑠𝑡 + 𝐶 𝑇−1∑︁ 𝑡=0 𝑟𝑡 . Dividing both sides by 𝜃 > 0 and rearranging the terms , we get 𝑇−1∑︁ 𝑡=0 𝑠𝑡 ≤ 𝑠 0 𝜃 + 𝐶 𝜃 𝑇−1∑︁ 𝑡=0 𝑟𝑡 .

Summary: This paper studies the extended versions of a previously proposed algorithm called EF21. The paper includes several variants of EF21 such as stochastic optimization, partial participation, variance reduction, momentum, etc. The theoretical analysis for each extension is given and experiments on real world datasets are conducted.