EF21: A New, Simpler, Theoretically Better, and Practically Faster Error Feedback

1 Introduction . In order to obtain state-of-the-art performance , modern machine learning models rely on elaborate architectures , need to be trained on data sets of enormous sizes , and involve a very large number of parameters . Some of the most successful models are heavily over-parameterized , which means that they involve more parameters than the number of available training data points [ Arora et al. , 2018 ] . Naturally , these circumstances should inform the design of optimization methods that could be most efficient to perform the training . First , the reliance on sophisticated model architectures , as opposed to simple linear models , generally leads to nonconvex optimization problems , which are more challenging than convex problems [ Jain and Kar , 2017 ] . Second , the need for very large training data sizes necessitates the use of distributed * King Abdullah University of Science and Technology , Thuwal , Saudi Arabia . †This paper was written while Ilyas Fatkhullin was an intern at KAUST . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . computing [ Verbraeken et al. , 2019 ] . Due to its enormous size , the data needs to be partitioned across a number of machines able to work in parallel . Typically , for further efficiency gains , each such machine further parallelizes its local computations using one or more hardware accelerators . Third , the very large number of parameters describing these models exerts an extra stress on the communication links used to exchange model updates among the machines . These links are typically slow compared to the speed at which computation takes place , and communication often forms the bottleneck of distributed systems even in less extreme situations than over-parameterized training where the number of parameters , and hence the nominal size of communicated messages , can be truly staggering . For this reason , modern efficient optimization methods typically employ elaborate lossy communication compression techniques to reduce the size of the communicated messages . Due to the above reasons , in this paper we are interested in solving the nonconvex distributed optimization problem min 𝑥∈R𝑑 [ ︃ 𝑓 ( 𝑥 ) def = 1 𝑛 𝑛∑︁ 𝑖=1 𝑓𝑖 ( 𝑥 ) ] ︃ , ( 1 ) where 𝑥 ∈ R𝑑 represents the parameters of a machine learning model we wish to train , 𝑛 is the number of workers/nodes/machines , and 𝑓𝑖 ( 𝑥 ) is the loss of model 𝑥 on the data stored on node 𝑖 . We specifically focus on the development of new and more efficient communication efficient firstorder methods for solving ( 1 ) utilizing biased compression operators , with a special emphasis on clean convergence analysis which removes the strong and often unrealistic assumptions , such as the bounded gradient assumption , which are currently needed to analyze such methods ( see Table 1 ) . The remainder of the paper is organized as follows . In Section 2 we describe the key concepts , results and open problems that form the motivation for our work , and summarize our main contributions . Our main theoretical results are presented in Section 4 . In Section 4.4 we establish a connection between EF and EF21 . Finally , experimental results are described in Section 5 . 2 Background and Motivation . To better motivate our approach and contributions , we first offer a concise walk-through over the key considerations , difficulties , advances and open problems in this area . 2.1 Two families of compression operators . Compression is typically performed via the application of a ( possibly randomized ) mapping 𝒞 : R𝑑 → R𝑑 , where 𝑑 is the dimension of the vector/tensor that needs to be communicated , with the property that it is much easier/quicker to transfer 𝒞 ( 𝑥 ) than it is to transfer the original message 𝑥 . This can be achieved in several ways , for instance by sparsifying the input vector [ Alistarh et al. , 2018 ] , or by quantizing its entries [ Alistarh et al. , 2017 , Horváth et al. , 2019a ] , or via a combination of these and other approaches [ Horváth et al. , 2019a , Beznosikov et al. , 2020 ] . There are two large classes of compression operators 𝒞 often studied in the literature : i ) unbiased compression operators satisfying a variance bound proportional to the square norm of the input vector , and ii ) biased compression operators whose square distortion is contractive with respect to the square norm of the input vector . In particular , we say that a ( possibly randomized ) map 𝒞 : R𝑑 → R𝑑 is an unbiased compression operator , or simply just unbiased compressor , if there exists a constant 𝜔 ≥ 0 such that E [ 𝒞 ( 𝑥 ) ] = 𝑥 , E [ ︀ ‖𝒞 ( 𝑥 ) − 𝑥‖2 ] ︀ ≤ 𝜔‖𝑥‖2 , ∀𝑥 ∈ R𝑑 . ( 2 ) The family of such operators will be denoted by U ( 𝜔 ) . Further , we say that a ( possibly randomized ) map 𝒞 : R𝑑 → R𝑑 is a biased compression operator , or simply just biased compressor , if there exists a constant 0 < 𝛼 ≤ 1 such that E [ ︀ ‖𝒞 ( 𝑥 ) − 𝑥‖2 ] ︀ ≤ ( 1 − 𝛼 ) ‖𝑥‖2 , ∀𝑥 ∈ R𝑑 . ( 3 ) The family of such operators will be denoted by B ( 𝛼 ) . It is well known that , in a certain sense , the latter class contains the former . In particular , it is easy to verify that if 𝒞 ∈ U ( 𝜔 ) , then ( 1 + 𝜔 ) −1𝒞 ∈ B ( 1/ ( 1+𝜔 ) ) . However , the latter class is strictly larger , i.e. , it contains compressors which do not arise via a scaling of an unbiased compressor . A canonical example of this is the Top-𝑘 compressor , which preserves the 𝑘 largest ( in absolute value ) entries of the input , and zeros out the remaining entries , and for which 𝛼 = 𝑘/𝑑 . We refer to [ Beznosikov et al. , 2020 , Safaryan et al. , 2021 , Table 1 ] for more examples of unbiased and biased compressors , and to Xu et al . [ 2020 ] for a systems-oriented survey . When used in an appropriate way , greedy biased compressors , such as Top-𝑘 , are often empirically superior to their unbiased counterparts [ Seide et al. , 2014 ] , such as Rand-𝑘 . Intuitively , such greedy compressors retain more of the “ information ” or “ energy ” contained within the message , and hence introduce less distortion . This is beneficial in practice , at least in the simplistic single node ( i.e. , non-distributed ) setting , albeit even here we do not have convincing theory that would explain this . Indeed , both Top-𝑘 and Rand-𝑘 impart the same distortion in the worst case , which happens when the energy is distributed uniformly across all entries of the input vector , and it is not easy to capture theoretically that this worst case situation will not happen repeatedly throughout the iterations . As a result , there is currently no separation between the worst case complexity of first order methods , such as gradient descent , combined with biased vs related unbiased compressors [ Beznosikov et al. , 2020 ] . If one makes a-priori statistical assumptions on the distribution of the messages/gradients that need to be compressed , such a separation can be made [ Beznosikov et al. , 2020 ] . While insightful , this is not satisfactory . Indeed , the gradients produced by methods such as gradient descent evolve in a non-stationary way , and hence modeling them as samples coming from a fixed distribution raises questions . Further , gradient compression affects the iterates and hence also the gradients that will be produced in all subsequent iterations , which is another phenomenon not captured by the aforementioned approach . 2.2 Error feedback : what it is good for , and what we still do not know The difference between what we know about unbiased and biased compressors is larger still in the distributed setting . In particular , unbiasedness turns out to be a very effective tool facilitating the analysis of distributed first order methods utilizing unbiased compressors , and for this reason , the landscape of methods using such compressors is very rich and relatively well understood . For example , using unbiased compressors we know how to i ) analyze distributed compressed gradient descent [ Khirirat et al. , 2018 , Mishchenko et al. , 2020 ] , ii ) remove the variance introduced by compression to achieve faster convergence [ Mishchenko et al. , 2019 , Horváth et al. , 2019b , Mishchenko et al. , 2020 ] , iii ) perform bidirectional compression at the workers and also at the master [ Horváth et al. , 2019a , Philippenko and Dieuleveut , 2020 , Gorbunov et al. , 2020b ] , iv ) develop a general theory for SGD which , besides more standard methods , also includes variants using unbiased compression of ( stochastic ) gradients [ Gorbunov et al. , 2020a , Khaled et al. , 2020 , Li and Richtárik , 2020 ] , v ) achieve Nesterov acceleration in the strongly convex regime [ Li et al. , 2020 ] , vi ) how to analyze these methods in the nonconvex regime [ Mishchenko et al. , 2019 , Horváth et al. , 2019a , Li and Richtárik , 2020 ] , vii ) achieve acceleration in the nonconvex regime [ Gorbunov et al. , 2021 ] , and even how to viii ) apply unbiased compressors to Hessian matrices to obtain communication-efficient secondorder methods [ Islamov et al. , 2021 ] . The situation with general biased compressors ( i.e. , those that do not arise from unbiased compressors via scaling ) is much more challenging . The key complication comes from the fact that their naive use within first order methods , such as gradient descent , can lead to divergence . We refer the reader to [ Karimireddy et al. , 2019 , Counterexamples 1-3 ] and [ Beznosikov et al. , 2020 , Example 1 ] for illustrative examples . In the latter , the gradient descent “ enhanced ” with the Top-1 compressor leads to exponential divergence when applied to the problem of minimizing the average of three strongly convex quadratics in R3 . However , divergence of gradient descent enhanced with biased compressors such as Top-𝑘 was observed empirically much sooner , and a fix for this problem , known as error feedback ( EF ) , or error compensation ( EC ) , was suggested by Seide et al . [ 2014 ] . This fix remained a heuristic until very recently . The first theoretical breakthroughs focused on the simpler single-node setting [ Stich et al. , 2018 , Alistarh et al. , 2018 ] . The first analysis in the general distributed heterogeneous data3 setting was performed by Beznosikov et al . [ 2020 ] , and was confined to the strongly convex regime . While without compression , one can expect a linear rate , the rate in [ Beznosikov et al. , 2020 ] is linear only in the special case of an over-parameterized regime ( i.e. , the regime in which the loss functions on all nodes share a common minimizer ) with a requirement of full gradient computations on each node . These deficiencies were later fixed by Gorbunov et al . [ 2020b ] , who developed the first linearly convergent methods EC-GD-DIANA and EC-LSVRG-DIANA , and also analyzed the convex case . Further , Qian et al . [ 2020a , b ] showed that error-compensated methods can be accelerated in the sense of Nesterov [ Nesterov , 2004 ] and can be combined with variance reduction techniques . However , these advances were achieved through the use of additional unbiased compressors or extra communication of full vectors , and hence via an increase in communication in each round . In the orthogonal line of works , Sun et al . [ 2020 ] , Magnússon et al . [ 2019 ] designed special types of biased quantization operators . However , these quantization schemes are unable to compress gradients below 𝒪 ( 𝑑 ) 4 , and the suggested methods require hyperparameters which can not be efficiently estimated .

Summary: This paper studies the Error Feedback (EF) technique in distributed optimization. The paper proposes a new version called EF21 and claims this is theoretically and practically better. A new scheme for EF compression named Markov compressor is proposed. The writing quality is great and the paper is easy to follow. However, some of the statements are flawed which substantially compromise the quality of this paper.