CANITA: Faster Rates for Distributed Convex Optimization with Communication Compression

O ( √ ( 1 + √ ω3 n ) L + ω ( 1 ) 1 3 ) , which improves upon the state-of-the-art non- accelerated rate O ( ( 1 + ωn ) L + ω2+ω ω+n 1 ) of DIANA [ 12 ] for distributed general convex problems , where is the target error , L is the smooth parameter of the objective , n is the number of machines/devices , and ω is the compression parameter ( larger ω means more compression can be applied , and no compression implies ω = 0 ) . Our results show that as long as the number of devices n is large ( often true in distributed/federated learning ) , or the compression ω is not very high , CANITA achieves the faster convergence rate O ( √ L ) , i.e. , the number of communication rounds is O ( √ L ) ( vs. O ( L ) achieved by previous works ) . As a result , CANITA enjoys the advantages of both compression ( compressed communication in each round ) and acceleration ( much fewer communication rounds ) . 1 Introduction . With the proliferation of edge devices , such as mobile phones , wearables and smart home appliances , comes an increase in the amount of data rich in potential information which can be mined for the benefit of humankind . One of the approaches of turning the raw data into information is via federated learning [ 15 , 29 ] , where typically a single global supervised model is trained in a massively distributed manner over a network of heterogeneous devices . Training supervised distributed/federated learning models is typically performed by solving an optimization problem of the form min x∈Rd { f ( x ) : = 1 n n∑ i=1 fi ( x ) } , ( 1 ) where n denotes the number of devices/machines/workers/clients , and fi : Rd → R is a loss function associated with the data stored on device i . We will write x∗ : = arg min x∈Rd f ( x ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . If more than one minimizer exist , x∗ denotes an arbitrary but fixed solution . We will rely on the solution concept captured in the following definition : Definition 1 A random vector x̂ ∈ Rd is called an -solution of the distributed problem ( 1 ) if E [ f ( x̂ ) ] − f ( x∗ ) ≤ , where the expectation is with respect to the randomness inherent in the algorithm used to produce x̂ . In distributed and federated learning problems of the form ( 1 ) , communication of messages across the network typically forms the key bottleneck of the training system . In the modern practice of supervised learning in general and deep learning in particular , this is exacerbated by the reliance on massive models described by millions or even billions of parameters . For these reasons , it is very important to devise novel and more efficient training algorithms capable of decreasing the overall communication cost , which can be formalized as the product of the number of communication rounds necessary to train a model of sufficient quality , and the computation and communication cost associated with a typical communication round . 1.1 Methods with compressed communication . One of the most common strategies for improving communication complexity is communication compression [ 37 , 1 , 40 , 8 , 30 , 9 , 26 , 24 ] . This strategy is based on the reduction of the size of communicated messages via the application of a suitably chosen lossy compression mechanism , saving precious time spent in each communication round , and hoping that this will not increase the total number of communication rounds . Several recent theoretical results suggest that by combining an appropriate ( randomized ) compression operator with a suitably designed gradient-type method , one can obtain improvement in the total communication complexity over comparable baselines not performing any compression . For instance , this is the case for distributed compressed gradient descent ( CGD ) [ 1 , 13 , 8 , 24 ] , and distributed CGD methods which employ variance reduction to tame the variance introduced by compression [ 7 , 30 , 9 , 24 , 6 ] . 1.2 Methods with acceleration . The acceleration/momentum of gradient-type methods is widely-studied in standard optimization problems , which aims to achieve faster convergence rates ( fewer communication rounds ) [ 33 , 31 , 32 , 17 , 28 , 2 , 18 , 16 , 23 , 20 ] . Deep learning practitioners typically rely on Adam [ 14 ] , or one of its many variants , which besides other tricks also adopts momentum . In particular , ANITA [ 20 ] obtains the current state-of-the-art convergence results for convex optimization . In this paper , we will adopt the acceleration from ANITA [ 20 ] to the distributed setting with compression . 1.3 Can communication compression and acceleration be combined ? . Encouraged by the recent theoretical success of communication compression , and the widespread success of accelerated methods , in this paper we seek to further enhance CGD methods with acceleration/momentum , with the aim to obtain provable improvements in overall communication complexity . Can distributed gradient-type methods theoretically benefit from the combination of gradient compression and acceleration/momentum ? To the best of our knowledge , no such results exist in the general convex regime , and in this paper we close this gap by designing a method that can provably enjoy the advantages of both compression ( compressed communication in each round ) and acceleration ( much fewer communication rounds ) . While there is abundance of research studying communication compression and acceleration in isolation , there is very limited work on the combination of both approaches . The first successful combination of gradient compression and acceleration/momentum was recently achieved by the ADIANA method of Li et al . [ 26 ] . However , Li et al . [ 26 ] only provide theoretical results for strongly convex problems , and their method is not applicable to ( general ) convex problems . So , one needs to both design a new method to handle the convex case , and perform its analysis . A-priori , it is not clear at all what approach would work . To the best of our knowledge , besides the initial work [ 26 ] , we are only aware of two other works for addressing this question [ 41 , 34 ] . However , both these works still only focus on the simpler and less practically relevant strongly convex setting . Thus , this line of research is still largely unexplored . For instance , the well-known logistic regression problem is convex but not strongly convex . Finally , even if a problem is strongly convex , the modulus of strong convexity is typically not known , or hard to estimate properly . 2 Summary of Contributions . In this paper we propose and analyze an accelerated gradient method with compressed communication , which we call CANITA ( described in Algorithm 1 ) , for solving distributed general convex optimization problems of the form ( 1 ) . In particular , CANITA can loosely be seen as a combination of the accelerated gradient method ANITA of [ 20 ] , and the variance-reduced compressed gradient method DIANA of [ 30 ] . Ours is the first work provably combining the benefits of communication compression and acceleration in the general convex regime . 2.1 First accelerated rate for compressed gradient methods in the convex regime . For general convex problems , CANITA is the first compressed communication gradient method with an accelerated rate . In particular , our CANITA solves the distributed problem ( 1 ) in O ( √ ( 1 + √ ω3 n ) L + ω ( 1 ) 1 3 ) communication rounds , which improves upon the current state-of-the-art result O ( ( 1 + ωn ) L + ω2+n ω+n 1 ) achieved by the DIANA method [ 12 ] . See Table 1 for more comparisons . Let us now illustrate the improvements coming from this new bound on an example with concrete numerical values . Let the compression ratio be 10 % ( the size of compressed message is 0.1 · d , where d is the size of the uncompressed message ) . If random sparsification or quantization is used to achieve this , then ω ≈ 10 ( see Section 3.1 ) . Further , if the number of devices/machines is n = 106 , 1In this strongly convex column , κ : = L µ denotes the condition number , where L is the smooth parameter and µ > 0 is the strong convexity parameter . 2Here QSGD [ 1 ] needs an additional bounded gradient assumption , i.e. , ‖∇fi ( x ) ‖2 ≤ G2 , ∀i ∈ [ n ] , x ∈ Rd . and the target error tolerance is = 10−6 , then the number of communication rounds of our CANITA method isO ( 103 ) , while the number of communication rounds of the previous state-of-the-art method DIANA [ 12 ] is O ( 106 ) , i.e. , O ( √ L ) vs. O ( L ) . This is an improvement of three orders of magnitude . Moreover , the numerical experiments in Section 6 indeed show that the performance of our CANITA is much better than previous non-accelerated compressed methods ( QSGD and DIANA ) , corroborating the theoretical results ( see Table 1 ) and confirming the practical superiority of our accelerated CANITA method . 2.2 Accelerated rate with limited compression for free . For strongly convex problems , Li et al . [ 26 ] showed that if the number of devices/machines n is large , or the compression variance parameter ω is not very high ( ω ≤ n1/3 ) , then their ADIANA method enjoys the benefits of both compression and acceleration ( i.e. , √ κ log 1 of ADIANA vs. κ log 1 of previous works ) . In this paper , we consider the general convex setting and show that the proposed CANITA also enjoys the benefits of both compression and acceleration . Similarly , if ω ≤ n1/3 ( i.e. , many devices , or limited compression variance ) , CANITA achieves the accelerated rate √ L vs. L of previous works . This means that the compression does not hurt the accelerated rate at all . Note that the second term ( 1 ) 1 3 is of a lower order compared with the first term √ L . 2.3 Novel proof technique . The proof behind the analysis of CANITA is significantly different from that of ADIANA [ 26 ] , which critically relies on strong convexity . Moreover , the theoretical rate in the strongly convex case is linear O ( log 1 ) , while it is sublinear O ( 1 ) or O ( √ 1 ) ( accelerated ) in the general convex case . We hope that our novel analysis can provide new insights and shed light on future work . 3 Preliminaries . Let [ n ] denote the set { 1 , 2 , · · · , n } and ‖ · ‖ denote the Euclidean norm for a vector and the spectral norm for a matrix . Let 〈u , v〉 denote the standard Euclidean inner product of two vectors u and v. We use O ( · ) and Ω ( · ) to hide the absolute constants . 3.1 Assumptions about the compression operators . We now introduce the notion of a randomized compression operator which we use to compress the gradients to save on communication . We rely on a standard class of unbiased compressors ( see Definition 2 ) that was used in the context of distributed gradient methods before [ 1 , 13 , 9 , 24 , 26 ] . Definition 2 ( Compression operator ) A randomized map C : Rd 7→ Rd is an ω-compression operator if E [ C ( x ) ] = x , E [ ‖C ( x ) − x‖2 ] ≤ ω‖x‖2 , ∀x ∈ Rd . ( 2 ) In particular , no compression ( C ( x ) ≡ x ) implies ω = 0 . It is well known that the conditions ( 2 ) are satisfied by many practically useful compression operators ( see Table 1 in [ 3 , 36 ] ) . For illustration purposes , we now present a couple canonical examples : sparsification and quantization . Example 1 ( Random sparsification ) . Given x ∈ Rd , the random-k sparsification operator is defined by C ( x ) : = d k · ( ξk x ) , where denotes the Hadamard ( element-wise ) product and ξk ∈ { 0 , 1 } d is a uniformly random binary vector with k nonzero entries ( ‖ξk‖0 = k ) . This random-k sparsification operator C satisfies ( 2 ) with ω = dk − 1 . By setting k = d , this reduces to the identity compressor , whose variance is obviously zero : ω = 0 . Example 2 ( Random quantization ) . Given x ∈ Rd , the ( p , s ) -quantization operator is defined by C ( x ) : = sign ( x ) · ‖x‖p · 1 s · ξs , where p , s ≥ 1 are integers , and ξs ∈ Rd is a random vector with i-th element ξs ( i ) : = { l + 1 , with probability |xi|‖x‖p s− l , l , otherwise . The level l satisfies |xi|‖x‖p ∈ [ l s , l+1 s ] . The probability is chosen so that E [ ξs ( i ) ] = |xi| ‖x‖p s. This ( p , s ) quantization operator C satisfies ( 2 ) with ω = 2+ d 1/p+d1/2 s . In particular , QSGD [ 1 ] used p = 2 ( i.e. , ( 2 , s ) -quantization ) and proved that the expected sparsity of C ( x ) is E [ ‖C ( x ) ‖0 ] = O ( s ( s+ √ d ) ) .

Summary: The paper propose a compressed and accelerated gradient method called CANITA for distributed optimization.  It improves the the state-of-the-art result which is achieved by DIANA in smooth and convex problems. The main contribution is the (near) optimal algorithm and its theoretical analysis.