A2BCD: Asynchronous Acceleration with Optimal Complexity

1 Introduction . In this paper , we propose and prove the convergence of the Asynchronous Accelerated Nonuniform Randomized Block Coordinate Descent algorithm ( A2BCD ) , the first asynchronous Nesterovaccelerated algorithm that achieves optimal complexity . No previous attempts have been able to prove a speedup for asynchronous Nesterov acceleration . We aim to find the minimizer x∗ of the unconstrained minimization problem : min x∈Rd f ( x ) = f ( x ( 1 ) , . . . , x ( n ) ) ( 1.1 ) where f is σ-strongly convex for σ > 0 with L-Lipschitz gradient ∇f = ( ∇1f , . . . , ∇nf ) . x ∈ Rd is composed of coordinate blocks x ( 1 ) , . . . , x ( n ) . The coordinate blocks of the gradient ∇if are assumed Li-Lipschitz with respect to the ith block . That is , ∀x , h ∈ Rd : ‖∇if ( x+ Pih ) −∇if ( x ) ‖ ≤ Li‖h‖ ( 1.2 ) where Pi is the projection onto the ith block of Rd . Let L̄ , 1n ∑n i=1 Li be the average block Lipschitz constant . These conditions on f are assumed throughout this whole paper . Our algorithm can also be applied to non-strongly convex objectives ( σ = 0 ) or non-smooth objectives using the black box reduction techniques proposed in Allen-Zhu & Hazan ( 2016 ) . Hence we consider only ∗Corresponding author : RobertHannah89 @ gmail.com †fei.feng @ math.ucla.edu ‡wotaoyin @ math.ucla.edu the coordinate smooth , strongly-convex case . Our algorithm can also be applied to the convex regularized ERM problem via the standard dual transformation ( see for instance Lin et al . ( 2014 ) ) : f ( x ) = 1 n n∑ i=1 fi ( 〈ai , x〉 ) + λ 2 ‖x‖ 2 ( 1.3 ) Hence A2BCD can be used as an asynchronous Nesterov-accelerated finite-sum algorithm . Coordinate descent methods , in which a chosen coordinate block ik is updated at every iteration , are a popular way to solve equation 1.1 . Randomized block coordinate descent ( RBCD , Nesterov ( 2012 ) ) updates a uniformly randomly chosen coordinate block ik with a gradient-descent-like step : xk+1 = xk − ( 1/Lik ) ∇ikf ( xk ) . The complexity K ( ) of an algorithm is defined as the number of iterations required to decrease the error E ( f ( xk ) −f ( x∗ ) ) to less than ( f ( x0 ) − f ( x∗ ) ) . Randomized coordinate descent has a complexity of K ( ) = O ( n ( L̄/σ ) ln ( 1/ ) ) . Using a series of averaging and extrapolation steps , accelerated RBCD Nesterov ( 2012 ) improves RBCD ’ s iteration complexity K ( ) to O ( n √ L̄/σ ln ( 1/ ) ) , which leads to much faster convergence when L̄σ is large . This rate is optimal when all Li are equal Lan & Zhou ( 2015 ) . Finally , using a special probability distribution for the random block index ik , the non-uniform accelerated coordinate descent method Allen-Zhu et al . ( 2015 ) ( NU_ACDM ) can further decrease the complexity to O ( ∑n i=1 √ Li/σ ln ( 1/ ) ) , which can be up to √ n times faster than accelerated RBCD , since some Li can be significantly smaller than L. NU_ACDM is the current state-of-the-art coordinate descent algorithm for solving equation 1.1 . Our A2BCD algorithm generalizes NU_ACDM to the asynchronous-parallel case . We solve equation 1.1 with a collection of p computing nodes that continually read a shared-access solution vector y into local memory then compute a block gradient ∇if , which is used to update shared solution vectors ( x , y , v ) . Proving convergence in the asynchronous case requires extensive new technical machinery . A traditional synchronous-parallel implementation is organized into rounds of computation : Every computing node must complete an update in order for the next iteration to begin . However , this synchronization process can be extremely costly , since the lateness of a single node can halt the entire system . This becomes increasingly problematic with scale , as differences in node computing speeds , load balancing , random network delays , and bandwidth constraints mean that a synchronous-parallel solver may spend more time waiting than computing a solution . Computing nodes in an asynchronous solver do not wait for others to complete and share their updates before starting the next iteration . They simply continue to update the solution vectors with the most recent information available , without any central coordination . This eliminates costly idle time , meaning that asynchronous algorithms can be much faster than traditional ones , since they have much faster iterations . For instance , random network delays cause asynchronous algorithms to complete iterations Ω ( ln ( p ) ) time faster than synchronous algorithms at scale . This and other factors that influence the speed of iterations are discussed in Hannah & Yin ( 2017a ) . However , since many iterations may occur between the time that a node reads the solution vector , and the time that its computed update is applied , effectively the solution vector is being updated with outdated information . At iteration k , the block gradient ∇ikf is computed at a delayed iterate ŷk defined as1 : ŷk = ( ( yk−j ( k,1 ) ) ( 1 ) , . . . , ( yk−j ( k , n ) ) ( n ) ) ( 1.4 ) 1Every coordinate can be outdated by a different amount without significantly changing the proofs . for delay parameters j ( k , 1 ) , . . . , j ( k , n ) ∈ N. Here j ( k , i ) denotes how many iterations out of date coordinate block i is at iteration k. Different blocks may be out of date by different amounts , which is known as an inconsistent read . We assume2 that j ( k , i ) ≤ τ for some constant τ < ∞ . Asynchronous algorithms were proposed in Chazan & Miranker ( 1969 ) to solve linear systems . General convergence results and theory were developed later in Bertsekas ( 1983 ) ; Bertsekas & Tsitsiklis ( 1997 ) ; Tseng et al . ( 1990 ) ; Luo & Tseng ( 1992 ; 1993 ) ; Tseng ( 1991 ) for partially and totally asynchronous systems , with essentially-cyclic block sequence ik . More recently , there has been renewed interest in asynchronous algorithms with random block coordinate updates . Linear and sublinear convergence results were proven for asynchronous RBCD Liu & Wright ( 2015 ) ; Liu et al . ( 2014 ) ; Avron et al . ( 2014 ) , and similar was proven for asynchronous SGD Recht et al . ( 2011 ) , and variance reduction algorithms Reddi et al . ( 2015 ) ; Leblond et al . ( 2017 ) ; Mania et al . ( 2015 ) ; Huo & Huang ( 2016 ) , and primal-dual algorithms Combettes & Eckstein ( 2018 ) . There is also a rich body of work on asynchronous SGD . In the distributed setting , Zhou et al . ( 2018 ) showed global convergence for stochastic variationally coherent problems even when the delays grow at a polynomial rate . In Lian et al . ( 2018 ) , an asynchronous decentralized SGD was proposed with the same optimal sublinear convergence rate as SGD and linear speedup with respect to the number of workers . In Liu et al . ( 2018 ) , authors obtained an asymptotic rate of convergence for asynchronous momentum SGD on streaming PCA , which provides insight into the tradeoff between asynchrony and momentum . In Dutta et al . ( 2018 ) , authors prove convergence results for asynchronous SGD that highlight the tradeoff between faster iterations and iteration complexity . Further related work is discussed in Section 4 . 1.1 Summary of Contributions . In this paper , we prove that A2BCD attains NU_ACDM ’ s state-of-the-art iteration complexity to highest order for solving equation 1.1 , so long as delays are not too large ( see Section 2 ) . The proof is very different from that of Allen-Zhu et al . ( 2015 ) , and involves significant technical innovations and complexity related to the analysis of asynchronicity . We also prove that A2BCD ( and hence NU_ACDM ) has optimal complexity to within a constant factor over a fairly general class of randomized block coordinate descent algorithms ( see Section 2.1 ) . This extends results in Lan & Zhou ( 2015 ) to asynchronous algorithms with Li not all equal . Since asynchronous algorithms complete faster iterations , and A2BCD has optimal complexity , we expect A2BCD to be faster than all existing coordinate descent algorithms . We confirm with numerical experiments that A2BCD is the current fastest coordinate descent algorithm ( see Section 5 ) . We are only aware of one previous and one contemporaneous attempt at proving convergence results for asynchronous Nesterov-accelerated algorithms . However , the first is not accelerated and relies on extreme assumptions , and the second obtains no speedup . Therefore , we claim that our results are the first-ever analysis of asynchronous Nesterov-accelerated algorithms that attains a speedup . Moreover , our speedup is optimal for delays not too large3 . The work of Meng et al . claims to obtain square-root speedup for an asynchronous accelerated SVRG . In the case where all component functions have the same Lipschitz constant L , the complexity they obtain reduces to ( n+ κ ) ln ( 1/ ) for κ = O ( τn2 ) ( Corollary 4.4 ) . Hence authors do not even obtain accelerated rates . Their convergence condition is τ < 14∆1/8 for sparsity parameter ∆ . Since the dimension d satisfies d ≥ 1∆ , they require d ≥ 2 16τ8 . So τ = 20 requires dimension d > 1015 . 2This condition can be relaxed however by techniques in Hannah & Yin ( 2017b ) ; Sun et al . ( 2017 ) ; Peng et al . ( 2016c ) ; Hannah & Yin ( 2017a ) 3Speedup is defined precisely in Section 2 In a contemporaneous preprint , authors in Fang et al . ( 2018 ) skillfully devised accelerated schemes for asynchronous coordinate descent and SVRG using momentum compensation techniques . Although their complexity results have the improved √ κ dependence on the condition number , they do not prove any speedup . Their complexity is τ times larger than the serial complexity . Since τ is necessarily greater than p , their results imply that adding more computing nodes will increase running time . The authors claim that they can extend their results to linear speedup for asynchronous , accelerated SVRG under sparsity assumptions . And while we think this is quite likely , they have not yet provided proof . We also derive a second-order ordinary differential equation ( ODE ) , which is the continuous-time limit of A2BCD ( see Section 3 ) . This extends the ODE found in Su et al . ( 2014 ) to an asynchronous accelerated algorithm minimizing a strongly convex function . We prove this ODE linearly converges to a solution with the same rate as A2BCD ’ s , without needing to resort to the restarting techniques . The ODE analysis motivates and clarifies the our proof strategy of the main result .

Summary: In distributed optimisation, it is well known that asynchronous methods outperform synchronous methods in many cases. However, the questions as to whether (and when) asynchronous methods can be shown to have any speed-up, as the number of nodes increases, has been open. The paper under review answers the question in the affirmative and does so very elegantly.