Lower Bounds and Optimal Algorithms for Smooth and Strongly Convex Decentralized Optimization Over Time-Varying Networks

1 Introduction . In this work we are solving the decentralized optimization problem min x∈Rd n∑ i=1 fi ( x ) , ( 1 ) where each function fi : Rd → Rd is stored on a compute node i ∈ { 1 , . . . , n } . We assume that the nodes are connected through a communication network . Each node can perform local computations based on its local state and data , and can directly communicate with its neighbors only . Further , we assume the functions fi to be smooth and strongly convex . Such decentralized optimization problems arise in many applications , including estimation by sensor networks ( Rabbat and Nowak , 2004 ) , network resource allocation ( Beck et al. , 2014 ) , cooperative control ( Giselsson et al. , 2013 ) , distributed spectrum sensing ( Bazerque and Giannakis , 2009 ) and power system control ( Gan et al. , 2012 ) . Moreover , problems of this form draw attention of the machine learning community ( Scaman et al. , 2017 ) , since they cover training of supervised machine ∗King Abdullah University of Science and Technology , Thuwal , Saudi Arabia . †Moscow Institute of Physics and Technology , Moscow , Russia ‡Institute for System Programming of the Russian Academy of Sciences , Research Center for Trusted Artificial Intelligence , Moscow , Russia 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . learning models through empirical risk minimization from the data stored across the nodes of a network as a special case . Finally , while the current federated learning ( Konečný et al. , 2016 ; McMahan et al. , 2017 ) systems rely on a star network topology , with a trusted server performing aggregation and coordination placed at the center of the network , advances in decentralized optimization could be useful in new-generation federated learning formulations that would rely on fully decentralized computation ( Li et al. , 2020 ) . 1.1 Time-varying Networks . In this work , we focus on the practically highly relevant and theoretically challenging situation when the links in the communication network are allowed to change over time . Such time-varying networks ( Zadeh , 1961 ; Kolar et al. , 2010 ) are ubiquitous in many complex systems and practical applications . In sensor networks , for example , changes in the link structure occur when the sensors are in motion , and due to other disturbances in the wireless signal connecting pairs of nodes . We envisage that a similar regime will be supported in future-generation federated learning systems ( Konečný et al. , 2016 ; McMahan et al. , 2017 ) , where the communication pattern among pairs of mobile devices or mobile devices and edge servers will be dictated by their physical proximity , which naturally changes over time . 1.2 Contributions . In this work we present the following key contributions : 1 . Lower bounds . We establish the first lower bounds on decentralized communication and local computation complexities for solving problem ( 1 ) over time-varying networks . Our results are summarized in Table 1 , and detailed in Section 3 ( see Theorems 2 and 3 therein ) . 2 . Optimal algorithms . Further , we prove that these bounds are tight by providing two new optimal algorithms4 which match these lower bounds : ( i ) a variant of the recently proposed algorithm ADOM ( Kovalev et al. , 2021 ) enhanced via a multi-consensus subroutine , and ( ii ) a novel algorithm , called ADOM+ ( Algorithm 1 ) , also featuring multi-consensus . The former method is optimal in the case when access to the dual gradients is assumed , and the latter one is optimal in the case when access to the primal gradients is assumed . See Sections 4 and 5 for details . To the best of our knowledge , ADOM with multi-consensus is the first dual based optimal decentralized algorithm for time-varying networks 3 . Experiments . Through illustrative numerical experiments ( see Section 6 , and the extra experiments contained in the appendix ) we demonstrate that our methods are implementable , and that they perform competitively when compared to existing baseline methods APM-C ( Rogozin et al. , 2020 ; Li et al. , 2018 ) and Acc-GT ( Li and Lin , 2021 ) . Related Work . When the communication network is fixed in time , decentralized distributed optimization in the strongly convex and smooth regime is relatively well studied . In particular , Scaman et al . ( 2017 ) established lower decentralized communication and local computation complexities for solving this problem , and proposed an optimal algorithm called MSDA in the case when an access to the dual oracle ( gradient of the Fenchel transform of the objective function ) is assumed . Under a primal oracle ( gradient of the objective function ) , current state of the art includes the near-optimal algorithms APM-C ( Li et al. , 2018 ; Dvinskikh and Gasnikov , 2019 ) and Mudag ( Ye et al. , 2020 ) , and a recently proposed optimal algorithm OPAPC ( Kovalev et al. , 2020 ) . The situation is worse in the time-varying case . To the best of our knowledge , no lower decentralized communication complexity bound exists for this problem . There are a few linearly-convergent algorithms , such as those of Nedic et al . ( 2017 ) and Push-Pull Gradient Method of Pu et al . ( 2020 ) , that assume a primal oracle , and the dual oracle based algorithm PANDA due to Maros and Jaldén ( 2018 ) . These algorithms have complicated theoretical analyses , which results in slow convergence rates . There are also several accelerated algorithms , which were originally developed for the fixed 4As a byproduct of our lower bounds , we show that the recently proposed method Acc-GT of Li and Lin ( 2021 ) is also optimal . This method appeared on arXiv in April 2021 , at the time when we already had a first draft of this paper , including all results . Their work does not offer any lower bounds . network case , and can be extended to the time-varying case . These include Acc-DNGD ( Qu and Li , 2019 ) , Mudag ( Ye et al. , 2020 ) and a variant of APM-C which was extended to the time-varying case by Rogozin et al . ( 2020 ) . Finally , there are two algorithms with state-of-the-art decentralized communication complexity : a dual based algorithm ADOM ( Kovalev et al. , 2021 ) , and a primal based algorithm Acc-GT ( Li and Lin , 2021 ) . 2 Notation and Assumptions . 2.1 Smooth and Strongly Convex Regime . Throughout this paper we restrict each function fi ( x ) to be L-smooth and µ-strongly convex . That is , we require the following inequalities to hold for all x , y ∈ Rd and i ∈ { 1 , . . . , n } : fi ( y ) + ⟨∇fi ( y ) , x− y⟩+ µ2 ∥x− y∥ 2 ≤ fi ( x ) ≤ fi ( y ) + ⟨∇fi ( y ) , x− y⟩+ L2 ∥x− y∥ 2 . ( 2 ) This naturally leads to the quantity κ = Lµ known as the condition number of function fi . Strong convexity implies that problem ( 1 ) has a unique solution . 2.2 Primal and Dual Oracle . In our work we consider two types of gradient oracles . By primal oracle we denote the situation when the gradients ∇fi of the objective functions fi are available . By dual oracle we denote the situation when the gradients ∇f∗i of the Fenchel conjugates5 f∗i of the objective functions fi are available . 2.3 Decentralized Communication . Let V = { 1 , . . . , n } denote the set of the compute nodes . We assume that decentralized communication is split into communication rounds . At each round q ∈ { 0 , 1 , 2 , . . . } , nodes are connected through a communication network represented by a graph Gq = ( V , Eq ) , where Eq ⊂ { ( i , j ) ∈ V×V : i ̸= j } is the set of links at round q . For each node i ∈ V we consider a set of its immediate neighbors at round q : N qi = { j ∈ V : ( i , j ) ∈ Eq } . At round q , each node i ∈ V can communicate with nodes from the set N qi only . This type of communication is known in the literature as decentralized communication . 5Recall that the Fenchel conjugate of fi is given as f∗i ( x ) = supy∈Rd [ ⟨x , y⟩ − fi ( y ) ] . Note , that f∗i is 1/µ-smooth and 1/L-strongly convex ( Rockafellar , 2015 ) . 2.4 Gossip Matrices . Decentralized communication between nodes is typically represented via a matrix-vector multiplication with a gossip matrix . For each decentralized communication round q ∈ { 0 , 1 , 2 , . . . } , consider a matrix W ( q ) ∈ Rn×n satisfying the following assumption . Assumption 1 . For any q ∈ { 0 , 1 , 2 , . . . } matrix W ( q ) ∈ Rn×n satisfies the following relations : 1 . W ( q ) i , j = 0 if i ̸= j and ( i , j ) /∈ Eq , 2. kerW ( q ) ⊃ { ( x1 , . . . , xn ) ∈ Rn : x1 = . . . = xn } , 3. rangeW ( q ) ⊂ { ( x1 , . . . , xn ) ∈ Rn : ∑n i=1 xi = 0 } , 4 . There exists χ ≥ 1 , such that ∥Wx− x∥2 ≤ ( 1− χ−1 ) ∥x∥2 for all x ∈ { ( x1 , . . . , xn ) ∈ Rn : ∑n i=1 xi = 0 } . ( 3 ) Throughout this paper we will refer to the matrix W ( q ) as the gossip matrix . A typical example of a gossip matrix is W ( q ) = λ−1max ( L ( q ) ) · L ( q ) , where L ( q ) the Laplacian of an undirected connected graph Gq . Note that in this case W ( q ) is symmetric and positive semi-definite , and χ can be chosen as an upper-bound on the condition number of the matrix L ( q ) defined by χ = supq λmax ( L ( q ) ) λ+min ( L ( q ) ) , where λmax ( L ( q ) ) and λ+min ( L ( q ) ) denote the largest and the smallest positive eigenvalue of L ( q ) , respectively . With a slight abuse of language , we will call χ the condition number of time-varying network even when the gossip matrices W ( q ) are not necessarily symmetric . While the notion of a gossip matrix is typically used in the fixed network setting ( Scaman et al. , 2017 ) , we adapt this notion to the time-varying regime in which the so-called mixing matrices ( Nedic et al. , 2017 ) are typically used instead . However , it turns out that there is no major difference between these notions . While we decided to present our main results in the language of gossip matrices , they can alternatively be described in the language of mixing matrices instead . We provide a detailed discussion of this in the appendix .

Summary: This paper studies a decentralized optimization problem. There is a set of nodes that collectively want to find the minimum of an objective function that is the sum of local costs. The nodes can communicate on underlying communication graph.  The authors study the trade-off communication v.s. precision. They show a lower bound on the quality of a solution that can be attained given a certain communication budget. They then provide two algorithms that attain this lower bound.