Distributed Saddle-Point Problems Under Data Similarity

· /µ·log ( 1/ '' ) rounds of communications , where > 0 measures the degree of similarity of the local functions , µ is their strong convexity constant , and is the diameter of the network . The lower communication complexity bound over mesh networks reads ⌦ 1/ p ⇢ · /µ · log ( 1/ '' ) , where ⇢ is the ( normalized ) eigengap of the gossip matrix used for the communication between neighbouring nodes . We then propose algorithms matching the lower bounds over either types of networks ( up to log-factors ) . We assess the effectiveness of the proposed algorithms on a robust regression problem . 1 Introduction . We study smooth ( strongly- ) convex- ( strongly- ) concave SPPs over a network of M agents : min x2X max y2Y f ( x , y ) : = 1 M MX m=1 fm ( x , y ) , ( P ) where X , Y ✓ Rd are convex and compact sets common to all the agents ; and fm ( x , y ) is the loss function of agent m , known only to the agent . Problem ( P ) has found a wide range of applications , including , game theory [ 42 , 10 ] , image deconvolution problems [ 7 ] , adversarial training [ 3 , 12 ] , and statistical learning [ 1 ] –see Sec . 2 for some motivating examples in the distributed setting . We are particularly interested in learning problems , where each fm is the empirical risk that measures the mismatch between the model to be learned and the local dataset owned by agent m. Since the functions fm can be accessed only locally and routing local data to other agents is infeasible or highly inefficient , solving ( P ) calls for the design of distributed algorithms that alternate between a local computation procedure at each agent ’ s side , and a round of communication among ( suitably chosen ) neighboring nodes . We address such a design considering explicitly two type of computational architectures , namely : ( i ) master/workers networks–these are centralized systems suitable for parallel computing ; for instance , they are the typical computational architecture arising ⇤Moscow Institute of Physics and Technology †ISP RAS Research Center for Trusted Artificial Intelligence 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . from federated learning applications ( e.g. , [ 17 ] ) , where data are split across multiple workers and computations are performed in parallel , coordinated by the master node ( s ) ; and ( ii ) mesh networks– these are distributed systems with no special topology ( modeled just as undirected graphs ) , which capture scenarios wherein there is no hierarchical structure ( e.g. , master nodes ) and each node can communicate only with its intermediate neighbors . Function similarity : Motivated in particular by machine learning applications , our design and analysis pertain to distributed algorithms for SPPs ( P ) where the local functions fm ’ s are related– quantities such as gradients and the second derivatives matrices of fm ’ s differ only by a finite quantity > 0 ; we will term such SPPs as -related SPPs . For instance , this is the typical situation in the aforementioned distributed empirical risk minimization setting [ 2 , 14 , 47 ] : when data are i.i.d . among machines , the fm ’ s reflect statistical similarities in the data residing at different nodes , resulting in a = Õ ( 1/ p n ) , where n is the local sample size ( Õ hides log-factors and dependence on d ) . While SPPs have been extensively studied in the centralized setting ( e.g. , [ 10 , 29 , 18 , 30 , 5 ] ) and more recently over mesh networks [ 23 , 27 , 22 , 26 , 36 , 4 , 6 ] , we are not aware of any analysis or ( distributed ) algorithm that explicitly exploit function similarity to boost communication efficiency–either lower complexity bounds or upper bounds . On the other hand , recent works for sum-utility minimization problems over networks ( e.g. , [ 2 , 38 , 35 , 45 , 43 , 11 , 47 , 14 , 39 , 20 ] ) show that employing some form of statistical preconditioning in the algorithm design provably reduces communication complexity . Whether these improvements are possible/achievable for -related SSPs in the form ( P ) remains unclear . This paper provides a positive answer to the above open problem . Major contributions : Our major results are summarized next . ( a ) Lower complexity bounds : Under mild structural assumptions on the algorithmic oracle ( satisfied by a variety of methods ) , we establish lower complexity bounds for the -related SPP ( P ) with µ-strongly-convex-strongly -concave , L-smooth ( twice-differentiable ) local functions : an `` precision on the optimality gap over master/workers system is achieved in ⌦ · /µ · log ( 1/ '' ) communication steps , where is the diameter of the network . The lower complexity bound over mesh networks reads ⌦ 1/ p ⇢ · /µ · log ( 1/ '' ) rounds of communications , where ⇢ is the ( normalized ) eigengap of the gossip matrix used for the communication between neighbouring nodes . These new lower bounds show a more favorable dependence on the optimization parameters ( via /µ ) than that of distributed oracles for SPPs ignoring function similarity [ 5 , 36 ] , whose communication complexity , e.g. , over mesh networks reads ⌦ 1/ p ⇢ · L/µ · log ( 1/ '' ) . The latter provides a pessimistic prediction when /µ ⌧ L/µ . This is the typical situation of ill-conditioned problems , such as many learning problems where the regularization parameter that is optimal for test predictive performance is so small that a scaling with L/µ is no longer practical while /µ is ( see , e.g. , [ 25 , 14 ] ) . ( b ) Near optimal algorithms : We proposed algorithms for such SPPs over master/workers and mesh networks that match the lower bounds up to logarithmic factors . They are provably faster than existing solution methods for µ-strongly-convex-strongly-concave , L-smooth SPPs , which do not exploit function similarity . Preliminary numerical results on distributed robust logistic regression support our theoretical findings . 1.1 Related works . Methods for SPPs ignoring function similarity : ( Strongly ) -convex- ( strongly ) -concave SPPs have been extensively studied in the optimization literature and as special instances of ( strongly ) monotone Variational Inequalities ( VI ) [ 10 , 16 ] . Several algorithms are available in the centralized setting , some directly imported from the VI literature ; representative examples include : the mirror-proximal algorithm [ 29 ] , Extragradient method [ 18 ] and the scheme in [ 30 ] –they are readily implementable on master/workers architectures as well . For SPPs with µ-strongly-convex-strongly-concave , L-smooth loss , all these schemes achieve iteration complexity of O L/µ · log ( 1/ '' ) , which has been shown to be optimal for first-order methods solving such a class of SPPs [ 46 , 34 ] . Lower bounds and optimal algorithms in the distributed setting for SPPs without similarity have been studied in [ 5 ] . Note that none of the above lower ( and upper ) complexity bounds or ( centralized or distributed ) algorithmic designs capture function similarity . As a consequence , convergence rates certified in the aforementioned works , when applicable to -related SPPs in the form ( P ) , provide quite pessimistic predictions , in the setting 1 + /µ ⌧ L/µ . Methods for sum-utility minimization exploiting function similarity : Several works exploited the idea of statistical preconditioning to provably improve communication complexity of solution methods for the minimization of the sum of -related , µ-strongly convex and L-smooth functions over mas- ter/workers networks . Lower complexity bounds are established in [ 2 ] , and read ⌦ p /µ log ( 1/ '' ) , which contrasts with O p L/µ log ( 1/ '' ) achievable by first-order ( Nesterov ) accelerated methods [ 31 ] , certifying thus faster rates whenever /µ < L/µ . Solutions methods exploiting function similarity are mirror proximal-like schemes , and include [ 38 , 35 , 45 ] ( for quadratic losses ) , [ 47 ] ( for self-concordant losses ) , [ 43 ] , and [ 11 ] ( for composite optimization ) , with [ 14 ] employing acceleration . None of these methods are implementable over mesh networks , because they rely on a centralized ( master ) node . To our knowledge , Network-DANE [ 20 ] and SONATA [ 39 ] are the only two methods that leverage statistical similarity to enhance convergence of distributed methods over mesh networks ; [ 20 ] studies strongly convex quadratic losses while [ 39 ] considers general objectives , achieving a communication complexity of eO ( ( 1/p⇢ ) · /µ · log ( 1/ '' ) ) , where eO hides logarithmic factors . None of the methods above however are applicable to the -related SPP ( P ) . 1.2 Notation . Given a positive integer M , we define [ M ] = { 1 , . . . , M } . We use hx , yi : = P d i=1 xiyi to denote standard inner product of x , y 2 Rd . It induces ` 2-norm in Rd in the following way kxk : = p hx , xi . We also introduce projZ ( z ) = minu2Z ku zk – the Euclidean projection onto Z . We order the eigenvalues of any symmetrix matrix A 2 Rm⇥m in nonincreasing fashion , i.e. , max ( A ) = 1 ( A ) . . . m ( A ) = min ( A ) , with max ( · ) [ resp . min ( · ) ] denoting the largest ( resp . smallest ) eigenvalue . 2 Setup and Background . Problem setting : We begin introducing the main assumptions underlying Problem ( P ) and some useful notation . Let us stack the x- and y-variables in the tuple z = ( x , y ) ; accordingly , define Z = X ⇥ Y and the vector-functions Fm , F : Z ! R2d : Fm ( z ) : = ✓ rxfm ( x , y ) ryfm ( x , y ) ◆ , and F ( z ) : = 1 M MX m=1 Fm ( z ) . ( 1 ) The following conditions are standard for strongly convex-strongly concave SPPs . Assumption 1 Given ( P ) , the following hold : ( i ) ; 6= Z is a convex set ; ( ii ) Each fm : R2d ! R is twice differentiable on ( an open set containing ) Z , with L-Lipschitz gradient : kFm ( z1 ) Fm ( z2 ) k Lkz1 z2k , for all z1 , z2 2 Z ; ( iii ) f ( z ) is µ-strongly convex-strongly concave on Z , i.e. , hF ( z1 ) F ( z2 ) , z1 z2i µkz1 z2k2 , for all z1 , z2 2 Z ; ( iv ) Each fm ( z ) is convex-concave on Z , i.e . 0-strongly convex-strongly concave . We are interested in finding the solution z⇤ = ( x⇤ , y⇤ ) of Problem ( P ) under function similarity . Assumption 2 ( -related fm ’ s ) The local functions are -related : for all ( x , y ) 2 Z , kr 2 xx fm ( x , y ) r 2 xx f ( x , y ) k , kr 2 xy fm ( x , y ) r 2 xy f ( x , y ) k , kr 2 yy fm ( x , y ) r 2 yy f ( x , y ) k . The interesting case is when 1 + /µ ⌧ L/µ . When the fm ’ s are empirical loss functions over local data sets of size n , under standard assumptions on data distributions and learning model ( e.g. , [ 47 , 14 ] ) , = Õ ( 1/ p n ) with high probability ( Õ hides log-factors and dependence on d ) –some motivating examples falling in this category are discussed in Sec . 2.1 below . While such examples represent important applications , we point out that our ( lower and upper ) complexity bounds are valid in all scenarios wherein Assumption 2 holds , not necessarily due to statistical arguments . Network setting : The communication network is modeled as a fixed , connected , undirected graph , G , ( V , E ) , where V , { 1 , . . . , M } denotes the vertex set–the set of agents–while E , { ( i , j ) | i , j 2 V } represents the set of edges–the communication links ; ( i , j ) 2 E iff there exists a communication link between agent i and j . We denote by the diameter of the graph . When it comes to distributed algorithms over mesh networks , we leverage neighbouring communications among adjoining nodes . Communications of d-dimensional vectors will be modeled as a matrix multiplication by a matrix W ( a.k.a . gossip matrix ) . The following assumptions on W are standard to establish convergence of distributed algorithms over mesh networks . Assumption 3 The matrix W 2 RM⇥M satisfies the following : ( a ) It is compliant with G , that is , ( i ) wii > 0 , 8i 2 [ M ] ; ( ii ) wij > 0 , if { j , i } 2 E ; and ( iii ) wij = 0 otherwise ; ( b ) It is symmetric and stochastic , that is , W1 = 1 ( and thus also 1 > W = 1 > ) . Notice that a direct consequence of Assumption 3 ( along with the fact that G is connected ) is that ⇢ , 1 max { 2 ( W ) , | min ( W ) | } < 1 , ( 2 ) where ⇢ is the eigengap between the first and second largest ( magnitude ) eigenvalue of W . Roughly speaking , ⇢ measures how fast the network mixes information ( the larger , the faster ) .

Summary: The paper studies saddle-point problems with objective being a sum of strongly convex-concave functions, $\min_x \max_y f(x,y) = \sum_{m = 1}^M f_m(x,y)$, in the setting of distributed communication. Specifically, each component in the sum of functions is local to an agent $m$, and communication between agents is expensive, which forces algorithms of two forms: master-worker type, and meshed grid type.   The paper's chief assumption is that the functions $f_i$ are $\delta$-similar, which means an upper bound of $\delta$ on  the operator norm  of the difference between the Hessians of $f_m$ and $f$, evaluated with respect to $(x,x)$, $(x, y)$, and $(y, y)$ directions. This assumption is reasonable in the case of robust linear regression and in proving adversarial robustness of neural networks.   The paper provides lower bounds and algorithms meeting these lower bounds for solving the saddle point problems to $\epsilon$ accuracy.