Title: Achieving Linear Convergence with Parameter-Free Algorithms in Decentralized Optimization
Venue: NeurIPS 2024
Accepted: yes
Keywords: decentralized optimization

linear convergence

parameter free
Abstract: This paper addresses the minimization of the sum of strongly convex, smooth
functions over a network of agents without a centralized server. Existing decentralized algorithms require knowledge of functions and network parameters, such as the Lipschitz constant of the global gradient and/or network connectivity, for
hyperparameter tuning. Agents usually cannot access this information, leading
to conservative selections and slow convergence or divergence. This paper introduces a decentralized algorithm that eliminates the need for specific parameter
tuning. Our approach employs an operator splitting technique with a novel variable
metric, enabling a local backtracking line-search to adaptively select the stepsize
without global information or extensive communications. This results in favorable
convergence guarantees and dependence on optimization and network parameters
compared to existing nonadaptive methods. Notably, our method is the first adaptive decentralized algorithm that achieves linear convergence for strongly convex,
smooth objectives. Preliminary numerical experiments support our theoretical
findings, demonstrating superior performance in convergence speed and scalability.

Achieving Linear Convergence with Parameter-Free Algorithms in Decentralized Optimization

Abstract

This paper addresses the minimization of the sum of strongly convex, smooth functions over a network of agents without a centralized server. Existing decentralized algorithms require knowledge of functions and network parameters, such as the Lipschitz constant of the global gradient and/or network connectivity, for hyperparameter tuning. Agents usually cannot access this information, leading to conservative selections and slow convergence or divergence. This paper introduces a decentralized algorithm that eliminates the need for specific parameter tuning. Our approach employs an operator splitting technique with a novel variable metric, enabling a local backtracking line-search to adaptively select the stepsize without global information or extensive communications. This results in favorable convergence guarantees and dependence on optimization and network parameters compared to existing nonadaptive methods. Notably, our method is the first adaptive decentralized algorithm that achieves linear convergence for strongly convex, smooth objectives. Preliminary numerical experiments support our theoretical findings, demonstrating superior performance in convergence speed and scalability.

1 Introduction

We study optimization across a network of m > 1 agents, modeled as an undirected, static graph, possibly with no centralized server. The agents cooperatively solve the following problem:

where f i : R n → R is the loss function of agent i , assumed to be strongly convex and smooth (i.e., with gradient being Lipschitz continuous), and accessible only to agent i .

This formulation applies to various fields, particularly emphasizing decentralized machine learning problems where datasets are produced and collected at different locations. Traditionally, statistical and computational methods in this domain have relied on a centralized paradigm, aggregating computational resources at a single, central location. However, this approach is increasingly unsuitable for modern applications with many machines, leading to server congestion, inefficient communication, and high energy consumption [27, 23]. This has motivated the surge of learning algorithms that target decentralized networks with no servers , a.k.a. mesh networks, which is the setting of this paper.

Decentralized convex optimization has a long history, with numerous algorithms applicable to Problem (P); recent tutorials include [34, 41, 6, 33, 45]. Lack of adaptivity: These methods share the hurdle of relying sensibly on the tuning of hyperparameters, such as the stepsize (a.k.a. learning rate), for both theoretical and practical convergence. Existing theories ensure convergence under generally conservative bounds on the stepsize, which depend on parameters like the Lipschitz constant of the global gradient, the spectral gap of the graph adjacency matrix, or other topological properties. Acquiring such information is challenging in practice, due to physical or privacy limitations and computational/communication constraints. This often leads to manual tuning, which is not only tedious but also results in less predictable, problem-dependent, and non-reproducible performance.

.

Parameter-free centralized methods: On the the hand, significant progress has been made in the centralized setting to automate the selection of the stepsize across various optimization and learning problem classes. (i) Traditional approaches in optimization-such as line-search methods [36], BarzilaiBorwein's stepsize [3], and Polyak's stepsize [37]-have been supplemented by recent adaptive stepsize rules based on estimates of local curvature [30] and subsequent techniques [31, 19, 20, 22, 50]. (ii) In the ML community, adaptive gradient methods such as AdaGrad [11], Adam [18], AMSGrad [40], NSGD-M [9], and variants [25, 44, 29] have gained significant attention for training large-scale learning models. These methods apply to stochastic, nonconvex optimization problems. (iii) Further advancements extend adaptivity to stochastic/online convex optimization problems, e.g., [5, 15].

Distributed adaptive methods: While variant of these centralized algorithms have been adapted to federated architectures (server-client systems), e.g., in [39, 24, 8], their application to mesh networks is not feasible . In federated learning, a central server aggregates local model updates, a process integral to its hierarchical structure. However, mesh networks, which lack a centralized coordinating node, do not support such a direct aggregation of large-scale vectors. Recent attempts to implement some form of stepsize adaptivity for stochastic (non)convex/online optimization problems over mesh networks are [32, 7, 21]. These methods generally achieve adaptivity by properly normalizing agents' gradients using past information. However, with the exception of [21], they rely on the strong assumption that the (population) losses are globally Lipschitz continuous (i.e., their gradients are bounded). In fact, Lipschitz continuity in convex optimization readily unlocks parameter-free convergence by using stepsize tuning of O (1 / √ k ) (here, k is the iteration index). Moreover, [32, 7] still require knowledge of some optimization parameters for the stepsize tuning, to guarantee convergence.

Attempts to introduce adaptivity in decentralized optimization for solving (P) have been explored in [12, 14, 13]. These methods bring the Barzilai-Borwein (BB)'s stepsize strategy into gradient tracking algorithms [43, 28, 35, 48]. (i) However, convergence of these algorithms is not guaranteed under the proposed BB strategy, unless the stepsizes remain uniformly bounded from below and above throughout the algorithm's trajectory-a condition the BB rule does not inherently satisfy in decentralized settings. Furthermore, these bounds for the stepsizes are typically unknown to the agents, as they depend on the strong convexity and smoothness constants of all agents' losses. Even with such knowledge, enforcing these conservative bounds contradicts the principle of adaptivity by potentially negating the advantages of a variable stepsize strategy that adapts based on local loss curvature, producing stepsize values significantly larger than theoretical thresholds used in nonadaptive methods. (ii) Additionally, to ensure contraction of the iterates, studies such as [13, 14] require multiple rounds of communications per iteration (gradient evaluation)-this demands the knowledge of network and optimization parameters at the agents' sides, making practical implementation unfeasible. (iii) None of these studies offer expressions of convergent rates for the explored algorithms, leaving it unclear whether the BB stepsize rule can provably outperform nonadaptive methods. (iv) Lastly, the methods discussed employ the traditional BB rule, which is only proven in centralized settings to produce convergence methods when minimizing quadratic losses. Simulations in [12, 13] are in fact performed only on quadratic functions.

Open questions and challenges: To our knowledge, no deterministic, parameter-free decentralized algorithms exist that solve Problem (P) over mesh networks, particularly achieving linear convergence when agents' functions are strongly convex and smooth. The current decentralized adaptive stochastic methods [32, 7, 21] discussed earlier do not adequately bridge this gap. Tailored for stochastic environments, these methods merely ensure that cumulative consensus errors along the iterations remain bounded, not necessarily decreasing . This typically involves either diminishing stepsizes or adjustments based on the final horizon to manage the bias-variance trade-off. These strategies fall short in deterministic scenarios like Problem (P), failing to ensure convergence to exact solutions, and achieve faster O (1 /k ) convergence rates in convex cases or linear rates in strongly convex scenarios.

Major contributions: This paper addresses this open problem. Our contributions are the following:

1. A new parameter-free decentralized algorithm: We propose a decentralized algorithm that eliminates the need for specific tuning of the stepsize. Our approach leverages a Forward-Backward operator splitting technique combined with a novel variable metric, enabling a local backtracking line-search procedure to adaptively select the stepsize at each iteration without requiring global information on optimization and network parameters or extensive communications. We are not aware of any other provable decentralized line-search methods over mesh networks.

Designing decentralized line-search procedures that are well-defined (terminating in a finite number of steps), locally implementable, and ensure algorithm convergence through satisfactory descent on an appropriate merit function presents significant challenges. A major issue is that line-search procedures merely based on the local curvature of agents' functions often fail to ensure convergence, producing excessively large , heterogeneous stepsizes that, e.g., poorly connected networks cannot support. This necessitates the identification of line-search directions and surrogate functions that encapsulate both optimization and network influences, aspects that have not yet formalized. Our design guidelines (cf., Sec. 3) are of independent interest; hopefully they will provide valuable insights for the development of other decentralized adaptive schemes, such as those based on alternative operator splittings.

2. Convergence guarantees: We have established linear convergence for the proposed decentralized adaptive method. Our analysis identifies critical quantities that capture the interplay between optimization conditions and network topology, directly influencing the convergence rates. Specifically: (a) In 'well-connected' networks, the convergence rate exhibits a separation property : the overall rate is dictated by the slower of either the centralized gradient algorithm solving the same problem or a consensus algorithm run on the same mesh network. (b) Conversely, in 'poorly' connected networks, the separation property vanishes, and the convergence rates are adversely affected by network degradation terms, still exhibiting a linear dependence on the condition number of the optimization loss. (ii) Unlike many existing distributed optimization frameworks, the optimization parameters in our rate expressions-such as smoothness and strong convexity constants-are localized to the convex hull of the traveled iterates. This localization arises from our adaptive stepsize strategy, which employs a line-search procedure tailored to local geometries, yielding more favorable dependencies on optimization parameters and thus enhanced convergence guarantees. (iii) Numerical experiments demonstrate superior performance of the proposed adaptive algorithm in convergence speed and scalability compared to existing non-adaptive methods.

1.1 Notation and paper organization

Capital letters denote matrices. Bold capital letters represent matrices where each row is an agent's variable, e.g., X = [ x 1 , . . . , x m ] ⊤ . For such matrices, the i -th row is denoted by the corresponding lowercase letter with the subscript i ; e.g., for X , we write x i (as column vector). Let S m , S m + , and S m ++ be the set of m × m (real) symmetric, symmetric positive semidefinite, and symmetric positive definite matrices, respectively; A † denotes the Moore-Penrose pseudoinverse of A . The eigenvalues of W ∈ S m are ordered in nonincreasing order, and denoted by λ 1 ( W ) ≥ · · · ≥ λ m ( W ) . For two operators A and B of appropriate size, ( A ◦ B )( · ) stands for A ( B ( · )) . Wedenote: [ m ] = { 1 , . . . , m } , for any integer m ≥ 1 ; [ x ] + := max( x, 0) , x ∈ R ; 1 m ∈ R m is the vector of all ones; I m (resp. 0 m ) is the m × m identity (resp. the m × m zero) matrix; the information on the dimension is omitted when not necessary; null ( A ) (resp. span ( A ) ) is the nullspace (resp. range space) of the matrix A . Let ⟨ X,Y ⟩ := tr ( X ⊤ Y ) , for any X and Y of suitable size ( tr ( · ) ) is the trace operator; and ∥ X ∥ M := √ ⟨ MX,X ⟩ , for any symmetric, positive definite M and X of suitable dimensions. We still use ∥ X ∥ M when M is positive semidefinite and X ∈ span ( M ) .

2 Problem Setup

We investigate Problem (P) over a network of [ m ] agents, modeled as an undirected, static, connected graph G = ([ m ] , E ) , where ( i, j ) ∈ E if there is communication link (edge) between i and j . For each agent i , we define by N i := { j : | ( i, j ) ∈ E , for some i ∈ [ m ] } ∪ { i } the set of immediate neighbors of agent i (including agent i itself).

Assumption 1. (i) Each function f i in (P) is L -smooth and µ -strong convex on R n , for some L ∈ (0 , ∞ ) and µ ∈ (0 , ∞ ) ; and (ii) each agent i has access only to its own function f i .

The following matrices are commonly utilized in the design of gossip-based algorithms.

Definition 2 (Gossip matrices) . Let W G denote the set of matrices ˜ W = [ ˜ W ij ] m i,j =1 that satisfy the following properties: (i) (compliance with G ) ˜ W ij > 0 if ( i, j ) ∈ E ; otherwise ˜ W ij = 0 . Furthermore, ˜ W ii > 0 , for all i ∈ [ m ] ; and (ii) (doubly stochastic) ˜ W ∈ S m and ˜ W 1 m = 1 m .

These matrices are standard in the literature on decentralized optimization algorithms, and several instances have been employed in practice; see [34, 41, 33] for some representative examples. Notice

that for any ˜ W ∈ W G (assuming G connected) it hold: (i) (null space condition) null ( I m -W ) = span (1 m ) ; and (ii) (eigen-spectrum distribution) 2 I ⪰ ˜ W + I ≻ 0 m .

3 Algorithm Design

Our approach to solving Problem (P) involves a saddle-point reformulation tackled via a variable metric operator splitting, implementable across the graph G . The innovative aspect of the proposed method lies in the selection of the variable metric that, coupled with a Forward Backward Splitting (FBS), enable adaptive stepsize selections through a decentralized line-search procedures.

Introducing local copies x i ∈ R d of the shared variable x (the i -th one is controlled by agent i ), and the stack matrix X := [ x 1 , . . . , x m ] ⊤ ∈ R m × n , let us consider the following auxiliary problem:

Here, Ł and K are m × m matrices that meet the following criteria: (c1) Ł ∈ S m and null ( Ł ) = span (1 m ) ; (c2) K ∈ S m ++ and null ( I -K ) = span (1 m ) ; and (c3) Ł and K commute. Conditions (c1) and (c2) ensure that (P) and (P ′ ) are equivalent. Specifically, any solution X ⋆ of (P ′ ) has the form of X ⋆ = 1 m ( x ⋆ ) ⊤ , where x ⋆ solves (P), and vice versa. While not essential, condition (c3) is postulated to simplify the algorithm derivation.

Primal-dual optimality for (P ′ ) reads, with Y being the dual-variable associated with the constraints,

Given X k , Y k at iteration k , the update X k +1 , Y k +1 via FBS with metric C ∈ S 2 m ++ reads [4]

Monotone operator theory [4] ensures convergence of (1) under the following conditions:

(c4) B is a monotone operator, C ∈ S 2 m ++ , and (c5) I -C -1 / 2 AC -1 / 2 is an averaged operator.

Condition (c4) is satisfied by construction; (c5) can be enforced through a suitable selection of C ∈ S 2 m ++ while leveraging the co-coercivity of A (implied by Assumption 1). Denoting by α > 0 the stepsize employed in the algorithm, we seek for C with the following structure:

to be determined. We proceed solving (1). Taking ( C + B ) -1 , we have

where

In addition to satisfying (c5), C 1 , C 2 ∈ S m ++ must be strategically chosen to facilitate the design of a decentralized line-search procedure for α . We propose the following guiding principles:

(c6) The range of admissible stepsize values α ensuring convergence-hence satisfying (c5)-should be independent of the network parameters; and

(c7) the operators ( I ) , ( II ) , and ( III ) in (2) should be independent of α .

At a high level, (c6) aims to decouple the line-search mechanism from network-dependent constraints. By doing so, it ensures that performing the line-search from the agents' sides requires no midprocess communications during backtracking, relying solely on local computations. Meanwhile, (c7) facilitates the identification of -(( II )( X k ) + ( III )( Y k )) in (2) as a potential line-search direction. This direction must be paired with an appropriate surrogate function, which we will define shortly.

Among several potential selections, in this paper, we consider the following for C 1 and C 2 :

which satisfy all the specified requirements. Using (4) and (c3), the operators in (3) simplify to

Notice that ( I ) , ( II ) , and ( III ) are independent of the stepsize. Substituting the above expressions in (2) and introducing D k := K -1 Ł Y k , the algorithm can be rewritten as

To make the above updates compliant with the graph G while satisfying (c1)-(c3), we set Ł 2 = ( I -˜ W ) , with ˜ W ∈ W G , and K = I -c Ł 2 , where c ∈ (0 , 1 / 2) is a free universal constant. Introducing W := (1 -c ) I m + c ˜ W ∈ W G , the final decentralized algorithm can be rewritten as

Finally, it can be verified that (c6) is met if ( √ αK -1 / 2 ) ◦ ∇ F ◦ ( √ αK -1 / 2 ) is nonexpansive, which holds if α < 1 /L , being independent on the network parameters. Next, we introduce a line-search procedure that enables the use of an adaptive stepsize α rather than a constant one satisfying the above more conservative bound.

Decentralized backtracking: It is not difficult to check that -D k +1 / 2 is a descent direction of F k ( X ) := F ( X )+ ⟨ D k , X ⟩ at X k +1 / 2 . This naturally suggests the following backtracking procedure for α : at iteration k , find the largest α k > 0 such that

where δ ∈ (0 , 1] is a tuning parameter. However, this condition would require a communication round for each backtracking step. To reduce the communication burden, we introduce a local stepsize for each agent i , denoted by α k i , determined by a backtracking line-search on the local function f k i ( x ) := f i ( x ) + ⟨ d k i , x ⟩ . Specifically, each α k i is the largest positive value satisfying

Clearly α k = min i ∈ [ m ] α k i also satisfies (6). Noticing that f k i has the same smooth (and strong convexity) constant(s) of f i , one can replace f k i in (7) with f i . The proposed decentralized algorithm is summarized in Algorithm 1, with the backtracking line-search procedure detailed in Algorithm 2.

3.1 Discussion

Several comments are in order.

̸

On the proposed algorithm: We emphasize that selecting K = I m in (P ′ ) marks a significant departure from the commonly used saddle-point reformulations of Problem (P), where K = I m , e.g., [46, 34, 33, 1]. Choosing K = I m , in conjunction with the novel variable metric C in the FBS as

̸

Algorithm 1

Data: (i) Initialization X 0 ∈ R m × n and D 0 = 0 ; (ii) initial value α -1 ∈ (0 , ∞ ) ; (iii) Backtracking parameters δ ∈ (0 , 1]0 ; (iv) nondecreasing sequence { γ k } k ⊆ [1 , ∞ ) (v) Gossip matrix W := (1 -c ) I m + c ˜ W , with ˜ W ∈ W G , and c ∈ (0 , 1 / 2] . Set the iteration index k = 0 .

1: (S.1) Communication step: Agents updates primal and dual variables via gossiping:

2: (S.2) Decentralized line-search : Each agent updates α k i according to

3: (S.3) Global min-consensus :

4: (S.4) Local updates of the primal and dual variables:

5: (S.5) If a termination criterion is not met, k ← k +1 and go to step (S.1) .

Algorithm 2 Backtracking( α , f , x , d , γ , δ )

1: α + := γα ;

2: x + := x + α + d ; set t = 1 ;

3: while f ( x + ) > f ( x ) + ⟨∇ f ( x ) , x + -x ⟩ + δ 2 α + ∥ x + -x ∥ 2 do

4: α + ← (1 / 2) α + ;

5: x + := x + α + d ;

6: t ← t +1 ;

return

α

+

.

specified in (4), is critical to obtain a valid line-search procedure that is also implementable across the network. For instance, popular decentralized algorithms such as EXTRA [42] and NIDS [26] can be interpreted as FBS with suitable metrics associated with the primal-dual reformulation of (P) as (P ′ ) but with K = I m . However, these schemes do not facilitate any suitable line-search, as no stepsizeindependent descent direction can be identified in their updates. Hopefully, our approach will provide principled guidelines for the design of other parameter-free decentralized algorithms, stemming from alternative decentralized formulations of (P) and their corresponding operator splittings.

On the backtracking: The following Lemma shows that the line-search procedure in Algorithm 2 is well-defined, as long as the function f therein is locally smooth.

Lemma 3. Let f in Algorithm 2 be any L f -smooth and µ f -strongly convex function on the segment [ x, x + γαd ] , where L f ∈ (0 , ∞ ) , µ f ∈ [0 , ∞ ) , and γ ∈ [1 , ∞ ) . The following hold for Algorithm 2: +

(i) The returned α satisfies

Therefore, the backtracking procedure terminates in at most max ( 1 , ⌈ log 2 2 L i γα δ ⌉ ) t -steps;

(ii) For any α + returned by Algorithm 2, any ¯ α + ∈ (0 , α + ] also satisfies the backtracking condition.

Notice that the last statement of the lemma guarantees that the each α k = min i ∈ [ m ] α k i satisfies the descent property (6) on the global loss F k , as each α k i meets the local condition (7).

The sequence { γ k } ∞ k =1 used in line 1 of the backtracking algorithm, with each γ k ≥ 1 , is introduced to favor nonmonotone, and thus potentially larger, stepsize values between two consecutive line-

search calls. Any sequence satisfying γ k ↓ 1 and ∏ ∞ k =1 γ k = ∞ , is advisable. In our experiments, we found the following rule quite effective: γ k = ( ( k + β 1 ) / ( k +1) ) β 2 , for some β 2 > 0 and β 1 ≥ 1 . One can also opt for γ k = 1 , for all k , thus eliminating this extra parameter, if simplicity is desired.

On the min-consensus: Step (S.3) involves a min-consensus across the network to establish a common stepsize, α k = min i ∈ [ m ] α k i , among the agents. This procedure is easily implemented in federated systems, where a server node facilitates information exchange between clients. Interestingly, this min-consensus protocol is also well-suited to current wireless mesh network technologies. Modern networks support multi-interface communications, including WiFi and LoRa (Low-Range) [17, 2, 16]. WiFi allows high-speed, short-range communications, supporting a mesh topology where nodes transmit large data volumes to immediate neighbors. Conversely, LoRa facilitates long-range but low-rate communications, ideal for communication flooding that reaches all network nodes in a single hop but transmits minimal information. Therefore, in multi-interface networks, the proposed algorithm operates by transmitting vector variables in Steps (S.1) via WiFi, while LoRa is used for the min-consensus in Step (S.3) . Furthermore, the values α k i 's can be quantized to their nearest lower values using a few bits before transmission. Based on Lemma 3(ii), this quantization ensures that the descent condition (6) is still met with the resultant min quantized stepsize. This approach renders the extra communication cost for implementing the global min-consensus step negligible.

For networks where LoRa technology cannot be used, Sec. 5 proposes a variant of Algorithm 1 wherein the global min-consensus step (S.3) is replaced by a local min-consensus procedure.

4 Convergence Results

We begin introducing a quantity of interest that helps identifying different operational regimes of the proposed algorithm. Let ( X ⋆ , D ⋆ ) be a fixed point of Algorithm 1 (whose existence is ensured by Assumption 1), and let { ( X k , D k ) } be the iterates generated by Algorithm 1. Define

̸

The following comments are in order. (i) Both D k and D ⋆ lie in the span ( I -˜ W ) = span ( I -W ) , for all k , and M is positive defined on this span. Consequently, ∥ D k -D ⋆ ∥ M > 0 for all D k = D ⋆ , and ∥ D k -D ⋆ ∥ M = 0 if and only if D k = D ⋆ . (ii) Under Assumption 1, X ⋆ = 1( x ⋆ ) ⊤ , where x ⋆ is the solution of (P). (iii) The quantity r k reflects the algorithm's convergence progress through the evolution of the dual variables and consensus error. Rewriting the update of the dual variables as D k +1 = D k + c α k ( I -˜ W ) X k -c ( I -˜ W ) ( ∇ F ( X k +1 / 2 ) + D k ) , we observe that small values of ∥ c α k ( I -˜ W ) X k -c ( I -˜ W ) ( ∇ F ( X k +1 / 2 ) + D k ) ∥ compared to ∥ D k -D ⋆ ∥ and ∥ X k -X ⋆ ∥ -hence small r k values-indicate slow convergence improvements (see Lemma 8 in the appendix).

We remark that r k need not be known by the agents; it is instrumental only for the analysis and posterior assessment of algorithm performance.

Linear convergence is established below via contraction of the following merit function

Theorem 4. Given Problem (P) under Assumption 1, let { ( X k , D k ) } be the iterates generated by Algorithm 1. Then, the following holds

If r k < 2 / 4 , then

where

Here µ k (resp. µ k +1 ) and L k are the strong convexity and smoothness constants of (each) f i along the segment [ x k +1 / 2 i , x ⋆ ] (resp. [ x k +1+1 / 2 i , x ⋆ ]) , respectively.

The theorem establishes linear convergence of Algorithm 1. As max(1 , ( α k /α k -1 ) 2 ) is bounded away from zero and uniformly upper bounded (with value depending on the sequence { γ k } )-see Lemma 3-the convergence rate is predominantly determined by { ρ k 1 } , { ρ k 2 } , and { r k } . Notice that, in the setting of the theorem, each ρ k 1 , ρ k 2 ∈ [0 , 1) . Intriguingly, the algorithm exhibits different operational regimes based on the range of values r k takes along the traveled trajectory. At the high level, if r k remains 'large', faster convergence can be guaranteed, as certified by (11); otherwise V k decreases every two consecutive iterations (see (12)), yielding to a slower convergence. The number of iterations required to reach a desired termination accuracy is given next.

Corollary 4.1. Instate the setting of Theorem 4, with now { γ k } being chosen such that γ k ≤ ( ( k + β 1 ) / ( k +1) ) β 2 , for all k and some β 1 ≥ 1 , β 2 > 0 . Then

for all k ≥ N ε , where N ε is given as follows:

otherwise,

Here κ is the condition number of each f i restricted to the convex hull of { x ⋆ , { x k i , x k +1 / 2 i } N ε k =0 } , and O hides the dependence on β 1 and β 2 .

Corollary 4.1 identifies the following two different operational regimes of the algorithm, resulting in difference performance based upon the network connectivity and optimization condition number.

(1) Strong connectivity regime: when r k ≥ r low , for all k , a fact that numerically has been observed for 'relatively good' network connectivity, the convergence rate exhibits a separation in the dependence on the network and optimization parameters. Since 1 -c (1 -λ m ( ˜ W )) > 1 -2 c , it follows that, when c (1 -λ 2 ( ˜ W )) ≥ (1 -2 c ) / √ κ , N ε reduces to O ( κ ) (omitting the dependence on ε ) , which matches the complexity of the centralized gradient algorithm. This suggests scenarios where the optimization problem is harder than a consensus problem over the same network, resulting in the bottleneck between the two. Conversely, when the condition number κ is large relative to the network connectivity 1 -λ 2 ( ˜ W ) , the rate is determined by that of the consensus algorithm running on the same network, that is, O ((1 -λ 2 ( ˜ W )) -1 ) . The above rate separation property mirrors that of certain nonadaptive primal-dual decentralized schemes including NEXT [10], AugDGM [47], Exact Diffusion [49] (with rate expression as improved in [46]), NIDS [26], and ABC [46].

(2) Worst-case regime: This regime reflects the algorithm's worst-case performance, typically registered in 'weakly' connected networks: the convergence rate reads O ( κ/ (1 -λ 2 ( ˜ W ))) , where optimization and network parameters are now mixed. This rate aligns with those of nonadaptive decentralized gradient-tracking schemes, such as DGing [35], SONATA [43] (subject to sufficiently small network connectivity), and [38].

In summary, the convergence rate of Algorithm 1 resembles in the form that of existing nonadaptive decentralized methods, but offers more favorable dependence on the condition number than that typically found in those algorithms. Specifically, the condition number in (13 ) and (14 ) is the local condition number, defined on the convex hull of the trajectory, which is generally much smaller than the global condition number governing decentralized algorithms in the literature. This demonstrates the algorithm's capability to adapt to the local geometry of the optimization problem.

5 From Global to Local Min-Consensus

This section extends Algorithm 1 to settings where the global min-consensus procedure in (S.3) is not implementable. For these cases, we propose to replace such a step with a local min-consensus procedure. The new algorithm is formally described in Algorithm 3 and briefly commented next.

In step (S.3) , each agent now computes its stepsize taking the minimum values among those of their immediate neighbors only (including itself). This produces possibly different stepsizes α k i for each agent, collected in the diagonal matrix Λ k = diag ( α k 1 . . . α k m ) . Because of that, in order to still guarantee D k ∈ span ( I -˜ W ) -a key property for the convergence of the algorithm-we slightly modified the updates of the dual variable in (S.4) , compared with the same step in Algorithm 1. Specifically, the updating direction of the dual variable as in Algorithm 1, ( α k ) -1 ( X k -X k +1 / 2 -α k ∇ F ( X k +1 / 2 )) , is replaced in Algorithm 3 by (Λ k ) -1 X k -X k +1 / 2 Λ - ∇ F ( X k +1 / 2 ) , where X k +1 / 2 Λ = W (Λ k ) -1 X k . Notice that, if all the stepsizes are equal, the update (S.4) in Algorithm 3 reduced to that in Algorithm 1. Finally, we point out that the computation of X k +1 / 2 Λ requires only the extra communication of neighboring stepsizes (thus scalar) values, which has a negligible cost.

Algorithm 3

Data: (i) Initialization X 0 ∈ R m × n and D 0 = 0 ; (ii) initial value α -1 ∈ (0 , ∞ ) ; (iii) Backtracking parameters δ ∈ (0 , 1] ; (iv) nondecreasing sequence { γ k } k ⊆ [1 , ∞ ) (v) Gossip matrix W := (1 -c ) I m + c ˜ W , with ˜ W ∈ W G , and c ∈ (0 , 1 / 2] . Set the iteration index k = 0 .

1: (S.1) Communication step: Agents updates primal and dual variables via gossiping:

2: (S.2) Decentralized line-search : Each agent updates α k i according to

3: (S.3) Local min-consensus :

Define Λ k = diag ( α k 1 . . . α k m ) ;

4: (S.4) Local updates of the primal and dual variables:

5: (S.5) If a termination criterion is not met, k ← k +1 and go to step (S.1) .

Convergence of Algorithm 3 is established in the following theorem.

Theorem 5. Instate assumptions in Theorem 4, applied now to Algorithm 3, with { γ k } being chosen such that γ k ≤ ( ( k + β 1 ) / ( k +1) ) β 2 , for all k and some β 1 ≥ 1 , β 2 > 0 . Further, suppose there exists a constant R > 0 such that V k ≤ R, for all k . Then

with

where N ε is defined as in Corollary 4.1 (replacing therein V 0 with R ).

Interestingly, Theorem 5 states that the degradation of the convergence rate when a local-min consensus is used instead of the global one is mild. Specifically, up to log factors, the total number of iterations to ε -optimality depends on d G (the diameter of the graph G ), if d G > N ε . This result is somehow expected, as min-consensus requires a number of iterations proportional to d G to propagate through the entire network. However, monotonicity in the decrease of the primal and dual errors can no longer be guaranteed when min-consensus is employed.

6 Numerical Results

This section presents some preliminary numerical results. We compare Algorithm 1 and Algorithm 3 with EXTRA [42] and NIDS [26] on a ridge regression problem using synthetic data. Further experiments are presented in the appendix. All experiments are run on Acer Swift 5 SF514-55TA56B6, with processor Intel(R) Core(TM) i5-8250U @ CPU 1.60GHz, 1800 MHz.

Ridge regression: It is an instance of (P), with f i ( x ) = ∥ A i x i -b i ∥ 2 + σ ∥ x i ∥ 2 2 , where we set A i ∈ R 20 × 300 , b i ∈ R 20 , and σ = 0 . 1 .The elements of A i , b i are independently sampled from the standard normal distribution; the regularization is set to σ = 0 . 1 . We simulated a network of m = 20 agents, and the following three different graph topologies, reflecting varying connectivity levels: (i) G 1 : Graph-path with m -1 edges and diameter m -1 , i.e., G = { [ m ] , { ( i, i +1) } m -1 i =1 } ; (ii) G 2 : Erd˝ os-Rényi graph, sparsely connected; and (iii) G 3 : Erd˝ os-Rényi graph, well-connected.

Results are summarized in Fig. 1 and Fig. 2. For EXTRA and NIDS we use a grid-search tuning, chosen to achieve the best practical performance. Algorithm 1 and Algorithm 3 are simulated under the following choice of the line-search parameters satisfying Corollary 4.1: γ k = ( k +2) / ( k +1) , δ = 1 . For all the algorithms we used the Metropolis-Hastings weight matrix W ∈ G W [34].

(b) Erd˝ os-Rényi Graph,

p

= 0

.

1

(c) Erd˝ os-Rényi Graph,

p

= 0

.

5

102

102

102

100

100

100

10-2

×

10-2

10-2

Algorithm3

Algorithm 3

Algorithm 3

10-6

NIDS (fine tuned)

Algorithm 1

10-6

Algorithm 1

NIDS (fine tuned)

10-6

Algorithm1

NIDS (fine tuned)

EXTRA (fine tuned)

EXTRA (fine tuned)

EXTRA (fine tuned)

10-8.

0

2000

#ofIterations

4000

6000

8000

10000

10-8

0

2000

#ofIterations

4000

6000

8000

10000

108

0

2000

#ofIterations

4000

6000

8000

10000

(b) Erd˝ os-Rényi Graph,

p

= 0

.

1

(c) Erd˝ os-Rényi Graph,

p

= 0

.

5

10000

10000

10000

Algorithm 1

Algorithm 1

Algorithm 1

NIDS (fine-tuned)

NIDS (fine-tuned)

NIDS (fine-tuned)

8000

EXTRA (fine-tuned)

8000

EXTRA (fine-tuned)

8000

EXTRA (fine-tuned)

Iterations

Iterations

Iterations

6000

6000

6000

4000

4000

4000

#

#

#

2000

2000

2000

0

0

200

Condition Number

400

600

800

0

0

200

Condition Number

400

600

800

0

200

Condition Number

400

600

800

The figures demonstrate that the proposed method consistently outperforms both EXTRA and NIDS, even when using the local min-consensus strategy, with a significant gap emerging as the condition number increases. This performance is particularly noteworthy given that Algorithm 1 and 3 operate effectively without requiring tedious tuning or global knowledge of the optimization and network parameters. Notably, Algorithms 1 and 3 exhibit different convergence behaviors: as predicted by Theorem 5, local min-consensus results in nonmonotonic error dynamics ∥ X k -X ⋆ ∥ 2 . However, the practical convergence speed remains largely unaffected compared to the global min-consensus.

Acknowledgment

The work of I. Kuruzov has been supported by the Grant No. 70-2021-00143 01.11.2021, IGK 000000D730324P540002.

References

S. A. Alghunaim, E. K. Ryu, K. Yuan, and A. H. Sayed. Decentralized proximal gradient algorithms with linear convergence rates. IEEE Transactions on Automatic Control , 66(6):2787-2794, June 2021.

A. Askhedkar, B. Chaudhari, and M. Zennaro. Hardware and software platforms for low-power wide-area networks , page 397-407. Elsevier, 2020.

J. Barzilai and J. M. Borwein. Two-point step size gradient methods. IMA Journal of Numerical Analysis , 8(1):141-148, 1988.

H. H. Bauschke and P.L. Combettes. Convex Analysis and Monotone Operator Theory in Hilbert Spaces . CMS Books in Mathematics. Springer New York, New York, NY, 2011.

Y. Carmon and O. Hinder. Making sgd parameter-free. In Proceedings of Thirty Fifth Conference on Learning Theory , volume 178 of Proceedings of Machine Learning Research , pages 23602389. PMLR, 02-05 Jul 2022.

T. Chang, M. Hong, H. Wai, X. Zhang, and S. Lu. Distributed learning in the nonconvex world: From batch data to streaming and beyond. IEEE Signal Processing Magazine , 37(3):26-38, 2020.

X. Chen, B. Karimi, W. Zhao, and P. Li. On the convergence of decentralized adaptive gradient methods. In Asian Conference on Machine Learning , pages 217-232. PMLR, 2023.

X. Chen, X. Li, and P. Li. Toward communication efficient adaptive gradient method. In Proceedings of the 2020 ACM-IMS on Foundations of Data Science Conference , page 119-128, Virtual Event USA, October 2020. ACM.

A. Cutkosky and H. Mehta. Momentum improves normalized SGD. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 2260-2268. PMLR, 13-18 Jul 2020.

P. Di Lorenzo and G. Scutari. NEXT: In-network nonconvex optimization. IEEE Trans. Signal Inf. Process. Netw. , 2(2):120-136, June 2016.

J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. In Proceedings of the 24th International Conference on Neural Information Processing Systems , pages 257-265, 2011.

Iyanuoluwa E. and Chinwendu E. Q-linear convergence of distributed optimization with barzilai-borwein step sizes. In 58th Annual Allerton Conference on Communication, Control, and Computing (Allerton) , pages 1-8, 2022.

J. Gao, XW. Liu, YH. Dai, HUang Y., and P. Yang. Achieving geometric convergence for distributed optimization with barzilai-borwein step sizes. Sci. China Inf. Sci. , 65:149-204, 2022.

J. Hu, X. Chen, L. Zheng, L. Zhang, and H. Li. (rectified version) the barzilai-borwein method for distributed optimization over unbalanced directed networks. arXiv:2305.11469v3 , 2024.

M. Ivgi, O. Hinder, and Y. Carmon. DoG is SGD's best friend: A parameter-free dynamic step size schedule. In Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 14465-14499. PMLR, 23-29 Jul 2023.

T. Janssen, N. BniLam, M. Aernouts, R. Berkvens, and M. Weyn. Lora 2.4 ghz communication link and range. Sensors , 20(16):4366, August 2020.

D.H. Kim, J.Y. Lim, and J.D. Kim. Low-power, long-range, high-data transmission using wi-fi and lora. In 2016 6th International Conference on IT Convergence and Security (ICITCS) , page 1-3, Prague, Czech Republic, September 2016. IEEE.

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. CoRR , abs/1412.6980, 2014.

P. Latafat, A. Themelis, and P. Patrinos. Adaptive proximal algorithms for convex optimization under local lipschitz continuity of the gradient. arXiv preprint arXiv:2301.04431 , 2023.

P. Latafat, A. Themelis, and P. Patrinos. On the convergence of adaptive first order methods: proximal gradient and alternating minimization algorithms. arXiv preprint arXiv:2311.18431 , 2023.

J. Li, X. Chen, S. Ma, and M. Hong. Problem-parameter-free decentralized nonconvex stochastic optimization. arXiv preprint arXiv:2402.08821 , 2024.

T. Li and G. Lan. A simple uniformly optimal method without line search for convex optimization. arXiv preprint arXiv:2310.10082 , 2023.

T. Li, A. K. Sahu, A. Talwalkar, and V. Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine , 37(3):50-60, 2020.

X. Li, B. Karimi, and P. Li. On distributed adaptive optimization with gradient compression. In International Conference on Learning Representations (ICLR) , 2022.

X. Li and F. Orabona. On the convergence of stochastic gradient descent with adaptive stepsizes. In Proceedings of the 22 nd International Conference on Artificial Intelligence and Statistics (AISTAT) . PMLR, 2019.

Z. Li, W. Shi, and M. Yan. A decentralized proximal-gradient method with network independent step-sizes and separated convergence rates. IEEE Transactions on Signal Processing , 67(17):4494-4506, 2019.

X. Lian, C. Zhang, H. Zhang, C. Hsieh, W. Zhang, and J. Liu. Can decentralized algorithms outperform centralized algorithms? A case study for decentralized parallel stochastic gradient descent. In Advances in Neural Information Processing Systems , pages 5330-5340, 2017.

P. Di Lorenzo and G. Scutari. NEXT: In-network nonconvex optimization. IEEE Transactions on Signal and Information Processing over Networks , 2(2):120-136, 2016.

L. Luo, Y. Xiong, Y. Liu, and X. Sun. Adaptive gradient methods with dynamic bound of learning rate. In Proceedings of the 7th International Conference on Learning Representations , New Orleans, Louisiana, May 2019.

Y. Malitsky and K. Mishchenko. Adaptive gradient descent without descent. In International Conference on Machine Learning , 2019.

Y. Malitsky and K. Mishchenko. Adaptive proximal gradient method for convex optimization. arXiv preprint arXiv:2308.02261 , 2024.

P. Nazari, D.A. Tarzanagh, and G. Michailidis. Dadam: A consensus-based distributed adaptive gradient method for online optimization. IEEE Transactions on Signal Processing , 70:6065-6079, 2022.

A. Nedic. Distributed gradient methods for convex machine learning problems in networks: Distributed optimization. IEEE Signal Processing Magazine , 37(3):92-101, 2020.

A. Nedi´ c, A. Olshevsky, and M. Rabbat. Network topology and communication-computation tradeoffs in decentralized optimization. Proceedings of the IEEE , 106:953-976, 2018.

A. Nedi´ c, A. Olshevsky, and W. Shi. Achieving geometric convergence for distributed optimization over time-varying graphs. SIAM Journal on Optimization , 27:2597-2633, July 2016.

J. Nocedal and S. Wright. Numerical Optimization . Springer, 2 edition, 2006.

B.T. Polyak. Minimization of unsmooth functionals. USSR Computational Mathematics and Mathematical Physics , 9(3):14-29, 1969.

G. Qu and N. Li. Harnessing smoothness to accelerate distributed optimization. IEEE Transactions on Control of Network Systems , 5(3):1245-1260, Sept 2018.

S. Reddi, Z. Burr Charles, M. Zaheer, Z. Garrett, K. Rush, J. Kone v cn , S. Kumar, and B. McMahan. Adaptive federated optimization. In International Conference on Learning Representations (ICLR) , 2021.

S. J. Reddi, S. Kale, and S. Kumar. On the convergence of adam and beyond. In International Conference on Learning Representations (ICLR) , 2018.

A. H. Sayed. Adaptation, learning, and optimization over networks. Foundations and Trends in Machine Learning , 7:311-801, January 2014.

W. Shi, Q. Ling, G. Wu, and W. Yin. EXTRA: An exact first-order algorithm for decentralized consensus optimization. SIAM J. on Optimization , 25(2):944-966, November 2015.

Y. Sun, G. Scutari, and A. Daneshmand. Distributed optimization based on gradient-tracking revisited: Enhancing convergence rate via surrogation. SIAM J. on Optimization , 32:354-385, 2022.

R. Ward, X. Wu, and L. Bottou. Adagrad stepsizes: Sharp convergence over nonconvex landscapes. The Journal of Machine Learning Research , 21:1-30, 2020.

R. Xin, S. Pu, A. Nedic, and U. A. Khan. A general framework for decentralized optimization with first-order methods. Proceedings of the IEEE , 108(11):1869-1889, November 2020.

J. Xu, Y. Tian, Y. Sun, and G. Scutari. Distributed algorithms for composite optimization: Unified framework and convergence analysis. IEEE Transactions on Signal Processing , 69:3555-3570, 2021.

J. Xu, S. Zhu, Y.-C. Soh, and L. Xie. Augmented distributed gradient methods for multiagent optimization under uncoordinated constant stepsizes. In Proceedings of the 54th IEEE Conference on Decision and Control , pages 2055-2060, 2015.

J. Xu, S. Zhu, Y. Chai Soh, and L. Xie. Convergence of asynchronous distributed gradient methods over stochastic networks. IEEE Trans. Automat. Contr. , 63(2):434-448, 2017.

K. Yuan, B. Ying, X. Zhao, and A. H. Sayed. Exact diffusion for distributed optimization and learning-part i: Algorithm development. IEEE Transactions on Signal Processing , 67(3):708723, 2018.

D. Zhou, S. Ma, and J. Yang. Adabb: Adaptive barzilai-borwein method for convex optimization. arXiv preprint arXiv:2401.08024 , 2024.

Appendix

A Proof of Lemma 3

The proof of the lemma is quite standard, and it is reported here for the sake of completeness.

(i) Smoothness of f implies that Algorithm 2 terminates when α + ≤ δ/L f . Therefore, it must be α + ≥ min( δ/ 2 L f , γα ) . Furthermore, it follows from the strong convexity of f that δ/ (2 α + ) ≥ µ/ 2 ; hence, α + ≤ min( δ/µ, γα ) . This proves (8).

Further, by the lower bound above, one infers that the backtracking procedure terminates when α + ≤ δ L i . Noting that α + = 2 -t +1 γ, we deduce that t = ⌊ log 2 2 L i γα δ ⌋ interations suffice.

(ii) Let ϕ ( α ) := f ( x + αd ) . Notice that ϕ is convex and ϕ ′ (0) = ⟨∇ f ( x ) , d ⟩ . The termination condition in Algorithm 2 can be equivalently rewritten in terms of ϕ as

Given λ ∈ [0 , 1] , let ¯ α = λα + . Invoking convexity of ϕ , we can write

which completes the proof.

B Proof of Theorem 4

We begin establishing the dynamics of V k defined in (10) along two consecutive updates. Lemma 6. The following holds along the update ( D k , X k ) → ( D k +1 , X k +1 ) :

Using the properties of the backtracking procedure (Lemma 3) and leveraging strong convexity and smoothness of F , the inner product in (16) can be bounded as follows.

Lemma 7. The following holds:

where µ k (resp. L k ) is the strong convexity (resp. smoothness) constant of each f i along the segment [ x k +1 / 2 i , x ⋆ ] .

Combining Lemma 6 and Lemma 7, after some algebraic manipulation, we obtain the following. Lemma 8. In the setting above, it holds

Proof. See Appendix E.3.

Lemma 8 suggests the path for the rest of the analysis: the decrease of V k +1 relies on the values of the terms and

relative to the the primal and dual optimality gaps ∥ X k -X ⋆ ∥ 2 and ( α k ) 2 ∥ D k -D ⋆ ∥ 2 M , respectively. At the high-level, one can say that 'higher' values of such quantities relative to ∥ X k -X ⋆ ∥ 2 and ( α k ) 2 ∥ D k -D ⋆ ∥ 2 M , determine larger decrease of V k +1 .

The above argument can be formally recorded by the following quantities:

and

Using r k and g k in (17), the next lemma establishes contraction of V k +1 , with a contraction factor depending in particular on such quantities.

Lemma 9. The following holds

where

and

The final expression (11) in Theorem 4 follows easily from ζ k ≥ δ ( r k ) 2 .

The above result ensures a 'sufficient' descent of V k +1 when r k (or g k ) is large enough. However, the contraction factor in (20) becomes vacuous for arbitrarily small values of r k (or g k ).

Next, we examine the unfavorable case where both r k and g k are 'small', leading to the proof of the decay of V k +1 as stated in (12) of Theorem 4. We build on the following key property of the sequence g k in this scenario: under low r k values, if g k is 'small', the subsequent value g k +1 cannot become arbitrarily small.

Lemma 10. Suppose

Then,

Proof. See Appendix E.5.

We infer from Lemma 10 that

where we used α k /α k +1 ≥ 1 /γ k (due to α k +1 ≤ γ k α k , see Step 1 of Algorithm 2). Notice that (i) the term in the parenthesis will be around one for small enough values of g k and r k ; and (ii) the sequence { γ k } is chosen being eventually uniformly lower bounded. Therefore, the above bound implies that r k and g k cannot both progressively diminish along the iterates. Consequently, in the unfavorable scenario described by (21), V k +1 still decreases, albeit over two consecutive iterations. This outcome is formalized in the following lemma.

Lemma 11. Suppose condition (21) holds. Then,

where

Here, µ k +1 is the strong convexity constants of (each) f i along the segment [ x ( k +1)+1 / 2 i , x ⋆ ] .

The final convergence result as stated in (12) is obtained using Lemma 9 and Lemma 11, with the variable g k , absorbed, as outlined next. We strengthen condition on r k in (21) by r k ≤ √ 2 / 4 . We consider two cases for the value of g k , in the above scenario. Specifically, (Case 1) g k is bounded away from zero, implying ζ k in (20) to be so. Therefore, ( 1 -min ( ρ k 1 , ζ k )) < 1 . This will be sufficient to ensure enough descent for V k +1 , though in two consecutive iterations. (Case 2): g k may be arbitrarily small; in this case, ˆ ρ k 2 in (23) remains bounded away from zero, ensuring contraction though from V k to V k +2 . More formally we have the following.

· Case 1: Consider (20), under (21) strengthened by r k ≤ √ 2 / 4 . If

then

Using lower bound above in (20), yields with

Notice that if the interval of admissible values for g k , as specified by (21) and (24) is empty, this case does not apply.

Case 2: Consider (23) under (21) strengthened by r k ≤ √ 2 / 4 . If

then where we used r k ≤ √ 2 / 4 . This implies

Using this lower bound in (23), yields with

Combining Case 1 and Case 2 above-taking the minimum between (25) and (26) and using the fact that γ k ≥ 1 , leads to the desired decay of V k +1 as in (12).

This completes the proof of Theorem 4.

C Proof of Corollary 4.1

Let us consider the first N > 0 iterations of Algorithm 1. Let us denote by L and µ the constants of smoothness and strong convexity of each f i restricted to the convex hull of { x ⋆ , { x k i , x k +1 / 2 i } N k =0 } . We proceed lower bounding ρ k 1 and ρ k 2 given in Theorem 4. We will use the following facts:

due to Lemma 7, and given λ max ( M ) = ( c (1 -λ 2 ( ˜ W )) -1 -1 ,

We can bound ρ k 1 and ρ k 2 as

Using (27), we can simplify the rate decay of V N in Theorem 4 as follows.

Case 1: Suppose

Substituting the lower bounds of ρ k 1 and r k in (11), we obtain the following simplified convergence rate:

Case 2: Condition (28) does not hold. For the values of k such that r k ≤ 1 √ 2 min ( 1 2 , 1 √ λ max ( M ) ) ≤ √ 2 4 , we can use (12). Substituting threin the lower bound for ρ 2 and γ k = (( k + β 1 ) / ( k +1)) β 2 ≥ β β 2 1 , yields

□

On the other hand, for k such that r k ≥ 1 √ 2 min ( 1 2 , 1 √ λ max ( M ) ) , using (11) on two consecutive iterations, we have

Therefore, in either situations of Case 2, one can ensure contraction after two consecutive iterations by a factor

Using V k +1 ≤ ( γ k ) 2 V k , we can merge (29) and (30) as follows:

Case 1 + Case 2: We can combine the rate expressions derived in the two cases above as follows:

where

Notice that ρ ∈ (0 , 1) .

Finally, we can obtain the desired asymptotic convergent rate noting that the growth of ∏ k γ k is dominated by the geometric decay of the contraction factor. This is formalized next.

Lemma 12. Let γ k = (( k + β 1 ) / ( k +1)) β 2 with β 1 ≥ 1 , β 2 ≥ 0 . Then the following holds:

Furthermore, for any given ρ ∈ (0 , 1) , we have

for all

Inequality (32) provides the asymptotic rate expression, as stated in the corollary where O hides the dependence on β 1 and β 2 . □

D Proof of Theorem 5

We begin noticing that if the stepsizes in Algorithm 3 are identical across agents, Algorithm 3 reduces to Algorithm 1. For the iterates where this happens, one can rely on the convergence guarantees established for Algorithm 1. Specifically, we have the following result, whose proof is straightforward.

Lemma 13. Suppose that exists some k ε ≥ 1 such that α k 1 = · · · = α k m , for k = k ε , . . . , k ε + N ε . Then one can guarantee

where N ε is defined as in Corollary 4.1 (replacing therein V 0 with V K ε ).

The remainder of the proof focuses on characterizing the properties of certain key events identified as detrimental for the local-min consensus algorithm to achieve convergence. We will demonstrate that the occurrence of such events within N consecutive iterations is of the order of log N , indicating that these are sporadic events relative to the total of N iterations.

Given α k i 's, as defined in Step (S.3) of Algorithm 3, let us denote their minimum across all agents at time k as

If the backtracking loop (steps 3-6 in Algorithm 2) is not activated at iteration k -1 in any of the agents' local line searches, then all output stepsizes α k i will increase by the same factor γ k , that is, α k i = γ k α k -1 i ; hence, does α k min . On the other hand, if all stepsizes are consensual at iteration k -1 and the backtracking procedure at some of the agents' side enters its steps- 3-6, a 'desynchronization' of the stepsizes occurs. This event can be detected by the condition

When the stepsize at time k -1 are not consensual, the condition above identifies increases in stepsize disagreements from iteration k -1 to k , measured by

This motivates the definition of the following index set: given N = 1 , 2 , . . . , let

The key properties of interest of this set are summarized below.

Lemma 14. For any given N = 1 , 2 , . . . , the following statements hold:

If α k i = α k min , for all i ∈ [ m ] , and k +1 / ∈ I N , then α k +1 i = α k +1 min , for all i ∈ [ m ] ;

If k ∈ I N , k < N -d G , and k +1 , . . . k + d G / ∈ I N , then α k + d G i = α k + d G min , for all i ∈ [ m ] ;

In words, the first statement confirms that consensus on the stepsizes is maintained if none of the local backtracking procedures are triggered. The second statement ensures that, the local-min consensus algorithm requires at most d G iterations to converge, from any initialization, provided that during those iterations no backtracking events alter the minimum stepsize across agents. Lastly, the third

assertion provides a limit on the maximum number of detrimental events that can occur during the N iterations under consideration. If this number is small relative to N , a fact that will be proved shortly, one we can find (multiple) window(s) of consecutive iterations wherein the stepsizes remain consensual across all agents. Within these windows, Lemma 13 can be applied, to establish convergence. This idea is formalized next.

Lemma 15. Suppose V k ≤ R. Then,

Here, N ε is defined as in Corollary 4.1 (replacing therein V 0 with R ).

To finalize our proof, let us simplify an upper bound of |I N | , when γ k = ( k + β 1 k +1 ) β 2 . For the sake of simplicity, we consider the case ln N ≥ 1 . Invoking Lemma 12, we have

where the constant ξ depends on the algorithm parameters. Therefore, one can guarantee (33), under the following condition

if

A sufficient condition for (35) is

Let N ∗ be the smallest iteration for which the above inequality holds. Then,

E Proof of the Intermediate Results in Appendix B

E.1 Proof of Lemma 6

Let us rewrite V k +1 in terms of ∥ X k -X ⋆ ∥ 2 and ( α k ) 2 ∥ D k -D ⋆ ∥ 2 M , to link it back to V k :

where the equality follows from ∥ a ∥ 2 = ∥ b ∥ 2 -∥ a -b ∥ 2 -2 ⟨ a, b -a ⟩ .

Notice that the negative terms on the RHS of (36) will contribute to the decrease of V k +1 . We are thus left to deal with term I and term II . The idea is to bound them so that they can overall being controlled by the backtracking inequality.

Let us proceed bounding term I and term II using the algorithm dynamics. We have the following:

where in the second equality we used ∇ F ( X ⋆ ) = -D ⋆ . Note that the last term in the expression above can be controlled through the backtracking procedure. Hence, we proceed bounding term II to 'cancel' out the other terms on the RHS of (37). Specifically,

where we used the update of X k +1 and the facts D k +1 -D ⋆ ∈ span ( I -˜ W ) and X ⋆ ∈ null ( I -˜ W ) . Summing up (37) and (38), we obtain

The statement of the Lemma follows readily substituting (39) in (36).

E.2 Proof of Lemma 7

We preliminary notice that, in view of Lemma 3, the backtracking inequality (6) on F k holds with α k = min i ∈ [ m ] α k i , where each α k i is the outcome of the backtracking procedure on the local f k i . Since F and F k have the same curvature, it follows that (6) holds also on F , that is,

We proceed now bounding 〈 ∇ F ( X k +1 / 2 ) -∇ F ( X ⋆ ) , X ⋆ -X k +1 〉 building on (40). To do so, we decompose the inner product as follows

□

We bound term I invoking strong convexity and co-coercivity of F while we use (40) to bound term II . Specifically,

Using (40), term II can be bounded as

Using (42) and (43) in (41), yields

This completes the proof.

E.3 Proof of Lemma 8

Combining Lemma 6 and Lemma 7, we can write

Next, we demonstrate that the sum of the last three terms contributes to the decrease of V k +1 . Using the definition of X k +1 and X k +1 / 2 , we can bound term III as follows:

Proceeding with ∥ D k -D k +1 ∥ 2 M , we have

□

where the second equality follows from the definition of M = c -1 ( I -˜ W ) † -I. Combining the three terms yields

which proves the statement of the lemma.

E.4 Proof of Lemma 9

The proof involves further bounding the RHS of (17) in Lemma 8, appropriately in terms of r k and g k . To achieve this, we construct two alternative bounds of the RHS of (17), as discussed below.

The first bound will reveal the dependence on r k , and it is based on using in (17)

We obtain

The second bound of the RHS of (17) aims to obtain an explicit dependence on g k . This is done by just neglecting the last two negative terms in the RHS of (17):

□

The final result follows combining the above two bounds while using the definition of ρ k 1 and ζ k . □

E.5 Proof of Lemma 10

Invoking the update of the primal variable in the form

where we used D ⋆ + ∇ F ( X ⋆ ) = 0 , and the definition of g k , we can write

Let us proceed lower bounding ∥ D k +1 -D ⋆ ∥ M . Using the update of the dual variable, in the form

we obtain

Using

the following holds for the last two terms on the RHS of (46):

where the last equality follows from g k ≤ 1 (as postulated in (21)). Finally, using √ 2 √ a 2 + b 2 ≥ a + b , we deduce

Substituting the above inequality in (45) yields the desired result.

□

E.6 Proof of Lemma 11

Using (22) and (44) in (17) (while neglecting therein the last two negative terms on the RHS), yields

where the last inequality follows from α k /α k +1 ≥ 1 /γ k . We deduce

The final statement of the lemma follows from the above inequality and

due to (20) and ρ k 1 < 1 .

E.7 Proof of Lemma 12

Let us consider the case N ≥ ⌈ β 1 ⌉ +2 . Using γ k = ( k + β 1 k +1 ) β 2 , we can bound the product of γ k 's as

Let us determine now N such that (32) holds. Condition (32) is met if the following inequality holds which proves (31). Notice that the above bound holds also if N ≤ ⌈ β 1 ⌉ +2 . 0

□

where we used that 1 -ρ ≤ (1 -ρ/ 2) 2 for ρ ∈ (0 , 1) . Bounding the LHS yields

It follows that (32) holds if and

A sufficient condition for the last inequality to hold is

This completes the proof.

F Proof of the Intermediate Results in Appendix D

F.1 Proof of Lemma 14

This assertion comes readily from the definition of I N and the backtracking procedure.

Let N i ( k ) be the set of neighbors of agent i that are at most k ≥ 1 hops away from agent i , including agent i itself. For notational consistency, N i (1) = N i ∪ { i } . Using N i ( k ) , we can rewrite the local-min consensus step of each agent i at iteration k as

where α k j is the stepsizes produced by the line-search of agent j at iteration k .

Let k > k be an iteration such that k +1 , . . . , k / ∈ I N . It must be

In particular, this implies which leads to

Finally, taking k = k + d G and noting N i ( d G ) = [ m ] , we proved the second statement of the lemma.

From the backtracking line-search and the definition of I N it follows that

Applying the above relation iteratively, yields

□

At the same time, it follows from Lemma 3 that

Combining the lower and upper bounds above, yields the desired result

F.2 Proof of Lemma 15

From (34), it follows

Then, according to the Dirichlet's principle there exists two iteration indices k 1 and k 2 such that

∀ k ∈ [ k 1 , k 2 ] ⇒ k / ∈ I N ; and 2. k 2 -k 1 ≥ N ε + d G .

Invoking Lemma 14.(1) and Lemma 14.(2), it follows that all agents' stepsizes reach consensus after k 1 + d G iterations and remain consensual for the subsequent N ε iterations. One can then invoke Lemma 13, and conclude

This concludes the proof.

G Additional Numerical Results

This section presents additional experiments for the Ridge Regression problem, introduced in Section 6. Here, we consider additional graph topologies, namely: 1) Ring Graphs; 2) Random Regular Graphs with degree 3; and 3) Random Regular Graphs with degree 10. The rest of the setup (including algorithms' tuning) is the same of that described in Section 6.

The experiments are summarized in Fig. 3. The findings corroborate the conclusions presented in

(b) Regular graph, degree 3

(c) Regular graph, degree 10

102

102

101

101

100

100

×

10-1

=10-1

=

10-2

10-2

1

1

10-3

Xll

10-3

10-4

Algorithm 3

Algorithm3

Algorithm 3

10-4

Algorithm1

Algorithm 1

Algorithm 1

10-5

EXTRA (fine tuned)

NIDS (fine tuned)

10-5

EXTRA (fine tuned)

NIDS (fine tuned)

10-6

EXTRA (fine tuned)

NIDS (fine tuned)

0

2000

#of Iterations

4000

6000

8000

10000

Q

2000

#of Iterations

4000

6000

8000

10000

0

2000

#of Iterations

4000

6000

8000

10000

Sec. 6: both Algorithm 1 and Algorithm 3 outperform EXTRA and NIDS, which were finely tuned for rapid practical convergence. Quite interestingly, the performance of the proposed methods appears to be less affected by network topology and depends primarily on network connectivity.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Abstract gives accurate presentation of our result. Part Major contributions of Introduction contains full description of our work.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer:[Yes]

Justification: The main limitaion of proposed procedure is min-consensus. The technology for its implementation is carefully discribed in part 3.1.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Main assumptions and definitions are presented in Section 2. All main theoretical results presented in Section 4 with all required assumptions. Proofs are placed in Appendix A-F because of their large size.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All setup for numerical experiments are described in Section 5. It is enough to reproduce all experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: code in the form of an attached archive.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Our paper demonstrates performance of optimization algorithm. Because of that, we do not need test some models. But Section 5 contains full information about our experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Numerical experiments demonstrate performance of optimization algorithm on a given problems. Besides, our algorithm is deterministic.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Information is given at the end of Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Authors are familiar with NeurIPS Code of Ethics and paper conform it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There are different methods of distributed optimization. The paper propose new method of distributed optimization that has no additional societal impact as the authors think.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The proposed method does not require safeguard.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Numerical experiments use one of datasets from LIBSVM. Authors cite corresponding work of owners (see reference [6] in Section 5 and References)

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification:contains contains README file with sufficient description.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Paper does not involve crowdsourcing nor research with human subjects

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper introduced a new parameter-free algorithm based on forward-backward splitting technique and variable metric for decentralized learning problems for convex locally smooth functions. Convergence guarantee with favorable rate and analysis are provided.

STRENGTHS:
1. The paper proposed the first parameter-free decentralized training algorithm integrating line search and splitting technique.
2. Convergence guarantee and analysis under milder conditions than previous studies are provided.
3. The guidelines of design seems to be of independent interest and further investigation.

WEAKNESSES:
1. The proposed algorithm is complicated, and it seems that in each iteration, a lot of computation needs to be carried out. How does the algorithm compares to the previous ones in terms of computational complexity?
2. The proposed algorithm needs more experimental evaluation.
3. In certain parts, the paper is a bit hard to follow.

QUESTIONS:
See weaknesses.

LIMITATIONS:
No.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-06-23

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a parameter-free method for decentralized learning and showed that the method converges to the optimal solution linearly without hyperparameter tuning.\n\nSTRENGTHS:\nThis paper proposed a novel decentralized method and the convergence rate is analyzed under the general setting.\n\nWEAKNESSES:\n1. Theorem 4 provided the convergence rate of the proposed method without hyperparameter tuning, but Corollary 4.1 provides the results with some $\\beta_1$ and $\\beta_2$. Thus, the reviewer thinks that the results in Corollary 4.1 require hyperparameter tuning. The author discussed the convergence rate in Corollary 4.1, but this result is not the convergence rate of the parameter-free method. The reviewer thinks that it is necessary to discuss the results in Theorem 4 and compare these results with one of the existing methods.\n2. The reviewer does not understand what the author would like to conclude from Theorem 5 (in the weakly convex setting). Theorem 5 only shows that the amount of parameter updates (i.e., $\\| X^j - X^{j+1} \\|^2$) is bounded from above by some factor and does not show that the proposed method converges to the optimal solution.\n3. The author did not tune the stepsize for EXTRA and NIDS in Fig. 1 and did not mention how to tune the stepsize in Fig. 2. The author claimed that the proposed method consistently outperforms EXTRA and NIDS in Figs 1 and 2, but these results are not convincing because the results of NIDS and EXTRA are suboptimal, at least in Fig. 1. The reviewer would like to see the results of EXTRA and NIDS with well-tuned stepsize and see the comparison between the proposed method and the prior methods. \n4. The line 3 in Algorithm 3 can not be calculated in a decentralized manner.\n\nQUESTIONS:\nSee the above comments.\n\nLIMITATIONS:\nSee the above comments.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-10"}

comments: SUMMARY:
This paper studies adaptive parameter determination in decentralized optimization. It is a meaningful and interesting topic to investigate. They propose a decentralized method to solve consensus optimization, develop an adaptive parameter strategy, and show the linear convergence of their algorithm.

STRENGTHS:
The main strength of the paper lies in the high value of the topic they studied, which is indeed a very meaningful and also a difficult topic to explore.

WEAKNESSES:
I was very excited when reading the abstract of this paper, because I know the difficulty of this topic. I was worried about the global min-consensus step, while the authors developed a local min-consensus strategy to replace it, which well addressed my concerns.

QUESTIONS:
No

LIMITATIONS:
No
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 1
DATE: 2024-07-11

comments: SUMMARY:
This paper introduces a new algorithm for decentralized optimization.  The main advantage over previous work in this domain is that it allows adaptive stepsize selection (via backtracking) that is independent of the properties of the functions being minimized.  The analysis of the algorithm recovers the linear rate for convergence for functions that are smooth and strongly convex, and the standard sublinear rate for the convex case.

STRENGTHS:
As the authors state, this seems to be the first fully decentralized algorithm with adaptive stepsize selection that has strong convergence guarantees.  I found the operator splitting method for solving the KKT conditions innovative, and it is nice that this leads naturally to a way to do local stepsize adaptation.

WEAKNESSES:
While the numerical experiments are sufficient, new results on ridge and logistic regression can only get one so excited.  

The exposition of the saddle point reformulation at the beginning of Section 3 could be improved.
Line 142: Does the optimization in (P') involve y at this point?  Should x be capitalized?
Line 143: Some discussion about why K is being introduced here would help the exposition a lot, along with a little more on why the solutions of (P) and (P') match.  A lot of work is being left to the reader at this critical point.

QUESTIONS:
It is outside the scope of this submission, but I would be interested to know if this type of adaptive stepsize selection made a difference in practice in the stochastic setting.

LIMITATIONS:
None.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-12