Title: Implicit Regularization of Decentralized Gradient Descent for Sparse Regression
Venue: NeurIPS 2024
Accepted: yes
Keywords: Decentralized optimization

Implicit regularization

Sparse regression
Abstract: We consider learning a sparse model from linear measurements taken by a network of agents. Different from existing decentralized methods designed based on the LASSO regression with explicit  $\ell_1$ norm regularization, we exploit the implicit regularization of decentralized optimization method applied to an over-parameterized nonconvex least squares formulation without penalization. Our first result shows that despite nonconvexity, if the network connectivity is good, the well-known decentralized gradient descent algorithm (DGD) with small initialization and early stopping can compute the statistically optimal solution. Sufficient conditions on the initialization scale,  choice of step size, network connectivity, and stopping time are further provided to achieve convergence. Our result recovers the convergence rate of gradient descent in the centralized setting, showing its tightness. 
Based on the analysis of DGD, we further propose a communication-efficient version, termed T-DGD, by truncating the iterates before transmission. In the high signal-to-noise ratio (SNR) regime, we show that T-DGD achieves comparable statistical accuracy to DGD, while the communication cost is logarithmic in the number of parameters. Numerical results are provided to validate the effectiveness of DGD and T-DGD for sparse learning through implicit regularization.

Implicit Regularization of Decentralized Gradient Descent for Decentralized Sparse Regression

Abstract

We consider learning a sparse model from linear measurements taken by a network of agents. Different from existing decentralized methods designed based on the LASSO regression with explicit ℓ 1 norm regularization, we exploit the implicit regularization of the decentralized optimization method applied to an overparameterized nonconvex least squares formulation without sparse penalization. Our first result shows that despite nonconvexity, if the network connectivity is good, the well-known decentralized gradient descent algorithm (DGD) with small initialization and early stopping can compute the statistically optimal solution. Sufficient conditions on the initialization scale, choice of step size, network connectivity, and stopping time are further provided to achieve convergence. Our result recovers the convergence rate of gradient descent in the centralized setting, showing its tightness. Based on the analysis of DGD, we further propose a communication-efficient version, termed T-DGD, by truncating the iterates before transmission. In the high signal-to-noise ratio (SNR) regime, we show that T-DGD achieves comparable statistical accuracy to DGD, while the communication cost is logarithmic in the number of parameters. Numerical results are provided to validate the effectiveness of DGD and T-DGD for sparse learning through implicit regularization.

1 Introduction

Modern deep learning is generally in the over-parameterized regime where the models have significantly more parameters than available training examples [6, 41]. Although deep learning models exhibit remarkable performance in multiple domains, the theoretical understanding of optimization and generalization for deep learning is still limited. Recent studies show that despite being overparameterized, gradient-based methods applied to minimize the emperical loss exhibit the implicit regularization phenomenon. For example, a line of works [3, 5, 11, 16] shows that with certain initialization, networks trained with gradient descent (GD) land in the 'kernel regime' and share similar behaviors to the kernel method. However, the literature [2, 9, 24] suggests that kernel regime analyses fall short in explaining the success of deep learning because neural networks analyzed in the kernel regime are almost linearized, thus hindering feature learning from data. Further, many works [35, 38, 40] start investigating the 'rich regime', showing that GD with small initialization induces structures on the solution, such as sparsity and low-rankness, that better explains the generalization capability of NNs. However, all aforementioned results are limited to the centralized setting, where data are stored on a single machine. Practical constraints such as limited computing and storage resources, data privacy and security, and regulation rules make the centralized learning framework increasingly inadequate for contemporary applications. Although a variety of decentralized learning algorithms can be applied to NN training, the questions of which solution they can converge, along with its generalization performance, are largely unclear.

In this paper, we study the sparse learning problem [34] in the overparameterized regime, which shares many key characteristics with deep learning models but is more tractable to analyze, as a prototype for understanding the computation and statistical guarantees of decentralized learning algorithms. Specifically, we consider learning a sparse model w ⋆ from its noisy linear measurements over m agents. These agents communicate over an undirected connected mesh network without a central coordinator, and each agent can only communicate with its one-hop neighbors. The i -th agent has its own n samples { ( x i,j , y i,j ) } n j =1 . Each j -th data pair ( x i,j , y i,j ) is generated according to the noisy linear model

where x i,j ∈ R d and y i,j ∈ R denotes respectively the j -th feature and its corresponding response at i -th agent, ξ i,j is the observation noise, and w ⋆ ∈ R d is the sparse model parameter to be learned, common to all agents, and has only s ( s ≪ d ) non-zero elements. We are interested in the highdimensional setting where the ambient dimension d is substantially larger than the total sample size N := mn , i.e., d ≫ N . By re-parameterizing w = u ⊙ u -v ⊙ v , the loss function can be formulated as minimizing the following regularization-free nonlinear least square problem:

where f i ( u , v ) corresponds to the loss function of i -th agent. Problem (2) can also regarded as the supervised learning problem on the diagonal linear network of degree-2 [38].

̸

Problem (2) is highly non-convex with u and v , however, recent works [12, 36, 42] demonstrate that if the design matrix satisfies the restricted isometric property (RIP) condition, the centralized GD without any regularization can yield the statistically optimal estimator with properly chosen initialization scale, step size, and early stopping time. This intriguing phenomenon is derived from the implicit regularization of GD. Roughly speaking, with small initialization, the gap | u 2 i -v 2 i | would increase with iteration for coordinate i such that w ⋆ i = 0 , where the remaining ones stay small enough before early stopping. As a result, GD identifies the support of w ⋆ as the algorithm processes. In the decentralized setting, general results from the pure optimization perspective can only certify convergence to the stationary points, implying neither global optimality nor generalizability. Given the encouraging result of GD achieved in the centralized setting, it is natural to ask if statistically optimal solutions are also computable by decentralized gradient-type algorithms, which algorithm can achieve the goal, and what are the regularity conditions.

This paper aims to analyse the renowned decentralized gradient algorithm (DGD) for minimizing (2) over the undirected mesh networks. The main contributions of this paper are detailed as follows.

Statistical guarantee. It is well established that even for convex objectives, DGD cannot compute an exact minimizer. It only converges to the neighborhood of the solution whose radius depends on the step size. However, we show that under specific conditions-namely, if the global design matrix satisfies the RIP condition, the initialization scale is sufficiently small, and the network is sufficiently connected-the solution computed by DGD with early stopping is statistically optimal.

Computational complexity. Our convergence analysis reveals that the early stopping time increases logarithmically with the ambient dimension d . While network connectivity does not affect statistical error when it satisfies mild conditions, it does influence the stopping time of DGD to find the optimal estimator. Networks with poor connectivity will delay the early stopping time, and thus increase the iteration complexity.

Technical analysis. Compared to the techniques used for analyzing the centralized GD [33, 38], proving the convergence of DGD faces the following challenge. Because the consensus error terms induced from the mesh network result in a perturbed version of the multiplicative update. Compared with the exact multiplicative updates, the challenge is that the additional error term outside of multiplication prevents applying the centralized analysis directly. In addition, the error terms within the multiplication have more complicated consensus error terms than that of the centralized setting, which requires bounding the consensus error terms carefully to control these error terms that can achieve the same order statistical error. To achieve this goal, we separately control the consensus errors on support S and non-support S c by the magnitudes of parameters on support S and non-support S c , respectively. Our fine-grained analysis for consensus errors is distinct

from existing decentralized optimization analyses that bound the consensus errors uniformly. The additional error term also complicates the transfer of proof from the simplified non-negative w ⋆ case to the general w ⋆ setting as the centralized setting, we conduct a comprehensive induction process to both u and v simultaneously for general w ⋆ .

Truncated DGD. We propose a communication-efficient truncated DGD (T-DGD) method that at each iteration, vectors being transmitted are truncated, keeping only s elements with the largest magnitudes nonzero. We prove that if each agent has sufficient samples and the signal-to-noise ratio is high enough, T-DGD can perform as well as the vanilla DGD while reducing communication complexity to logarithmic dependence on ambient dimension d .

2 Related works

We categorize the existing works most relevant to our study into three main groups.

Implicit regularizations for sparse regression. The recent study in [15] reparameterized the model parameter through overparameterized Hadamard product and discovered encouraged empirical performance by the first-order optimization algorithms. The statistical and convergence guarantees for this phenomenon are established in [36, 42] under mild conditions. Woodworth et al. [38] studied the impact of initialization scale on solutions. Scott et al.[27] demonstrated the benefit of stochasticity of SGD in sparse regression and explored the impact of momentum in [25]. The more recent process in understanding the linear diagonal networks can be found in references [7, 10, 12, 19, 23, 26, 43]. To the best of our knowledge, existing works have only discussed implicit regularizations induced by centralized optimization methods in linear diagonal networks. However, the question of whether decentralized algorithms induce implicit regularizations, and what type of implicit regularization they may induce has not been studied so far.

Decentralized sparse regression with explicit regularization. For estimating ground truth w ⋆ in high-dimensional sparse linear regression under the decentralized setting, Ji et al. [18] proposed DGD-CTA algorithm for tackling LASSO objective with consensus penalty and proved linear convergence rate to the neighbor of the statistical optimal estimator, but the convergence rate has polynomial dependence on ambient dimension d . Further Sun et al. [31] proposed the NetLASSO based on the gradient tracking method and obtained d -independent convergence rate and optimal statistical accuracy. To complement work [18], Ji et al. proposed DGD-ATC by mixing the local gradients along iterations and achieved logarithmic dependence on d in [17]. Maros et al. [21] proposed DGD 2 method based on a double mixing for solving decentralized LASSO and obtained similar theoretical guarantees in [17, 31]. To improve the computation efficiency, Maros et al. [20] integrated accelerated proximal gradient descent with gradient tracking to solve decentralized LASSO. Despite these developments, it remains unclear whether leveraging the unregularized overparameterization and implicit regularization of decentralized optimization methods can achieve the optimal statistical guarantee over mesh networks.

Implicit regularizations of decentralized optimization. Implicit bias or regularizations of centralized optimization methods for overparameterized models have been extensively studied [13, 30], but only a few works have investigated the implicit regularization of decentralized optimization methods. Richards et al. [29] studied the implicit regularization for decentralized stochastic gradient descent for solving general unregularized convex problems. Zhu et al. [44] demonstrated that decentralized stochastic gradient descent implicitly executes the sharpness-aware minimization algorithm for general non-convex problems. Taheri et al. [33] studied the implicit regularization of DGD in overparameterized classification for separable data. Recent work [22] demonstrated the implicit regularization of the DGD 2 in solving the overparameterized matrix sensing problem. Different from these works, we establish the statistical and computational results for specific non-convex sparse regression problem.

3 Preliminaries

In this section, we will introduce the basic notations used in this paper, and then formulate the DGD for solving the problem (2). Finally, we provide the necessary assumptions and definitions for the decentralized sparse regression problem.

3.1 Notations

Throughout this paper, we use [ m ] to denote set { 1 , · · · , m } for given positive integer m , 1 d denotes d -dimensional vector that all elements are one and I d denotes d -dimensional identity matrix. For ground truth parameter w ⋆ , the relevant notations are support set S := { j | w ⋆ j = 0 } , positive support set S + := { j | w ⋆ j > 0 } , negative support set S -:= { j | w ⋆ j < 0 } and non-support set S c := { j | w ⋆ j = 0 } , w ⋆ max := max j ∈ S | w ⋆ j | and w ⋆ min := min j ∈ S | w ⋆ j | . ∀ x ∈ R d , x S := 1 S ⊙ x where 1 S denotes a vector equal to one for all coordinates j ∈ S and equal to zero everywhere else. Symbol ' ⊙ ' denotes Hadamard product that ( a ⊙ b ) j = a j b j , ∀ a , b ∈ R d . The averaged signal is defined as w t := 1 m ∑ m i =1 w t,i and similar notations can be extended to u t , v t .

̸

X := [ X 1 ; · · · ; X m ] denotes the concatenated sample matrix, where each row of X i represents one feature vector in agent i . ∥·∥ denotes the Frobenius norm for vector and the spectral norm (maximum singular value) for matrix. ∥ A ∥ ∞ := max i,j | A ij | denotes infinity norm. We use a = O ( b ) to denote that inequality a ≤ Cb holds with some absolute constants C that do not depend on any parameters of the problem. The notation a ≲ b shares the same meaning as a = O ( b ) . Finally, we use a ≳ b if there exists a universal constant c such that a ≥ cb .

3.2 Method and Assumptions

We focus on DGD solving problem (2) over mesh network, modeled as an undirected graph G = {V , E} where nodes V = { 1 , · · · , m } represent the set of agents and edges E ⊂ V × V represent the communication links. An unordered pair { i, j } is included in E if and only if there is a bidirectional communication link between agent i and j . The set of one-hop neighbors for agent i is denoted by N i := { j ∈ V| ( i, j ) ∈ E} ⋃ { i } .

DGD allows each agent to independently update its parameters based on local gradient descent and then synchronize with neighboring agents by weighted averaging these updates. The recursive iteration of DGD for each agent is described as follows.

where the w t,i := u t,i ⊙ u t,i -v t,i ⊙ v t,i denotes the local estimator in agent i at t th iteration, the initialization is u 0 ,i = v 0 ,i = α 1 d , ∀ i ∈ [ m ] and η is constant step size. W is the nonnegative weight mixing matrix for the undirected mesh network, where W ij > 0 if there is a link between agents i and j , and W ij = 0 otherwise. The mixing matrix W related to the undirected graph satisfies the following assumption.

Assumption 1. The communication network G is connected. The weight matrix W = [ w ij ] m i,j =1 for this graph has the following properties: (i) w ij = 0 for all pairs ( i, j ) that are not in E ; (ii) it is double stochastic that 1 T m W = 1 T m and W 1 m = 1 m ; (iii) the spectral gap ρ := ∥ ∥ W -1 m 1 m 1 T m ∥ ∥ ≤ 1 .

This assumption is common in decentralized optimization literature [17, 22]. We need the following RIP condition which is a key condition to obtain the optimal estimator for sparse regression.

Definition 1. The global design matrix X / N ∈ R N × d satisfies the ( δ, s ) -Restricted Isometry Property (RIP) if for any s -sparse vector w ∈ R d , there is (1 -δ ) ∥ w ∥ 2 ≤ ∥ ∥ ∥ Xw / √ N ∥ ∥ ∥ 2 ≤ (1 + δ ) ∥ w ∥ 2 .

√

The RIP condition was first introduced in the compressed sensing literature in [8] which is a little more restrictive condition to achieve optimal statistical rate than the restricted eigenvalue condition in [1]. We inherit this assumption in the centralized setting [36, 42] to achieve optimal estimator error under the condition that parameter δ is upper bounded. Besides the global RIP condition, we have the following local RIP condition for local design matrices { X i } m i =1 .

Definition 2. The local design matrices X 1 / √ n, · · · , X m / √ n ∈ R n × d satisfy the local ( δ max , s ) -(RIP) condition, if for any s -sparse vector w ∈ R d and any local design matrix X i / √ n , there is (1 -δ max ) ∥ w ∥ 2 ≤ ∥ X i w / √ n ∥ 2 ≤ (1 + δ max ) ∥ w ∥ 2 .

The definition of the local RIP condition is just for ease of proof presentation, as we do not necessitate any upper bound on the local RIP parameter δ max .

4 Main Result

Based on the above method and assumptions, we now give theoretical guarantees of DGD in solving problem (2) for sparse regression problem (1) as follows.

Theorem 1. Considering the sequence generated by (3) and (4) based on DGD for solving problem (2) and ∀ ϵ > 0 , if the global design matrix X / √ N satisfies ( δ, s + 1) -RIP condition with bounded RIP parameter δ ≲ 1 √ s , the local design matrices { X i / √ n } m i =1 satisfy local ( δ max , s +1) -RIP condition, and the mesh network satisfies assumption 1, the initialization satisfies α ≲ min { 1 , ϵ 2 (12 d +1) 2 , ϵ w ⋆ max , ζ 6( w ⋆ max ) 2 , w ⋆ min 4 } , the constant stepsize η satisfies

and the spectral gap ρ satisfies

then after running t = O ( 1 ηζ log 1 α ) iterations. There would be

where ς := max {∥ ∥ ∥ X T ξ N ∥ ∥ ∥ ∞ , ϵ } , ζ := max { w ⋆ min 5 , 960 ς } .

Mechanism to promote sparsity . The consensus errors induced from decentralized network complicate the multiplicative updates, which becomes inexact multiplicative updates as u t +1 = u t ⊙ ( 1 -4 η ( u t ⊙ u t -w ⋆ + ˆ p t + ˆ b t )) + e t . Compared with the exact multiplicative updates of GD in [36], the challenge is that the extra error term e t outside of the multiplication prevents applying the centralized analysis trivially. In addition, the perturbation error terms ˆ p t , ˆ b t within the multiplication are much more complicated than that of the centralized setting due to additional multiple consensus errors. This requires bounding the consensus error terms carefully, which should control the complicated perturbation errors ˆ p t , ˆ b t , e t not to be large. Thus, we can use network connectivity to control the consensus errors to bound these three perturbation errors small enough to make the distance between two trajectories obtained by inexact and exact multiplicative updates within statistical accuracy, which can promote sparsity in the decentralized setting. The detailed theoretical mechanism of promoting sparsity has been demystified in Proposition 3.

Statistical Guarantee . Based on the result in (7) and conditions in (5) and (6), we can observe that if the initialization α is small enough and network connectivity is sufficiently well, the DGD with early stopping can obtain the desired estimator for sparse ground truth parameter w ⋆ that achieves the same order of statistical error as the centralized setting in [36]. The formula in (7) not only

illustrates that we establish the network-independent estimator error bound but also inherits the benefit of implicit regularization, which indicates that if the signal-to-noise is high enough, the statistical error is independent of ambient dimension d . In contrast, existing results in decentralized LASSO methods [17, 18, 21, 31], have consistent dependence on d in any case.

Computational Complexity. The iteration complexity of early stopping is network-dependent that is because the t = O ( 1 ηζ log 1 α ) has the dependence on stepsize, which should satisfy the condition in (5). This suggests that poorer network connectivity leads to higher computational complexity. Although the initialization has no dependence on network connectivity, 1 α has polynomial dependence on d , and the dependence of complexity on d is just logarithmic, which is similar to DGD in solving LASSO in [17, 21] and improves the polynomial dependence on d in [18].

Dependence on network connectivity. For accurate estimation, it is essential that the network should be well-connected, as specified in condition (6). When this condition is not satisfied, we can run multiple rounds of communication per iteration. It is observable that the smaller ratio between the global RIP parameter δ and the local RIP parameter δ max , and smaller ratio between the local noise and the global noise magnitude, necessitate a higher degree of network connectivity. This can be understood from the perspective of heterogeneity, where smaller ratios indicate a significant disparity between local and global design matrices. Consequently, condition (6) is reasonable as it suggests that higher levels of heterogeneity necessitate improved network connectivity. In numerical experiments, we can observe that if ρ does not satisfy the condition as (6), obtaining optimal statistical error is not achievable, which indicates the optimal statistical error undergoes a phase transition with the network connectivity.

Our results demonstrate the benefit of overparameterization for DGD. Theorem 1 shows that standard DGDis sufficient to provide a satisfied statistical estimator with efficient computation without gradient correction techniques. This finding challenges the widely held belief in decentralized optimization literature that extra techniques like gradient tracking and other gradient-correction-based methods are necessary for heterogeneous scenarios [39, 32]. The following corollary considers the well-known instance where the design matrix and noise are generated from sub-Gaussian distribution, which indicates that DGD with early stopping can achieve the minimax optimal statistical rate under the ℓ 2 metric.

Corollary 1. Suppose that entries of global design matrix X generated from i.i.d 1-sub-Gaussian distribution, and the total sample size satisfies N ≳ s ( s log ed s +log dN m ) . The noise vector ξ is generated from independent σ 2 -sub-Gaussian entries, and the initialization is set as Theorem 1 with

then after running t = O   w ⋆ max √ N σ √ log d ( 1 -( 1+ √ ρ 2 ) 1 4 ) log 1 α   iterations, the sequence generated by (3)

and (4) based on DGD for solving problem (2) would obtain estimator that ∥ ∥ w t -w ⋆ ∥ ∥ ≲ σ √ s log d N with probability at least 1 -3 8 d 3 .

Corollary 1 indicates that in the sub-Gaussian setting, network-independent statistical error obtained by DGD matches optimal rate O ( σ √ s log d N ) under ℓ 2 metric in the centralized setting [28]. In this context, the condition for network connectivity implies that the smaller ρ is required as the number of agents m increases. This is reasonable because when the total sample size N is fixed, an increase in the number of agents results in fewer samples assigned to each agent. Consequently, better network connectivity is necessary to achieve optimal estimation.

5 Communication Efficient DGD via Truncation

It is apparent that iterations in (3) and (4) of DGD, each agent has to transmit two d -dimensional vectors u t,i and v t,i to its neighboring agents per iteration. Because we are considering the highdimensional regime where the feature has ultra-high dimension, which leads to the O ( d · 1 ηζ log 1 α ) high communication complexity (in terms of the bits transmitted) for DGD. The primary idea is

whether it is possible to transmit fewer partial elements instead of the entire d -dimensional vectors for u t,i , v t,i in all rounds of communication. Since all elements of u t,i and v t,i equal to α at initialization, we can utilize the one step of local gradient descent step in each agent to distinguish the support set and non-support based on changes of magnitudes for each element. The intuition is that the elements on the support would grow more rapidly than those on the non-support. Thus, we propose the T runcated D ecentralized G radient D escent (T-DGD) as

for ∀ i ∈ [ m ] , where Trun s ( x ) is the operator that preserves only the s largest magnitude elements of the vector x while setting all other elements to zero. The following proposition shows the benefit of T-DGD in sparse regression under proper conditions.

Proposition 1. With the same setup in Corollary 1, if the ground truth w ⋆ satisfies w ⋆ min 2 ≳ √ sδ max w ⋆ max + σ √ log d n , then the sequence generated by T-DGD as (8) for solving problem (2) would obtain estimator that ∥ ∥ w t -w ⋆ ∥ ∥ ≲ σ √ s log d N with probability at least 1 -3 8 d 3 . However, the communication complexity in terms of transmitted bits would be at most O ( s · ( w ⋆ max √ N )/ ( σ √ log d ( 1 -( 1+ √ ρ 2 ) 1 4 )) log 1 α ) .

To ensure condition w ⋆ min 2 ≳ √ sδ max w ⋆ max + σ √ log d n satisfied, it is necessary to require that each agent has sufficient samples and SNR is high enough such that δ max ≲ w ⋆ min √ sw ⋆ max and σ √ log d n ≲ w ⋆ min . This proposition enables each agent to transmit only s elements of d -dimensional vector per communication round, which can achieve optimal statistical rate and eliminate the d -linear increasing communication complexity. The result in Proposition 1 validates the usefulness of the Hadamard product over-parameterization in decentralized gradient-based optimization.

6 Numerical Results

6.1 Simulations on DGD

We organize the experiments as follows: 1) We visualize the dynamics of averaged variables and consensus errors that allow us to evaluate the implicit regularization of DGD and the soundness of our technical analysis. 2) We check whether DGD can achieve optimal statistical error, the impacts of ambient dimension d and initialization scale α on statistical and computational properties. 3) We evaluate the condition of (6) that reveals the relationship between network connectivity and network scale for achieving the statistical accuracy of centralized setting.

Dynamics of w t , u t , v t and u t,i -u t , v t,i -v t . In this case, we set d = 2000 , s = 10 , m = 10 , N = 400 , ρ = 0 . 1778 , α = 10 -6 . Fig. 1(a) demonstrates the convergence of averaged w t in DGD, showing successful convergence of elements on support S and maintenance of small

magnitudes for elements on non-support S c . Fig. 1(b) and Fig. 1(c) further illustrate how DGD utilizes u t and v t to fit parameters on positive and negative support, respectively. Additionally, the magnitudes of u t and v t on non-positive and non-negative support remain small enough as the initialization. Consensus errors u t,i -u t and v t,i -v t are depicted in Fig. 1(d) and Fig. 1(e), respectively. The trends in these curves correspond to the magnitudes of the model parameter, affirming the validity of our analysis.

(a)

w

t

0

50

100

150

200

250

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

(b)

(

u

t

)

2

0

50

100

150

200

250

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

(c)

(

v

t

)

2

0

50

100

150

200

250

0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

(d)

u

t,i

-

u

t

0

50

100

150

200

250

0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

(e)

v

t,i

-

v

t

Impact of d and α on optimal estimation. We vary the dimension of d (4 × 10 2 , 4 × 10 3 , 4 × 10 4 ) to access effect of d on both statistical and computational properties. With s = ⌈ log d ⌉ and N chosen to satisfy s log d/N ≈ 0 . 25 , we aim to maintain the same order optimal statistical error O ( σ √ s log d N ) . Testing is conducted on two networks with m = 20 but different ρ . For each d , we select the maximum initialization α that achieves optimal statistical error, resulting in α = 10 -8 for d = 4 × 10 2 , α = 10 -8 . 5 for d = 4 × 10 3 and α = 10 -9 for d = 4 × 10 4 . The results for ρ = 0 . 1778 and ρ = 0 . 7519 are displayed in Fig. 2(a) and Fig. 2(b), respectively. It is observable that DGD obtains estimators with statistical error matching that of the centralized setting, with computational complexity remaining largely unaffected by ambient dimension d across different network conditions. To assess the influence of α , we set d = 2000 , s = 10 , m = 20 , N = 400 , and different values for α on network with m = 20 , ρ = 0 . 1778 . The results in Fig. 2(c) illustrate that it is necessary to use small enough initialization to obtain optimal estimator.

0

50

100

150

200

250

300

-5

-4

-3

-2

-1

0

1

2

3

centralized statistical error

(a)

m

= 20

, ρ

= 0

.

1778

0

100

200

300

400

500

600

700

800

900

1000

-5

-4

-3

-2

-1

0

1

2

3

centralized statistical error

(b)

m

= 20

, ρ

= 0

.

7519

0

50

100

150

200

250

300

350

400

-5

-4

-3

-2

-1

0

1

2

3

centerlized statistical error

(c)

m

= 20

, ρ

= 0

.

1778

Dependence on ρ and m . We set d = 2000 , s = 10 , N = 200 , α = 10 -6 and test on networks with different numbers of agents. The results are shown in Fig. 3 where Fig. 3(a) and Fig.

0

100

200

300

400

500

600

700

800

900

1000

-5

-4

-3

-2

-1

0

1

2

(a)

0

100

200

300

400

500

600

700

800

900

1000

-5

-4

-3

-2

-1

0

1

2

(b)

0

100

200

300

400

500

600

-5

-4

-3

-2

-1

0

1

2

(c)

3(b) display the performance with varied numbers of agents under fixed ρ = 0 . 9400 and fixed ρ = 0 . 1778 , respectively. Fig. 3(a) indicates that DGD would not obtain the optimal estimator as the centralized setting when the number of agents is large which violates the condition in (6). When network connectivity is sufficiently connected as ρ = 0 . 1778 , Fig. 3(b) conveys that this can allow a larger scale of agents to attain optimal statistical error. In Fig. 3(c), we fix m = 10 and observe the phenomenon under varied ρ by choosing proper stepsizes to achieve the best statistical error. Fig. 3(c) illustrates that ρ would influence the stopping time when DGD can obtain the optimal estimator. The worse the network, the more iterations it takes to find the optimal estimator.

6.2 Simulations on T-DGD

In this section, we evaluate the effectiveness of T-DGD. Initially, we vary the values of N , keeping the other parameters consistent with the simulations in Fig. 1. Fig. 4(a) illustrates that when each agent has inadequate samples ( N = 100 , n = 10 ), T-DGD would fail in achieving optimal estimation. However, with increasing local samples ( N = 400 , n = 10 ), T-DGD matches DGD in both statistical accuracy and convergence performance. Subsequently, we set the magnitudes of the ground truth on support as 100 and N = 300 . The performance is depicted as dashed lines in Fig. 4(c), indicating failure of T-DGD under higher noise level ( σ = 0 . 5 ). We further reduce the noise magnitude to σ = 0 . 1 , and solid lines in Fig. 4(c) demonstrate the usefulness of T-DGD in sparse regression. These observations validate the statement in Proposition 1.

0

50

100

150

200

250

300

-5

-4

-3

-2

-1

0

1

2

3

(a)

0

50

100

150

200

250

300

-6

-5

-4

-3

-2

-1

0

1

2

3

(b)

0

50

100

150

200

250

300

-8

-6

-4

-2

0

2

4

6

8

10

12

(c)

6.3 Comparison with explicit regularization

We have compared our proposed method with three existing decentralized methods, namely: CTADGD (LASSO) [18], ATC-DGD (LASSO) [17], and DGT (NetLASSO) [31]. These methods are all derived based on the LASSO formulation with explicit regularization. The numerical results presented in Fig. 5 compare all four methods under three different network connectivity settings. For each method, we tuned the step size to achieve the best performance. Our proposed method demonstrated the best recovery performance in all network settings with minimal iterations.

0

500

1000

1500

2000

2500

3000

3500

4000

4500

-4

-2

0

2

4

6

8

(a)

ρ

= 0

.

9428

0

500

1000

1500

2000

2500

3000

3500

4000

4500

-6

-4

-2

0

2

4

6

8

(b)

ρ

= 0

.

6367

0

200

400

600

800

1000

1200

1400

1600

1800

2000

-6

-4

-2

0

2

4

6

8

(c)

ρ

= 0

.

1384

We further compared T-DGD with existing methods with truncated versions of existing methods: Trun-CTA-DGD (LASSO), Trun-ATC-DGD (LASSO), and Trun-DGT (NetLASSO) which use the same Tops truncation operator. As shown in Fig. 6, our proposed method is the only one to achieve successful recovery, while all other truncated decentralized methods failed. The numerical evidence demonstrates that naively combining sparsification with decentralized algorithms is not granted to converge . This is precisely one of the motivations of this work: to provide communication-efficient algorithms with both provably statistical and computational guarantees. This result also demonstrates the unique benefit of overparameterization and implicit regularization for decentralized learning setting, which has not been explored in the literature of learning theory.

0

20

40

60

80

100

120

140

160

180

200

-10

-8

-6

-4

-2

0

2

4

6

8

7 Conclusion

In this paper, we study the implicit regularization of decentralized gradient descent for decentralized sparse regression in the unpenalized and overparameterized regimes. We establish both statistical and computational guarantees for the decentralized estimator under mild conditions of network connectivity, underscoring the utility of DGD in addressing overparameterized models. Furthermore, the proposed truncated DGD (T-DGD) offers a promising idea to reduce communication complexity while maintaining performance. In future work, exploring the possibility of relaxing the RIP condition in our assumption and leveraging the restricted eigenvalue condition to achieve optimal estimator in the decentralized setting is an interesting topic. Additionally, investigating alternative forms of implicit regularizations in decentralized optimization algorithms for more complicated overparameterized models is another intriguing direction.

References

Alekh Agarwal, Sahand Negahban, and Martin J Wainwright. Fast global convergence rates of gradient methods for high-dimensional statistical recovery. Advances in Neural Information Processing Systems , 23, 2010.

Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. In The Eleventh International Conference on Learning Representations , 2022.

Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over-parameterization. In International conference on machine learning , pages 242-252. PMLR, 2019.

Maksym Andriushchenko, Aditya Vardhan Varre, Loucas Pillaud-Vivien, and Nicolas Flammarion. Sgd with large step sizes learns sparse features. In International Conference on Machine Learning , pages 903-925. PMLR, 2023.

Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning , pages 322-332. PMLR, 2019.

Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machinelearning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences , 116(32):15849-15854, 2019.

Raphaël Berthier. Incremental learning in diagonal linear networks. Journal of Machine Learning Research , 24(171):1-26, 2023.

Emmanuel J Candes and Terence Tao. Decoding by linear programming. IEEE transactions on information theory , 51(12):4203-4215, 2005.

Yuan Cao, Zixiang Chen, Misha Belkin, and Quanquan Gu. Benign overfitting in two-layer convolutional neural networks. Advances in neural information processing systems , 35:2523725250, 2022.

Hung-Hsu Chou, Johannes Maly, and Holger Rauhut. More is less: inducing sparsity via overparameterization. Information and Inference: A Journal of the IMA , 12(3):1437-1460, 2023.

Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International conference on machine learning , pages 1675-1685. PMLR, 2019.

Jianqing Fan, Zhuoran Yang, and Mengxin Yu. Understanding implicit regularization in over-parameterized single index model. Journal of the American Statistical Association , 118(544):2315-2328, 2023.

Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro. Characterizing implicit bias in terms of optimization geometry. In International Conference on Machine Learning , pages 1832-1841. PMLR, 2018.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision , pages 1026-1034, 2015.

Peter D Hoff. Lasso, fractional norm and structured sparse estimation using a hadamard product parametrization. Computational Statistics & Data Analysis , 115:186-198, 2017.

Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems , 31, 2018.

Yao Ji, Gesualdo Scutari, Ying Sun, and Harsha Honnappa. Distributed (atc) gradient descent for high dimension sparse regression. IEEE Transactions on Information Theory , 2023.

Yao Ji, Gesualdo Scutari, Ying Sun, and Harsha Honnappa. Distributed sparse regression via penalization. Journal of Machine Learning Research , 24(272):1-62, 2023.

Jiangyuan Li, Thanh Nguyen, Chinmay Hegde, and Ka Wai Wong. Implicit sparse regularization: The impact of depth and early stopping. Advances in Neural Information Processing Systems , 34:28298-28309, 2021.

Marie Maros and Gesualdo Scutari. Acceleration in distributed sparse regression. Advances in Neural Information Processing Systems , 35:36832-36844, 2022.

Marie Maros and Gesualdo Scutari. Dgdˆ 2: A linearly convergent distributed algorithm for high-dimensional statistical recovery. Advances in Neural Information Processing Systems , 35:3475-3487, 2022.

Marie Maros and Gesualdo Scutari. Decentralized matrix sensing: Statistical guarantees and fast convergence. Advances in Neural Information Processing Systems , 36, 2024.

Mor Shpigel Nacson, Kavya Ravichandran, Nathan Srebro, and Daniel Soudry. Implicit bias of the step size in linear diagonal neural networks. In International Conference on Machine Learning , pages 16270-16295. PMLR, 2022.

Eshaan Nichani, Yu Bai, and Jason D Lee. Identifying good directions to escape the ntk regime and efficiently learn low-degree plus sparse polynomials. Advances in Neural Information Processing Systems , 35:14568-14581, 2022.

Hristo Georgiev Papazov, Scott Pesme, and Nicolas Flammarion. Leveraging continuous time to understand momentum when training diagonal linear networks. In Proceedings of the 27th International Conference on Artificial Intelligence and Statistics (AIS-TATS) 2024, , 2024.

Scott Pesme and Nicolas Flammarion. Saddle-to-saddle dynamics in diagonal linear networks. Advances in Neural Information Processing Systems , 36, 2024.

Scott Pesme, Loucas Pillaud-Vivien, and Nicolas Flammarion. Implicit bias of sgd for diagonal linear networks: a provable benefit of stochasticity. Advances in Neural Information Processing Systems , 34:29218-29230, 2021.

Garvesh Raskutti, Martin J Wainwright, and Bin Yu. Minimax rates of estimation for highdimensional linear regression over ℓ _ q -balls. IEEE transactions on information theory , 57(10):6976-6994, 2011.

Dominic Richards and Patrick Rebeschini. Graph-dependent implicit regularisation for distributed stochastic subgradient descent. Journal of Machine Learning Research , 21(34):1-44, 2020.

Ohad Shamir. The implicit bias of benign overfitting. In Conference on Learning Theory , pages 448-478. PMLR, 2022.

Ying Sun, Marie Maros, Gesualdo Scutari, and Guang Cheng. High-dimensional inference over networks: Linear convergence and statistical guarantees. arXiv preprint arXiv:2201.08507 , 2022.

Ying Sun, Gesualdo Scutari, and Amir Daneshmand. Distributed optimization based on gradient tracking revisited: Enhancing convergence rate via surrogation. SIAM Journal on Optimization , 32(2):354-385, 2022.

Hossein Taheri and Christos Thrampoulidis. On generalization of decentralized learning with separable data. In International Conference on Artificial Intelligence and Statistics , pages 4917-4945. PMLR, 2023.

Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology , 58(1):267-288, 1996.

Gal Vardi. On the implicit bias in deep-learning algorithms. Communications of the ACM , 66(6):86-93, 2023.

Tomas Vaskevicius, Varun Kanade, and Patrick Rebeschini. Implicit regularization for optimal sparse recovery. Advances in Neural Information Processing Systems , 32, 2019.

Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge university press, 2019.

Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Conference on Learning Theory , pages 3635-3673. PMLR, 2020.

Kun Yuan, Qing Ling, and Wotao Yin. On the convergence of decentralized gradient descent. SIAM Journal on Optimization , 26(3):1835-1854, 2016.

Chulhee Yun, Shankar Krishnan, and Hossein Mobahi. A unifying view on implicit bias in training linear neural networks. In International Conference on Learning Representations , 2020.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations , 2016.

Peng Zhao, Yun Yang, and Qiao-Chu He. Implicit regularization via hadamard product overparametrization in high-dimensional linear regression. arXiv preprint arXiv:1903.09367 , 2(4):8, 2019.

Mo Zhou and Rong Ge. Implicit regularization leads to benign overfitting for sparse linear regression. In International Conference on Machine Learning , pages 42543-42573. PMLR, 2023.

Tongtian Zhu, Fengxiang He, Kaixuan Chen, Mingli Song, and Dacheng Tao. Decentralized sgd and average-direction sam are asymptotically equivalent. In International Conference on Machine Learning , pages 43005-43036. PMLR, 2023.

A Appendix.A

Section A.1, A.2 give the additional notations and useful basic lemmas, respectively. Section A.3 provides the properties for the simplified setting where all the elements of ground truth parameter w ⋆ are non-negative. Section A.4 gives the key proposition for the general case where w ⋆ contains both positive and negative elements on support S . Final Section A.5 concludes the proofs for the Theorem 1, Corollary 1, and Proposition 1 in the main paper based on Proposition 3 in Section A.4.

A.1 Full Notations

In addition to introduced notations from Section 3.1 in the main paper, we need additional notations to present the proof. The consensus error is denoted as ∆ t,i u := u t,i -u t . ∆ t,i u , S + = ∆ t,i u ⊙ 1 S + and notations ∆ t,i u , S -, ∆ t,i u , S c can be defined similarly. The additional notations are defined as

Above definitions can be also extended to variable v similarly.

A.2 Premilary Lemmas

Lemma 1. (Theorem 6.5 in [37]) Consider the random feature vector x ∈ R d that all entries obey i.i.d. 1 -sub-Gaussian distribution, if the sample size satisfies n ≳ δ -2 ( s log ed s +log 2 ϵ ) , then the sample covariance matrix ˆ X = 1 n ∑ n i =1 x i x T i satisfies ( δ, s ) -RIP condition with probability 1 -ϵ .

Lemma 2. (Lemma A.3 in [36]) Suppose that X √ n ∈ R n × d satisfies the ( δ, s +1) -RIP, if w ∈ R d is a s -sparse vector, then ∥ ∥ ( X T X n -I ) w ∥ ∥ ≤ √ sδ ∥ w ∥ ∞ .

Lemma 3. (Lemma A.4 in [36]) Suppose that 1 √ X ∈ R n × d that satisfies ( δ, 1) -RIP with 0 ≤ δ ≤ 1

n , ∥ ∥ .

Lemma 4. (Lemma B.5 in [42]) Let ξ ∈ R n is a vector of independent σ -sub-Gaussian random variables and all ℓ 2 norm of column vectors of X ∈ R n × d are bounded, then with high probability

A.3 Non-negative Case

We consider the simplified setting where all the elements on support are positive for ground truth w ⋆ . The following lemma shows the recursion of average variable u t on support S and non-support S c .

Lemma 5. Consider the sequence { u t,i } generated according to (3) and (4) by DGD for solving loss function in (2) , the average signal u t on support S and non-support S c are updated according to the following formulas

(9)

where the perturbed error terms p t , q t , g t , f t induced from decentralized network are defined as

Proof. Based on the updating of DGD, one-step iteration of the averaged parameter is

The gradient difference has the formula as

where the last inequality is due to the definition of ∆ t,i . Substituting the above equality into (16) would have

Because there are 1 m ∑ m i =1 u t S ⊙ ∆ t,i S = 0 and 1 m ∑ m i =1 ∆ t,i S ⊙ ( u t S ⊙ u t S -w ⋆ ) = 0 , the above formula can be simplified as

Thus, this would obtain the recursion (10) for support averaged signal.

The recursion of optimization error on non-support S c becomes

Rearranging the above equality would obtain the (11).

The following lemma shows the recursion of consensus error on support S and non-support S c .

Lemma 6. Consider the sequence { u t,i } generated according to (3) and (4) by DGD for solving loss function in (2) , the consensus error ∆ t on support S and non-support S c have following recursion

Proof. Based on the iteration of DGD, there is

Thus, the consensus error on support S has recursion as follows

where ( i ) is due to the defined spectral gap of network in Assumption 1 and ( ii ) uses the gradient difference formula in (17) and

(26)

and local and global RIP conditions. The ( iii ) is summing up terms involved ∥ ∥ ∆ t +1 S ∥ ∥ ∞ and ∥ ∥ u t S ∥ ∥ ∞ separately.

The recursion of consensus error on non-support S c part is as follows

The following proposition shows the dynamics of average variable u t and consensus error ∆ t,i in the form of an inductive hypothesis. Before showing the proposition, we define the following quantities. We define T := 1 ηw ⋆ max log 1 α 4 and for any integer k ≥ -1 , T k := 2 k T and T k := ∑ k i =0 T i with T -1 = 0 where T k denotes the number of iterations between ( k -1) -th and k -th induction step. Defining K := ⌈ log 2 w ⋆ max ζ ⌉ as the number of induction steps, B k := w ⋆ max 40 × 2 k denotes the upper bound of perturbed error in ( k -1) -th induction step and constant scale parameter β := 32 1 - √ ρ .

Proposition 2. With the same setting as Theorem 1, the following claims hold k = 0 , 1 , · · · , K -1 steps.

(a) For T k -1 ≤ t < T k that ∀ k ∈ [ K ] , there is ∥ ∥ u t S ⊙ u t S -w ⋆ ∥ ∥ ∞ ≤ w ⋆ max 2 k .

(b) For ∀ k ∈ [ K ] , there is ∥ ∥ ∥ u T k -1 S ⊙ u T k -1 S -w ⋆ ∥ ∥ ∥ ∞ ≤ w ⋆ max 2 k .

(c) For T k -1 ≤ t < T k that ∀ k ∈ [ K ] , it has ∥ ∥ ∆ t S ∥ ∥ ∞ ≤ 4 βρ 3 4 η ∥ ∥ u t S ∥ ∥ ∞ B k . In addition, the refined element-wise bound is | ∆ t j | ≤ 4 βρ 3 4 η | u t j | B k , ∀ j ∈ S .

(d) For T k -1 ≤ t < T k that ∀ k ∈ [ K ] , it has ∥ ∥ ∆ t S c ∥ ∥ ∞ ≤ 4 βρ 3 4 η ∥ ∥ u t S c ∥ ∥ ∞ B k . In addition, the refined element-wise bound is | ∆ t j | ≤ 4 βρ 3 4 η | u t j | B k , ∀ j ∈ S c .

(e) For ∀ k ∈ [ K ] and ∀ j ∈ S , α 3 ≤ u T k -1 j ≤ w ⋆ j +4 B k .

Proof. Proof idea: Inductions (a), (b), (e) indicate that if the connectivity of the network is sufficiently well ( ρ is small enough), the trajectory of the averaged signal u t would mimic that of the centralized case [36]. Different from the centralized setting, these three claims are based on inductions (c) and (d), which guarantee that the consensus error along both support S and non-support S c can be controlled based on the magnitude of the respective signals. We utilize this property to reparameterize consensus error by Hadamard product based on the averaged signal. Thus, the perturbed error terms induced by the decentralized network in the recursion of the averaged signal can be quantitatively through the reparameterized consensus error. Then conditions on network connectivity ρ and step size η can guarantee that the averaged signal in decentralized would have properties in inductions (a), (b), (e) based on inductions (c), (d).

Base case : As the initialization u 0 ,i = α 1 d , ∀ i ∈ [ m ] . Due to the condition on α , the base case is true.

Induction Step : If the above (a)-(e) induction hypotheses hold all until some 0 ≤ k ≤ K -1 , we should prove they still hold at k +1 -th induction step.

(a) The magnitude of p t in (12) under this induction step can be bounded based on inductions (c), (d). ∀ T k -1 ≤ t < t +1 < T k , if ∥ ∥ u t S c ∥ ∥ ∞ keep same order as initialization, then there is

where the first inequality is based on Lemma 2 and Lemma 3, the last inequality is due to step size, value of β , ∥ ∥ u t S ∥ ∥ 2 ∞ ≤ 2 w ⋆ max , network connectivity condition and global RIP condition that ρ 1 4 √ sδ max ≤ √ sδ ≤ 1 .

For the perturbation q t in (13), which is an error term outside the multiplicative updates in (10), based on induction (a), fine-grained upper in (c), there is ∀ j ∈ S

where the first inequality is due to (28) by comparing formula of q t with p t and last inequality is due to step size condition and such that ρ 1 4 max i ∥ ∥ ∥ X T i ξ i n ∥ ∥ ∥ ∞ ≤ ∥ ∥ ∥ X T ξ N ∥ ∥ ∥ ∞ < w ⋆ max . Then q t could be reparameterized as q t = r t q ⊙ u t S where ∥ ∥ r t q ∥ ∥ ∞ ≤ √ ρB k 16 for ∀ t that T k -1 ≤ t < T k , the perturbed optimization recursion on support over decentralized network in (10) becomes

where the perturbation errors E t 2 and E t 3 are defined as

Because there is ∥ ∥ E t 2 ∥ ∥ ∞ + ∥ ∥ E t 3 ∥ ∥ ∞ + ∥ p t ∥ ∞ + ∥ ∥ r t q ∥ ∥ ∞ ≤ B k , which is based on the upper bound in (70). Then the proof is divided into the following two cases based on the magnitude of the element in w ⋆ S + .

∀ j that w ⋆ j ≥ 20 B k , based on induction hypothesis (e) that ( u T k -1 j ) 2 ≤ w ⋆ j + 4 B k , there is ( u T k -1 j ) 2 ≤ 6 5 w ⋆ j , which illustrates that it satisfies the conditions in Lemma B.10 in [36]. Then because induction hypothesis (a) and (b) are true until t -th iteration, then if B k < ∥ ∥ ∥ ( u t j ) 2 -w ⋆ j ∥ ∥ ∥ ∞ ≤ w ⋆ max 2 k , then ∥ ∥ ∥ ( u t +1 j ) 2 -w ⋆ j ∥ ∥ ∥ ∞ ≤ ∥ ∥ ∥ ( u t j ) 2 -w ⋆ j ∥ ∥ ∥ ∞ ≤ w ⋆ max 2 k , else if ∥ ∥ ∥ ( u t j ) 2 -w ⋆ j ∥ ∥ ∥ ∞ ≤ B k , then ∥ ∥ ∥ ( u t +1 j ) 2 -w ⋆ j ∥ ∥ ∥ ∞ ≤ B k . Combined with two cases, we can conclude that (a) also holds for ( t +1) -th iteration for j that w ⋆ j ≥ 20 B k .

For arbitrary j -th elements whose magnitude is not sufficiently larger than the perturbation that w ⋆ j ≤ 20 B k , based on the upper bound in induction (e), perturbation bound and monotonic property in Lemma B.6 in [36], we can guarantee that ( u t j ) 2 would keep staying in (0 , w ⋆ j +4 B k ] . With condition w ⋆ j ≤ 20 B k , we can conclude that ∥ ∥ ∥ ( u t +1 j ) 2 -w ⋆ j ∥ ∥ ∥ ∞ ≤ max { w ⋆ j , 4 B k } ≤ 20 B k ≤ w ⋆ max 2 k . Combined these two cases would finish proof of (a).

(b) To prove this statement, we should guarantee that there are sufficient iterative steps in the ( k -1) -th induction that can make ∥ ∥ ∥ ∥ ( u T k -1 S ) 2 -w ⋆ ∥ ∥ ∥ ∥ ∞ decrease at least by half from the beginning iteration of current induction stage to that of next induction stage. The proof is also divided into two cases.

The one case is that ∀ j that it already has ∣ ∣ ∣ ( u T k -1 j ) 2 -w ⋆ j ∣ ∣ ∣ < w ⋆ max 2 k +1 , then with similar proof in (a), we can guarantee that ∀ t ≥ T k -1 , ∣ ∣ ∣ ( u t j ) 2 -w ⋆ j ∣ ∣ ∣ < w ⋆ max 2 k +1 . Thus, for these supports, we prove the ( k +1) -th induction also holds.

The second case is ∀ j that there is ∣ ∣ ∣ ( u T k -1 j ) 2 -w ⋆ j ∣ ∣ ∣ ≥ w ⋆ max 2 k +1 = 20 B k . Based on the upper bound in induction (e) that ( u T k -1 j ) 2 ≤ w ⋆ j +4 B k , then u T k -1 j , w ⋆ j must satisfy 0 ≤ ( u T k -1 j ) 2 ≤ w ⋆ j -20 B k , w ⋆ j ≥ 20 B k , respectively. This means u T k -1 j is far away from w ⋆ j at least 20 B k distance. According to Lemma B.12 in [36], to achieve ∣ ∣ ∣ ( u T k -1 j ) 2 -w ⋆ j ∣ ∣ ∣ ≤ 20 B k , the sufficient condition for the number of iterations t in current induction stage is t ≥ 15 32 ηw ⋆ j log ( w ⋆ j ) 2 19 ( u T k -1 j ) 2 B k . Now we verify the setting of T k as follows

where the first inequality is due to the definition of B k and small initialization condition that α ≤ ζ 3( w ⋆ max ) 2 and the second inequality is due to w ⋆ j ≥ 20 B k and lower bound in induction (e) that ( u T k -1 j ) 2 ≥ α 3 . The last inequality is because ζ 3 ≤ 16 40 w ⋆ max 2 K -1 ≤ 16 40 w ⋆ max 2 k = 16 B k . Thus, combining the above two cases and similar proof, we can conclude that ∀ t ≥ T k , there is ∥ ∥ ∥ ( u t S ) 2 -w ⋆ ∥ ∥ ∥ ∞ ≤ w ⋆ max 2 k +1 . This completes the proof of induction (b).

(c) To make the consensus error satisfy the above induction, ∀ t that T k -1 ≤ t < t +1 < T k , based on one step iteration in (24) of Lemma 6, induction (a), (c) and step size condition η ≤ 1 - √ ρ 64 ρ 1 2 w ⋆ max , it has

Then the recursion (24) in Lemma 6 becomes

where the last inequality is based on induction such as if ∥ ∥ ∆ t S ∥ ∥ ∞ ≤ 4 βρ 3 4 η ∥ ∥ u t S ∥ ∥ ∞ B k , then based (30) and step size condition η ≤ c 10 ( 1 - √ ρ ) w ⋆ max , there is

To guarantee that (34) holds induction (c), the sufficient condition is

The c 10 should be chosen that 1+ ρ 2 ( 1 -c 10 ( 1 - √ ρ )) 2 < 1 , which means that c 10 should satisfy c 10 ≤ 1 -√ 1+ √ ρ 2 1 - √ ρ . Then we can set c 10 = 1 -( 1+ √ ρ 2 ) 1 4 1 - √ ρ , which results in 1 ( 1 -c 10 ( 1 - √ ρ )) 2 = √ 2 1+ √ ρ . Based on (36), the lower bound for β is β ≥ 2 (√ 2 ( 1+ √ ρ ) +1+ √ ρ ) 1 -ρ .

Unfolding the recursion in (24) from the beginning of ( k -1) -th induction to the beginning of k -th induction based on (34) and combining induction (a) would have

where both the second and third inequalities use lower bound in (35). To guarantee that the last inequality satisfies the induction (c), the β and ρ should satisfy the following condition

One sufficient condition for achieving above inequality is 8 1 - √ ρ ≤ β 4 . Combine above lower bound of β , we can verify that β = 32 1 - √ ρ satisfies this condition and following inequality is attained

which implies that the step size η should satisfy

One sufficient condition for achieving inequality is η ≤ log 1 α 4 ( 1 -√ 1+ √ ρ 2 ) 4 w ⋆ max based on inequality that ln ( 1 + √ 1+ √ ρ 2 -1 ) ≤ √ 1+ √ ρ 2 -1 . Combining all the above conditions on η , we obtain the upper bound as shown in condition (5).

(d) ∀ t that T k -1 ≤ t < t +1 < T k , based on one step iteration in (27) of Lemma 6 and inductions (a), (c), there is

where the second inequality is due to 4 βρ 3 4 ηB j ≤ 4 × 32 1 - √ ρ × 1 - √ ρ 256 w ⋆ max B j ≤ 1 and third inequality is due to global RIP condition on δ that is order of 1 √ s and βη ∥ ∥ u t S ∥ ∥ 2 ∞ ≤ 1 , the last inequality is

based on induction such that if ∥ ∥ ∆ t S c ∥ ∥ ∞ ≤ 4 γρ 3 4 η ∥ ∥ u t S c ∥ ∥ ∞ B k , then based (45), if η ≤ c 10 (1 - √ ρ ) w ⋆ max , we have

With the same derivation in proving induction (c), the lower bound for γ is the same as that of β .

Then also unrolling the recursion in (27) from beginning of ( k -1) -th induction to beginning of k -th induction and combining induction (a) would have

The derivation of above inequality is similar with (37). To guarantee the above two inequality satisfy induction (d), the γ and ρ can have the same value as in (38) that γ = β = 32 1 - √ ρ .

(e) For the upper bound, the proof is divided into three cases.

One of case is ( u T k -1 j ) 2 ≤ w ⋆ j + B k , based on the Lemma B.6 in [36], we can conclude that ∀ t ≥ T k -1 , ( u t j ) 2 would keep below w ⋆ j + B k .

Another case is w ⋆ j + B k ≤ ( u T k -1 j ) 2 ≤ w ⋆ j + 2 B k , in this case ∀ t ≥ T k -1 , Either 0 ≤ ( u t j ) 2 ≤ w ⋆ j + B k , which can use result in case (1) that ∀ t ′ ≥ t , 0 ≤ ( u t ′ j ) 2 ≤ w ⋆ j + B k or that w ⋆ j + B k ≤ ( u t j ) 2 ≤ w ⋆ j +2 B k , which would guarantee that ( u t +1 j ) 2 ≤ ( u t j ) 2 .

The last case is w ⋆ j +2 B k < ( u t j ) 2 ≤ w ⋆ j +4 B k , then based on Lemma B.14 [36], after the sufficient number of iterations that ∀ t ≥ 1 10 ηB k = 4 × 2 k ηw ⋆ max and we can check that T k is large enough that satisfies this condition, it can keep that ( u T k -1 + t j ) 2 ≤ w ⋆ j +2 B k .

Following the above three cases, we can guarantee that there exists ( u T k j ) 2 ≤ w ⋆ j + 1 2 × 4 B k = w ⋆ j +4 B k +1 .

For the lower bound, we can also have bound ∥ g t ∥ ∞ ≤ 3 √ ρB k 8 by comparing formulas between p t and g t , ∀ t that T k -1 ≤ t < T k . For perturbation f t , based on induction (a), fine-grained bound in (d), ∀ j ∈ S c it has

where the first inequality is due to bound for ∥ g t ∥ ∞ and the last inequality is the same reason as (29). f t can also be reparameterized as f t = r t f ⊙ u t S c where ∥ ∥ ∥ r t f ∥ ∥ ∥ ∞ ≤ √ ρB k 16 for ∀ t that T k -1 ≤ t < T k , the perturbed optimization recursion on non-support S c over decentralized network in (11) becomes

Based on the similar lower bound (79) and upper bound in (80), we can conclude that ∀ t ≤ O ( 1 ηζ log 1 α 4 ) , there is

where this inequality guarantees that the averaged signal on non-support remains ∥ ∥ u t S c ∥ ∥ ∞ ≤ √ α until early stopping. ∀ j ∈ S , we use j k to denote the largest index that w ⋆ j ≤ B j k + α 3 . As B 0 = w ⋆ max , the existence of j k is guaranteed. Then for t = 0 , · · · , T j k -1 and based on (30), the ( u t j ) 2 would shrinkage from initialization α 2 , to obtain the lower bound, we should consider the maximum shrinkage as follows

where the last inequality is due to (46), ( u i j ) 2 ≤ α 2 , ∀ i = 1 , · · · , T j k -1 and step size condition that η ( ( u i j ) 2 + ∥ ∥ E i 2 ∥ ∥ ∞ + ∥ ∥ E i 3 ∥ ∥ ∞ + ∥ ∥ g i ∥ ∥ ∞ + ∥ ∥ ∥ r i f ∥ ∥ ∥ ∞ ) ≤ 3 40 and (1 -5 x )(1+8 x ) ≥ 1 , ∀ x ∈ [0 , 3 40 ] . Thus, we have ( u t j ) 2 ≥ α 3 for ∀ t = 0 , · · · , T j k .

For t = T j k + 1 , · · · , T j k +1 -1 and ∀ j ∈ S , let consider the auxiliary iterations that ( ˆ u t j ) 2 = ( ˆ u t -1 j ) 2 ⊙ ( 1 -4 η ( ( ˆ u t -1 j ) 2 -( w ⋆ j -B j k +1 ) )) 2 where ˆ u T j k j = u T j k j ≥ α 3 as above proved. Based on definition of j k , there is w ⋆ j -B j k +1 ≥ α 3 . According to monotonic property in Lemma B.6 in [36], we can guarantee that ( ˆ u T j k +1 j ) 2 ∈

( where the first inequality is due to the squeezing property in Lemma B.9 in [36]. Then we can follow the same analysis to prove that left i -th induction step that i = j k +2 , · · · , m -1 based on the monotonic property of B k .

A.4 General Case

This section considers the general setting where the ground truth w ⋆ includes both positive and negative elements in its support. The analysis here is more complex than in the previous section due to the presence of v t,i and its consensus error terms. The following formulates the recursion of perturbed average variable u t and similar derivation could be applied to v t .

Lemma 7. Consider the sequence { u t i , v t i } , i ∈ [ m ] generated according to (3) and (4) by DGD for solving problem (2) , the average signal { u t } on positive support S + , negative support S -and non-support S c are updated according to the following formulas

where the perturbed error terms p t u , q t u , y t u , z t u , g t u , f t u induced from decentralized network are defined as

Proof. The average of optimization for u t,i , i ∈ [ m ] under DGD are as follows

Based on error decomposition on positive support S + , negative support S -and non-support S c , there has

With same decomposition on S + , S -, S c for Π 1 ,i and Π 2 ,i , there are

Substituting (59) into (57) would obtain the perturbed recursion of averaged u t S + , u t S -, u t S c in Lemma 7.

The following lemma separately bounds the consensus errors on S + , S -, and S c by the corresponding magnitudes of parameters, which is different from the current analysis in the decentralized optimization literature.

(61)

where the C t err is defined as

Proof. Based on the updating of DGD, there is

Thus, there is

where the second inequality is due to (64) and the last inequality is substituting into the formula Π 1 ,i , Π 2 ,i in (59) and gradient difference at averaged pair. Merging terms involved ∥ ∥ u t S + ∥ ∥ ∞ and ∥ ∥ ∆ t u , S + ∥ ∥ ∞ separately would obtain the result in (60). Performing analogous proof would yield results on negative support S -in (49) and non-support S c in (50).

The following proposition describes the dynamics of u t,i , v t,i by conducting inductive hypothesis for both u t,i and v t,i , which is different with centralized setting [36]. Because the complicated consensus errors prevent transferring the proof of non-negative case to general case trivially.

Proposition 3. This proposition inherits Proposition 2 for the general case with considering dynamic of v t i for learning negative part signal w ⋆ S -. Recall the definitions of T, T k , T k , B k , K, β in Proposition 2, the following statements hold in each induction step.

(b) For ∀ k ∈ [ K ] , there is max {∥ ∥ ∥ ∥ ( u T k -1 S + ) 2 -w ⋆ S + ∥ ∥ ∥ ∥ ∞ , ∥ ∥ ∥ ∥ ( v T k -1 S -) 2 -w ⋆ S -∥ ∥ ∥ ∥ ∞ } ≤ w ⋆ max 2 k .

(c) ∀ t that T k -1 ≤ t < T k with ∀ k ∈ [ K ] , there are max i ∥ ∥ ∥ ∆ t,i u , S + ∥ ∥ ∥ ∞ ≤ 4 βρ 3 4 η ∥ ∥ u t S + ∥ ∥ ∞ B k and max i ∥ ∥ ∥ ∆ t,i v , S -∥ ∥ ∥ ∞ ≤ 4 βρ 3 4 η ∥ ∥ v t S -∥ ∥ ∞ B k .

(d) For any T k -1 ≤ t < T k that ∀ k ∈ [ K ] , there are max i ∥ ∥ ∥ ∆ t,i u , S c ∥ ∥ ∥ ∞ ≤ 4 βρ 3 4 η ∥ ∥ u t S c ∥ ∥ ∞ B k and max i ∥ ∥ ∥ ∆ t,i v , S c ∥ ∥ ∥ ∞ ≤ 4 βρ 3 4 η ∥ ∥ v t S c ∥ ∥ ∞ B k .

(e) For any T k -1 ≤ t < T k that ∀ k ∈ [ K ] , there are max i ∥ ∥ ∥ ∆ t,i u , S -∥ ∥ ∥ ∞ ≤ 4 βρ 3 4 η ∥ ∥ u t S -∥ ∥ ∞ B k and max i ∥ ∥ ∥ ∆ t,i v , S + ∥ ∥ ∥ ∞ ≤ 4 βρ 3 4 η ∥ ∥ v t S + ∥ ∥ ∞ B k .

(f) ∀ t that T k -1 ≤ t < T k with k ∈ [ K ] , there are refined element-wise bounds for consensus errors as | ∆ t u ,j | ≤ 4 βρ 3 4 η | u j | B k and | ∆ t v ,j | ≤ 4 βρ 3 4 η | v j | B k , ∀ j ∈ S .

(g) For ∀ j ∈ S + and k ∈ [ K ] , there is α 3 ≤ ( u T k -1 j ) 2 ≤ w ⋆ j +4 B k . For ∀ j ∈ S -and k ∈ [ K ] , there is α 3 ≤ ( v T k -1 j ) 2 ≤ w ⋆ j +4 B k .

(h) ∀ t ≤ O ( 1 ηζ log 1 α ) , max {∥ ∥ u t S c ∥ ∥ ∞ , ∥ ∥ v t S c ∥ ∥ ∞ } ≤ √ α .

(i) ∀ t ≤ O ( 1 ηζ log 1 α ) and ∀ j ∈ S , there is u t j v t j ≤ α 3 2 .

(j) ∀ t ≤ O ( 1 ηζ log 1 α ) , it has max {∥ ∥ u t S -∥ ∥ ∞ , ∥ ∥ v t S + ∥ ∥ ∞ } ≤ √ α .

Proof. Proof idea: The key difference in the proof between centralized and decentralized settings lies in the fact that we cannot directly transfer the proof from the simpler non-negative w ⋆ to the general w ⋆ . This is because the error terms induced in (48), (49),(50) by the decentralized network in Lemma 7 prevent obtaining the results in inductions (i), (j) without induction steps as Lemma B.16 in [36], which allows to apply proof of non-negative case to the general case directly in the centralized case. Therefore, we have to conduct the comprehensive induction process for the general w ⋆ , which can ensure that the magnitudes of u t on the negative support S -and v t on the positive support S + remain small up to the early stopping iteration steps. The inductions (a)-(h) show that the averaged signal u t on the positive support S + and the averaged signal v t on the negative support exhibit dynamics similar to those in non-negative case, as outlined in Proposition 2.

Base case: As the initialization u 0 ,i = v 0 ,i = α 1 d , ∀ i ∈ [ m ] . Due to the condition on α , the base case is true.

Induction Step : Under the assumption that all above induction hypotheses hold until some 0 ≤ k ≤ K -1 , we should prove they still should hold at k +1 -th induction.

and ∥ ∥ ∥ ( v t S -) 2 -w ⋆ S -∥ ∥ ∥ ∞ ≤ w ⋆ max 2 k still holds. If the condition is true for t -th iteration, then based on claims (c),(d), (e), (h) and (j) the ∥ p t u ∥ ∞ in (51) under current induction step could be bounded as

where the first inequality is due to definition of local RIP condition and Lemma 2 and Lemma 3, second inequality is due to hypothesis (a) holds at t -th iteration that ∥ ∥ u t S + ∥ ∥ 2 ∞ ≤ 2 w ⋆ max , ∥ ∥ v t S -∥ ∥ 2 ∞ ≤ 2 w ⋆ max , the last inequality is due to definitions of β, B k , step size condition and hypothesis that ∥ ∥ u t S c ∥ ∥ ∞ , ∥ ∥ v t S c ∥ ∥ ∞ , ∥ ∥ u t S -∥ ∥ ∞ , ∥ ∥ v t S + ∥ ∥ ∞ would keep below √ α up to early stopping under hypothesis (h) and (j).

The element-wise bound for q t u in (52) under current induction step can be bounded as follows that ∀ j ∈ S + , there is

where the first inequality is based on refined bound in hypothesis (c), result in (66) and inequality ∥ x + y ∥ 2 ∞ ≤ 2 ∥ x ∥ 2 ∞ +2 ∥ y ∥ 2 ∞ and step size condition that 4 βρ 3 4 ηB k < 1 . The second inequality uses hypothesis (h) and (j). The last inequality is because of hypothesis (a), the step size condition, network connectivity condition, small initialization α and definition of B k .

Then q t u could be reparameterized as q t u = r t q u ⊙ u t S + where ∥ ∥ r t q u ∥ ∥ ∞ ≤ √ ρB k 32 , ∀ t, T k -1 ≤ t < T k . Then the perturbed recursion for u t on positive support S + over decentralized network in (48) becomes

where the new perturbed error terms E t 2 s , E t 2 s are defined as

Then the total error terms in (68) can be bounded as follows

where ( i ) is because of Lemma 2 and Lemma 3 under the global RIP condition and induction hypothesis (a) and substituting the results in (66) and (67). ( ii ) is due to condition on global RIP parameter that δ ≤ C γ 2 √ s ( log ⌈ w⋆ max ζ ⌉ +1 ) , condition α ≤ ζ (3 d +1) 2 and network connectivity condition ρ ≤ 1 36 ( √ sδ max +3 ) 4 3 . ( iii ) is due to definition of ζ and ( iv ) is based on definition of K that ζ ≤ w ⋆ max 2 K -1 ≤ w ⋆ max 2 k -1 . ( v ) is because definition of B k and C γ , C b are determined later that are small enough that C γ +2 C b ≤ 1 80 .

Based on iteration (68) and perturbation bound in (70), we would prove the ∥ ∥ ∥ ( u t +1 S + ) 2 -w ⋆ S + ∥ ∥ ∥ ∞ ≤ w ⋆ max 2 k should be also true based on the proof of hypothesis (a) in Proposition 2 by replacing S with S + . We can obtain similar results for v t S -. Combined two cases and induction on t , we have conclusion that ∀ t ≥ T k -1 , there is max {∥ ∥ ∥ ( u t S + ) 2 -w ⋆ S + ∥ ∥ ∥ ∞ , ∥ ∥ ∥ ( v t S -) 2 -w ⋆ S -∥ ∥ ∥ ∞ } ≤ w ⋆ max 2 k . (71) Thus, we finish the proof of induction (a).

(b) Comparing the (68), (70) with (30), the proof of this hypothesis can follow the proof of hypothesis (b) in Proposition 2.

(

where ( i ) is due to the claim (a) and step size condition that 4 βρ 3 4 ηB k ≤ 1 and δ ≤ δ max . ii ) is due to 4 βρ 3 4 ηB k ≤ 1 , condition that all ∥ ∥ u t S c ∥ ∥ ∞ , ∥ ∥ v t S c ∥ ∥ ∞ , ∥ ∥ u t S -∥ ∥ ∞ , ∥ ∥ v t S + ∥ ∥ ∞ are not larger than √ α before early stopping in hypothesis (h), (j) and ∥ ∥ ∥ X T ξ N ∥ ∥ ∥ ∞ ≤ max i ∥ ∥ ∥ X T i ξ i n ∥ ∥ ∥ ∞ .

max {∥ ∥ ∥ X T ξ N ∥ ∥ ∥ ∞ , ϵ } ≤ w ⋆ max . The ( iv ) is because in the current induction step hypothesis (c) holds in t -th iteration and step size condition that satisfies η ≤ 1 - √ ρ 160 ρ 1 4 w ⋆ max such that ρ 1 2 ( ρ 1 4 +80 ρ 1 4 ηw ⋆ max ) ≤ (1 -(1 -ρ 1 2 ))(1 + 1 - √ ρ 2 ) ≤ 1+ √ ρ 2 and 16 √ sδ max βρη ∥ ∥ v t S -∥ ∥ 2 ∞ ≤ B k 4 where ∥ ∥ v t S -∥ ∥ 2 ∞ ≤ 2 w ⋆ max is according to claim (a). The last inequality ( v ) is based on the (68) and step size condition η ≤ c 10 ( 1 - √ ρ ) w ⋆ max such that 4 η (∥ ∥ ∥ ( u t S + ) 2 -w ⋆ S + ∥ ∥ ∥ ∞ + B k ) ≤ 1 based on hypothesis (a). Comparing the formula of (72) with (34), we can follow the poof of hypothesis (c) in Proposition 2 to finish the proof.

(d) The proof is similar to that of hypothesis (c). ∀ t that T k -1 ≤ t < t +1 ≤ T k , according to one step iteration in (49) and hypothesis (a), we have

where the first two inequalities have the same reasons in (72). The third inequality is due to 16 βρ 3 4 B k ( ∥ ∥ u t S + ∥ ∥ 2 ∞ + ∥ ∥ v t S -∥ ∥ 2 ∞ ) ≤ 8 5 w ⋆ max and ρ 1 4 √ sδ max ≤ √ sδ 8 ≤ 1 640 due to upper bound of global RIP condition in Proposition 1 with C γ ≤ 1 80 . The last inequality is due to (79). Then we can use analogized proof for hypothesis (c) to finish the proof.

(e) The proof is similar to hypothesis (d). ∀ t that T k -1 ≤ t < t +1 ≤ T k , based on one step iteration in (50) and hypothesis (a), there is

where the last inequality and the left proof have analogized derivations in proof of hypothesis (d).

(f) To absorb the perturbed error q t u in (48), we need to prove the fine-grained bound for consensus error in hypotheses (c), (d), (d). The idea is that we focus on the ∀ j ∈ S + , then the upper bound on the consensus error on j -th entry is

where the last inequality is due to substituting the crude bound and analogous derivation in (72). Following the proof of crude bound would finish the proof. The proof for ∀ j ∈ S -and ∀ j ∈ S c can combine this proof with proofs of hypotheses (e) and (d), respectively. The fine-grained bound for v t can be proved by using analogous proofs for u t .

(g) The proof can follow the proof of hypothesis (e) in Proposition 2 by replacing the S with S + . The proof of v t S -is analogous to u t S + .

(h) ∀ t that T k -1 ≤ t < t +1 < T k . First, we bound the perturbation error ∥ g t u ∥ ∞ , ∥ ∥ f t u ∥ ∥ ∞ in (50) induced from decentralized network

where the second inequality is substituting the upper bound in (66) and ∥ ∥ u t S + ∥ ∥ ∞ ≤ 2 w ⋆ max due to hypothesis (a). ∀ j ∈ S c , there is

where the first inequality is due to hypotheses (a), (c)-(f), (h), (g) and step size condition. The second inequality is due to the definition of B k , the condition on ρ , and small initialization for α .

Then f t u could be reparameterized as f t u = r t f u ⊙ u t S c , where ∥ ∥ ∥ r t f u ∥ ∥ ∥ ∞ ≤ √ ρB k 32 , ∀ t that T k -1 ≤ t < T k . Then the perturbed recursion for u t on non-support S c over decentralized network in (50) becomes

Thus, we can conduct similar calculation in (70) and obtain the upper bound ∥ ∥ E t 2 s ∥ ∥ ∞ + ∥ ∥ E t 3 s ∥ ∥ ∞ + ∥ g t u ∥ ∞ + ∥ ∥ ∥ r t f u ∥ ∥ ∥ ∞ ≤ B k . Based on the step size condition that η ≤ c 10 (1 - √ ρ ) w ⋆ max , therefore there is a similar result in (35) as

Based on the recursion in (78), ∀ k -th induction stage, we have the following upper bound

where ( i ) is due to ∀ x 1 , x 2 , · · · , x n ≥ 0 , there is 1 + ∑ n i =1 x i ≤ ∏ n i =1 (1 + x i ) , substituting the upper bound in (76) and definition of global RIP parameter. For ( ii ) , we use T k ≤ 2 T k based on definition of T k , T k , induction (a) with Lemma B.13 in [36], step size condition, small initialization for α , definition of B k and refined upper bound for ∥ ∥ ∥ r t f u ∥ ∥ ∥ ∞ as ∥ ∥ f t u ∥ ∥ ∞ ≤

ρ 3 4 ( √ sδ max B k +32 dα +max i ∥ ∥ ∥ X T i ξ i n ∥ ∥ ∥ ∞ ) ∥ ∥ u t S c ∥ ∥ ∞ . ( iii ) is because network connectivity condition that ρ ≤ min { ( √ sδ √ sδ max +4 ) 4 3 , ∥ X T ξ ∥ ∞ 8 m max i ∥ X T i ξ i ∥ ∞ } . ( iv ) , ( v ) are due to condition i ≤ K -1 and inequality 1 + ∑ K i =1 x i ≤ ∏ K i =1 (1 + x i ) that x 1 = , · · · , x K = 32 ηC b ζ K . ( vi ) is based on the definition of K that ζ ≤ w ⋆ max 2 K -1 and condition that 4 C b ≤ C γ . The last inequality results from the Lemma A.2 in [36] with upper bound is set as B = 2 C γ w ⋆ max K × 2 K -1 and initialization x 0 = α 2 , then for t ≤ K × 2 K -1 64 ηC γ w ⋆ max log 1 α 4 , there is

Thus, we select universal constant C γ ≤ 1 1280 such that K × 2 K -1 64 ηC γ w ⋆ max log 1 α 4 ≥ 20 K × T K -1 , which would lead to the last inequality.

(i) Concentrate on the iteration by DGD algorithm on support S , based on (57), we have the recursive formula as

where the E t d 1 , E t d 2 , E t d 3 and E t,i d 4 are defined as

For the negative part v t , with similar decomposition in (57), there is

After tedious calculation, its formula is as follows

Based on induction (c) and (d), we can reparameterize ∆ t,i u , S and ∆ t,i v , S as ∆ t,i u , S = r t,i u , S ⊙ u t S and ∆ t,i v , S = r t,i v , S ⊙ v t S , where max {∥ ∥ ∥ r t,i u , S ∥ ∥ ∥ ∞ , ∥ ∥ ∥ r t,i v , S ∥ ∥ ∥ ∞ } ≤ 4 βρ 3 4 ηB k for T k -1 ≤ t < T k , Thus, the (82) and (85) can be reformulated as

Based on hypotheses (a), (c)-(f), (j) and step size condition, we have the following bound for additional perturbed error terms

where the last inequality is due to network connectivity condition that ρ ≤ min { ( √ sδ √ sδ max +4 ) 4 3 , ∥ X T ξ ∥ ∞ 8 m max i ∥ X T i ξ i ∥ ∞ } .

Then multiplying (86) and (87), we can obtain that

where the ( i ) is due to (88) and 4 η ∥ ∥ ∥ ( u t S ) 2 -( v t S ) 2 -w ⋆ + E t d 1 + E t d 2 + E t d 3 ∥ ∥ ∥ ∞ ≤ 1 . The ( ii ) is due to induction (a) and Lemma B.13 in [36]. The reason for ( iii ) is the same as the that of last inequality in (80) because of the setting of step size.

(j) Recall the recursive formula of u t S -in (49) and compare the definition of the perturbation y t u with perturbations p t u , g t u , we can obtain ∥ y t u ∥ ∞ ≤ 150 ρ 3 4 B k ( √ sδ max +4 ) 600 + ρ 3 4 dα 32 . Compare the outer perturbation z t u with outer perturbations f t u , q t u , we can obtain ∥ z t u ∥ ∞ ≤ √ ρB k ∥ u t S -∥ ∞ 16 . With similar reparameterization as z t u = r t z u ⊙ u t S -, the (49) would become

∀ j ∈ S -and ∀ t ≤ O ( 1 ηζ log 1 α ) . Let 0 ≤ τ ≤ t be the largest τ such that ( v τ j ) 2 > -w ⋆ j . If there is no such τ exists or τ = t , then

where the last inequality is due to the similar bound in (80).

If τ < t , then unrolling (90) to τ -th iteration would have

where ( i ) is based on condition for initialization that α ≤ w ⋆ min 4 and induction (h) that u τ j ≤ α 3 2 v τ j ≤ α 3 2 √ -w ⋆ j ≤ α 2 . ( ii ) is based on the induction (a), step size condition that ( 1 -4 η ( -w ⋆ j -( v τ j ) 2 + E τ 2 s,j + E τ 3 s,j + y τ s,j + r τ z s ,j )) 2 ≤ 4 and ∀ i > τ that ( v i j ) 2 < -w ⋆ j which is based on definition of τ . The last inequality has the same reason as (91).

A.5 Proof of Main Results

A.5.1 Proof of Theorem 1

Based on the proof of hypothesis (a) in Proposition 3, we can conclude that ∀ t ≥ T K , there would be max {∥ ∥ ∥ ∥ ( u T k -1 S + ) 2 -w ⋆ S + ∥ ∥ ∥ ∥ ∞ , ∥ ∥ ∥ ∥ ( v T k -1 S -) 2 -w ⋆ S -∥ ∥ ∥ ∥ ∞ } ≤ w ⋆ max 2 K ≤ ζ , where the last inequality is due to the definition of K . The total computational complexity is the value of T K -1 = 2 K -1 ηw ⋆ max log 1 α 4 = O ( 1 ηζ log 1 α ) .

Due to the definition of ζ in the Theorem 1, if ζ ≥ w ⋆ max , then our result holds at t = 0 based on the small initialization condition. When ζ ≤ w ⋆ max , we consider the two cases where the first case is the magnitudes of parameters is strong enough that ζ = 1 5 w ⋆ min ≥ 960 ς , where the value 960 is due to under the condition 2 C b + C γ ≤ 1 80 , 4 C b ≤ C γ , we set C b = 1 480 , C γ = 1 120 , we can use run DGD T ′ 1 iterations to obtain the estimator max {∥ ∥ ∥ ∥ ( u T k -1 S + ) 2 -w ⋆ S + ∥ ∥ ∥ ∥ ∞ , ∥ ∥ ∥ ∥ ( v T k -1 S -) 2 -w ⋆ S -∥ ∥ ∥ ∥ ∞ } ≤ w ⋆ max 2 K ≤ 1 5 w ⋆ min , then based on Proposition 2 in [36], we can also run DGD with another 45 32 ηw ⋆ min log w ⋆ min ϵ iterations to obtain the dimension-independent bound | w t j -w ⋆ j | ≤ max { √ sδ max i ∈S B i , B j , ϵ } , ∀ j ∈

S where the B j := ∥ ∥ ∥ X T ξ N ⊙ 1 j ∥ ∥ ∥ ∞ . The total iterations are T K -1 + 45 32 ηw ⋆ min log w ⋆ min ϵ ≤ O ( 1 ηw ⋆ min log 1 α ) = O ( 1 ηζ log 1 α ) where the first inequality is due to the condition of α .

The second case is ζ = 960 ς ≥ 1 5 w ⋆ min , then running of DGD with O ( 1 ηζ ) iterations would obtain the result in Theorem 1.

A.5.2 Proof of Corollary 1

Because the entries of global design matrix generated from i.i.d. 1 -Sub-Gaussian distribution, we have the upper bound of global RIP parameter as δ ≲ 1 √ s probability at least 1 -1 8 d 3 , which is based on the Lemma 1 and sample size lower bound in Corollary 1 matches the condition of sample complexity in Lemma 1. For the local design matrix { X i / √ n } m i =1 , local sample size satisfies n = N m ≳ (√ m δ ) -2 ( s ln ed s +ln( dn ) ) , we can bound the local RIP parameter as δ max ≤ √ m δ with probability as least 1 -1 8 d 3 based on the union bound and Lemma 1. Based on condition in (6), the ρ ≲ 1 m 4 holds with probability at least 1 -1 4 d 3 . Based on the Lemma 4, we have the upper bound for the noise level as ∥ ∥ ∥ X T ξ N ∥ ∥ ∥ ∞ ≲ σ √ log d N with probability at least 1 -1 8 d 3 . Thus, with probability at least 1 -3 8 d 3 , we have ∥ ∥ w t -w ⋆ ∥ ∥ 2 2 ≲ sϵ 2 + dϵ 2 (2 d +1) 2 ≲ sσ 2 log d N where in the last inequality we select ϵ = 4 σ √ 2 log(2 d ) N in Theorem 1.

A.5.3 Proof of Proposition 1

Because each agent has same initialization u 0 ,i = v 0 ,i = α 1 d , ∀ i ∈ [ m ] , after one step of local gradient descent from (3) and (4), there are

Because the perturbed error bound

holds with probability at least 1 -3 8 d 3 based on Lemma 2 and Lemma 4, we denote ν i = ( X T i X i n -I ) w ⋆ + X T i ξ i n .

Consider u 1 ,i , for ∀ p ∈ S + , ∀ q ∈ S c , there is w ⋆ p -| ν i p | > ϕ > | ν i q | > 0 based on the condition in Proposition 1. In addition, ∀ j ∈ S -, there is -| w ⋆ j | + | ν j | < 0 based on the the condition on Proposition 1. Thus, the growth of elements on positive support S + would be larger than these of S -, S c , and the Trun k operator would identify the S + . The analogous analysis could also applied to v 1 ,i that the Trun k operator would identify the S -. Because each agent can identify the S + and S -and based on results in Proposition 3, the Trun k would also obtain the optimal statistical error with probability at least 1 -3 8 d 3 .

B Appendix.B

B.1 Kernel to rich regime transition

Previous works show a transition from the kernel regime to the rich regime by varying the initialization scale in the gradient descent method [38]. Experimental results in Fig. 7 show that the transition phenomenon also appears in DGD. We can observe that when we increase the initialization scale α gradually, DGD would converge to the minimal ℓ 2 norm solution w ⋆ ℓ 2 . On the contrary, when we

decrease the α , the DGD would converge to the minimal ℓ 1 norm solution (sparse solution) w ⋆ ℓ 1 . Thus, Fig. 7 demonstrates the existence of phase transition from kernel to rich regime for DGD when decreasing initialization α . Since we focus on sparse recovery, the small initialization would achieve this aim with better generalization performance.

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

0

10

20

30

0

2

4

6

B.2 More implicit regularizations of DSGD

To verify the widespread existence of implicit regularization in decentralized optimization, we add two extra experiments on general overparameterized neural network architecture trained by DSGD, which are motivated from [38].

(1) The first one is that we use vanilla decentralized SGD(DSGD) to train a depth-2, 5000 hidden ReLU network with the cross-entropy loss on the MNIST dataset until each agent model reaches almost 100% training accuracy. The number of agents is 10 and network connectivity is ρ = 0 . 178 . 60000 total training samples and 10000 test samples are uniformly allocated to agents. To evaluate the implicit regularization of SGD under varying initialization scales, the network weights are initialized as α w 0 , w 0 ∼ N (0 , √ 2 n in ) , which is suggested by [14] and n in denotes the number of units in the last year. Each agent uses the same batch size 256 to train in DSGD. The step sizes were optimally tuned for each α individually to achieve the best validation error. We plot the average test error (which is defined as the summation of the test error of each agent's model, then divided by the number of agents) vs. α in Fig. 8(a). The figure shows a visible phase transition for generalization ( ≈ 98% for α ≤ 6 , and ≈ 96 . 6% error for α ≥ 100 ). Fig. 8(a) shows that the transition from the kernel regime to the rich regime by varying the initialization scale may also exist in complex fully connected neural networks.

(2) The second one is that we use vanilla DSGD with batch size 128 to train the VGG11-like deep convectional neural network on CIFAR10 with small step size 10 -4 for 2000 epochs such that each local model achieves almost 100% training accuracy. The network setting is the same as the first experiment. The VGG11-like architecture is the same as [38]. Weights were initialized using Uniform He initialization multiplied by α . The Fig. 8(b) plots the average test accuracy vs. α . In addition, we adopt the sparse feature learning measure [4] to monitor the sparsity of learned features in all agents along the epoch in Fig. 8(c). From Fig. 8(c) and Fig. 8(d), we can observe that the implicit regularization of DGD under small initialization may promote neural networks to learn sparse features that can have better generalization performance, which can be regarded as the complementary observation in [4] under decentralized learning setting. How to prove this kind of implicit regularization of DSGD theoretically will be an interesting future work.

(a) MNIST test accuracy vs

α

(b) CIFAR10 test accuracy vs

α

(c) Feature sparsity vs

α

(d) Training curves for CIFAR10

MNIST Test Accuracy vs Scale

64

0.980

62

Averaged Test Accuracy (

(%)

0.978

0.976 -

Test Accuracy (%)

60

0.974

58

0.972

56

0.970

54

0.968

52

0.966

101

100

101

102

103

OS

1.0

1.2

1.4

9T

8T

2.0

α

Sparsity vs Epochs for Different Alpha Values

Test Accuracy vs Epochs for Different Alpha Values

0.925

60

α = 1

α = 1.5

= 1.75

0.900

α= 2

50

0.875

Test Accuracy (%)

Krysjeds

0.850

α = 1.5

α = 1

40

α = 1.75

0.825

α = 2

30

0.800

20

0.775

10

0.750

250

500

750

Epoch

1000

1250

1500

1750

2000

0

250

500

750

Epoch

1000

1250

1500

1750

2000

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer:[NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper proves the convergence of the decentralized gradient descent (DGD) algorithm under RIP condition when the initialization scale is small. The authors also propose a truncated version of the algorithm with a cheaper cost but comparable performance (in certain situations) to the original DGD.

STRENGTHS:
The theory seems sound and nontrivial. The authors prove the convergence for DGD, while previously it seems that only centralized GD results exist.

WEAKNESSES:
This paper is a little bit technical.

QUESTIONS:
There are already many technical explanations and comparisons in the paper. However, it would be better if more intuitive understanding of the convergence of DGD, and the comparison with centralized GD can be added.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-07

comments: SUMMARY:
This manuscript studies a decentralized optimization method for training linear sparse models using a network of agents that collect linear measurements. Unlike decentralized methods relying on L1 regularization, this approach leverages implicit regularization inherited in the gradient descent process. The authors propose a communication-efficient variant referred to as Truncated Decentralized Gradient Descent T-DGD. The authors analyze their decentralized version of gradient descent applied to a non-convex least squares formulation. The manuscript concludes with numerical results that validate the effectiveness of both DGD and T-DGD for sparse learning tasks.

STRENGTHS:
The manuscript is well-written and provides an interesting perspective on decentralized optimization of a linear least squares system.

WEAKNESSES:
It remains unclear to me how the approach promotes sparsity. This has neither been discussed nor numerically explored. Here, I would expect a comparison to l1-type regularization approaches promoting sparsity in w* but such comparison and sparsity levels are not provided.

QUESTIONS:
See weaknesses

LIMITATIONS:
yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on deriving the implicit regularization effects of decentralized gradient descent (DGD) for minimizing an objective function over undirected mesh networks. In particular, this paper establishes the fact that the solution returned by DGD with early stopping is statistically optimal under certain conditions. In addition, this paper also proposes a new method, the T-DGD, that can have better performance than vanilla DGD.\n\nSTRENGTHS:\n1.\tThe studying of the DGD setting is novel, as most works in the direction of implicit regularization of algorithms typically do not pay attention to DGD. The main contributions are clearly reflected in Theorem 1, showing the statistical guarantee and computational complexity of DGD. \n2.\tThe proposed TDGD can be seen as an interesting application of the theoretical observation, which additionally makes the theoretical claims inspiring.\n\nWEAKNESSES:\n1.\tThe simulations are only for diagonal linear networks, which limits the generalizability of the proposed methods and theoretical conclusions. \n2.\tThough results for the setting of this paper are novel, the motivation for studying such a setting is unclear. It would be better to specify clearly why and how the setting could be useful in practice. Currently it is hard for to see how the theoretical results could be important.\n\nQUESTIONS:\n1.\tCould you demonstrate your theoretical conclusions in more general architectures to further elaborate the implications of the theoretical claims?\n2.\tPrevious works shows a transition from kernel regime to rich regime by varying the initialization scale, does such phenomenon exist in the DGD setting?\n\nLIMITATIONS:\n1.\tThe motivation and implications of the theoretical claims are not clear.\n2.\tNumerical experiments are only conducted in simple architectures.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
The paper shows that the implicit regularization enjoyed by a well-known reparameterizion of least squares extends to the decentralized setting.  Convergence guarantees are provided, and it is also shown that communication can be limited by thresholding vectors before they are communicated to neighbors.

STRENGTHS:
The convergence guarantess end up having the form that one would hope.

WEAKNESSES:
I don't quite understand how this paper is contributing to our understanding of machine learning.  That running gradient descent on the nonlinear least-squares problem in (2) gives you sparse solutions that are statistically near-oprimal is interesting, but is of course known and written about a lot already.  That you can make gradient descent for non-convex problems decentralized using consesus averaging has also been written about a lot already.  That communications can be reduced using truncation when we are solving sparse regression problems is also an idea that has received plenty of attention.  This paper puts all of these things together, but we don't seem to learn anything new about implicit regularization or decentralized optimization.  The algorithm is not compared against existing decentralized sparse solvers.  If it happened to perform better even on stylized examples, that would at least be something new.

QUESTIONS:
none

LIMITATIONS:
none
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-13