Title: Does Worst-Performing Agent Lead the Pack? Analyzing Agent Dynamics in Unified Distributed SGD
Venue: NeurIPS 2024
Accepted: yes
Keywords: Distributed Optimization

Agent Dynamics

Federated Learning

Central Limit Theorem

Efficient Sampling
Abstract: Distributed learning is essential to train machine learning algorithms across *heterogeneous* agents while maintaining data privacy. We conduct an asymptotic analysis of Unified Distributed SGD (UD-SGD), exploring a variety of communication patterns, including decentralized SGD and local SGD within Federated Learning (FL), as well as the increasing communication interval in the FL setting. In this study, we assess how different sampling strategies, such as *i.i.d.* sampling, shuffling, and Markovian sampling, affect the convergence speed of UD-SGD by considering the impact of agent dynamics on the limiting covariance matrix as described in the Central Limit Theorem (CLT). Our findings not only support existing theories on linear speedup and asymptotic network independence, but also theoretically and empirically show how efficient sampling strategies employed by individual agents contribute to overall convergence in UD-SGD. Simulations reveal that a few agents using highly efficient sampling can achieve or surpass the performance of the majority employing moderately improved strategies, providing new insights beyond traditional analyses focusing on the worst-performing agent.

Does Worst-Performing Agent Lead the Pack? Analyzing Agent Dynamics in Unified Distributed SGD

Abstract

Distributed learning is essential to train machine learning algorithms across heterogeneous agents while maintaining data privacy. We conduct an asymptotic analysis of Unified Distributed SGD (UD-SGD), exploring a variety of communication patterns, including decentralized SGD and local SGD within Federated Learning (FL), as well as the increasing communication interval in the FL setting. In this study, we assess how different sampling strategies, such as i.i.d. sampling, shuffling, and Markovian sampling, affect the convergence speed of UD-SGD by considering the impact of agent dynamics on the limiting covariance matrix as described in the Central Limit Theorem (CLT). Our findings not only support existing theories on linear speedup and asymptotic network independence, but also theoretically and empirically show how efficient sampling strategies employed by individual agents contribute to overall convergence in UD-SGD. Simulations reveal that a few agents using highly efficient sampling can achieve or surpass the performance of the majority employing moderately improved strategies, providing new insights beyond traditional analyses focusing on the worst-performing agent.

1 Introduction

Distributed learning deals with the training of models across multiple agents over a communication network in a distributed manner, while addressing the challenges of privacy, scalability, and highdimensional data [11, 55]. Each agent i ∈ [ N ] holds a private dataset X i and an agent-specified loss function F i : R d × X i → R that depends on the model parameter θ ∈ R d and a data point X ∈ X i . The goal is then to find a local minima θ ∗ of the objective function f ( θ ) ≜ 1 N ∑ N i =1 f i ( θ ) , where agent i 's loss function f i ( θ ) ≜ E X ∼D i [ F i ( θ, X )] and D i represents the target distribution of data for agent i . 1 Each agent i can locally compute the gradient ∇ F i ( θ, X ) ∈ R d w.r.t. θ for every sampled data point X ∈ X i . Due to the distributed nature, {D i } i ∈ [ N ] and {X i } i ∈ [ N ] are not necessarily identically distributed over [ N ] so that the minima of each local function f i ( θ ) can be far away from L . This is particularly relevant in decentralized training data, e.g., Federated Learning (FL) with heterogeneous data across data centers or devices [81, 31].

In this paper, we focus on Unified Distributed SGD (UD-SGD), where each agent i ∈ [ N ] updates its model parameter θ i n +1 in a two-step process:

where γ n denotes the step size, X i n is the data sampled by agent i at time n (i.e., agent dynamics), and W n =[ w n ( i, j )] i,j ∈ [ N ] represents the doubly-stochastic communication matrix satisfying w n ( i, j ) ≥ 0 and 1 T W n = 1 T , W n 1 = 1 . In the special case of N = 1 , (1) simplifies to the vanilla SGD where W n = 1 for all n . UD-SGD covers a wide range of distributed algorithms, e.g., decentralized SGD (DSGD) [71, 80, 61, 68], distributed SGD with changing topology (DSGD-CT) [24, 43], local SGD (LSGD) in FL [55, 76], and its variant aimed at reducing communication costs (LSGD-RC) [51].

Versatile Communication Patterns { W n } : For visualization, we depict the scenarios of UD-SGD (1) in Figure 1. In DSGD, each agent (node) in the graph communicates with its neighbors after each SGD computation via W n , representing the underlying network topology. As a special case, central server-based aggregation, forming a fully connected network, translates W n into a rank-1 matrix W n = 11 T /N . To minimize communication expenses, FL variants allow each agent to perform multiple SGD steps before aggregation [55, 67, 76], resulting in a communication interval of length K and a consistent pattern W n = W for n = mK, ∀ m ∈ N , and W n = I N otherwise. In particular, i)

𝒊

𝑿𝒏

𝒊

𝒊

𝒋

𝒘𝒏(𝒊, 𝒋)

𝒘𝒏(𝒋, 𝒊)

𝑿𝒏

𝒋

𝒋

dataset

dataset

𝒳𝒋

Communication

pattern

𝑾𝑛

Heterogeneous

sampling

strategy

𝑋 𝑖

, 𝑋

𝑗

W = 11 T /N corresponds to LSGD with full agent participation (LSGD-FP) [76, 42, 51]; ii) W is a random matrix generated by partial agent participation (LSGD-PP) [55, 18, 74]; iii) W is generated by Metropolis-Hasting algorithm in decentralized setting, e.g., hybrid LSGD (HLSGD) [37, 32] and decentralized FL (DFL) [46, 77, 16]. We defer further discussion of W to Appendix F.1.

Markovian vs i.i.d. Sampling: Agents typically employ i.i.d. or Markovian sampling, as illustrated in the bottom brown box of Figure 1. In cases where agents have full access to their data, DSGD with i.i.d sampling has been extensively studied [60, 43, 61, 47]. In FL, many application-oriented LSGD variants have been investigated [51, 18, 77, 32, 37, 53]. However, these works solely focus on i.i.d. sampling, restricting their applicability to Markovian sampling scenarios.

Markovian sampling, which has received increased attention in limited settings (see Table 1), is vital where agents lack independent data access. For instance, in statistical applications, agents with an unknown a priori distribution often use Markovian sampling over i.i.d. sampling [40, 63]. In HLSGDacross device-to-device (D2D) networks [32, 37], random walks reduce communication costs compared to the frequent aggregations required by Gossip algorithms [38, 28, 4]. For single-agent scenarios, vanilla SGD with Markovian noise, as applied in a D2D network, has shown improved communication efficiency and privacy [69, 28, 35]. In contrast, for agents with full data access, Markov Chain Monte Carlo (MCMC) methods can be more efficient than i.i.d. sampling, especially in high-dimensional spaces with constraints [27, 40], where acceptance-rejection methods [12] lead to computational inefficiency (e.g., wasted samples) due to multiple rejections before obtaining a sample that satisfies constraints [26, 69]. In addition, shuffling methods can be considered as high-order Markov chains [38], which achieves faster convergence than i.i.d. sampling [1, 79, 78].

Limitations of Non-Asymptotic Analysis on Agent's Sampling Strategy: Recent studies on the non-asymptotic behavior of DSGD and LSGD variants under Markovian sampling, as summarized in Table 1, have made significant strides. However, these works often fall short in accurately revealing the statistical influence of each agent dynamics { X i n } on the performance of UD-SGD. For instance,

[71, 68] proposed the error bound O ( 1 / log 2 (1 /ρ ) n 1 -a ) , where a ∈ (0 . 5 , 1] and ρ denotes the identical mixing rate for all agents, overlooking agent heterogeneity in sampling strategy. A similar assumption to ρ is also evident in [42]. More recent contributions from [80, 72] have attempted to relax these constraints by considering a finite-time bound of O ( τ 2 mix / ( n +1)) , where τ mix is the mixing time of the slowest agent. This approach, however, inherently focuses on the worst-performing agent , neglecting how other agents with faster mixing rates might positively influence the system. 2 Such an analysis fails to capture the collective impact of other agents on the overall system performance,

a crucial aspect in large-scale applications where identifying and managing the worst-performing agent is challenging due to privacy concerns or sporadic unreachability. Since agents in distributed learning have the freedom to choose their sampling strategies, it's vital to understand how each agent's improved sampling approach contributes to the overall convergence speed of the UD-SGD algorithm. This understanding is key to enhancing system performance, particularly in large-scale machine learning scenarios where agent heterogeneity is a defining feature.

Rationale for Asymptotic Analysis: Recent trends in convergence analysis have leaned towards non-asymptotic methods, yet it's crucial to recognize the complementary role of asymptotic analysis for a better understanding of convergence behaviors, as highlighted in [9, 56, 25, 39]. For vanilla SGD, [59, 17] emphasized that central limit theorem (CLT) is far less asymptotic than it may appear under both i.i.d. and Markovian sampling. Notably, the limiting covariance matrix, a key statistical feature in vanilla SGD's CLT, also prominently features in high-probability bound [59], explicit finite-time bound [17] and 1 -Wasserstein distance in the non-asymptotic CLT [66]. [38] further underscored this by numerically showing that the limiting covariance matrix provides a more precise depiction of convergence than the mixing rates often used in finite-time upper bounds [26, 69]. Moreover, they argued that finite-time analysis may not suitably apply to certain efficient high-order Markov chains, due to the lack of comparative mixing-rate metrics.

Our Contributions: We present an asymptotic analysis of the UD-SGD algorithm (1) under heterogeneous agent dynamics { X i n } and a large family of communication patterns { W n } . Specifically,

Under appropriate assumptions, all agents performing (1) asymptotically reach the consensus and find θ ∗ : ∀ i ∈ [ N ] , θ n ≜ 1 N ∑ N i =1 θ i n denotes the average model parameter among all agents, we have

Moreover, we derive the CLT of UD-SGD in the form of

Our framework addresses technical challenges in quantifying consensus error under various communication patterns and slowly increasing communication interval. This shows a substantial extension compared to previous studies [58, 43, 51], particularly in regulating the growth of communication

intervals (Assumption 2.3-ii) and in proving the scaled consensus error's boundedness (Lemma B.1). Furthermore, we reformulate UD-SGD as a stochastic approximation-like iteration and tackle the Markovian noise term using the Poisson equation, a technique previously confined only to vanilla SGD with Markovian sampling [17, 38, 52]. The key here is to devise the noise decomposition that separates the consensus error among all agents from the error caused by the bias from the Markov chain, which aligns with the target distribution only asymptotically at infinity, not at finite times.

In analyzing (3), we derive the exact form of V as 1 N 2 ∑ N i =1 V i . Here, V i is the limiting covariance matrix of agent i , which depends mainly on its sampling strategy { X i n } . This allows us to show that improving individual agents' sampling strategy can reduce the covariance in CLT, which in turn implies a smaller mean-square error (MSE) for large time n . This is a significant advancement over previous finite-sample bounds that only account for the worst-performing agent and do not fully capture the effect of individual agent dynamics on overall system performance. Our CLT result (3) also treats recent findings in [38] as a very special case with N = 1 , where the relationship therein between the sampling efficiency of the Markov chain and the limiting covariance matrix in the CLT of vanilla SGD, can carry over to our UD-SGD.

Wedemonstrate that our analysis supports recent findings from studies such as [42], which exhibited linear speedup scaling with the number of agents under LSGD-FP with Markovian sampling; and [62, 61], which examined the notion of 'asymptotic network independence' for DSGD with i.i.d. sampling, where the convergence of the algorithm (1) at large time n depends solely on the left eigenvector of W n ( 1 N 1 considered in this paper) rather than the specific communication network topology encoded in W n , but now under Markovian sampling. We extend these findings in view of CLT to a broader range of communication patterns { W n } and general sampling strategies { X i n } .

We conduct numerical experiments using logistic regression and neural network training with several choices of agents' sampling strategies, including a recently proposed one via nonlinear Markov chain [25]. Our results uncover a key phenomenon: a handful of compliant agents adopting highly efficient sampling strategies can match or exceed the performance of the majority using moderately improved strategies. This finding is crucial for practical optimization in large-scale learning systems, moving beyond the current literature that only considers the worst-performing agent in more restrictive settings.

2 Preliminaries

Basic Notations: We use ∥ v ∥ to indicate the Euclidean norm of a vector v ∈ R d and ∥ M ∥ to indicate the spectral norm of a matrix M ∈ R d × d . The identity matrix of dimension d is denoted by I d , and the all-one (resp. all-zero) vector of dimension N is denoted by 1 (resp. 0 ). Let J ≜ 11 T /N . The diagonal matrix with the entries of v on the main diagonal is written as diag ( v ) . We also use ' ⪰ ' for Loewner ordering such that A ⪰ B is equivalent to x T ( A -B ) x ≥ 0 for any x ∈ R d .

Asymptotic Covariance Matrix: Asymptotic variance is a widely used metric for evaluating the second-order properties of Markov chains associated with a scalar-valued test function in the MCMC literature, e.g., Chapter 6.3 [12], and asymptotic covariance matrix is its multivariate version for a vector-valued function. Specifically, we consider a finite, irreducible, aperiodic and positive recurrent (ergodic) Markov chain { X n } n ≥ 0 with transition matrix P and stationary distribution π , and the estimator ˆ µ n ( g ) ≜ 1 n ∑ n -1 s =0 g ( X s ) for any vector-valued function g : [ N ] → R d . According to the ergodic theorem [12, 13], we have lim n →∞ ˆ µ n ( g ) = E π ( g ) a.s.. As defined in [13, 38], the asymptotic covariance matrix Σ X ( g ) for a vector-valued function g ( · ) is given by

where ∆ n ≜ ∑ n -1 s =0 ( g ( X s ) -E π ( g )) . By following the algebraic manipulations in [12, Theorem 6.3.7] for asymptotic variance (univariate version), we can rewrite (4) in a matrix form such that

where G ≜ [ g (1) , · · · , g ( N )] T ∈ R N × d and Z ≜ [ I N -P + 1 π T ] -1 . This matrix form explicitly shows the dependence on the transition matrix P and its stationary distribution π , and will be utilized in our Theorem 3.3.

Model Description: The UD-SGD in (1) can be expressed in a compact iterative form, i.e., we have

̸

at each time n , where each agent i samples according to its own Markovian trajectory { X i n } n ≥ 0 with stationary distribution π i such that E X ∼ π i [ F i ( θ, X )] = f i ( θ ) . Let K l denote the communication interval between the ( l -1) -th and l -th aggregation among N agents, and n l ≜ ∑ l m =1 K m be the time instance for the l -th aggregation. We also define τ n ≜ min l { l : n l ≥ n } as the index of the upcoming aggregation at time n such that K τ n indicates the communication interval for the τ n -th aggregation, or more precisely, the length of the communication interval that includes the time index n . The communication pattern follows that W n = I n if n = n l and W n = W otherwise for l ≥ 1 , where the examples of W will be discussed in Appendix F.1. Note that i) when K l = 1 , (6) reduces to DSGD; ii) when K l = K > 1 , (6) becomes the local SGD in FL. iii) When K l increases with l , we recover some choices of K l studied in [51] beyond LSGD-RC with i.i.d. sampling. This increasing communication interval aims to further reduce the frequency of aggregation among agents for lower communication costs, but now under a Markovian sampling setting and a wider range of communication patterns. We below state the assumptions needed for the main theoretical results.

Assumption 2.1 (Regularity of the gradient) . For each i ∈ [ N ] and X ∈ X i , the function F i ( θ, X ) is L-smooth in terms of θ , i.e., for any θ 1 , θ 2 ∈ R d ,

In addition, we assume that the objective function f is twice continuously differentiable and µ -strongly convex only around the local minima θ ∗ ∈ L , i.e.,

Assumption 2.1 imposes the regularity conditions on the gradient ∇ F i ( · , X ) and Hessian matrix of the objective function f ( · ) , as is commonly assumed in [10, 45, 29, 38]. Note that (7) requires per-sample Lipschitzness of ∇ F i and is stronger than the Lipschitzness of its expected version ∇ f i , which is commonly assumed under i.i.d sampling setting [73, 50, 30]. However, we remark that this is in line with previous work on DSGD and LSGD-FP under Markovian sampling as well [71, 42, 80], because ∇ F i ( θ, X ) is no longer the unbiased stochastic version of ∇ f i ( θ ) and the effect of { X i n } has to be taken into account in the analysis. The local strong convexity at the minimizer is commonly assumed to analyze the convergence of the algorithm under both asymptotic and non-asymptotic analysis [10, 29, 38, 45, 52, 80].

̸

Assumption 2.2 (Ergodicity of Markovian sampling) . { X i n } n ≥ 0 is an ergodic Markov chain with stationary distribution π i such that E X ∼ π i [ F i ( θ, X )] = f i ( θ ) , and is independent from { X j n } n ≥ 0 , j = i .

The ergodicity of the underlying Markov chains, as stated in Assumption 2.2, is commonly assumed in the literature [26, 69, 80, 42, 38]. This assumption ensures the asymptotic unbiasedness of the loss function F i ( θ, · ) , which takes i.i.d. sampling as a special case.

Assumption 2.3 (Decreasing step size and slowly increasing communication interval) . i) For bounded communication interval K τ n ≤ K, ∀ n , we assume the polynomial step size γ n = 1 /n a and a ∈ (0 . 5 , 1] ; Or ii) If K τ n →∞ as n →∞ , we assume γ n = 1 /n and define η n = γ n K L +1 τ n , where the sequence { K l } l ≥ 0 satisfies ∑ n η 2 n < ∞ , K τ n = o ( γ -1 / 2( L +1) n ) , and lim l →∞ η n l +1 /η n l +1 +1 = 1 .

In Assumption 2.3, the polynomial step size γ n is standard in the literature and it has the property ∑ n γ n = ∞ , ∑ n γ 2 n < ∞ [17, 38]. Inspired by [51], we introduce η n to control the step size within each l -th communication interval with length K l to restrict the growth of K l . Specifically, ∑ n η 2 n < ∞ and K τ n = o ( γ -1 / 2( L +1) n ) ensure that η n → 0 and K τ n does not increase too fast in n . lim l →∞ η n l +1 /η n l +1 +1 = 1 sets the restriction on the increment from n l to n l +1 . Several practical forms of K l suggested by [51], including K l ∼ log( l ) and K l ∼ log log( l ) , also satisfy Assumption 2.3-ii). We defer to Appendix A the mathematical verification of these two types of K l , together with the practical implications of increasing communication interval K l .

Remark 1. In Assumption 2.3, we incorporate an increasing communication interval along with a step size γ n = 1 /n . This complements the choice of step size γ n in [51, Assumption 3 . 3 ], where γ n = 1 /n a for a ∈ (0 . 5 , 1) . It is important to note, however, that the increasing communication interval specified in [51, Assumption 3 . 2 ] is applicable only in i.i.d sampling. Under the Markovian

sampling framework, the expression ∇ F i ( θ, X ) -∇ f i ( θ ) loses its unbiased and Martingale difference properties. Consequently, the Martingale CLT application as utilized by [51] does not directly extend to Markovian sampling. To address this, we adapted techniques from [29, 58] to accommodate the increasing communication interval within the Markovian sampling setting and various communication patterns. This adaptation necessitates γ n = 1 /n , a specification not covered in [51]. Exploring more general forms of K l that could relax this assumption is outside the scope of our current study.

Assumption 2.4 (Stability on model parameter) . We assume sup n ∥ θ i n ∥ < ∞ almost surely ∀ i ∈ [ N ] .

Assumption 2.4 claims that the sequence of { θ i n } always remains in a path-dependent compact set. It is to ensure the stability of the algorithm that serves the purpose of analyzing the convergence, which is often assumed under the asymptotic analysis of vanilla SGD with Markovian noise [23, 29, 52]. As mentioned in [58, 70], checking Assumption 2.4 is challenging and requires case-by-case analysis, even under i.i.d. sampling. Only recently the stability of SGD under Markovian sampling has been studied in [9], but the result for UD-SGD remains unknown in the literature. Thus, we analyze each agent's sampling strategy in the asymptotic regime under this stability condition.

Assumption 2.5 (Contraction property of communication matrix) . i). { W n } n ≥ 0 is independent of the sampling strategy { X i n } n ≥ 0 for all i ∈ [ N ] and is assumed to be doubly-stochastic for all n ; ii). At each aggregation step n l , W n l is independently generated from some distribution P n l such that ∥ E W ∼P n l [ W T W ] -J ∥≤ C 1 < 1 for some constant C 1 .

The doubly-stochasticity of W n in Assumption 2.5-i) is widely assumed in the literature [54, 24, 43, 80]. Assumption 2.5-ii) is a contraction property to ensure that agents employing UD-SGD will asymptotically achieve the consensus, which is also common in [7, 24, 80]. Examples of W that satisfy Assumption 2.5-ii), e.g., Metropolis-Hasting matrix, partial agent participation in FL, are deferred to Appendix F.1 due to space constraint.

3 Asymptotic Analysis of UD-SGD

Almost Sure Convergence: Let θ n ≜ 1 N ∑ N i =1 θ i n represent the consensus among all the agents at time n , we establish the asymptotic consensus of the local parameters θ i n , as stated in Lemma 3.1.

Lemma 3.1. Under Assumptions 2.1, 2.3, 2.4 and 2.5, the consensus error θ i n -θ n diminishes to zero at the rate specified below: Almost surely, for every agent i ∈ [ N ] ,

Lemma 3.1 indicates that all agents asymptotically reach consensus at a rate of O ( γ n ) (or O ( η n ) ). This finding extends the scope of [58, Proposition 1], incorporating considerations for Markovian sampling, FL settings, and increasing communication interval K l . The proof, detailed in Appendix B, primarily tackles the challenge of establishing the boundedness of the sequences { γ -1 n ( θ i n -θ n ) } (or { η -1 n ( θ i n -θ n ) } ) almost surely for all i ∈ [ N ] . This is specifically analyzed in Lemma B.1. Next, with additional Assumption 2.2, we are able to obtain the almost sure convergence of θ n to θ ∗ ∈ L .

Theorem 3.2 is achieved by decomposing the Markovian noise term ∇ F i ( θ i n , X i n ) -∇ f i ( θ i n ) , using the Poisson equation technique as discussed in [6, 29, 17], into a Martingale difference noise term, along with additional noise terms. We then reformulate (6) into an iteration akin to stochastic approximation, as depicted in (56). The subsequent step involves verifying the conditions on these noise terms under our stated assumptions. Crucially, this theorem also establishes that UD-SGD ensures an almost sure convergence of each agent to a local minimum θ ∗ ∈ L , even in scenarios where the communication interval K l gradually increases, in accordance with Assumption 2.3-ii). The detailed proof of this theorem is provided in Appendix C.

Central Limit Theorem: Let U i ≜ Σ X i ( ∇ F i ( θ ∗ , · )) represent the asymptotic covariance matrix (defined in (5)) associated with each agent i ∈ [ N ] , given their sampling strategy { X i n } and function ∇ F i ( θ ∗ , · ) . Define U ≜ 1 N 2 ∑ N i =1 U i . We assume the polynomial step-size γ n ∼ γ ⋆ /n a , a ∈ (0 . 5 , 1] and γ ⋆ > 0 . In the case of a = 1 , we further assume γ ⋆ > 1 / 2 µ , where µ is defined in (8). For notational simplicity, and without loss of generality, our remaining CLT result is stated while conditioning on the event that { θ n → θ ∗ } for some θ ∗ ∈ L .

Theorem 3.3. Let Assumptions 2.1 - 2.5 hold. Then,

where the limiting covariance matrix V is in the form of

Here, we have M = -H if a ∈ (0 . 5 , 1) , or M = I d / 2 γ ⋆ -H if a = 1 , where H is defined in (8) .

Moreover, let ¯ θ n = 1 n ∑ n -1 s =0 θ s and V ′ = H -1 UH -1 . For a ∈ (0 . 5 , 1) , we have

The proof, presented in Appendix D, addresses the technical challenges in deriving the CLT for UD-SGD, specifically the second-order conditions in decomposing the Markovian noise term, which is not present in the i.i.d. sampling case [58, 43, 51]. We decompose ∇ F i ( θ n , X i n ) -∇ f i ( θ n ) into three parts in (48) using Poisson equation: e i n +1 , ν i n +1 , ξ i n +1 . The consensus error θ i n -θ n embedded in noise terms e i n +1 and ξ i n +1 is a new factor, whose characteristics have been quantified in our Lemma 3.1 but are not present in the single-agent scenario analyzed as an application of stochastic approximation in [22, 29]. The specifics of this analysis are expanded upon in Appendices D.1 to D.3. We require γ ⋆ > 1 / 2 µ for a =1 to ensure that the largest eigenvalue of M is negative, as this is a necessary condition for the existence of V in (12) (otherwise integration diverges). In the case where there is only one agent ( N =1 ), V and V ′ reduce to the matrices specified in the CLT result of vanilla SGD [29, 38, 52]. In addition, for a special case of constant communication interval in Assumption 2.3-i) and i.i.d. sampling as shown in Table 1, we recover the CLT of LSGD-RC in [51]. See Appendix E for detailed discussions.

Theorem 3.3 has significant implications for the MSE of { θ n } for large time n , i.e., E [ ∥ θ n -θ ∗ ∥ 2 ]= ∑ d i =1 e T i E [( θ n -θ ∗ )( θ n -θ ∗ ) T ] e i ≈ γ n ∑ d i =1 e T i Ve i = γ n Tr ( V ) , where e i is the d -dimensional vector of all zeros except 1 at the i -th entry. This indicates that a smaller limiting covariance matrix V , according to the Loewner order, results in a smaller trace of V and consequently in a reduced MSE for large n . Consideration for smaller V will be presented in the next section, where agents have the opportunity to improve their sampling strategies.

Remark 2. Studies by [62, 61] have shown that in DSGD with a fixed doubly-stochastic matrix W , the influence of communication topology diminishes after a transient period. Our Theorem 3.3 extends these findings to Markovian sampling and a broader spectrum of communication patterns as in Table 1. This extension is based on the fact that the consensus error, impacted primarily by the communication pattern, decreases faster than the CLT scale O ( √ γ n ) and is thus not the dominant factor in the asymptotic regime, as suggested by Lemma 3.1.

Remark 3. Recent studies have highlighted linear speedup with increasing number of agents N in the dominant term of their finite-sample error bounds under DSGD-CT with i.i.d. sampling [43] and LSGD-FC with Markovian sampling [42]. However, our Theorem 3.3 demonstrates this phenomenon under more diverse communication patterns and Markovian sampling in Table 1 via the leading term

V in our CLT. Specifically, it scales with 1 /N , i.e. V V /N , where V = N i =1 V i denotes the average limiting covariance matrices across all N agents and V i = ∫ ∞ 0 e M t U i e M t dt , suggesting that the MSE E [ ∥ θ n -θ ∗ ∥ 2 ] will be improved by 1 /N . A similar argument also applies to V ′ in (13) , i.e., V ′ = ¯ V ′ /N , where ¯ V ′ = 1 N ∑ N i =1 V ′ i and V ′ i = H -1 U i H -1 .

Impact of Agent's Sampling Strategy: In the literature, the mixing time-based technique has been widely used in the non-asymptotic analysis in SGD, DSGD and various LSGD variants in FL [26, 69, 68, 80, 42], i.e., for each agent i ∈ [ N ] and some constant C ,

where ρ i is the mixing rate of the underlying Markov chain. However, typical non-asymptotic analyses often rely on ρ ≜ max i ρ i among N agents, i.e., the worst-performing agent in their finite-time bounds [80, 72], or assume an identical mixing rate across all N agents [42, 68].

In contrast, Remark 3 highlights that each agent holds its own limiting covariance matrices V i and V ′ i , which are predominantly governed by the matrix U i , capturing the agent's sampling strategy

{ X i n } and contributing equally to the overall performance of UD-SGD. For each agent i , denote by U X i and U Y i the asymptotic covariance matrices associated with two candidate sampling strategies { X i n } and { Y i n } , respectively. Let V X and V Y be the limiting covariance matrices of the distributed system in (12), where agent i employs { X i n } and { Y i n } , respectively, while keeping other agents' sampling strategies unchanged. Then, we have the following result.

Corollary 3.4. For agent i , if there exist two sampling strategies { X i n } n ≥ 0 and { Y i n } n ≥ 0 such that U X i ⪰ U Y i , we have V X ⪰ V Y .

Corollary 3.4 directly follows from the definition of Loewner ordering, and Loewner ordering being closed under addition (i.e., A ⪰ B implies A + C ⪰ B + C ). It demonstrates that even a single agent improves its sampling strategy from { X i n } to { Y i n } , it leads to an overall reduction in V (in terms of Loewner ordering), thereby decreasing the MSE and benefiting the entire group of N agents. The subsequent question arises: How do we identify an improved sampling strategy { Y i n } over the baseline { X i n } ?

This question has been partially addressed by [57, 48, 38], which qualitatively investigates the 'efficiency ordering' of two sampling strategies. In particular, [38, Theorem 3 . 6 (i)] shows that sampling strategy { Y n } is more efficient than { X n } if and only if Σ X ( g ) ⪰ Σ Y ( g ) for any vector-valued function g ( · ) ∈ R d . Consequently, in the UD-SGD framework, employing a more efficient sampling strategy { Y i n } over the baseline { X i n } by agent i leads to Σ X i ( ∇ F i ( θ ∗ , · )) ⪰ Σ Y i ( ∇ F i ( θ ∗ , · )) , thus satisfying U X i ⪰ U Y i . This finding, as per Corollary 3.4, implies an overall improvement in UD-SGD.

For illustration purposes, we list a few examples where two competing sampling strategies follow efficiency ordering: i) When an agent has complete access to the entire dataset (e.g., deep learning), shuffling techniques like single shuffling and random reshuffling are more efficient than i.i.d. sampling [38, 78]; ii) When an agent works with a graph-like data structure and employs a random walk, e.g., agent i in Figure 1, using non-backtracking random walk (NBRW) is more efficient than simple random walk (SRW) [48]. iii) A recently proposed self-repellent random walk (SRRW) is shown to achieve near-zero sampling variance, indicating even higher sampling efficiency than NBRW and SRW [25]. 3 This random-walk-based sampling finds a particular application in large-scale FL within D2D networks (e.g., mobile networks, wireless sensor networks), where each agent acts as an edge server or access point, gathering information from the local D2D network [37, 32]. Employing a random walk over local D2D network for each agent constitutes the sampling strategy.

Theorem 3.3 and Corollary 3.4 not only qualitatively compare these sampling strategies but also allow for a quantitative assessment of the overall system enhancement. Since every agent contributes equally to the limiting covariance matrix V of the distributed system as in Remark 3, a key application scenario is to encourage a subset of compliant agents to adopt highly efficient strategies like SRRW, potentially yielding better performance than universally upgrading to slightly improved strategies like NBRW. This approach, more feasible and impactful in large-scale machine learning scenarios where some agents cannot freely modify their sampling strategies, is a unique aspect of our framework not addressed in previous works focusing on the worst-performing agent [80, 42, 68, 72].

4 Experiments

In this section, we empirically evaluate the effect of agents' sampling strategies under various communication patterns in UD-SGD. We consider the L 2 -regularized binary classification problem

where the feature vector x i,j and its corresponding label y i,j are held by agent i , with a penalty parameter κ set to 1 . We use the ijcnn1 dataset [14] with 22 features in each data point and 50 k data points in total, which is evenly distributed to two groups with 50 agents each ( N = 100 agents

10000

12500

15000

17500

20000

22500

25000

27500

30000

Number of steps (n)

1.0

1.5

2.0

2.5

3.0

MSE

n

*

2

1e

5

50 SRW

50 NBRW

40 SRW, 10 SRRW

30 SRW, 20 SRRW

10 1

10 2

10 3

Number of steps (n)

10

5

10

4

10

3

10

2

10

1

10 0

MSE

n

*

2

DSGD-VT

LSGD-FP

DFL

in total) and each agent holds B = 500 distinct data points. Each agent in the first group has full access to its entire dataset, and thus can employ i.i.d. sampling (baseline) or single shuffling. On the other hand, each agent in the other group has a graph-like structure and uses SRW (baseline), NBRWor SRRW with reweighting to sample its local dataset with uniform weight. In this simulation, we assume that agents can only communicate through a communication network using the DSGD algorithm. This scenario with heterogeneous agents, as depicted in Figure 1, is of great interest in large-scale machine learning [37, 32]. In addition, we employ a decreasing step size γ n = 1 /n in our UD-SGD framework (1) because it is typically used for the strongly convex objective function and is tested to have the fastest convergence in this simulation setup. Due to space constraints, we defer detailed simulation setup, including the introduction of SRW, NBRW, and SRRW, to Appendix G.1.

The simulation results are obtained through 120 independent trials. In Figure 2(a), we assume that the first group of agents perform either i.i.d. sampling or shuffling method, while the other group of agents all change their sampling strategies from baseline SRW to NBRW and SRRW, as shown in the legend. This plot shows that improved sampling strategy leads to overall convergence speedup since NBRW and SRRW are more efficient than SRW [38, 25]. Furthermore, it illustrates that SRRW is significantly more efficient than NBRW in this simulation setup, i.e., SRRW ≫ NBRW > SRW in terms of sampling efficiency. While keeping the second group of agents unchanged, we can see that shuffling method outperforms i.i.d. sampling with smaller asymptotic MSE. However, shuffling method may not perform perfectly for small time n due to slow mixing behavior in the initial period, which is also observed in the single-agent scenario in [65, 1, 38]. The error bar therein also indicates that the random-walk sampling strategy has a significant impact on the overall system performance and SRRW has smaller variance than NBRW and SRW.

̸

In Figure 2(b), we let the first group of agents perform i.i.d. sampling while only changing a portion of agents in the second group to upgrade from SRW to SRRW, e.g., 30 SRW 20 SRRW in the legend means that there are 30 agents using SRW while the rest 20 agents in the second group upgrade to SRRW. We observe that more agents willing to upgrade from SRW to SRRW lead to smaller asymptotic MSE, as predicted by Theorem 3.3 and Remark 3. This improvement in MSE reduction doesn't scale linearly with more agents adopting SRRW because each agent holds its own dataset that are not necessarily identical, resulting in different individual limiting covariance matrices V i = V j .

While maintaining i.i.d. sampling for the first group of agents, we compare the performance when the second group of agents in Figure 2(c) employ NBRW or SRRW. Remarkably, the case with only 10 agents out of 50 agents in the second group adopting far more efficient sampling strategy ( 40 SRW, 10 SRRW) through incentives or compliance already produces a smaller MSE than all 50 agents using slightly better strategy ( 50 NBRW). The performance gap becomes even more pronounced

when 20 agents upgrade from SRW to SRRW ( 30 SRW, 20 SRRW). We show that the performance of a distributed system can be improved significantly when a small proportion of agents adopt highly efficient sampling strategies.

Figure 2(d) empirically illustrates the asymptotic network independence property via four algorithms under our UD-SGD framework: Centralized SGD (communication interval K = 1 , communication matrix W = 11 T /N ); LSGD-FP (FL with full client participation, K = 5 , W = 11 T /N ); DSGDVT (DSGD with time-varying topologies, randomly chosen from 5 doubly stochastic matrices); DFL (decentralized FL with fixed MH-generated W and increasing communication interval K l = max { 1 , log( l ) } after l -th aggregation). We fix the sampling strategy (shuffling, SRRW) throughout this plot. All four algorithms overlap around 1000 steps, implying that they have entered the asymptotic regime with similar performance where the CLT result dominates, implying the asymptotic network independence in the long run.

Figure 2(e) and 2(f) show the performance of different sampling strategies in DSGD-VT and DFL algorithms in terms of MSE. Both plots consistently demonstrate that improving agent's sampling strategies (e.g., shuffling > iid sampling, and SRRW > NBRW > SRW) leads to faster convergence with smaller MSE, supporting our theory.

Furthermore, in Appendix G.2, we simulate an image classification task with CIFAR-10 dataset [44] by training a 5 -layer CNN and ResNet-18 model collaboratively through a 10 -agent network. The result is illustrated in Figure 3, where SRRW outperforms NBRW and SRW as expected. In summary, we find that upgrading even a small portion of agents to efficient sampling strategies (e.g., shuffling method, NBRW, SRRW under different dataset structures) improves system performance in UD-SGD. These results are consistent in binary and image classification tasks, underscoring that every agent matters in distributed learning.

5 Conclusion

In this work, we develop an UD-SGD framework that establishes the CLT of various distributed algorithms with Markovian sampling. We overcome technical challenges such as quantifying consensus error under very general communication patterns and decomposing Markovian noise through the Poisson equation, which extends the analysis beyond the single-agent scenario. We demonstrate that even if only a few agents optimize their sampling strategies, the entire distributed system will benefit with a smaller limiting covariance in the CLT, suggesting a reduced MSE. This finding challenges the current established upper bounds where the worst-performing agent leads the pack. Future studies could pivot towards developing fine-grained finite-time bounds to individually characterize each agent's behavior, and theoretically analyze the effect of SRRW in UD-SGD.

6 Acknowledgments and Disclosure of Funding

We thank the anonymous reviewers for their constructive comments. This work was supported in part by National Science Foundation under Grant Nos. CNS-2007423, IIS-1910749, and IIS-2421484.

References

Kwangjun Ahn, Chulhee Yun, and Suvrit Sra. Sgd with shuffling: optimal rates without component convexity and large epoch requirements. In Proceedings of the 34th International Conference on Neural Information Processing Systems , pages 17526-17535, 2020.

Noga Alon, Itai Benjamini, Eyal Lubetzky, and Sasha Sodin. Non-backtracking random walks mix faster. Communications in Contemporary Mathematics , 9(04):585-603, 2007.

Christophe Andrieu, Éric Moulines, and Pierre Priouret. Stability of stochastic approximation under verifiable conditions. SIAM Journal on control and optimization , 44(1):283-312, 2005.

Ghadir Ayache, Venkat Dassari, and Salim El Rouayheb. Walk for learning: A random walk approach for federated learning from heterogeneous data. IEEE Journal on Selected Areas in Communications , 41(4):929-940, 2023.

Anna Ben-Hamou, Eyal Lubetzky, and Yuval Peres. Comparing mixing times on sparse random graphs. In Proceedings of the Twenty-Ninth Annual ACM-SIAM Symposium on Discrete Algorithms , pages 1734-1740. SIAM, 2018.

Albert Benveniste, Michel Métivier, and Pierre Priouret. Adaptive algorithms and stochastic approximations , volume 22. Springer Science & Business Media, 2012.

Pascal Bianchi, Gersende Fort, and Walid Hachem. Performance of a distributed stochastic approximation algorithm. IEEE Transactions on Information Theory , 59(11):7405-7418, 2013.

Patrick Billingsley. Convergence of probability measures . John Wiley & Sons, 2013.

Vivek Borkar, Shuhang Chen, Adithya Devraj, Ioannis Kontoyiannis, and Sean Meyn. The ode method for asymptotic statistics in stochastic approximation and reinforcement learning. arXiv preprint arXiv:2110.14427 , 2021.

V.S. Borkar. Stochastic Approximation: A Dynamical Systems Viewpoint: Second Edition . Texts and Readings in Mathematics. Hindustan Book Agency, 2022.

Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein, et al. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends® in Machine learning , 3(1):1-122, 2011.

Pierre Brémaud. Markov chains: Gibbs fields, Monte Carlo simulation, and queues , volume 31. Springer Science & Business Media, 2013.

Steve Brooks, Andrew Gelman, Galin Jones, and Xiao-Li Meng. Handbook of markov chain monte carlo . CRC press, 2011.

Chih-Chung Chang and Chih-Jen Lin. Libsvm: a library for support vector machines. ACM transactions on intelligent systems and technology (TIST) , 2(3):1-27, 2011.

VijaySekhar Chellaboina and Wassim M Haddad. Nonlinear dynamical systems and control: A Lyapunov-based approach . Princeton University Press, 2008.

Vishnu Pandi Chellapandi, Antesh Upadhyay, Abolfazl Hashemi, and Stanislaw H Zak. On the convergence of decentralized federated learning under imperfect information sharing. arXiv preprint arXiv:2303.10695 , 2023.

Shuhang Chen, Adithya Devraj, Ana Busic, and Sean Meyn. Explicit mean-square error bounds for monte-carlo and linear stochastic approximation. In International Conference on Artificial Intelligence and Statistics , pages 4173-4183. PMLR, 2020.

Wenlin Chen, Samuel Horváth, and Peter Richtárik. Optimal client sampling for federated learning. Transactions on Machine Learning Research , 2022.

Anna Choromanska, Mikael Henaff, Michael Mathieu, Gérard Ben Arous, and Yann LeCun. The loss surfaces of multilayer networks. In Artificial intelligence and statistics , pages 192-204, 2015.

Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional nonconvex optimization. In Advances in neural information processing systems , volume 27, 2014.

Burgess Davis. On the intergrability of the martingale square function. Israel Journal of Mathematics , 8:187-190, 1970.

Bernard Delyon. Stochastic approximation with decreasing gain: Convergence and asymptotic theory. Technical report, 2000.

Bernard Delyon, Marc Lavielle, and Eric Moulines. Convergence of a stochastic approximation version of the em algorithm. Annals of statistics , pages 94-128, 1999.

Thinh Doan, Siva Maguluri, and Justin Romberg. Finite-time analysis of distributed td (0) with linear function approximation on multi-agent reinforcement learning. In International Conference on Machine Learning , pages 1626-1635. PMLR, 2019.

Vishwaraj Doshi, Jie Hu, and Do Young Eun. Self-repellent random walks on general graphsachieving minimal sampling variance via nonlinear markov chains. In International Conference on Machine Learning . PMLR, 2023.

John C Duchi, Alekh Agarwal, Mikael Johansson, and Michael I Jordan. Ergodic mirror descent. SIAM Journal on Optimization , 22(4):1549-1578, 2012.

Martin Dyer, Alan Frieze, Ravi Kannan, Ajai Kapoor, Ljubomir Perkovic, and Umesh Vazirani. A mildly exponential time algorithm for approximating the number of solutions to a multidimensional knapsack problem. Combinatorics, Probability and Computing , 2(3):271-284, 1993.

Mathieu Even. Stochastic gradient descent under markovian sampling schemes. In International Conference on Machine Learning , 2023.

Gersende Fort. Central limit theorems for stochastic approximation with controlled markov chain dynamics. ESAIM: Probability and Statistics , 19:60-80, 2015.

Yann Fraboni, Richard Vidal, Laetitia Kameni, and Marco Lorenzi. A general theory for client sampling in federated learning. In International Workshop on Trustworthy Federated Learning in conjunction with IJCAI , 2022.

Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. In Proceedings of the 34th International Conference on Neural Information Processing Systems , pages 19586-19597, 2020.

Yuanxiong Guo, Ying Sun, Rui Hu, and Yanmin Gong. Hybrid local SGD for federated learning with heterogeneous communications. In International Conference on Learning Representations , 2022.

P. Hall, C.C. Heyde, Z.W. Birnbauam, and E. Lukacs. Martingale Limit Theory and Its Application . Communication and Behavior. Elsevier Science, 2014.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Hadrien Hendrikx. A principled framework for the design and analysis of token algorithms. In International Conference on Artificial Intelligence and Statistics , pages 470-489. PMLR, 2023.

Roger A. Horn and Charles R. Johnson. Topics in Matrix Analysis . Cambridge University Press, 1991.

Seyyedali Hosseinalipour, Sheikh Shams Azam, Christopher G Brinton, Nicolo Michelusi, Vaneet Aggarwal, David J Love, and Huaiyu Dai. Multi-stage hybrid federated learning over large-scale d2d-enabled fog networks. IEEE/ACM Transactions on Networking , 2022.

Jie Hu, Vishwaraj Doshi, and Do Young Eun. Efficiency ordering of stochastic gradient descent. In Advances in Neural Information Processing Systems , 2022.

Jie Hu, Vishwaraj Doshi, and Do Young Eun. Accelerating distributed stochastic optimization via self-repellent random walks. In The Twelfth International Conference on Learning Representations , 2024.

Mark Jerrum and Alistair Sinclair. The markov chain monte carlo method: an approach to approximate counting and integration. Approximation Algorithms for NP-hard problems, PWS Publishing , 1996.

Thomas Kailath, Ali H Sayed, and Babak Hassibi. Linear estimation . Prentice Hall, 2000.

Sajad Khodadadian, Pranay Sharma, Gauri Joshi, and Siva Theja Maguluri. Federated reinforcement learning: Linear speedup under markovian sampling. In International Conference on Machine Learning , pages 10997-11057, 2022.

Anastasia Koloskova, Nicolas Loizou, Sadra Boreiri, Martin Jaggi, and Sebastian Stich. A unified theory of decentralized sgd with changing topology and local updates. In International Conference on Machine Learning , pages 5381-5393, 2020.

Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, 2009.

Harold Kushner and G George Yin. Stochastic approximation and recursive algorithms and applications , volume 35. Springer Science & Business Media, 2003.

Anusha Lalitha, Shubhanshu Shekhar, Tara Javidi, and Farinaz Koushanfar. Fully decentralized federated learning. In Advances in neural information processing systems , 2018.

Batiste Le Bars, Aurélien Bellet, Marc Tommasi, Erick Lavoie, and Anne-Marie Kermarrec. Refined convergence and topology learning for decentralized sgd with heterogeneous data. In International Conference on Artificial Intelligence and Statistics , pages 1672-1702. PMLR, 2023.

Chul-Ho Lee, Xin Xu, and Do Young Eun. Beyond random walk and metropolis-hastings samplers: why you should not backtrack for unbiased graph sampling. ACM SIGMETRICS Performance evaluation review , 40(1):319-330, 2012.

Jure Leskovec and Andrej Krevl. Snap datasets: Stanford large network dataset collection, 2014.

Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. In International Conference on Learning Representations , 2020.

Xiang Li, Jiadong Liang, Xiangyu Chang, and Zhihua Zhang. Statistical estimation and online inference via local sgd. In Proceedings of Thirty Fifth Conference on Learning Theory , volume 178 of Proceedings of Machine Learning Research , pages 1613-1661, 02-05 Jul 2022.

Xiang Li, Jiadong Liang, and Zhihua Zhang. Online statistical inference for nonlinear stochastic approximation with markovian data. arXiv preprint arXiv:2302.07690 , 2023.

Yunming Liao, Yang Xu, Hongli Xu, Lun Wang, and Chen Qian. Adaptive configuration for heterogeneous participants in decentralized federated learning. In IEEE INFOCOM 2023 . IEEE, 2023.

Adwaitvedant S Mathkar and Vivek S Borkar. Nonlinear gossip. SIAM Journal on Control and Optimization , 54(3):1535-1557, 2016.

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR, 2017.

Sean Meyn. Control systems and reinforcement learning . Cambridge University Press, 2022.

Antonietta Mira. Ordering and improving the performance of monte carlo markov chains. Statistical Science , pages 340-350, 2001.

Gemma Morral, Pascal Bianchi, and Gersende Fort. Success and failure of adaptation-diffusion algorithms with decaying step size in multiagent networks. IEEE Transactions on Signal Processing , 65(11):2798-2813, 2017.

Wenlong Mou, Chris Junchi Li, Martin J Wainwright, Peter L Bartlett, and Michael I Jordan. On linear stochastic approximation: Fine-grained polyak-ruppert and non-asymptotic concentration. In Conference on Learning Theory , pages 2947-2997. PMLR, 2020.

Giovanni Neglia, Chuan Xu, Don Towsley, and Gianmarco Calbi. Decentralized gradient methods: does topology matter? In International Conference on Artificial Intelligence and Statistics , pages 2348-2358. PMLR, 2020.

Alex Olshevsky. Asymptotic network independence and step-size for a distributed subgradient method. Journal of Machine Learning Research , 23(69):1-32, 2022.

Shi Pu, Alex Olshevsky, and Ioannis Ch Paschalidis. Asymptotic network independence in distributed stochastic optimization for machine learning: Examining distributed and centralized stochastic gradient descent. IEEE signal processing magazine , 37(3):114-122, 2020.

Christian P Robert, George Casella, and George Casella. Monte Carlo statistical methods , volume 2. Springer, 1999.

Sheldon M Ross, John J Kelly, Roger J Sullivan, William James Perry, Donald Mercer, Ruth M Davis, Thomas Dell Washburn, Earl V Sager, Joseph B Boyce, and Vincent L Bristow. Stochastic processes , volume 2. Wiley New York, 1996.

Itay Safran and Ohad Shamir. How good is sgd with random shuffling? In Conference on Learning Theory , pages 3250-3284. PMLR, 2020.

R. Srikant. Rates of Convergence in the Central Limit Theorem for Markov Chains, with an Application to TD Learning. arXiv e-prints , page arXiv:2401.15719, January 2024.

Sebastian U Stich. Local sgd converges fast and communicates little. In International Conference on Learning Representations , 2018.

Tao Sun, Dongsheng li, and Bao Wang. On the decentralized stochastic gradient descent with markov chain sampling. IEEE Transactions on Signal Processing , PP, 07 2023.

Tao Sun, Yuejiao Sun, and Wotao Yin. On markov chain gradient descent. In Advances in neural information processing systems , volume 31, 2018.

MVidyasagar. Convergence of stochastic approximation via martingale and converse lyapunov methods. arXiv preprint arXiv:2205.01303 , 2022.

Hoi-To Wai. On the convergence of consensus algorithms with markovian noise and gradient bias. In 2020 59th IEEE Conference on Decision and Control (CDC) , pages 4897-4902. IEEE, 2020.

Han Wang, Aritra Mitra, Hamed Hassani, George J Pappas, and James Anderson. Federated temporal difference learning with linear function approximation under environmental heterogeneity. arXiv preprint arXiv:2302.02212 , 2023.

Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H. Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. In Advances in Neural Information Processing Systems , volume 33, pages 7611-7623, 2020.

Shiqiang Wang and Mingyue Ji. A unified analysis of federated learning with arbitrary client participation. In Advances in neural information processing systems , 2022.

Stephan Wojtowytsch. Stochastic gradient descent with noise of machine learning type part i: Discrete time analysis. Journal of Nonlinear Science , 33(3):45, 2023.

Blake Woodworth, Kumar Kshitij Patel, Sebastian Stich, Zhen Dai, Brian Bullins, Brendan Mcmahan, Ohad Shamir, and Nathan Srebro. Is local sgd better than minibatch sgd? In International Conference on Machine Learning , pages 10334-10343. PMLR, 2020.

Hao Ye, Le Liang, and Geoffrey Ye Li. Decentralized federated learning with unreliable communications. IEEE Journal of Selected Topics in Signal Processing , 16(3):487-500, 2022.

Chulhee Yun, Shashank Rajput, and Suvrit Sra. Minibatch vs local SGD with shuffling: Tight convergence bounds and beyond. In International Conference on Learning Representations , 2022.

Chulhee Yun, Suvrit Sra, and Ali Jadbabaie. Open problem: Can single-shuffle sgd be better than reshuffling sgd and gd? In Proceedings of Thirty Fourth Conference on Learning Theory , volume 134, pages 4653-4658. PMLR, Aug 2021.

Sihan Zeng, Thinh T Doan, and Justin Romberg. Finite-time convergence rates of decentralized stochastic approximation with applications in multi-agent and multi-task learning. IEEE Transactions on Automatic Control , 2022.

Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint arXiv:1806.00582 , 2018.

A Discussion of Assumption 2.3-ii)

A.1 Suitable choices of K l

When wet let K l ∼ log( l ) (resp. K l ∼ log log( l ) ), as suggested by [51], it trivially satisfies K τ n = o ( γ -1 / 2( L +1) n ) = o ( n 1 / 2( L +1) ) since by definition K τ n < K n ∼ log( n ) (resp. log log( n ) ), and log( n ) = o ( n ϵ ) (resp. log log( n ) = o ( n ϵ ) ) for any ϵ > 0 . Besides, ∑ n η 2 n = ∑ n γ 2 n K 2( L +1) τ n ≲ ∑ n n -2 n 2( L +1) ϵ = ∑ n n 2( L +1) ϵ -2 . To ensure ∑ n η 2 n < ∞ , it is sufficient to have 2( L +1) ϵ -2 < -1 , or equivalently, ϵ < 1 / 2( L + 1) . Since ϵ can be arbitrarily small to satisfy the condition, ∑ n η 2 n < ∞ is satisfied. When K l ∼ log( l ) , we can rewrite the last condition as

where we have n l ∼ log( l !) such that K l +1 /n l = log( l +1) / log( l !) → 0 and log( l +1) / log( l ) → 1 as l → ∞ , which leads to lim n →∞ η n l +1 /η n l +1 +1 = 1 . Similarly, for K l ∼ log log( l ) , we have n l ∼ log( ∏ l s =1 log( s )) such that K l +1 /n l ∼ log log( l +1) / log log( ∏ l s =1 log( s )) → 0 and log log( l +1) / log log( l ) → 1 as l →∞ , which also leads to lim n →∞ η n l +1 /η n l +1 +1 = 1 .

A.2 Practical implications of Assumption 2.3-ii)

In this assumption, we allow the number of local iterations to go to infinity asymptotically. In distributed learning environments such as mobile, IoT, and wireless sensor networks, where nodes are often constrained by battery life, increasing communication interval in Assumption 2.3-ii) plays a crucial role in balancing energy costs with communication effectiveness. It allows agents to communicate more frequently early on, leading to a faster initial convergence to the neighborhood of θ ∗ . Then, we slow down the communication frequency between agents to conserve energy, leveraging the diminishing returns on accuracy improvements from additional communications.

Consider the scenario where devices across multiple clusters collaborate on a distributed optimization task, utilizing local datasets. Devices within each cluster form a communication network that allows a virtual agent to perform a heterogeneous Markov chain trajectory via random walk, or an i.i.d. sequence in a complete graph with self-loops, depending on the application context. Each cluster features an edge server that supports the exchange of model estimates with neighboring clusters. By performing K local updates before uploading these to the cluster's edge server, the model benefits from reduced communication overhead. As the frequency of updates between devices and edge servers decreases - optimized by gradually increasing K -weeffectively lower communication costs, particularly as the model estimation θ n is close to θ ∗ .

B Proof of Lemma 3.1

Let J ⊥ ≜ I N -J ∈ R N × N and J ⊥ ≜ J ⊥ ⊗ I d ∈ R Nd × Nd , where ⊗ is the Kronecker product. Let Θ n = [( θ 1 n ) T , · · · , ( θ N n ) T ] T ∈ R Nd . Then, motivated by [58], we define a sequence ϕ n ≜ η -1 n +1 J ⊥ Θ n ∈ R Nd in the increasing communication interval case (resp. ϕ n ≜ γ -1 n +1 J ⊥ Θ n in the bounded communication interval case ), where η n +1 is defined in Assumption 2.3-ii). J ⊥ Θ n = Θ n -1 N ( 11 T ⊗ I d )Θ n represents the consensus error of the model.

We first give the following lemma that shows the pathwise boundedness of ϕ n .

Lemma B.1. Let Assumptions 2.1, 2.3, 2.4 and 2.5 hold. For any compact set Ω ⊂ R Nd , the sequence ϕ n satisfies sup n E [ ∥ ϕ n ∥ 2 ∩ j ≤ n -1 { Θ j ∈ Ω } ] < ∞ .

✶ Lemma B.1 and Assumption 2.3-ii) imply that for any n ≥ 0 , E [ ∥J ⊥ Θ n ∥ 2 ✶ ∩ j ≤ n -1 { Θ j ∈ Ω } ] = η 2 n +1 E [ ∥ ϕ n ∥ 2 ✶ ∩ j ≤ n -1 { Θ j ∈ Ω } ] ≤ Cη 2 n +1 for some constant C that depends on C 1 and Ω . Along with Assumption 2.4 such that ∥J ⊥ Θ n ∥ is always bounded per each trajectory, it means

Let { Ω m } m ≥ 0 be a sequence of increasing compact subset of R Nd such that ⋃ m Ω m = R Nd . Then, we know that for any m ≥ 0 ,

(17) indicates either one of the following two cases:

there exists some trajectory-dependent index m ′ such that each trajectory { Θ n } n ≥ 0 is always within the compact set Ω m ′ , i.e., ✶ ∩ j ≤ n { Θ j ∈ Ω m ′ } = 1 (satisfied by the construction of increasing compact sets { Ω m } m ≥ 0 and Assumption 2.4), and we have ∥J ⊥ Θ n ∥ = O ( η n ) such that lim n →∞ J ⊥ Θ n = 0 ;

Θ n will escape the compact set Ω m eventually for any m ≥ 0 in finite time such that ∩ j ≤ n -1 { Θ j ∈ Ω m } = 0 when n is large enough.

✶ We can see the second case contradicts Assumption 2.4 because we assume every trajectory { Θ n } n ≥ 0 is within some compact set. Therefore, (17) for any m ≥ 0 is equivalent to showing ∥J ⊥ Θ n ∥ = O ( η n ) and lim n →∞ J ⊥ Θ n = 0 . Under Assumption 2.3-i) we can obtain similar result ∥J ⊥ Θ n ∥ = O ( γ n ) by following the same steps as above, which completes the proof of Lemma 3.1.

Proof of Lemma B.1. We begin by rewriting (6) in the matrix form,

where X n ≜ ( X 1 n , X 2 n , · · · , X N n ) and ∇ F (Θ n , X n ) ≜ [ ∇ F 1 ( θ 1 n , X 1 n ) T , · · · , ∇ F N ( θ N n , X N n ) T ] T ∈ R Nd . Recall θ n ≜ 1 N ∑ N i =1 θ i n ∈ R d and we have [ θ T n , · · · , θ T n ] T = 1 N ( 11 T ⊗ I d )Θ n ∈ R Nd .

Case 1 (Increasing communication interval K τ n ): By left multiplying (18) with 1 N ( 11 T ⊗ I d ) , along with γ n +1 = η n +1 /K L +1 τ n +1 in Assumption 2.3-ii), we have the following iteration

where the equality comes from 1 N ( 11 T ⊗ I d ) W n = 1 N ( 11 T W n ⊗ I d ) = 1 N ( 11 T ⊗ I d ) . With (18) and (19), we have

where the second equality comes from W n -1 N ( 11 T ⊗ I d ) = ( W n -1 N 11 T ) ⊗ I d = J ⊥ W n ⊗ I d and ( J ⊥ W n ⊗ I d ) J ⊥ = J ⊥ W n J ⊥ ⊗ I d = J ⊥ W n ⊗ I d . Let a n ≜ η n /η n +1 , dividing both sides of (20) by η n +2 gives

Define the filtration {F n } n ≥ 0 as F n ≜ σ { Θ 0 , X 0 , W 0 , Θ 1 , X 1 , W 1 , · · · , X n -1 , W n -1 , Θ n , X n } . Recursively computing (21) w.r.t the time interval [ n l , n l +1 ] gives

where ∏ is the backward multiplier, the second equality comes from J ⊥ W n J ⊥ = J ⊥ W n and W k = I N for k / ∈ { n l } . In Assumption 2.5, we have ∥ E W ∼P n l [ W T J ⊥ W ] ∥ = ∥ E W ∼P n l [ W T W -J ] ∥ ≤ C 1 < 1 . Then,

where the first inequality comes from J T ⊥ J ⊥ = J ⊥ and η k +2 ≥ η n l +1 +1 for k ∈ [ n l , n l +1 -1] . Then, we analyze the norm of the gradient ∥∇ F (Θ k , X k ) ∥ in the second term on the RHS of (23) conditioned on F n l . By Assumption 2.4, we assume Θ n l is within some compact set Ω at time n l such that sup i ∈ [ N ] ,X i ∈X i ∇ F i ( θ i n l , X i ) ≤ C Ω for some constant C Ω . For n = n l +1 and any X ∈ X 1 ×X 2 ×··· × X N , we have

Considering ∥∇ F (Θ n l , X ) ∥ , we have sup X ∥∇ F (Θ n l , X ) ∥ 2 ≤ ∑ N i =1 sup X i ∈X i ∥∇ F i ( θ i n l , X i ) ∥ 2 ≤ NC 2 Ω such that ∥∇ F (Θ n l , X ) ∥ ≤ √ NC Ω . In addition, we

have

For n = n l +2 and any X , we have

Similar to the steps in (24), we have

Then, ∥∇ F (Θ n l +2 , X ) ∥ ≤ (1 + γ n l +2 L ) sup X ∥∇ F (Θ n l +1 , X ) ∥ and, together with (25), we have ∥∇ F (Θ n l +2 , X ) ∥ ≤ (1 + γ n l +2 L )(1 + γ n l +1 L ) √ NC Ω . (27) √

By induction, ∥∇ F (Θ n l + m , X ) ∥ ≤ ∏ m s =1 (1 + γ n l + s L ) NC Ω for m ∈ [1 , K l +1 -1] . The next step is to analyze the growth rate of ∏ m s =1 (1 + γ n l + s L ) . By 1 + x ≤ e x for x ≥ 0 , we have

For step size γ n = 1 /n , we have L ∑ m s =1 γ n l + s = L ∑ m s =1 1 / ( n l + s ) < L ∑ m s =1 1 /s < L (log( m )+ 1) such that ∏ m s =1 (1 + γ n l + s L ) < ( em ) L . Then,

where the last inequality comes from ∑ K l +1 -1 m =0 m L < K l +1 ( K l +1 -1) L < K L +1 l +1 . We can see the sum of the norm of the gradients are bounded by √ Ne L C Ω , which only depends on the compact set Ω at time n = n l .

Let δ 1 ∈ ( C 1 , 1) . Since from Assumption 2.3-ii), lim l →∞ η n l +1 /η n l +1 +1 = 1 , there exists some large enough l 0 such that ( η n l +1 η n l +1 +1 ) 2 C 1 < δ 1 < δ 2 := ( δ 1 +1) / 2 < 1 for any l > l 0 . Note that δ 1 depends only on C 1 and is independent of F n . Then, let ˜ C Ω := √ Ne L C Ω , we can rewrite (23) as

where M Ω satisfies M Ω > 8 ˜ C 2 Ω / (1 -δ 1 ) + δ 1 ˜ C 2 Ω , which is derived from rearranging (29) as M Ω ≥ ( δ 1 -δ 2 ) ∥ ϕ n l ∥ 2 + 2 δ 1 ˜ C Ω ∥ ϕ n l ∥ + δ 1 ˜ C 2 Ω and upper bounding the RHS. Upon noting that ✶ ∩ j ≤ n l { Θ j ∈ Ω } ≤ ∩ j ≤ n l -1 { Θ j ∈ Ω } , we obtain

√

The induction leads to E [ ∥ ϕ n l +1 ∥ 2 ✶ ∩ j ≤ n l { Θ j ∈ Ω } ] ≤ δ n l +1 -n l 0 2 E [ ∥ ϕ n l 0 ∥ 2 ✶ ∩ j ≤ n l 0 -1 { Θ j ∈ Ω } ] + M/ (1 -δ 2 ) < ∞ for any l ≥ l 0 . Besides, for m ∈ ( n l , n l +1 ) , by following the above steps (23) applied to (21), we have

By (28) we already show that ∥ ∑ n l +1 -1 k = n l ∇ F (Θ k , X k ) K L +1 l +1 ∥ < ∞ conditioned on F n l . Therefore, E [ ∥ ϕ m ∥ 2 ✶ ∩ j ≤ n l { Θ j ∈ Ω } ] < ∞ for m ∈ ( n l , n l +1 ) . This completes the boundedness analysis of E [ ∥ ϕ n ∥ 2 ∩ j ≤ n -1 { Θ j ∈ Ω } ] .

✶ Case 2 (Bounded communication interval K τ n ≤ K ): In this case, we do not need the auxiliary step size η n and can directly work on γ n = 1 /n a for a ∈ (0 . 5 , 1] . Similar to (20), we have

and let b n ≜ γ n /γ n +1 , dividing both sides of above equation by γ n +2 gives

Then, by following the similar steps in (22) and (23), we obtain

Also similar to (25) - (28), we can bound the sum of the norm of the gradients as

Now that K l is bounded above by K , ∏ k s = n l (1 + γ s +1 L ) ≤ e L ∑ k s = n l γ s +1 < e L ∑ K -1 s =0 γ s +1 := C K . Then, we further bound (35) as

The subsequent proof is basically a replication of (29) - (31) and is therefore omitted.

C Proof of Theorem 3.2

We focus on analyzing the convergence property of θ , which is obtained by left multiplying (18) with 1 N ( 1 T ⊗ I d ) , i.e.,

where the second equality comes from W n being doubly stochastic and 1 N ( 1 T ⊗ I d ) W n = 1 N ( 1 T W n ⊗ I d ) = 1 N ( 1 T ⊗ I d ) .

For self-contained purpose, we first give the almost sure convergence result for the stochastic approximation that will be used in our proof.

Theorem C.1 (Theorem 2 [23]) . Consider the stochastic approximation in the form of

Assume that

C1. w.p.1, the closure of { θ n } n ≥ 0 is a compact subset of R d ;

C2. { γ n } is a decreasing sequence of positive number such that ∑ n γ n = ∞ ;

C3. w.p.1, lim p →∞ ∑ p n =1 γ n e n exists and is finite. Moreover, lim n →∞ r n = 0 .

C4. vector-valued function h is continuous on R d and there exists a continuously differentiable function V : R d → R such that ⟨∇ V ( θ ) , h ( θ ) ⟩ ≤ 0 for all θ ∈ R d . Besides, the interior of V ( L ) is empty where L ≜ { θ ∈ R d : ⟨∇ V ( θ ) , h ( θ ) ⟩ = 0 } .

We can rewrite (37) as

and work on the converging behavior of the third and fourth term. By definition of function ∇ f ( · ) , we have

By the Lipschitz continuity of function ∇ F i ( · , X ) in (7), we have

where the second inequality comes from the Cauchy-Schwartz inequality. In Appendix B, we have shown lim n J ⊥ Θ n = 0 almost surely such that lim n →∞ r n = 0 almost surely.

Next, we further decompose the fourth term in (39). For an ergodic transition matrix P and a function v associated with the same state space X , define the operator P k v ( x ) ≜ ∑ y ∈X P k ( x, y ) v ( y ) for the k -step transition probability P k ( x, y ) . Denote by P 1 , · · · , P N the underlying transition matrices of all N agents with corresponding stationary distribution π 1 , · · · , π N . Then, for every function ∇ F i ( θ i , · ) : X i → R d , there exists a corresponding function m θ i ( · ) : X i → R d such that

The solution of the Poisson equation (42) has been studied in the literature, e.g., [17, 38]. For self-contained purpose, we derive the closed-form m θ i ( x ) from scratch. First of all, we can obtain function m θ i ( x ) in the recursive form as follows,

It is not hard to check that (43) satisfies (42). Note that by induction we get

Then, we can further simplify (43), and the closed-form expression of m θ i ( x ) is given as

where the fourth equality comes from (44). Note that the so-called 'fundamental matrix' ( I -P i + 1 ( π i ) T ) -1 exists for every ergodic Markov chain X i from Assumption 2.2. Since function ∇ F i is Lipschitz continuous, we have the following lemma.

Lemma C.2. Under assumption (A1), functions m θ i ( x ) and P i m θ i ( x ) are both Lipschitz continuous in θ i for any x ∈ X i .

Proof. By (45), for any θ i 1 , θ i 2 ∈ R d and x ∈ X i , we have

where the second inequality holds for a constant C i that is the largest absolute value of the entry in the matrix ( I -P i + 1 ( π i ) T ) -1 . Therefore, m θ i ( x ) is Lipschitz continuous in θ i . Moreover, following the similar steps as above, we have

such that P i m θ i ( x ) is also Lipschitz continuous in θ i , which comletes the proof.

Now with (42) we can decompose

Here { γ n e i n } is a Martingale difference sequence and we need the martingale convergence theorem in Theorem C.3.

Theorem C.3 (Theorem 6 . 4 . 6 [64]) . For an F n -Martingale S n , set X n -1 = S n -S n -1 . If for some 1 ≤ p ≤ 2 ,

then S n converges almost surely.

We want to show that ∑ n γ 2 n +1 E [ ∥ e i n +1 ∥ 2 |F n ] < ∞ such that ∑ n γ n e i n converges almost surely by Theorem C.3. As we can see in (45), with Lemma C.2 and Assumption 2.4, for a sample path ( Θ n within a compact set Ω ), sup n ∥ m θ i n ( x ) ∥ < ∞ and sup n ∥ P i m θ i n ( x ) ∥ < ∞ almost surely for all x ∈ X i . This ensures that e i n +1 is an L 2 -bounded martingale difference sequence, i.e., sup n ∥ e i n +1 ∥ ≤ sup n ( ∥ m θ i n ( X i n +1 ) ∥ + ∥ P i m θ i n ( X i n ) ∥ ) ≤ D Ω < ∞ . Together with Assumption 2.3, we get

and thus ∑ n γ n e i n converges almost surely.

Next, for the term ν i n we have

As is shown before, for a given sample path, ∥ P i m θ i n ( x ) ∥ is bounded almost surely for all n and x ∈ X i such that sup n ∥ ν i n ∥ < ∞ almost surely. Since lim n →∞ ( γ n +1 -γ n ) = 0 , we have lim n →∞ ( γ n +1 -γ n ) ν i n = 0 . Note that there exists a path-dependent constant C (that bounds ∥ ν i n ∥ ) such that for any n ≥ m ,

Since lim n →∞ γ n = 0 , there exists a positive integer M such that for all n ≥ m ≥ M , γ m < ϵ/C and ∥ ∑ n k = m ( γ k +1 -γ k ) ν i k ∥ < ϵ for every ϵ > 0 . Therefore, { ∑ p k =0 ( γ k +1 -γ k ) ν i k } p ≥ 0 is a Cauchy sequence and ∑ ∞ k =0 ( γ k +1 -γ k ) ν i k converges by Cauchy convergence criterion. The last term of (51) tends to zero. Therefore, ∑ ∞ k =0 γ k +1 ( ν i k -ν i k +1 ) converges and is finite.

For the last term ξ i n , Lemma C.2 leads to

for the Lipschitz constant C ′ of P i m θ i ( x ) . However, the relationship between θ n and θ n +1 is not obvious in the D-SGD and FL setting due to the update rule (18) with communication matrix W n , unlike the classical stochastic approximation shown in (38). We come up with the novel decomposition of ξ i n , which takes the consensus error into account, to solve this issue, i.e.,

Using the Lipschitzness property of P i m θ ( X ) in Lemma C.2, we have

In Appendix B we have shown lim n →∞ J ⊥ Θ n = 0 almost surely. Moreover, ∥ 1 N ( 1 T ⊗ I d ) ∇ F (Θ n , X n ) ∥ is bounded per sample path. Therefore, lim n →∞ 1 N ∑ N i =1 ∥ ξ i n +1 ∥ = 0 such that lim n →∞ 1 N ∑ N i =1 ξ i n +1 = 0 almost surely.

To sum up, we decompose (39) into

Now that lim p →∞ ∑ p n =1 1 N ∑ N i =1 γ n e i n and lim p →∞ ∑ p n =0 1 N ∑ N i =1 γ n +1 ( ν i n -ν i n +1 ) converge and are finite, lim n →∞ r n = 0 , lim n →∞ 1 N ∑ N i =1 ξ i n = 0 , all the conditions of C3 in Theorem C.1 are satisfied. Additionally, Assumption 2.4 corresponds to C1, Assumption 2.3 meets C2, and C4 is automatically satisfied when we choose the lyapunov function V ( θ ) = f ( θ ) . Therefore, lim sup n inf θ ∗ ∈L ∥ θ n -θ ∗ ∥ = 0 .

D Proof of Theorem 3.3

To obtain Theorem 3.3, we need to utilize the existing CLT result for general SA in Theorem D.1 and check all the necessary conditions therein.

Theorem D.1 (Theorem 2.1 [29]) . Consider the stochastic approximation iteration (38) , assume

C1. Let θ ∗ be the root of function h , i.e., h ( θ ∗ ) = 0 , and assume lim n →∞ θ n = θ ∗ . Moreover, assume the mean field h is twice continuously differentiable in a neighborhood of θ ∗ , and the Jacobian H ≜ ∇ h ( θ ∗ ) is Hurwitz, i.e., the largest real part of its eigenvalues B < 0 ;

C2. The step size ∑ n γ n = ∞ , ∑ n γ 2 n < ∞ , and either (i). log( γ n -1 /γ n ) = o ( γ n ) , or (ii). log( γ n -1 /γ n ) ∼ γ n /γ ⋆ for some γ ⋆ > 1 / 2 | B | ;

C3. sup n ∥ θ i n ∥ < ∞ almost surely for any i ∈ [ N ] ;

C4. (a) { e n } n ≥ 0 is an F n -Martingale difference sequence, i.e., E [ e n |F n -1 ] = 0 , and there exists τ > 0 such that sup n ≥ 0 E [ ∥ e n ∥ 2+ τ |F n -1 ] < ∞ ;

(b) E [ e n +1 e T n +1 |F n ] = U + D ( A ) n + D ( B ) n , where U is a symmetric positive semi-definite matrix and

C5. Let r n = r (1) n + r (2) n , r n is F n -adapted, and

Then, where

Note that the matrix U in the condition C4(b) of Theorem D.1 was assumed to be positive definite in the original Theorem 2.1 [29]. It was only to ensure that the solution V to the Lyapunov equation (60) is positive definite, which was only used for the stability of the related autonomous linear ODE (e.g., Theorem 3.16 [15] or Theorem 2.2.3 [36]). However, in this paper, we do not need strict positive definite matrix V . Therefore, we extend U to be positive semi-definite such that V is also positive semi-definite (see Lemma D.2 for the closed form of matrix V ). Such kind of extension does not change any of the proof steps in [29].

D.1 Discussion about C1-C3

Our Assumption 2.1 corresponds to C1 by letting function h ( θ ) = -∇ f ( θ ) therein. We can also let γ ⋆ in Theorem 3.3 large enough to satisfy C2. The typical form of step size, also indicated in [29], is polynomial step size γ n ∼ γ ⋆ /n a for a ∈ (0 . 5 , 1] . Note that a ∈ (0 . 5 , 1) satisfies C2 (i) and a = 1 satisfies C2 (ii). Assumption 2.4 corresponds to C3. 4

D.2 Analysis of C4

To check condition C4, we need to analyze the Martingale difference sequence { e i n } . Recall e i n +1 = m θ i n ( X i n ) -P i m θ i n ( X i n -1 ) such that there exists a constant C ,

Since ∥ m θ i n ( Y ) ∥ < ∞ almost surely by Assumption 2.4 and X i is a finite state space, at all time n , we have

and there exists another constant C ′ such that by definition of P i m θ i n ( X i n -1 ) , we have

Therefore, E [ ∥ e i n +1 ∥ 2+ τ |F n ] < ∞ a.s. for all n and C4.(a) is satisfied.

̸

We now turn to C4.(b). Note that for any i = j , we have E [ e i n +1 ( e j n +1 ) T |F n ] = E [ e i n +1 |F n ] · E [( e j n +1 ) T |F n ] = 0 due to the independence between agent i and j , and E [ e i n +1 |F n ] = 0 . Then, we have

The analysis of E [ e i n +1 ( e i n +1 ) T |F n ] is inspired by Section 4 [29] and Section 4.3.3 [22], where they constructed another Poisson equation to further decompose the noise terms therein. 5 Here, expanding

Denote by

and let its expectation w.r.t the stationary distribution π i be g i ( θ i ) ≜ E x ∼ π i [ G i ( θ i , x )] , we can construct another Poisson equation, i.e.,

for some matrix-valued function φ i : R d ×X i → R d × d . Following the similar steps shown in (42) (45), we can obtain the closed-form expression

Then, we can decompose (65) into

(69)

Let U ≜ 1 N 2 ∑ N i =1 U i , D (1) n ≜ 1 N 2 ∑ N i =1 D (1) 1 ,n , D (2 ,a ) n ≜ 1 N 2 ∑ N i =1 D (2 ,a ) i,n , and D (2 ,b ) n ≜ 1 N 2 ∑ N i =1 D (2 ,b ) i,n , we want to prove that D (1) n satisfies the first condition in C4, and D (2 ,a ) n , D (2 ,b ) n meet the second condition in C4.

We now show that for all i , G i ( θ i , x ) is Lipschitz continuous in θ i ∈ Ω for some compact subset Ω ⊂ R d . For any x ∈ X i and θ i 1 , θ i 2 ∈ Ω , we can get

for some constant C , where the last inequality comes from ∥ m θ i 1 ( x ) ∥ < ∞ since θ i 1 ∈ Ω and the Lipschitz continuous function m θ i ( x ) . Similarly, we can get ∥ P i m θ i 1 ( x ) -P i m θ i 2 ( x ) ∥ ≤ C ∥ θ i 1 -θ i 2 ∥ . Therefore, G i ( θ i , x ) and g i ( θ i ) are Lipschitz continuous in θ i ∈ Ω for any x ∈ X i .

For the sequence { D (1) i,n } n ≥ 0 , by applying Theorem 3.2 and conditioned on lim n →∞ θ n = θ ∗ for an optimal point θ ∗ ∈ L , we have lim n →∞ ∥ g i ( θ i n ) -g i ( θ ∗ ) ∥ ≤ lim n →∞ C ∥ θ i n -θ ∗ ∥ = 0 . This implies D (1) i,n → 0 for every i ∈ [ N ] and thus D (1) n → 0 as n →∞ almost surely, which satisfies the first condition in (57).

For the Martingale difference sequence { D (2 ,a ) i,n } n ≥ 0 , we use Burkholder inequality (e.g., Theorem 2.10 [33], [21]) such that for p ≥ 1 and some constant C p ,

By the definition (66) and Assumption 2.4, for a sample path, sup n ∥ G i ( θ i n , x ) ∥ < ∞ for any x ∈ X i , as well as sup n ∥ g i ( θ i n ) ∥ < ∞ , which leads to sup n ∥ φ i θ i ( x ) ∥ < ∞ for any x ∈ X i because of (68).

n

Then, we have sup n ∥ D (2 ,a ) i,n ∥ ≤ C < ∞ for the path-dependent constant C . Taking p = 1 and we have

Thus, Lebesgue dominated convergence theorem gives

and we have n →∞ n i =1 i,n

For the sequence { D (2 ,b ) i,n } n ≥ 0 , we have

Since G i ( θ i , x ) and g i ( θ i ) are Lipschitz continuous in θ i ∈ Ω , φ i θ i ( x ) is also Lipschitz continuous in θ i ∈ Ω and is bounded. We have

where ∥ φ i θ i 0 ( X i 0 ) ∥ + ∥ φ i θ i n ( X i n ) ∥ ≤ D 1 for a given sample path, D 2 is the Lipschitz constant of φ i θ i ( x ) , and ∥∇ F i ( x i , X i ) ∥ ≤ D Ω for any x i ∈ Ω and X i ∈ X i . Then,

because γ n ∑ n k =1 γ k = O ( n 1 -2 a ) by assumption 2.3. Therefore, the second condition of C4 is satisfied.

D.3 Analysis of C5

Wenowanalyze condition C5. The decreasing rate of each term in (56) has been proved in Appendix C. Specifically, by assumption 2.4, there exists a compact subset for a given sample path, and

For 1 N ∑ N i =1 ξ i n , in the case of increasing communication interval, 1 N ∑ N i =1 ξ i n = O ( γ n + η n ) , by Assumption 2.3-ii), we know ( γ n + η n ) / √ γ n = √ γ n + √ γ n K L +1 τ n = o (1) such that ∥ 1 N ∑ N i =1 ξ i n ∥ = o ( √ γ n ) almost surely. On the other hand, in the case of bounded communication interval, 1 N ∑ N i =1 ξ i n = O ( γ n ) such that ∥ 1 N ∑ N i =1 ξ i n ∥ = o ( √ γ n ) a.s.

Since sup n ∥ ν i n ∥ < ∞ almost surely, we have sup p ∥ 1 N ∑ N i =1 ∑ p k =0 ( ν i k -ν i k +1 ) ∥ = sup p ∥ 1 N ∑ N i =1 ( ν i 0 -ν i p +1 ) ∥ < ∞ almost surely. Then, √ γ p ∥ 1 N ∑ N i =1 ∑ p k =0 ( ν i k -ν i k +1 ) ∥ = O ( √ γ p ) leads to √ γ p ∥ 1 N ∑ N i =1 ∑ p k =0 ( ν i k -ν i k +1 ) ∥ = o (1) a.s.

Let r (1) n ≜ r ( A ) n + 1 N ∑ N i =1 ξ i n and r (2) n ≜ 1 N ∑ N i =1 ( ν i k -ν i k +1 ) . From above, we can see that C5 in Theorem D.1 is satisfied and we show that all the conditions in Theorem D.1 have been satisfied.

D.4 CLosed Form of Limitimg Covariance Matrix

Lastly, we need to analyze the closed-form expression of U as in C4 (b) of Theorem D.1. Recall that U = 1 N 2 ∑ N i =1 U i and U i = g i ( θ ∗ ) in (69). We now give the exact form of function g i ( θ ∗ ) as follows:

where the second equality comes from the recursive form of m θ i ( x ) in (45), and that the process { X n } n ≥ 0 is in its stationary regime, i.e., X 0 ∼ π i from the beginning. The last equality comes from rewriting Cov ( ∇ F i ( θ ∗ , X i ) , ∇ F i ( θ ∗ , X j )) in a matrix form. Note that g i ( θ ∗ ) is exactly the asymptotic covariance matrix of the underlying Markov chain { X i n } n ≥ 0 associated with the test function ∇ F i ( θ ∗ , · ) . By utilizing the following lemma, we can obtain the explicit form of V as defined in (60).

Lemma D.2 (Lemma D.2.2 [41]) . If all the eigenvalues of matrix M have negative real part, then for every positive semi-definite matrix U there exists a unique positive semi-definite matrix V satisfying U + MV + VM T = 0 . The explicit solution V is given as

D.5 CLT of Polyak-Ruppert Averaging

We now consider the CLT result of Polyak-Ruppert averaging ¯ θ n = 1 n ∑ n -1 k =0 θ k . The steps follow similar way by verifying that the conditions in the related CLT of Polyak-Ruppert averaging for the stochastic approximation are satisfied. The additional assumption is given below.

C6. For the sequence { r n } in (38), n -1 / 2 ∑ n k =0 r (1) k → 0 with probability 1 .

Then, the CLT of Polyak-Ruppert averaging is as follows.

Theorem D.3 (Theorem 3.2 of [29]) . Consider the iteration (38) , assume C1, C3, C4, C5 in Theorem D.1 are satisfied. Moreover, assume C6 is satisfied. Then, with step size γ n ∼ γ ⋆ /n a for a ∈ (0 . 5 , 1) , we have

where V ′ = H -1 UH -T .

Discussion about C1 and C3 can be found in Section D.1. Condition C4 has been analyzed in Section D.2 and condition C5 has been examined in Section D.3. The only condition left to analyze is C6, which is based on the results obtained in Section D.3. In view of (56), r (1) n = r ( A ) n + 1 N ∑ N i =1 ξ i n +1 , so C6 is equivalent to

In Section D.3, we have shown that ∥ ∥ ∥ r ( A ) n ∥ ∥ ∥ = O ( η n ) , 1 N ∑ N i =1 ξ i n = O ( γ n ) . Note that by Assumption 2.3, we consider bounded communication interval for step size γ n ∼ γ ⋆ /n a for a ∈ (0 . 5 , 1) , and hence, η n = O ( γ n ) such that ∥ ∥ ∥ r ( A ) n ∥ ∥ ∥ = O ( γ n ) . We then know that

such that

which proved (79) and C6 is verified. Therefore, Theorem D.3 is proved under our Assumptions 2.1 2.5.

E Discussion on the comparison of Theorem 3.3 to the CLT result in [51]

As a byproduct of our Theorem 3.3, we have the following corollary.

Corollary E.1. Under Assumptions 2.1 - 2.5, for the sub-sequence { n l } l ≥ 0 where K l = K for all l , we have

Proof. Since K l = K for all l , we have n l = Kl . There is an existing result showing the CLT result of the partial sum of a sub-sequence (after normalization) has the same normal distribution as the partial sum of the original sequence.

Theorem E.2 (Theorem 14.4 of [8]) . Given a sequence of random variable θ 1 , θ 2 , · · · with partial sum S n ≜ ∑ n k =1 θ k such that 1 √ n S n dist. - - - - → n →∞ N (0 , V ) . Let n l be some positive random variable taking integer value such that θ n l is on the same space as θ n . In addition, for some sequence { b l } l ≥ 0 going to infinity, n l /b l → c for a positive constant c . Then, 1 √ n l S n l dist. - - - → l →∞ N (0 , V ) .

From Theorem E.2 and our Theorem 3.3, we have 1 √ n l ∑ l k =1 ( ¯ θ n k -θ ∗ ) dist. - - - → l →∞ N (0 , V ′ ) .

Recently, [51] studied the CLT result under the LSGD-FP algorithm with i.i.d sampling (with slightly different setting of the step size). We are able recover Theorem 3.1 of [51] under the constant communication interval while adjusting their step size to make a fair comparison. We state their algorithm below for self-contained purpose. During each communication interval n ∈ ( n l , n l +1 ] ,

The CLT result associated with (83) is given below.

Theorem E.3 (Theorem 3.1 of [51]) . Under LSGD-FP algorithm with i.i.d. sampling, we have

where ν ≜ lim l →∞ 1 l 2 ( ∑ l k =1 K l )( ∑ l k =1 K -1 l ) .

Note that ν = 1 for constant K . We can rewrite (84) as

such that

Note that the step size in (83) keeps unchanged during each communication interval, while ours in (1) keeps decreasing even in the same communication interval. This makes our step size decreasing faster than theirs. To make a fair comparison, we only choose a sub-sequence { n Kl } l ≥ 0 in (86) such that it is 'equivalent' to see that our step sizes become the same at each aggregation step. In this case, we again use Theorem E.2 to obtain

such that

Therefore, our Corollary E.1 also recovers Theorem 3.1 of [51] under the constant communication interval K , but with more general communication patterns and Markovian sampling.

F Discussion on Communication Patterns

F.1 Examples of Communication Matrix W

̸

Metropolis Hasting Algorithm: In the decentralized learning such as D-SGD, HLSGD and DFL, W at the aggregation step can be generated locally using the Metropolis Hasting algorithm based on the underlying communication topology, and is deterministic [62, 43, 80]. Specifically, each agent i exchanges its degree d i with its neighbors j ∈ N ( i ) , forming the weight W ( i, j ) = min { 1 /d i , 1 /d j } for j ∈ N ( i ) and W ( i, i ) = 1 -∑ j = N ( i ) W ( i, j ) . In this case, W is doubly stochastic and symmetric. By Perron-Frobenius theorem, its SLEM λ 2 ( W ) < 1 . Then, ∥ W T W -J ∥ = ∥ W 2 -J ∥ = λ 2 2 ( W ) < 1 , which satisfies Assumption 2.5-ii). It is worth noting that this algorithm is robust to time-varying communication topologies.

Client Sampling in FL: For LSGD-FP studied in [67, 76, 42], W = 11 T /N trivially satisfies Assumption 2.5-ii). For LSGD-PP on the other hand, only a small fraction of agents participate in each aggregation step for consensus [50, 30]. Denote by S a randomly selected set of agents (without replacement) of fixed size |S| ∈ { 1 , 2 , · · · , N } at time n and W S plays a role of aggregating θ i n

̸

for agent i ∈ S . Additionally, the central server needs to broadcast updated parameter θ n +1 to the newly selected set S ′ with the same size, which results in a bijective mapping σ (for S → S ′ and [ N ] / S → [ N ] / S ′ ) and a corresponding permutation matrix T S→S ′ . Then, the communication matrix becomes W = T S→S ′ W S . 6 Specifically, T S→S ′ ( i, j ) = 1 if j = σ ( i ) and T S→S ′ ( i, j ) = 0 otherwise. Besides, W S ( i, j ) = 1 / |S| for i, j ∈ S , W S ( i, i ) = 1 for i / ∈ S , and W S ( i, j ) = 0 otherwise. Note that W S is now a random matrix, since S is a randomly chosen subset of size |S| . Clearly, for each choice of S , W S is doubly stochastic, symmetric and W 2 S = W S . Taking the expectation of W S w.r.t the randomly selected set S gives E S [ W S ]( i, i ) = 1 -( |S| 1) /N for i ∈ [ N ] and E S [ W S ]( i, j ) = ( |S| 1) /N ( N -1) for i = j . Note that E S [ W S ] has all positive entries. Therefore, we use the fact T T T = I for permutation matrix T such that ∥ E [ W ] -J ∥ = ∥ E S , S ′ [ W T S T T S , S ′ T S , S ′ W S ] -J ∥ = ∥ E S [ W T S W S ] -J ∥ = ∥ E S [ W S ] -J ∥ < 1 by Perron-Frobenius theorem and eigendecomposition, which satisfies Assumption 2.5-ii).

F.2 Discussion on partial client sampling

The commonly used partial client sampling algorithm in the FL literature [50, 30] is FedAvg as follows:

At time n , the central server updates its global parameter θ n = 1 |S| ∑ i ∈S θ i n from the agents in the previous set S . Then, the central server selects a new subset of agents S ′ and broadcasts θ n to agent i ∈ S ′ , i.e., θ i n = θ n ;

Each selected agent i computes K steps of SGD locally and consecutively updates its local parameter θ i n +1 , · · · , θ i n + K according to (1a);

Each selected agent i ∈ S ′ uploads θ i n + K to the central server.

Then, the central server repeats the above three steps with θ n + K and a new set of selected agents.

In our client sampling scheme, at the aggregation step n , the design of W S results in ˜ θ i n = 1 |S| ∑ j ∈S θ j n for a selected agent i ∈ S , and ˜ θ i n = θ i n for an unselected agent i / ∈ S . Meanwhile, the central server updates the global parameter ˜ θ n = ˜ θ i n for i ∈ S . Then, the permutation matrix T S→S ′ ensures that the newly selected agent i ∈ S ′ will use ˜ θ n as the initial point for its subsequent SGD iterations. Consequently, from the selected agents' perspective, the communication matrix W = T S→S ′ W S corresponds to step 1 in FedAvg. As we can observe, both algorithms update the global parameter identically from the central server's viewpoint, rendering them mathematically equivalent regarding the global parameter update.

We acknowledge that under the i.i.d sampling strategy, the behavior of unselected agents in our algorithm differs from FedAvg. Specifically, unselected agents are idle in FedAvg, while they continue the SGD computation in our algorithm (despite not contributing to the global parameter update). Importantly, when an unselected agent is later selected, the central server overwrites its local parameter during the broadcasting process. This ensures that the activities of agents when they are unselected have no impact on the global parameter update.

To our knowledge, the FedAvg algorithm under the Markovian sampling strategy remains unexplored in the FL literature. Extrapolating the behavior of unselected agents in FedAvg from i.i.d sampling to Markovian sampling suggests that unselected agents would remain idle. In contrast, our algorithm enables unselected agents to continue evolving X i n . These additional transitions contribute to faster mixing of the Markov chain for each unselected agent and a smaller bias of F i ( θ, X i n ) relative to its mean field f i ( θ ) , potentially accelerating the convergence.

G Additional Simulation

G.1 Simulation Setup in Section 4

This simulation is performed on a PC with an AMD R9 5950X, RTX 3080 and 128 GB RAM. In this simulation, we assume that agents follow the DSGD algorithm (1). In Figure 2(a) - 2(c), each

agent holds a disjoint local dataset (non-overlapping data points for every agent), while we distribute the ijcnn1 dataset [14] with more varied distribution among 100 agents by leveraging Dirichlet distribution with the default alpha value of 0 . 5 in Figure 2(d) - 2(f).

Moreover, we assume that all agents are distributed over a communication network. In order to create this network among 100 agents and the graph-like dataset structure held by each agent, we utilize connected sub-graphs from the real-world graph Facebook in SNAP [49]. All 100 agents collaborate together to generate a deterministic communication matrix W = [ W ij ] with Metropolis Hasting algorithm of the following form: For i, j ∈ [ N ] , we have

where d i represents the degree of agent i in the graph. The communication interval K is set to 1 , as is the usual choice in DSGD [71, 80, 61, 68].

For the first group of agents, we assume they have full access to their datasets, thus performing i.i.d. sampling or single shuffling. In particular,

i.i.d. sampling employed by agent i means that the data point X i n is independently and uniformly sampled from its dataset X i at each time n .

Single shuffling, by its name, only shuffles the dataset once and adheres to that specific order throughout the training process.

On the other hand, within the second group of agents, we assume that they hold graph-like datasets. Now, we introduce simple random walk (SRW), non-backtracking random walk (NBRW), and self-repellent random walk (SRRW) in order:

SRW refers to the walker that chooses one of the neighboring nodes uniformly at random.

NBRW, as studied in [2, 48, 5], is a variation of SRW, which selects one of the neighbors uniformly at random, with the exception of the one visited in the last step.

SRRW, recently proposed by [25], is designed with a nonlinear transition kernel K [ x ] ∈ [0 , 1] N × N of the following form:

where matrix P = [ P ij ] is the transition kernel of the baseline Markov chain and µ = [ µ i ] is its corresponding stationary distribution. Additionally, α denotes the force of self repellence, and larger α leads to stronger force of self repellence, thus higher sampling efficiency [25, Corollary 4.3]. Moreover, vector x ∈ R N is in the interior of probability simplex, representing the empirical distribution, in other words, the visit frequency of each node in the graph. The update rule of this empirical distribution is in the following form:

where β n ≜ ( n +1) -b is the step size of SRRW iterates. b = 1 was original proposed in [25] and is recently extended to b ∈ (0 . 5 , 1) in [39]. In this simulation, we use SRW as the baseline Markov chain of SRRW, and in turn µ is proportional to the degree distribution. We also assume x 0 = 1 /N , i.e., each node has been visited once, and choose the step size β n = ( n +1) -0 . 8 , force of self repellence α = 20 according to the suggestion in [39, Section 4].

Since SRW, NBRW, and SRRW all admit the stationary distribution that is proportional to degree distribution, in order to obtain unfirom target in (15), we need to reweight the gradient computed by each agent i in order to maintain an asymptotic unbiased gradient. Thus, agent i should modify its SGD update from (1a) to the following:

0

25

50

75

100

125

150

175

200

Number of rounds (n)

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2.0

2.2

training loss

Shuffling, SRW

Shuffling, NBRW

Shuffling, SRRW

0

25

50

75

100

125

150

175

200

Number of rounds (n)

0.0

0.5

1.0

1.5

2.0

training loss

Shuffling, SRRW with 10 clients

Shuffling, SRRW with 20 clients

Shuffling, SRRW with 30 clients

Shuffling, SRRW with 40 clients

0

25

50

75

100

125

150

175

200

Number of rounds (n)

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

training loss

Shuffling, SRW

Shuffling, NBRW

Shuffling, SRRW(

=10)

G.2 Image Classification Task

In this part, we perform the image classification task through a 5 -layer neural network, where the CIFAR10 dataset [44] with 50 k image data is evenly distributed to 10 agents. Each agent possesses 5 k images, which are further divided into 200 batches, each batch with 25 images.

The Convolutional Neural Network (CNN) model used in this simulation encompasses:

Two convolutional layers (i.e., nn.Conv2d(3, 6, 5) and nn.Conv2d(6, 16, 5) ), each followed by ReLU activation functions to introduce non-linearity and max pooling (i.e., nn.MaxPool2d(2, 2) ) to reduce spatial dimensions.

Three fully connected (linear) layers, concluding with a softmax output to handle the multi-class classification problem.

Similar to the simulation setup in Section 4, among the 10 participating agents, five have unrestricted access to their respective data allocations, enabling them to utilize the shuffling method to iterate through their batches. The other five agents are designed to simulate limited data access scenarios. Their data access is structured using five distinct graph topologies extracted from the SNAP dataset collection [49], each graph simulating a unique communication pattern among the batches (nodes) of data. Within these topologies, agents adopt one of three random walk strategies - SRW, NBRW, and SRRW, all with importance reweighting - to sample the batches for training.

Local model training is conducted for five epochs at each agent before aggregating the updates at a central server - a process repeated for a total of 200 communication rounds. Each epoch consists of a full traversal of the local dataset of agents in the first group, or 200 batches sampled for training in the second group of agents. To mimic realistic conditions, we also introduce partial agent participation where only 40% of agents are selected randomly to transmit their updates in each round, reflecting the intermittent communication in real-world FL deployments. Lastly, the selection of the step size β n for SRRW iterates (90) is a critical aspect of our experiments. In this simulation, we experiment with various values of b ∈ { 0 . 501 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 } to determine the most advantageous setting for maximizing the benefits of the SRRW strategy. Based on our findings, the best choice for the SRRW step size is b = 0 . 501 , in other words, β n = ( n +1) -0 . 501 .

The simulation result is quantified by averaging the training loss across ten repeated trials for each configuration. As depicted in Figure 3(a), the training results are consistent with our previous findings in Figure 2(a) in the context of the FL framework and the training of neural networks: the use of a more effective sampling approach, even for a portion of the agents, results in significant enhancements in the overall training of the model, and this improvement is further enhanced through the highly efficient sampling strategy SRRW.

In Figure 3(b) and 3(c), we perform image classification experiments in the FL setting with partial client participation. Only 4 random agents will participate in the training process at each aggregation phase. In Figure 3(b), we fix the sampling strategy (shuffling, SRRW with α = 10 ) and test the linear speedup effect for the 5 -layer CNN model by duplicating 10 agents to N agents with N ∈ { 10 , 20 , 30 , 40 } , keeping the same participation ratio 0 . 4 . As can be seen from the plot, the training loss is inversely proportional to the number of agents, i.e., at 200 rounds, the training loss is 0 . 52 for 10 agents, 0 . 23 for 20 agents, 0 . 18 for 30 agents, and 0 . 12 for 40 agents. In Figure 3(c), we extend the current simulation from 5-layer CNN model to ResNet-18 model [34] in order to

numerically test the performance of different sampling strategies in a more complex neural network training task. By fixing the shuffling in the first group of agents, we observe that improving Markovian sampling from SRW to NBRW, then to SRRW, gives accelerated training process with smaller training loss.

H Limitations

Our study provides crucial insights into the identification of nuanced agents' sampling behaviors in UD-SGD, where improving each agent's sampling strategy speeds up overall system performance without additional computational burden except the additional storage for the visit counts used for sampling their datasets. Our UD-SGD is scalable in terms of larger datasets as the sampling strategy (i.e., random walk) utilized by each agent leverages only local information for its dataset. However, our work has two limitations that should be acknowledged.

Assumption 2.4 posits that the parameter trajectory { θ n } is almost surely bounded, which is a strong assumption. This is crucial for guaranteeing the well-defined nature of all related quantities. Some mechanisms such as projections onto a compact subset [45, Chapter 5.1], or truncation-based method with expanding compact subsets can do the trick to ensure that the iteration is always bounded [3]. As mentioned in our discussion after Assumption 2.4, only recently the stability of SGD under Markovian sampling has been guaranteed to hold for some class of objective function f [9], while the discussion on stability issue under multi-agent scenario with Markovian sampling remains an open problem and we do not pursue to remove this stability assumption in this paper.

Asymptotic analysis: The main results of our work, i.e., almost sure convergence and central limit theorem in distributed optimization, are based on asymptotic analysis and might not accurately represent the finite-sample performance of each contributing agent in the system. The state-of-the-art finite-sample analysis in the literature only focuses on the worst-performing agent that cannot capture the nuanced differences between agents' sampling strategies, with the explanation detailed in Footnote 2. Thus, a finite-sample error bound that distinguish every agent's dynamics is still unknown and regarded as a future direction.

Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and the introduction clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. The claims made match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The 'Limitations' section is provided in Appendix H.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All the theorems, formulas, and proofs in the paper have been numbered and cross-referenced. All assumptions have been clearly stated or referenced in the statement of any theorems. The complete and correct proofs have been provided in appendices.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The detailed simulation setups have been provided in Appendix G.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We do not provide clean source codes, but detailed instructions to perform the experiment have been provided in Appendix G.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The full details have been provided in Appendix G.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We plot the error bars in Figure 2(a). However, error bars are intentionally omitted in Figure 2 in the main body to avoid a cluttered plot and deliver the main message.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We specify the platform that runs the simulation in Appendix G.1.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips . cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We fully obey the NeurIPS Code of Ethics.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This work mainly aims to demonstrate how the improvement of partial agents' sampling strategies can accelerate the overall system's performance. It advocates for improving common wealth and has no negative social impact.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited correctly the datasets ijcnn1 and CIFAR-10 used for experiments in our paper.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides an asymptotic analysis of Unified Distributed SGD (UD-SGD) under heterogeneous agent dynamics and a large family of communication topology. It shows that under certain assumptions: i) regularity of the gradient, **ii) Ergodicity of Markovian sampling**, iii) decreasing step size and intervals not increasing so fast, iv) stability on model parameter v) contraction property of douly-stochastic communication matrix, UD-SGD algorithm guarantees that every agent parameter $\\theta_n$ converges to the same minimum value $\\theta^*$ with some variance matrix $V$ and the mean value of all agents $\\overline{\\theta}=\\sum_{i=1}^n\\theta_i$ also converges to $\\theta_i$ with some variance matrix $V'$.\n\nSTRENGTHS:\n1. Although some previous works discuss the asymptotic analysis for distributed learning, most of them focus on some specific algorithm under a given communication topology and i.i.d sampling. This work is more general. It discusses the asymptotic analysis for more diverse communication topologies with the Markov sampling method.\n2. In order to discuss the theoretical properties of a general distributed learning framework under diverse communication topologies, the author introduce the Unified Distributed SGD (UD-SGD). The clear definition for a unified version of DSGD helps readers better understand the universality of this property.\n3. Let $V_i$ be the limiting covariance matrix of agent i and $V$ be the covariance matrix of the mean of all agents ($V=\\frac{1}{n^2}\\sum_{i=1}^{n} V_i$). The author provides the exact for of $V$.\n4. The provides numerical experiments on logistic regression and neural networks with different sampling strategies to support their theory part.\n\nWEAKNESSES:\n1. In line 26, the authors claim that $\\mathcal{L}$ represents the collection of local minima of objective function $f$. In line 246, the authors claim that $\\theta^*\\in \\mathcal{L}$. Since when $f$ is non-convex, $\\mathcal{L}$ do not have a single element and $\\theta^*$ could have a lot of selections. How could we ensure that the convergence of consensus $\\theta_n$ is unique? And how to choose $\\theta^*$? Will that be the global minima?\n2. The asymptotic analysis Theorem 3.3 is mainly based on Assumption 2.2, which assumes that the dynamic agent $\\{X_{n}^i\\}$ is an ergodic Markov chain. Could the author give some sampling strategies that satisfy this property and are now widely used in DSGD? What are the advantages of Markov sampling compared with the i.i.d sampling method?\n3. In figure 2 of the experimental section, the authors only compare the performance of different Markov sampling methods. Could the author also compare with the iid sampling and sampling shuffle strategies?\n\nQUESTIONS:\nSame as the weakness.\n\nLIMITATIONS:\nSame as the weakness.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

comments: SUMMARY:
This paper studies the asymptotic convergence behavior of federated learning under the UD-SGD framework with Markovian data. The authors establish a new central limit theorem that considers the strategy of every agent, which goes beyond the existing bounds that only focus on the worst-performing agent. Their theory emphasizes the importance of every individual and also explains linear speedup and asymptotic network independence.

STRENGTHS:
1. This paper established theories for a more general federated learning framework and gives a refined analysis to encode the behavior of every agent into their bound.
2. The new asymptotic analysis could explain linear speedup scaling with the number of agents and asymptotic network independence.
3.  The new analysis shows that upgrading only a small group of agents could benefit the whole system and the authors conduct experiments to validate it.
4. The paper is well-written.

WEAKNESSES:
1. As the author mentioned, Assumption 2.4 seems strong and finite-sample analysis is preferred,
2. As far as I could see in the experiments, the author only implemented DSGD but the UD-SGD framework has more algorithms.
3. The linear speedup scaling with the number of agents and asymptotic network independence are only mentioned in theory but do not have empirical justification.
4. The experiments for neural networks are somewhat toy.

QUESTIONS:
1. What is the main technical challenge when utilizing Poisson equation to prove Theorem 3.3? Specifically, what is new compared with the analysis in reference [23] and [30]?
2. Do the authors believe the current asymptotic analysis technique could be extended to (federated) reinforcement learning? I saw some finite-sample analyses for federated RL in your references like [72].

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-09

comments: SUMMARY:
This paper conducts an asymptotic analysis of Unified Distributed SGD (UD-SGD), which has a generalized communication patterns (modelled with a doubly stochastic communication matrix). The paper investigates several different sampling strategies, such as i.i.d. sampling, shuffling, and Markovian sampling

STRENGTHS:
- Rigorous theory: I went through most of the proofs in the appendix and as far as I can tell the theory is sound.
- Well presented: Despite the heavy derivation I find this paper easy to follow as the proof strategies and theoretical insights are discussed clearly in the main exposition.

WEAKNESSES:
Empirical results are quite sparse. Not that this is a big problem for a theory paper, but maybe some additional studies would make it more interesting. I have some suggestions:
- Are all of the results obtained using the same communication matrix (referring to line 982 in appendix G1)? Since the theory is generic for all double stochastic W maybe the authors could present the average result over 5 random W?
- How was local data allocated to each client? The text said each client holds 500 data points from a bigger dataset -- are they iid? If so, it might be interesting to investigate non iid local data (using Dirichlet distribution, which is a common setting in FL)
- How does client dropout (also a common setting in FL) affect the theory and empirical results?

QUESTIONS:
I have no further question.

LIMITATIONS:
No negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-18