Hybrid Local SGD for Federated Learning with Heterogeneous Communications

1 INTRODUCTION Federated learning ( FL ) is a distributed machine learning paradigm in which multiple edge devices or clients cooperate to learn a machine learning model under the orchestration of a central server , and enables a wide range of applications such as autonomous driving , extended reality , and smart manufacturing ( Kairouz et al. , 2019 ) . Communication is a critical bottleneck in FL as the clients are typically connected to the central server over bandwidth-limited networks . Standard optimization methods such as distributed SGD are often not suitable in FL and can cause high communication costs due to the frequent exchange of large-size model parameters or gradients . To tackle this issue , local SGD , in which clients update their models by running multiple SGD iterations on their local datasets before communicating with the server , has emerged as the de facto optimization method in FL and can largely reduce the number of communication rounds required to train a model ( McMahan et al. , 2017 ; Stich , 2019 ) . However , the communication benefit of local SGD is highly sensitive to non-iid data distribution as observed in prior work ( Rothchild et al. , 2020 ; Karimireddy et al. , 2020 ) . Intuitively , taking many local iterations of SGD on local dataset that is not representative of the overall data distribution will lead to local over-fitting , which will hinder convergence . In particular , it is shown in ( Zhao et al. , 2018 ) that the convergence of local SGD on non-iid data could slow down as much as proportionally to the number of local iteration steps taken . Therefore , local SGD with a large aggregation period can converge very slow on non-iid data distribution , and this may nullify its communication benefit ( Rothchild et al. , 2020 ) . Local SGD assumes a star network topology where each device connects to the central server for model aggregation . In modern communication networks , rather than only communicating with the server over slow communication links , devices are increasingly connected to others over fast communication links . For instance , in 5G-and-beyond mobile networks , mobile devices can directly communicate with their nearby devices via device-to-device links of high data rate ( Asadi et al. , 2014 ) . Also , edge devices within the same local-area network ( LAN ) domain can communicate with each other rapidly without traversing through slow wide-area network ( WAN ) ( Yuan et al. , 2020 ) . This gives the potential to accelerate the FL convergence under non-iid data distribution by leveraging fast D2D cooperation so that the total training time can be reduced in FL over bandwidthlimited networks . Motivated by the above observation , this paper proposes hybrid local SGD ( HL-SGD ) , a new distributed learning algorithm for FL with heterogeneous communications , to speed up the learning process and reduce the training time . HL-SGD extends local SGD with fast gossip-style D2D communication after local iterations to mitigate the local over-fitting issue under non-iid data distribution and accelerate convergence . A hybrid model aggregation scheme is designed in HL-SGD to integrate both fast device-to-device ( D2D ) and slow device-to-server ( D2S ) cooperations . We analyze the convergence of HL-SGD in the presence of heterogeneous data for general nonconvex settings , and characterize the relationship between the optimality error bound and algorithm parameters . Our algorithm and analysis are general enough and subsume previously proposed SGD variations such as distributed SGD , local SGD and gossip SGD . Specifically , we consider the FL setting in which all devices are partitioned into disjoint clusters , each of which includes a group of connected devices capable of communicating with each other using fast D2D links . The clustering can be a natural result of devices belonging to different LAN domains so that those devices connected to the same LAN domain are considered as one cluster . In another example , clustering is based on the geographic locations of mobile devices so that devices in a cluster are connected to each other through D2D communication links . In summary , the paper makes the following main contributions : • We propose a novel distributed learning algorithm for FL called HL-SGD to address the communication challenge of FL over bandwidth-limited networks by leveraging the availability of fast D2D links to accelerate convergence under non-iid data distribution and reduce training time . • We provide the convergence analysis of HL-SGD under general assumptions about the loss function , data distribution , and network topology , generalizing previous results on distributed SGD , local SGD , and gossip SGD . • We conduct extensive empirical experiments on two common benchmarks under realistic network settings to validate the established theoretical results of HL-SGD . Our experimental results show that HL-SGD can largely accelerate the learning process and speed up the runtime . 2 BACKGROUND AND RELATED WORK Large-scale machine learning based on distributed SGD has been well studied in the past decade , but often suffers from large network delays and bandwidth limits ( Bottou et al. , 2018 ) . Considering that communication is a major bottleneck in federated settings , local SGD has been proposed recently to reduce the communication frequency by running SGD independently in parallel on different devices and averaging the sequences only once in a while ( Stich , 2019 ; Lin et al. , 2019 ; Haddadpour et al. , 2019 ; Yu et al. , 2018 ; Wang et al. , 2021 ) . However , they all assume the client-server architecture and do not leverage the fast D2D communication capability in modern communication networks . Some studies ( Liu et al. , 2020 ; Abad et al. , 2020 ; Castiglia et al. , 2020 ) develop hierarchical FL algorithms that first aggregate client models at local edge servers before aggregating them at the cloud server or with neighboring edge servers , but they still rely on D2S communication links only and suffer from the scalability and fault-tolerance issues of centralized setting . On the other hand , while existing works on decentralized or gossip SGD consider D2D communications ( Tsitsiklis , 1984 ; Boyd et al. , 2006 ) , they assume a connected cluster with homogeneous communication links and will converge very slow on the large and sparse network topology that is typically found in FL settings . Unlike previous works , HL-SGD leverages both D2S and D2D communications in the system . Some recent studies aim to encapsulate variants of SGD under a unified framework . Specifically , a cooperative SGD framework is introduced in ( Wang & Joshi , 2018 ) that includes communication reduction through local SGD steps and decentralized mixing between clients under iid data distribution . A general framework for topology-changing gossip SGD under both iid and non-iid data distributions is proposed in ( Koloskova et al. , 2020 ) . Note that all of the above works assume undirected network topology for communications in every iteration . In comparison , our proposed HL-SGD is different : the D2S communication is asymmetric due to the use of device sampling and model broadcasting in each global aggregation round and can not be modeled in an undirected graph . Therefore , the convergence analysis of HL-SGD does not fit into the prior frameworks and is much more challenging . Moreover , our major focus is on the runtime of the algorithm rather than its convergence speed in iterations . 3 SYSTEM MODEL In this section , we introduce the FL system model , problem formulation , and assumptions we made . Notation . All vectors in this paper are column vectors by default . For convenience , we use 1 to denote the all-ones vector of appropriate dimension , 0 to denote the all-zeros vector of appropriate dimension , and [ n ] to denote the set of integers { 1 , 2 , . . . , n } with any positive integer n. Let ‖·‖ denote the ` 2 vector norm and Frobenius matrix norm and ‖·‖2 denote the spectral norm of a matrix . We consider a FL system consisting of a central server and K disjoint clusters of edge devices . Devices in each cluster k ∈ [ K ] can communicate with others across an undirected and connected graph Gk = ( V , Ek ) , where Vk denotes the set of edge devices in the cluster , and edge ( i , j ) ∈ Ek denotes that the pair of devices i , j ∈ Vk can communicate directly using D2D as determined by the communication range of D2D links . Besides , each device can directly communicate with the central server using D2S links . Denote the set of all devices in the system as V : = ⋃ k∈ [ K ] Vk , the number of devices in each cluster k ∈ [ K ] as n : = |Vk| , and the total number of devices in the system as N : = ∑ k∈ [ K ] n 1 . The FL goal of the system is to solve an optimization problem of the form : min x∈Rd f ( x ) : = 1 N ∑ i∈V fi ( x ) : = 1 K ∑ k∈ [ K ] f̄k ( x ) , ( 1 ) where fi ( x ) : = Ez∼Di [ ` i ( x ; z ) ] is the local objective function of device i , f̄k ( x ) : = ( 1/n ) ∑ i∈Vk fi ( x ) is the local objective function of cluster k , and Di is the data distribution of device i . Here ` i is the ( non-convex ) loss function defined by the learning model and z represents a data sample from data distribution Di . When applying local SGD to ( 1 ) in FL with heterogeneous communications , the communications between the server and devices in FL are all through D2S links that are bandwidth-limited , particularly for the uplink transmissions . Therefore , the incurred communication delay is high . Due to the existing of high-bandwidth D2D links that are much more efficient than low-bandwidth D2S links , it would be highly beneficial if we can leverage D2D links to reduce the usage of D2S links such that the total training time can be reduced . This motivates us to design a new learning algorithm for FL with heterogeneous communications . 4 HYBRID LOCAL SGD In this section , we present our HL-SGD algorithm suitable for the FL setting with heterogeneous communications . Algorithm 1 provides pseudo-code for our algorithm . At the beginning of r-th global communication round , the server broadcasts the current global model xr to all devices in the system via cellular links ( Line 4 ) . Note that in typical FL systems , the downlink communication is much more efficient than uplink communication due to the larger bandwidth allocation and higher data rate . Therefore , devices only consume a smaller amount of energy when receiving data from the server compared with transmitting data to the server . After that , devices in each cluster initialize their local models to be the received global model and run τ iterations of gossip-based SGD via D2D links to update their local models in parallel ( lines 5– 9 ) . Let xr , si denote the local model of device i at the r-th local iteration of s-th round . Here 1For presentation simplicity , we assume each cluster contains the same number of devices here . The results of this paper can be extended to the case of clusters with different device numbers as well . Algorithm 1 HL-SGD : Hybrid Local SGD Input : initial global model x0 , learning rate η , communication graph Gk and mixing matrix Wk for all clusters k ∈ [ K ] , and fraction of sampled devices in each cluster p. Output : final global model xR 1 : for each round r = 0 , . . . , R− 1 do 2 : for each cluster k ∈ [ K ] in parallel do 3 : for each device i ∈ Vk in parallel do 4 : xr,0i = x r 5 : for s = 0 , . . . , τ − 1 do 6 : Compute a stochastic gradient gi over a mini-batch ξi sampled from Di 7 : x r , s+ 12 i = x r , s i − ηgi ( x r , s i ) . local update 8 : xr , s+1i = ∑ j∈Nki ( Wk ) i , jx r , s+ 12 j . gossip averaging 9 : end for 10 : end for 11 : end for 12 : for each cluster k ∈ [ K ] do 13 : m← max ( p · n , 1 ) 14 : Srk ← ( random set of m clients in Vk ) . device sampling 15 : end for 16 : xr+1 = 1K ∑ k∈ [ K ] 1 m ∑ i∈Srk xr , τi . global aggregation 17 : end for 18 : return xR each gossip-based SGD iteration consists of two steps : ( i ) SGD update , performed locally on each device ( lines 6–7 ) , followed by a ( ii ) gossip averaging , where devices average their models with their neighbors ( line 8 ) . In the gossip averaging protocol , N ki denotes the neighbors of device i , including itself , on the D2D communication graph Gk of cluster k , and Wk ∈ [ 0 , 1 ] n×n denotes the mixing matrix of cluster k with each element ( Wk ) i , j being the weight assigned by device i to device j . Note that ( Wk ) i , j > 0 only if devices i and j are directly connected via D2D links . Next , a set Srk of m devices are sampled uniformly at random ( u.a.r . ) with probability p without replacement from each cluster k ∈ [ K ] by the server ( lines 13–14 ) , and their final updated local models { xr , τi , ∀i ∈ Srk } are sent to the server via D2S links . After that , the server updates the global model xr+1 by averaging the received local models from all sampled devices ( line 16 ) . Note that only m devices per cluster will upload their models to the server in each round to save the usage of expensive D2S uplink transmissions . The intuition is that after multiple iterations of gossip-based SGD , devices have already reached approximate consensus within each cluster , and the sampled average can well represent the true average . By trading D2D local aggregation for D2S global aggregation , the total communication cost can be reduced . We will empirically validate such benefits later in the experiments . It is worth noting that HL-SGD inherits the privacy benefits of classic FL schemes by keeping the raw data on device and sharing only model parameters . Moreover , HL-SGD is compatible with existing privacy-preserving techniques in FL such as secure aggregation ( Bonawitz et al. , 2017 ) , differential privacy ( McMahan et al. , 2018 ) , and shuffling ( Girgis et al. , 2021 ) since only the sum rather than individual values is needed for the local and global model aggregation steps . 4.1 RUNTIME ANALYSIS OF HL-SGD We now present a runtime analysis of HL-SGD . Here we ignore the communication time of downloading models from the server by each device since the download bandwidth is often much larger than upload bandwidth for the D2S communication in practice ( Kairouz et al. , 2019 ) . In each round of HL-SGD , we denote the average time taken by a device to compute a local update , perform one round of D2D communication and one round of D2S communication as ccp , cd2d and cd2s , respectively . Assume the uplink bandwidth between the server and devices is fixed and evenly shared among the sampled devices in each round , then cd2s is linearly proportional to the sampling ratio p. Similarly , ccp depends on the D2D network topology Gk and typically increases with the maximum node degree ∆ ( Gk ) . The total runtime of HL-SGD after R communication rounds is R× [ τ × ( ccp + cd2d ) + cd2s ] . ( 2 ) The specific values of ccp , cd2d and cd2s depend on the system configurations and applications . In comparison , the total runtime of local SGD after R communication rounds is R× [ τ × ccp + cd2s ] . 4.2 PREVIOUS ALGORITHMS AS SPECIAL CASES When devices do not communicate with each other , i.e. , Wk = I , ∀k ∈ [ K ] , and sampling ratio p = 1 , HL-SGD reduces to distributed SGD ( when τ = 1 ) or local SGD ( when τ > 1 ) where each device only directly communicates with the server with D2S links . Also , when τ → ∞ , HLSGD reduces to gossip SGD where devices only cooperate with their neighboring devices through a gossip-based communication protocol with D2D links to update their models without relying on the server . Therefore , HL-SGD subsumes existing algorithms and enables us to strike the best balance between runtime and model accuracy by tuning τ , Wk , and p. However , due to the generality of HL-SGD , there exist significantly new challenges in its convergence analysis , which constitutes one of the main contributions of this paper as elaborated in the following section . 5 CONVERGENCE ANALYSIS OF HL-SGD In this section , we analyze the convergence of HL-SGD with respect to the gradient norm of the objective function f ( · ) , specifically highlighting the effects of τ and p. Before stating our results , we make the following assumptions : Assumption 1 ( Smoothness ) . Each local objective function fi : Rd → R is L-smooth for all i ∈ V , i.e. , for all x , y ∈ Rd , ‖∇fi ( x ) −∇fi ( y ) ‖ ≤ L‖x− y‖ , ∀i ∈ V. Assumption 2 ( Unbiased Gradient and Bounded Variance ) . The local mini-batch stochastic gradient in Algorithm 1 is unbiased , i.e. , Eξi [ gi ( x ) ] = ∇fi ( x ) , and has bounded variance , i.e. , Eξi‖gi ( x ) − ∇fi ( x ) ‖2 ≤ σ2 , ∀x ∈ Rd , i ∈ V , where the expectation is over all the local minibatches . Assumption 3 ( Mixing Matrix ) . For any cluster k ∈ [ K ] , the D2D network is strongly connected and the mixing matrix Wk ∈ [ 0 , 1 ] n×n satisfies Wk1 = 1 , 1 > Wk = 1 > , null ( I −Wk ) = span ( 1 ) . We also assume ||Wk − ( 1/n ) 11 > ||2 ≤ ρk for some ρk ∈ [ 0 , 1 ) . Assumption 4 ( Bounded Intra-Cluster Dissimilarity ) . There exists a constant k ≥ 0 such that ( 1/n ) ∑ i∈Vk ‖∇fi ( x ) − ∇f̄k ( x ) ‖ 2 ≤ 2k for any x ∈ Rd and k ∈ [ K ] . If local functions are identical to each other within a cluster , then we have k = 0 . Assumption 5 ( Bounded Inter-Cluster Dissimilarity ) . There exist constants α ≥ 1 , ≥ 0 such that ( 1/K ) ∑ k∈ [ K ] ‖∇f̄k ( x ) ‖2 ≤ α2 ‖∇f ( x ) ‖ 2 + 2g for any x ∈ Rd . If local functions are identical to each other across all clusters , then we have α = 1 , g = 0 . Assumptions 1–3 are standard in the analysis of SGD and decentralized optimization ( Bottou et al. , 2018 ; Koloskova et al. , 2019 ) . Assumptions 4–5 are commonly used in the federated optimization literature to capture the dissimilarities of local objectives ( Koloskova et al. , 2020 ; Wang et al. , 2020 ) . 5.1 MAIN RESULTS We now provide the main theoretical results of the paper in Theorem 1 and Theorem 2 . The detailed proofs are provided in the appendices . Define the following constants : ρmax = max k∈ [ K ] ρk , Dτ , ρ = min { 1 1− ρmax , τ } , ̄2L = 1 K K∑ k=1 2k ( 3 ) and let r0 = 8 ( f ( x 0 ) − f ( x ? ) ) , r1 = 16L ( σ2 N ) , r2 = 16C1L 2τ2 2g + 16C1L 2 ( τρ2maxDτ , ρ̄ 2 L + τσ 2 ( 1 n + ρ2max ) ) . ( 4 ) Theorem 1 ( Full device participation ) . Let Assumptions 1–5 hold , and let L , σ , ̄L , g , Dτ , ρ , ρmax , r0 , r1 , and r2 be as defined therein . If the learning rate η satisfies η = min { 1 4C1α · 1 τL , ( r0 r1τR ) 1 2 , ( r0 r2τR ) 1 3 } , ( 5 ) then for any R > 0 , the iterates of Algorithm 1 with full device participation for HL-SGD satisfy min r∈ [ R ] min s=0 , ... , τ−1 E‖∇f ( x̄r , s ) ‖2 =O σ√ NτR + ( τ2 2g + τρ 2 maxDτ , ρ̄ 2 L + τ ( 1 n + ρ2max ) σ2 ) 1 3 ( τR ) 2 3 + 1 R , ( 6 ) where x̄r , s = 1N ∑N i=1 x r , s i . In the following , we analyze the iteration complexity of HL-SGD and compare it with those of some classic and state-of-the-art algorithms relevant to our setting in Table 1 . First , we consider two extreme cases of HL-SGD where ρmax = 0 and ρk = 1 , ∀k ∈ [ K ] , and show that our analysis recovers the best known rate of local SGD . Fully Connected D2D networks . In this case , ρmax = 0 , and each cluster can be viewed as a single device , and thus HL-SGD reduces to local SGD with K devices . Substuting ρmax = 0 into ( 6 ) , the iteration complexity of HL-SGD reduces to O ( σ/ √ NτR+ ( τ2 2g + τ · ( σ2/n ) ) 1/3 / ( τR ) 2/3 + 1/R ) . This coincides with the complexity of local SGD provided in Table 1 with device number K and stochastic gradient variance σ2/n thanks to the fully intra-cluster averaging . Disconnected D2D networks . In this case , HL-SGD reduces to local SGD with N devices . Substituting ρmax = 1 into ( 6 ) , the iteration complexity of HL-SGD becomes O ( σ/ √ NτR + ( τ2 ( 2g + ̄ 2 L ) + τσ 2 ) 1/3 / ( τR ) 2/3 + 1/R ) . This coincides with the complexity of local SGD with N devices , stochastic gradient variance σ2 , and gradient heterogeneity of order 2g + ̄ 2 L. Table 1 : Comparison of Iteration Complexity . 2 Local SGD O ( σ√ NτR + ( τ2 2+τσ2 ) 1 3 ( τR ) 2 3 + ττR ) Gossip SGD O ( σ√ NτR + ρ 2 3 2 3 ( τR ) 2 3 ( 1−ρ ) 2 3 + ρ 2 3 σ 2 3 ( τR ) 2 3 ( 1−ρ ) 1 3 + ρ ( 1−ρ ) τR ) Gossip PGA ( Chen et al. , 2021a ) O ( σ√ NτR + C 1 3 τ , ρD 1 3 τ , ρ′ρ 2 3 2 3 ( τR ) 2 3 + C 1 3 τ , ρρ 2 3 σ 2 3 ( τR ) 2 3 + ρDτ , ρ′ τR ) HL-SGD ( this work ) O ( σ√ NτR + ( τ2 2g+τρ 2 maxDτ , ρ̄ 2 L ) 1 3 ( τR ) 2 3 + ( τ ( 1n+ρ 2 max ) σ 2 ) 1 3 ( τR ) 2 3 + ττR ) Next , we compare the complexities of HL-SGD , local SGD , gossip SGD and gossip PGA . Comparison to Local SGD . Comparing ( 6 ) and the complexity of local SGD , we can see the intracluster D2D communication provably improves the iteration complexity by reducing the transient iterations . This is reflected in the smaller coefficient associated with the O ( ( τR ) −2/3 ) term . In particular , improving D2D communication connectivity will lead to a smaller ρmax and consequently , mitigate the impact of both local data heterogeneity and stochastic noise on the convergence rate . Comparison to Gossip SGD . Under the condition that ρ = ρmax , i.e. , the connectivity of D2D network in gossip SGD is the same as that of HL-SGD , Table 1 shows HL-SGD outperforms gossip SGD when τ/n ≤ ρ2/ ( 1−ρ ) . In other words , HL-SGD is beneficial for weakly connected networks , which is the case in FL settings where a large number of devices are often loosely connected or disconnected into several disjoint clusters via D2D communications only . 1The convergence rates for gossip SGD and local SGD are from ( Koloskova et al . ( 2020 ) ) . The parameters in the table are given by the following : σ2 : stochastic gradient variance ; ρ : network connectivity ; 2 : data heterogeneity of order 2g + ̄2L ; Cτ , ρ , ∑τ−1 k=0 ρ k , Dτ , ρ′ = min { 1/ ( 1− ρ ) , τ } . Note that Dτ , ρ 6= Dτ , ρ′ . Comparison to Gossip PGA . Gossip PGA improves local SGD by integrating gossiping among all devices in one round using a connected network . Compared to gossip SGD , gossip PGA has one extra full averaging step with period τ . The complexity of gossip PGA improves both by reducing the transient iterations . HL-SGD ( full participation ) differs from gossip PGA in the sense that gossiping is performed within multiple clusters instead of a single one . The benefit comes from the fact that for many commonly used D2D network topologies , the spectral gap 1− ρ decreases as the network size decreases , see Table 2 . Therefore , when employing the same D2D network topology , HL-SGD enjoys a smaller connectivity number ρmax than ρ . Considering the scenario where τ and n are fixed while the cluster number K grows , the total device number N = nK grows and hence ρ → 1 for gossip PGA . In the case when τ = Dτ , ρ′ ≈ Cτ , ρ , the fastest decaying O ( 1/τR ) terms are comparable for both algorithms . However , the O ( ( τR ) −2/3 ) term of gossip GPA can be larger than that of HL-SGD since ρ increases with N . This observation shows for large-scale networks , it is advantageous to use HL-SGD with multiple connected clusters instead of gossip GPA with a single cluster under the D2D network topology . Our next result shows the iteration complexity of HL-SGD with partial device participation . We assume the devices participate in synchronizing their models at the end of each FL round following the sampling rule given by Assumption 6 . Assumption 6 ( Sampling strategy ) . Each Srk contains a subset ofm indices uniformly sampled from { 1 , . . . , n } without replacement . Furthermore , Srk is independent of Sr ′ k′ for all ( k , r ) 6= ( k′ , r′ ) . Theorem 2 ( Partial device participation ) . Let Assumptions 1–6 hold , and let L , σ , ̄L , g , Dτ , ρ , ρmax , r0 , r1 , and r2 be as defined therein . If the network connectivity satisfies ρmax ≤ 1− 1/τ , ( 7 ) then for suitably chosen learning rate η , the iterates of Algorithm 1 with partial device participation for HL-SGD satisfy min r∈ [ R ] min s=0 , ... , τ−1 E‖∇f ( x̄r , s ) ‖2 =O σ + E ( g , ̄L , σ , ρmax ) √ NτR + ( τ2 2g + τρ 2 maxDτ , ρ̄ 2 L + τ ( 1 n + ρ2max ) σ2 ) 1 3 ( τR ) 2 3 + 1 R , ( 8 ) where x̄r , s = 1N ∑N i=1 x r , s i , E2 ( g , ̄L , σ , ρmax ) = ( 2gDτ , ρ + ρmaxDτ , ρ̄2L + σ2 ) ·G′pDτ , ρρmaxN + n m · 1 τ ρ2maxσ 2 . ( 9 ) and G′p = n−m m ( n− 1 ) + 1 τ2 . ( 10 ) Compared to Theorem 1 , Theorem 2 shows partial device participation deteriorates the rate by O ( E ( g , ̄L , σ , ρmax ) / √ NτR ) . From the expression of E , we observe that as ρmax → 0 , E ( g , ̄L , σ , ρmax ) vanishes , which indicates that the loss caused by device sampling can be compensated by increasing network connectivity uniformly for all clusters . The next corollary finds the critial ρmax so that E2 = O ( 1 ) , and the order of convergence rate of partial device participation matches that of the full participation case . Corollary 1 . Under the same assumptions as Theorem 2 , if the network connectivity satisfies ρmax ≤ 1 4N min { m , τ − 1 } , ( 11 ) then min r∈ [ R ] min s=0 , ... , τ−1 E‖∇f ( x̄r , s ) ‖2 =O σ + g + ̄L√ NτR + ( τ2 2g + τρ 2 maxDτ , ρ̄ 2 L + τ ( 1 n + ρ2max ) σ2 ) 1 3 ( τR ) 2 3 + 1 R . ( 12 ) Corollary 1 reveals the tradeoff between sampling intensity and network connectivity . More connected D2D networks result in smaller ρmax , and thus ( 11 ) can be satisfied by a smaller m. This means we can sample fewer devices at the end of each round and reduce the D2S communication delay when the D2D network is more connected . 6 EXPERIMENTAL EVALUATION 6.1 EXPERIMENTAL SETTINGS We use two common datasets in FL literature ( McMahan et al. , 2017 ; Reddi et al. , 2020 ; Wang et al. , 2020 ) : Federated Extended MNIST ( Caldas et al. , 2018 ) ( FEMNIST ) and CIFAR-10 ( Krizhevsky et al. , 2009 ) . The 62-class FEMNIST is built by partitioning the data in Extended MNIST ( Cohen et al. , 2017 ) based on the writer of the digit/character and has a naturally-arising device partitioning . CIFAR-10 is partitioned across all devices using a Dirichlet distribution Dir ( 0.1 ) as done in ( Hsu et al. , 2019 ; Yurochkin et al. , 2019 ; Reddi et al. , 2020 ; Wang et al. , 2020 ) . We evaluate our algorithms by training CNNs on both datasets , and the CNN models for FEMNIST and CIFAR-10 were taken from ( Caldas et al. , 2018 ) and ( McMahan et al. , 2017 ) with around 6.5 and 1 million parameters , respectively . For each dataset , the original testing set ( without partitioning ) is used to evaluate the generalization performances of the trained global model . We consider a FL system consisting of a central server and 32 devices . The devices are evenly divided into four clusters , and each cluster has a ring topology by default , which provides a conservative estimation for the cluster connectivity and convergence speed . In our experiments , the mixing matrix of each cluster Wk is set according to the Metropolis-Hastings weights ( Nedić et al. , 2018 ) . According to the real-world measurements in ( Yuan et al. , 2020 ; Yang et al. , 2021 ) , we set the average time for a device to perform a local update , a round of D2D communication under ring topology , and a round of D2S communication with one device sampled per cluster to be ccp = 0.01h , cd2d ( ∆ = 2 ) = 0.005h and cd2s ( p = 1/8 ) = 0.05h , respectively , in the runtime model ( 2 ) . For arbitrary device sampling ratio and D2D network topology , we consider a linear-scaling rule ( Wang et al. , 2019 ) and let cd2d ( ∆ ) = ( ∆/2 ) × 0.005h and cd2s ( p ) = 8p× 0.05h . We compare HL-SGD with local SGD in the experiments . For local SGD , devices will only communicate with the central server periodically . In all experiments , we let the local iteration period τ to be the same for both local SGD and HL-SGD to have a fair comparison . On the FEMNIST dataset , we fix the batch size as 30 and tune the learning rate η from { 0.005 , 0.01 , 0.02 , 0.05 , 0.08 } for each algorithm separately . On the CIFAR-10 dataset , we fix the batch size as 50 and tune η from { 0.01 , 0.02 , 0.05 , 0.08 , 0.1 } for each algorithm separately . We run each experiment with 3 random seeds and report the average . All experiments in this paper are conducted on a Linux server with 4 NVIDIA RTX 8000 GPUs . The algorithms are implemented by PyTorch . More details are provided in Appendix F. 6.2 EXPERIMENTAL RESULTS We first compare the convergence speed and runtime of HL-SGD and local SGD while fixing τ = 50 and p = 1 . We measure the test accuracy of the global model on the server in every FL round . Figure 1 shows the convergence process . From the figure , we can observe that HL-SGD can largely accelerate the model convergence while improving model accuracy in FL . On FEMNIST , the best accuracy of HL-SGD achieved over 100 rounds is 4.78 % higher than that of local SGD ( i.e. , 83.76 % vs. 79.94 % ) , and its runtime necessary to achieve a target test-set accuracy of 75 % is only 17.64 % of that of the baseline ( i.e. , 5.67× speedup ) . On CIFAR-10 , the best accuracy of HL-SGD achieved over 100 rounds is 9.32 % higher than that of local SGD ( i.e. , 70.13 % vs. 64.15 % ) , and its runtime necessary to achieve a target test-set accuracy of 60 % is 18.36 % less than that of local SGD ( i.e. , 1.225× speedup ) . Next , we investigate the effect of D2D network speed on the runtime benefit of HL-SGD . We consider the same FL setting as before and choose cd2d from { 0.003 , 0.005 , 0.02 } h. As illustrated in Figure 2 , the training speed of HL-SGD can be further improved by using faster D2D links . For instance , it takes HL-SGD 4.2 hours to reach 75 % test accuracy on FEMNIST with cd2d = 0.003h , which is 1.5× faster than HL-SGD with cd2d = 0.01h . On CIFAR-10 , to reach a target level of 60 % test accuracy , HL-SGD with cd2d = 0.003h needs 21 hours , which is 1.14× faster than HL-SGD with cd2d = 0.005 . Note that the runtime can not proportionally decrease with the increase of D2D network speed because local update also takes substantial computation time . Also , when cd2d is relatively large compared with cd2s , the convergence acceleration benefit of gossip averaging could be offset by the increased communication time in each round of HL-SGD , leading to an inferior total runtime as shown in Figure 2b . Finally , we investigate how the sampling ratio p affects the performance of HL-SGD . We select p from { 0.125 , 0.25 , 0.5 , 1 } , corresponding to sampling { 1 , 2 , 4 , 8 } devices from each cluster to upload models to the server . Figure 3 depicts the best value of test-set accuracy achieved over all prior rounds . As can be observed from the figures , sampling one device per cluster only results in slightly lower model accuracy , e.g. , 1.92 % drop compared to full participation on CIFAR-10 . This matches the theoretical result in Corollary 1 that device sampling does not affect the order of convergence rate under certain conditions . However , decreasing p can lead to faster training speed due to its shorter D2S communication delay as observed in Figure 3d . In practice , the optimal value of p needs to be tuned to strike a good balance between model accuracy and runtime . 7 CONCLUSION In this paper , we have proposed a new optimization algorithm called HL-SGD for FL with heterogeneous communications . Our algorithm leverages the D2D communication capabilities among edge device to accelerate the model convergence while improving model accuracy in FL . We have provided the theoretical convergence analysis of HL-SGD and conducted experiments to demonstrate the benefits of HL-SGD . In the future , we plan to extend HL-SGD to handle straggler issues under device heterogeneity and provide rigorous privacy protection for HL-SGD . REFERENCES Mehdi Salehi Heydar Abad , Emre Ozfatura , Deniz Gunduz , and Ozgur Ercetin . Hierarchical federated learning across heterogeneous cellular networks . In ICASSP 2020-2020 IEEE International Conference on Acoustics , Speech and Signal Processing ( ICASSP ) , pp . 8866–8870 . IEEE , 2020 . Arash Asadi , Qing Wang , and Vincenzo Mancuso . A survey on device-to-device communication in cellular networks . IEEE Communications Surveys & Tutorials , 16 ( 4 ) :1801–1819 , 2014 . Keith Bonawitz , Vladimir Ivanov , Ben Kreuter , Antonio Marcedone , H Brendan McMahan , Sarvar Patel , Daniel Ramage , Aaron Segal , and Karn Seth . Practical secure aggregation for privacypreserving machine learning . In proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security , pp . 1175–1191 , 2017 . Léon Bottou , Frank E Curtis , and Jorge Nocedal . Optimization methods for large-scale machine learning . Siam Review , 60 ( 2 ) :223–311 , 2018 . Stephen Boyd , Arpita Ghosh , Balaji Prabhakar , and Devavrat Shah . Randomized gossip algorithms . IEEE transactions on information theory , 52 ( 6 ) :2508–2530 , 2006 . Sebastian Caldas , Sai Meher Karthik Duddu , Peter Wu , Tian Li , Jakub Konečnỳ , H Brendan McMahan , Virginia Smith , and Ameet Talwalkar . Leaf : A benchmark for federated settings . arXiv preprint arXiv:1812.01097 , 2018 . Timothy Castiglia , Anirban Das , and Stacy Patterson . Multi-level local sgd for heterogeneous hierarchical networks . arXiv preprint arXiv:2007.13819 , 2020 . Yiming Chen , Kun Yuan , Yingya Zhang , Pan Pan , Yinghui Xu , and Wotao Yin . Accelerating gossip sgd with periodic global averaging . In International Conference on Machine Learning . PMLR , 2021a . Yiming Chen , Kun Yuan , Yingya Zhang , Pan Pan , Yinghui Xu , and Wotao Yin . Accelerating gossip sgd with periodic global averaging . In Marina Meila and Tong Zhang ( eds . ) , Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pp . 1791–1802 . PMLR , 18–24 Jul 2021b . Gregory Cohen , Saeed Afshar , Jonathan Tapson , and Andre Van Schaik . Emnist : Extending mnist to handwritten letters . In 2017 International Joint Conference on Neural Networks ( IJCNN ) , pp . 2921–2926 . IEEE , 2017 . Antonious Girgis , Deepesh Data , Suhas Diggavi , Peter Kairouz , and Ananda Theertha Suresh . Shuffled model of differential privacy in federated learning . In International Conference on Artificial Intelligence and Statistics , pp . 2521–2529 . PMLR , 2021 . Farzin Haddadpour , Mohammad Mahdi Kamani , Mehrdad Mahdavi , and Viveck Cadambe . Local sgd with periodic averaging : Tighter analysis and adaptive synchronization . In Advances in Neural Information Processing Systems , pp . 11082–11094 , 2019 . Tzu-Ming Harry Hsu , Hang Qi , and Matthew Brown . Measuring the effects of non-identical data distribution for federated visual classification . arXiv preprint arXiv:1909.06335 , 2019 . Peter Kairouz , H Brendan McMahan , Brendan Avent , Aurélien Bellet , Mehdi Bennis , Arjun Nitin Bhagoji , Keith Bonawitz , Zachary Charles , Graham Cormode , Rachel Cummings , et al . Advances and open problems in federated learning . arXiv preprint arXiv:1912.04977 , 2019 . Sai Praneeth Karimireddy , Satyen Kale , Mehryar Mohri , Sashank Reddi , Sebastian Stich , and Ananda Theertha Suresh . Scaffold : Stochastic controlled averaging for federated learning . In International Conference on Machine Learning , pp . 5132–5143 . PMLR , 2020 . Anastasia Koloskova , Sebastian Stich , and Martin Jaggi . Decentralized stochastic optimization and gossip algorithms with compressed communication . In International Conference on Machine Learning , pp . 3478–3487 , 2019 . Anastasia Koloskova , Nicolas Loizou , Sadra Boreiri , Martin Jaggi , and Sebastian U Stich . A unified theory of decentralized sgd with changing topology and local updates . arXiv preprint arXiv:2003.10422 , 2020 . Alex Krizhevsky , Geoffrey Hinton , et al . Learning multiple layers of features from tiny images . 2009 . Tao Lin , Sebastian U Stich , Kumar Kshitij Patel , and Martin Jaggi . Don ’ t use large mini-batches , use local sgd . In International Conference on Learning Representations , 2019 . Lumin Liu , Jun Zhang , SH Song , and Khaled B Letaief . Client-edge-cloud hierarchical federated learning . In ICC 2020-2020 IEEE International Conference on Communications ( ICC ) , pp . 1–6 . IEEE , 2020 . Brendan McMahan , Eider Moore , Daniel Ramage , Seth Hampson , and Blaise Aguera y Arcas . Communication-efficient learning of deep networks from decentralized data . In Artificial Intelligence and Statistics , pp . 1273–1282 , 2017 . H Brendan McMahan , Daniel Ramage , Kunal Talwar , and Li Zhang . Learning differentially private recurrent language models . In International Conference on Learning Representations , 2018 . Angelia Nedić , Alex Olshevsky , and Michael G Rabbat . Network topology and communicationcomputation tradeoffs in decentralized optimization . Proceedings of the IEEE , 106 ( 5 ) :953–976 , 2018 . Sashank Reddi , Zachary Charles , Manzil Zaheer , Zachary Garrett , Keith Rush , Jakub Konečnỳ , Sanjiv Kumar , and H Brendan McMahan . Adaptive federated optimization . arXiv preprint arXiv:2003.00295 , 2020 . Daniel Rothchild , Ashwinee Panda , Enayat Ullah , Nikita Ivkin , Ion Stoica , Vladimir Braverman , Joseph Gonzalez , and Raman Arora . Fetchsgd : Communication-efficient federated learning with sketching . In International Conference on Machine Learning , pp . 8253–8265 . PMLR , 2020 . Sebastian Urban Stich . Local sgd converges fast and communicates little . In International Conference on Learning Representations ( ICLR ) , 2019 . John Nikolas Tsitsiklis . Problems in decentralized decision making and computation . Technical report , Massachusetts Inst of Tech Cambridge Lab for Information and Decision Systems , 1984 . Jianyu Wang and Gauri Joshi . Cooperative sgd : A unified framework for the design and analysis of communication-efficient sgd algorithms . arXiv preprint arXiv:1808.07576 , 2018 . Jianyu Wang , Anit Kumar Sahu , Zhouyi Yang , Gauri Joshi , and Soummya Kar . Matcha : Speeding up decentralized sgd via matching decomposition sampling . arXiv preprint arXiv:1905.09435 , 2019 . Jianyu Wang , Qinghua Liu , Hao Liang , Gauri Joshi , and H Vincent Poor . Tackling the objective inconsistency problem in heterogeneous federated optimization . Advances in neural information processing systems , 2020 . Jianyu Wang , Zachary Charles , Zheng Xu , Gauri Joshi , H Brendan McMahan , Maruan Al-Shedivat , Galen Andrew , Salman Avestimehr , Katharine Daly , Deepesh Data , et al . A field guide to federated optimization . arXiv preprint arXiv:2107.06917 , 2021 . Chengxu Yang , Qipeng Wang , Mengwei Xu , Zhenpeng Chen , Kaigui Bian , Yunxin Liu , and Xuanzhe Liu . Characterizing impacts of heterogeneity in federated learning upon large-scale smartphone data . In Proceedings of the Web Conference 2021 , pp . 935–946 , 2021 . Bicheng Ying , Kun Yuan , Yiming Chen , Hanbin Hu , Pan Pan , and Wotao Yin . Exponential graph is provably efficient for decentralized deep training . arXiv preprint arXiv:2110.13363 , 2021 . Hao Yu , Sen Yang , and Shenghuo Zhu . Parallel restarted sgd with faster convergence and less communication : Demystifying why model averaging works for deep learning , 2018 . Jinliang Yuan , Mengwei Xu , Xiao Ma , Ao Zhou , Xuanzhe Liu , and Shangguang Wang . Hierarchical federated learning through lan-wan orchestration . arXiv preprint arXiv:2010.11612 , 2020 . Mikhail Yurochkin , Mayank Agarwal , Soumya Ghosh , Kristjan Greenewald , Nghia Hoang , and Yasaman Khazaeni . Bayesian nonparametric federated learning of neural networks . In International Conference on Machine Learning , pp . 7252–7261 . PMLR , 2019 . Yue Zhao , Meng Li , Liangzhen Lai , Naveen Suda , Damon Civin , and Vikas Chandra . Federated learning with non-iid data . arXiv preprint arXiv:1806.00582 , 2018 . A PRELIMINARIES Intra-cluster dynamics . To facilitate the analysis , we introduce matrices Xk ∈ Rn×d and Gk ∈ Rn×d constructed by stacking respectively xi and gi for i ∈ Vk row-wise . Similarly , we define the pseudo-gradient ∇Fk ( Xk ) ∈ Rn×d associated to cluster k by stacking ∇fi ( xi ) for i ∈ Vk row-wise . In addition , define the following intra-cluster averages for each cluster k : x̄k , 1 n ∑ i∈Vk xi and ḡk , 1 n ∑ i∈Vk gi . ( 13 ) The update within each cluster then can be written compactly in matrix form as Xr , s+1k = Wk ( X r , s k − ηG r , s k ) , ∀k = 1 , . . . , K. ( 14 ) Since each Wk is bi-stochastic , we obtain the following update of the intra-cluster average x̄r , s+1k = x̄ r , s k − η · ḡ r , s k . ( 15 ) We proceed to derive the update of the intra-cluster consensus error . Define the averaging matrix J = 1 n 1 · 1 > with 1 = [ 1 , . . . , 1︸ ︷︷ ︸ n ] . ( 16 ) Multiplying both sides of ( 14 ) from the left by ( I−J ) leads to the following update of the consensus error : ( I − J ) Xr , s+1k︸ ︷︷ ︸ Xr , s+1k , ⊥ = ( I − J ) Wk ( Xr , sk − ηG r , s k ) = ( Wk − J ) ( Xr , sk − ηG r , s k ) = ( Wk − J ) ( Xr , sk , ⊥ − ηG r , s k ) . ( 17 ) Global average dynamics . Define the global average among all xi ’ s as x̄ , 1 N N∑ i=1 xi . ( 18 ) Then accordingly to ( 15 ) we have the following update of x̄ for all s = 0 , . . . , τ − 1 : x̄r , s+1 = 1 N K∑ k=1 nx̄r , s+1k = 1 N K∑ k=1 n ( x̄r , sk − ηḡ r , s k ) = 1 N N∑ i=1 ( xr , si − ηg r , s i ) = x̄ r , s − η 1 N N∑ i=1 gr , si . ( 19 ) Filtration . Let G = [ G1 ; . . . ; GK ] ∈ RN×d be the matrix constructed by stacking all the stochastic gradients . We introduce the following filtration Fr , s = σ ( G0,0 , . . . , G0 , τ−1 , S0 , G 1,0 , . . . , G1 , τ−1 , . . . , Sr−1 , G r,0 , . . . , Gr , s ) Fr = σ ( G0,0 , . . . , G0 , τ−1 , S0 , G 1,0 , . . . , G1 , τ−1 , . . . , Sr−1 ) . ( 20 ) Therefore we have xr,0i = x r ∈ Fr for r ≥ 1 , and xr , si ∈ Fr , s−1 for 1 ≤ s ≤ τ . For simplicity the conditional expectation E ( · |Fr , s ) is denoted as Er , s , and we define the noise in the stochastic gradient as ξr , si , g r , s i −∇fi ( x r , s i ) . ( 21 ) Since at the end of round r all nodes are picked with equal probability , the sampling procedure preserves average in expectation : Er , τ−2xr+1 = E ( E ( xr+1|Fr , τ−1 ) |Fr , τ−2 ) = E E 1 K K∑ k=1 1 m ∑ i∈Srk xr , τi |Fr , τ−1 |Fr , τ−2 = E ( E ( 1 K K∑ k=1 1 m ∑ i∈Vk I ( i ∈ Srk ) x r , τ i |Fr , τ−1 ) |Fr , τ−2 ) = Er , τ−2 ( x̄r , τ ) ( 22 ) where the last equality holds since P ( i ∈ Srk|i ∈ Vk ) = mn . B CONVERGENCE ANALYSIS To prove the convergence we first establish in Sec . B.1 that the objective value Ef ( xr ) is descending at each round r , up to some consensus error terms . Subsequently , bounds on the error terms are provided in Sec . B.2-B.4 . Based on these results , the proof of convergence of Algorithm 1 with full and partial device participation are given in Sec . B.5 and B.6 , respectively . The proofs of the main propositions are given in Sec . C and that of the supporting lemmas are deferred to Sec . D. B.1 OBJECTIVE DESCENT Lemma 1 . Let { xr , si } be the sequence generated by Algorithm 1 under Assumptions 1-6 . If η > 0 , then the following inequality holds for all r ∈ N+ : Ef ( xr+1 ) ≤ Ef ( xr ) − η 4 τ−1∑ s=0 E‖∇f ( x̄r , s ) ‖2 − η 4 τ−1∑ s=0 E ∥∥∥ 1 N N∑ i=1 ∇fi ( xr , si ) ∥∥∥2 − η 4 τ−1∑ s=0 E ∥∥∥ 1 K K∑ k=1 ∇f̄k ( x̄r , sk ) ∥∥∥2 + η 4 τ−1∑ s=0 E ∥∥∥∇f ( x̄r , s ) − 1 N N∑ i=1 ∇fi ( xr , si ) ∥∥∥2 + η 4 τ−1∑ s=0 E ∥∥∥∇f ( x̄r , s ) − 1 K K∑ k=1 ∇f̄k ( x̄r , sk ) ∥∥∥2 + η 4 τ−1∑ s=0 E ∥∥∥ 1 N N∑ i=1 ∇fi ( xr , si ) − 1 K K∑ k=1 ∇f̄k ( x̄r , sk ) ∥∥∥2 + τ−2∑ s=0 E ( L 2 ‖x̄r , s+1 − x̄r , s‖2 ) + E ( L 2 ‖xr+1 − x̄r , τ−1‖2 ) . ( 23 ) Proof . The proof is a standard application of the descent lemma and the sampling rule applied at iteration τ to obtain xr+1 . See Appendix D.1 . Lemma 1 shows the objective value f ( xr ) is descending in expectation up to the following error terms : T1 = E ∥∥∥∇f ( x̄r , s ) − 1 N N∑ i=1 ∇fi ( xr , si ) ∥∥∥2 , T2 = E∥∥∥∇f ( x̄r , s ) − 1 K K∑ k=1 ∇f̄k ( x̄r , sk ) ∥∥∥2 T3 = E ∥∥∥ 1 K K∑ k=1 ∇f̄k ( x̄r , sk ) − 1 N N∑ i=1 ∇fi ( xr , si ) ∥∥∥2 T4 = E‖x̄r , s+1 − x̄r , s‖2 , T5 = E‖xr+1 − x̄r , τ−1‖2 . ( 24 ) In the sequel , we will show these quantities can be bounded by the optimality gap measured in terms of the gradient norms ‖∇f ( x̄r , s ) ‖2 , ‖ ( 1/K ) ∑K k=1∇f̄k ( x̄ r , s k ) ‖2 , and ‖ ( 1/N ) ∑N i=1∇fi ( x r , s i ) ‖2 . B.2 BOUNDING T1 , T2 AND T3 . Define ρmax = max k=1 , ... , K ρk . ( 25 ) Therefore it holds 0 ≤ ρmax ≤ 1 by Assumption 3 . Since each fi is L-smooth by Assumption 1 , we have f̄k and f are also L-smooth . Using this fact and the convexity of ‖ · ‖2 we can bound T1 , T2 and T3 as T1 = E ∥∥∥∇f ( x̄r , s ) ± 1 K K∑ k=1 ∇f̄k ( x̄r , sk ) − 1 N N∑ i=1 ∇fi ( xr , si ) ∥∥∥2 ≤ 2 1 K K∑ k=1 L2E‖x̄r , s − x̄r , sk ‖ 2 + 2 K∑ k=1 1 N ∑ i∈Vk L2E‖x̄r , sk − x r , s i ‖ 2 , T2 = E ∥∥∥∇f ( x̄r , s ) − 1 K K∑ k=1 ∇f̄k ( x̄r , sk ) ∥∥∥2 ≤ 1 K K∑ k=1 L2E ∥∥∥x̄r , s − x̄r , sk ∥∥∥2 , T3 = E ∥∥∥ 1 K K∑ k=1 ∇f̄k ( x̄r , sk ) − 1 N N∑ i=1 ∇fi ( xr , si ) ∥∥∥2 ≤ K∑ k=1 1 N ∑ i∈Vk L2E‖x̄r , sk − x r , s i ‖ 2 . ( 26 ) Clearly , in order to bound T1,2,3 we first need to bound the inter-cluster consensus error ‖x̄r , s−x̄r , sk ‖ and the intra-cluster consensus error ‖x̄r , sk − x r , s i ‖ . Lemma 2 ( Inter-Cluster Consensus Error Bound ) . Let { xr , si } be the sequence generated by Algorithm 1 under Assumptions 1 , 2 , 3 , and 5 . If the learning rate η > 0 satisfies η2 ≤ 1 24τ ( 4τ − 1 ) L2 , ( 27 ) then for all s = 0 , . . . , τ − 1 it holds 1 K K∑ k=1 E‖x̄r , s+1 − x̄r , s+1k ‖ 2 ≤ Cτ 1 K K∑ k=1 E‖x̄r , s − x̄r , sk ‖ 2 + 12τη2L2 ( 1 N K∑ K=1 E ∥∥∥Xr , sk , ⊥∥∥∥2 ) + 12τη2 ( α2E‖∇f ( x̄r , s ) ‖2 + 2g ) + η2 K − 1 N σ2 ( 28 ) where Cτ , 1 + 3 2 · 1 4τ − 1 . ( 29 ) Proof . See Appendix D.2 . Lemma 3 ( Intra-Cluster Consensus Error Bound ) . Let { xr , si } be the sequence generated by Algorithm 1 under Assumptions 1-5 . If η > 0 , then for all s = 0 , . . . , τ − 1 it holds 1 N K∑ k=1 E‖Xr , s+1k , ⊥ ‖ 2 ≤ ( max k∈ [ K ] ρ2k ( 1 + ζ −1 k ) + η 2ρL · 4L2 ) 1 N K∑ k=1 E‖Xr , sk , ⊥‖ 2 + 4η2ρLL 2 1 K K∑ k=1 E‖x̄r , sk − x̄ r , s‖2 + 4η2ρL ( α2E‖∇f ( x̄r , s ) ‖2 + 2g ) + 4η2ρL̄2L + η2σ2ρ2max , ( 30 ) where ρmax is defined in ( 25 ) and ρL , max k=1 , ... , K { ρ2k ( 1 + ζk ) } , ̄2L , 1 K K∑ k=1 2k ( 31 ) with ζk > 0 being a free parameter to be chosen properly for all k = 1 , . . . , K. Proof . See Appendix D.3 . Combining Lemma 2 and 3 we can obtain the following bound on the sum of intra- and interconsensus errors using gradient ‖∇f ( x̄r , s ) ‖2 . Proposition 1 . Let { xr , si } be the sequence generated by Algorithm 1 under Assumptions 1-5 . If the learning rate η > 0 satisfies η2 ≤ 1 24τ ( 4τ − 1 ) L2 , ( 32 ) then for all s = 0 , . . . , τ − 1 it holds 1 K K∑ k=1 E‖x̄r , s+1 − x̄r , s+1k ‖ 2 + K∑ k=1 1 N ‖Xr , s+1k , ⊥ ‖ 2 ≤ s∑ ` =0 C1η 2 ( τ + ρ2maxDτ , ρ ) ( α2E‖∇f ( x̄r , ` ) ‖2 + 2g ) + C1η2τρ2maxDτ , ρ̄2L + C1τη 2ρ2maxσ 2 + C1 ( τ +D 2 τ , ρτ −1ρ2max ) η 2 1 n σ2 ( 33 ) where Dτ , ρ , min { τ , 1 1− ρmax } ( 34 ) and C1 > 0 is some universal constant . Proof . See Appendix C.1 . Notice that according to ( 26 ) the gradient difference terms in Lemma 1 can be bounded as η 4 E ∥∥∥∇f ( x̄r , s ) − 1 N N∑ i=1 ∇fi ( xr , si ) ∥∥∥2 + η 4 E ∥∥∥∇f ( x̄r , s ) − 1 K K∑ k=1 ∇f̄k ( x̄r , sk ) ∥∥∥2 + η 4 E ∥∥∥ 1 N N∑ i=1 ∇fi ( xr , si ) − 1 K K∑ k=1 ∇f̄k ( x̄r , sk ) ∥∥∥2 ≤η 4 ( 2 K K∑ k=1 L2E‖x̄r , s − x̄r , sk ‖ 2 + K∑ k=1 2 N ∑ i∈Vk L2E‖x̄r , sk − x r , s i ‖ 2 ) + η 4 ( 1 K K∑ k=1 L2E ∥∥∥x̄r , s − x̄r , sk ∥∥∥2 + K∑ k=1 1 N ∑ i∈Vk L2E‖x̄r , sk − x r , s i ‖ 2 ) ≤ηL2 ( 1 K K∑ k=1 E‖x̄r , s − x̄r , sk ‖ 2 + K∑ k=1 1 N ‖Xr , sk , ⊥‖ 2 ) ( 35 ) for all s = 1 , . . . , τ . Therefore Proposition 1 immediately leads to the following result . Corollary 2 . Under the same setting as Proposition 1 , it holds τ∑ s=0 η 4 E ∥∥∥∇f ( x̄r , s ) − 1 N N∑ i=1 ∇fi ( xr , si ) ∥∥∥2 + τ∑ s=0 η 4 E ∥∥∥∇f ( x̄r , s ) − 1 K K∑ k=1 ∇f̄k ( x̄r , sk ) ∥∥∥2 + τ∑ s=0 η 4 τ−1∑ s=0 E ∥∥∥ 1 N N∑ i=1 ∇fi ( xr , si ) − 1 K K∑ k=1 ∇f̄k ( x̄r , sk ) ∥∥∥2 ≤ τ−1∑ s=0 C1L 2η3 ( τ2 + τρ2maxDτ , ρ ) ( α2E‖∇f ( x̄r , s ) ‖2 + 2g ) + C1L2η3τ2ρ2maxDτ , ρ̄2L + C1τ 2L2η3ρ2maxσ 2 + C1L 2 ( τ2 +D2τ , ρρ 2 max ) η 3σ 2 n . ( 36 ) We conclude this section by providing a separate bound on the consensus error 1 N ∑K k=1 Er‖X r , τ−1 k , ⊥ ‖2 that will be useful in bounding T5 . Proposition 2 . Under the same setting as Proposition 1 , if ρmax ≤ 1− 1τ , then we have 1 N E K∑ k=1 ‖Xr , τ−1k , ⊥ ‖ 2 ≤ 2C2 τ−2∑ s=0 D2τ , ρρmaxη 2 ( α2‖∇f ( x̄r , s ) ‖2 + 2g ) + C2D 2 τ , ρτρ 2 maxη 2̄2L + C2 ( 1 n + ρmax ) ρmaxτDτ , ρη 2σ2 . ( 37 ) for some universal constant C2 > 0 . Proof . See Appendix C.2 . Proposition 2 shows that the average intra-cluster consensus error 1N ∑K k=1 ‖X r , τ−1 k , ⊥ ‖2 decreases as the network connectivity improves , and vanishes if ρmax goes to zero . B.3 BOUNDING T4 Proposition 3 . Under the same setting as Lemma 1 , we have E‖x̄r , s+1 − x̄r , s‖2 = η2E ∥∥∥∥∥ 1N N∑ i=1 ∇fi ( xr , si ) ∥∥∥∥∥ 2 + η2 σ2 N ( 38 ) for s = 0 , . . . , τ − 1 and r ∈ N+ . Proof . Recall the algorithmic update at iteration s for all s = 0 , . . . , τ − 1 : Xr , s+1k = WkX r , s k − ηWkG r , s k x̄r , s+1k = x̄ r , s k − ηḡ r , s k . ( 39 ) Therefore , it holds under Assumption 2 that E‖x̄r , s+1 − x̄r , s‖2 =E ∥∥∥∥∥ ηN N∑ i=1 ( gr , si ±∇fi ( x r , s i ) ∥∥∥∥∥ 2 = E ∥∥∥∥∥ ηN N∑ i=1 ∇fi ( xr , si ) ∥∥∥∥∥ 2 + η2 σ2 N . ( 40 ) B.4 BOUNDING T5 We provide the bound on T5 separately for the full device participation and partial participation cases . Full participation . When the sampling probability p = 1 , we have xr+1 = 1 N N∑ i=1 xr , τi = x̄ r , τ . In this case , it follows from Proposition 3 that E‖xr+1 − x̄r , τ−1‖2 = η2E ∥∥∥∥∥ 1N N∑ i=1 ∇fi ( xr , τ−1i ) ∥∥∥∥∥ 2 + η2 σ2 N . ( 41 ) Partial participation . We proceed to bound T5 for 1 ≤ m ≤ n− 1 . ( 42 ) Define p = m/n . Recall the algorithmic update at iteration τ − 1 : Xr , τk = WkX r , τ−1 k − ηWkG r , τ−1 k ( 43 ) and xr+1 = 1 K K∑ k=1 1 m ∑ i∈Srk xr , τi = 1 Np K∑ k=1 ∑ i∈Srk xr , τi . ( 44 ) Therefore , with ( Wk ) i , j being the ij-th element of matrix Wk we have under Assumption 2 : E‖xr+1 − x̄r , τ−1‖2 =E ∥∥∥∥∥∥ 1Np K∑ k=1 ∑ i∈Srk xr , τi − x̄ r , τ−1 ∥∥∥∥∥∥ 2 =E ∥∥∥∥∥∥ 1Np K∑ k=1 ∑ i∈Srk ( ∑ j∈Vk ( Wk ) i , j ( x r , τ−1 j − ηg r , τ−1 j ) ) − x̄r , τ−1 ∥∥∥∥∥∥ 2 =E ∥∥∥∥∥∥ 1Np K∑ k=1 ∑ i∈Vk I ( i ∈ Srk ) ( ∑ j∈Vk ( Wk ) i , j ( x r , τ−1 j − η∇fj ( x r , τ−1 j ) ) − x̄r , τ−1 ∥∥∥∥∥∥ 2 ︸ ︷︷ ︸ A2,1 + η2 E ∥∥∥∥∥∥ 1Np K∑ k=1 ∑ i∈Vk I ( i ∈ Srk ) ( ∑ j∈Vk ( Wk ) i , j ( ∇fj ( xr , τ−1j ) − g r , τ−1 j ) ) ∥∥∥∥∥∥ 2 ︸ ︷︷ ︸ A2,2 . ( 45 ) Proposition 4 . Let { xr , si } be the sequence generated by Algorithm 1 under Assumptions 1-6 . If the learning rate η > 0 satisfies η2 ≤ 1 24τ ( 4τ − 1 ) L2 , ( 46 ) then we have the following bounds on A2,1 : A2,1 ≤2η2E ∥∥∥∥∥ 1K K∑ k=1 ∇f̄k ( x̄r , τ−1k ) ∥∥∥∥∥ 2 + 8 ( Gp + 1 τ2 ) ( 1 N K∑ k=1 E‖Xr , τ−1k , ⊥ ‖ 2 ) ; ( 47 ) where Gp , n−m m ( n− 1 ) . ( 48 ) Proof . See Appendix C.3 . Proposition 5 . Under the same setting as Proposition 4 , A2,2 can be bounded as A2,2 ≤ σ2 N ( 2 + n m · ρ2max ) . ( 49 ) Proof . See Appendix C.4 B.5 PROOF OF THEOREM 1 ( FULL PARTICIPATION ) We first prove the descent of the objective value under suitable choice of η . Proposition 6 . If the learning rate satisfies η ≤ 1 4C1α · 1 τL , ( 50 ) then we have Ef ( xr+1 ) ≤ Ef ( xr ) − η 8 τ−1∑ s=0 E‖∇f ( x̄r , s ) ‖2 +Rfull ( η ) , ( 51 ) where Rfull ( η ) =C1L2η3τ2 ( τ + ρ2maxDτ , ρ ) 2g + 2C1L 2η3 ( τ2ρ2maxDτ , ρ̄ 2 L + τ 2σ2 ( 1 n + ρ2max ) ) + η2Lτ σ2 N . ( 52 ) C1 > 0 is some universal constant . Proof . See Appendix C.5 . To attain the expression of the convergence rate , we sum ( 51 ) over r = 0 , . . . , R : min r∈ [ R ] min s=0 , ... , τ−1 E‖∇f ( x̄r , s ) ‖2 ≤8 ( f ( x 0 ) − f ( x ? ) ) ητ ( R+ 1 ) + 8Rfull ( η ) ητ = 8 ( f ( x0 ) − f ( x ? ) ) ητ ( R+ 1 ) + 16ηL σ2 N︸ ︷︷ ︸ centralized SGD + 16C1L 2τ2η2 2g + 16C1L 2η2 ( τρ2maxDτ , ρ̄ 2 L + τσ 2 ( 1 n + ρ2max ) ) ︸ ︷︷ ︸ network effect . ( 53 ) The first two terms of ( 53 ) corresponds to the impact of stochastic noise and is of the same order as the centralized SGD algorithm . The last term is of order η2 and corresponds to the deterioration of convergence rate due to the fact that we are not computing the average gradients of all devices at each iteration . Denote r0 = 8 ( f ( x 0 ) − f ( x ? ) ) , r1 = 16L ( σ2 N ) , r2 = 16C1L 2τ2 2g + 16C1L 2 ( τρ2maxDτ , ρ̄ 2 L + τσ 2 ( 1 n + ρ2max ) ) . ( 54 ) The rest of the proof follows the same argument as ( Chen et al. , 2021b , Appendix B.5 ) and thus we omit the details . B.6 PROOF OF THEOREM 2 AND COROLLARY 1 ( PARTIAL PARTICIPATION ) Proposition 7 . Let { xr , si } be the sequence generated by Algorithm 1 under Assumption 1-5 . If the learning rate η and the network connectivity satisfies η ≤ 1 C3α2 · 1 τL and ρmax ≤ 1− 1 τ , ( 55 ) then Ef ( xr+1 ) ≤ Ef ( xr ) − η 8 τ−1∑ s=0 E‖∇f ( x̄r , s ) ‖2 +R ( 1 ) part ( η ) +R ( 2 ) part ( η ) ( 56 ) with R ( 1 ) part ( η ) =C1L2η3τ2 ( τ + ρ2maxDτ , ρ ) 2g + 2C1L 2η3 ( τ2ρ2maxDτ , ρ̄ 2 L + τ 2σ2 ( 1 n + ρ2max ) ) + Lτη2 σ2 N , ( 57 ) R ( 2 ) part ( η ) = ( 8C2G ′ pLD 2 τ , ρτρmax ) η2 2g + 4C2LG ′ pη 2 ( D2τ , ρτρ 2 max̄ 2 L + ( 1 n + ρmax ) ρmaxτDτ , ρσ 2 ) + L 2 ( n m ρ2max ) η2 σ2 N . C1 , C3 > 0 are some universal constants , and G′p = Gp + 1 τ2 , with Gp = n−m m ( n− 1 ) . ( 58 ) Proof . See Appendix C.6 Comparing ( 56 ) to ( 51 ) we can see thatR ( 1 ) part ( η ) is of the same order asR ( 1 ) full ( η ) , whileR ( 2 ) part ( η ) is an extra loss term introduced by sampling . Following the same steps as the proof of Theorem 1 gives min r∈ [ R ] min s=0 , ... , τ−1 E‖∇f ( x̄r , s ) ‖2 =O σ + E ( g , ̄L , σ , ρmax ) √ NτR + ( τ2 2g + τρ 2 maxDτ , ρ̄ 2 L + τ ( 1 n + ρ 2 max ) σ2 ) 1 3 ( τR ) 2 3 + 1 R , ( 59 ) where E2 ( g , ̄L , σ , ρmax ) = ( 2gDτ , ρ + ρmaxDτ , ρ̄2L + σ2 ) ·G′pDτ , ρρmaxN + n m · 1 τ ρ2maxσ 2 . ( 60 ) Our last step simplifies the overall conditions on ρmax so that E2 ( g , ̄L , σ , ρmax ) = O ( 1 ) : ρmax ≤ 1− 1 τ , G′pD 2 τ , ρρmax ≤ 1 N , ρmax ≤ m n · τ . ( 61 ) We claim to fulfill ( 61 ) it suffices to require ρmax ≤ 1 4N min { m , τ − 1 } . ( 62 ) When τ = 1 , the condition trivially requires ρmax = 0 . We then consider the case for τ ≥ 2 . By definition , it can be verified that G′p ≤ 1 m + 1 τ2 . ( 63 ) First notice that m 4N ≤ 1 4 ≤ 1− 1 τ and τ 4N ≤ m n · τ . ( 64 ) Therefore , it remains to prove ( 62 ) implies G′pD 2 τ , ρρmax ≤ 1 N . ( 65 ) Using the fact that under ( 62 ) ρmax ≤ 1/4 we have G′pD 2 τ , ρρmax = G ′ pρmax · 1 ( 1− ρmax ) 2 ≤16 9 ρmax ( 1 m + 1 τ2 ) ≤ 16 9 ( 1 m + 1 τ2 ) 1 4N min { m , τ } ≤ 1 N . ( 66 ) This proves the claim . C PROOF OF MAIN PROPOSITIONS C.1 PROOF OF PROPOSITION 1 Denote for short Mr , s , 1 N K∑ k=1 E‖Xr , sk , ⊥‖ 2 1 K K∑ k=1 E‖x̄r , sk − x̄ r , s‖2 . ( 67 ) Invoking Lemma 2 and Lemma 3 we obtain that under the condition that the learning rate η > 0 satisfies η2 ≤ 1 24τ ( 4τ − 1 ) L2 , ( 68 ) the following inequality is satisfied for all s = 0 , . . . , τ − 1 : Mr , s+1 ≤ G ·Mr , s +Br , s , ( 69 ) where G = ( maxk∈ [ K ] ρ 2 k ( 1 + ζ −1 k ) + η 2ρL · 4L2 η2ρL · 4L2 12τη2L2 Cτ ) ( 70 ) Br , s = ( 4ρLη 2 ( α2E‖∇f ( x̄r , s ) ‖2 + 2g ) + 4η2ρL̄2L + η2ρ2maxσ2 12τη2 ( α2E‖∇f ( x̄r , s ) ‖2 + 2g ) + η2 σ 2 n . ) ( 71 ) The inequality in ( 69 ) is defined elementwise . Unrolling ( 69 ) yields Mr , s+1 ≤ s∑ ` =0 G ` Br , s− ` , ( 72 ) where we have used the fact that Mr,0 = 0 due to full synchronization of the xi ’ s at the beginning of each round r. We first provide a bound on the sum of the two elements of G ` Br , s− ` . For simplicity we omit the round index r in the superscript for the rest of this section . Lemma 4 . Let bs− ` 1 and b s− ` 2 be the first and second element of B s− ` , respectively . Suppose the learning rate η > 0 then ( 1 , 1 ) G ` Bs− ` ≤ λ ` 2 ( bs− ` 1 + b s− ` 2 ) + λ ` 2 − λ ` 1 λ2 − λ1 η2 · ( 12τL2bs− ` 1 + 4ρLL 2bs− ` 2 ) ( 73 ) where λ1 ≤ λ2 are the eigenvalues of G ; and ρL is defined in ( 31 ) . Proof . See Appendix D.4 . From Lemma 4 we immediately get s∑ ` =0 ( 1 , 1 ) ·G ` Bs− ` ≤ s∑ ` =0 ( λ ` 2 ( b s− ` 1 + b s− ` 2 ) + λ ` 2 − λ ` 1 λ2 − λ1 η2 · ( 12τL2bs− ` 1 + 4ρLL 2bs− ` 2 ) ) . ( 74 ) Since λ2 ≥ Cτ > 1 , we have λ ` 2 − λ ` 1 λ2 − λ1 = λ ` −12 ` −1∑ s=0 ( λ1 λ2 ) s ≤ λ ` −12 min { λ2 λ2 − λ1 , ` } ≤ λ ` 2 min { 1 λ2 − λ1 , ` } ( 75 ) and thus s∑ ` =0 ( 1 , 1 ) ·G ` Bs− ` ≤ s∑ ` =0 λ ` 2 ( b s− ` 1 + b s− ` 2 ) + s∑ ` =0 ( λ ` 2 min { 1 λ2 − λ1 , ` } ) η2 · ( 12τL2bs− ` 1 + 4ρLL 2bs− ` 2 ) . ( 76 ) Recall the definition of ρL given by ( 31 ) : ρL = max k=1 , ... , K ρ2k ( 1 + ζk ) . ( 77 ) By the Gershgorin ’ s theorem , since η > 0 , we can upperbound λ2 as λ2 ≤ max { max k∈ [ K ] ρ2k ( 1 + ζ −1 k ) + η 2ρL · 8L2 , Cτ + 12τη2L2 } ≤ max { max k∈ [ K ] ρ2k ( 1 + ζ −1 k ) + ρL ( 4τ − 1 ) 3τ , 1 + 2 4τ − 1 } , ( 78 ) where the last inequality is due to the bound on η : η2 ≤ 1 24τ ( 4τ − 1 ) L2 . ( 79 ) Define constant Dτ , ρ = min { τ , 1 1− ρmax } . ( 80 ) We consider two cases . • Case 1 : ρmax ≤ 1− 1 τ ⇒ 1 1− ρmax ≤ τ . ( 81 ) Thus Dτ , ρ = 1/ ( 1− ρmax ) . We let ζk = ρk/ ( 1− ρk ) and it gives max k∈ [ K ] ρ2k ( 1 + ζ −1 k ) = ρmax , ρL = max k=1 , ... , K { ρ2k 1− ρk } = ρ2max 1− ρmax = ρ2maxDτ , ρ . ( 82 ) Substituting into the bound of λ2 [ cf . ( 78 ) ] gives λ2 ≤ max { ρmax + ρ2max ( 1− ρmax ) 3τ ( 4τ − 1 ) , 1 + 2 4τ − 1 } ≤ max 1− 1τ + ( 1− 1τ ) 2 3 ( 4τ − 1 ) , 1 + 2 4τ − 1 < 1 + 34τ − 1 , ( 83 ) where in the second inequality we used the condition ( 81 ) . Since s ≤ τ and λ2 ≥ 1 , we obtain the following bound s∑ ` =0 λ ` 2b s− ` 1 ≤ ( ( 1 + 3 4τ − 1 ) τ ) · ( s∑ ` =0 b ` 1 ) ≤ 3 · ( s∑ ` =0 b ` 1 ) . ( 84 ) Moreover , since ρmax + η 2ρL · 4L2 ≤ ρmax + ρ2max ( 1− ρmax ) ( 4τ − 1 ) 6τ ( 81 ) ≤ 1− 1 τ + ( 1− 1τ ) 2 6 ( 4τ − 1 ) ≤ Cτ , ( 85 ) we can bound λ2 − λ1 as λ2 − λ1 ≥ Cτ − ρmax − η2ρL · 4L2 ≥ Cτ − ( ρmax + ρ2max ( 1− ρmax ) ( 4τ − 1 ) 6τ ) ( 81 ) ≥ Cτ − ( ρmax + ρmax · 1− 1τ 6 ( 4τ − 1 ) ) ≥ 1 + 1 4τ − 1 − ( ρmax + ρmax · 1 4τ − 1 ) = ( 1− ρmax ) ( 1 + 1 4τ − 1 ) ≥ 1− ρmax . ( 86 ) Collecting ( 84 ) and ( 86 ) we can bound ( 76 ) as s∑ ` =0 ( 1 , 1 ) ·G ` Bs− ` ≤ s∑ ` =0 ( b ` 1 + b ` 2 ) · 3 + s∑ ` =0 η2 · ( 12τL2b ` 1 + 4ρLL 2b ` 2 ) · 3 ( min { 1 λ2 − λ1 , τ } ) ≤ s∑ ` =0 ( b ` 1 + b ` 2 ) · 3 + s∑ ` =0 η2 · ( 12τL2b ` 1 +Dτ , ρρ 2 max4L 2b ` 2 ) · 3Dτ , ρ ( 79 ) ≤ s∑ ` =0 ( b ` 1 + b ` 2 ) · 3 + s∑ ` =0 ( 12τb ` 1 +Dτ , ρρ 2 max4b ` 2 ) · 3Dτ , ρ 1 ( 4τ − 1 ) 24τ ≤ s∑ ` =0 ( b ` 1 + b ` 2 ) · 3 + s∑ ` =0 ( 12τb ` 1 +Dτ , ρρ 2 max4b ` 2 ) · 1 8 Dτ , ρ τ2 = s∑ ` =0 ( b ` 1 + b ` 2 ) · 3 + s∑ ` =0 ( 12τb ` 1 +Dτ , ρρ 2 max4b ` 2 ) · 1 8 Dτ , ρ τ2 . ( 87 ) Substituting the expression of b ` 1 and b ` 2 gives s∑ ` =0 ( 1 , 1 ) ·G ` Bs− ` ≤ s∑ ` =0 ( b ` 1 + b ` 2 ) · 5 + s∑ ` =0 D2τ , ρ τ2 · ρ2maxb ` 2 = s∑ ` =0 5 ( 4η2 ( ρL + 3τ ) ( α 2E‖∇f ( x̄r , ` ) ‖2 + 2g ) + 4η2ρL̄2L + η2ρ2maxσ2 + η2 1 n σ2 ) + s∑ ` =0 D2τ , ρ τ2 · ρ2max ( 12τη2 ( α2E‖∇f ( x̄r , ` ) ‖2 + 2g ) + η2 1 n σ2 ) ≤ s∑ ` =0 C1 2 η2 ( ρL + τ + ρ 2 maxD 2 τ , ρτ −1 ) ( α2E‖∇f ( x̄r , ` ) ‖2 + 2g ) + C1 2 η2τρL̄ 2 L + C1 2 τη2ρ2maxσ 2 + C1 2 ( τ +D2τ , ρτ −1ρ2max ) η 2 1 n σ2 ≤ s∑ ` =0 C1η 2 ( τ + ρ2maxDτ , ρ ) ( α2E‖∇f ( x̄r , ` ) ‖2 + 2g ) + C1η2τρ2maxDτ , ρ̄2L + C1τη 2ρ2maxσ 2 + C1 ( τ +D 2 τ , ρτ −1ρ2max ) η 2 1 n σ2 ( 88 ) where C1 is some universal constant . The last inequality holds since ρL = ρ2maxDτ , ρ and Dτ , ρ ≤ τ . • Case 2 : ρmax > 1− 1 τ ⇒ Dτ , ρ = τ . ( 89 ) In such a case , we let ζk = ( 4τ − 1 ) and thus max k∈ [ K ] ρ2k ( 1 + ζ −1 k ) = ρ 2 max ( 1 + ( 4τ − 1 ) −1 ) , ρL = 4τρ2max = 4ρ2maxDτ , ρ . ( 90 ) Substituting into the bound of λ2 given in ( 78 ) , applying again the learning rate condition ( 79 ) and using the fact that Dτ , ρ = τ : λ2 ≤ max { ρ2max ( 1 + ( 4τ − 1 ) −1 ) + 4ρ2max 3 ( 4τ − 1 ) , 1 + 2 ( 4τ − 1 ) } ≤ 1 + 3 4τ − 1 . ( 91 ) Therefore by ( 76 ) , ( 79 ) , ( 84 ) , and the fact that min { 1 λ2 − λ1 , ` } ≤ τ = Dτ , ρ ( 92 ) we obtain s∑ ` =0 ( 1 , 1 ) ·G ` Bs− ` ≤ s∑ s=0 ( b ` 1 + b ` 2 ) · 3 + s∑ ` =0 η2 · ( 12τL2b ` 1 + 16ρ 2 maxDτ , ρL 2b ` 2 ) · 3Dτ , ρ ≤ s∑ ` =0 ( b ` 1 + b ` 2 ) · 3 + s∑ ` =0 ( 12τb ` 1 + 16ρ 2 maxDτ , ρb ` 2 ) 1 8 Dτ , ρ τ2 ≤ s∑ ` =0 ( b ` 1 + b ` 2 ) · 5 + s∑ ` =0 2 D2τ , ρ τ2 ρ2maxb ` 2 . ( 93 ) Substituting the expression of b1 and b2 and using the fact that ρL = 4τρ 2 max = 4ρ 2 maxDτ , ρ we arrive at the same bound as in Case 1 , possibly with a different constant C1 . C.2 PROOF OF PROPOSITION 2 We are in Case 1 described in the proof of Proposition 1 . By letting ζk = ρk/ ( 1− ρk ) we have G = ( ρmax + ρ2max 1−ρmax · η 2 · 4L2 ρ 2 max 1−ρmax · η 2 · 4L2 12τη2L2 Cτ ) ( 94 ) and the following bound on the difference of the eigenvalues of G : λ2 − λ1 ≥ 1− ρmax . ( 95 ) Notice that according to ( 72 ) and ( 149 ) 1 N K∑ k=1 ‖Xr , τ−1k , ⊥ ‖ 2 = 1 det ( T ) τ−2∑ ` =0 tτ−2− ` 1 . Therefore 1 N K∑ k=1 ‖Xr , τ−1k , ⊥ ‖ 2 = 1 12τη2L2 ( λ2 − λ1 ) τ−2∑ ` =0 ( ( Cτ − λ1 ) ( λ ` 112τη 2L2bτ−2− ` 1 − λ ` 1 ( λ2 − Cτ ) b τ−2− ` 2 ) ) + 1 12τη2L2 ( λ2 − λ1 ) τ−2∑ ` =0 ( ( λ2 − Cτ ) ( λ ` 212τη 2L2bτ−2− ` 1 + λ ` 2 ( Cτ − λ1 ) bτ−2− ` 2 ) ) ≤ 1 12η2L2τ ( 1− ρmax ) τ−2∑ ` =0 ( ( Cτ − λ1 ) λ ` 1 ( 12τη2L2bτ−2− ` 1 − ( λ2 − Cτ ) b τ−2− ` 2 ) ) + 1 12η2L2τ ( 1− ρmax ) τ−2∑ ` =0 ( ( λ2 − Cτ ) λ ` 2 ( 12τη2L2bτ−2− ` 1 + ( Cτ − λ1 ) b τ−2− ` 2 ) ) . ( 96 ) In the following , we bound λ1 and λ2 − Cτ as a function of ρmax . For notation simplicity we omit the subscript of ρmax in the rest of the proof . Further , we introduce the following shorthand notation for the elements of G : f ( ρ ) = ρ+ ρ2 1− ρ · η2 · 4L2 , g ( ρ ) = ρ 2 1− ρ · η2 · 4L2 , and h ( τ ) = 12τη2L2 . ( 97 ) The eigenvalues of G then can be written in closed form as λ1,2 = f ( ρ ) + Cτ ± √ ( Cτ − f ( ρ ) ) 2 + 4g ( ρ ) h ( τ ) 2 . ( 98 ) Since we have proved in Proposition 1 that Cτ ≥ f ( ρ ) [ cf . ( 85 ) ] , this gives us the following upperbounds on λ2 as : Cτ ≤ λ2 = f ( ρ ) + Cτ + √ ( Cτ − f ( ρ ) ) 2 + 4g ( ρ ) h ( τ ) 2 ≤ Cτ + √ g ( ρ ) h ( τ ) . ( 99 ) Furthermore , applying the Gershgorin ’ s theorem we obtain λ1 ≥ min { ρ , Cτ − 12τη2L2 } ≥ ρ ≥ 0 . ( 100 ) under the learning rate condition ( 79 ) . Substituting the bounds into ( 96 ) gives 1 N K∑ k=1 ‖Xr , τ−1k , ⊥ ‖ 2 ≤ 1 12η2L2τ ( 1− ρ ) τ−2∑ ` =0 ( Cτλ ` 1 ( 12τη2L2bτ−2− ` 1 − ( λ2 − Cτ ) b τ−2− ` 2 ) ) + 1 12η2L2τ ( 1− ρ ) τ−2∑ ` =0 ( ( λ2 − Cτ ) λ ` 2 ( 12τη2L2bτ−2− ` 1 + Cτ b τ−2− ` 2 ) ) ≤ 1 12η2L2τ ( 1− ρ ) τ−2∑ ` =0 ( Cτf ( ρ ) ` ( 12τη2L2bτ−2− ` 1 ) ) + 1 12η2L2τ ( 1− ρ ) τ−2∑ ` =0 ( √ g ( ρ ) h ( τ ) λ ` 2 ( 12τη2L2bτ−2− ` 1 + Cτ b τ−2− ` 2 ) ) ≤ 1 1− ρ Cτ ( τ−2∑ ` =0 b ` 1 ) + 1 1− ρ √ g ( ρ ) h ( τ ) λτ2 ( τ−2∑ ` =0 b ` 1 ) + Cτ 12η2L2τ ( 1− ρ ) √ g ( ρ ) h ( τ ) λτ2 ( τ−2∑ ` =0 b ` 2 ) ≤Dτ , ρCτ ( τ−2∑ ` =0 b ` 1 ) + 3Dτ , ρη 2L2ρ √ Dτ , ρ · 48τ ( τ−2∑ ` =0 b ` 1 ) + Dτ , ρCτ 12τ ρ √ Dτ , ρ · 48τ ( τ−2∑ ` =0 b ` 2 ) . ( 101 ) where we have used the bound λ1 ≤ f ( ρ ) < 1 , λ2 > 1 and λτ2 < 3 . Plug in the expression of b1 and b2 and using the fact that Cτ < 2 , ρL = ρ2Dτ , ρ gives 1 N K∑ k=1 ‖Xr , τ−1k , ⊥ ‖ 2 ≤ ( 2Dτ , ρ + 3Dτ , ρη 2L2ρ √ Dτ , ρ · 48τ ) ρ2 τ−2∑ s=0 ( 4η2Dτ , ρ ( α 2‖∇f ( x̄r , s ) ‖2 + 2g ) + 4η2Dτ , ρ̄2L + η2σ2 ) + Dτ , ρCτ 12τ ρ √ Dτ , ρ · 48τ τ−2∑ s=0 ( 12τη2 ( α2‖∇f ( x̄r , s ) ‖2 + 2g ) + η2 σ2 n ) ≤ ( 2Dτ , ρ + Dτ , ρ τ ρ ) τ−2∑ s=0 ρ2 ( 4η2Dτ , ρ ( α 2‖∇f ( x̄r , s ) ‖2 + 2g ) + 4η2Dτ , ρ̄2L + η2σ2 ) + 2 Dτ , ρ τ ρ √ Dτ , ρ · τ τ−2∑ s=0 ( 12τη2 ( α2‖∇f ( x̄r , s ) ‖2 + 2g ) + η2 1 n σ2 ) ≤3Dτ , ρρ2 τ−2∑ s=0 ( 4η2Dτ , ρ ( α 2‖∇f ( x̄r , s ) ‖2 + 2g ) + 4η2Dτ , ρ̄2L + η2σ2 ) + 2Dτ , ρρ √ Dτ , ρτ−1 τ−2∑ s=0 ( 12τη2 ( α2‖∇f ( x̄r , s ) ‖2 + 2g ) + η2 1 n σ2 ) , ( 102 ) where the second inequality holds since 3τ2η2L2 ≤ 3τ2L2 · 1 24τ ( 4τ − 1 ) L2 ≤ 3τ2L2 · 1 24τ ( 2τ ) L2 < 1 8 ( 103 ) ⇒3Dτ , ρη2L2ρ √ Dτ , ρ · 48τ ≤ Dτ , ρ 8τ2 √ Dτ , ρ · 48τρ ≤ Dτ , ρ τ ρ ( 104 ) and the last inequality holds since Dτ , ρ ≤ τ . Tidy up the expression gives 1 N K∑ k=1 ‖Xr , τ−1k , ⊥ ‖ 2 ≤ τ−2∑ s=0 3Dτ , ρρ 2 ( 4η2Dτ , ρ ( α 2‖∇f ( x̄r , s ) ‖2 + 2g ) + 4η2Dτ , ρ̄2L + η2σ2 ) + τ−2∑ s=0 2Dτ , ρρ √ Dτ , ρτ−1 ( 12τη 2 ( α2‖∇f ( x̄r , s ) ‖2 + 2g ) + η2 1 n σ2 ) ≤ C2 τ−2∑ s=0 ( D2τ , ρρ 2 + ( Dτ , ρ ) 1.5τ−0.5ρ ) η2 ( α2‖∇f ( x̄r , s ) ‖2 + 2g ) + C2 ( Dτ , ρ ) 2τρ2η2̄2L + C2τDτ , ρρ 2η2σ2 + C2τ · ( Dτ , ρ ) 1.5τ−0.5ρη2 1 n σ2 ≤ 2C2 τ−2∑ s=0 D2τ , ρρη 2 ( α2‖∇f ( x̄r , s ) ‖2 + 2g ) + C2 ( Dτ , ρ ) 2τρ2η2̄2L + C2 ( 1 n + ρ ) ρτDτ , ρη 2σ2 ( 105 ) for some C2 > 0 . The last inequality holds since ( Dτ , ρ ) 1.5τ−0.5 ≤ Dτ , ρ . ( 106 ) C.3 PROOF OF PROPOSITION 4 We prove ( 47 ) by splitting the terms A2,1 follows : A2,1 ( a ) =E ∥∥∥∥∥∥ 1Np K∑ k=1 ∑ i∈Vk I ( i ∈ Srk ) ( ∑ j∈Vk ( Wk ) i , j ( x r , τ−1 j − η∇fj ( x r , τ−1 j ) − x̄ r , τ−1 k ) ∥∥∥∥∥∥ 2 ≤2E ∥∥∥∥∥ 1Np K∑ k=1 p · nη∇f̄k ( x̄r , τ−1k ) ∥∥∥∥∥ 2 + 2E ∥∥∥ 1 Np K∑ k=1 ∑ i∈Vk I ( i ∈ Srk ) rik ∥∥∥2 , ( 107 ) where rik , ∑ j∈Vk ( Wk ) i , j ( x r , τ−1 j − x̄ r , τ−1 k − η ( ∇fj ( x r , τ−1 j ) −∇f̄k ( x̄ r , τ−1 k ) ) . ( 108 ) Equality ( a ) holds since K∑ k=1 ∑ i∈Srk ∑ j∈Vk ( Wk ) i , j x̄ r , τ−1 k = K∑ k=1 ∑ i∈Srk x̄r , τ−1k = K∑ k=1 m · x̄r , τ−1k = K∑ k=1 p ∑ i∈Vk xr , τ−1i = Npx̄ r , τ−1 , ( 109 ) and similarly , ∑ i∈Srk ∑ j∈Vk ( Wk ) i , j∇f̄k ( x̄r , τ−1k ) = np∇f̄k ( x̄ r , τ−1 k ) . ( 110 ) Since samples are taken according to the rule specified by Assumption 6 , the following probabilities hold : P ( i ∈ Srk | i ∈ Vk ) = p , P ( i , j ∈ Srk | , i , j ∈ Vk ) = p · np− 1 n− 1 , ( 111 ) P ( i ∈ Srk , j ∈ Sr ` | i ∈ Vk , j ∈ V ` , k 6= ` ) = p2 . ( 112 ) Consequently , we can evaluate the second term in ( 107 ) and obtain A2,1 =2E ∥∥∥∥∥ 1K K∑ k=1 η∇f̄k ( x̄r , τ−1k ) ∥∥∥∥∥ 2 + 2 ( Np ) 2 E p K∑ k=1 ∑ i∈Vk ‖rik‖2 + p · np− 1 n− 1 K∑ k=1 ∑ i , j∈Vk r > ikrjk + 2 ( Np ) 2 · p2 ∑ k 6= ` ∑ i∈Vk ∑ j∈V ` E ( r > ikrj ` ) =2E ∥∥∥∥∥ 1K K∑ k=1 η∇f̄k ( x̄r , τ−1k ) ∥∥∥∥∥ 2 + 2 ( Np ) 2 E ( p · np− 1 n− 1 ∥∥∥ K∑ k=1 ∑ i∈Vk rik ∥∥∥2 + p ( 1− p ) n n− 1 K∑ k=1 ∑ i∈Vk ‖rik‖2 ) + 2 ( Np ) 2 p ( 1− p ) n− 1 E ∑ k 6= ` ∑ i∈Vk ∑ j∈V ` r > ikrj ` ≤2E ∥∥∥∥∥ 1K K∑ k=1 η∇f̄k ( x̄r , τ−1k ) ∥∥∥∥∥ 2 + 2 ( Np ) 2 E ( p · np− 1 n− 1 ∥∥∥ K∑ k=1 ∑ i∈Vk rik ∥∥∥2 + p ( 1− p ) n n− 1 K∑ k=1 ∑ i∈Vk ‖rik‖2 ) + 2 ( Np ) 2 p ( 1− p ) n− 1 E ∑ k 6= ` ∑ i∈Vk ∑ j∈V ` 1 2 ‖rik‖2 + 1 2 ‖rj ` ‖2 ≤2E ∥∥∥∥∥ 1K K∑ k=1 η∇f̄k ( x̄r , τ−1k ) ∥∥∥∥∥ 2 + 2 ( Np ) 2 E ( p · np− 1 n− 1 ∥∥∥ K∑ k=1 ∑ i∈Vk rik ∥∥∥2 + p ( 1− p ) n n− 1 K∑ k=1 ∑ i∈Vk ‖rik‖2 ) + 2 ( Np ) 2 p ( 1− p ) n− 1 ( K − 1 ) n K∑ k=1 ∑ i∈Vk E‖rik‖2 . ( 113 ) By substituting the expression of rik we can bound terms ‖ ∑K k=1 ∑ i∈Vk rik‖ 2 and∑K k=1 ∑ i∈Vk ‖rik‖ 2 as ∥∥∥ K∑ k=1 ∑ i∈Vk rik ∥∥∥2 = ∥∥∥ K∑ k=1 ∑ i∈Vk ∑ j∈Vk ( Wk ) i , j ( x r , τ−1 j − x̄ r , τ−1 k − η ( ∇fj ( x r , τ−1 j ) −∇f̄k ( x̄ r , τ−1 k ) ) ∥∥∥2 = η2 ∥∥∥∥∥∥ K∑ k=1 ∑ i∈Vk ∑ j∈Vk ( Wk ) i , j ( ∇fj ( xr , τ−1j ) −∇f̄k ( x̄ r , τ−1 k ) ) ∥∥∥∥∥∥ 2 = η2 ∥∥∥∥∥∥ K∑ k=1 ∑ i∈Vk ∑ j∈Vk ( Wk ) i , j ( ∇fj ( xr , τ−1j ) −∇fj ( x̄ r , τ−1 k ) ) ∥∥∥∥∥∥ 2 ≤ η2N K∑ k=1 ∑ i∈Vk ∑ j∈Vk ( Wk ) i , jL 2‖xr , τ−1j − x̄ r , τ−1 k ‖ 2 ≤ η2N K∑ k=1 ∑ i∈Vk L2‖xr , τ−1i − x̄ r , τ−1 k ‖ 2 = η2 ·NL2 K∑ k=1 ‖Xr , τ−1k , ⊥ ‖ 2 ( 114 ) and K∑ k=1 ∑ i∈Vk ‖rik‖2 = K∑ k=1 ∑ i∈Vk ∥∥∥∥∥∥ ∑ j∈Vk ( Wk ) i , j ( x r , τ−1 j − x̄ r , τ−1 k − η ( ∇fj ( x r , τ−1 j ) −∇f̄k ( x̄ r , τ−1 k ) ) ∥∥∥∥∥∥ 2 ≤ K∑ k=1 ∑ i∈Vk ∑ j∈Vk ( Wk ) i , j ∥∥∥xr , τ−1j − x̄r , τ−1k − η ( ∇fj ( xr , τ−1j ) −∇fj ( x̄r , τ−1k ) ) ∥∥∥2 ≤ K∑ k=1 ∑ i∈Vk ∑ j∈Vk ( Wk ) i , j ( 2 ∥∥∥xr , τ−1j − x̄r , τ−1k ∥∥∥2 + 2η2 ∥∥∥∇fj ( xr , τ−1j ) −∇fj ( x̄r , τ−1k ) ∥∥∥2 ) ≤ K∑ k=1 ∑ i∈Vk 2 ∥∥∥xr , τ−1i − x̄r , τ−1k ∥∥∥2 + 2η2L2 K∑ k=1 ∑ i∈Vk ‖xr , τ−1i − x̄ r , τ−1 k ‖ 2 = K∑ k=1 2 ( 1 + η2L2 ) ‖Xr , τ−1k , ⊥ ‖ 2 . ( 115 ) Tidy up the expression leads to the following bound of A2,1 : A2,1 ≤2E ∥∥∥∥∥ 1K K∑ k=1 η∇f̄k ( x̄r , τ−1k ) ∥∥∥∥∥ 2 + 2 ( Np ) 2 ( p · np− 1 n− 1 E ∥∥∥ K∑ k=1 ∑ i∈Vk rik ∥∥∥2 ) + 2 ( Np ) 2 p ( 1− p ) N n− 1 K∑ k=1 ∑ i∈Vk E‖rik‖2 ≤2E ∥∥∥∥∥ 1K K∑ k=1 η∇f̄k ( x̄r , τ−1k ) ∥∥∥∥∥ 2 + 2 ( Np ) 2 ( p · np− 1 n− 1 η2 ·NL2 K∑ k=1 E‖Xr , τ−1k , ⊥ ‖ 2 ) + 2 ( Np ) 2 p ( 1− p ) N n− 1 ( K∑ k=1 2 ( 1 + η2L2 ) E‖Xr , τ−1k , ⊥ ‖ 2 ) ≤2η2E ∥∥∥∥∥ 1K K∑ k=1 ∇f̄k ( x̄r , τ−1k ) ∥∥∥∥∥ 2 + 2 N η2L2 K∑ k=1 E‖Xr , τ−1k , ⊥ ‖ 2 + 4 N 1− p p ( n− 1 ) ( K∑ k=1 ( 1 + η2L2 ) E‖Xr , τ−1k , ⊥ ‖ 2 ) ≤2η2E ∥∥∥∥∥ 1K K∑ k=1 ∇f̄k ( x̄r , τ−1k ) ∥∥∥∥∥ 2 + 8 N ( 1− p p ( n− 1 ) + 1 τ2 ) ( K∑ k=1 E‖Xr , τ−1k , ⊥ ‖ 2 ) , ( 116 ) where the last inequality holds under the learning rate condition η2 ≤ 1 24τ ( 4τ − 1 ) L2 . ( 117 ) This completes the proof of ( 47 ) . C.4 PROOF OF PROPOSITION 5 We bound A2,2 in following the same rationale as Proposition 4 . A2,2 =E ∥∥∥ 1 Np K∑ k=1 ∑ i∈Vk I ( i ∈ Srk ) ( ∑ j∈Vk ( Wk ) i , j ( ∇fj ( xr , sj ) − g r , s j ) ) ︸ ︷︷ ︸ eik ∥∥∥2 = 1 ( Np ) 2 ( p · np− 1 n− 1 E ∥∥∥ K∑ k=1 ∑ i∈Vk eik ∥∥∥2 + p ( 1− p ) n n− 1 K∑ k=1 ∑ i∈Vk E‖eik‖2 ) + 2 ( Np ) 2 p ( 1− p ) n− 1 E ∑ k 6= ` ∑ i∈Vk ∑ j∈V ` e > ikej ` = 1 ( Np ) 2 ( p · np− 1 n− 1 E ∥∥∥ K∑ k=1 ∑ i∈Vk eik ∥∥∥2 + p ( 1− p ) n n− 1 K∑ k=1 ∑ i∈Vk E‖eik‖2 ) , ( 118 ) where the last equality is due to fact that the inter-cluster stochastic noise is zero mean and independent . Recall the definition ξr , si , g r , s i −∇fi ( x r , s i ) . Using again the independence of the ξi ’ s we get E ∥∥∥ K∑ k=1 ∑ i∈Vk eik ∥∥∥2 = E∥∥∥ K∑ k=1 ∑ i∈Vk ∑ j∈Vk ( Wk ) i , jξ r , s j ∥∥∥2 = E∥∥∥ N∑ i=1 ξr , si ∥∥∥2 = Nσ2 , ( 119 ) and K∑ k=1 ∑ i∈Vk E‖eik‖2 = K∑ k=1 ∑ i∈Vk E ∥∥∥ ∑ j∈Vk ( Wk ) i , jξ r , s j ∥∥∥2 = K∑ k=1 ‖Wk‖2σ2 = K∑ k=1 σ2 n∑ j=1 d2j ≤ K∑ k=1 σ2 ( 1 + ( n− 1 ) ρ2k ) ≤ Kσ2 + ( n− 1 ) Kρ2maxσ2 = ( 1 + ( n− 1 ) ρ2max ) Kσ2 . ( 120 ) where d1 ≤ d2 ≤ · · · ≤ dn = 1 are the singular values of Wk . Therefore , A2,2 ≤ 1 ( Np ) 2 ( p · np− 1 n− 1 ·Nσ2 + p ( 1− p ) n n− 1 ( 1 + ( n− 1 ) ρ2max ) Kσ2 ) = 1 Np ( np− 1 n− 1 σ2 ) + 1 Np ( 1− p n− 1 ( 1 + ( n− 1 ) ρ2max ) σ2 ) ≤ σ 2 N + σ2 N p−1 − 1 n− 1 ( 1 + ( n− 1 ) ρ2max ) ≤ σ 2 N ( 2 + p−1ρ2max ) . ( 121 ) The last inequality is due to p ≥ 1/n . C.5 PROOF OF PROPOSITION 6 Invoking the descent inequality Lemma 1 and the error bound for T1-T5 given by Corollary 2 , Proposition 3 and Eq . ( 41 ) : Ef ( xr+1 ) ≤Ef ( xr ) − η 4 τ−1∑ s=0 E‖∇f ( x̄r , s ) ‖2 − η 4 τ−1∑ s=0 E ∥∥∥ 1 N N∑ i=1 ∇fi ( xr , si ) ∥∥∥2 − η 4 τ−1∑ s=0 E ∥∥∥ 1 K K∑ k=1 ∇f̄k ( x̄r , sk ) ∥∥∥2 + C1L 2η3τ ( τ + ρ2maxDτ , ρ ) τ−1∑ s=0 ( α2‖∇f ( x̄r , s ) ‖2 + 2g ) + C1L2η3τ2ρ2maxDτ , ρ̄2L + C1τ 2L2η3ρ2maxσ 2 + C1L 2 ( τ2 +D2τ , ρρ 2 max ) η 3σ 2 n + η2L 2 τ−1∑ s=0 E ∥∥∥∥∥ 1N N∑ i=1 ∇fi ( xr , si ) ∥∥∥∥∥ 2 + η2L 2 τ σ2 N ≤Ef ( xr ) − η 4 τ−1∑ s=0 E‖∇f ( x̄r , s ) ‖2 + C1L 2η3τ ( τ + ρ2maxDτ , ρ ) τ−1∑ s=0 ( α2E‖∇f ( x̄r , s ) ‖2 + 2g ) + C1L2η3τ2ρ2maxDτ , ρ̄2L + C1τ 2L2η3ρ2maxσ 2 + C1L 2 ( τ2 +D2τ , ρρ 2 max ) η 3σ 2 n + η2Lτ σ2 N . ( 122 ) The last inequality holds under the condition that η ≤ 1 2L . ( 123 ) If we further enforce C1L 2η3τ ( τ + ρ2maxDτ , ρ ) α2 ≤ η 8 ⇔ η2 ≤ 1 8C1L2τ ( τ + ρ2maxDτ , ρ ) α 2 , ( 124 ) then Ef ( xr+1 ) ≤Ef ( xr ) − η 8 τ−1∑ s=0 E‖∇f ( x̄r , s ) ‖2 + C1L 2η3τ2 ( τ + ρ2maxDτ , ρ ) 2g + C1L 2η3τ2ρ2maxDτ , ρ̄ 2 L + C1τ 2L2η3ρ2maxσ 2 + C1L 2 ( τ2 +D2τ , ρρ 2 max ) η 3σ 2 n + η2Lτ σ2 N =Ef ( xr ) − η 8 τ−1∑ s=0 E‖∇f ( x̄r , s ) ‖2 + C1L2η3τ2 ( τ + ρ2maxDτ , ρ ) 2g + C1L 2η3 ( τ2ρ2max ( Dτ , ρ̄ 2 L + σ 2 ) + ( τ2 +D2τ , ρρ 2 max ) σ2 n ) + η2Lτ σ2 N ≤Ef ( xr ) − η 8 τ−1∑ s=0 E‖∇f ( x̄r , s ) ‖2 + η2Lτ σ 2 N + C1L 2η3τ2 ( τ + ρ2maxDτ , ρ ) 2g + 2C1L 2η3 ( τ2ρ2maxDτ , ρ̄ 2 L + τ 2σ2 ( 1 n + ρ2max ) ) , ( 125 ) the last inequality is due to Dτ , ρ ≤ τ and ρmax ≤ 1 . C.6 PROOF OF PROPOSITION 7 Invoking the descent inequality Lemma 1 and the error bound for T1-T5 given by Corollary 2 , Proposition 3 , and 4 : Ef ( xr+1 ) ≤ Ef ( xr ) − η 4 τ−1∑ s=0 E‖∇f ( x̄r , s ) ‖2 − η 4 τ−1∑ s=0 E ∥∥∥ 1 N N∑ i=1 ∇fi ( xr , si ) ∥∥∥2 − η 4 τ−1∑ s=0 E ∥∥∥ 1 K K∑ k=1 ∇f̄k ( x̄r , sk ) ∥∥∥2 + η 4 τ−1∑ s=0 E ∥∥∥∇f ( x̄r , s ) − 1 N N∑ i=1 ∇fi ( xr , si ) ∥∥∥2 + η 4 τ−1∑ s=0 E ∥∥∥∇f ( x̄r , s ) − 1 K K∑ k=1 ∇f̄k ( x̄r , sk ) ∥∥∥2 + η 4 τ−1∑ s=0 E ∥∥∥ 1 N N∑ i=1 ∇fi ( xr , si ) − 1 K K∑ k=1 ∇f̄k ( x̄r , sk ) ∥∥∥2 + τ−2∑ s=0 E ( L 2 ‖x̄r , s+1 − x̄r , s‖2 ) + E ( L 2 ‖xr+1 − x̄r , τ−1‖2 ) ≤Ef ( xr ) − η 4 τ−1∑ s=0 E‖∇f ( x̄r , s ) ‖2 − η 4 τ−1∑ s=0 E ∥∥∥ 1 N N∑ i=1 ∇fi ( xr , si ) ∥∥∥2 − η 4 τ−1∑ s=0 E ∥∥∥ 1 K K∑ k=1 ∇f̄k ( x̄r , sk ) ∥∥∥2 + C1L 2η3τ ( τ + ρ2maxDτ , ρ ) τ−1∑ s=0 ( α2‖∇f ( x̄r , s ) ‖2 + 2g ) + C1L2η3τ2ρ2maxDτ , ρ̄2L + C1τ 2L2η3ρ2maxσ 2 + C1L 2 ( τ2 +D2τ , ρρ 2 max ) η 3σ 2 n + L 2 η2 τ−2∑ s=0 E ∥∥∥∥∥ 1N N∑ i=1 ∇fi ( xr , si ) ∥∥∥∥∥ 2 + L 2 η2 ( τ − 1 ) σ 2 N + Lη2E ∥∥∥∥∥ 1K K∑ k=1 ∇f̄k ( x̄r , τ−1k ) ∥∥∥∥∥ 2 + L 2 ( Gp + 1 τ2 ) ( 8 N K∑ k=1 E‖Xr , τ−1k , ⊥ ‖ 2 ) + L 2 η2 σ2 N ( 2 + n m ρ2max ) . ( 126 ) Denote for short G′p , Gp + 1 τ2 . ( 127 ) Further applying the bounds on the consensus error derived in Proposition 2 : Ef ( xr+1 ) ≤Ef ( xr ) − η 4 τ−1∑ s=0 E‖∇f ( x̄r , s ) ‖2 − η 4 τ−1∑ s=0 E ∥∥∥ 1 N N∑ i=1 ∇fi ( xr , si ) ∥∥∥2 − η 4 τ−1∑ s=0 E ∥∥∥ 1 K K∑ k=1 ∇f̄k ( x̄r , sk ) ∥∥∥2 + C1L 2η3τ ( τ + ρ2maxDτ , ρ ) τ−1∑ s=0 ( α2‖∇f ( x̄r , s ) ‖2 + 2g ) + C1L2η3τ2ρ2maxDτ , ρ̄2L + C1τ 2L2η3ρ2maxσ 2 + C1L 2 ( τ2 +D2τ , ρρ 2 max ) η 3σ 2 n + L 2 η2 τ−2∑ s=0 E ∥∥∥∥∥ 1N N∑ i=1 ∇fi ( xr , si ) ∥∥∥∥∥ 2 + L 2 η2 ( τ − 1 ) σ 2 N + Lη2E ∥∥∥∥∥ 1K K∑ k=1 ∇f̄k ( x̄r , τ−1k ) ∥∥∥∥∥ 2 + 4LG′p ( 2C2 τ−2∑ s=0 D2τ , ρρmaxη 2 ( α2‖∇f ( x̄r , s ) ‖2 + 2g ) ) + 4LG′p ( C2D 2 τ , ρτρ 2 maxη 2̄2L + C2 ( 1 n + ρmax ) ρmaxτDτ , ρη 2σ2 ) + L 2 η2 σ2 N ( 2 + n m ρ2max ) ( 128 ) Rearranging terms and tidy up the expression we have Ef ( xr+1 ) ≤Ef ( xr ) − η 4 τ−1∑ s=0 E‖∇f ( x̄r , s ) ‖2 − η 4 τ−1∑ s=0 E ∥∥∥ 1 N N∑ i=1 ∇fi ( xr , si ) ∥∥∥2 − η 4 τ−1∑ s=0 E ∥∥∥ 1 K K∑ k=1 ∇f̄k ( x̄r , sk ) ∥∥∥2 + L 2 η2 τ−2∑ s=0 E ∥∥∥∥∥ 1N N∑ i=1 ∇fi ( xr , si ) ∥∥∥∥∥ 2 + Lη2E ∥∥∥∥∥ 1K K∑ k=1 ∇f̄k ( x̄r , τ−1k ) ∥∥∥∥∥ 2 + C1L 2η3τ ( τ + ρ2maxDτ , ρ ) τ−1∑ s=0 ( α2‖∇f ( x̄r , s ) ‖2 + 2g ) ( 129 ) + 8C2G ′ pLD 2 τ , ρρmaxη 2 ( τ−2∑ s=0 ( α2‖∇f ( x̄r , s ) ‖2 + 2g ) ) + C1L 2η3 ( τ2Dτ , ρρ 2 max̄ 2 L + τ 2ρ2maxσ 2 + ( τ2 +D2τ , ρρ 2 max ) σ2 n ) + L 2 ( τ − 1 ) η2σ 2 N + L 2 ( 2 + n m ρ2max ) η2 σ2 N + 4C2LG ′ pη 2 ( D2τ , ρτρ 2 max̄ 2 L + ( 1 n + ρmax ) ρmaxτDτ , ρσ 2 ) . Notice that if the following conditions on the learning rate are satisfied η 4 ≥ η2L , η 8 ≥ C1L2η3τ ( τ + ρ2maxDτ , ρ ) α2 + 8C2G ′ pLD 2 τ , ρρmaxη 2α2 , ( 130 ) then the terms associated to the gradients will be negative and Ef ( xr+1 ) ≤Ef ( xr ) − η 8 τ−1∑ s=0 E‖∇f ( x̄r , s ) ‖2 + C1L 2η3τ2 ( τ + ρ2maxDτ , ρ ) 2g + 2C1L 2η3 ( τ2ρ2maxDτ , ρ̄ 2 L + τ 2σ2 ( 1 n + ρ2max ) ) + Lτη2 σ2 N + ( 8C2G ′ pLD 2 τ , ρτρmax ) η2 2g + 4C2LG ′ pη 2 ( D2τ , ρτρ 2 max̄ 2 L + ( 1 n + ρmax ) ρmaxτDτ , ρσ 2 ) + L 2 ( n m ρ2max ) η2 σ2 N . In the last step we clean the condition on the learning rate η . Collecting all the conditions on η : η2 ≤ 1 24L2τ ( 4τ − 1 ) , ( 131 ) η 4 ≥ η2L , ( 132 ) η 8 ≥ C1L2η3τ ( τ + ρ2maxDτ , ρ ) α2 + 8C2G ′ pLD 2 τ , ρρmaxη 2α2 . ( 133 ) Clearly , ( 131 ) implies ( 132 ) . To ensure ( 133 ) it suffices to require η 16 ≥ C1L2η3τ ( τ + ρ2maxDτ , ρ ) α2 η 16 ≥ 8C2G′pLD2τ , ρρmaxη2α2 . ( 134 ) Recall the definition of G′p : G′p = Gp + 1 τ2 , and Gp = n−m m ( n− 1 ) . ( 135 ) It can be verified that G′p ≤ Gp + 1 τ2 ≤ 1 m + 1 ≤ 2 . ( 136 ) Using the fact that Dτ , ρ ≤ τ and ρmax ≤ 1 we can see that it suffices to require η ≤ 1 C3α2τL . ( 137 ) for some C3 > 0 large enough to satisfy both ( 131 ) and ( 133 ) . D SUPPORTING LEMMAS D.1 PROOF OF LEMMA 1 Since the global average of the local copies follows the update [ cf . ( 19 ) ] : x̄r , s+1 = x̄r , s − η 1 N N∑ i=1 gr , si , ∀s = 0 , . . . , τ − 1 . ( 138 ) Under Assumption 1 , we can apply the descent lemma at points x̄r , s+1 and x̄r , s for s = 0 , . . . , τ−2 , conditioned on Fr , s−1 : Er , s−1f ( x̄r , s+1 ) ≤ f ( x̄r , s ) +∇f ( x̄r , s ) > Er , s−1 ( x̄r , s+1 − x̄r , s ) + Er , s−1 ( L 2 ‖x̄r , s+1 − x̄r , s‖2 ) ( a ) =f ( x̄r , s ) − η∇f ( x̄r , s ) > ( 1 N N∑ i=1 ∇fi ( xr , si ) ) + Er , s−1 ( L 2 ‖x̄r , s+1 − x̄r , s‖2 ) =f ( x̄r , s ) − 1 2 η∇f ( x̄r , s ) > ( 1 N N∑ i=1 ∇fi ( xr , si ) ) − 1 2 η∇f ( x̄r , s ) > ( 1 N N∑ i=1 ∇fi ( xr , si ) ± 1 K K∑ k=1 ∇f̄k ( x̄r , sk ) ) + Er , s−1 ( L 2 ‖x̄r , s+1 − x̄r , s‖2 ) ( b ) ≤f ( x̄r , s ) − η 2 ( 1 2 ‖∇f ( x̄r , s ) ‖2 + 1 2 ∥∥∥ 1 N N∑ i=1 ∇fi ( xr , si ) ∥∥∥2 − 1 2 ∥∥∥∇f ( x̄r , s ) − 1 N N∑ i=1 ∇fi ( xr , si ) ∥∥∥2 ) − η 2 ( 1 2 ‖∇f ( x̄r , s ) ‖2 + 1 2 ∥∥∥ 1 K K∑ k=1 ∇f̄k ( x̄r , sk ) ∥∥∥2 − 1 2 ∥∥∥∇f ( x̄r , s ) − 1 K K∑ k=1 ∇f̄k ( x̄r , sk ) ∥∥∥2 ) + η 2 ‖∇f ( x̄r , s ) ‖ ∥∥∥ 1 N N∑ i=1 ∇fi ( xr , si ) − 1 K K∑ k=1 ∇f̄k ( x̄r , sk ) ∥∥∥ + Er , s−1 ( L 2 ‖x̄r , s+1 − x̄r , s‖2 ) ( c ) ≤f ( x̄r , s ) − η 4 ‖∇f ( x̄r , s ) ‖2 − η 4 ∥∥∥ 1 N N∑ i=1 ∇fi ( xr , si ) ∥∥∥2 − η 4 ∥∥∥ 1 K K∑ k=1 ∇f̄k ( x̄r , sk ) ∥∥∥2 + η 4 ∥∥∥∇f ( x̄r , s ) − 1 N N∑ i=1 ∇fi ( xr , si ) ∥∥∥2 + η 4 ∥∥∥∇f ( x̄r , s ) − 1 K K∑ k=1 ∇f̄k ( x̄r , sk ) ∥∥∥2 + η 4 ∥∥∥ 1 N N∑ i=1 ∇fi ( xr , si ) − 1 K K∑ k=1 ∇f̄k ( x̄r , sk ) ∥∥∥2 + Er , s−1 ( L 2 ‖x̄r , s+1 − x̄r , s‖2 ) , where ( a ) is due to Assumption 2 , ( b ) is due to 2ab = ‖a‖2 + ‖b‖2 − ‖a − b‖2 and ab ≤ ‖a‖‖b‖ , and ( c ) is due to ‖a‖‖b‖ ≤ 12‖a‖ 2 + 12‖b‖ 2 . Notation a ± b stands for adding and subtracting , i.e. , a± b = a+ b− b . For the pair ( x̄r , τ−1 , xr+1 ) we have according to ( 19 ) and ( 22 ) : Er , τ−2xr+1 = Er , τ−2 ( x̄r , τ ) = Er , τ−2 ( x̄r , τ−1 − η 1 N N∑ i=1 gr , τ−1i ) . Applying the descent lemma in the same way as before yields Er , τ−2f ( xr+1 ) ≤f ( x̄r , τ−1 ) +∇f ( x̄r , τ−1 ) > Er , τ−2 ( xr+1 − x̄r , τ−1 ) + L 2 Er , τ−2‖xr+1 − x̄r , τ−1‖2 ≤f ( x̄r , τ−1 ) − η 4 ‖∇f ( x̄r , τ−1 ) ‖2 − η 4 ∥∥∥ 1 N N∑ i=1 ∇fi ( xr , τ−1i ) ∥∥∥2 − η 4 ∥∥∥ 1 K K∑ k=1 ∇f̄k ( x̄r , τ−1k ) ∥∥∥2 + η 4 ∥∥∥∇f ( x̄r , τ−1 ) − 1 N N∑ i=1 ∇fi ( xr , τ−1i ) ∥∥∥2 + η 4 ∥∥∥∇f ( x̄r , τ−1 ) − 1 K K∑ k=1 ∇f̄k ( x̄r , τ−1k ) ∥∥∥2 + η 4 ∥∥∥ 1 N N∑ i=1 ∇fi ( xr , τ−1i ) − 1 K K∑ k=1 ∇f̄k ( x̄r , τ−1k ) ∥∥∥2 + Er , τ−2 ( L 2 ‖xr+1 − x̄r , τ−1‖2 ) . Taking expectation , summing over the iterations in round r over s = 0 , . . . , τ − 1 and using the fact that xr = x̄r,0 completes the proof . D.2 PROOF OF LEMMA 2 Recall the average update of the k-th cluster and that of the global average given by ( 15 ) and ( 19 ) , respectively , for s = 0 , . . . , τ − 1 : x̄r , s+1k = x̄ r , s k − η · ḡ r , s k ( 139 ) x̄r , s+1 = x̄r , s − η · 1 N N∑ i=1 gr , si . ( 140 ) Taking the difference gives E‖x̄r , s+1 − x̄r , s+1k ‖ 2 =E ∥∥∥∥∥ ( x̄r , s − x̄r , sk ) − η ( 1 n ∑ i∈Vk ∇fi ( xr , si ) − 1 N N∑ i=1 ∇fi ( xr , si ) ) ∥∥∥∥∥ 2 + η2E ∥∥∥∥∥ 1n ∑ i∈Vk ξr , si − 1 N N∑ i=1 ξr , si ∥∥∥∥∥ 2 ≤ ( 1 + ) E‖x̄r , s − x̄r , sk ‖ 2 + ( 1 + −1 ) η2E ∥∥∥∥∥ 1n ∑ i∈Vk ∇fi ( xr , si ) − 1 N N∑ i=1 ∇fi ( xr , si ) ∥∥∥∥∥ 2 + η2E ∥∥∥∥∥ 1n ∑ i∈Vk ξr , si − 1 N N∑ i=1 ξr , si ∥∥∥∥∥ 2 , ( 141 ) where > 0 is some constant to be chosen . Averaging over k = 1 , . . . , K : 1 K K∑ k=1 E‖x̄r , s+1 − x̄r , s+1k ‖ 2 ≤ ( 1 + ) 1 K K∑ k=1 E‖x̄r , s − x̄r , sk ‖ 2 + ( 1 + −1 ) η2 1 K K∑ k=1 E ∥∥∥∥∥ 1n ∑ i∈Vk ∇fi ( xr , si ) − 1 N N∑ i=1 ∇fi ( xr , si ) ∥∥∥∥∥ 2 + η2 1 K K∑ k=1 E ∥∥∥∥∥ 1n ∑ i∈Vk ξr , si − 1 N N∑ i=1 ξr , si ∥∥∥∥∥ 2 ( a ) = ( 1 + ) 1 K K∑ k=1 E‖x̄r , s − x̄r , sk ‖ 2 + ( 1 + −1 ) η2 ( 1 K K∑ k=1 E ∥∥∥ 1 n ∑ i∈Vk ∇fi ( xr , si ) ∥∥∥2 − E∥∥∥ 1 N N∑ i=1 ∇fi ( xr , si ) ∥∥∥2 ) + η2 ( 1 K K∑ k=1 E ∥∥∥ 1 n ∑ i∈Vk ξr , si ∥∥∥2 − E∥∥∥ 1 N N∑ i=1 ξr , si ∥∥∥2 ) ≤ ( 1 + ) 1 K K∑ k=1 E‖x̄r , s − x̄r , sk ‖ 2 + ( 1 + −1 ) η2 ( 1 K K∑ k=1 E ∥∥∥ 1 n ∑ i∈Vk ∇fi ( xr , si ) ∥∥∥2 ) + η2K − 1 N σ2 = ( 1 + ) 1 K K∑ k=1 E‖x̄r , s − x̄r , sk ‖ 2 + ( 1 + −1 ) η2 ( 1 K K∑ k=1 E ∥∥∥ 1 n ∑ i∈Vk ( ∇fi ( xr , si ) ±∇f̄k ( x̄ r , s k ) ±∇f̄k ( x̄ r , s ) ) ∥∥∥2 ) + η2K − 1 N σ2 ≤ ( 1 + ) 1 K K∑ k=1 E‖x̄r , s − x̄r , sk ‖ 2 + ( 1 + −1 ) η2 ( 3 K K∑ k=1 E ∥∥∥ 1 n ∑ i∈Vk ( ∇fi ( xr , si ) −∇f̄k ( x̄ r , s k ) ) ∥∥∥2 ) + ( 1 + −1 ) η2 3 K K∑ k=1 E ∥∥∥∇f̄k ( x̄r , sk ) −∇f̄k ( x̄r , s ) ∥∥∥2 + ( 1 + −1 ) η2 3K K∑ k=1 E‖∇f̄k ( x̄r , s ) ‖2 + η2 K − 1 N σ2 ( b ) ≤ ( 1 + ) 1 K K∑ k=1 E‖x̄r , s − x̄r , sk ‖ 2 + ( 1 + −1 ) η2 ( 3 N K∑ K=1 L2E ∥∥∥Xr , sk , ⊥∥∥∥2 ) + ( 1 + −1 ) η2 3 K K∑ k=1 L2E ∥∥∥x̄r , sk − x̄r , s∥∥∥2 + ( 1 + −1 ) η2 3K K∑ k=1 E ∥∥∥∇f̄k ( x̄r , s ) ∥∥∥2 + η2K − 1 N σ2 = ( 1 + + 3L2η2 ( 1 + −1 ) ) 1 K K∑ k=1 E‖x̄r , s − x̄r , sk ‖ 2 + ( 1 + −1 ) η2 ( 3 N K∑ K=1 L2E ∥∥∥Xr , sk , ⊥∥∥∥2 ) + ( 1 + −1 ) η2 3 K K∑ k=1 E ∥∥∥∇f̄k ( x̄r , s ) ∥∥∥2 + η2K − 1 N σ2 . ( 142 ) In ( a ) we used the fact that 1 K K∑ k=1 1 n ∑ i∈Vk ∇fi ( xr , si ) = 1 N N∑ i=1 ∇fi ( xr , si ) , 1 K K∑ k=1 1 n ∑ i∈Vk ξr , si = 1 N N∑ i=1 ξr , si . ( 143 ) and K∑ i=1 ‖xi − x̄‖2 = K∑ i=1 ‖xi‖2 −K‖x̄‖2 with x̄ = 1 K K∑ k=1 xi . ( 144 ) In ( b ) we applied the L-smoothness of fi and f̄k . Choosing = 14τ−1 and using the condition that η2 ≤ 1 24τ ( 4τ − 1 ) L2 we have 1 K K∑ k=1 E‖x̄r , s+1 − x̄r , s+1k ‖ 2 ≤ ( 1 + 1 4τ − 1 + 1 2 ( 4τ − 1 ) ) 1 K K∑ k=1 E‖x̄r , s − x̄r , sk ‖ 2 + 12τη2L2 ( 1 N K∑ K=1 E ∥∥∥Xr , sk , ⊥∥∥∥2 ) + 12τη2 1 K K∑ k=1 E ∥∥∥∇f̄k ( x̄r , s ) ∥∥∥2 + η2K − 1 N σ2 ≤Cτ 1 K K∑ k=1 E‖x̄r , s+1 − x̄r , s+1k ‖ 2 + 12τη2L2 ( 1 N K∑ K=1 E ∥∥∥Xr , sk , ⊥∥∥∥2 ) + 12τη2 ( α2E‖∇f ( x̄r , s ) ‖2 + 2g ) + η2 K − 1 N σ2 . In the last inequality we applied Assumption 5 on the inter-cluster heterogeneity . D.3 PROOF OF LEMMA 3 We follow the perturbed average consensus analysis . Recall the update equation of the consensus error given in ( 17 ) : Xr , s+1k , ⊥ = ( Wk − J ) ( X r , s k , ⊥ − ηG r , s k ) . ( 145 ) Squaring both sides and conditioning : E‖Xr , s+1k , ⊥ ‖ 2 = E ( E ( ‖ ( Wk − J ) ( Xr , sk , ⊥ ± η∇Fk ( X r , s k ) − ηG r , s k ) ‖ 2|Fr , s−1 ) ) ≤ E‖ ( Wk − J ) ( Xr , sk , ⊥ − η∇Fk ( X r , s k ) ) ‖ 2 + η2ρ2knσ 2 ≤ ρ2k ( 1 + ζ−1k ) · E‖X r , s k , ⊥‖ 2 + ρ2k ( 1 + ζk ) η 2E‖∇Fk ( Xr , sk ) ‖ 2 + η2ρ2knσ 2 , where ζk > 0 is some free parameter to be properly chosen . Next , we bound the norm of the pseudo-gradient ∇Fk ( Xr , sk ) . ‖∇Fk ( Xr , sk ) ‖ 2 = ∑ i∈Vk ‖∇fi ( xr , si ) ‖ 2 = ∑ i∈Vk ‖∇fi ( xr , si ) ±∇fi ( x̄ r , s k ) ±∇f̄k ( x̄ r , s k ) ±∇f̄k ( x̄ r , s ) ‖2 ≤ ∑ i∈Vk ( 4‖∇fi ( xr , si ) −∇fi ( x̄ r , s k ) ‖ 2 + 4‖∇fi ( x̄r , sk ) −∇f̄k ( x̄ r , s k ) ‖ 2 + 4‖∇f̄k ( x̄r , sk ) −∇f̄k ( x̄ r , s ) ‖2 ) + ∑ i∈Vk 4‖∇f̄k ( x̄r , s ) ‖2 ( 146 ) ≤ ∑ i∈Vk ( 4‖∇fi ( x̄r , sk ) −∇f̄k ( x̄ r , s k ) ‖ 2 + 4L2‖xr , si − x̄ r , s k ‖ 2 + 4L2‖x̄r , sk − x̄ r , s‖2 + 4‖∇f̄k ( x̄r , s ) ‖2 ) ≤ 4L2‖Xr , sk , ⊥‖ 2 + 4L2n‖x̄r , sk − x̄ r , s‖2 + 4n‖∇f̄k ( x̄r , s ) ‖2 + 4n 2k . The last inequality is due to Assumption 4 on the intra-cluster heterogeneity . Averaging over k = 1 , . . . , K clusters : 1 N K∑ k=1 E‖Xr , s+1k , ⊥ ‖ 2 ≤ 1 N K∑ k=1 ρ2k ( 1 + ζ −1 k ) E‖X r , s k , ⊥‖ 2 + 1 N K∑ k=1 ρ2k ( 1 + ζk ) η 2E‖∇Fk ( Xr , sk ) ‖ 2 + η2 ( 1 K K∑ k=1 ρ2k ) σ2 ≤ 1 N K∑ k=1 ρ2k ( 1 + ζ −1 k ) E‖X r , s k , ⊥‖ 2 + η2 1 N K∑ k=1 ρ2k ( 1 + ζk ) · 4L2E‖X r , s k , ⊥‖ 2 + η2 1 K K∑ k=1 ρ2k ( 1 + ζk ) 4L 2E‖x̄r , sk − x̄ r , s‖2 + η2 1 K K∑ k=1 ρ2k ( 1 + ζk ) 4E‖∇f̄k ( x̄r , s ) ‖2 + η2 1 K K∑ k=1 ρ2k ( 1 + ζk ) 4 2 k + η 2ρ2maxσ 2 ( 147 ) ≤ ( max k∈ [ K ] ρ2k ( 1 + ζ −1 k ) + η 2 · 4L2 max k∈ [ K ] { ρ2k ( 1 + ζk ) } ︸ ︷︷ ︸ ρL ) 1 N K∑ k=1 E‖Xr , sk , ⊥‖ 2 + η2 max k∈ [ K ] { ρ2k ( 1 + ζk ) } · 4L2 1 K K∑ k=1 E‖x̄r , sk − x̄ r , s‖2 + η2 max k∈ [ K ] { ρ2k ( 1 + ζk ) } · 4 ( α2E‖∇f ( x̄r , s ) ‖2 + 2g ) + η2 1 K K∑ k=1 ρ2k ( 1 + ζk ) 4 2 k + η 2ρ2maxσ 2 . D.4 PROOF OF LEMMA 4 To simplify the notation we omit the superscript in Br , s− ` in this section . Let Λ = diag ( λ1 , λ2 ) and the eigendecomposition of G = TΛT−1 , we can obtain the closed form expression of T as T = ( λ1 − Cτ λ2 − Cτ 12τη2L2 12τη2L2 ) and T−1 = 1 det ( T ) ( 12τη2L2 −λ2 + Cτ −12τη2L2 λ1 − Cτ ) , where det ( T ) = 12τη2L2 ( λ1 − λ2 ) . ( 148 ) Consequently det ( T ) ·G ` B = det ( T ) · TΛ ` T−1B = ( λ1 − Cτ λ2 − Cτ 12τη2L2 12τη2L2 ) ( λ ` 1 0 0 λ ` 2 ) ( 12τη2L2 −λ2 + Cτ −12τη2L2 λ1 − Cτ ) ( b1 b2 ) = ( λ1 − Cτ λ2 − Cτ 12τη2L2 12τη2L2 ) ( λ ` 1 0 0 λ ` 2 ) ( 12τη2L2b1 + ( −λ2 + Cτ ) b2 −12τη2L2b1 + ( λ1 − Cτ ) b2 ) = ( λ1 − Cτ λ2 − Cτ 12τη2L2 12τη2L2 ) ( λ ` 112τη 2L2b1 + λ ` 1 ( −λ2 + Cτ ) b2 −λ ` 212τη2L2b1 + λ ` 2 ( λ1 − Cτ ) b2 ) = ( t1 t2 ) ( 149 ) with t1 = ( λ1 − Cτ ) ( λ ` 112τη 2L2b1 + λ ` 1 ( −λ2 + Cτ ) b2 ) + ( λ2 − Cτ ) ( −λ ` 212τη2L2b1 + λ ` 2 ( λ1 − Cτ ) b2 ) , ( 150 ) t2 = 12τη 2L2 ( λ ` 112τη 2L2b1 + λ ` 1 ( −λ2 + Cτ ) b2 ) + 12τη2L2 ( −λ ` 212τη2L2b1 + λ ` 2 ( λ1 − Cτ ) b2 ) . ( 151 ) Therefore det ( T ) ( 1 , 1 ) TΛ ` T−1B = t1 + t2 = ( λ1 − Cτ ) ( λ ` 1L 212τη2b1 + λ ` 1 ( −λ2 + Cτ ) b2 ) + ( λ2 − Cτ ) ( −λ ` 2L212τη2b1 + λ ` 2 ( λ1 − Cτ ) b2 ) + L212τη2 ( λ ` 1L 212τη2b1 + λ ` 1 ( −λ2 + Cτ ) b2 ) + L212τη2 ( −λ ` 2L212τη2b1 + λ ` 2 ( λ1 − Cτ ) b2 ) . ( 152 ) Substituting the expression of det ( T ) and dividing both sides of the equality by 12τη2 ( λ1−λ2 ) we have L2 ( 1 , 1 ) TΛ ` T−1B = 1 12τη2 ( λ1 − λ2 ) ( λ1 − Cτ ) ( λ ` 1L 212τη2b1 + λ ` 1 ( −λ2 + Cτ ) b2 ) + 1 12τη2 ( λ1 − λ2 ) ( λ2 − Cτ ) ( −λ ` 2L212τη2b1 + λ ` 2 ( λ1 − Cτ ) b2 ) + 1 ( λ1 − λ2 ) L2 ( λ ` 1L 212τη2b1 + λ ` 1 ( −λ2 + Cτ ) b2 ) + 1 ( λ1 − λ2 ) L2 ( −λ ` 2L212τη2b1 + λ ` 2 ( λ1 − Cτ ) b2 ) = 1 ( λ1 − λ2 ) ( λ1 − Cτ ) λ ` 1L2b1 + −1 ( λ1 − λ2 ) ( λ2 − Cτ ) λ ` 2L2b1 + 1 ( λ1 − λ2 ) L4λ ` 112τη 2b1 − 1 ( λ1 − λ2 ) L4λ ` 212τη 2b1 + 1 12τη2 ( λ1 − λ2 ) ( λ1 − Cτ ) λ ` 1 ( −λ2 + Cτ ) b2 + 1 12τη2 ( λ1 − λ2 ) ( λ2 − Cτ ) λ ` 2 ( λ1 − Cτ ) b2 + 1 ( λ1 − λ2 ) L2λ ` 1 ( −λ2 + Cτ ) b2 + 1 ( λ1 − λ2 ) L2λ ` 2 ( λ1 − Cτ ) b2 = 1 ( λ1 − λ2 ) ( −λ ` 2 ( λ2 − Cτ ) − λ ` 1 ( Cτ − λ1 ) ) L2b1 + 1 ( λ1 − λ2 ) 12τη2 ( λ ` 1 − λ ` 2 ) L4b1 + 1 12τη2 ( λ1 − λ2 ) ( λ1 − Cτ ) λ ` 1 ( −λ2 + Cτ ) b2 + 1 12τη2 ( λ1 − λ2 ) ( λ2 − Cτ ) λ ` 2 ( λ1 − Cτ ) b2 + 1 ( λ1 − λ2 ) L2λ ` 1 ( −λ2 + Cτ ) b2 + 1 ( λ1 − λ2 ) L2λ ` 2 ( λ1 − Cτ ) b2 ≤λ ` 2L2b1 + λ ` 2 − λ ` 1 λ2 − λ1 · 12τη2L4b1 + λ ` 2 − λ ` 1 λ2 − λ1 ( λ2 − Cτ ) ( Cτ − λ1 ) 1 12τη2 b2 + λ ` 2L 2b2 , ( 153 ) where in the last inequality we used the fact that λ1 ≤ Cτ ≤ λ2 . Note that ( λ2 − Cτ ) ( Cτ − λ1 ) =− C2τ − λ1λ2 + ( λ1 + λ2 ) Cτ =− C2τ − det ( G ) + Tr ( G ) Cτ =− C2τ − ( Cτ ( max k∈ [ K ] ρ2k ( 1 + ζ −1 k ) + η 2ρL · 4L2 ) − 48ρLτη4L4 ) + Cτ ( max k∈ [ K ] ρ2k ( 1 + ζ −1 k ) + η 2ρL · 4L2 + Cτ ) = 48ρLτη 4L4 . ( 154 ) Therefore , we further obtain L2 ( 1 , 1 ) TΛ ` T−1B ≤λ ` 2L2 ( b1 + b2 ) + λ ` 2 − λ ` 1 λ2 − λ1 η2 · ( 12τL4b1 + 4ρLL 4b2 ) ( 155 ) Dividing both sides by L2 completes the proof . E NETWORK CONNECTIVITY CONDITIONS IN THEOREMS AND COROLLARY Both Theorem 2 and Corollary 1 impose some sufficient conditions on the network connectivity ρmax for convergence . This can be satisfied in practice as follows . For Theorem 2 , as long as ρmax < 1 , we can choose τ large enough so that ( 7 ) is fulfilled . Corollary 1 strengthens the result of Theorem 2 by requiring no loss in the order of convergence rate compared to full device participation . This naturally leads to a more stringent condition on ρmax given by ( 11 ) . For any given D2D network topology , this can be satisfied by running multiple D2D gossip averaging steps per SGD update in Algorithm 1 . Since the right hand side of ( 11 ) depends only on the algorithmic parameters , we can choose the suitable gossip averaging steps to fulfill this condition before launching the algorithm . F MORE EXPERIMENTS DETAILS In this section , we provide additional experimental results on CIFAR-10 dataset . We follow the same CNN model and non-iid data partition strategy as before and run each experiments for 3 times with different random seeds to report the mean values of best test accuracy . Instead of using a constant learning rate , we decay the local learning rate η by half after finishing 50 % and 75 % of the communication rounds and tune the initial learning rate from { 0.01 , 0.02 , 0.05 , 0.08 , 0.1 } for each algorithm . First , we evaluate the convergence processes of HL-SGD and local SGD under varying D2D network topologies in Figure 4 . We generate random network topologies by Erdős-Rényi model with edge probability from { 0.2 , 0 , 5 , 0.8 , 1 } and use Metropolis-Hastings weights to setWk , corresponding to spectral norm ρmax = { 0.9394 , 0.844 , 0.5357 , 0 } . As observed in Figure 4a , a more connected D2D network topology ( i.e. , a smaller value of ρmax ) generally accelerates the convergence and leads to a higher model accuracy achieved over 100 communication rounds in HL-SGD . However , in terms of runtime , a more connected D2D network topology corresponds to a larger D2D communication delay cd2d per round , and hence the total runtime is larger as well , which can be clearly observed in Figure 4b . Therefore , to achieve a target level of model accuracy within the shortest time in HLSGD , a sparse D2D network topology could work better than the fully connected one in practice . Second , to give a more comprehensive analysis on the runtime benefits of HL-SGD , we vary τ from { 5 , 10 , 20 , 50 } and compare the performances of HL-SGD and local SGD in Figure 5 . From the figure , we can observe that HL-SGD can consistently outperform local SGD across a wide range of τ . In particular , on CIFAR-10 , the best accuracy of HL-SGD achieved over 100 rounds is 2.49 % , 3.99 % , 4.05 % , and 7 % higher than that of local SGD , respectively , as τ increases from 5 to 50 . At the same time , the runtime of HL-SGD needed to achieve a target test-set accuracy of 60 % is 9.66 % , 19.76 % , 33.46 % , and 45.88 % less than that of local SGD , respectively . Third , we consider an extension of HL-SGD by allowing each device to perform multiple SGD updates before the gossip averaging step in Algorithm 1 and empirically evaluate its performance . Specifically , each device performs l = { 1 , 5 , 10 } steps of SGD update before aggregating models with their neighbors in the same cluster . Note that l = 1 corresponds to the original version of HLSGD in Algorithm 1 . As observed in Figure 6a , when communicating and aggregating models with neighbors more frequently , HL-SGD with l = 1 has the best convergence speed and will converge to the highest level of test accuracy . In terms of runtime , choosing a value of l > 1 might be favorable in some cases due to the reduced D2D communication delay per round . For instance , to achieve a target level of 60 % test accuracy , HL-SGD with l = 5 needs 5.22 % less amount of time than l = 1 . It is an interesting direction to rigorously analyze the convergence properties of HL- SGD with arbitrary l and find the best hyperparameter tuning method for minimizing the runtime to achieve a target level of model accuracy in the future .

Summary: The paper proposes a new method to address the heterogeneous communication setting in a hierarchical distributed system.  The system consists of a central server and several disjoint clusters, and each cluster is a local network connection between multiple edge devices. The communications are heterogeneous in the sense that in cluster device to device (D2D) communication are much faster than the device to server (D2S) communication. The proposes algorithm is a hybrid method where decentralized method is applied within clusters (on edge devices) and federated average is applied between cluster and server. Convergence analysis is provided showing the benefit of variance reduction using decentralization within cluster. Empirical studies are conducted to demonstrate the improvement under synthetic federated learning environment.