Title: Global Update Tracking: A Decentralized Learning Algorithm for Heterogeneous Data
Venue: NeurIPS 2023
Accepted: yes
Keywords: Federated Learning

Decentralized Learning

Non-IID Data

Heterogeneous data distribution

Peer-to-peer connectivity
Abstract: Decentralized learning enables the training of deep learning models over large distributed datasets generated at different locations, without the need for a central server. However, in practical scenarios, the data distribution across these devices can be significantly different, leading to a degradation in model performance. In this paper, we focus on designing a decentralized learning algorithm that is less susceptible to variations in data distribution across devices. We propose Global Update Tracking (GUT), a novel tracking-based method that aims to mitigate the impact of heterogeneous data in decentralized learning without introducing any communication overhead. We demonstrate the effectiveness of the proposed technique through an exhaustive set of experiments on various Computer Vision datasets (CIFAR-10, CIFAR-100, Fashion MNIST, and ImageNette), model architectures, and network topologies. Our experiments show that the proposed method achieves state-of-the-art performance for decentralized learning on heterogeneous data via a 1-6% improvement in test accuracy compared to other existing techniques.

Global Update Tracking: A Decentralized Learning Algorithm for Heterogeneous Data

Abstract

Decentralized learning enables training of deep learning models over large distributed datasets generated at different locations, without the need for a central server. However, in practical scenarios, the data distribution across these devices can be significantly different, leading to a degradation in model performance. In this paper, we focus on designing a decentralized learning algorithm that is less susceptible to variations in data distribution across devices. We propose Global Update Tracking (GUT), a novel tracking-based method that aims to mitigate the impact of heterogeneous data in decentralized learning without introducing any communication overhead. We demonstrate the effectiveness of the proposed technique through an exhaustive set of experiments on various Computer Vision datasets (CIFAR-10, CIFAR-100, Fashion MNIST, and ImageNette), model architectures, and network topologies. Our experiments show that the proposed method achieves state-of-the-art performance for decentralized learning on heterogeneous data via a 1 -6% improvement in test accuracy compared to other existing techniques.

1 Introduction

Decentralized learning is a branch of distributed optimization which focuses on learning from data distributed across multiple agents without a central server. It offers many advantages over the traditional centralized approach in core aspects such as data privacy, fault tolerance, and scalability [18]. It has been demonstrated that decentralized learning algorithms [15] can perform comparable to centralized algorithms on benchmark vision datasets. Decentralized Parallel Stochastic Gradient Descent (DSGD) presented in [15] combines SGD with a gossip averaging algorithm [26]. Further, the authors analytically show that the convergence rate of DSGD is similar to its centralized counterpart [5]. A momentum version of DSGD referred to as Decentralized Momentum Stochastic Gradient Descent (DSGDm) was proposed in [3]. The authors in [2] introduce Stochastic Gradient Push (SGP) which extends DSGD to directed and time-varying graphs. Recently, a unified framework for analyzing gossip-based decentralized SGD methods and the best-known convergence guarantees was presented in [11].

One of the key assumptions to achieve state-of-the-art performance by all the above-mentioned decentralized algorithms is that the data is independently and identically distributed (IID) across the agents. In particular, the data is assumed to be distributed in a uniform and random manner across the agents. This assumption does not hold in most real-world settings where the data distributions across the agents are significantly different (non-IID/heterogeneous) [9]. The effect of heterogeneous data in a peer-to-peer decentralized setup is a relatively under-studied problem and an active area of research.

Recently, there have been few efforts to bridge the performance gap between IID and non-IID data for a decentralized setup [16, 23, 19, 7, 1, 24]. Cross Gradient Aggregation [7] and Neighborhood

Gradient Clustering [1] algorithms utilize the concept of cross-gradients to reduce the impact of heterogeneous data and show significant improvement in performance (test accuracy). However, these techniques incur 2 × communication cost than the standard decentralized algorithms such as DSGD. D 2 algorithm proposed in [23] is shown to be agnostic to data heterogeneity and can be employed in deep learning tasks. One of the major limitations of D 2 is that its convergence requires mixing topologies with negative eigenvalue bounded from below by -1 3 . Additionally, it has been shown that D 2 performs worse than DSGD in some cases [16].

Tracking mechanisms such as Gradient Tracking (GT) [6, 19] and Momentum Tracking (MT) [22] have been proposed to tackle heterogeneous data in decentralized settings. But these algorithms improve the performance at the cost of 2 × communication overhead. The authors in [16] introduce Quasi-Global Momentum (QGM), a decentralized learning method that mimics the global synchronization of momentum buffer to mitigate the difficulties of decentralized learning on heterogeneous data. Recently, RelaySGD was presented in [24] that replaces the gossip averaging step with RelaySum. Since RelaySGD deals with the gossip averaging step, it is orthogonal to the aforementioned algorithms and can be used in synergy with them. QG-DSGDm [16] which incorporates QGMinto DSGDm sets the current state-of-the-art for decentralized learning on heterogeneous data without increasing the communication cost. This work investigates the following question: Can we improve decentralized learning on heterogeneous data through a tracking mechanism without any communication overhead?

To that effect, we present Global Update Tracking (GUT) , a novel decentralized learning algorithm designed to improve performance under heterogeneous data distribution. Motivated by, yet distinct from, the gradient tracking mechanism, we propose to track the consensus model ( ¯ x t ) by tracking global/average model updates, where x t i is the model parameters on agent i at time step t and ¯ x is the averaged model parameters. In the traditional tracking-based methods [19, 22] that track average gradients, each agent communicates both model parameters x t i and the tracking variable y t i with its neighbors resulting in 2 × communication overhead. The proposed GUT algorithm overcomes this bottleneck by allowing agents to store a copy of their neighbors' model parameters and then tracking the model updates instead of the gradients. This results in communicating only the tracking variable y t i that yields the model update ( x t i -x t -1 i ). We demonstrate the effectiveness of the proposed algorithm through an exhaustive set of experiments on various datasets, model architectures, and graph topologies. We also provide a detailed convergence analysis showing that the convergence rate of GUT algorithm is consistent with the state-of-the-art decentralized learning algorithms. Further, we show that QG-GUTm - Global Update Tracking with Quasi-Global momentum beats the current state-of-the-art decentralized learning algorithm (i.e., QG-DSGDm) on heterogeneous data under iso-communication cost.

1.1 Contributions

In summary, we make the following contributions.

We propose Global Update Tracking (GUT) , a novel tracking-based decentralized learning algorithm to mitigate the impact of heterogeneous data distribution.

We theoretically establish the non-asymptotic convergence rate of the proposed algorithm to a first-order solution.

Through an exhaustive set of experiments on various datasets, model architectures, and graph topologies, we establish that the proposed Global Update Tracking with Quasi-Global momentum ( QG-GUTm ) outperforms the current state-of-the-art decentralized learning algorithm on a spectrum of heterogeneous data.

2 Background

In this section, we provide the background on the decentralized setup with peer-to-peer connections.

The main goal of decentralized machine learning is to learn a global model using the knowledge extracted from the locally stored data samples across n agents while maintaining privacy constraints. In particular, we solve the optimization problem of minimizing the global loss function f ( x ) distributed across n agents as given in (1). Note that F i is a local loss function (for example, cross-entropy loss)

defined in terms of the data ( d i ) sampled from the local dataset D i at agent i .

The optimization problem is typically solved by combining stochastic gradient descent [4] with global consensus-based gossip averaging [26]. The communication topology is modeled as a graph G = ([ N ] , E ) with edges { i, j } ∈ E if and only if agents i and j are connected by a communication link exchanging the messages directly. We represent N ( i ) as the neighbors of agent i including itself. It is assumed that the graph G is strongly connected with self-loops i.e., there is a path from every agent to every other agent. The adjacency matrix of the graph G is referred to as a mixing matrix W where w ij is the weight associated with the edge { i, j } . Note that, weight 0 indicates the absence of a direct edge between the agents, and the elements of the Identity matrix are represented by I ij . Similar to the majority of previous works in decentralized learning, the mixing matrix is assumed to be doubly stochastic. Further, the initial models and all the hyperparameters are synchronized at the beginning of the training. The communication among the agents is assumed to be synchronous.

Traditional decentralized algorithms such as DSGD [15] assume the data across the agents to be Independent and Identically Distributed (IID). In DSGD, each agent i maintains local parameters x t i ∈ R d and updates them as follows.

We focus on a decentralized setup with non-IID/heterogeneous data. In particular, the heterogeneity in the data distribution comes in the form of skewed label partition similar to [9]. Decentralized learning with the DSGD algorithm on heterogeneous data distribution results in performance degradation due to huge variations in the local gradients across the agents. To tackle this, authors in [16] propose a momentum-based optimization technique (QG-DSGDm) introducing Quasi-Global momentum as shown in (3).

QG-DSGDm improves the performance of decentralized learning on heterogeneous data without any communication overhead and is used as a baseline for comparison in this work.

Gradient Tracking (GT) mechanisms [19, 22] are also known to improve decentralized learning on heterogeneous data by reducing the variance between the local gradient and the averaged (global) gradient. To achieve this, the gradient tracking algorithm introduces a tracking variable y t i that approximates the total gradient and is used to update the local parameters x t i (refer to (4)).

The update rule of tracking variable is such that it recursively adds a correction term (∑ j ∈N ( i ) w ij y t -1 j -g t -1 i ) to the local gradient g t i , pushing y t i to be closer to the global gradients ( 1 n ∑ n j =1 g t j ). This requires each agent i to communicate two sets of parameters x t i and y t i with its neighbors. Thus, the gradient tracking algorithm improves the decentralized learning on non-IID data at the cost of 2 × communication overhead.

3 Global Update Tracking

We present Global Update Tracking (GUT) , a novel algorithm for decentralized deep learning on non-IID data distribution. GUT is a communication-free tracking mechanism that aims to mitigate the difficulties of decentralized training when the data distributed across the agents is heterogeneous.

In order to attain the benefits of gradient tracking without communication overhead, we propose to apply the tracking mechanism with respect to the model updates x t i -x t -1 i instead of the gradients

Algorithm 1 Global Update Tracking ( GUT )

Input: Each agent i ∈ [1 , n ] initializes model parameters x 0 i and neighbors' copy ˆ x 0 j , step size η , GUT scaling factor µ , mixing matrix W = [ w ij ] i,j ∈ [1 ,n ] , N ( i ) represents neighbors of i including itself, and note ˆ x t i = x t i .

Each agent simultaneously implements the TRAIN( ) procedure

procedure TRAIN( )

for t = 0 , 1 , . . . , T -1 do

d t ∼ D

g t i . Firstly, to design a tracking mechanism without additional communication cost, each agent i communicates model updates instead of model parameters to its neighbors. An agent i stores a copy of its neighbor's parameters as ˆ x j and updates it using the received model updates to retrieve the current version of the neighbor's parameters as shown in line-9 of Algorithm 1. A memoryefficient implementation of the algorithm (Algorithm 4 in Appendix B) requires each agent to store s i = ∑ j ∈N ( i ) w ij ˆ x j instead of storing each neighbor's copy separately requiring only O (1) additional memory [12].

Now, we define a variable δ t i on each agent i that accumulates the local gradient update g t i and the gossip averaging update ∑ j ( w ij -I ij )ˆ x t j as shown in line-5 of Algorithm 1. Note that we can recover the DSGD update defined in (2) by using δ t i in the update rule i.e., x t +1 i = x t i -ηδ t i . We then proceed to compute the tracking variable y t i , as described in line-6 of Algorithm 1, using the combined model update (local gradient part and gossip averaging part) reflected by δ t i . The gossip averaging part of the update for each agent i i.e., ∑ j w ij (ˆ x t j -x t i ) is computed with respect to its own model weights. To account for this in the computation of tracking variable y t i , the agents have to adjust the information received from the neighbors (i.e., y t j 's) to change the reference to itself. This is reflected as an additional term 1 η (ˆ x t j -x t i ) in the update rule given by line-6 of Algorithm 1. Further, we scale the correction term of the tracking variable by a factor µ , a hyper-parameter, which is tuned to extract the maximum benefits of the proposed algorithm.

In summary, the update scheme of GUT can be re-formulated in the following matrix form where X = [ x 1 , . . . , x n ] ∈ R d × n are the model parameters and G = [ g 1 , . . . , g n ] ∈ R d × n are stochastic gradients.

Finally, we show that integrating the proposed GUT algorithm with Quasi-Global Momentum improves the current state-of-the-art significantly without any communication overhead. The pseudocode for the momentum version of our algorithm ( QG-GUTm ) is presented in Appendix B.

4 Convergence Guarantees

This section provides the convergence analysis for the proposed GUT Algorithm. We assume that the following standard assumptions hold:

Assumption 1 (Lipschitz Gradients) . Each function f i ( x ) is L-smooth i.e., ||∇ f i ( y ) -∇ f i ( x ) || ≤ L || y -x || .

Assumption 2 (Bounded Variance) . The stochastic gradients are unbiased and their variance is assumed to be bounded.

Assumption 3 (Doubly Stochastic Mixing Matrix) . The mixing matrix W is a real doubly stochastic matrix with λ 1 ( W ) = 1 and

where λ i ( W ) is the i th largest eigenvalue of W and ρ is the spectral gap.

The above assumptions are commonly used in most decentralized learning setups. Theorem 1 presents the convergence of the proposed GUT algorithm and the proof is detailed in Appendix A.

Theorem 1. (Convergence of GUT algorithm) Given Assumptions 1, 2, and 3 let step size η ≤ ρ 7 L and the scaling factor µ 1 -µ ≤ ρ 42 . For all T ≥ 1 , we have

where f (¯ x 0 ) -f ∗ is the sub-optimality gap, ¯ x is the average/consensus model parameters.

The result of the Theorem 1 shows that the averaged gradient of the averaged model is upperbounded by the sub-optimality gap (the difference between the initial objective function value and the optimal value), the sampling variance ( σ ), and gradient variations across the agents representing data heterogeneity ( ζ ). Further, we present a corollary to show the convergence rate of GUT in terms of the number of iterations.

Corollary 1. Suppose that the step size satisfies η = O ( √ n T ) For a sufficiently large T we have,

Corollary 1 indicates that the GUT algorithm achieves linear speedup with a convergence rate of O ( 1 √ nT ) when T is sufficiently large and is independent of communication topology. In other words, the communication complexity to find an ϵ -first order solution, i.e., E ∥∇ f (¯ x ) ∥ 2 ≤ ϵ is O ( σ 2 nϵ 2 ) . This convergence rate is similar to the well-known best result for decentralized SGD algorithms [15] in the literature.

5 Experiments

In this section, we analyze the performance of the proposed GUT and QG-GUTm techniques and compare them with the baseline DSGD algorithm [15] and the current state-of-the-art QG-DSGDm [16] respectively. The source code is available at https://github.com/aparna-aketi/ global_update_tracking

5.1 Experimental Setup

The efficiency of the proposed method is demonstrated through our experiments on a diverse set of datasets, model architectures, graph topologies, and graph sizes. We present the analysis on (a) Datasets: CIFAR-10, CIFAR-100, Fashion MNIST, and Imagenette. (b) Model architectures: VGG-11, ResNet-20, LeNet-5 and, MobileNet-V2. All the models use Evonorm [17] as the activationnormalization layer as it is shown to be better suited for decentralized learning on heterogeneous data.

(c) Graph topologies: Ring graph with 2 peers per agent, Dyck graph with 3 peers per agent, and Torus graph with 4 peers per agent (refer Figure 1). (d) Number of agents: 16-40 agents. We use the Dirichlet distribution to generate disjoint non-IID data across the agents. The created data partition across the agents is fixed, non-overlapping, and never shuffled across agents during the training. The degree of heterogeneity is regulated by the value of α - the smaller the α the larger the non-IIDness across the agents. We report the test accuracy of the consensus model averaged over three randomly chosen seeds. The details of the decentralized setup and hyperparameters for all the experiments are presented in Appendix C.

5.2 Average Consensus Task

We first consider an average consensus task that is isolated from the learning through stochastic gradient descent. Here the aim is that all the agents should reach a consensus which is the average value of the initial information each agent holds. The following equations show the simplified version of GUT (11) and QG-GUTm (12) after removing the gradient update part.

Note that setting the hyper-parameter µ as 0 in the (11), 12 gives simple gossip[26] and quasi-global gossip [16] respectively and all the agents communicate X t -X t -1 at iteration t with their neighbors.

Figure.2 shows the average consensus error i.e., 1 n || X t -¯ X || 2 F over time for the average consensus task on the Ring topology with respect to various algorithms. We observe that the gossip averaging with GUT converges faster than simple gossip averaging. Figure.2(c) illustrates that for graphs with a smaller spectral gap (which corresponds to more agents), the proposed QG-GUTm can converge faster than quasi-global gossip (gossip with QGM) resulting in better decentralized optimization.

0

250

500

750

1000

Iteration

10

3

10

1

10 1

10 3

Consensus Error

simple gossip

gossip with GUT

gossip with QGM

gossip with QG-GUTm

(a) 64 agents

0

375

750

1125

1500

Iteration

10

3

10

2

10

1

10 0

10 1

10 2

10 3

10 4

Consensus Error

simple gossip

gossip with GUT

gossip with QGM

gossip with QG-GUTm

(b) 128 agents

0

500

1000

1500

2000

Iteration

10

2

10

1

10 0

10 1

10 2

10 3

10 4

Consensus Error

simple gossip

gossip with GUT

gossip with QGM

gossip with QG-GUTm

(c) 256 agents

5.3 Decentralized Deep Learning Results

We evaluate the efficiency of GUT and its quasi-global momentum version QG-GUTm with the help of an exhaustive set of experiments. We compare GUT with DSGD and the momentum version QGGUTm with QG-DSGDm to show that the proposed method outperforms the current state-of-the-art.

Table. 1 shows the average test accuracy for training ResNet-20 and VGG-11 models on the CIFAR-10 dataset with varying degrees of non-IIDness over ring topology of 16 and 32 agents. We observe that GUT consistently outperforms DSGD for all models, graph sizes, and degree of heterogeneity with a significant performance gain varying from 1 -18% . The quasi-global momentum version of our algorithm, QG-GUTm , beats QG-DSGDm with 1 -3 . 5% improvement in the case of the CIFAR-10 dataset partitioned with a higher degree of heterogeneity ( α = 0 . 1 , 0 . 01 ).

Table 1: Test accuracy of different decentralized algorithms evaluated on CIFAR-10, distributed with different degrees of heterogeneity (non-IID) for various models over ring topologies. The results are averaged over three seeds where the standard deviation is indicated. We also include the results of the IID baseline as DSGDm (IID) where the local data is randomly partitioned independent of α .

We present the experimental results on various graph topologies and datasets to demonstrate the scalability and generalizability of QG-GUTm . We train the CIFAR-10 dataset on ResNet-20 over the Dyck graph and Torus graph to exemplify the impact of connectivity on the proposed technique. As shown in Table. 2, we obtain 0 . 5 -3 . 5% performance gains with varying connectivity (or spectral gap).

Further, we evaluate QG-GUTm on various image datasets such as Fashion MNIST, and Imagenette and on challenging datasets such as CIFAR-100. Table. 3 shows that QG-GUTm outperforms QGDSGDm by 0 . 2 -6 . 2% across various datasets. Therefore, in a decentralized deep learning setup, the proposed GUT and QG-GUTm algorithms are more robust to heterogeneity in the data distribution and can outperform all the comparison methods with an average improvement of 2% .

5.4 Ablation Study

First, we analyze different ways of utilizing or tracking model update information as shown in Table. 4. We present two different update rules apart from GUT and DSGD [15] and also compare them with gradient tracking [19]. Rule-a applies the proposed tracking mechanism on model updates but does not change the reference of tracking variable y j received from the neighbors to itself (refer sec. 3 for details on changing the reference). In the case of Rule-b, each agent computes the difference between the averaged neighborhood model update ( W ( X t -X t -1 ) ) along with its own model update ( X t -X t -1 ) and adds the difference between the two as a bias correction. Table. 4 shows that such naive ways of tracking or bias correction update rules (rule-a,b) do not improve the performance of decentralized learning on heterogeneous data. This confirms our findings that the GUT technique is an effective and provable way to track the consensus model and can outperform the gradient tracking mechanism without any communication overhead.

We then proceed to investigate the effect of different variants of momentum with GUT . From Table. 5 (refer to Appendix D for more results), we can conclude that the quasi-global variant of Global Update Tracking always surpasses the other methods. This indicates that the proposed GUT algorithm accelerates decentralized optimization and can be used in synergy with quasi-global momentum to achieve maximal performance gains.

Furthermore, Figure 3(a) illustrates the effect of scaling µ on the test accuracy with QG-GUTm and note that µ = 0 shows the test accuracy for QG-DSGDm. Figure 3(b), 3(c) showcase the scalability

of QG-GUTm on different graph sizes and model sizes. QG-GUTm outperforms QG-DSGDm by ∼ 1 . 7% over different graph sizes and ∼ 1 . 4% over different model sizes.

0.00

0.02

0.04

0.06

0.08

Scaling value (

)

76

78

80

82

84

86

88

90

Test Accuracy

skew(

)=0.1

skew(

)=0.01

(a)

n

= 16

, ResNet-20

16

24

32

40

Number of agents (n)

80

81

82

83

84

85

86

87

88

Test Accuracy

QG-GUTm

QG-DSGDm

(b)

α

= 0

.

1

, ResNet-20

20

32

44

56

Depth of ResNet

82

83

84

85

86

87

88

89

90

Test Accuracy

QG-GUTm

QG-DSGDm

(c)

α

= 0

.

1

,

n

= 16

6 Discussion and Limitations

Wedemonstrated the superiority of the Global Update Tracking ( GUT ) algorithm through an elaborate set of experiments and ablation studies. In our experiments, we focused on doubly-stochastic and symmetric graph structures. The proposed GUT algorithm can be easily extended to directed and time-varying graphs by combining it with stochastic gradient push (SGP) [2]. Further, the additional terms added by GUT can also be interpreted as a bias correction mechanism where the added bias pushes the local model towards the consensus (averaged) model. The matrix representation of this interpretation of GUT is given by (13). and analyzed in Lemma 1.

Lemma 1. Given assumptions 3, we define ¯ b t = B t 1 n ✶✶ T , where ✶ is a vector of all ones. For all t , we have: ¯ b t = µ ¯ b t -1 .

A complete proof for Lemma 1 can be found in Appendix A.2. Lemma 1 highlights that the average bias added by GUT is zero as ¯ b 0 is zero. Hence, the GUT algorithm crucially preserves the average value of the decentralized system. A feature we leverage to establish Theorem 1.

There are two potential limitations of the GUT algorithm - a) memory overhead and b) introduction of an additional hyper-parameter. GUT requires the agents to keep a copy of the averaged model parameters of their neighbors which adds an extra memory buffer of the size of model parameters. The storage of the tracking variable also adds to the memory overhead, requiring additional memory equivalent to the size of model parameters. We also introduce a new hyper-parameter µ which has to be tuned similarly to the learning rate or momentum coefficient tuning. Besides, the theoretical analysis presented for the GUT algorithm does not consider momentum and assumes the communication to be synchronous. We leave the theoretical analysis of QG-GUTm and formulation of the asynchronous version of GUT as a future research direction.

7 Conclusion

Decentralized learning on heterogeneous data is the key to launching ML training on edge devices and thereby efficiently leveraging the humongous amounts of user-generated private data. In this paper, we propose Global Update Tracking ( GUT ), a novel decentralized algorithm designed to improve learning over heterogeneous data distributions. The convergence analysis presented in the paper shows that the proposed algorithm matches the best-known rate for decentralized algorithms. Additionally, the paper introduces a quasi-global momentum version of the algorithm, QG-GUTm, to further enhance the performance gains. The empirical evidence from experiments on different model architectures, datasets, and topologies demonstrates the superior performance of both algorithms. In summary, the proposed algorithm and its quasi-global momentum version have the potential to facilitate more scalable and efficient decentralized learning on edge devices.

References

Sai Aparna Aketi, Sangamesh Kodge, and Kaushik Roy. Neighborhood gradient clustering: An efficient decentralized learning method for non-iid data distributions. arXiv preprint arXiv:2209.14390 , 2022.

Mahmoud Assran, Nicolas Loizou, Nicolas Ballas, and Mike Rabbat. Stochastic gradient push for distributed deep learning. In International Conference on Machine Learning , pages 344-353. PMLR, 2019.

Aditya Balu, Zhanhong Jiang, Sin Yong Tan, Chinmay Hedge, Young M Lee, and Soumik Sarkar. Decentralized deep learning using momentum-accelerated consensus. In ICASSP 20212021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 3675-3679. IEEE, 2021.

Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT'2010 , pages 177-186. Springer, 2010.

Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc'aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, et al. Large scale distributed deep networks. Advances in neural information processing systems , 25, 2012.

Paolo Di Lorenzo and Gesualdo Scutari. Next: In-network nonconvex optimization. IEEE Transactions on Signal and Information Processing over Networks , 2(2):120-136, 2016.

Yasaman Esfandiari, Sin Yong Tan, Zhanhong Jiang, Aditya Balu, Ethan Herron, Chinmay Hegde, and Soumik Sarkar. Cross-gradient aggregation for decentralized learning from non-iid data. In International Conference on Machine Learning , pages 3036-3046. PMLR, 2021.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip Gibbons. The non-IID data quagmire of decentralized machine learning. In Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 43874398. PMLR, 13-18 Jul 2020.

Hamel Husain. Imagenette - a subset of 10 easily classified classes from the imagenet dataset. https://github.com/fastai/imagenette , 2018.

Anastasia Koloskova, Nicolas Loizou, Sadra Boreiri, Martin Jaggi, and Sebastian Stich. A unified theory of decentralized sgd with changing topology and local updates. In International Conference on Machine Learning , pages 5381-5393. PMLR, 2020.

Anastasia Koloskova, Sebastian Stich, and Martin Jaggi. Decentralized stochastic optimization and gossip algorithms with compressed communication. In Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 3478-3487. PMLR, 09-15 Jun 2019.

Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar (canadian institute for advanced research). http://www.cs.toronto.edu/ kriz/cifar.html , 2014.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsieh, Wei Zhang, and Ji Liu. Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent. Advances in Neural Information Processing Systems , 30, 2017.

Tao Lin, Sai Praneeth Karimireddy, Sebastian Stich, and Martin Jaggi. Quasi-global momentum: Accelerating decentralized deep learning on heterogeneous data. In Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 6654-6665. PMLR, 18-24 Jul 2021.

Hanxiao Liu, Andy Brock, Karen Simonyan, and Quoc Le. Evolving normalization-activation layers. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances

in Neural Information Processing Systems , volume 33, pages 13539-13550. Curran Associates, Inc., 2020.

Angelia Nedic. Distributed gradient methods for convex machine learning problems in networks: Distributed optimization. IEEE Signal Processing Magazine , 37(3):92-101, 2020.

Shi Pu and Angelia Nedi´ c. Distributed stochastic gradient tracking methods. Mathematical Programming , 187:409-457, 2021.

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4510-4520, 2018.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

Yuki Takezawa, Han Bao, Kenta Niwa, Ryoma Sato, and Makoto Yamada. Momentum tracking: Momentum acceleration for decentralized deep learning on heterogeneous data. arXiv preprint arXiv:2209.15505 , 2022.

Hanlin Tang, Xiangru Lian, Ming Yan, Ce Zhang, and Ji Liu. d 2 : Decentralized training over decentralized data. In International Conference on Machine Learning , pages 4848-4856. PMLR, 2018.

Thijs Vogels, Lie He, Anastasiia Koloskova, Sai Praneeth Karimireddy, Tao Lin, Sebastian U Stich, and Martin Jaggi. Relaysum for decentralized deep learning on heterogeneous data. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 28004-28015. Curran Associates, Inc., 2021.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017.

Lin Xiao and Stephen Boyd. Fast linear iterations for distributed averaging. Systems & Control Letters , 53(1):65-78, 2004.

A Convergence Rate Proof

In this work, we solve the optimization problem of minimizing global loss function f ( x ) distributed across n agents as given below. Note that F i is a local loss function (for example, cross-entropy loss) defined in terms of the data sampled ( d i ) from the local dataset D i at agent i .

We reiterate the update scheme of GUT presented in Algorithm. 1 in a matrix form:

where W is the mixing matrix, I is the identity matrix, X = [ x 1 , x 2 , . . . , x n ] ∈ R d × n is the matrix containing model parameters, x i ∈ R d is model parameters of agent i , Y = [ y 1 , y 2 , . . . , y n ] ∈ R d × n is the matrix containing tracking variables, G = [ g 1 , g 2 , . . . , g n ] ∈ R d × n is the matrix containing local gradients, µ is the GUT scaling factor, η is the learning rate. Now, we rewrite the above equation in the form of a bias correction update,

A.1 Assumptions

We assume that the following statements hold:

Assumption 1 - Lipschitz Gradients: Each function f i ( x ) is L-smooth i.e., ||∇ f i ( y ) -∇ f i ( x ) || ≤ L || y -x || . Equivalently,

Assumption 2 - Bounded Variance: The variance of the stochastic gradients is assumed to be bounded.

Assumption 3 - Doubly Stochastic Mixing Matrix: The mixing matrix W is a real doubly stochastic matrix with λ 1 ( W ) = 1 and

where λ i ( W ) is the i th largest eigenvalue of W and ρ is the spectral gap. The mixing matrix satisfies E W || ZW -¯ Z || 2 F ≤ (1 -ρ ) || ZW -¯ Z || 2 F , where ¯ Z = Z 1 n ✶✶ T . We also have W ✶ = ✶ and W T = .

✶ ✶ Further, we define the average gradients ¯ g t = 1 n ∑ n i =1 ∇ F i ( x t i , d t i ) where d t i is sampled mini-batch of data on node i

A.2 Proof of Lemma 1

Lemma 1: Given assumptions 3, we define ¯ b t = B t 1 n ✶✶ T , where ✶ is a vector of all ones. For all t , we have: ¯ b t = µ ¯ b t -1 .

Proof. Starting from the definition of B t

Given that ¯ b 0 = 0 , the average bias is zero at each iteration. This indicates that the proposed algorithm GUT preserves the average of the system in an average consensus task.

A.3 Proof of Theorem 1

This section presents the detailed proof of the convergence bounds of GUT algorithm given by Theorem 1. Firstly, we analyze the one-step progress of the averaged model parameters ¯ x . Note that, ¯ X t = [¯ x t , ¯ x t , . . . , ¯ x t ] ∈ R d × n and ¯ x t = 1 n ∑ n i =1 x t i

Lemma 2. Given assumptions 1-3 and η ≤ , we have

Proof. From the definition of X t +1 , we have

using L-smoothness assumption given by (16)

(a) uses assumption-2 ((6)). (b) uses the fact that -2 ⟨ a, b ⟩ = -|| a || 2 - || b || 2 + || a -b || 2 . (c) uses Jensen's inequality. (d) uses L-smoothness condition. (e) follows from the assumption that η ≤ 1 4 L .

Now, we proceed to bound the consensus error through Lemma 3.

Lemma 3. Given assumptions 1-3 and η ≤ ρ , we have

Proof. Starting from the update step 15

(a) follows from the fact that || a + b || 2 ≤ (1 + α ) || a || 2 +(1 + 1 α ) || b || 2 ∀ α > 0 and let α = ρ 2 . (b) uses E W || ZW -¯ Z || 2 F ≤ (1 -ρ ) || ZW -¯ Z || 2 F and 1 + 2 ρ ≤ 3 ρ . (c) uses assumption-2 (7). (d) uses L-smoothness condition. (e) follows from the assumption that η ≤ ρ 7 L

The next step is to find an upper bound for the bias term E || B t || 2 F .

Lemma 4. Given assumptions 1-3 and µ ≤ ρ

Proof. starting from the update step 15

Now,

Multiplying both sides with 6 η 2 µ 2 ρ (1 -µ )

Note that W -I < I , I -W < 2 I , ( W -I ) ¯ X t = 0 and ( W -I ) ¯ G t = 0 . (a) follows from the fact that || a + b || 2 ≤ (1 + α ) || a || 2 +(1 + 1 α ) || b || 2 ∀ α > 0 and let α = 1 -µ µ . (b) uses the fact that || AB || 2 F ≤ σ 2 max ( A ) || B || 2 F where A = 2 W + I , B = ( I -W ) X t and σ 2 max ( A ) = 9 . (c) uses the assumption µ 1 -µ ≤ ρ 42 and η ≤ ρ 7 L . This implies that 24(9+ η 2 L 2 ) µ 2 ρ (1 -µ ) 2 ≤ ρ 8 and 48 µ 2 ρ (1 -µ ) 2 ≤ ρ 8

We present the proof for Theorem 1 using Lemmas. 2, 3, and 4. Adding Lemmas. 3, and 4 and simplifying, we get

Finally, define a function Φ t as shown below

Now adding Lemma 2 and (17), we have the following

Summing over t

This concludes the proof of the Theorem 1.

A.4 Proof of Corollary. 1

In the proof of Theorem 1, we assumed the following the following constraints on learning rate η and scaling factor µ :

We assume that the step size η is O ( √ n T ) , where n is the total number of agents and T is the number of iterations. Given this assumption, we have the following order of each term in (19) of Theorem 1.

For the remaining terms we have,

Therefore, by omitting the constant n in this context of higher order terms, there exists a constant C > 0 such that the overall convergence rate is as follows:

which suggests when T is sufficiently large, GUT enables the convergence rate of O ( 1 √ nT ) .

B Algorithmic details

In this section, we present the pseudo-code for QG-GUTm which combines the proposed GUT algorithm with quasi-global momentum as Algorithm 2 and its PyTorch implementation version is shown in Algorithm 3. We also summarize the memory-efficient implementation of GUT in Algorithm 4.

Algorithm 2 Global Update Tracking with momentum ( QG-GUTm )

Input: Each agent i ∈ [1 , n ] initializes model weights x (0) i and neighbors' copy ˆ x (0) j , momentum buffer m (0) i , step size η , momentum coefficient β , mixing matrix W = [ w ij ] i,j ∈ [1 ,n ] , GUT scaling factor µ , I ij are elements of n × n identity matrix, N ( i ) represents neighbors of i including itself, and note ˆ x t i = x t i .

Each agent simultaneously implements the TRAIN( ) procedure

procedure TRAIN( )

for t= 0 , 1 , . . . , T -1 do

d t i ∼ D i

x t +1 i = x t i -ηm t i

SENDRECEIVE( m t i )

end

return

Algorithm 3 Global Update Tracking with momentum ( QG-GUTm ) - Pytorch Implementation

Input: Each agent i ∈ [1 , n ] initializes model weights x (0) i and neighbors' copy ˆ x (0) j , momentum buffer m (0) i , step size η , momentum coefficient β , mixing matrix W = [ w ij ] i,j ∈ [1 ,n ] , GUT scaling factor µ , I ij are elements of n × n identity matrix, N ( i ) represents neighbors of i including itself, and note ˆ x t i = x t i .

Each agent simultaneously implements the TRAIN( ) procedure

procedure TRAIN( )

for t= 0 , 1 , . . . , T -1 do

d t i ∼ D i

m t i = βm t -1 i + y t i

x t +1 i = x t i -ηm t i

SENDRECEIVE( m t i )

end

return

Algorithm 4 Global Update Tracking (Memory Efficient Implementation)

Input: Each agent i ∈ [1 , n ] initializes model parameters x 0 i and weighted model parameters of neighborhood s 0 i , step size η , GUT scaling factor µ , mixing matrix W = [ w ij ] i,j ∈ [1 ,n ] , N ( i ) represents neighbors of i including itself.

Each agent simultaneously implements the TRAIN( ) procedure

C Decentralized Learning Setup

For the decentralized setup, we use an undirected ring, undirected Dyck graph, and undirected torus graph topologies with a uniform mixing matrix. The undirected ring topology for any graph size has 3 peers per agent including itself and each edge has a weight of 1 3 . The undirected Dyck topology with 32 agents has 4 peers per agent including itself and each edge has a weight of 1 4 . The undirected torus topology with 32 agents has 5 peers per agent including itself and each edge has a weight of 1 5 . All our experiments were conducted on a system with Nvidia GTX 1080ti card with 4 GPUs except for ImageNette simulations. We used NVIDIA A40 card with 4 GPUs for ImageNette simulations.

C.1 Datasets

In this section, we give a brief description of the datasets used in our experiments. We use a diverse set of datasets each originating from a different distribution of images to show the generalizability of the proposed techniques.

CIFAR-10: CIFAR-10 [13] is an image classification dataset with 10 classes. The image samples are colored (3 input channels) and have a resolution of 32 × 32 . There are 50 , 000 training samples with 5000 samples per class and 10 , 000 test samples with 1000 samples per class.

CIFAR-100: CIFAR-100 [13] is an image classification dataset with 100 classes. The image samples are colored (3 input channels) and have a resolution of 32 × 32 . There are 50 , 000 training samples with 500 samples per class and 10 , 000 test samples with 100 samples per class. CIFAR-100 classification is a harder task compared to CIFAR-10 as it has 100 classes with very few samples per class to learn from.

Fashion MNIST: Fashion MNIST [25] is an image classification dataset with 10 classes. The image samples are in greyscale (1 input channel) and have a resolution of 28 × 28 . There are 60 , 000 training samples with 6000 samples per class and 10 , 000 test samples with 1000 samples per class.

Imagenette: Imagenette [10] is a 10-class subset of the ImageNet dataset. The image samples are colored (3 input channels) and have a resolution of 224 × 224 . There are 9469 training samples with roughly 950 samples per class and 3925 test samples.

C.2 Network Architecture

We replace ReLU+BatchNorm layers of all the model architectures with EvoNorm-S0 as it was shown to be better suited for decentralized learning over non-IID distributions.

VGG-11: We modify the standard VGG-11 [21] architecture by reducing the number of filters in each convolutional layer by 4 × and using only one dense layer with 128 units. Each convolutional layer is followed by EvoNorm-S0 as the activation-normalization layer. VGG-11 has 0 . 58 M trainable parameters.

ResNet-20: For ResNet-20 [8], we use the standard architecture with 0 . 27 M trainable parameters except that BatchNorm+ReLU layers are replaced by EvoNorm-S0.

LeNet-5: For LeNet-5 [14], we use the standard architecture with 61 , 706 trainable parameters.

MobileNet-V2: We use the the standard MobileNet-V2 [20] architecture used for CIFAR dataset with 2 . 3 M parameters except that BatchNorm+ReLU layers are replaced by EvoNorm-S0.

C.3 Hyper-parameters

This section presents a detailed description of the hyper-parameters used in our experiments. All the experiments were run for three randomly chosen seeds. We decay the step size by 10x after 50% and 75% of the training, unless mentioned otherwise. The hyper-parameter µ is set to 0.9 for all the experiments using GUT optimizer. We used grid search to choose the hyper-parameter µ for QG-GUTm .

Hyper-parameters for experiments in Table 1: All the experiments have the stopping criteria set to 200 epochs. The initial learning rate is set to 0.1. We decay the step size by 10 × in multiple steps at 100 th and 150 th epoch. Table 6 presents values of the scaling factor µ used in the experiments. For all the experiments, we use a mini-batch size of 32 per agent. The stopping criteria is a fixed number of epochs. We have used a momentum of 0.9 for all QG-DSDm and QG-GUTm experiments.

Hyper-parameters for experiments in Table 2: All the experiments have the stopping criteria set to 200 epochs. The initial learning rate is set to 0.1. We decay the step size by 10 × in multiple steps at 100 th and 150 th epoch. Table 7 presents values of the scaling factor µ used in the experiments. For all the experiments, we use a mini-batch size of 32 per agent. The stopping criteria is a fixed number of epochs. For all QG-DSDm and QG-GUTm experiments, we have used a momentum of 0.9 and Nesterov is set to False. We did not use any regularization in our experiments on non-IID data i.e., weight decay is set to zero. We set the weight decay to be 1 e -4 for experiments on IID data (DSGDm case in Table. 1).

Hyper-parameters for experiments in Table 3: All the experiments with Fashion-MNIST and Imagenette datasets have the stopping criteria set to 100 epochs where as CIFAR-100 experiments

have the stopping criteria as 200 epochs. The initial learning rate is set to 0.1 for experiments on CIFAR-100 and Fashion MNIST datasets. The initial learning rate is set to 0.01 for experiments on the Imagenette dataset. We decay the step size by 10 × in multiple steps at 50 th and 75 th epoch. Table 8 presents values of the scaling factor µ used in the experiments. For all the experiments, we use a mini-batch size of 32 per agent. The stopping criteria is a fixed number of epochs. We have used a momentum of 0.9 for all QG-DSDm and QG-GUTm experiments.

D Additional Results

Table. 9 shows that the quasi-global variant of Global Update Tracking surpasses the other existing methods even for a higher degree of heterogeneity i.e., α = 0 . 01 . We also observe that the Nesterov momentum hurts the performance when the heterogeneity in the data distribution is high. Table. 10 compares the DSGDm baseline with the proposed GUT and QG-GUTm . It shows that for a higher degree of heterogeneity, GUT outperforms even the momentum version of DSGD. Table. 11 evaluates the proposed method on various datasets trained on same model architecture (ResNet-20).

E Hardware Efficiency

In this section, we present the quantitative results on communication, memory, and compute overheads of the various decentralized algorithms. The communication cost incurred by the proposed GUT

and QG-GUTm methodologies is the same as the DSGD or QG-DSGDm techniques. All the abovementioned algorithms communicate a vector with the size same as the model parameters. However, gradient tracking incurs 2 × communication cost in terms of model size. The communication cost and memory requirements for all the experiments are reported in Table. 12 and 13.

We report the numbers for communication, memory, and compute overheads for the memory-efficient implementation of GUT Algorithm. 4 in Table. 14. Memory overhead is reported as the percentage of additional memory required per agent during training with a batch size of 32.

The total memory includes the memory required to store model parameters, activations, gradients, gossip buffer, tracking variable ( y i ), and weighted neighbors' parameters ( s i ). We observe that for compact models such as ResNet and MobileNet, the memory overhead is less than 2% . However, for larger models such as VGG-11, the memory overhead shoots up to 14% . The computational overhead is reported as the percentage of additional FLOPs required per sample per agent during training.

The total compute includes the forward pass, backward pass, model updates, gossip averaging, and tracking variable computation flops. We observe that for compact models such as ResNet and MobileNet, the compute overhead is around 2% . However, for larger models such as VGG-11, the compute overhead shoots up to 15% .

Peer reviews:
comments: SUMMARY:
This work proposes a novel decentralized learning algorithm based on gradient tracking mechanism, called Global Update Tracking (GUT), which aims to mitigate the impact of heterogeneous data distribution. The proposed GUT algorithm overcomes the bottleneck of communication overhead by allowing agents to store a copy of their neighbors’ model parameters and then tracking the model updates instead of the gradients. Numerous experiments have proven that the proposed Global Update Tracking with Quasi-Global momentum (QG-GUTm) outperforms the current state-of-the-art decentralized learning algorithm on a spectrum of heterogeneous data.

STRENGTHS:
1.	This paper proposes a powerful global update tracking method and achieves good performance on multiple benchmarks.
2.	The authors provide a detailed sensitivity study of the method's performance with respect to model architecture and hyper-parameters.
3.	The paper is well written and easy for the reader to read and understand.

WEAKNESSES:
1.	In the experimental part, the quantitative experiments about communication parameters(cost) are not clear enough. Can the authors provide specific quantitative results for comparison so that readers can easily understand them?

QUESTIONS:
Please refer to the weaknesses section.

LIMITATIONS:
This paper does not have any specific societal impact, other than the ones associated with other algorithms.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe performance of decentralized learning is limited to the different distribution over devices. To address this issue, this paper proposes a method that is less susceptible to variations in data distribution, named GUT. The proposed GUT tracks the global/average model updates. Experiments show GUT achieves good performance on some benchmarks with various models. The paper also analyzes the convergence of the proposed method.\n\nSTRENGTHS:\n1) The issue of heterogeneous data on devices in decentralized learning is critical, the paper shows a good motivation to address this.\n2) On the given benchmarks and models, the proposed method seems to perform well.\n3) The analysis of convergence of the method is detailed.\n\nWEAKNESSES:\n1) There are too many descriptions of related work in the introduction, making it difficult for readers to find the gist.\n\n2) The analysis of method strengths is not clear enough. Why the proposed method is better than previous methods? Why tracking the model updates instead of the gradients can save twice the communication overhead? Can differences in the data distribution of all agents be mitigated by communicating with neighboring agents?\n\n3) References to employed benchmarks (CIFAR-10, CIFAR-100, Fashion MNIST, and Imagenette) and models (VGG-11, ResNet-20, LeNet-5 and, MobileNet-V2) are missed.\n\n4) The evaluated benchmarks are relatively small. More large-scale benchmarks should be supplemented, such as ImageNet.\n\n5) Some larger and state-of-the-art models should be evaluated to verify the effectiveness of the proposed method, such as ResNet-50, ResNet-100, ViT.\n\n6) The evaluations on Fashion MNIST, CIFAR-100, Imagenette employ different model. The models should be unified. \n\n7) Since one of the main contributions of the method is to reduce overhead, the article lacks a quantitative analysis of overhead.\n\nQUESTIONS:\nIn rebuttal, the reviewer hope to see the response to 2),4)-7) in weaknesses.\n\nLIMITATIONS:\nThe paper provides a compelling discussion about the limitation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new decentralized learning algorithm called Global Update Tracking (GUT) that can mitigate the impact of heterogeneous data in decentralized learning without introducing any communication overhead. The proposal was demonstrated 4 computer vision datasets.\n\nSTRENGTHS:\nThe paper proposes an important and useful application for training deep learning models in decentralized way. There are many strengths of the proposal including the following: \n\n1. GUT is less susceptible to variations in data distribution across devices as shown by different experiments and ablation study\n\n2. The performance is better than DSGD and DSGDm. However, it is shown only for CIFAR-10.\n\nWEAKNESSES:\nThere are several questions: \n\n1. The paper claims that GUT does not introduce any communication overhead but no results are shown to justify that. However using Figure 2, authors have claimed that their method converges faster than quasi global gossip. The comparison/improvement is not clear from the figure. \n\n2. The author claim that agents in GUT store copy of their neighbors. How often this copy is updated? What is the communication overhead in that?\n\n3. Performance comparison with related methods is not shown on all 4 datasets.\n\n4. The data partition is fixed, non-overlapping and never shuffled across agents during training. How does this partition is defined?\n\nQUESTIONS:\nPlease see weaknesses.\n\nLIMITATIONS:\nThere are several limitations that the authors should address: \n\n1. The data partition across agents is not clearly explained. Necessary details for all datasets is required. For example, number of samples per class per agent. Implementation details are also missing from the main paper.\n\n2. Detailed comparison with the related methods is missing. For example, Table 3 compare performance on 3 datasets but no related method is shown.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an algorithm for decentralized learning, where training datasets with heterogenous distributions are collected across different devices. They propose a global update tracking (GUT)-based approach where the IID assumption for the data is removed, and aim to generalize their method to non-IID data by tracking the consensus model while reducing the communication overhead by their GUT algorithm, at the expense of additional memory for storing the parameters for the neighboring model. They validate their method by experiments on various  datasets.\n\nSTRENGTHS:\n- Proposed method is applicable to more realistic settings for decentralized learning, where non-IID heterogeneous distribution is more common for the real world data.\n\n- Their GUT-based approach adds minimal computational overhead on the overall agents, and they show the performance of their algorithm on multiple datasets in the experimental results.\n\n- They provide ablation studies for further analysis on the hyperparameter sensitivity, different levels of heterogeneity, and various implementations of momentum.\n\n- Authors provide the source code for their implementation in the supplementary materials.\n\nWEAKNESSES:\n- Since one of the main claims of this approach is its generalizability to non-IID heterogeneous data distributions for each agent, what aspect in the proposed formulation corrects the gap between the individual distributions?\n\n- Compared to previous decentralized learning approaches where the agent tracks gradients, momentum of gradients, or gloabl gradient obtained from the adjacent agents, proposed GUT tracks model updates. What statistical characteristic/effect makes the model update ($x^{t+1}_i - x^{t}_i$) beneficial for tracking? A simple toy (e.g. Gaussian) example experiment with update with/without model update tracking would suffice.\n\n- Are there any visualizations for non-IID characteristic on the datasets under varying degree of $\\alpha$? Also, are there experimetal results of additional memory consumption and computational overhead due to the additional update terms?\n\nQUESTIONS:\nPlease refer to the questions in the weaknesses section. Since I am not an expert in this field, my positive and negative opinions are mixed.\n\nAdditionally, repeating the full paper in the supplementary materials seems unnecessary.\n\nLIMITATIONS:\nThe authors included a separate discussions and limitations section in their paper, with adequate explanations and descriptions on the weaknesses and possible future directions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}