Momentum Tracking: Momentum Acceleration for Decentralized Deep Learning on Heterogeneous Data
Authors: Yuki Takezawa
Han Bao
Kenta Niwa
Ryoma Sato
Makoto Yamada
Venue: Submitted to ICLR 2023 (2023)
Keywords: Decentralized Optimization
Non-Convex Stochastic Optimization
Momentum Acceleration

Abstract: SGD with momentum acceleration is one of the key components for improving the performance of neural networks. For decentralized learning, a straightforward approach using momentum acceleration is Distributed SGD (DSGD) with momentum acceleration (DSGDm). However, DSGDm performs worse than DSGD when the data distributions are statistically heterogeneous. Recently, several studies have addressed this issue and proposed methods with momentum acceleration that are more robust to data heterogeneity than DSGDm, although their convergence rates remain dependent on data heterogeneity and decrease when the data distributions are heterogeneous. In this study, we propose Momentum Tracking, which is a method with momentum acceleration whose convergence rate is proven to be independent of data heterogeneity. More specifically, we analyze the convergence rate of Momentum Tracking in the standard deep learning setting, where the objective function is non-convex and the stochastic gradient is used. Then, we identify that it is independent of data heterogeneity for any momentum coefficient $\beta \in [0, 1)$. Through image classification tasks, we demonstrate that Momentum Tracking is more robust to data heterogeneity than the existing decentralized learning methods with momentum acceleration and can consistently outperform these existing methods when the data distributions are heterogeneous.

Under review as a conference paper at ICLR 2023
MOMENTUM TRACKING: MOMENTUM ACCELERA- TION FOR DECENTRALIZED DEEP LEARNING ON HET- EROGENEOUS DATA
Anonymous authors Paper under double-bling blind review
ABSTRACT
SGD with momentum acceleration is one of the key components for improving the performance of neural networks. For decentralized learning, straightforward approach using momentum acceleration is Distributed SGD (DSGD) with mo- mentum acceleration (DSGDm). However, DSGDm performs worse than DSGD when the data distributions are statistically heterogeneous. Recently, several stud- ies have addressed this issue and proposed methods with momentum acceleration that are more robust to data heterogeneity than DSGDm, although their conver- gence rates remain dependent on data heterogeneity and decrease when the data distributions are heterogeneous. In this study, we propose Momentum Tracking, which method with momentum acceleration whose convergence rate is proven to be independent of data heterogeneity. More specifically, we analyze the con- vergence rate of Momentum Tracking the standard deep learning setting, where the objective function is non-convex and the stochastic gradient is used. Then, we identify that it is independent of data heterogeneity for any momentum coefficient 3 E [0, 1). Through image classification tasks, demonstrate that Momen- tum Tracking is more robust to data heterogeneity than the existing decentralized learning methods with momentum acceleration and consistently outperform these existing methods when the data distributions are heterogeneous.
INTRODUCTION
Neural networks have achieved remarkable success in various fields such as image processing (Si-) monyan & Zisserman 2015 Chen et al. (0200) and natural language processing (Devlin et al. 2019). To train neural networks, we need collect large amounts of training data. However, because of privacy concerns, is often difficult to collect large amounts data such as medical images on one server. In such scenarios, decentralized learning has attracted significant attention because it allows us to train neural networks without aggregating all the data onto one server. Recently, decentral- ized learning has been studied from various perspectives, including data heterogeneity (Tang al., 2018b 8b] Esfandiari et al. [222)) communication compression (Tang et al. 20 8a Lu & De Sa 2020 Liu et al. [021] Takezawa al. 2022a). and network topologies (Ying et al. 2021).
One of the key components for improving the performance of neural networks is SGD with momen- tum acceleration (SGDm). Whereas SGD updates the model parameters using a stochastic gradient, SGDm updates the model parameters using the moving average of the stochastic gradient, which called the momentum. Because SGDm can accelerate convergence and improve generalization performance, SGDm has become indispensable tool, enabling neural networks to achieve high accuracy (He al. 2016). Recently, SGDm been improved many studies, and methods such Adam (Kngmma & Ba 2015) and RAdam (Liu al. 2020a) have been proposed.
In decentralized learning, the straightforward approach to using the momentum is Distributed SGD (DSGD) with momentum acceleration (DSGDm) (Gao & Huang 2020). When the data distributions held by each node (i.e., the server) are statistically homogeneous, DSGDm works well and can improve the performance as well as SGDm (Lin al. 2021). However, in real-world decentralized learning settings, the data distributions may be heterogeneous (Hsich et al., 2020). such cases, DSGDm performs worse than DSGD (i.e., without momentum acceleration) Yuan et al.) 2021).

---

Under review as a conference paper at ICLR 2023
Table I: Comparison of the convergence rates. In the "Data--Heterogeeeitt"" column, "v" indicates that the convergence rate is independent of data heterogeneity, and "(v)" indicates that it is inde- pendent, but there is discussion about data heterogeneity either theoretically experimentally. In the "Momentum," "Stochastic," " and "Non-Convex" columns, "V" respectively indicates that the method is accelerated using momentum, the convergence rate provided when the stochastic gra- dient is used, and the convergence rate is provided when the objective function non-convex.
Data-Heierogeneiity
Momentum
Stochastic
Non-Convex
DSGD Lian et al. 2017
Gradient Tracking (Knloskova al.
2021
DSGDm (Cao & Huang
2020
QG-DSGDm (Lin cl al.
202T
DecentLaM (Yuan et al.
2021
ABm (inn & Khan 2020)
<33
GTAdam (Crnevalc el al. 2022
Momentum Tracking (our work)
This is because, when the data distributions are heterogeneous and we use the momentum instead of the stochastic gradient, each model parameter is updated in further different directions and drifts away more easily. As a result, the convergence rate of DSGDm falls below that of DSGD. To address this issue, [in et al. (202T) and Yuan al. (20T)) modified the update rules of the momentum in DSGDm and proposed methods that are more robust to data heterogeneity than DSGDm. However, their convergence rates remain dependent on data heterogeneity, and our experiments revealed that their performance are degraded when the data distributions are strongly heterogeneous (Sec. 4.
Data heterogeneity for decentralized learning has been well studied from both experimental and theoretical perspectives (Hsich al., 2020) Koloskova et al., (220)). Subsequently, many methods including Gradient Tracking (Lorenzo & Scutari 2016 6, Nedić et al. 2017) have been proposed and it has been shown that their convergence rates do not depend data heterogeneity (Tang et al. 2018b Vogels et al., [2021] Koloskova et al., (021). However, these studies considered only the case not used, and remains unclear whether these methods are robust to data
where the momentum heterogeneity when the momentum is applied.
In the convex optimization literature, Xin & Khan (220) Carnevae al. 2022) proposed com- bining Gradient Tracking with momentum or Adam and analyzed the convergence rates. However, they considered only the case where the objective function strongly convex and the full gradient is used, which does not hold in the standard deep learning setting, where the objective function is non-convex and only the stochastic gradient is accessible. Hence, their convergence rates are still unknown in standard deep learning settings, and it remains unclear whether their convergence rates are independent of data heterogeneity. Furthermore, they did not discuss data heterogeneity, either theoretically or experimentally.
In this work, we propose decentralized learning method with momentum acceleration, which we call Momentum Tracking, whose convergence rate is proven to be independent of data hetero- geneity in the standard deep learning setting. More specifically, we provide the convergence rate of Momentum Tracking in a setting in which the objective function is non-convex and the stochastic gradient is used. Then, we identify that the convergence rate of Momentum Tracking is independent of data heterogeneity for any momentum coefficient 3 E [0. 1). In Table D compare the conver- gence rate of Momentum Tracking with those of existing methods. To the best of our knowledge, Momentum Tracking is the first decentralized learning method with momentum acceleration whose convergence rate has been proven to be independent of data heterogeneity in the standard deep learning setting. Experimentally, we demonstrate that Momentum Tracking is more robust to data heterogeneity than the existing decentralized learning methods with momentum acceleration and can consistently outperform these existing methods when the data distributions are heterogeneous.
PRELIMINARIES AND RELATED WORK
2.1 DECENTRALIZED LEARNING
Let G E) be an undirected graph that represents the underlying network topology, where V denotes the set of nodes and E denotes the set of edges. Let N = |V be the number nodes, and label each node in V by set of integers (1,2, .N for simplicity. We define Ne = (j
2

---

Under review as a conference paper at ICLR 2023
A (i.j) E E) the set of neighbor nodes of node i and define A == U [+). In decentralized learning, node has local data distribution D, and local objective function ft R + R, and can communicate with node j if and only if (i, E E. Then, decentralized learning aims to minimize the average of the local objective functions as follows:
min zERd
f(x)
= N Σ Ma))
f((x)) = RENND (2:(
where æ is the model parameter, & is the data sample that follows D;, and local objective function fa( is defined as the expectation of Fx E,) over data sample & the following. (F((
and (z) == D, (a; {,)] denote the stochastic and full gradient respectively.
Distributed SGD (DSGD) Lian et al. 2017) is one of the most well-known algorithms for decen- tralized learning. Formally, the update rules of DSGD are defined as follows:
(r+1)
Σ
(r) Wij m 1, (m(ee((((?)))) :)))))
(1)
jEA,
where 77 > 0 is the step size and Wij E [0. 1] is the weight of edge (i.j). Let W E [0, NxN be the matrix whose (elemet is W,; if ( E and 0 otherwise. In general, mixing matrix is used for W (i.e., W W W1 1, and W 1 1). Lian et al. (2018) extended DSGD in the case where each node communicates asynchronously and analyzed the convergence rate. Koloskovl let al. (2020) analyzed the convergence rate of DSGD when the network topology changes over time. These results revealed that the convergence rate of DSGD decreases and the performance is degraded when the data distributions held by each node are statistically heterogeneous. This is because the local gradients f. are different across nodes and each model parameter æ; tends to drift away when the data distributions are heterogeneous. To address this issue, D2 (Tang et al.) 2018b). Gradient Tracking (Lorenzo & Scutari, 2016: Nedić et al., 2017). and primal-dual algorithms (Niwa et al., 2020 2021 Takezawa et al. 2022b) were proposed to correct the local gradient V to the global gradient V f. As different approach, Vogels al. (202I) proposed novel averaging method to prevent each model parameter a, from drifting away. has been shown that the convergence rates of these methods do not depend on data heterogeneity and do not decrease, even when the data distributions held by each node are statistically heterogeneous. However, these methods do not consider the case in which momentum is used.
2.2 MOMENTUM ACCELERATION
The methods with momentum acceleration were originally proposed by PPlyyk (1964), and SGD with momentum acceleration (SGDm) has achieved successful results in training neural networks (imonnyan & Zisserman 2015 He et al. 2016); [Wang et al. (0200). In decentralized learning, a straightforward approach using the momentum DSGD with momentum acceleration (DSGDm)
(Gao & Huang 2020). The update rules of DSGDm are defined as follows:
u (r+1) = =Bu + (F
(2)
(r+1)
Σ Wij (x)))) -
(3)
JENT
where ue is the local momentum of node and 8 E 10. 1) is momentum coefficient. In addi- tion, several variants of DSGDm were studied by Yu et al. (2019) Assran al, (2019); Wang et al. (2000a); Singh et al. (202T). When the data distributions held by each node are statistically homoge- neous. DSGDm works well and can improve the performance as well as SGDm. However, when the data distributions are statistically heterogeneous, DSGDm leads to poorer performance than DSGD. This is because when the data distributions held by each node are statistically heterogeneous (i.e., ff varies significantly across nodes), the difference in the updated value of the model parameter across the nodes (i.e., ryu,) is amplified by the momentum (Lin et al. 2021).
To address this issue, Yuan et al. (2221) and Lin et al. 2202 proposed methods to modify the update rules of the momentum in DSGDm, called Decent LaM and QG- DG-DDG respectively. They further experimentally demonstrated that these methods more robust to data heterogeneity than DSGDm. However, their convergence rates have been shown to still depend on data heterogeneity and decrease when the data distributions are heterogeneous.
3

---

Under review as a conference paper at ICLR 2023
2.3 GRADIENT TRACKING
One of the most well-known methods whose convergence rate does not depend on data heterogene- ity is Gradient Tracking (Lorenzo Scutari 2016). Whereas DSGD exchanges only the model parameter Ti Gradient Tracking exchanges the model parameter æ; and local (stochastic) gradi- ent f. and then updates the model parameters while estimating global gradient V.. Nedić et al. (017)) and Qu & Li (2018) analyzed the convergence rate of Gradient Tracking when the objec- tive function is (strongly) convex and the full gradient used. [u & Nedic (202T) analyzed the convergence rate when the objective function is strongly convex and the stochastic gradient is used. Recently, Koloskova et al. (2021) analyzed the convergence rates of Gradient Tracking in a standard deep learning setting. where the objective function is non-convex and the stochastic gradient is used. There is also a line of research to combine Gradient Tracking with variance reduction methods (Xin) et al. (022). They showed that the convergence rate of Gradient Tracking does not depend data heterogeneity. However, these studies only consider the case without momentum acceleration, and the convergence analysis for Gradient Tracking with momentum acceleration has not been explored thus far in the aforementioned studies.
In the convex optimization literature, Xin & Khan (0200) and Carnevale et al. 2022 proposed
combination of Gradient Tracking and the momentum or Adam (Kingma & Ba they only considered the case where the objective function is strongly convex and the full gradient used. The convergence rate is still unclear in the standard deep learning setting, where the objective function is con-convex and the stochastic gradient is used. Furthermore, there no discussion about data heterogeneity in these studies, either theoretically or experimentally.
2015 However,
3 PROPOSED METHOD
In this section, we propose Momentum Tracking, which is decentralized learning method with momentum acceleration whose convergence rate is proven to independent the data heterogene- ity in the standard deep learning setting.
3.1 SETUP
We assume that the following four standard assumptions hold: Assumption 1.
There exists constant f > -oo that satisfies f(x)) 2 1* for all x E RRd.
Assumption 2. There exists constant p (0, that satisfies for all $1,'". NN Rd.
|XW~ X 1 S (1 p)|| -
where X = (I1."" ,NN) ) E RR and X = XX11..
Assumption 3. There exists constant L > 0 that satisfies for all i E V and æ, y E R4,
[V)(x) - V.((i)ll S L. - yll-
(5)
Assumption 4. There exists constant x² that satisfies for all V and æ; E Rd.
DE (F v((=))) o².
(6)
Assumptions 1.2 2 3 and are commonly used for decentralized learning algorithms (Lian et al.) 2017 Yu et al. 2019: Koloskova et al.] 2021 Lin et al.) (02T). Additionally, the following as- sumption, which represents data heterogeneity, is commonly used in the convergence analysis of decentralized learning algorithms (Lian et al. 2017 Yu et al. 2019 Lin et al. (221))
Assumption 5. There exists constant <² that satisfies for all Rd,
N
V(()) S
N
Under Assumption 5 the convergence rates of DSGD iLian et al. 2017). DSGDm (Gao & Huang 2020 Yuan al. 202T), QG-DSGDM (Lin et al. 2021 and Decentl.at (Yuan al. 2021) were
4

---

Under review as a conference paper at ICLR 2023
shown to be dependent data heterogeneity ç² and decrease as 52 increases. By contrast, in Sec. we prove that Momentum Tracking converges without Assumption and the convergence rate independent of data heterogeneity <² In addition, do not assume the convexity of the objective functions (() and (ar). Therefore, / (ar) and fi) are potentially non-convex functions (e.g.. the loss functions of neural networks).
3.3
3.2
MOMENTUM TRACKING
In this section, we propose Momentum Tracking, which robust to data heterogeneity and accel- erated by the momentum. The update rules of Momentum Tracking are defined follows:
(r+1)
BBu (r)
+ (a ; :
u;
a
(r+1)
Σ
W
(r) 17 (r+1) -c(").
(8)
JEN!
(r+1)
= Σ We (c/)) -uj ( + u; (r+1)
JENT
where 33 E [0, 1) is a momentum coefficient. The pseudo-code for Momentum Tracking is presented Sec. A In Momentum Tracking, C, corrects the local momentum u, the global momentum N Dj Uj and prevents each model parameter æ; from drifting, even when the data distributions are statistically heterogeneous (i.e., the local momentum u, varies significantly across nodes).
Because Momentum Tracking is equivalent to Gradient Tracking when ,3 O, Momentum Track- ing is a simple extension of Gradient Tracking. Hence, when / = 0. it has been shown that the convergence rate of Momentum Tracking independent of data heterogeneity Ç² (Koloskova al., 2021). However, because data heterogeneity is amplified when the momentum is used instead of the stochastic gradient (i.e., 3 > 0) (Lin et al. [021]] Yuan et al., (021), is unclear whether the conver- gence rate of Momentum Tracking is independent of data heterogeneity (² for any 3 [0, 1) or for only a restricted range of In Sec. [3.3] we provide the convergence rate of Momentum Tracking and prove that it is independent of data heterogeneity (² for any 3 [0. 1).
3.3 CONVERGENCE ANALYSIS
Under Assumptions I and Theorem I provides the convergence rate of Momentum Tracking in the standard deep learning setting. All proofs are presented in Sec. [
Theorem 1 (Convergence Rate in Non-Convex Setting). Suppose that Assumptions 7
2 and
hold, each model parameter 24 is initialized with the same parameters, and both u: and C, are (0),
initialized T-3 ( (xx :(!") - Σ/= (a) :) Then, for any 3 E 10, and R > 1,
(0),
there exists step size 17 such that the average parameter 2 - EE x; generated by Eqs. 7 satisfies
E
(10)
R
Too2L
11 (1- (1 (1+ (1+1-1) 8) ))
r T200L2
p33
+ Lro (1 3>-3
G2
1+ (1 1B-3pp3p
NR
where ro ((z(00)--''
Remark 1. Combinations of Gradient Tracking with the momentum or Adam have also been pro- posed by Xin & Khan (202 and Carnevale et al. (2022). However, they considered only the setting in which the objective function is strongly convex and the full gradient is used. By contrast, our study focuses on the deep learning setting. Hence, our proof strategies are completely different from those in these previous studies, and Theorem provides the convergence rate in the setting where the objective function is non-convex and the stochastic gradient is used.
(220)
Remark 2. The convergence rate of Gradient Tracking in the standard deep learning setting was provided by Koloskova et al. (2020). However, they did not consider the case where the momentum used, and it is not trivial to provide the convergence rate of Momentum Tracking from the results in this previous work.
5

---

Under review as a conference paper at ICLR 2023
3.4 DISCUSSION
Comparison with Gradient Tracking:
Theorem indicates that the convergence rate of Momen-
tum Tracking does not depend on data heterogeneity 5² for any 3 E [0, 1) and does not decrease when the data distributions are statistically heterogeneous (i.c., ç² > 0). Therefore, Theorem iidicaaes that Momentum Tracking is theoretically robust to data heterogeneity for any / E [0, 1). Although Momentum Tracking is simple extension of Gradient Tracking, our work is the first to identify that the combination of Gradient Tracking and the momentum converges without being affected by data heterogeneity Ç² for any 8 E [0, 1) in the standard deep learning setting.
Because the convergence rate of Momentum Tracking Eq. (TO) is optimal when [ = 0. Theorem T does not show that the convergence rate is improved by using the momentum. However, the convergence rates of DSGDm and QG-DSGDm provided by Gao & Huang (2020) and Lin et al. (2021) are also optimal when consistent with the experimental results that the convergence rates are improved when 3 > 0. As in these studies, we experimentally demonstrate that convergence is accelerated when B > in Sec. and leave for future work to show the theoretical benefits of using 3 > O.
O. Moreover, they do not provide theoretical results that are
Comparison with Existing Algorithms with Momentum Acceleration: Next, we compare the convergence rate of Momentum Tracking with those of existing decentralized learning algorithms with momentum acceleration: DSGDm (Gao Huang 2020), Decentl LaM (Yuan et al. 202T), and QG-DSGDm (Lin et al. 2021). Here, only show the convergence rate of QG-DSGDm, but the same discussion holds for the other methods. The convergence rate of QG-DSGDm is as follows:
((Lin et al., 2021)). Suppose that Assumptions 1 also holds. Then, for any B E [0, ztrpl and R Z there exists step size n such that the average parameter x= = N Σx æi generated by QG-DSGDm satisfie
Theorem 2
3 and A hold, and Assumption
( /TooPT. NNR +
+ (1812((+00)) R²R²
Lro
1
B
R E
R p 1-3 (1- By3
+
where ro f( f
Data heterogeneity <² appears in the second term, and the convergence rate of QG-DSGDm depends on data heterogeneity Ç Therefore, the convergence rate of QG-DSGDm decreases when the data distributions held by each node are statistically heterogeneous. By contrast, the convergence rate of Momentum Tracking Eq. (10) does not depend on data heterogeneity Ç². Therefore, Momentum Tracking is more robust to data heterogeneity than QG-DSGDm. Because the convergence rates of DSGDm and DecentLaM also depend on data heterogeneity Ç² the same discussion holds for DSGDm and DeC Hence, Momentum Tracking is more robust to data heterogeneity than these methods. To the best of knowledge, Momentum Tracking is the first decentralized learning method with momentum acceleration whose convergence rate has been proven to be independent of data heterogeneity Ç² in the standard deep learning setting.
Next, we discuss the range of B. The convergence rates of QG-DSGDm and DecentLaM provided by Lin et al. (202I) and Yuan et al. (202I) hold only when the range of is restricted. For instance, 0.05). However, these restrictions on the range of 3 do not hold in practice. (Typically, 3 is set to 0.9.) Therefore, the convergence rates of QG-DSGDm and DecentLaM are unclear such practical cases. By contrast, Theorem can provide the convergence rate of Momentum Tracking that holds for any B E [0, 1).
Theorem 2 assumes that 3 < 2 2
Comparison with SGDm: Next, we compare the convergence rate of Momentum Tracking with that of SGDm. In setting where the objective function is non-convex and the stochastic gradi- ent is used, SGDm has been proven to converge to the stationary point with ((1/VR) R)) (Yan et al. 2018 Liu et al.) 2020b). By contrast, Theorem [ indicates that if the number of rounds is suffi- ciently large, Momentum Tracking converges with (11/ N R). Therefore, Momentum Tracking can achieve linear speedup with respect to the number of nodes N, which is common and impor- tant property in decentralized learning methods (Lian et al. 2018 Koloskova et al.) 2020).
For simplicity, we set the additional hyperparameter j for QG-DSGDm 3.

---

Under review as a conference paper at ICLR 2023
75
Accurary
50
Accuracy
Accuracy
DSGD Tracking
30
DSGDm
20
QG D5GDm
55
DecentLaM
Momentum Tracking
300
400
560
300
Epoch
k-class
(a) CIFAR-10 (10-class)
(b) CIFAR-10 (4-class)
(c) Average accuracy for datasets
Figure 1: (a) Learning curve on CIFAR-I0 with LeNet in the 10-class (i.e., homogeneous) setting. We evaluated the test accuracy per 10 epochs. (b) Learning curve in the 4-class (i.e., heterogeneous) setting. (c) Average test accuracy for all datasets (i.e., FashionMNIST, SVHN, and CIFAR-10).
4 EXPERIMENT
In this section, we present the results of an experimental evaluation of Momentum Tracking and demonstrate that Momentum Tracking is more robust to data heterogeneity than the existing decen- tralized learning methods with momentum acceleration. In this section, focus test accuracy, and more detailed evaluation about the convergence rate is presented in Sec. C.4
4.1 SETUP
Comparison Methods: (1) DSGD (Lian al. 2017): the method descrihed in Sec. 2.1 (2) DS-
GDm (Gao & Huang 2020): the method described in Sec. 2.2 (3) QG-DSGDm (Lin et al. 2021): )
method in which the update rule of the momentum in DSGDm is modified to be more robust to data heterogeneity than DSGDm; (4) DecentLaM (Yuan et al. 202T]: method in which the update
rule of the momentum in DSGDm modified to be more robust to data heterogeneity; (5) Gradi- ent Tracking (Nedić et al., 2017): method without momentum acceleration that is robust to data heterogeneity; (6) Momentum Tracking: the proposed method described Sec.
We evaluated Momentum Tracking using three 10-class image classification tasks: FaShIonMNIST (Xiao et al.] 2017), SVHN (Netzer et al.) al.] [0TT), and CIFAR-10 (Krizhevsky) 2009). Following the previous work (Niwa al., 2020). distributed the data to nodes such that each node was given data randomly selected & classes. When & = 10, the data distributions held by each node can be regarded as statistically homogeneous. When k < 10, the data distributions are regarded as statistically heterogeneous. We evaluated the comparison methods by setting & to (4, 6,8, 8, 10} and changing data heterogeneity. Note that a smaller & indicates that the data distribu- tions are more heterogeneous. For the neural network architecture, we used LeNet (LeCun et al.) 1998) with group normalization (Wu & He, 2018) in Sec. 4.2 In Sec. 4.3 present more de- tailed evaluation by varying the neural network architecture (c.g.. VGG-11 (Simonyan & Zisserman 2015) and ResNet-34 (He al. (0166)) For each comparison method, we used 10% the training data for validation and individually tuned the step size. For DSGDm, QG-DSGDm, DecentLaM, and Momentum Tracking, we set 3 to 0.9. All experiments were repeated using three different seed values, and we report their averages. More detailed hyperparameter settings are presented in See E
Dataset and Model:
Network Topology and Implementation: In Secs. 4.2 and we present the results of setting the underlying network topology to a ring consisting of eight nodes (i.e., N = 8). In Sec. C. we present more detailed evaluation by varying the network topology. All comparison methods were implemented using PyTorch and run eight GPUs (NVIDIA RTX 3090).
4.2 EXPERIMENTAL RESULTS
Table 2 lists the test accuracy for FashionMNIST, SVHN and CIFAR-10. Fig. (a) the learning curves for CIFAR-10 and Fig. (c) presents the average test accuracy for all datasets.
(a) and (b) present
7

---

Under review as a conference paper at ICLR 2023
Table Test accuracy FashionMNIST, SVHN, and CIFAR-IO with LeNet. "k-class" means that each node has only the data of randomly selected classes. Bold font means the highest accuracy.
FashionMNIST
10-class
8-class
6-class
4-class
DSGD
85.6 + 0.49
85.6 0.41
82.7 = 1.12
78.1 = 1.56 84.9 + 0.22
Gradient Tracking
85.0 + 0.49
85.4 = 0.26
85.0 + 0.37
DSGDm
89.5 + 0.15
89.3 = 0.21
82.1 = 3.23
68.7 = 5.02
QG-DSGDm
89.6 0.10
89.5 0.47
86.9 1.59
80.8 + 2.94
DecentLaM
89.5 + 0.14
89.3 = 0.36
89.2 = 0.41
84.3 + 3.05
Momentum Tracking
89.5 + 0.36
89.4 = 0.05
88.9 0.47
86.8 + 1.56
SVHN
10-class
8-class
6-class
4-class
DSGD)
90.1 + 0.17
89.5 0.61
87.6 + 1.94
78.8 + 8.55
Gradient Tracking
90.1 + 0.30
89.8 = 0.38
89.8 = 0.39
89.4 0.47
DSGDm
92.6 + 0.35
92.4 = 0.19
88.1 + 4.38
67.2 + 9.69
QG-DSGDm
92.5 + 0.22
92.5 0.17
90.9 1.67
83.5 7.14
DecentLaM
92.4 + 0.21
92.2 0.39
92.0 = 0.48
88.2 + 4.75
Momentum Tracking
92.6 + 0.32
92.4 0.40
92.3 0.23
91.7 + 0.53
CIFAR-10
10-class
8-class
6-class
4-class
DSGD
63.1 + 0.60
64.1 0.52
61.2 : 1.16
47.6 5.77 +
Gradient Tracking
62.3 + 0.73
62.0 = 0.80
61.9 - 0.58
DSGDm
72.9 + 0.41
72.5 = 0.20
63.8 - 6.24
38.8 1.61
QG-DSGDm
72.4 + 0.87
73.1 0.16
69.6 : 2.42
55.3 + 5.30
DecentLaM
73.2 = 0.36
72.9 0.14
11.71.111
63.1 = 5.43
Momentum Tracking
72.9 = 0.59
73.0 = 0.49
0.41
70.7 = 1.38
Comparison of Momentum Tracking and Gradient Tracking: First, we discuss the results of Momentum Tracking and Gradient Tracking. Table Fig. indicaate that Momentum Tracking achieves higher accuracy faster than Gradient Tracking and outperforms Gradient Tracking in all settings. When the data distributions are homogeneous (i.e., 10-class). Momentum Tracking outperforms Gradient Tracking by 5.8% average. When the data distributions are heterogeneous (e.g.. 4-class), Momentum Tracking outperforms Gradient Tracking by 4.4% on average. Therefore, the results show that Momentum Tracking can consistently outperform Gradient Tracking regardless of data heterogeneity.
Comparison of Momentum Tracking and DSGDm: Next, discuss the results of Momentum Tracking and DSGDm. The results show that when the data distributions are homogeneous (i.e., 10-class), Momentum Tracking and DSGDm are comparable and outperform DSGD and Gradient Tracking. However, when the data distributions are heterogeneous (e.g., 4-class), the test accuracy of DSGDm decreases even more than that of DSGD, and DSGDm underperforms DSGD by 9.9% on average. By contrast, the results indicate that Momentum Tracking consistently outperforms DSGD and Gradient Tracking by 14.9% and 4.4% respectively when the data distributions are het- erogeneous. The results indicate that Momentum Tracking is more robust to data heterogeneity than DSGDm and outperforms DSGDm by 24.9% on average.
Comparison of Momentum Tracking, QG-DSGDm, and DecentLaM: When the data distribu- tions are homogeneous (i.c., 10-class), Momentum Tracking, QG-DSGDm, and DecentLaM are comparable and outperform DSGD and Gradient Tracking. By contrast, when the data distributions are heterogeneous (e.g., 4-class), Momentum Tracking consistently outperforms QG-DSGDm and DecentLaM by 9.9% and 4.5% respectively, whereas QG-DSGDm and DecentLaM to data heterogeneity than DSGDm. Hence, these results are consistent with our theoretical analysis, discussed in Secs. 3.3 and 3.4
robust
8

---

Under review as a conference paper at ICLR 2023
Table 3: Test accuracy on CIFAR-10 with VGG-11 and ResNet-34. "k-class" indicates that each node has only the data of randomly selected & classes, and bold font indicates the highest accuracy.
CIFAR-I0 VGG-11
CIFAR-10 ResNet-34
10-class
4-class
2-class
10-class
4-class
2-class
DSGD
89.7 +9.7+
85.8 2.39
70.3 2.73
93.4 + 0.05
87.5 3.27
64.1 + 0.54
Gradient Tracking
88.1 0.19
86.0 +: 0.45
82.9 0.13
85.1 + 1.07
81.2 + 0.73
75.9 0.41
DSGDm
92.2 0.09
77.3 4.05
39.6
95.8 = 0.26
3.09
2.83
QG-DSGDm
92.0 0.04
89.4 + 1.04
77.8 + 1.96
95.6 + 0.37
94.0 1.02
+
DecentLaM
92.1 1 0.09
90.9 1 0.65
85.2 0.67
95.9 1 0.04
95.2 0.51
89.2 2.26
Momentum Tracking
91.9 0.06
90.9 + 0.60
87.0 0.48
95.0 0.13
94.4 0.52
89.9 0.73
VGG-11
ResNet-34
M Aw
Accuracy
Accuracy
50-
50
DSGD
Gradient Tracking
DSGDm
30
QG DG-DSSDD
DecentLaM
10
10
Momentur Tracking
200
400
600
800
1000
200
400
600
750
Epoch
Epoch
Figure 2: Learning curves for CIFAR-I0 with VGG-11 and ResNet-34 in the 2-class setting.
In summary, when the data distributions are homogeneous, DSGDM, QG-DSGDM, Decent and Momentum Tracking are comparable and outperform DSGD and Gradient Tracking. When the data distributions heterogeneous, Momentum Tracking is more robust data heterogeneity than DSGDm, QG-DSGDm, and DecentLaM, and can outperform all comparison methods.
4.3 RESULTS WITH VARIOUS NEURAL NETWORK ARCHITECTURES
Next, evaluated Momentum Tracking in detail by varying the neural network architecture. Table lists the test accuracy with VGG-11 (Simonyan Zisserman, 2015) and ResNet-34 (He al. 2016 when we set to {2, 4, 10). and Fig. shows the learning curves.
For both neural network architectures, Table reveals that when the data distributions are homo- geneous (i.e., 10-class), Momentum Tracking comparable with DSGDm, QG-DSGDm, and De- centLaM and outperforms DSGD and Gradient Tracking. By contrast, when the data distributions are heterogeneous (e.g.. 2-class), Table and Fig. 2 reveal that Momentum Tracking outperforms all comparison methods for both neural network architectures. In particular, Fig. 2 indicates that DSGDm, QG-DSGDm, and DecentL.aM are unstable and continue oscillate the final training phase, whereas Momentum Tracking converges stably. These results are consistent with those of LeNet presented in Table 2] Therefore, the results indicate that Momentum Tracking is more ro- bust to data heterogeneity than DSGDm, QG-DSGDm, and DecentLaM, and can outperform these methods regardless of the neural network architecture.
CONCLUSION
In this study, we propose Momentum Tracking, which method with momentum acceleration whose convergence rate is proven to be independent of data heterogeneity. More specifically, we provide the convergence rate of Momentum Tracking in the standard deep learning setting, in which the objective function is non-convex and the stochastic gradient is used. Our theoretical analysis reveals that the convergence rate Momentum Tracking is independent data heterogeneity for any 8 E [0, 1). Through image classification tasks, we demonstrated that Momentum Tracking can consistently outperform the decentralized learning methods without momentum acceleration regardless of data heterogeneity. Moreover, showed that Momentum Tracking is more to data heterogeneity than existing decentralized learning methods with momentum acceleration and can consistently outperform these existing methods when the data distributions heterogeneous.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Mahmoud Assran, Nicolas Loizou, Nicolas Ballas, and Mike Rabbat. Stochastic gradient push for distributed deep learning. In International Conference on Machine Learning. 2019.
Guido Carnevale, Francesco Farina, Ivano Notarnicola, and Giuseppe Notarstefano. Gtadam: Gra- dient tracking with adaptive momentum for distributed online optimization. In arXiv, 2022.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT; Pre-training of deep bidirectional transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguistics, 2019.
Yasaman Esfandiari, Sin Yong Tan, Zhanhong Jiang, Aditya Balu, Ethan Herron, Chinmay Hegde, and Soumik Sarkar. Cross- goas-ggradien aggregation for decentralized learning from non-iid data. In International Conference Machine Learning, 2021.
Hongchang Gao and Heng Huang. Periodic stochastic gradient descent with momentum for decen- tralized training. In arXiv, 2020.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip Gibbons. The non-llD data quagmire of decentralized machine learning. In International Conference on Machine Learning, 2020.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International
Conference on Learning Representations, 2015.
Anastasia Koloskova, Nicolas Loizou, Sadra Boreiri, Martin Jaggi, and Sehastian Stich. A unified theory of decentralized SGD with changing topology and local updates. In International Confer- ence on Machine Learning. 2020.
Anastasia Koloskova, Tao Lin, and Scbastian U Stich. An improved analysis of gradient tracking for decentralized machine learning. In Advances in Neural Information Processing Systems, 2021.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. In IEEE, 1998.
Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsich, Wei Zhang, and Ji Liu. Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent. In Advances Neural Information Processing Systems, 2017.
Xiangru Lian, Wei Zhang, Ce Zhang, and Ji Liu. Asynchronous decentralized parallel stochastic gradient descent. In International Conference on Machine Learning, 2018.
Tao Lin, Sai Praneeth Karimireddy, Sebastian Stich, and Martin Jaggi. Quasi-global momentum: Accelerating decentralized deep learning on heterogeneous data. In International Conference on Machine Learning. 2021.
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. In International Conference Learning Representations, 2020a.
Xiaorui Liu, Yao li, Rongrong Wang, Jiliang Tang, and Ming Yan. Linear convergent decentralized optimization with compression. In International Conference on Learning Representations, 2021.
Yanli Liu, Yuan Gao, and Wotao Yin. An improved analysis of stochastic gradient descent with momentum. In Advances in Neural Information Processing Systems, 2020b.
10

---

Under review as a conference paper at ICLR 2023
Paolo Di Lorenzo and Gesualdo Scutari. NEXT: in-network nonconvex optimization. In IEEE Transactions on Signal and Information Processing over Networks, 2016.
Yucheng Lu and Christopher De Sa. Moniqua: Modulo quantized communication in decentralized
SGD. In International Conference Machine Learning, 2020.
Angelia Nedić, Alexander Olshevsky, and Wei Shi. Achieving geometric convergence for distributed optimization over time-narying graphs. In SIAM Journal on Optimization, 2017.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In Advances in Neural Information Processing Systems Workshop, 2011.
Kenta Niwa, Noboru Harada, Guoqiang Zhang, and W. Bastiaan Kleijn. Edge-consensus learning: Deep learning on p2p networks with nonhomogeneous data. In International Conference on Knowledge Discovery and Data Mining, 2020.
Kenta Niwa, Guoqiang Zhang, W. Bastiaan Kleijn, Noboru Harada, Hiroshi Sawada, and Akinori Fujino. Asynchronous decentralized optimization with implicit stochastic variance reduction. In International Conference Machine Learning, 2021.
Boris T. Polyak. Some methods of speeding up the convergence of iteration methods. In USSR Computational Mathematics and Mathematical Physics, 1964.
Shi and Angelia Nedic. Distributed stochastic gradient tracking methods. Math. Program., 2021.
Guannan Qu and Na Li. Harnessing smoothness to accelerate distributed optimization. In IEEE Transactions on Control of Network Systems, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference Learning Representations, 2015.
Navjot Singh, Deepesh Data, Jemin George, and Suhas Diggavi. Squarm-sgd: Communication-
efficient momentum sgd for decentralized optimization. In JEEE International Symposium on Information Theory, 2021.
Yuki Takezawa, Kenta Niwa, and Makoto Yamada. Communication compression for decentralized learning with operator splitting methods. In arXiv, 2022a.
Yuki Takezawa, Kenta Niwa, and Makoto Yamada. Theoretical analysis of primal-dual algorithm for non-convex stochastic decentralized optimization. In arXiv, 2022b.
Hanlin Tang, Shaoduo Gan, Ce Zhang, Tong Zhang, and Liu. Communication compression for decentralized training. In Advances in Neural Information Processing Systems, 201 8a.
Hanlin Tang, Xiangru Lian, Ming Yan, Ce Zhang, and Ji Liu. de: Decentralized training over decentralized data, In International Conference on Machine Learning, 2018b.
Thijs Vogels, Lie He, Anastasia Koloskova, Sai Praneeth Karimireddy, Tao Lin, Sebastian U Stich, and Martin Jaggi. RelaySum for decentralized deep learning on heterogeneous data. In Advances in Neural Information Processing Systems, 2021.
Jianyu Wang, Vinayak Tantia, Nicolas Ballas, and Michael Rabbat. SlowMo: Improving communication-efiiieell distributed sgd with slow momentum. In International Conference on Learning Representations, 2020a.
Jun-Kun Wang, Chi-Heng Lin, and Jacob Abernethy. Escaping saddle points faster with stochastic momentum. In International Conference Learning Representations, 2020b.
Yuxin Wu and Kaiming He. Group normalization. In European Conference on Computer Vision, 2018.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mmin a novel image dataset for benchmark- ing machine learning algorithms. In arXiv, 2017.

---

Under review as a conference paper at ICLR 2023
Ran Xin and Usman A. Khan. Distributed heavy-ball: A generalization and acceleration of first- order methods with gradient tracking. In IEEE Transactions on Automatic Control, 2020.
Ran Xin, Usman A. Khan, and Soummya Kar. Fast decentralized nonconvex finite-sum optimization with recursive variance reduction. In SIAM Journal Optimization, 2022.
Yan Yan, Tianbao Yang. Zhe Li, Qihang Lin, and Yi Yang. A unified analysis of stochastic mo- mentum methods for deep learning. In International Joint Conference on Artificial Intelligence, 2018.
Bicheng Ying. Kun Yuan, Yiming Chen, Hanbin Hu, Pan Pan, and Wotao Yin. Exponential graph is provably efficient for decentralized deep training. In Advances in Neural Information Processing Systems, 2021.
Hao Yu, Rong Jin, and Sen Yang. On the linear speedup analysis of communication efficient mo- mentum SGD for distributed non-convex optimization. In International Conference on Machine Learning, 2019.
Kun Yuan, Yiming Chen, Xinmeng Huang, Yingya Zhang, Pan Pan, Yinghui Xu, and Wotao Yin. DecentLaMe Decentralized momentum sgd for large-batch deep training. In International Con ference Computer Vision, 2021.
12


Reviews:
[{"review_id": "nHxM5aKYP1", "reviewer": "Reviewer_AZPB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes momentum tracking in decentralized learning. The authors establish a main theorem that shows the proposed momentum tracking method is invariant to the data heterogeneity bound. The authors conduct several experiments comparing momentum tracking with other baseline methods including gradient tracking (a special case with $\\beta=0$), DSGDm, QG-DSGDm and DecentLaM.", "summary_of_the_review": "I believe the paper has good topic and some interesting analysis, but I'd like to inquire the authors on the main claim they make (momentum tracking is invariant to the data heterogeneity); please refer to the weakness 1 for details.", "strength_and_weaknesses": "Strength\n1. The idea of tracking momentum rather than gradient in decentralized learning is interesting, and is more close to real-world applications. In many applications, especially vision tasks, applying momentum is a must.\n2. The authors compare both empirical and theoretical results with multiple baselines. Although there are a few missing (details in weakness), but I appreciate the authors for detailed discussion with these mentioned baseline works.\n\nWeakness\n1. My main concern is on the claim of \"momentum tracking is invariant to the data heterogeneity\". In fact, this is not a new result in the decentralized learning. Many algorithms including D2 (https://arxiv.org/pdf/1803.07068.pdf), DeTAG (https://arxiv.org/pdf/2006.08085.pdf), DSGT (https://arxiv.org/pdf/1909.02712.pdf) are able to achieve this with only gradient tracking type methods. The difference is that these existing rates will depend on the $\\zeta$ at step 0, which is the norm bound on the $$\\zeta_0=\\left\\|\\nabla F_i(x_i^{(0)};\\xi_i^{(0)}) - \\frac{1}{N}\\sum_{j=1}^{N}\\nabla F_j(x_j^{(0)};\\xi_j^{(0)})\\right\\|$$. \nIt seems Theorem 1 in the paper solves this problem with no $\\zeta_0$ shown, but it appears that $u_i$ and $c_i$ are initialized to $\\zeta_0$. This raises two problems: 1) if $u_i$ is initialized to $0$ (which is usual in practice), it appears the obtained rate is no better than the rates given in the previous works mentioned (D2, DeTAG, DSGT); 2) In practice, it is generally impossible to initialize $u_i$ to such quantity, unless all the nodes all reduce their first gradients.\n2. In the experiments, I believe the authors should also compare with the naive baseline where gradient tracking and local momentum are enabled. This way, we can know clearly whether it is tracking or momentum (local) that makes the difference in small-class case.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TF7mpeZat2f", "reviewer": "Reviewer_uTVi", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper developed a new decentralized optimization algorithm to address the heterogeneous data distribution issue. In particular, it introduces a correction term for the local stochastic gradient to alleviate the distribution shift issue. The authors further provided theoretical analysis for the convergence rate and conducted empirical results to demonstrate the performance of the proposed algorithm. \n\n", "summary_of_the_review": "Overall, this paper developed a new decentralized optimization approach. But some parts are not quite clear, the theoretical bound is not tight, and the novelty is incremental.  ", "strength_and_weaknesses": "Pros:\n1. The problem studied is important and the proposed algorithm looks effective based on the empirical results. \n2. The authors provided extensive experimental results. \n\nCons:\n1. The authors claim that the proposed algorithm becomes gradient tracking when $\\beta=0$. However, it is not clear to me. In particular, in gradient tracking, there should be a difference between the stochastic gradient in two consecutive iterations on each device. But there is no such a term. \n2. Introducing a correction term to the local gradient is a commonly used approach to address the distribution shift issue. For instance, SCAFFOLD also employs such a strategy. What is the difference between your approach and SCAFFOLD?\n3. As for the convergence rate, existing algorithms only have a quadratic dependence on the spectral gap. But the proposed algorithm has a worse dependence. Is it possible to improve this dependence? If not, what is the reason? More discussions are needed.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SDQBOf0XSx", "reviewer": "Reviewer_i18n", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The manuscript tries to address the issue of decentralized deep learning with heterogeneous local data, by extending the idea of gradient tracking to the momentum buffer.\nThe convergence rate of the proposed idea has no dependence on data heterogeneity and can achieve better empirical performance than existing methods.", "summary_of_the_review": "The reviewer enjoys reading the manuscript and believes it is a crucial question for decentralized deep learning.\n\nThe proposed solution may be a valid method to alleviate the data heterogeneity. However, the current manuscript still needs to improve its experimental settings and compare it fairly with STOA methods.\n\nThe reviewer will reconsider the rating once the issues pointed out in the weakness part have been addressed.\n\n\n### post-rebuttal\nstill a borderline paper (though some authors' responses have addressed the reviewer's concern, and have raised the score to 6).", "strength_and_weaknesses": "## Strength\n* The paper is very clear and well-written.\n* The attempts introduced in the manuscript may alleviate the issue of data heterogeneity; some empirical results also justify its potential.\n\n## Weaknesses\n1. Missing comparison with RelaySum (Vogels et al 2021). The reviewer agrees with the authors that the original paper of RelaySum did not provide the convergence rate for DSGDm; however, all their empirical evaluations are on top of DSGDm. As a paper proposed to address the issue of decentralized deep learning, it is equally important to justify the performance gain of the proposed methods over all existing SOTA methods, in terms of their empirical performance.\n2. The manuscript considers an uncommon experimental setting for data heterogeneity, unlike the recent standard Dirichlet distributions in the community of decentralized deep learning and federated learning. The authors are required to justify their design choice and at least include some experiments for CIFAR, in terms of alpha=10, 1, 0.1. The authors can also take the chance to evaluate other datasets with more classes and consider other neural architectures (e.g. Vit), as the current evaluations are quite limited. \n3. The hyper-parameter settings stated in Appendix E seem a bit random to the reviewer and thus make results less convincing. For example, why 1000 epochs and 750 epochs are used for VGG-11 and ResNet-34, with a relatively small initial learning rate like 0.05? The reviewer also checked the prior work like QG-DSGDm and RelaySum, where in their experiments all settings were well-tuned for each baseline, making the comparison fair enough.\n4. Limited topology scale. Only 8 workers are considered for the evaluation, and it is unclear to the reviewer if some issues will occur when scaling up. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2CwCmsZ45lX", "reviewer": "Reviewer_hspD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a novel momentum-tracking algorithm, whose convergence rate is independent of data heterogeneity. This paper present this algorithm is independent of data heterogeneity for any momentum coefficient. Moreover, experiments find this paper presents a stronger method than previous decentralized learning methods.", "summary_of_the_review": "Seems to be an incremental application of the momentum tracking technique.", "strength_and_weaknesses": "I'm not an expert in this domain. In fact, one of my major research fields is tracking (object tracking). And object tracking almost has nothing to do with momentum tracking. Honestly speaking, I might not have enough skills to assess this paper. However, I hope my comments can help the authors improve their paper. Moreover, it is exciting to me to see if my assessments could align with the professional optimization people because my engineering practice might also benefit the theory design.\n\nStrengths:\n1. It seems that this paper has novel components than previous optimizers. And theorems prove that they are robust to data heterogeneity.\n2. In general this paper is well written and it is well grounded to the literature. I think all the relevant papers have been referenced and discussed.\n3. The experimental results seem promising and somewhat support the authors' claims that this optimizer can augment the decnetralized learning algorithms.\n\nWeaknesses: \n1. I do not think this paper is targeting an important enough and the improvements seem incremental. In other words, they do not highlight the core problems in the baselines. Since the data heterogeneity is already well studied. In the introduction, the authors' writing flow seems to be this:\n(1) Neural networks are powerful and decentralized learning is studied due to privacy concerns.\n(2) DSGDm can work poorly because the parameter drifts away easily.\n(3) Gradient tracking literature does not consider the data heterogeneity problem.\nHowever, I do not think the above can lead to the major contribution to this paper and the improvements seem to be A+B (decentralized learning + gradient tracking stuff). It is unclear how important the proposed approach is in the context of decentralized learning literature. Is it a simple borrow from the standard optimization to the decentralized learning setting? If so, how to reflect the novelty and significance of such a borrow? Would it lead to a major change in the results?\nBesides, it is unclear to me how important the data heterogeneity problem is and what the key insight for this solution. There is only one sentence in the introduction talking about the technical part of momentum tracking. It seems the technical contribution is weak.\n2. The experimental results are not convincing. Can the proposed approach work on ImageNet? Can the proposed optimizer work for object detection and other tasks? I also think the CIFAR10's experimental results are not convincing enough. For the 10-class cases, the training curves almost overlap with the baseline. The 4 class cases are not as important as the 10 class cases in my point of view. In other words, the improvements are not so significant.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject