Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The manuscript tries to address the issue of decentralized deep learning with heterogeneous local data, by extending the idea of gradient tracking to the momentum buffer.
The convergence rate of the proposed idea has no dependence on data heterogeneity and can achieve better empirical performance than existing methods.
strength_and_weaknesses: ## Strength
* The paper is very clear and well-written.
* The attempts introduced in the manuscript may alleviate the issue of data heterogeneity; some empirical results also justify its potential.

## Weaknesses
1. Missing comparison with RelaySum (Vogels et al 2021). The reviewer agrees with the authors that the original paper of RelaySum did not provide the convergence rate for DSGDm; however, all their empirical evaluations are on top of DSGDm. As a paper proposed to address the issue of decentralized deep learning, it is equally important to justify the performance gain of the proposed methods over all existing SOTA methods, in terms of their empirical performance.
2. The manuscript considers an uncommon experimental setting for data heterogeneity, unlike the recent standard Dirichlet distributions in the community of decentralized deep learning and federated learning. The authors are required to justify their design choice and at least include some experiments for CIFAR, in terms of alpha=10, 1, 0.1. The authors can also take the chance to evaluate other datasets with more classes and consider other neural architectures (e.g. Vit), as the current evaluations are quite limited. 
3. The hyper-parameter settings stated in Appendix E seem a bit random to the reviewer and thus make results less convincing. For example, why 1000 epochs and 750 epochs are used for VGG-11 and ResNet-34, with a relatively small initial learning rate like 0.05? The reviewer also checked the prior work like QG-DSGDm and RelaySum, where in their experiments all settings were well-tuned for each baseline, making the comparison fair enough.
4. Limited topology scale. Only 8 workers are considered for the evaluation, and it is unclear to the reviewer if some issues will occur when scaling up. 
clarity,_quality,_novelty_and_reproducibility: Please check the comments above.
summary_of_the_review: The reviewer enjoys reading the manuscript and believes it is a crucial question for decentralized deep learning.

The proposed solution may be a valid method to alleviate the data heterogeneity. However, the current manuscript still needs to improve its experimental settings and compare it fairly with STOA methods.

The reviewer will reconsider the rating once the issues pointed out in the weakness part have been addressed.


### post-rebuttal
still a borderline paper (though some authors' responses have addressed the reviewer's concern, and have raised the score to 6).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper developed a new decentralized optimization algorithm to address the heterogeneous data distribution issue. In particular, it introduces a correction term for the local stochastic gradient to alleviate the distribution shift issue. The authors further provided theoretical analysis for the convergence rate and conducted empirical results to demonstrate the performance of the proposed algorithm. 


strength_and_weaknesses: Pros:
1. The problem studied is important and the proposed algorithm looks effective based on the empirical results. 
2. The authors provided extensive experimental results. 

Cons:
1. The authors claim that the proposed algorithm becomes gradient tracking when $\beta=0$. However, it is not clear to me. In particular, in gradient tracking, there should be a difference between the stochastic gradient in two consecutive iterations on each device. But there is no such a term. 
2. Introducing a correction term to the local gradient is a commonly used approach to address the distribution shift issue. For instance, SCAFFOLD also employs such a strategy. What is the difference between your approach and SCAFFOLD?
3. As for the convergence rate, existing algorithms only have a quadratic dependence on the spectral gap. But the proposed algorithm has a worse dependence. Is it possible to improve this dependence? If not, what is the reason? More discussions are needed.

clarity,_quality,_novelty_and_reproducibility: Clarity: It is not clear how the proposed algorithm relates to gradient tracking.

Novelty: The idea of using a correction term to address the distribution shift issue has been extensively studied in FL. It would be good to provide more discussions. 

Reproducibility: Good. 
summary_of_the_review: Overall, this paper developed a new decentralized optimization approach. But some parts are not quite clear, the theoretical bound is not tight, and the novelty is incremental.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes momentum tracking in decentralized learning. The authors establish a main theorem that shows the proposed momentum tracking method is invariant to the data heterogeneity bound. The authors conduct several experiments comparing momentum tracking with other baseline methods including gradient tracking (a special case with $\beta=0$), DSGDm, QG-DSGDm and DecentLaM.
strength_and_weaknesses: Strength
1. The idea of tracking momentum rather than gradient in decentralized learning is interesting, and is more close to real-world applications. In many applications, especially vision tasks, applying momentum is a must.
2. The authors compare both empirical and theoretical results with multiple baselines. Although there are a few missing (details in weakness), but I appreciate the authors for detailed discussion with these mentioned baseline works.

Weakness
1. My main concern is on the claim of "momentum tracking is invariant to the data heterogeneity". In fact, this is not a new result in the decentralized learning. Many algorithms including D2 (https://arxiv.org/pdf/1803.07068.pdf), DeTAG (https://arxiv.org/pdf/2006.08085.pdf), DSGT (https://arxiv.org/pdf/1909.02712.pdf) are able to achieve this with only gradient tracking type methods. The difference is that these existing rates will depend on the $\zeta$ at step 0, which is the norm bound on the $$\zeta_0=\left\|\nabla F_i(x_i^{(0)};\xi_i^{(0)}) - \frac{1}{N}\sum_{j=1}^{N}\nabla F_j(x_j^{(0)};\xi_j^{(0)})\right\|$$. 
It seems Theorem 1 in the paper solves this problem with no $\zeta_0$ shown, but it appears that $u_i$ and $c_i$ are initialized to $\zeta_0$. This raises two problems: 1) if $u_i$ is initialized to $0$ (which is usual in practice), it appears the obtained rate is no better than the rates given in the previous works mentioned (D2, DeTAG, DSGT); 2) In practice, it is generally impossible to initialize $u_i$ to such quantity, unless all the nodes all reduce their first gradients.
2. In the experiments, I believe the authors should also compare with the naive baseline where gradient tracking and local momentum are enabled. This way, we can know clearly whether it is tracking or momentum (local) that makes the difference in small-class case.
clarity,_quality,_novelty_and_reproducibility: Please refer to the previous section.
summary_of_the_review: I believe the paper has good topic and some interesting analysis, but I'd like to inquire the authors on the main claim they make (momentum tracking is invariant to the data heterogeneity); please refer to the weakness 1 for details.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a novel momentum-tracking algorithm, whose convergence rate is independent of data heterogeneity. This paper present this algorithm is independent of data heterogeneity for any momentum coefficient. Moreover, experiments find this paper presents a stronger method than previous decentralized learning methods.
strength_and_weaknesses: I'm not an expert in this domain. In fact, one of my major research fields is tracking (object tracking). And object tracking almost has nothing to do with momentum tracking. Honestly speaking, I might not have enough skills to assess this paper. However, I hope my comments can help the authors improve their paper. Moreover, it is exciting to me to see if my assessments could align with the professional optimization people because my engineering practice might also benefit the theory design.

Strengths:
1. It seems that this paper has novel components than previous optimizers. And theorems prove that they are robust to data heterogeneity.
2. In general this paper is well written and it is well grounded to the literature. I think all the relevant papers have been referenced and discussed.
3. The experimental results seem promising and somewhat support the authors' claims that this optimizer can augment the decnetralized learning algorithms.

Weaknesses: 
1. I do not think this paper is targeting an important enough and the improvements seem incremental. In other words, they do not highlight the core problems in the baselines. Since the data heterogeneity is already well studied. In the introduction, the authors' writing flow seems to be this:
(1) Neural networks are powerful and decentralized learning is studied due to privacy concerns.
(2) DSGDm can work poorly because the parameter drifts away easily.
(3) Gradient tracking literature does not consider the data heterogeneity problem.
However, I do not think the above can lead to the major contribution to this paper and the improvements seem to be A+B (decentralized learning + gradient tracking stuff). It is unclear how important the proposed approach is in the context of decentralized learning literature. Is it a simple borrow from the standard optimization to the decentralized learning setting? If so, how to reflect the novelty and significance of such a borrow? Would it lead to a major change in the results?
Besides, it is unclear to me how important the data heterogeneity problem is and what the key insight for this solution. There is only one sentence in the introduction talking about the technical part of momentum tracking. It seems the technical contribution is weak.
2. The experimental results are not convincing. Can the proposed approach work on ImageNet? Can the proposed optimizer work for object detection and other tasks? I also think the CIFAR10's experimental results are not convincing enough. For the 10-class cases, the training curves almost overlap with the baseline. The 4 class cases are not as important as the 10 class cases in my point of view. In other words, the improvements are not so significant.
clarity,_quality,_novelty_and_reproducibility: In general, I think this paper is clear, and reproducibility can be ensured because the method is not difficult to implement.
However, I do not think this paper is particularly novel, and it seems that this paper only applies the tracking tricks in optimizers to the decentralized training setting. 
As for the overall quality, I think it is weakly below the borderline. The ICLR should have more novel or more significant papers.
summary_of_the_review: Seems to be an incremental application of the momentum tracking technique.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper introduces a momentum tracking approach for decentralized learning. The authors prove a theorem demonstrating that the proposed method is insensitive to data heterogeneity. They compare momentum tracking to other baseline methods, including gradient tracking, DSGDm, QG-DSGDm, and DecentLaM, through experimental evaluation.

The reviewers raised concerns on technical novelty and the significance of the results, especially on the dependence factor of hetergeneity. The concerns remain after the discussion. The paper can be significantly improved if it highlights the technical contributions and make in-depth comparison with existing work.
justification_for_why_not_higher_score: NA
justification_for_why_not_lower_score: NA
Decision: Reject