Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a novel algorithm to deal with Byzantine resilient decentralized optimization for data training. 
strength_and_weaknesses: + The algorithm dose not rely on additional graph connectivity
+ The performance is better than some existing algorithms

- The algorithm does not guarantee accuracy of the optimization process
- The idea behind the algorithm is unclear. In particular, the Clipping idea involves a parameter \tau but its role is not well explained and illustrated. 
- The paper has an adaptive selection process for \tau_i, but it cannot be computed only rely on local information. In particular, it involves \delta_max, which seems not an available information. 
clarity,_quality,_novelty_and_reproducibility: The work idea is original.  In general, the paper is easy to follow, except for the \tau issue. 
summary_of_the_review: The paper proposes a novel algorithm to deal with Byzantine resilient decentralized optimization for data training, which has better performance compared with some existing works. However, the idea behind the algorithm is not clearly explained. In particular, the role of the \tau parameter and the adaptive selection process for \tau_i is unclear to me.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This submission discusses using the technique ClippedGossip in the decentralized optimization setting to counter potential Byzantine agents in the network. 

The authors provide convergence guarantee in terms of first-order condition to show the attainment of consensus in the network, and give numerical experiments to show the effectiveness of the ClippedGossip technique in providing robustness in several attack/data heterogeneity settings.
strength_and_weaknesses: Strength:

-> The paper is well-structured and the narrative flows finely.

Weakness:

-> There can be further clarification about the experimental settings.

-> Some proof steps do not naturally connect. Specific comments are in the following section.

-> I suggest the authors submit core code for numerical experiments.
clarity,_quality,_novelty_and_reproducibility: Regarding theory/statements/proofs:

-> Lemma 3. It should be $\forall i,j \in [n-b]$

-> Please justify Lemma 4.

-> In subsection E.1, bullets 4 and 5 have the same descriptions.

-> Page 27, Lemma 8 proof.  Applying inequality "(16)" to the last term, not inequality (18).

-> Page 29, Lemma 10 proof, I do not follow the second inequality when bounding $A_1$. Where does that square come from, and how does $\tau_i^{t+1}$ get into the argument? 

-> Page 30, proof to theorem I': I do not understand why it is reasonable to apply Lemma 10 with $\eta = 0$. When the step size is 0, no update happens. As this bound needs to hold for every $t$, this step does not seem reasonable.

Regarding experiments:

-> It is not obvious that how the spectral gap $\gamma$ can be explicitly fixed or controlled. Could authors explain this issue?

-> In terms of comparison methods, I wonder if authors have thought of gradient tracking, which is a potential practice for decentralized optimization (ref. [A general framework for decentralized optimization with first-order methods by Xin et al., Proceedings of the IEEE (Volume: 108, Issue: 11, November 2020)]). 

-> It is not immediately obvious to me how aggregators are implemented in this case. In the centralized setting, aggregators are usually applied on the server. I wonder if authors apply the aggregator on every node when they iterate?
summary_of_the_review: There are several places where further clarification about theoretical proof and numerical experiments can be helpful for me to determine the correctness of the submission. It is not immediately clear to me that the selected numerical baseline is the most appropriate baseline, and I would like to examine the proof closer.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies byzantine robust decentralized learning. In this setting, a fraction of workers is byzantine. First, the paper argues why the consensus is vulnerable to byzantine attacks, then they propose DISSENSUS, a decentralized attack to steer away from the true consensus. Finally, they come up with a robust called aggregation self-centered clipping which converges to a neighborhood of a stationary point. 
strength_and_weaknesses: -Weaknesses
1)The aggregation strategy proposed in the paper has been proposed before and is not a novelty of this paper. Moreover, the idea of centralized clipping is not new, for example, paper [1] looked at this idea before in distributed setting, and this paper is an extension of that to the decentralized setting.
[1]Karimireddy, Sai Praneeth, Lie He, and Martin Jaggi. "Byzantine-Robust Learning on Heterogeneous Datasets via Bucketing." arXiv preprint arXiv:2006.09365 (2020).
-Strength
1)The reviewer did a good job reviewing relevant papers.
2)The paper does a good job of motivating the problem.
3)The experimental results are sound.

clarity,_quality,_novelty_and_reproducibility: The only problem with this paper is contributions are only marginally significant, and the assistance of this paper in comparison to [1] is not substantial.
[1]Karimireddy, Sai Praneeth, Lie He, and Martin Jaggi. "Byzantine-Robust Learning on Heterogeneous Datasets via Bucketing." arXiv preprint arXiv:2006.09365 (2020).
summary_of_the_review: The aggregation strategy proposed in the paper has been suggested before and is not a novelty of this paper. But, the authors provided a study of this method in a decentralized setting and did a good job comparing it with previous methods and providing experiments that support their idea. Compared to the previous versions of the paper, the authors add more comparisons with previous results in control/optimization literature. In summary, the strengths outweigh the weaknesses.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work studies clipping of gossip style algorithms for decentralized optimization problems in order to mitigate the effect of byzantine agents. Convergence analysis is provided and experiments corroborate the merits of the proposed approach.

Strengths are in establishing robustness to byzantine agents.

Weaknesses are in failing to rigorously contrast with prior approaches to establish robustness to byzantine agents, and that many of the technical results presented here are already known or exceptionally similar.
justification_for_why_not_higher_score: NA
justification_for_why_not_lower_score: NA
Decision: Reject