Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: My review got deleted because the title kept creating an unexplained error.
Here's another shorter attempt

I haven't really followed along the literature for this. But from the results, it's not immediately clear to me what practical setup this is useful in. The authors assume perfect network synchrony, they have a 25% overhead on TensorFlow and they have a comparison to another algorithm that operates under different assumptions.

Who would ever use this and why? What's the plan for getting data to the untrusted workers?
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper considers distributed stochastic gradient descent, where some (unknown) compute nodes may be unreliable. New heuristics for filtering out replies from unreliable servers are introduced alongside a new protocol that helps keeping nodes in sync.

In general, I miss a more clear indication of how the individual contributions are different from other methods. I am also missing more detailed ablation studies showing which of the new ideas contribute the most to efficient learning. As far as I can tell, the experiments do not really show an improvement over existing methods in this domain.

This is not my area of expertise, but I cannot recommend the paper for publication in its current form as
(a) it's not clear to me that the paper improves on existing methods, and
(b) it's not clear to me what the real novelty of the work is.

Post-rebuttal:
I acknowledge the response of the authors. They clarified some aspects for me, and the paper appears to have improved over the rebuttal period.
I did not change my rating, but I want to emphasize that this is only because my knowledge of this field is so limited. My rating is largely based on "gut feeling" rather than actual knowledge, and I won't argue against acceptance.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper introduces an algorithm to build distributed SDG-based training algorithm that are robust to Byzantine workers and servers.

I am not very familiar with this area of research, but I feel the authors did a good job providing clear explanations and introducing all the relevant concepts needed to understand the proposed algorithm. Overall, I found the paper an interesting read.

The experimental section of the paper is lacking in some aspects:
- One of the main ideas introduced in the paper is that of filters to check the legitimacy of models from model servers. While these ideas are sensible from a technical point of view, I feel the experimental section is not properly demonstrating all the robustness claims made in the paper. For example, in the beginning of training with high learning rates the models will change a lot, are these filters effective in this situation as well? How are these filters working in terms of false positive/negatives in the experiments?
- How are models corrupted during training? What's the performances of the filters with different corruption techniques (e.g. adversarial attacks)?
- What's the impact of the choice of T in the experiments?
review_title: Official Blind Review #2

Meta-review: This paper is concerned with learning in the context of so-called Byzantine failures. This is relevant for for example distributed computation of gradients of mini-batches and parameter updates. The paper introduces the concept and Byzantine servers and gives theoretical and practical results for algorithm for this setting.

The reviewers had a hard time evaluating this paper and the AC was unable to find an expert reviewer. Still, the feedback from the reviewers painted a clear picture that the paper did not do enough to communicate the novel concepts used in the paper.

Rejection is recommended with a strong encouragement to use the feedback to improve the paper for the next conference.
Decision: Reject