Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work studies the problem of distributed training with large batches in a communication-bottlenecked setup, where regular versions of algorithms such as LAMB become a constraint. Authors propose a new algorithm, which compresses the gradient momentum before aggregation and then reconstructs the gradients to recover the scaling coefficients for LAMB. This idea allows authors to achieve faster convergence compared to naive compression for LAMB while maintaining higher communication efficiency than a non-compressed version.
main_review: Strengths:
* In general, the results demonstrated in the work show measurable improvements in efficiency over the baselines: the gains of 2-3x might be quite significant for practical scenarios of large-scale learning. Authors also compare their method with several reasonably chosen baselines to demonstrate the importance of their contributions: while there definitely could be more pretraining tasks and models to be considered, given the space limits I believe the current scope of experiments is sufficient to justify the claims made in the paper.
* Moreover, the theoretical analysis of the algorithm allows to validate its scalability properties and to establish convergence rates mostly equivalent to those seen in previous works on distributed optimization.

Weaknesses:
* In the text, I could not find any information about the actual compression algorithm used for experiments. I think this is a quite important part of the algorithm which should be described in detail, since both the empirical results and the analysis (Assumption 1.3) rely on it.
* Although the simplicity of the method is definitely a plus, some parts of the contribution of this work appear to lack any significant novelty. For instance, it was not clear to me how the primitive described in Section 5 differs from regular All-Reduce and whether this contribution is solely a software engineering one (that is, porting the existing protocol from MPI to NCCL). The method also seems to generally follow from 1-bit Adam, with the exception of a gradient recovery mechanism (although authors indeed demonstrate that direct application of the techniques from prior work yields worse results).
* The presentation of the method can be slightly improved: for instance, is it necessary to mention a server as a separate entity if in practice all communication happens by means of All-Reduce?

Questions:
* Do I understand correctly that Algorithm 1 mostly reduces to 1-bit Adam if we remove all lines required to update $c_t^{(l)}$ (in particular, lines 15–20) and change LAMB to ADAM for the preconditioning phase?
summary_of_the_review: Although this paper proposes a straightforward yet definitely useful method for speeding up distributed training, in my opinion the clarity of presentation both for method and results can be improved. I give a score of a weak accept for now, with possible improvements if authors address the issues outlined in the review.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose 1-bit LAMB, which combines lamb optimizer with communication compression, with convergence analysis and experiment results.
main_review: The paper is well-written. The idea of 1-bit lamb is well-supported by the convergence analysis and good empirical results. In overall, I think this is a good paper.
Here are my concerns:
1. All the experiments are limited to BERT, which makes it unknown whether the proposed algorithms could be applied to other training tasks including cv and nlp. To justify the effectiveness of the proposed algorithm, I recommend to add at least 1 CV task, such as resnet on cifar100 or imagenet.
2. My major concern is in the theoretical analysis. Some part of the theoretical analysis is confusing to me. I cannot find the definition of $V_{max}$, so I simply guess it is the maximum value of $V$. However, since the diagonal elements of $V$ are actually $1/v$, $V_{max}$ is actually assuming a lower bound of the gradient $\|\nabla f(x)\|$. Such a lower bound seems unreasonable to me since the convergence of the algorithm actually means that $\|\nabla f(x)\|$ should approach 0, which contradicts the lower bound of $\|\nabla f(x)\|$. Could the authors resolve such an issue? 
summary_of_the_review: The paper is well-written. The idea of 1-bit lamb is well-supported by the convergence analysis and good empirical results. In overall, I think this is a good paper. Some other experiments is recommended. The main issue is in the assumption of the theoretical analysis.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a communication-efficient distributed LAMB optimizer with 1-bit compression. It follows previous work to first warm-up the variance, but proposes to inference the scaling factor based on reconstructed variance. Experiments show training speedup due to communication compression. The proposed 1-bit LAMB achieves similar model performance to full-precision LAMB.
main_review: Strengths

1. An interesting insight that the norm of LAMB's variance is more unstable than Adam

2. Comprehensive experimental results to validate the claims, including the motivation and proposed method.

Weaknesses

1. Given the prior work 1-bit Adam, the novelty becomes somewhat incremental. The novel part is that 1-bit LAMB will try to adjust the scaling coefficient of LAMB. The efficient communication of momentum follows existing communication-efficient momentum SGD method.

The reconstruction of the gradient means that other compressors (e.g., Top-K) may not be incorporated into the proposed method.

2. Does 1-bit LAMB preserve the same performance as LAMB with different number of workers and batch size? It would be great to have a table summarizing the performance with different #worker and batch size settings. Figure 3 only gives the training throughput.

3. The theoretical analysis does not seem to validate the coefficient scaling in 1-bit LAMB. It may need further heuristic and theoretical elaboration.

4. The 3rd assumption in the theoretical analysis does not seem very common to me in communication-efficient distributed training. The author needs to validate it. A more commonly-used assumption about the compressor is $||C(x)-x||^2 \leq (1-\delta)||x||^2$, from which we may know more about the effect of the compression ratio. Therefore, the convergence analysis of 1-bit LAMB seems less solid to me.
summary_of_the_review: The paper propose a communication-efficient large-batch LAMB optimizer to accelerate distributed training. The experimental results looks promising, but there could be some improvements both empirically and theoretically.
review_title: null

Meta-review: The authors propose a communication-efficient distributed LAMB optimizer using a 1-bit compression. This work is similar in spirit to other prior work, eg 1-bit Adam. Although the algorithm works reasonably well it is a bit unclear how much compression is achieved. Overall, the algorithmic novelty is limited, given the prior work, and the benefits of the algorithm don't shine through as the experiments are quite limited in their data sets and models. The theoretical results are also of unclear usefulness due to the assumptions made.
Decision: Reject