Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors proposed 0/1 Adam to reduce communication during training. 0/1 Adam addresses the existing issues by adaptively freezing variance and linearly approximating momentum and parameter update locally. The authors provide theoretical convergence guarantee and experimental results on large-scale training tasks, showing that the proposed optimizer can achieve similar accuracy compared to fp16 training at a higher throughput. 
strength_and_weaknesses: Strength:
1. Firstly, the paper is generally well-written and easy to follow. 
2. The evaluation is comprehensive. The authors provided experimental results on language model training (transfer learning and zero-shot evolution) and ImageNet classification, where the proposed optimizer can match the performance of fp16 Adam. 
3. The proposed method achieves higher training throughputs across different benchmarks and hardware setups, which seems to be pretty general.
4. The motivation of the method is 

Weakness:
1. The adjustment of hyper-parameter T_v and T_u seems a bit non-trivial (especially the latter one). Can the author confirms that the same hyper-parameter can scale across model scales without careful tuning? For example, does the same hyper-parameter scale to training on the CIFAR dataset with the same learning schedule?
clarity,_quality,_novelty_and_reproducibility: The clarity and quality of the paper are satisfying.  The method contains enough technical novelty to outperform existing work (1-bit Adam). The reproducibility should be good since the code is released and integrated into open-source repos. 
summary_of_the_review: Generally, I think this is a good paper. It contributes to the distributed training area by reducing communication to increase throughput. The method outperforms existing work by further reducing the high-precision training steps. My only concern is the difficulty of hyper-parameter adjustment.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper introduces 0/1 Adam, a variant of the 1-bit Adam algorithm.  While the 1-bit Adam algorithm leveraged a fixed burn-in period wherein vanilla Adam was run before the variance term stabilized and was frozen, 0/1 Adam proposes an adaptive variance-freezing policy.  Additionally, 0/1 Adam leverages local SGD wherein synchronicity rounds among workers are skipped.  Convergence results similar to 1-bit Adam are provided for 0/1 Adam, and experimental results showing the efficacy of the algorithm for Imagenet and BERT are provided.
strength_and_weaknesses: Strengths
-The paper extends 1-bit Adam in a natural and intuitive way.  Both adaptive variance-freezing and local SGD are promising directions to improve the original 1-bit Adam algorithm
-The results are compelling

Weaknesses
-Presentation of results could be significantly improved (see clarity section for further information).

-The writing and description of the novelty of the presented work requires extensive reevaluation.  The description of 0/1 Adam as a novel algorithm is not correct, it builds upon 1-bit Adam (also borrowing a large amount the analysis from the corresponding paper) and this point should be made clear throughout the paper (however, adaptive variance freezing is an important and novel contribution to the 1-bit Adam algorithm). Furthermore, 0/1 Adam is not the first work to consider local SGD for distributed algorithm.

-Even after reading the paper, it is unclear how adaptive variance-freezing or local SGD (i.e., skipping rounds/local steps) may be set in practice.  Indeed, these two policies must be known \emph{before} running Algorithm~\ref{1}.

-Claims are made without substantiation or are misleading.  E.g.:

         1) "the change of variance over steps in Adam is generally smooth" <- Please substantiate this claim.  In the 1-bit Adam paper, they 
              assumed this was the case and stated so explicitly.

         2) "While this paradigm avoids compressing non-linear information with a one-time frozen variance, the experimental results from 
             (Tang et al., 2021) indicate the fullprecision stage still incurs non-trivial overhead." <- Overhead discussed in the paper was with 
              regards to vanilla Adam.  Please remove this claim or provide additional evidence to substantiate it

         3) "Furthermore, 1-bit Adam is restricted in the scope of gradient compression, and cannot be trivially adapted when other 
              techniques are used, such as local steps." <- In the 1-bit Adam paper, several variants were 
              implemented and contrasted against, including compressing the gradient and compressing the momentum.  Why can it not be 
              trivially adapted?  Furthermore, how to leverage local SGD with Adam is not an open problem, previous work in [1] has tackled 
              this. 1-bit Adam could trivially be combined with CADA.  Please reassess, and support, your claim.

         4) "Besides, the empirical success of (Tang et al., 2021) was not substantiated on generative models (GPT-style models), for instance, 
              175B GPT-3 Brown et al. (2020), 530B MT-NLG Smith et al. (2022), etc." <- This is disingenuous; 1-bit Adam was evaluated on 
              GANs (generative model) as well large-scale models (BERT).  530B MT-NLG Smith et al. was also 
              published after the 1-bit Adam paper, and neither GPT-3 no MT-NLG are evaluated using 0/1 Adam.  I would recommend revising 
              this claim, as well as elaborating why BERT is not a significant enough benchmark that GPT model evaluations are required.

References:
[1] Chen, Tianyi, et al. "Cada: Communication-adaptive distributed adam."
International Conference on Artificial Intelligence and
Statistics. PMLR, 2021.
clarity,_quality,_novelty_and_reproducibility: Clarity:
Presentation of results has significant room for improvement and requires more explanation.  For instance, it is unclear exactly what is being demonstrated in Figure 5: is "w/o Skipping Round" equivalent to w/o Local Steps?  What are 1.2x/1.3x denoting?  Similarly, in Figure 2 (a-c), what is being denoted by 1.6x, 2x, and 1.4x?  One suggestion is to describe each figure and the corresponding results after they are presented.  As it stands, Figures 2-5 all appear before a single figures' results are described in the text.

Quality:
Writing quality can be significantly improved.  Furthermore, comparing against Adam in all results would greatly improve the paper.  E.g., "Since Tang et al. (2021) already demonstrated that 1-bit Adam has similar statistical results to Adam, we omit the comparison of end-to-end model accuracy to Adam for brevity." <- This is unreasonable; 1-bit Adam has been shown to provide near Adam accuracy, and so the comparison between the two may be ommitted.  However, the newly described algorithm (which is not equivalent to 1-bit Adam) has not been shown to achieve similar performance to Adam, and thus this must be demonstrated.  For the purposes of the paper, including Adam as a baseline in Figures 4 and 5 would not overcomplicate these results, but would significantly strengthen the paper.

Novelty:
0/1 Adam is an extension of the 1-bit Adam algorithm.  Additionally, much of the motivation and theoretical analysis strongly follow the 1-bit Adam paper.  The paper needs to do a better job of establishing that 0/1 Adam builds on this previous work, what exact theoretical results were presented in the 1-bit Adam algorithm and how the presented theoretical results stand as novel contributions, and exactly what are the novel contributions being made.  E.g., "We propose 0/1 Adam, a novel optimization method that addresses the non-linearity challenges in Adam when applying aggressive 1-bit quantization and local steps" <- disingenuous to call 0/1 Adam novel, but rather "0/1 Adam builds upon the 1-bit Adam algorithm."  Furthermore, the use of local SGD/steps for Adam has been previously addressed by the Cada algorithm of Chen et al.

Reproducibility:
The 1-bit Adam has been open-sourced in a deep learning library by the authors.  However, the presented experiments are non-trivial, and providing experimental code to reproduce the featured results would greatly help other researchers to further build on their work.
summary_of_the_review: The paper makes some natural extensions to the 1-bit Adam algorithm, while providing convergence guarantees for the resulting 0/1 Adam algorithm.  Furthermore, the results seem compelling.  However, the presentation of the paper's novelty and contributions requires serious reevaluation.  Also, the general presentation and writing has significant room for improvement, particularly discussion of the presented results.  Important key details regarding figures are lacking or insufficient to understand exactly what is being stated.  Finally, while the extensions made from 1-bit Adam to produce 0/1 Adam make intuitive sense, it is unclear how exactly these are (or can be made) adaptive in practice, particularly given their predefined values in Algorithm 1.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: With the deep-learning revolution, large-scale training has been at the forefront of innovation. As the data sizes grow, the training time and cost explode. There have been techniques such as compression and bit quantization to alleviate these problems. These techniques work well in linear information but not in non-linear information. The paper proposes a new method called 0/1 Adam, leveraging insights within updates and stationary via steps. 

strength_and_weaknesses: They provide convergence guarantees with theoretical analysis and experimental results comparing 1-bit Adam on language modeling training and image-net training showing a much faster reduction in training loss in both cases. The authors also open-sourced their algorithm. 

One missing piece in this setup is the generalization, I think. I believe the experimental setup might be decent, and the problems are large enough to demonstrate efficacy. But I am not sure about the broader applicability to various other applications of AI, such as siamese networks (wide-and-deep models) or GANs, where the scale might come from a different angle than language models.
clarity,_quality,_novelty_and_reproducibility: The paper is written well. The experimental setup and parameter settings seem to be decent and explained thoroughly. I liked the discussion about policy, convergence speed, and quality analysis. Novelty is not impressive but decent and impactful enough. With the given implementation and parameter details, it seems that there is no reproducibility issue. 
summary_of_the_review: The paper focuses on an important problem to speed up model training. Though there are some minor limitations on generalization, I think it is good enough to be represented in the venue. I wanted to vote 7/10 (but only 6 or 8 are available).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: **Summary:** This paper proposes a couple of improvements to one-bit Adam, which is a popular way to reduce the bandwidth requirements of training large language models. Specifically, the paper proposes (1) to freeze the variance over short intervals as opposed to one-bit adam's technique of freezing it permanently after a certain point in training and (2) to do local updates taking advantage of this frozen variance. This also eliminates the warmup period during which one-bit adam must maintain adam state at full precision.

**Strengths:** The technique is a nice incremental improvement on one-bit adam that further reduces bandwidth requirements, and - in bandwidth-restricted settings - training time. It appears to work quite well, producing similar model performance to one-bit adam.

**Weaknesses:** The paper introduces new hyperparameters that need to be set. This may be annoying for one-off runs, but I suspect that it won't be too hard to find good settings before large hero runs. It would be nice if the paper showed how to efficiently find hyperparameter values here, or even explored whether values found in small-scale models (e.g., 125M GPT model) are applicable to larger-scale models (e.g., 1B GPT model). It would be nice to clarify whether these costs will make the technique difficult to use in practice.

**Suggestions for improving the paper:** The paper could really benefit from having a standard adam baseline as well for context. There's no reason _not_ to have it. The paper could also use error bars and multiple replicates - some of these differences are potentially in the noise. I acknowledge that these are potentially experiments, but I implore the authors to make their best effort here so that the paper is easier for future researchers to build on.
justification_for_why_not_higher_score: It's a productive but incremental result on top of a popular paper. It's not earthshattering, and probably doesn't merit extra attention in the broader community.
justification_for_why_not_lower_score: The paper is a productive improvement to one-bit adam that practitioners will probably find useful. I could see it being sufficiently incremental to merit rejection if absolutely needed.
Decision: Accept: poster