Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies the problem of quantized training where the gradients are quantized as well in addition to quantization aware training methods. The paper builds up on an interesting observation related to the detrimental effect of batch normalisation in quantized training. It is shown that batch norm amplifies the accumulated gradient quantization noise during the backpropagation. Based on this observation, a rectification method is proposed to reduce the negative effect of accumulated gradient quantization noise.
strength_and_weaknesses: Strengths:
- The paper is easy to follow and well-written.
- There has been a lot of research focused on quantization aware training methods in the literature but this paper aims to solve the challenging task of quantized training. It is an important problem to increase training efficiency and this paper makes a good attempt at it.
The proposed method achieves improvements consistently on multiple datasets.

Weaknesses:
- The computational overhead of the approach has been shown to be not significant but it would be useful if the authors put some empirical comparisons on training time of the proposed method vs the baselines.
- In table 1, the results for Mobilenet-V2 are considerably worse than UI8 and DAINT8. Can the authors explain the reason why the proposed method is worse in that setup?

clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is easy to follow and clearly written.
Quality and Novelty: Technically solid and proposed approach is novel to the best of my knowledge. The empirical results are ok but not too strong.
Reproducibility: I did not find the code with the paper. I would suggest the authors to release the code for the sake of reproducibility during the review process.
summary_of_the_review: Overall the paper makes a good attempt at the problem of quantized training and the proposed idea seems novel as well as technically solid to me. Though the results are not particularly strong. Still I would recommend weak acceptance at this stage.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors consider fully quantized training (FQT), a subfield of quantization-aware training (QAT), where even the gradients are quantized during backprop. A common issue in FQT is that training can be very unstable. 

The authors study the role of batch normalization (BN) in causing training instability, and they identify BN as a significant culprit. Hence, the authors propose adding a regularization term in the loss function of models using BN, and they empirically demonstrate their approach's effectiveness.
strength_and_weaknesses: ## Strengths
The authors propose a solution to a relevant problem, stabilizing FQT, which should be of interest to many people in the ML community. The proposed solution is simple to implement and is intuitively clear: the regularization term they propose doesn't allow the normalizing standard deviation term in BN to become "too small" for an appropriate notion of "small", which prevents instability during training. Finally, they compare their proposed approach on a reasonable selection of benchmarks and verify their method's effectiveness from different angles.

## Weaknesses
 - Simply by virtue of the extremely sloppy mathematical setup, Thm 1 cannot possibly be correct. The theorem makes a statement about the variance of deterministic quantities. Halfway through the proof in the appendix, the authors suddenly assume that these quantities are random and state assumptions on them. I am not claiming that the statement of Thm 1 is "morally wrong" because perhaps it can be shown to be correct with a proper setup. However, as the authors present it, it is uninterpretable.
 - Given how well the authors' solution appears to perform on the benchmarks they consider, I think it would be very useful to show the performance gap between models trained with their method or that are trained without using quantization.
clarity,_quality,_novelty_and_reproducibility: The quality of writing is low, with many typos and hard-to-parse sentences. The general presentation of the paper needs significant improvement. Some of the more notable issues are:
 - "On the other hand, theoretical calculations on the BitOps computation costs..." - I am unfamiliar with BitOps; is it some benchmark? In any case, it should be clarified what these are or at least a reference would be needed.
 - "Empirical data1 shows backward sometimes even costs more in practice." - I presume the authors mean backdrop.
 - In Eq 1, the $\mathbf{g_{x_n}}$ terms are undefined. Are they $g_x = \nabla_x \hat{x}$? Also, the second and third terms involving the sums contain mistakes, as the sums use $i$ as an index, but this index is not used in the summands. What is the relevance of the earlier defined $y$ symbol?
 - In Eqs 1 and 2, what is the difference between $g$ and $g_{x}$? Is $g$ the gradient operator, and $g_{x}$ is the gradient evaluated at $x$?
 - In Eq 2, I would avoid using $\Delta$ as it is usually reserved for the Laplacian operator.
 - "Gradients play a crucial role in back-propagation based optimization and make a huge impact on training stability and convergence speed." - I don't disagree with this statement, but I wonder if the authors might actually be referring to the error in the gradients.
 - "As the bit-width decreases, the quantization noise injected in error signal at each single layer increases exponentially." - This is a very confusing sentence, as it is unclear what the authors mean. In the following sentence, they explain that the error accumulates multiplicatively, and in the sentence after that, they finally explain what they mean by the exponential increase.
 - Figure 2 is not great at demonstrating the claimed exponential increase. I suggest the authors redesign it to plot bit width against gradient error since that is what they are analyzing.
 - The colour scheme in Figure 3 is confusing, especially the colour gradients for the $\delta$ terms.
 - The "Prior efforts reducing variances in backprop" paragraph in Section 5 should be moved (perhaps even verbatim) to the related works section, as it doesn't pertain to the proclaimed content of the section at all.
 - The symbol $D$ is used in the text as the "statistical variance" operator, but the authors' whole mathematical setup is deterministic, so it is unclear what they mean.
 - The font sizes of the labels of Figures 3 - 6 should be increased as they are currently difficult to read.
 - X-axis labels are missing in Figure 3
 - Table 3 is not refereced in the main text.
summary_of_the_review: The authors propose a solution to a relevant problem and demonstrate its effectiveness empirically. However, the paper suffers heavily in terms of clarity of writing and mathematical formalism. Most importantly, it contains mathematical statements that are, at best uninterpretable and flat-out wrong at worst. Hence, the paper is unpublishable in its current form.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The author studies the batchnorm's negative effect when training with quantized gradient. To address the problem, the author proposes to add an additional loss to control the variance of batchnorm. Empirical results show improvement when quantizing weight, activation and gradient.
strength_and_weaknesses: Strength:
1. The observation of the batchnorm's negative effect on gradient explosion seems novel.
2. The empirical results are comprehensive.

Weaknesses:
1. Does theorem 1 take weight and activation quantization into consideration? Why is the result not affected by the quantization?
2. Corollary 1.1: should be not greater -> should not be greater
3. Please provide a formal definition of D in Theorem 1.
4. Eq. (4): why should $\sigma_l$ approaches $\sigma_*$? Theorem 1 seems to suggest that $\eta$ should be as large as possible.
5. From Table 4, the performance is extremely sensitive to the choice of $\lambda$, considering the gap in Table 1 and 2 is not very obvious compared to $\lambda$. Therefore, repeated experiments should be conducted for fair comparison in Table 1 and 2.
6. How does it work for training full-precision networks with quantized gradients?
clarity,_quality,_novelty_and_reproducibility: The motivation seems novel, the empirical results are comprehensive, and the paper is easy to follow.
Codes are not provided, so the reproducibility cannot be confirmed.
summary_of_the_review: The authors make a novel observation that batchnorm may contribute to gradient explosion in FQT. But I think more comprehensive theoretical and empirical results may be needed to make it more convincing. The method proposed seem to be too sensitive to the choice of the new hyper-parameter $\lambda$, which makes it hard to validate its better performance since the improvements in Table 1 and 2 become negligible compared with the variance in Table 3.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: ### Decision

The paper addresses an interesting and challenging problem and demonstrates experimental results that appear promising.
The paper received low or borderline initial scores. The main reason for rejection by reviewer FcHZ seems to be the lack of clarity and rigorousness. The author's rebuttal addresses this to some extent. The reviewer GegD was not convinced by the paper, expecting a more comprehensive theoretical and experimental analysis. The average rating implies rejection. I have elaborated on two issues in detail the "AC comment" below, which are in my opinion serious and also imply rejection.

justification_for_why_not_higher_score: Serious flaws
Decision: Reject