Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a novel integer training method and shows that it is effective to train deep learning models with integer-only arithmetic. More specifically, the proposed method is designed to address and tackle the common challenges observed in integer gradient computation: how to guarantee that the gradients are properly scaled, the numerical error is  unbiased and the training is distribution independent.
strengths_and_weaknesses: Strengths

* The paper is well-written and the structure is clear.
* Instead of doing quantization, the proposed method directly switches to the number representation of computations and forms a fully integer training pipeline, which is quite novel to me.
* The paper provides a clean theoretical analysis and its motivation is clearly explained in the contribution section.
* The empirical results look good. Compared with baseline, the integer arithmetic performs well.


Weaknesses

* Though the paper is targeted for the experts with quantization computation experience, it would be nice if the authors could elaborate the background and introduce more related work to help the readers to understand the general setup and goal of this area. For a machine learning conference lots of researchers are not that familiar with low-bit integer arithmetic. 
* The experiment setup is a little bit too simple. It seems the authors only compare their method with the baselines but not other state-of-art quantization or integer arithmetic training methods. The lack of comparison make the results much less convincible.
* I believe part of the motivation to introduce the integer training pipeline is to reduce the memory complexity and speed up the training. However, I could not see any reports reflecting on these aspects.
questions: I have listed most of my concerns in previous section and there are several additional questions that might help improve the score.
* Though the authors claim the pipeline is easy to implement and the method is hardware-friendly, it is not that obvious to me as I found the process could be quite complex in practice.
* For Appendix A.2, the bound from equation (18) could become arbitrarily large as $M^q$ and $M^q_v$ are now dependent on the norm of $X$. I am not sure if the bound still makes sense in that case. Maybe there should be some additional assumptions required.
limitations: It seems there is no negative social impact of the work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a fixed-point based model training pipeline for deep neural networks. It uses linear fixed-point mapping and non-linear inverse mapping to convert the numbers (weights, activations, gradients) during training to fixed-point and back. It conducted theoretical analysis on the noise introduced by the quantization process and the convergence speed with SGD. Experiments are done with several tasks: classification, detection and segmentation. Models include Resnet18, MobileNetV2, ViT-B, DeepLab and FasterRCNN.
strengths_and_weaknesses: Strength:
-	The proposed method achieves pretty impressive results. On ImageNet, the average performance drop for a int8 training pipeline is only around 0.5%. On semantic segmentation, performance drop is around  0.2% and on detection, drop at around 0.3 – 1%.
-	Comprehensive analysis is provided on the impact of int8 training for the SGD convergence, gradient noise and loss landscapes. Although the analysis is rather simplified (e.g. only strong convexity is discussed on the loss landscape part), it still provides interesting insight to the process.
Weaknesses:
-	There are some existing work on using integer arithmetic for training deep models such as (NITI: Training Integer Neural Networks Using Integer-only Arithmetic. TPDS). There is no directly comparison with these methods. The NITI work also uses stochastic rounding which is similar to this work.
-	The authors claim to be using “int8” integer operations for the computation. However, this is misleading since a lot of computation are not “int8”. For example, the multiplication of two int8 numbers/vectors will be using an int16 accumulator. The SGD part is also using int6 arithmetics (line 266). For ViT, the attention computation which contains softmax layers seems not to be using int8 operations (it is not described in the paper).

questions: Other issues:
-	On line 33: shouldn’t int8 range be [-127, 128]?
-	Figure 2. On the second block, the right side should be sign2, exponent 2 etc.
-	Table 2&3, the decimal places are not unified.
Overall, my major concern of this work is the over-claim of "int8" pipeline where a lot of operations are not actually int8. And how is the used pipeline different from existing works. More clarifications need to be made. The "int8" bragging and boasting needs to be taken down.
limitations: Not sure.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: * The authors have proposed a new integer arithmetic for neural network training. What's different from the previous work is that instead of using the scale from the linear floating point to integer mapping they used the maximum floating-point exponent as the scale. The mapping between the integer and floating point values is thus dramatically different from the existing quantization approaches. 
* Previous approaches tried to keep both the forward propagation and the back propagation entirely in the int8 domain, but the training suffered from significant accuracy loss. With the new integer arithmetic proposed in this paper which also keeps the forward propagation and the back propagation entirely in the int8 domain, a few experiments have been performed on vision models demonstrating that the new approach sacrifices negligible accuracy.
* Theoretical analysis has also been performed on the new approach.

strengths_and_weaknesses: Strengths
* The approach proposed is simple yet seems surprisingly effective in practice.
* The authors were aware of the limitations of the assumptions for the proofs. The proofs are relatively speaking clean.

Weaknesses
* This is not necessarily a weakness. As I mentioned in the following discussions, there are a few places in the paper that can be further elaborated and clarified to make the paper even better.

questions: * Line 33 why the dynamic range for int8 is [-127, 126] instead of [-128, 127]? Please put some explanations somewhere in the paper.
* When describing mantissa, please align with the IEEE standards. For example, for FP32 values, the bit-width of mantissa is 23 instead of 24. I understand that the authors are counting the leading hidden bit. But please describe it explicitly in the paper.
* There is a lack of clearance in explaining the stochastic rounding of mantissas, i.e., how the the 24-bit single floating-point mantissas are further rounded to 7-bit mantissas. If it is possible, please provide a few concrete examples in the paper, just like the one in line 147. The stochastic rounding is a key for the proofs shown later in the paper. If $\mathbb{E}\\{\hat{A}_i\\} = A_i$ cannot be well justified by the stochastic rounding, most of the proofs in the paper will be invalid.
* If shifting bits in Figure 1(a) is linear mapping, why the alignment module in Figure 1(b) which also shift bits is non-linear? The naming was a little bit confusing.
* Please also show a table of comparing the proposed approach with the previous "SOTA"s in the paper. I believe this is beneficial for the paper because previous work does not always stay in the int8 domain for training.
* Please also discuss about how even lower bit-width, such as 4-bit, will affect the training accuracy. Depending on the authors' implementation, it is possible to test it in the experiments as well.
* In section 4.1, assumption 2 (iii, b), is it possible that $M \geq 0$, $M^q \leq 0$ and $M + M^q \geq 0$? Same for $M_V$ and $M_V^q$. If it is possible, it might alter the remark 2 and 3. If it is not possible, please elaborate in the paper.
* Line 235, with "some" constant $c > 0$.
* The authors claimed that "by keeping the variance bound $M^q$ relatively small, we can theoretically achieve the original performance". How could we measure or ensure this for training different neural networks in practice? In line 53, the authors claimed that the optimality gap is only shifted by a negligible amount. I did not see the concrete amount and how the amount is measured.


limitations: * Although this work is targeted for neural network training, the inference arithmetic is slightly different from the existing popular symmetric quantization arithmetic, which makes most of the existing integer inference frameworks incompatible with the new approach. The author should discuss about the impact of the new approach to the existing inference infrastructures, since it will require new software to make it efficient.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper provides mathematical analysis and experimental results to demonstrate an INT8 neural network that are trained with int8 arithmetic. The proposed method is hardware friendly and does not rely on training data and weights distribution.
strengths_and_weaknesses: Strengths
1. The proposed method is interesting and the mathematical deduction is sound to me.
2. The results of the compact models are comparable with the FP32 models.

Weaknesses
1. Please compare with other INT8 methods instead of only FP32 models.
2. The title and abstract are misleading to me. There are also floating-point arithmetic in the model, so it is not integer arithmetic only.
3. I suggest to replace cifar10 experiments with cifar100. Though, ImageNet experiments can demonstrate the method's effectiveness on large datasets.
questions: N/A
limitations: Yes.
review_title: null

Meta-review: metareview: This paper proposes methods for using integer arithmetic to train deep learning models. reviewers arrived at a consensus to accept the paper. Concerns do exist on "The title and abstract are misleading to me. There are also floating-point arithmetic in the model, so it is not integer arithmetic only". Hope the author can fix it. 
confidence: Certain
Decision: Accept