Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: 
This paper introduced a log-barrier based regularization method to reduce the dynamic range of data types (activation, weight, error, gradient, and input) in neural networks. The authors claim that such regularization is important to avoid overflow or swamping in the accumulation of matrix multiplication. 

However, the reviewer is afraid to think that there are serious technical issues with this claim. Here are a few details regarding this concern.

- The main claim of this paper seems to be that by reducing the dynamic range of the neural network data, they can avoid overflow. However, it does not seem to be (always) true; what matters more would be the resolution of information you need (or want) to keep, and the number of data elements you accumulate. E.g., assume that we use 8-bit -- even if the magnitude of data is constrained to be around 2^-3, if the resolution of info is 2^-8, there will be an overflow after 8 times of accumulation. Also, note that the magnitude of activation or weight of a layer is often not important due to scale-invariance of the normalization techniques (like batchnorm). Thus it is not clear if constraining the magnitude of data is always beneficial.  

- The authors claim that previous reduced-precision training techniques (like [Sun et al., NeurIPS19]) have dynamic quantization range across the layers. This is NOT true, since the reduced-precision floating point format has the fixed dynamic range; e.g., [Sun et al., NeurIPS] employs (1-5-2) format for representing back-prop error, implying the fixed dynamic range of 2^(-15) -- 2^(16). In fact, one very compelling benefit of using the reduced-precision floating-point over the fixed-point format is that the floating-point format does not need scaling of the magnitude across the layers. 

- It is not clear if the authors' understanding of "overflow" or "swamping" is the same as what is reported in prior work (e.g., [Sun et al., NeurIPS19]).  First of all, overflow and swamping issues are different - the former issue is for the fixed-point accumulation where the accumulated value wraps around to the small value when it exceeds the largest value representable by the given accumulation precision. Whereas, the latter is for the accumulation of floating-point numbers, where the small magnitude value is ignored (or truncated) when it is added to the large magnitude sum. Thus, it is very confusing when the authors use these two different terms interchangeably in the text (e.g., the 3rd paragraph of Intro). 

- There are many confusing points in the analysis of the overflow in Sec 3. First of all, the authors seem to claim that Prob(z > 2) is the overflow condition... but why? I couldn't find the notion of bit-width in this analysis... but then how can we check the overflow?

- The authors mention that they employ STE, which does not make much sense in case of the quantized training; what does it mean to take STE for the quantization of back-prop error?

- It is not clear how the proposed method is evaluated. The computational graph in Fig 2 does not seem to include the accumulation quantization. Also, more detailed information should be provided about the following; how the MU8 format is implemented to use 8-bit hardware? how the proposed method is implemented in the deep learning framework?


To sum up, this paper seems to have several technical flaws, which should be carefully addressed. 

review_title: Not sure if this paper is technically sound

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper is addressing an important and challenging problem of end-to-end training of deep nets in fixed-point, in this case, with 8-bit precision. A good solution to this problem can have a major impact on the deployability of deep nets on embedded hardware. The basic idea is to introduce an additional term (the log-barrier constraint) in the loss function to constrain the allowable range over which model parameters are allowed to take values. The authors use of mu-encoding to assign non-uniform quantization levels to minimize the quantization error. The main results are in Table 2 showing that the method eliminates overflow in practice and allows quantized networks to approach the accuracy of full-precision networks on the MNIST, CIFAR-10 and ImageNet.

A few comments/questions:

1) How was the quantization range ([-u,+u]) chosen? There will be a trade-off between the precision (8-b vs. 6-b) and the quantization range u. Is there any notion of optimality of the chosen value of u for 8-b quantization.
2) How important is mu-law coding? How much accuracy is lost when uniform coding within the quantization range is applied? Please note, doing arithmetic on non-uniformly quantized variables is harder than with uniform coding, which has a bearing on hardware realizations. It will be good to quantify the accuracy loss in the absence of mu-law quantization.
3) The results in Table 2 are sparse. It further indicates that the accuracy of MobileNet using the proposed method is quite significantly lower (66% vs. 72% for Sun) compared to previous works. MobileNet is designed for embedded IoT type hardware. 8-b or even 9-b quantized MobileNet results would put this work firmly in the domain of end-to-end fixed-point training of lightweight networks and hence make it more useful. These results will strengthen Table 2 significantly.
4) Please comment on the choice of hyperparameters and initialization. Is this method robust to these choices?
5) How does 8-b fixed-point compare with say MiniFloat-8 or equivalent? Such low-precision floating-point realizations are an attractive choice and a competitor to fixed-point.  
6)The paper is missing comparisons with a couple of highly relevant papers on fixed-point training listed below:

[1] Zhang et al., Fixed-point Back Propagation Training, CVPR 2020.
[2] Sakr et al., Per Tensor Fixed-Point Quantization of the Back Propagation Algorithm, ICLR 2019.

Overall a very nice paper but needs more work (indicated above) to strengthen the results.

review_title: The paper is addressing an important and challenging problem of end-to-end training of deep nets in fixed-point, in this case, with 8-bit precision. The results promising but sparse. The work needs additional results/comparisons before it becomes ready for prime time.

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper presents a set of well crafted regularization techniques that improve the accuracy of QAT.

The overview of related works, particularly QAT is very well done. We clearly see what the present limitations are. I would like to point out that there have been works that have analytically determined suitable fixed quantization ranges in the context of backprop. The authors claim otherwise, I would suggest including works such as [1] in section 2.2, and revisit that claim.

The proposed method method is very nice, Lagrangian optimization techniques are used to ensure weights do not fall into the tails of the distribution. This reduces the probability of overflows. 

I wish to point out that the term 'swamping' does not describe an overflow phenomenon per se, it is rather an underflow that occurs when adding two very different floating-point numbers. Avoiding tail samples is definitely useful to reduce swamping, but some of the wording in section 3.3 and in the introduction probably ought to be revisited. It is not a very serious issue, but the clarity in that regard could be improved. Otherwise, the analysis in Section 3 is excellent.

One question about eq. (9): the tail distribution is defined as P(z>2) (subscripts aside for convenience). Shouldn't it be P(|z|>2)? Won't large negative numbers be problematic as well? In the grand scheme of things this won't affect the conclusions, but I think this correction (or at least a comment) should be made for extra rigor. 

The experimental results are good. I wonder if the authors could spare a few sentence to discuss how they simulated accumulation quantization. This issue is not as straightforward as representation quantization that would apply to activations/weights/gradients. The cited papers that have looked at swamping previously (Wang & Sakr) do mention something about modifying the deep learning framework at the GEMM level. Is the same technique used?

One question about a statement made in the conclusion: Throughout the paper, a clipping level of 2 has been chose, but in the final discussion, it is stated that the weights are contrained to the range [-1.5,2.5]. Why the disparity? Is this an error or something expected? If so, can it be explained?

Minor issue:
It is stated in section 3 that \phi_t approaches the indicator function when t approaches infinity. How so? I get that z<0 -> \phi_\ifty = 0 but z>0 is not well defined. Maybe the definition of \phi for z>0 can be explicitly stated.

[1] Sakr, C., & Shanbhag, N. Per-tensor fixed-point quantization of the back-propagation algorithm. In 7th International Conference on Learning Representations, ICLR 2019.


=======================================================================================================
Comments Post Rebuttal:
I still find the technical contribution of this paper to be a good one. However, As stated in my original review, there were some clarity issues with the manuscript. While, I personally was able to follow along, the other reviewers are correct in their claim that the writing can be quite confusing (Reviewer 4 makes a strong case). To that end, I have decided to decrease my score to a 5 because I can no longer say the manuscript is ready for publication. Nonetheless, I urge the authors not to be disappointed. The presented work has many merits, and I am sure that with a thorough "clean up" of the text, this paper can be a great one!
review_title: Log barrier extensions to improve QAT

final_score: rating: 3: Clear rejection
review_content: The authors introduce a log-barrier extension loss term enforcing soft constraints on the range of values to enable fully end-to-end quantization-aware training. 

Strengths of the paper: 

- The paper addresses an important topic, because there are increasing concerns in performing fully end-to-end low precision training to deploy on low-precision hardware. 
- The method has a practical goal and could be interesting for practitioners 

Weaknesses of the paper: 

- Lack of positioning with respect to the SOTA quantization-aware training(QAT) and post-training quantization(PTQ) schemes, there are plenty of missing related literatures on both quantization schemes. Some statements in the background and related work could be wrong. For example, QAT also focuses on efficient inference as well as PTQ. The levels of practical applicability of a variety of quantization solutions have been introduced in DFQ(Nagel etal., 2019). Survey on the related work is not sufficient. 
- Having a benchmark would be interesting if it will include some SOTA methods and evaluates with them. The comparison targets are mostly out-of-date. It is lack of convincing evaluation results to support the proposed scheme.
- Organizing the whole contents is ok but not good enough for the readers to easily follow and understand. 

Detailed comments: 

(1) The terminology on swamping might not be familiar with the ML community. Explaining the criticality of swamping problem is not good enough in the intro. You should provide how critical the problem is on the low-precision hardware with the other SOTA quantization schemes. For example, the probability of occuring swamping without applying the proposed scheme, etc.

(2) Evaluation results provided in the paper are just for comparing accuracy. Accuracy loss is intrinsic in fully end-to-end low-precision training. The benefits of employing the proposed scheme would be beyond accuracy, say memory or energy-saving constraints for on-device training. Experimental evaluations to support the necessity and merits of the proposed scheme should be provided. 

(3) The quantization range is fixed in the proposed scheme. Is it a merit that the proposed scheme does not need to adjust the range and precision either per-layer or per-channel during training as in other SOTA methods?

(4) Writing on the constrained optimization formulation is a bit verbose and not properly formulated. 

(5) Inducing the tail bound of distribution to demonstrate that the probability of swamping can be controlled, several assumptions and approximations have been applied for the worst-case upper bound. Are the assumptions reasonable to work in practice? For example, assuming that the weight distribution is Gaussian is too strong to be practical. 

(6) The paper has a conceptual overlap with other quantization approaches and some of the proposed scheme is not entirely novel resulting in a weak contribution. 

(7) In Table 2, the MobilNet has more severe degradation of accuracy than the ResNet on the low-precision(8-bit) setting. Could you explain why this happens?

Minors: 
- Several typos: There was been in p.2, to soft threshold the range of in p.3, theta-i in eq.(3), some more in p.7
review_title: Official Blind Review #4

Meta-review: This paper introduced a log-barrier based regularization method to reduce the dynamic range of data types in neural networks. As pointed out by the reviewers, there are many technical issues. The authors agree with the reviewers in the rebuttal, though claimed that they are fixed in the revised version of the paper.

Experimental results are not convincing. It is not clear how the proposed method is evaluated. Accuracy of MobileNet using the proposed method is quite significantly lower compared to previous works. The work needs additional results/comparisons with other highly relevant papers on fixed-point training.

There are also many clarity issues that need to be fixed.
Decision: Reject