Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The premise of this paper is that quantization plays an important role in the deployment of deep neural networks; ie in the inference stage. However, errors due to quantization affect different neural architectures differently. It would be useful if we could predict ahead of time which models are more amenable to quantization. I think this is a very interesting premise and the paper is very well motivated.

The paper is also very clear and well written, making the claims precise and backing these up with experiments.

At the heart of the paper is the replacement of floating point numbers with inexact values, which are treated as random variables and defined precisely in equation 4. This definition enables the authors to apply Monte Carlo methods to obtain network predictions as shown in equation (10) and figure 2, and subsequently carry out sensitivity analysis. The experiments show that a measure of sensitivity (K) is indeed a good augmentation to cross-validation for model selection for the purpose of trading-off accuracy and resource consumption when launching deep neural networks with floating point rounding errors.

One question I have for the authors is the following: There has been a large body of literature on Monte Carlo methods for Bayesian neural networks. Could those works have something to say in addressing some of the challenges posed in Section 4.1? 

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The authors propose a scalable method based on Monte Carlo arithmetic for quantifying the sensitivity of trained neural networks to floating point rounding errors. They demonstrate that the loss of significance metric K estimated from the process can be used for selecting networks that are more robust to quantization, and compare popular architectures (AlexNet, ResNet etc.) for their varying sensitivities.

Strengths:
- The paper tackles an important problem of analyzing sensitivity of networks to quantization and offers a well-correlated metric that can be computed without actually running models on quantized mode
- Experiments cover a wide range of architectures in image recognition

Weaknesses:
- The proposed method in Section 4.2 appears to be a straightforward modification to MCA for NN
- Experiments only demonstrate model selection and evaluating trained networks. Can this metric be used in optimization? For example, can you optimize for lowering K (say with fixed t) during training, so you can find a well-performing weight that also is robust to quantization? 1000 random samples interleaved in training may be slow, but perhaps you can use coarse approximation. This could significantly improve the impact of the paper. Some Bayesian NN literatures may be relevant (dropout, SGLD etc). 

Other Comments:
- How is the second bullet point in Section 4.1 addressed in the proposed method?
- Can you make this metric task-agnostic or input-distribution-agnostic (e.g. just based on variance in predictions over some input datasets)? (e.g. you may pick a difference loss function or different test distribution to evaluate afterwards) 
- Does different t give different K? If so, what’s the K reported? (are those points on Figure 3)?
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Summary:

The paper studies the sensitivity of a neural network with respect to quantizing its weights and activations. The idea is to use Monte Carlo Arithmetic (MCA) in order to calculate the number of significant bits in the training loss (e.g. cross entropy) that are lost due to floating-point arithmetic. The results show that the number of significant bits lost correlates with the reduction in classification accuracy when quantizing the weights and activations of the neural network.

Decision:

Overall, this is an interesting paper with interesting results. However, I think there is considerable room for improvement, and that more details are needed in order to assess the significance of the results, as I detail in the rest of my review. For these reasons, I recommend weak reject for now, but I encourage the authors to continue working on improving the paper and to provide more details in the updated version.

Contribution:

The paper considers an important problem, that of quantizing the weights and activations of a neural network in order to reduce computational and memory cost, while maintaining the machine-learning performance as high as possible. 

In my opinion, the main contribution of the paper is the experimental findings, and in particular that the sensitivity of the training loss with respect to the precision of the weights and activations correlates with the accuracy of the network. It seems to me that these results may relate to work on Bayesian neural networks, sharp vs flat minima, and minimum-description length approaches to variational inference. Work on these areas has also shown that sensitivity of the training loss with respect to the precision of the weights (which intuitively happens when the network is at a "sharp" local minimum vs a "flat" one) is related to poor generalization performance, and vice versa. I would encourage the authors to explore the potential relationship of their work with these areas, and possibly discuss them in an updated version of the paper.

Originality:

The paper describes a method for assessing the sensitivity of a neural network with respect to the precision of the weights and activations. The method is a straightforward application of Monte Carlo Arithmetic (MCA) to neural networks. I believe that the application of MCA to neural networks for this particular purpose is novel, and that the results are original. However, the introduction of the paper gives the impression that the proposed method is brand new, and even uses the acronym MCA to refer to the proposed method, which can be confusing to readers. I would suggest to the authors to rewrite the introduction so as to reflect more accurately that the contribution is not a brand-new method, but rather the application of an existing method (MCA) in a novel way.

Writing quality:

The paper is generally easy to read, but there is considerable room for improvement. There are mistakes, and often the writing is sloppy and imprecise. I give some more specific suggestions on what to improve later on.

Technical quality:

The method is well motivated and the experiments seem reasonable. However, there is very little detail on the experiments, which makes it hard to assess their correctness/significance. I would suggest to the authors to rewrite the experimental section with full detail, or put more details in an appendix. In particular:
- Is each Monte Carlo trial done on the same batch of training images or a different one? If different, how are the trials averaged, and does that mean that the standard deviation over trials also includes a contribution due to different batches?
- In sections 5.1 and 5.2, how were the results for different t combined/aggregated? Did you use linear-regression analysis as in section 5.3?
- When you say "accuracy", do you mean accuracy on the training set, validation set, or test set? This is particularly important for assessing the significance of the results, and is something that is currently missing from the description of the experiments.
- How was the quantization of the neural networks performed? It would be good to explain this at least on a high level, in addition to citing Wang et al., (2018).
- In section 5.2, how was the model selection for each method performed exactly? In the baseline method, was the model to be quantized selected based on validation performance before quantization or after quantization?

Specific suggestions for improvement:

The citation format, i.e. (Smith et al., (2019)), is unusual and uses unnecessarily many parentheses. Use \citep for (Smith et al., 2019), and \citet for Smith et al. (2019).

The illustration of fig. 1 is not fully convincing as a motivation for floating-point arithmetic. Even though it makes the case that Float(7, 7) is more efficient than Float(8, 8) and Float (9, 9), the comparison between Float(7, 7) and Fixed(12, 12) is hard to interpret, as we can't conclude whether the efficiency gain is due to reducing the number of bits or to switching from fixed-point to floating-point arithmetic. A more convincing illustration would compare fixed-point with floating-point arithmetic using the same number of bits.

It would be better if fig. 1 were 2D, as 3D doesn't add anything but makes it harder to compare sizes visually.

Please avoid exaggerations, such as "exquisitely sensitive" or "extremely sensitive", when "sensitive" would suffice.

Section 2 is grammatically sloppy:
- arihtmetic --> arithmetic
- Last line of page 2 seems to be missing a verb.
- this has lead --> this has led

The related-work section is too short and in many cases it doesn't explain what previous work has actually done. For example, "rounding of inexact values to their nearest FP approximation has been studied in several publications" is vague: what exactly these publication have done? This lack of detail makes it hard to assess the originality of the current paper, and how it differs from existing work.

Section 3 is often unclear with imprecise mathematical notation:
- "e is the base-2 exponent of x in binary floating point arithmetic": surely, the exponent is represented as an integer?
- (bs, be1, be2, ..., bex, bm1, bm2, ..., bmx) is sloppy, as it indicates that the indices run from 1 to x.
- Bx = sx + ex + mx  is also sloppy; what is meant here is the number of bits to represent sx, ex, mx and not the values themselves.
- F(x) = x(1 + θ), shouldn't θ be δ?
- "which is typically the cause of horrific numerical inaccuracy from numerical analysis literature", the phrase "from numerical analysis literature" doesn't make much sense here.
- In eq. (4), substituting the expression for x from eq. (1) doesn't yield the same result.
- "The number of trials is an important consideration because [...] it can produce adverse effects on results". What is meant by "adverse effects"? Do you mean that with few trials Monte Carlo doesn't give accurate results? Please be more specific.
- "we can determine the expected number of significant binary digits available from a p-digit FP system as p ≥ −log2(δ)". I'm unable to follow this statement, please explain further. Also, from applying logs to δ ≤ 2^{−p} one gets an inequality that doesn't match the one in p ≥ −log2(δ).
- "The relative error of an MCA operation is, for virtual precision t, is δ ≤ 2^-t", "is" is used twice here.
- "the expected number of significant binary digits in a t-digit MCA operations is at least t", operations --> operation. Also, shouldn't it be at most t, otherwise K becomes negative?
-  is discussed in the section --> is discussed in the next section.

Some mistakes in section 4.1:
- y = (x; w) --> y = f(x; w)
- Eq. (8) is sloppy, it uses X for both the set and its size. Use |X| or something similar for the size.
- In the caption of fig. 2, baes --> base

I'm not convinced by the second bullet point in section 4.1, that the averaging over many images used to obtain the accuracy is the reason why MCA doesn't work well. Surely, the training loss (cross entropy) is also an average over many images? To me it would seem more plausible that the main reason MCA works with training loss but not accuracy is because accuracy is discrete, whereas training loss is continuous.

Fig. 3 would be much easier to read if the axes were labelled, and if the axes had the same range (so that different plots can be compared visually).

Fig. 5 would be easier to read if the networks were sorted with respect to K.

In section 5, CIFAR-10 is sometimes written as CIFAR10.

Appendix A is empty, so it should be removed.

review_title: Official Blind Review #3

Meta-review: The paper studies the impact of rounding errors on deep neural networks. The                                                       
authors apply Monte Carlos arithmetics to standard DNN operations.                                                                 
Their results indeed show catastrophic cancellation in DNNs and that the resulting loss of                                         
significance in the number representation correlates with decrease in validation                                                   
performance, indicating that DNN performances are sensitive to rounding errors.                                                    
                                                                                                                                   
Although recognizing that the paper addresses an important problem (quantized /                                                    
finite precision neural networks), the reviewers point out the contribution of                                                     
the paper is somewhat incremental.                                                                                                 
During the rebuttal, the authors made an effort to improve the manuscript based                                                    
on reviewer suggestions, however review scores were not increased.                                                                 
                                                                                                                                   
The paper is slightly below acceptance threshold, based on reviews and my own                                                      
reading, as the method is mostly restricted to diagnostics and cannot yet be used                                                  
to help training low-precision neural networks.
Decision: Reject