Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The manuscript imposes a SMOL framework during training in an attempt to curtail energy consumption of deep learning models and inference time of deployed models by:

- Accounting for task objectives and 
- Quantizing parameters into low-precision numeric representations 
- Instantiating network weights with full-precision values and auxiliary parameters i.e., learnable noise magnitudes;
- Further, employing above mentioned noise magnitudes as differential proxies for discrete bit precisions which further replace noisy weights (post training)

thus yielding heterogeneous mixed-precision allocation across parameters on ResNet-20 and DCGAN (CIFAR-10), ResNet-18 and ResNet-50 (Imagenet) with benchmarks such as weight quantized CNNs for image classification.

Further exploration of Layer-wise precision allocation and learning structured precision maps are presented in the appendix. 
strengths_and_weaknesses: **Merits**:
- The paper addresses an interesting and _relatively_ less explored problem i.e., lower-level, mixed-precision numeric representations within machine learning. 
- The intuitions regarding precisions and noise perturbations are well-explained! 
- The paper additionally accommodates pruning/zero-precision weights.
- Formalization of precision/quantization have been addressed in Appendix.

**Demerits**:
- Claims regarding **energy efficiency, inference time, storage/memory considerations** have not been quantified/mathematically demonstrated/estimated. Provided results (table 2 and table 4) are incremental at best.
- There is no attempt towards **comparisons with standard/widely-used metrics** such as [float16*](https://en.wikipedia.org/wiki/Bfloat16_floating-point_format). The results are not demonstrated on standard networks such as VGG/CNN etc. 
*There is a mixed-precision version available as well.

__Minor Corrections:__

- Section 1, Introduction, line 58, the our -> our*
- Section 1, Introduction, line 59, The Appendix provide -> The Appendix provides*
- Section 4, Results, line 187, based on our preeliminary experiments -> based on our preliminary* experiments
- Table 3, Image Generation on CIFAR-10, needs to include the “* represents results with Zero Precision Allocation.” 
- In general, highlighting SMOL’s results (through bold font etc.) in tables, helps. 
- Supplementary material, Section A.1: Preliminaries, Line 463, the reviewer believes this should be: L($\theta$) = 1/N $\sum_{i=1}^Nl(f_\theta(x_i, y_i))$ where $(x_i, y_i)_{i=1}^N$ is a fixed dataset*. 
- Supplementary material, Section A.2: Stochastic Approximation, Point 4, via performs truncation -> via truncation* (?)
questions: - The authors are further requested to comment on the **computational efficiency** of the proposed method.
- The claims regarding **higher generation quality** of DCGANs with SMOL has not been algorithmically justified. Why does lower precision representation yield higher generation quality while the loss of the network does not change significantly?  
- Supplementary material, Section A.2: Stochastic Approximation, Line 531, what is the **interpretation** of function R?
- Optionally, comment on how the proposed method could potentially **generalize to large language models** (multiplicity of inputs), if at all?
-------------------------------------------------------------------------------------------------
**Additional Note Post-Rebuttal**:
Score has been bumped from _5: Borderline accept_ (Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation) to *6: Weak Accept* (Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations).

 

limitations: - Inability to represent zero-precision weights has been addressed. 
- Computational Complexity is a potential limitation that needs to be addressed (see details above)
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This submission proposed to allocate quantization precision for each parameters. It formulated precision allocation as the magnitude of disturbance into parameters: the larger magnitude the parameters can endure, the lower precision should be assigned. During training, a penalty to constraint allocation within budget is added, and the estimated disturbance limit is estimated. 
strengths_and_weaknesses: Strengths:
1. It is interesting to interpret precision allocation as maximum disturbance in parameters estimation, and the formulation is straightforward.


Weaknesses:
1. The writting is confusing, please refer to the Question part.
questions: 1. The author mention allocating a precision to each parameter. What is the meaning of assigning precision to each parameter instead of to a layer where parameters within share a same precision ? In my understanding, learn precision in quantization is to fully utilize computational and storage budget: some layers require more precision (quantization bits) to maintain performance while others not. And the quantization's advantage in inference takes effect only when parameters use a same precision in a layer (i.e. in one matrix multiplication). Please explain how the learnt precision is applied to **each** parameter. And if so, how the advantage of quantization is realized.
limitations: N.A.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a technique for dynamically allocating quantization precision for neural networks. Their approach works by randomly generating a uniform noise tensor of the same shape as the parameters, and rescaling the noise by a per-parameter scalar. Based on the tolerable perturbation, it is possible to deduce what the appropriate precision should be. They provide results on CIFAR and ImageNet with standard models.

strengths_and_weaknesses: I like this work, although I have a few concerns. It's well written and I like the approach overall.

Strengths

I do find the approach intuitive and satisfying. Unifying quantization with pruning is interesting from a theoretical perspective. I do think this work could inspire many new ideas in this community.

The results are good too, although it would be good if the main paper had more. Too many interesting results are shoved in the appendix (more on this below)

The approach is simple enough to be relatively easy to implement

The paper is well written. Figure 1 is very good :-)

Weaknesses

The most obvious issue with this work is that main setup presented is per-parameter precisions. This is not a realistic setup in the real world, and is not hardware efficient. The authors do consider layerwise precisions in the appendix (although more experiments would be desirable). Why are these not in the main paper? You have half a page left!

The evaluation only considers ResNets. Some other standard models would be useful. For example MobileNet/EfficientNet family are a good one to check, since they are difficult to quantize. Following this line of thought, what about other models (Transformers?)

Not enough discussion on activation quantization is given in the main text. Some is provided in the appendix. Quantizing to 4 bits is fine, although it might be interesting to consider other values. Some discussion of how to extend the proposed approach to activations would be nice to read (note: I am not actually asking you to do it -- just some comments).
questions: See previous section.

I'm happy to increase score if you make some progress on them.

=== Post rebuttal:

these have mostly been answered and I'm happy. Score will be increased.
limitations: The authors check this box on the checklist but I am not convinced there is a clear description of the methods disadvantages. It would be nice to add a summary (I certainly won't penalise you).

==== Post rebuttal

discussed below
review_title: null

Meta-review: metareview: The paper addresses an important yet less explored problem (optimizing numeric precision per parameters/activations in architectures) with thorough and impressive experiments over multiple architectures (ResNet and Transformers in the rebuttal) and different tasks (ImageNet and machine translation). Reviewers unanimously thinks the paper should be accepted. 
confidence: Certain
Decision: Accept