Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper summarizes a technique to allow quantizing a weight-sharing superset to with flexible quantization policies to allow choosing different network architecture and quantization policies for desired efficiency-accuracy trade-off. The key observation of this method is that quantizers needs to estimate the maximum and minimum values of a activations. However, the extreme values of activations are dependent on the activation tensor's size. During training, the activation size changes as we sample different architectures, so the extreme value do not converge when using EMA estimator, and the quantizer become unstable during training.  To fix this, this paper proposes a simple method to estimate the extreme values (therefore quantizer parameters) based on current batch statistics. This leads to better stability and performance. 
main_review: Strength:
+ This paper proposes a method that effectively combines superset-based architecture search with mixed precision quantization in one-shot, allowing more flexible network configuration for different accuracy-efficiency trade-off.
+ This paper is established a convincing loop from observation (input size difference -> unstable quantizer parameters) to techniques (use batch statistics) and results (improved performance). The story is very clear. 

Weaknesses:
- The mathematical derivation in Equation (3, 4) needs improving. E.g., what is the "z" variable in Equation (3)? It is not clearly defined. Also, according to Equation (4), since E(M_n) ~ log (n), seems that the expectation will saturates if n (number of elements in activation) is large enough? Does this imply that if we only sample layers with large activations, we can somehow bypass this problem? 
- Any explanation why Equation (8) is the only working estimator? 
- MFLOPs calculation questionable. In L308, this paper states that the FLOP operation reported by this paper is computed by mn/a/64. I understand that it is convenient to use one single metric to evaluate the efficiency, but this should not be called as FLOP, which exactly mean floating-point operations. 
- Configurations of the optimal networks? It will be valuable to share the configurations of Pareto-optimal network configurations, which is not available in this paper. 

Overall this is a neat paper. 
limitations_and_societal_impact: No need for discussion.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Authors propose a robust quantizer formulation BatchQuant, which utilizes the batch statistics to provide a more stable quantizer for different quantization levels. With benefit of BatchQuant, they train a compact, single-shot, mix-precision, weight-sharing supernet for NAS. They evaluate BatchQuant on ImageNet.
main_review: The paper is well written,

In neural architecture search, it is very challenging to include the mixed precision quantization as a part of policy, since it enlarges the search spaces and increases the quantization instability during the training. BatchQuant helps to alleviate the  instability introduced by quantization and enables to train a large supernet considering different quantization levels of each layer.

BatchQuant estimates the max and min of activations using the current batch statistics and also learns two scalars gamma and beta for quantization. The method is solid and simple.

In experiment, they evaluate on ImageNet and compare with SPOS, APQ, OQA methods. BatchQuant achieves better performance in terms of FLOPS and Top 1 acc. 

For rebuttal,
1. Is there any other dataset or other task to evaluate the method on? It is good that when BatchQuant evaluates on other datasets as well
2. Authors choose [2, 3, 4, 32] as bitwidth choices. Can authors elaborate the reason for choosing these bitwidths? It seems there is a huge gap between 32 bits and 4 bits.
3. Even the authors claim that the re-training is not required. But if we re-train the networks, will the performance still get improved?


limitations_and_societal_impact: Yes, they have mentioned and discussed the limitation in Sec 6
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposed to conduct quantization-aware NAS: in the meanwhile of neural architecture search (basically it is searching subnet within a supernet), it searches for bitwidth allocation for different layer. To stablize the training for quantized activation, it proposed BatchQuant, following similar method of batch normalization.
main_review: 1. The protocol of NAS follows previous work, which is less novel. The proposed BatchQuant is interesting: Basically, quantization-aware NAS is adding quantization bit search into NAS. However, BatchQuant is clamined to be necessary for quantization-aware NAS. 
2. Can BatchQuant be applied on normal network quantization training ? If possible, are there any experiments?
3. MobileNetV3 is used as spernet, following the convention used in previous work. Is other network possible for supernet? 
4. In Line 297-298, accuracy predictor is trained using quantized subnets, how are subnets processed to be inputs for predictor?
limitations_and_societal_impact: N.A.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper presents the use of batch statics to help stabilize the NAS+Quantization process. It is based on the one-shot NAS idea, but applies the batch idea to assist the quantization policy search during the Neural Architecture Search. The results are compared to some alternative methods and show a faster convergence.
main_review: Strengths: 
* The problem tackled in the paper, NAS+Quantization combination, is challenging. 
* The idea seems to help speed up the convergence.

Weaknesses:
* Incremental novelty.
* No theoretical analysis on the effects of the introduced technique.
* The evaluation is limited.

Solving the two problems (NAS+Quantization) in a combined manner is challenging. The area has however been explored by prior studies. The proposed idea of using batch statistics to help stabilize the process appears to be a small change rather than anything fundamental. The paper gives no theoretical analysis of the effects of the technique. So it is hard to tell whether the technique is always going to help. A strong comprehensive evaluation could have mitigate the concern. But the evaluation in the paper is quite lightweight, leaving the practical impact of the work unclear.
limitations_and_societal_impact: The reviewer didn't find discussions of the limitations of the work or societal impact in the paper. The limitations would be good to add, while the societal impact isn't that important to have for the nature of the work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a combination of neural architecture search and per-batch quantization statistics calibration in order to quantize neural networks to low bit-width using mixed precision. The proposed NAS method is applied to ImageNet and pareto-optimal networks in the accuracy vs complexity space are found.
main_review: Overall an interesting idea and a generally well written paper. Some questions/comments that I have:

I wish to point out that in eq. (4) $\frac{1-u^N}{1-u} = \sum_{k=0}^{N-1}u^k$ for $u\in[0,1]$. Therefore, we get $E[M_N] = \int_0^1 \frac{1}{\lambda} \sum_{k=0}^{N-1}u^k du = \frac{1}{\lambda} \sum_{k=0}^{N-1}\int_0^1u^k du = \frac{1}{\lambda}\left(1+ \sum_{k=1}^{N-1}\frac{1}{k}\right)$. The conclusion that follows eq. (4) doesn't change much but the correction should be made nonetheless. I also checked reference [38] and did not find a justification for modeling activations as being Laplacian distributed. Can we get some comments on the validity of this assumption. Otherwise, section 3.3 is a good one?

Regarding BatchQuant itself (section 3.4), I am having a hard time understanding how it differs from standard QAT approaches that observe the max value in the instantaneous tensor (such as DoReFaNet). This seems to be what BatchQuant is doing based on the description in lines 202-203. In BatchNorm, there is a moving window that averages mean and variance of the batches temporally and then uses the converged values post-training for inference. That doesn't seem to be what is going on here. Instead, it appears that simple per-tensor scaling is employed. Can we please get some clarification?

Regarding the use of NAS, is there a specific contribution by which mixed-precision is made possible? The introduction and abstract seem to suggest so, however, the description in Section 4 indicates that this is done simply by using prior arts. Either of the two description needs to be corrected for exactness and consistency.

Post-rebuttal: I thank the reviewers for their responses. I am willing to keep my score of a weak accept. But I wish to ask that the authors double check all the math (and crucially its exposition) as well as the details of BatchNorm vs BatchQuant. Ideally, there should be no room for a doubt.
limitations_and_societal_impact: Yes, there is a nice discussion section on that on Page 9.
review_title: null

Meta-review: This paper combines the idea of mixed precision quantization and single shot NAS method, and proposed a single-shot weight sharing quantization method that requires no training during search. 3 out of 4 reviewers acknowledges that the idea is solid and simple. Concerns remain about the novelty. This paper is recommended for acceptance given the effectiveness and satisfying result. 
Decision: Accept (Poster)