Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents One Quantize for all framework. The framework claims to search for the network and the quantization without the need for retraining. 

Results are promising although it is not clear to me if the comparisons are fair (different bit size). 
Would also be great to see the resulting architecture and how it is different from other NAS approaches. It is not clear to me why the number of FLOPS is so low compared to related methods. 

In the experiments, I missed the computational cost for training and how it compares to the other approaches (as this paper suggests there is no need for retraining or fine-tuning). 


review_title: Jointly Network search and quantization outperforming state of the art results

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposed a method to train quantized supernets which can be directly deployed without retraining. The motivation is to have a supernet with a given quantization bit-width which only train once and can be deployed with different architectures (under different FLOPs budget). This paper made a bunch of experiments showing that the proposed once quantized for all method can find DNN architectures which have SOTA performance with low bit-width. The paper also shows that when training lower-bits supernet, it is helpful to use the weights from the trained higher-bits supernet.

Pros:

1. This paper targets a very practical problem that quantization is actually required for most resource-constrained devices. Recently, supernets (e.g. OFA, BigNAS) validate that it is possible to directly obtain DNNs with different FLOPs budget from a single big supernet without retraining. This saves lots of training time in the cases that one want to have DNNs with different FLOPs. Combining quantization-aware training and supernet is an effective approach to save training time when we want search different DNNs (with different size/FLOPs) which are quantized with certain bit-width.

2. The authors also show that it is very important to use the pretrained weights (from a trained higher-bits supernet) as initialization when training the quantized supernet. This observation is also meaningful for the cases that low-bits quantization-aware training is hard or unstable.

3. Using the proposed once-quantized-for-all method, the authors get several quantized DNNs which have SOTA results on ImageNet. The authors compared both "SOTA architectures + quantization aware training" and recent "quantization-aware NAS method". The accuracy / flops of the searched DNNs is better than the compared methods.

Cons:

1. It's natural to apply quantization-aware training on supernets when we want a supernet to be quantization-aware. Both quantization-aware training and supernets are ready-to-use techniques and the combination is straightforward. The benefit of bit-inheritance is a good observation, while using pretrained weights as initialization is kind of a common practice in quantization or model compression. At this point, the contribution in terms of the novelty is limited.

2. The proposed method can also outperform methods that can also search layer-wise bit-width (e.g., Table 2), although the method in this paper only uses the same bitwidth for all the layers. It's not clear what is the main factor in this comparison. Are all the methods using the same experiment setup? E.g., quantization algorithm (LSQ or min-max), architecture search spaces. It will be better to have an ablation study to understand which part of the proposed algorithm plays the key role to the better performance.


In general, I think this paper did a great job on the experiments of quantization-aware supernet, but the novelty contribution is slightly under the criteria of ICLR. So my rating is borderline. I hope the authors can give some response to the cons listed above, and I'd like to consider changing my rating if I missed something important.


review_title: borderline

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a new method to search for quantized neural networks. This method is different from others that it results in quantized weights which can be deployed without post-process such as fine-tuning. Proposed method first trains a 4-bit quantized supernet, and search for the best performance sub-net using the validation dataset. Then, the method initialize the 3-bit supernet using the 4-bit supernet, and trains 3-bit supernet using the knowledge distilation method. Proposed method iterates the initialization, training, and search process until the goal bit resolution is achieved.

I find that the idea of constructing quantized super-nets using Bit Inheritance is interesting and the paper is well written. However, I think more precise description of the training method and additional analysis is required to improve the paper.

1. In section 3.4, you described the K to K-1 supernet inheritance process as "During training, we use the K and K − 1 bit-width supernets as teacher and student, and train them in a knowledge distillation way to further reduce the quantization error between the K −1 and K bit-width parameters.". However, the knowledge distillation method is not specified after the statement. Is it similar to QKD? Or is it more of a traditional knowledge distillation approach? More precise description will be helpful.

2. In the introduction, you described that "... This two-stage procedure will undesirably increase the number of models to be retrained if we have multiple deployment constraints and hardware bit-widths...". However, there is no analysis on the search cost under such scenarios. Since the proposed method induces more supernet training process compared to NAS-then-Quantize approaches such as APQ, it is unclear whether your method will acheive lower search cost or not. I believe that additional analysis on the benefit of deploying the quantized weights without retraining in the mean of search cost must be given in the paper as one of the main contribution of the paper is that the proposed method allows the deployment without retraining.

3. Minor: In section 3.4, you described that "... where the parameters of the K − 1 bit network inherit from the parameters of the K − 1 bit network.". I think the later K - 1 must be K.
review_title: Interesting paper, but some clarifications needed

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:
	This paper performs a joint optimisation for DNN models, making the NAS scheme is aware of both the quantisation and architectural search spaces. The paper presented a large range of comparisons to different quantisation strategies and ran a lot of experiments to support their claims. However, the writing quality of this paper is worrying. Also, I am a little worried about the novelty of this paper.

Strength:
1. There are a lot of experiments with the proposed method, showing a great empirical value for researchers in this field. I consider results shown in Figure 2 and Table 2 very supportive evidence of the effectiveness of the proposed method.
2. It is nice to see a large scale study (1.5K architectures) on some common properties of network architectures and their interactions with quantisation.
3. To my knowledge, this paper does present a state-of-the-art number for low-precision ImageNet classification.

Weakness:
1. The writing quality of this paper is worrying. This is not simply to do with the use of language, but also on the clarity of some matters. I strongly recommend the authors to have a serious polish of their paper, since they do present valuable results and STOA numbers.
2. To me, the novelty of this paper is limited, it seems like an extension to Once-for-all, and the authors also cited this work. The teacher-student technique is also a published idea. The authors claim this is the first piece of work of NAS without re-training. However, they are iteratively reducing the bit-width K, which implies a large training cost and is somehow equivalent to re-training. The method in the paper looks like a combination of a number of well-known techniques, which might limit the novelty claim in this paper. However, I have to say I am not very troubled with combining a bunch of existing techniques if it show new STOA that is outperforming by a significant margin. This weakness is only minor to me.

My suggestions & confusions:
1. It seems like you can boost the performance of quantised networks from a) jointly search for architectures and quantisation and b) teacher-student alike quantisation training with inherited weights. Could you test these two parts in isolation and quantify the contributions of each technique?
2. Why you quantise activations to unsigned numbers (Page 4)? Don’t you consider activations like leakyrelu in your activation search space? Or you do not search activations at all?
‘... NAS methods suffer from more unreliable order preserving’, Who are you comparing to in this case? Is it more unreliable compared to RL based NAS?
3. What is your Flops reported in Table 2? Flops means floating point operations, do you mean bitops? or you somehow scaled flops with respect to bitwidths?
4. ‘we focus on the efficient models under one fixed low bit-width quantization strategy’ Do you mean the network is uni-precision? So no layer-wise mixed-precision is allowed?
5. I spotted a number of misused languages, and will strongly recommend authors to check mistakes like:
a) Ambiguity: 
 i) ‘with high floating-point performance’: do you mean floating-point models? Or you mean customised floating-point models? Describe floating-point as high is very misleading.
ii) ‘quantize the network with retraining’: I guess I understand what you mean, but you might say “retrain the quantised models” to be less ambiguous. 

b) Grammar:
i). ‘different bit-width’ -> ‘different bit-widths’
ii) ‘quantization supernet’ -> ‘quantized supernet’ and so on.

c) Do not assume readers have prior knowledge:
i) ‘we use sandwich rules’ -> ‘we use the sandwich rule’ and maybe you should consider explain what it is.

I cannot present all the mistakes here, these are just examples, I would iterate again that I would strongly recommend you to polish the paper since I do like the results you are presenting and think if the code is open-sourced, they will benefit the community.






review_title: The paper shows promising results, but needs further polishing

Meta-review: This paper proposed a method to train quantized supernets which can be directly deployed without retraining. A main concern is that there is limited novelty. The proposed method looks like a combination of well-known techniques. Experimental results are promising. However, it is not clear if the comparisons are fair and if all the methods are using the same setup. It is desirable to have additional analysis and ablation studies. The writing can also be improved.
Decision: Reject