Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:
- This paper proposes to consider the importance of each parameter in the post-training quantization of weights. The authors propose to use weight magnitude as the importance indicator and to minimize the weighted distance between the full-precision weights and quantized weights. Experiments are performed on various NLP models and tasks.

Strengths: 
- The paper is structured clearly and the proposed method is simple.

Weaknesses: 
1. It is not clear to me why the change in the loss in (7) and (8) is necessarily related to the magnitude of the weights. From (8), the loss change \delta L  is only related to the hessian, and each weight's perturbation, but not the weight's magnitude.

2.  The second concern comes from the novelty of the proposed method. Indeed, loss-aware (or importance-aware as in this paper) quantization which considers the quantization effect of each parameter to the loss has already been proposed several years ago in [1,2]. In [1,2], they also approximate the loss using the second-order Taylor expansion like in equation (7) in this submission. Moreover, the proposed importance-aware quantization solution in equation (5)  is exactly the same as equation (8) in [1], except that their importance is derived from the diagonal hessian instead of weight magnitude (which is not quite reasonable, refer to the first point).

3. One other post-training quantization method GOBO in [3], which also uses codes and codebooks to quantize language models, is also not compared. From their reported results, under the same number of bits, GOBO has higher accuracy than the proposed method. E.g., GOBO has 83.76% accuracy for the 3-bit quantization on MNLI  while the proposed method only has 82.9%.

4. From Table 2, there is no winning configuration of (E,C,P) that works well on all studied models and tasks. How to determine these hyperparameters for  new tasks empirically? Tuning these parameters separately for each task can be inefficient.

Reference:

[1] Hou et al. "Loss-aware binarization of deep networks." International Conference on Learning Representations. 2017.

[2] Hou et al. "Loss-aware weight quantization of deep networks." International Conference on Learning Representations. 2017.

[3] Zadeh, Ali Hadi, and Andreas Moshovos. "GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inference." arXiv:2005.03842, 2020.
review_title: Inconsistent loss formulation and importance indicator and limited novelty. Lack of comparison with a similar post-training quantization method for NLP tasks.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a weighted quantization framework that could be applied to general neural networks for language models. One core idea is that different parameters might show different sensitivity towards a loss function change. Thus it would make sense to take use of importance metrics to do weighted quantization across parameters.

Clarity:

The paper is clearly written. With a good introduction of related work and a pretty self-inclusive references to experiment setup. I also like how the paper is organized to motivate the inclusion of weight importance into quantization. 

Originality:

Originality in this paper is mostly from introducing the concept of parameter (weight) importance and how it is defined and applied to the quantization problem. 
This paper is based on an earlier paper (also new), e.g., BiQGEMM (Jeon et al., 2020), which paved the foundation  to support binary-coding-based quantization techniques to accelerate quantized neural networks. A magnitude-based importance metric is proposed and approved effective in the form of the binary codes. To fine-tune model accuracy, three hyper-parameters are explored and empirically investigated to discover best performance (regarding model quality).

Significance:

Based on the experiment result, I agree that this paper's contribution is significant. If we combine the effort of quantization with distillation, as shown with the metrics in DistillBert, we could achieve great ability of inference speed, a very smaller model size but still maintain reasonable model quality.

review_title: Importance weighting integrated into weight quantization

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper employs the binary-coding-based post-training quantization (without retraining) for language modeling. The key contribution is that weight importance is considered while determining binary code (a, B). Two methods, Greedy and Alternating, are also modified to use the importance. The algorithm uses a novel normalized importance, which directly uses weight magnitude and some hyper-parameters. Because the performance is sensitive to hyperparameters, Bayesian optimization is used to find task- and model-specific settings.

Adopting weight importance to previous algorithms is quite novel and makes sense. Binary-coding-based quantization is recently introduced but seems to be quite a promising direction, so this paper has some significance. The idea to replace pruned parameters with the smallest ones is also smart.

The choice of E, C, P as controllable parameters seems reasonable. However, it looks like these parameters are sensitive and differs much task-by-task and model-by-model, that the robustness to the hyper-parameter is not ensured.

Major questions:

1)	I am not sure that in Section 4.1, Equation (7) and (8) properly support the state “weight magnitude can be a dominating factor for the loss function perturbation”. ‘Delta-L’ is a function of both ‘h’ and ‘delta-w’, not ‘w’ itself directly. (I might be misunderstood…)

2)	Can you provide some explanation (or guess)? (a) why sometimes greedy method is better than alter method (b) when does the pruning helps.

Minor comments:

1)	It would be great if a detailed scaling factor sharing scheme inside the parameter is given. For example, row-wise/column-wise/block tiling for a weight matrix?

2)	When the sample dataset (all or some) is not given (that’s why post-training quantization is valuable), how can we select proper hyper-parameters? BO maybe not applicable.

3)	The term “Original” in tables may be misleading that the values are from the floating-point baseline. For example, for Table 1, I recommend clarifying three cases: float baseline, greedy without importance (previously “Original”), greedy with importance (E=1.0).

review_title: Good work, extends and improves recent approach with novel features

final_score: rating: 5: Marginally below acceptance threshold
review_content: Based on a previous classic binary coding scheme, this paper proposed to introduce a modification $m_i$ on the binarization scaling factor $\alpha$, by considering the weight magnitude. It further use 3 hyperparamters to refine $m_i$ by constraining its upper/lower bound and exponent. Besides,  this work spent lengthy content to describe how to determine the hyperparamters.

This paper contains the following drawbacks:
1. The proposed post-training quantization method has no connection to language model. All we can say is that: Post-training weighted quantization of neural network *in* (or applied in) language models. As author also mentioned that this method also works well in image tasks.
2. There is a confusion on experimental setting: The quantization should be applied to the *fine-tuned* model. But it seems that author didn't pay attention to the disambiguation of pre-trained and fine-tuned, as I am quite lost in Sec.4.3 where author mention that "perform post-training ... using *pre-trained* models of BERT-base ... on MNLI and MRPC." And didn't emphasize that the quantization is conducted on the fine-tuned models by MNLI/MRPC.
3. Author spent much effort on how to determine the hyperparameters, which is also one defect of the method: it requires exquisite tuning on hyperparameters, which are sensitive as shown in Table 2.

Question:
1. How does the work deal with parameters in batch normalization layers?
review_title: Less Innovative Post-Training Quantization Method

Meta-review: The paper builds upon a recent paper BiQGEMM, providing a binary coding based post training quantization technique. The authors show how to combine magnitude-based importance metrics to these techniques and achieve superior performance. The use of importance metrics for quantization and pruning is not new, and magnitude-based metrics are among the more common metrics. With that in mind, the novelty of the paper is in the integration of importance metrics to the techniques of BiQGEMM. The provided methods lead to several hyper-parameters and the task of tuning these can be non-trivial and time consuming. Due to this the authors devote a detailed section showing how to properly tune these hyper-parameters. This is appreciated and indeed alleviates the problem coming with new hyper-parameters.

The paper received mixed opinions by the reviewers related to its overall novelty, but the resulting conclusion is that although the combination of binary coding based quantization with importance scores is not trivial, the challenges faced relate more to correct implementation as opposed to scientific novelty. Combined with other issues raised by the reviewers such as a need for further comparison with existing work, this lead me to recommend rejection for this paper.

Decision: Reject