Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes to reduce the computational requirement and memory usage of the state of the art transformer based models. A novel quantization and pruning strategy with proper initialization is proposed and number of experiments support the claim of the paper.
strengths_and_weaknesses: Strength:
- A novel method is presented.
- An important and challenging task is addressed

Weakness:
- The text could be improved 
questions: - I think it is helpful to elaborate more on the process of fine tuning . You could maybe make a figure with some blocks visualizing the pretrained model, the customized model and explain the fine tuning and of course how the proposed method is employed.
- In general, it is more clear to report the relative improvements in the results. For example, in line 88 it is stated that the model achieved less than %1 accuracy loss; this 1% should be read next to the original accuracy.
- Regarding figure 3, it is stated that outliers play an important role in transformer training. Could we also say that the close to mean values are not as important?
- Some comments regarding the text:
  - what is q in equation 2?
  - Could you provide a reference for the zero alignment explained in 2.1.1?
  - in line 114: can be find --> can be found
  - Table 1 caption: accuracy --> performance (you are not reporting accuracy in this table)
limitations: NA
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The study makes a substantial contribution to more equitable and accessible natural language processing, computer vision, and speech signal processing by introducing compression methods that allow deploying the leading Transformer models up to 16 times less compute (i.e., model simplification using less bits) while maintaining model accuracy.
strengths_and_weaknesses: The paper seems to make substantial contributions and is easy to follow, although I am not an expert in compressing models. What could be clarified is the experimental design and its justification. For example, it would be beneficial for the reader to better understand why demonstrating impressive compression rates in three different data modalities (i.e., text of SQuAD1.1, speech of Librispeech, and images of ImageNet1k) is the way to go as opposed to, let us say, compressing on text from three different datasets (e.g., different tasks/problems/levels of difficulty) for natural language processing. A clearer articulation of the compression goals would have been helpful to better evaluate the achieved compression rates; I felt uncertain what high, excellent, outstanding, and groundbreaking success would look like in general and in particular for each of the three experimented modalities. That is, some more justifications are needed to truly acknowledge and appreciate experimental contributions made in this study. Moreover, a more explicit addressing of the authors' right to use these datasets and methods for the purposes of this study would have been beneficial what comes to "Ethics Review Area" flagging below. Finally, the end of the manuscript calls for some further revisions to discuss the limitations and broader impact of this study and its contributions before concluding it.
questions: * Remember to introduce all acronyms used in the paper at their first occurrence (in particular, Section 1 could be enhanced in this way).

* You seem to have missed a paragraph break in Section 1 before the following topic sentence: "In summary, we make the following contributions towards accurate..."

* Please elaborate in Section 3 Experimental Setup and Results on the described experiments by their further justifications. Why demonstrating impressive compression rates in three different data modalities (i.e., text of SQuAD1.1, speech of Librispeech, and images of ImageNet1k) is the way to go as opposed to, let us say, compressing on text from three different datasets (e.g., different tasks/problems/levels of difficulty) for natural language processing? 

* Moreover, a more explicit addressing of the authors' right to use these datasets and methods for the purposes of this study in Section 3 Experimental Setup and Results would have been beneficial what comes to "Ethics Review Area" flagging below. 

* Consider dividing this section to methods and results more clearly to ease the reader in feeling convinced that the experiments are correct and complete prior to presenting the results. I was little confused what the compression goals were and where the difference between high, excellent, outstanding, and groundbreaking were in general and in particular for each domain/task/modality addressed.

* A more traditional Discussion section before the conclusion would have probably worked better. Namely, some further revisions are needed to discuss the limitations and broader impact of this study and its contributions before concluding it.

* Double check that capitalisation is consistent in all section headings

limitations: The authors have not addressed the limitations and potential negative societal impact of their work. As stated above, a more traditional Discussion section before the conclusion would have probably worked better. Namely, some further revisions are needed to discuss the limitations and broader impact of this study and its contributions before concluding it.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents authors' investigation on compressing pre-trained transformers with quantization/sparsity-aware fine-tuning and optimized quantizers, initialization, data format etc. Experimental results on NLP, speech and vision tasks show effectiveness of the proposed approach.
strengths_and_weaknesses: The topic of compressing transformer models is of broad interests in the machine learning community, given transformer-based models are being widely researched in many areas, and it's also important to compress models for resource constraint use cases. This paper proposes an effective approach which is generally applicable to different areas. This work is technically sound, with carefully designed experiments and analysis. This paper is clearly written as well.

There are different approaches to compress models, and it would be helpful to provide more contexts on the chosen quantization + sparsity vs. other techniques (see questions below).
questions: 1. In this paper authors list limitations with teacher-student distillation (e.g. lines 192-195). Still, teacher-student distillation could be applied to train a student model with a lower-complexity architecture, and its overhead in training/tuning may not necessarily be a blocker for using it, and it has no direct impacts on inference. Do authors have additional reasons why not consider teacher-student distillation for compression?

2. Singular value decomposition (SVD) is also widely applied for model compression, and could be easily combined with other model compression techniques. Do authors have a say on comparing/combining it with the proposed approach?

3. In check list it's said "key results are averaged results over random seeds". For this case, could error bars be included directly in results tables?
limitations: N/A
review_title: null

Meta-review: metareview: The paper presents quantization and sparsifying (50%) for transformers, and studies simple schemes for both quite extensively. The novelty is rather low but the value of the paper is in presenting results in 3 domains (NLP, ASR, image classification) and in paying attention to details of what matters to do quantization on pre-trained models with minimal loss of accuracy, as well as efficient implementation of those schemes. Overall, the paper is suitable for NeurIPS and I recommend acceptance.
confidence: Certain
Decision: Accept