Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a Specialization framework to create optimized transformer models for a given downstream task. The framework systematically uses accuracy-driven pruning. The authors proposed two ways to reduce model parameters, 1) Hierarchical pruning. Start from analyzing entire feed-forward and self-attention blocks, and inspect them at finer granularity (attention heads and neurons) only
when required. 2) Replacing soft-attention with hard-attention. The proposed method significantly improves benchmark models, BERT, Q8BERT and DistillBERT.
main_review: Strength:
1. The proposed method is interesting, and it can be adopted by different models with significant improvement.
2. The accuracy-driven pruning is reasonable and different from previous pruning-based methods.
3. Leveraging hard attention is an interesting idea. 
4. The paper is easy to read and the experiments are solid. The authors compare the method with multiple pruning methods including regularization and lottery ticket based methods. And the experiment shows that the proposed method Is better.

Weakness:
1. It would be better to have some more analysis on hard attention, such as some statistical comparison between soft-attention and hard-attention. Also, more analysis on training and inference speed between two attention mechanisms would be helpful.

summary_of_the_review: Overall, the proposed method is novel and reasonable. The experiments are quite solid by adopting the method to multiple structures and comparing it with SOTA pruning methods.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces a framework to prune parameters in transformer-based structures. The authors claim that the proposed method leads to a smaller model with faster and more accurate performance. It first replaces soft self-attention with hard self-attention by checking whether the replacement leads to a smaller loss and benefits more than N samples in the validation set. Using a similar way, it further prunes attention block, feed-forward block, and their neurons. 
 
main_review: Pros:
1. The proposed method does not require further fine-tuning.
2. It is simple enough, easy to apply on the general transformer-based network.
3. Speed, size, and performance could be improved at the same time. That is interesting. 

Cons:
1. The pruning method is not very novel. 
2. Here I quote from the paper, "where N is set to be greater than at least three-fourths the number of samples in the dataset. In order to reduce the effect of outliers, the exact value of N is tuned for the different datasets, based on the validation loss of each sample." Could you introduce the details of tuning N? 
3. Lack mechanism analysis. For example, how does the attention to each word change after the pruning of each element?
4. Not enough ablation study. Not sure what really matters. The pruning of which element matters the most? 
5. "For example, sentiment analysis requires only local context, and long-range information often ends up confusing the model, since sentiments often change rapidly; it is also unlikely that syntatic and symantic information are needed. Hence, we place the final layer at the front of the queue, and work our way backwards towards the first layer, since blocks in the final layers are more likely to hinder performance on sentiment analysis". Here I have two questions: 1. How do you get this conclusion about the final layers and sentiment analysis? I am not convinced by the short reasoning process. 2. Considering the conclusion is correct, then "Place the final layer at the front of the queue" is based on some prior knowledge about the task. Therefore, can I assume that without these inductive biases, the performance will be worse? Should I introduce the inductive bias for each downstream task?
6. It does not compare with some of the latest work. Like "Know what you don't need: Single-Shot Meta-Pruning for attention heads". Also, could it be combined with other works like compression? For example, your method then compression? Or does the implementation of your method hinder the implementation of other pruning/compression/quantization methods?
7. (Minor) Some tables are using figures. I think these tables can be generated via latex.

summary_of_the_review: Overall, it is an interesting paper. An easy method to reduce the model size and increase the performance in the downstream tasks. Not sure whether it can be generalized to general transformer-based models. It looks more like some technology that is not so novel. There is also not enough detailed research analysis.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes Specialized Transformer that identifies harmful parts and prunes a pre-trained transformer in a greedy and hierarchical manner to boost accuracy in a downstream task. According to the paper, the proposed Specialized Transformer is faster, smaller, and more accurate than a standard fine-tuned model without any retraining.
main_review: The method is a *greedy* method with unjustified heuristics. However, how can it perform well without getting stuck in a local minimum and have low variance? I am pretty suspicious about the results because many essential details and explanations are missing.

As far as I understand, it prunes a transformer fine-tuned for a downstream task. (I’m a little bit confusing because the abstract mentions pruning a pre-trained transformer.) Then, it is already overfitted in some sense. I am not sure how pruning can improve the generalization.

I am curious whether similar pruned architectures are obtained with different random seeds. Is the pruning process deterministic or stochastic? If it is deterministic, is it measuring with differently fine-tuned transformers? Do you have any explanation about low variance?

Are pruning and hard self-attention separate each other? What is the order of them?

The method does not perform any retraining. Simply thinking, eliminating one layer or even a smaller portion of parameters could change the representation distribution of the following layers. How can it perform well?

The paper should be revised further. The related work section is just a listing of works that improve the efficiency of the transformer. The authors should categorize them and provide clear relation with their work. Method and result sections include many long paragraphs only divided by headings.

I don’t understand how to use Taylor expansion to estimate the importance of each parameter. Is it more than a first-order gradient?
summary_of_the_review: The results look very promising, but the presentation is bad. The method lacks crucial details for implementation. Their methods seem too naive to be effective, but the authors do not fully explain why they result in a good performance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new framework for pruning a pretrained Transformer on downstream tasks for better performance and efficiency. The resulted model is 2.5x faster and 3.2x smaller than a fine-tuned model. The authors test the effectiveness on GLUE and SQUAD with BERT-base, Q8BERT, DistilBERT and XLNet.
main_review: ### Pros
1. The paper is overall well-written and easy to follow.
2. The evaluation is thorough with SQUAD and GLUE. The method shows effectiveness on both benchmarks and with different models by outperforming the fine-tuned counterparts. 
3. The paper proposes a progressive and hierarchical strategy that provides a solution for both self-attention and FFNN in a Transformer, which has the potential to be adopted in more tasks (e.g., generation, translation).
4. Figure 3 is quite interesting - it looks like DistilBERT is more "dense" than an uncompressed model and it warrants further exploration.
5. The method is practical since it does not require re-training and works on downstream tasks thus enabling the reuse of nearly all off-the-shelf pretrained models.

### Cons
1. My main concern is novelty - the pruning technique is not new and pruning a neural network progressively is also not a novel idea. That being said, the paper is still informative in my opinion.
2. Although specialization does not require a complete re-training, it does introduce extra overheads for training such a model. The authors mentioned this problem on page 8 but I'd like to see more discussion with analysis. 

### Missing References 
- https://arxiv.org/abs/2002.10957
- https://arxiv.org/abs/2005.07683
- https://arxiv.org/abs/2006.04152
- https://arxiv.org/abs/2109.04838
summary_of_the_review: This paper proposes a new pruning framework for pretrained Transformers with good evaluation and analysis. Although the idea is not completely new, I would like to recommend weak acceptance for this paper.
review_title: null

Meta-review: The authors propose a simple and effective technique for task-specific pruning of transformer models that identifies which model components to prune by minimizing validation loss. Weaknesses of the paper include (1) related work reads more like a list and doesn’t compare and contrast the proposed approach with related work, (2) authors don’t compare to other structured pruning methods (that use different objectives) (3) lack of novelty — main difference with existing work is using validation loss to optimize and (4) one reviewer was unconvinced that the results should be possible given the approach. I share these concerns, and, in particular, I think they might be related. Given that the models are pruned using the development set (essentially equivalent to training on the development set), it seems infeasible that this approach could have been developed without looking at the testing data, and I’m concerned that this explains the unprecedentedly high accuracy compared to previous pruning approaches. At the very least, comparing to a baseline that trains on development data would be prudent in order to understand the result.
Decision: Reject