Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The main goal of this paper is to introduce a simple methodology for optimizing transformer based models for efficiency and effectiveness. 

The paper introduces two main ideas:

1)A top-down strategy for pruning components of a transformer model: Given a specific focus, say speed, the strategy is to consider pruning large coarse-grained components first followed by smaller finer-grained components. The pruning decision is made based on a “significance analysis” -- a component is considered significant for pruning if it from the model does not result in a substantial increase in the model’s loss (as decided by a pruning threshold).

2) Pruning and approximating techniques for different components: For example feed-forward networks are pruned by removing weights in groups (determined via a hyperparameter). For approximating self-attention a sign-matching technique for deciding which top K keys to use for computing Query x Key dot products. 

The main strengths of this work are as follows:

1) The techniques do not require training networks from scratch and can be applied directly during fine-tuning. 

2) The techniques are simple and should apply widely to most transformer-based models. 

3) The empirical results support the claim that the technique can yield significant speed-up and memory-reductions while maintaining accuracy and even provide improvements in accuracy if that is the pruning goal. They show that technique is orthogonal to other models explicitly designed for speed and memory footprint (Q8BERT, DistillBERT) and can provide further improvements in both efficiency and effectiveness. 

4) This is a practical and useful approach that should be widely applicable along with many useful insights about optimizing transformer-based systems. 

I appreciate that the experimental results are reported with averages across multiple runs!

I don’t see any major weaknesses in the paper. Here are some areas that can be improved:

1) The description of the pruning strategies was hard to follow and needed to be tightened up. Possibly adding equations and some pseudo-code to the description should help.

2) I am curious to know what components get pruned cross the different models that were optimized. I wonder if there are systematic differences between original and distilled models and between auto-regressive (GPT) and auto-encoding style models.

3) Also some level of ablation analysis on the strategies used will be helpful. For example if the elements were not ordered based on the granularity would the results be any different? Since this is an iterative strategy the order should play an important role in selection and utility of the subsequent pruning steps. Same goes for the set of pruning strategies. A related question would be what gives the biggest gains. 

4) What is the impact on the fine-tuning time? The baseline only requires one fine-tuning pass. Does this method require multiple fine-tuning passes? Or can the loss thresholds be computed on a smaller subset of the target data? This may be a good future work to look into for tasks where the training data is relatively large, where one cannot afford to exhaustively search through all the pruning strategies. 

review_title: A simple yet impactful optimization approach using a suite of pruning methods. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: After reading the rebuttal, some of my concerns are addressed by the additional experiments. But I also agree with other reviewers that the result is not very surprising. As R4 mentioned, the proposed method depends on the a specific downstream task where the "small" "general" BERT can be further pruned. For a fair comparison to previous work, baselines that are applied to a specific fine-tuning task need to be compared. 

=====

This paper presents a new framework for creating small fine-tuned pre-trained language models. 
The framework has 3 components: 
1. a set of transformer components to be pruned
2. a significant analysis for identifying unimportant elements. 
3. a set of techniques to prune or approximate the transformer element.

Pros:
1. The framework is very adaptive by considering different basic elements of the transformer. 
2. The framework is very efficient by removing large components (e.g., layers, attention blocks, ffd layers) at first and small components (e.g., weight group) later.
3. The framework gathers multiple different pruning/approximation techniques and tries to explore the limit of pruning pre-trained models, which is appreciated. 

Cons/Questions:
1. Is the loss used in significant analysis computed using the development set? If the validation loss is used, the experiment results in Table 1 are not reliable. 
2. There are many BERT pruning papers. Providing comparison to these papers is very important to evaluate the proposed method. Can the model prune more weight at the same performance level? or  Can the model perform better at the same pruning ratio? 
3. It is also helpful to present how much computing resource is needed to prune the network. E.g., how many prune-finetune cycles are needed. 
4. Lack of results of pruning BERT-base on GLUE, which is a very standard and common setting. 
5. In Figure 3, why Q8BERT + Speed Focus is even larger/slower than Q8BERT? With the same speed, Q8BERT + Speed Focus is significantly worse than Q8BERT.

Minor:
Page 5: less the minimum loss seen ==> less 'than' the minimum loss
review_title: A new framework for pruning fine-tuned BERT by ensembling multiple tricks

final_score: rating: 7: Good paper, accept
review_content: This paper presents a method for improving a fine-turned Transformer in terms of a specific metric such as size, speed, or accuracy. The candidates of removed elements are considered hierarchically with some heuristics and are evaluated in terms of training and validation loss to determine whether they should actually be removed from the model. The authors apply their method to several state-of-the-art Transformer models and show that they can produce fast and compact models without losing much accuracy.

Although the individual techniques employed to realize the whole pruning process are not particularly novel, the paper presents a well-thought-out approach to combine those and reports very promising experimental results. I think this is a nice contribution to the community, given that the computation cost is increasingly important in dealing with BERT-like models. 

It seems to me that the authors used transformers whose weights are shared between different layers like Universal Transformers or ALBERT. Maybe I missed something, but I think the authors should clarify if this is really the case in the manuscript.

The entire process of pruning is a bit vague and hard to replicate. Would it be possible to describe the whole process in pseudo code? (Is Algorithm 1 the whole process?)

I think the authors should also describe the computational cost (or maybe wallclock time) required to perform the proposed pruning processes. It seems to me that the search space is rather large and requires a considerable amount of computation.

> p.5 … we prune the element only if the training/validation loss

I think you should be more specific here. How did you actually use both the training and validation loss?  Why do you need to look at the training loss when you are interested in the generalization error?

> p.5 … weight groups of (Wn) …   

Why is this Wn? I thought this should be W.

Minor comments: 
p.5 less the -> less than the?
p.6 doesn’t -> does not
p.6 ’’attention -> ``attention
p.7 second order -> second-order?

review_title: Respectable engineering effort with promising experimental results

final_score: rating: 4: Ok but not good enough - rejection
review_content: Thanks to the authors for the detailed feedback! I still have concerns about the clarity of the presentation, and some contributions of the papers are not strong enough, so I'll keep my score.

===

Summary:

This paper presents a framework to systematically perform pruning and layer approximation. The framework includes a queue of potential elements for compression. At each time step, the framework will evaluate the head element of the queue, try to prune the whole element or perform approximation (quantizing, pruning attention heads, and approximating with sign-matching attention), and keep the transformation only if the loss in performance is acceptable. The paper performs experiments with various models on GLUE and shows speedups or compressions compared to the original model.

Reasons for score: 

The techniques used in the paper are not novel, and the choices on how to apply multiple compression techniques need more justification. The experiment results are okay but not surprising. The presentation of the paper needs to be polished. See below for more details.

Pros: 

1. I like the insight that {approximate, fine-tune, approximate} cycles doesn’t work for fine-tuning.
2. I like the insights used to determine which elements to be examined first: start from the larger blocks and later layers. I hope this point can be emphasized more and compared with more brute-force and less-efficient algorithms. For example, for each round, one can choose a layer that causes the least loss of performance to prune. You can compare your greedy algorithm with this algorithm to show the gain of using a less efficient algorithm is not significant.
3. The sign-matching attention proposed in the paper is new. I would like to see more emphasis and ablation studies on the effectiveness of this special module.

Cons: 

1. It is well-known that compressing the model is easier during the fine-tuning phase [1, 2]. I don’t think this should be a contribution to emphasize for the paper.
2. The whole compression framework has a single global error bound. Combining this with the greedy layer-by-layer approach taken by the framework, will the following case be possible: a layer that is early in the queue causes a huge drop of accuracy and thus makes all the future layers impossible to remove because the global error bound has been reached. A better way is to only remove the layer with the lowest loss reduction. It will be better to justify this point with an ablation study, or at least show the final pruned model doesn’t have this issue in the paper.
3. At the end of page 5: “When optimizing for speed, however, removing weight groups with low significance from arbitrary locations does not help, since it introduces unstructured sparsity in the weight matrix that can be difficult to exploit to achieve speedups.” It’s true that if you remove random entries in a matrix will not help for the actual speedups, but you can remove an arbitrary set of rows of the matrix, and then restructure the weight matrix (i.e. concatenate all the remaining rows to form a new matrix) to make it efficient for modern parallel hardwares. 
4. I don’t really understand the point of using accuracy as the final goal. If the framework is for compression, the goal should be about speedup or size. If accuracy really matters, it should be enforced as the threshold instead of as the final goal. Also, I don’t see the difference in the framework between using speedup or size as the goal, since all the thresholds are defined by loss.
5. The results in the paper are okay, but compared to previous works in computer vision [3], it seems that the model size can be further compressed.
6. There are multiple places where the presentation can be improved:

    a. It’s more clear to use a pseudo-code instead of a diagram in Figure 2. 

    b. It should be more clear to present Table 1 as multiple tables.

    c. It’s better to put the results comparing with previous works in a table (in the middle of page 8).

Minor comments: 

- On page 5, 3rd paragraph from the bottom, “less the minimum loss” -> “less than minimum loss”

References:

[1] Jiao, Xiaoqi, et al. "Tinybert: Distilling bert for natural language understanding." arXiv preprint arXiv:1909.10351 (2019).

[2] Shen, Sheng, et al. "Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT." AAAI. 2020.

[3] Han, Song, Huizi Mao, and William J. Dally. "Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding." arXiv preprint arXiv:1510.00149 (2015).

review_title: Techniques need more justification and presentation needs polishing

Meta-review: This paper introduces a set of techniques that can be used to obtain smaller models on downstream tasks, when fine-tuning large pre-trained models such as BERT. Some reviewers have noted the limited technical novelty of the paper, which can be seen more as a combination of existing methods. This should not be a reason for rejection alone, but unfortunately, the results in the experimental section are also a bit weak (eg. see [1-4]), there are not very insightful analysis and it is hard to compare to existing work. For these reasons, I believe that the paper should be rejected.


[1] DynaBERT: Dynamic BERT with Adaptive Width and Depth

[2] Training with quantization noise for extreme model compression

[3] MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices

[4] SqueezeBERT: What can computer vision teach NLP about efficient neural networks?
Decision: Reject