Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper explores extreme parameter compression for pre-trained language model, especially BERT. It introduces and compares several tensor decomposition methods and proposes to leverage Tucker decomposition as the final solution. The compressed BERT model achieves much smaller size with promising performance.
main_review: Large scale pre-trained language models have demonstrated their effectiveness. However the large model size makes it difficult to deploy and compressing such models have drawn a lot of interest. This paper aims to compress PLMs to extremely small size mainly from the perspective of decomposition. It introduces several decomposition methods and makes a comprehensive comparison among them from the perspective of compressing Transformer layers. The Tucker decomposition is chosen to be the final solution due to its compression ratio.

The motivation is clear and the methods are technically sound. Though the introduced decomposition methods are not new, the adaption to the Transformer layers and corresponding analysis are comprehensive. The experimental results demonstrate the effectiveness of the method. Especially, the compressed model size is really competitive.

Some weaknesses:
1. The authors do not include embedding layer and prediction layer size in experiments, while only report the Transformer encoder size. I know that this can make the size of compressed model really amazing (e.g., 1.8M) and the compression ratio amazing (e.g., 86M/12.3M=7) but is not fair as the whole model including the embedding layer are used when deploying. If the embedding layer is added, the model size will increase a lot, and the compression ratio will decrease, which make the experimental results less surprising. But this should be made clear.
2. The authors name a lot of related works, but compare only very few of them in the experiments.
3. Some other method(s) are missing in the related works. For example: [1]

Some typos:
1. Section 5.1, "...are not exactly equal to the the raw weights...", duplicate "the"?
2. Section 6.2, "...outperforms ALBERT - the latter needs...while the latter does not...", two "latter"?

reference:
[1] Xu, Jin, et al. "NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Search."
summary_of_the_review: The paper presents extreme compression on pre-trained language models. Though the introduced methods are not new, the adaptation to the Transformer layers and the analysis are interesting, and the experiments are convincing. Though there exist some weaknesses, I think the paper is of good quality, if the authors could mitigate them.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to use tensor decomposition to jointly compress the model weights in all attention and FFN layers of a Transformer model, which reaches similar performance as the original BERT model while marking the model much smaller.
main_review: Reasons for score:

I think the idea proposed in the paper is novel, but some design choices can be further elaborated and there should be more experiments on larger models and more ablation studies. Detailed comments:

Strengths:

- Applying tensor decomposition across layers to utilize the similarity between layers is novel. This is a valuable contribution to the model compression community.
- The paper analyzes the optimal way to perform matrix multiplication given the compression method proposed in the paper.

Weaknesses:

- The paper proposed multiple potential ways of compressing weight matrices (matrix decomposition and tensor train decomposition) as some alternatives to the proposed Tucker decomposition. However, the author didn't compare with tensor train decomposition due to time constraints. I believe the paper will be more solid by adding ablation studies on tensor train decomposition.
- The paper only performs experiments on BERT-base and TinyBERT models, but I believe that the compression method proposed in the paper should be more demanded by larger models.
- Some design choices in the paper seem arbitrary. For example, why do you jointly compress all the layers, including all the FFN weights and attention weights? I can understand the similarity of the attention query weight vectors across the layers, but I can’t understand why all the weights are merged. In addition, the reason for splitting each FFN weight matrix into 4 seems to make it possible to combine it with attention weights.

Other comments:

- How does this method compare with previous works in terms of training/inference latency?
summary_of_the_review: The paper would be better with more experiments on larger models and ablation studies. Also, the presentation and the rationale behind the idea are not clear to me.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to use tensor decomposition to compress the multi-head attention (MHA) and FFN layers in transformer architecture. 

Premise:

Drawing from previously published research and their own experiments using PCA, the paper shows that the MHA and FFN layers are over-parameterized and exist in a lower dimensional subspace. Further, the authors talk about where these redundancy might come from and point to the decomposability of FFN and MHA layers and how that can lead to different parts learning similar behavior. Using PCA the authors show that these layers exhibit both inter-matrix and intra-matrix redundancy. Inter-matrix redundancy calls for low dimensional representation of each matrix in the layer, while intra-matrix redundancy implies possibility of parameter sharing between matrices.

Proposed Solution:

Based on this observation, the authors discuss the merit of various methods for decomposition of MHA and FFN matrices. Specifically, they discuss matrix decomposition, tensor train decomposition and tucker decomposition. The last technique will have the largest impact on compression and also has the most marginal cost of adding a new layer. 

Results:

1. Impressive accuracy/parameter pareto frontier. Can compress BERT-base by ~50x with 1.5 to 2% loss in accuracy (Table 4)
2. Ablations on the need for 2 stage knowledge distillation - stage 1 to ensure attention maps and output of last layer are similar and stage two for task distillation

main_review: Strengths:

1. The framework of decomposability and tensor decomposition allows this paper to encompass and explain the benefit of multiple previous work using a single viewpoint
2. The results of compression are strong. Possibility of 50x compression, albeit at 1.5-2% accuracy loss opens a lot of possibilities for on device deployment


Weakness:

1. The use of tensor decomposition for compressing neural networks has been explored extensively for CNNs, RNNs and Embeddings. The use here to compress transformers is a natural extension of the idea
2. Decomposability and low-rank nature of FFN and MHA layers has been discussed previously in the literature. The authors themselves refer to these prior works
3. Cordonnier et al 2021, further discusses the use of tucker decomposition to express MHA layers, albeit in a slightly different context. 

In order to improve the paper, I recommend providing more insights into the workings of the method and the bias that the fixed structure like tucker decomposition can lead to. I also recommend exploring the systems impact of running training and inference using tucker decomposed layers and why say 50x reduction in parameter count, does not lead to a 50x improvement in RPS (Table 4). Further, I would encourage the authors to explore and understand why finetuning with KD in Table 6 leads to such large accuracy drop. GD+TD should lead to better accuracy, but improving accuracy by 40% is an interesting data point.

Questions:

1. In section 4.2, the authors say "During the inference phase, the terns that do not involve batch size b or seq length n could be calculated in an offline way...". Could you expand on what you were referring to?
2. Table 4 should have comparisons to sparsified BERT, esp for data points with 2-3x parameter compression. Both structured sparsity and random sparsity could achieve said compression. Eg - https://arxiv.org/abs/2109.04838
3. Tucker decomposition of matrices across layers forces common parameters between the matrices. A possible way this methodology could go wrong is if these matrices have different scales. Is the norm value across matrices similar for different layers? Have you looked at the problem from this point of view?
summary_of_the_review: Overall, I think this paper is an incremental improvement to previous state-of-the-art. Tucker decomposition has been used extensively in NN to compress RNNs, CNNs and Embeddings. Thus use of tucker decomposition and its ability to compress BERT MHA and FFN layers is incremental improvement over the previous results, especially given the fact that prior work has also shown that MHA and FFN layers can be decomposed in a low rank structure and talked about the redundancy in the parameters in those layers.  However, the results of the paper are interesting from an engineering point of view.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: After author rebuttal: thank you very much for the detailed response to both my and the other reviewers' comments. I have updated my score. 
------------
This paper proposes an approach to compressing the Transformer family of pretrained language models via tensor decomposition. Compared to existing work, the main differences are: (1) performing global tensor decomposition which takes into account matrices across different layers, and (2) using a matrix bank to enable even greater model compression. The approach, when combined with distillation, outperforms existing methods for compressing BERT models on the popular GLUE benchmark.

main_review: Strengths:
- The paper is well written and the motivating analyses (e.g. Fig 1) are interesting. I also appreciated the thorough appendix.
- The main technical contributions of the paper (i.e. using all the matrices to perform matrix decomposition, using a bank of matrices) is novel to my knowledge. However, I have some reservations about whether this is sound (see weaknesses).

Weaknesses:
- The paper introduces many variants of decomposing the matrices (Table 2). However the results only seem to be based on one of the methods (i.e. IV). I realize that some of variants have already been studied in the literature (e.g. Mao et al., Noach and Goldberg), but since the setup is not identical, it is crucial that the proposed approach is compared against both II and III. Therefore, it is not clear whether the improvement in performance is coming from the actual proposed method, or something else.
- Since the matrices across layers often have different scaling, and since tensor decomposition is approximating some reconstruction error (i.e. L2 in the case of SVD), it's not clear that performing decomposition with all the matrices makes sense. 
- The method requires a pipelined approach where one must first perform generalized distillation against the BERT model before doing task-specific distillation.

Questions:
- In GD, did you try distilling also the masked LM logits, in addition to (or instead of) the last layer hidden states and attention maps?
- Have you checked the norms of the matrices? Are they of similar scales? If not, do you obtain better performance by normalizing the matrices such that they are in the same scale?

summary_of_the_review: A new approach to model compression with strong-ish empirical results. Some reservations about the soundness of the method. And it is furthermore not clear that the improvements are coming from the proposed approach as opposed to something else, since an ablation study across the different decomposition methods is missing.
review_title: null

Meta-review: This paper reviews a number of parameter decomposition methods for BERT style contextual embedding models. The authors argue for the application of Tucker decomposition to the attention and feedforward layers of such models. Evaluation is performed for a range of models on the GLUE benchmark. Further ablation studies indicate that the distillation procedure employed is crucial for obtaining competitive results and the raw decomposition approaches are ineffective at directly approximating the original pre-trained model.

Strengths: The reviewers generally agree that the methods explored and results presented in this paper are interesting and could be of use to those deploying large embedding models. The authors review a range of possible decomposition methods and use this to motivate their approach. The resulting levels compression are high while maintaining good performance, while the ablation study clearly shows the contribution of the various steps of the training pipeline.

Weaknesses: The main weakness identified by the reviewers is the incremental nature of this work in comparison to previous works applying various decomposition and compression techniques to neural networks. They also highlight that many of the techniques discussed early in the paper are not compared in the evaluation. The authors have effectively responded to this issue by providing further comparisons and justification for their modelling choices (e.g. not compressing the embedding layers). 

Overall, despite the incremental nature of this work, I believe that there are enough though provoking ideas and results presented to warrant publication. Interestingly, as the authors emphasise in their response, the ablation study highlights that this work is not really about approximating the original models weights, as all of the work appears to be being done by the distillation procedure in concert with the choice weight decomposition. In general I wonder whether this paper would be better presented as exploring a structured distillation procedure rather than weight compression.
Decision: Accept (Poster)