Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes DRONE, a data-aware low-rank compression algorithm for BERT, achieving a speed-up with marginal accuracy loss.
main_review: DRONE effectively reduces the computational cost with the observation that intermediate feature vectors lie on a low-dimensional manifold. DRONE significantly outperforms the vanilla SVD method.

I presume how to distribute r into each module R_{l, i} is important for the final performance. Algorithm 2 is duplicated in Section 3.3 and Appendix B. Although there are explanations about Algorithm 2, it would be better to describe its rationale. I think the method is somewhat heuristic. Do you have any results on other design choices? Small errors in a lower layer can be propagated to a higher layer resulting in a big difference to the final representation. Is there any consideration for this effect in the proposed method? Also, in this sense, I wonder whether DRONE is also effective in deeper networks (e.g., BERT-Large).

The combination of DRONE with other compression methods is practically valuable. The authors show that DRONE can be used orthogonally with knowledge distillation and quantization. DRONE could be compared with Linformer which also leverages low-rank structures.

Detailed analysis on ranks for each position (self-attention or feed-forward / layer) would give an insight for a future design of new architectures or compression methods.

Figure 2 only shows singular values at the very first and last layer. I am curious whether the same trends appear at intermediate layers. Considering a space limit, the authors may report the accumulation ratio of singular values choosing the top 50% rank, for instance.

Minor: It is strange that the body of the appendix is aligned to the center.
limitations_and_societal_impact: Developing a better algorithm for the distribution of loss ratio would be an interesting research direction. Instead of equally distributing in a deterministic manner, dynamic programming based method can be used for optimal distribution.

Inspired by many works on iterative pruning, I am curious whether DRONE can be performed iteratively (e.g., DRONE → finetune → DRONE → ...) for better final performance despite the increased computational cost.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposed a data-aware low rank compression for BERT. It is stated that data-aware low rank compression could perform better than standard low rank compression like SVD since weights for matrix multiplications in BERT are not low-rank and SVD will fail. It seems that the motivation makes sense, and experiments evidenced its effectiveness.

main_review: 
It is not fair to say 12.3 times speed up in the Abstract, because  DRONE is used on TinyBERT but it seems that the speedup is calculated upon standard BERT. The speedup mainly comes from TinyBERT.

The example in Sec.3 (between the line 135 and 136) assumes that X is low rank. I am wondering whether this is true in real-world applications.

It seems that data-aware low rank compression is not specific to pre-trained language models. Is there any previous work that evidences its effectiveness in some simple scenarios like MLP or CNN, especially models are not needed to be pre-trained? By doing so, it may better investigate the behavior of  "data-aware low rank compression".

Can the authors show some information about the ranks {k} in different weights?

In Algo.2 both m_i and G are called 'module'. What is  G like? In the second for-loop, we find a subscript i in M and we also find i in the third for-loop.


In Table 2, why speedup in tinyBERT varies in different tasks? And RTE, why is tinyBERT only 1.8x times faster than BERT? but in other tasks, 10 more times speedup is achieved.
limitations_and_societal_impact: There is nothing related to Societal Impact. The Limitation is that the speedup is not notable while the performance is a little bit worse. This limits its applications in the real world. But the idea itself is intertesting.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces DRONE, which is a data-driven low-rank decomposition method. Experiments are conducted using LSTM on PTB dataset and BERT on GLUE tasks. The method performs better compared to naive SVD.
main_review: Strengths:
- The method is simple and orthogonal to other methods.
- Writing is clear and straightforward.
- Authors report the performance of DRONE combined with distillation and quantization methods, which demonstrate the potential for production.

Weaknesses & Questions:
- For the detailed time cost (of BERT), I only can find the numbers of DRONE, can you also provide the costs of baseline?
- How do you measure speed-up? What batch-size is used? Line 231-232 looks not very clear. I guess you use large batch-size for evaluation, then use the average time per sample? If yes, can you also provide the speed-up of single sample inference? (As single-sample inference time is important to real-world)
- In real-world products, we often need to re-train / re-finetuning the model by the latest data, due to distribution changes. As DRONE is data-depend, will distribution-change cause more performance loss?
- Did you try DRONE on language generation tasks? How does it perform?
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes DRONE, a data-aware low-rank model compression method designed specifically for large language models, e.g., BERT. The method is motivated by an observation that the weight matrices in a pre-trained BERT model are generally not low rank, but the data/intermediate results usually lie in a low-rank space. DRONE leverages the above observation and arrives at a close form solution to approximate the layer weights using low-rank factorization. Theoretical analysis is provided and experimental results are shown to demonstrate that DRONE successfully accelerate the inference speed of BERT model on a various range of down stream tasks.
main_review: Pros:
1. The paper is well-written, and the research direction of compressing the large language models and improving the inference speed is promising.
2. The experimental results are promising. I am convinced that DRONE can work well in practice.

Cons:
My major concern on the proposed method is about its novelty. The data-aware low-rank compression method has been explicitly proposed in PCN [1] (which is not even cited by this paper). In my view, DRONE is almost identical to PCN apart from the fact that PCN has not been deployed in language models yet. The authors are expected to elaborate the novelty of DRONE further. Other concerns are summarized as follows:
1. Decomposing one layer into two small low-rank layers can effectively increase the depth of the model (which reducing width), which is known to make the model harder to train. In such a sense, how dose the training behavior of DRONE look like? And how does that compare to the vanilla BERT training.
2. On the memory perspective of view, deploying low rank approach reduces the model size. But the activation values for each factorize layers need to be saved. How does the memory footprint look like?
3. It has been studied that starting from a “warm-up” stage of full-rank model can be helpful for the low-rank model [2-3]. Does it also help DRONE?
4. In the current draft, DRONE is not compared with any other pruning/sparsification based methods. How does DRONE compare to the structured pruning method, e.g., [3]?
5. The current experiments only study BERT models, whose scales are not that large. How does DRONE perform on GPT-2/ViT/MLP-Mixer models?
6. It is easy to imagine that one can use DRONE to speed up the BERT pre-training stage. Adding even an preliminary result on such a direction can be really helpful.
Minor issues:
Missing references: [1-2], [4].

[1] https://arxiv.org/abs/2006.13347
[2] https://arxiv.org/abs/2103.03936
[3] https://arxiv.org/pdf/1910.04732.pdf
[4] https://openreview.net/forum?id=KTlJT1nof6d
limitations_and_societal_impact: My major suggestions are summarized in the “Cons” of “Main review”. Please take a close look on it. Moreover, as the state-of-the-art architectures in computer vision and NLP fields tend to be unified, e.g., Transformers/pure MLP networks. It would be important to include both NLP and computer vision results, e.g., ViT/MLP-Mixer on CIFAR and ImageNet to show that the method generalize well.

I am happy to increase my overall evaluation scores if my concerns are addressed.
review_title: null

Meta-review: This paper proposes DRONE, a data-aware low-rank compression algorithm for BERT, achieving a significant speed-up with marginal accuracy loss. The authors show that DRONE can be used orthogonally with knowledge distillation and quantization. In this sense, the paper has its clear value to the community. The reviewers raised some concerns on the technical novelty, the precise quantification of the speed-up, and the experimental details. Overall speaking, the authors did a good job in their rebuttal, and some of the reviewers’ concerns were successfully addressed. The reviewers made several rounds of discussions, and the conclusion is that it would be good to accept the paper as a poster. 
Decision: Accept (Poster)