Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: This paper proposes a method to compress large-scale neural language models (e.g., BERT) without losing too much downstream performance. In contrast to prior work that seeks to compress just the parameters W in the standard linear layer h = Wx + b, this work incorporates the input by approximating Wx instead. The authors also extend this idea to approximations of attention, which enables them to experiment with Transformer architectures. Experiments across the GLUE benchmark confirm that the method does not significantly harm downstream performance, and it also seems to result in inference speedups. However, the biggest flaw with this paper is that its timing experiments are reported on a single CPU thread, while the majority of practical scenarios rely on GPUs. There are other issues with the choices made for the timing experiments (large batch size despite using CPU?) which make me skeptical that the method works as well as claimed.  As such, I cannot recommend the paper's acceptance.

comments:
- i'm not quite sure how the proposed method is more "generic" than existing related work. why does having to "train the proposed structure from scratch" (pg 1) reduce the practicality of a method? in general, these types of applications prioritize inference speedups over training time, so i don't get the criticism. i feel like touting the method as "generic" is a little misleading; the proposed method here is complementary to prior work on compression / distillation and should be presented as such.
- the related work comparisons are similarly exaggerated / misleading (sec 2). implying that pruning methods "might not" reduce inference speed is strange given that much prior work on pruning BERT-like methods yields significant inference speedups. 
- this method assumes that the data lies in a low-dimensional subspace, which might not be a valid assumption for many tasks / datasets. i would have liked to see more discussion of this, although the paper does evaluate over many different tasks. 
- the GLUE speedup ratios are interesting; do they correlate to something like the complexity of a particular dataset? 
- a HUGE concern with the timing experiments is that they were done on a single thread of a CPU, not a GPU. especially with a large batch size (100 is fairly large), this definitely disadvantages the CPU computation for larger models. DRONE is not very relevant if the same speedups don't occur on a GPU; no one is serving BERT models on CPU. this makes me skeptical of the value of the proposed method, and is by itself enough to reject the paper. 
- it's even more strange because "DRONE-retrain" performs 1 epoch of fine-tuning, presumably on a GPU, so why would a CPU be used for timing?
review_title: interesting idea but has experimental issues

final_score: rating: 5: Marginally below acceptance threshold
review_content: The goal of this paper is to accelerate large-scale NLP models. This paper reduces the computational complexity by exploiting the data distribution. They claim that exploiting the data distribution enables us to perform low-rank approximation on feed-forward networks. Furthermore, they use that idea to reduce the complexity of the dot-product of attention modules. They experimentally show that they achieve faster inference time while retaining original accuracy. In addition, they show that their method can be combined with distillation methods.

Strength
- This paper is well organized. We easily understand the goal, challenges, and main idea. In addition, they provide an example of their observation which leads to their main idea.
- They achieve faster inference time with similar accuracy to a given BERT model; they combine low-rank approximation with a distillation method. They show the experimental results of BERT models as well as LSTM models.


Weakness
- Why not report the preprocessing time? It would not be a fair comparison to compare only the inference time.
- The accuracy loss seems to be non-negligible.
- In the last line of proof of Theorem 1, how can we get the right equation from the left equation? U and V are column-orthogonal matrices (not orthogonal matrix) so that UU^T and VV^T are not equal to identity matrices. If Theorem 1 is incomplete, need to re-investigate the method and experiments.
- They sub-sample 10% of the training data to perform low-rank approximation. The ratio affects the performance of a compressed model. I would like to see the performance change with respect to the sampling ratio.

review_title: Limited contribution

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:
This paper studies a technique to increase the inference speed and decrease model sizes of pretrained NLP models such as BERT. Since most operations in BERT consist of matrix multiplications, the authors conduct empirical experiments to show that while matrices themselves are not low-rank, the learned representation of each layer lies in a low-dimensional space. These empirical insights lead the authors to propose an approach based on data-aware low-rank compression of pretrained weight matrices and can be applied to fully-connected and self-attention layers. Their approach is able to improve both model size and inference speed
with limited loss of accuracy on the Glue dataset

Strengths:
1. The paper studies an important problem and is well-motivated. The writing is generally clear and supported by figures and experimental results.
2. The algorithm is presented very clearly and is very intuitive. Their algorithm is also supported by examples and proofs when necessary.
3. The proposed approach is simple to implement and can be applied for different network architectures. This is good and the results are also promising, but there are some issues with evaluation (see below).
4. Pseudocode is provided in the appendix and the approach seems easy to implement and quite flexible.

Weaknesses:
1. Figure 3 tradeoff between performance and speedup is not very useful unless the plots are also made for existing baselines and compared with the author's proposed approach.
2. The only empirical comparison is with TinyBERT but it seems like there are many recent papers studying similar compression algorithms on BERT:
https://arxiv.org/abs/1909.11687
https://arxiv.org/abs/2001.04246
https://arxiv.org/abs/2002.02925
https://arxiv.org/abs/2002.08307
that I think should be mentioned and possibly compared to as well.
3. What are the additional pre-processing time/space requirements in solving for low-rank compression? Some analysis of this would be good, and further comparing the tradeoffs between performance, pre-processing time, and inference time/space.
4. Tables should include some bolding or highlighting otherwise it's hard to tell what the reader is supposed to look at.
5. Figure and table captions can be made more informative and provide standalone arguments.

================Post rebuttal================

I thank the work done by the authors during the rebuttal. However, my main concerns regarding novelty and experimental comparisons still stand - I think it is insufficient to say that your method is 'complementary to most if not all existing methods, so we could easily combine the proposed method to the others to further accelerate' without showing this or comparing to current approaches. The other reviewers have also brought up important concerns regarding experimental details which I concur with.
review_title: review

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work introduces a low-rank based compression method, called DRONE, to accelerate the inference of large NLP models. Instead of decomposing weight matrices in a model, DRONE proposes to exploit the low-rank decomposition by considering the input vectors, which can be in a low-rank space. To compress the whole model, DRONE performs low-rank decomposition from the lower layers to the upper layers in a sequential order. DRONE also proposes a way to extend the low-rank decomposition to dot-product attention. In the evaluation, the paper evaluates their approach against BERT-base and LSTM, and shows that their approaches can obtain better latency-vs-accuracy trade-off versus SVD-based compression method.

Strengths: 
- The paper aims to address an important problem in large NLP model inference.
- It is an interesting idea to apply matrix decomposition by exploiting the low-rank in the input data rather than the weight matrix. 

Weakness:
- The evaluation is inadequate as the baseline does not seem to represent real inference scenarios. 
- Important references are missing, making it not clear the advantage of this work as compared with existing approaches that also compress Transformer networks. 

Comments:

The idea of doing data-aware low-rank compression is certainly interesting. However, the evaluation is largely inadequate to make it convincing that the data-aware low-rank compression will bring a benefit in NLP inference. 

First, it is not clear why a large batch size (e.g., 100) is chosen for inference. In real online inference scenarios, queries come in one-by-one, so the batch size is often just 1 or at most a few in order to meet latency constraint. Is large batch size chosen because DRONE cannot obtain much speedup with small batch sizes? What is the latency when batch size equals to 1? It would be better to add discussions on the choice of batch sizes and show speedup ratios using batch sizes that are close to real settings.  

Second, the evaluation is done "with single-thread on CPU", which does not represent real scenario either and makes the whole evalution less convincing. Modern servers often have multi-core CPUs and GPU with thousands of cores, and DL frameworks often support parallel execution on different hardwares. One caveat is that reducing the width of matrix-multiplication, e.g., using low-rank decomposition, may not necessarily lead to significant latency reduction, as general matrix-multiplication can execute quite efficiently through highly optimized GEMM kernels (e.g., in cuBLAS and MKL) on GPU and CPU. Therefore, it is important to report the performance of DRONE with parallelism enabled. It is important because if the latency reduction is relatively small on real hardware setting, it becomes difficult to justify the 3% accuracy loss on downstream tasks using compression. 

Finally, it does not seem to be accurate to claim "the first work to discuss the generic matrix computation acceleration on NLP tasks", given that low-rank decomposition has been used quite a bit for model compression (e.g., SVD and its variants), and there is at least some work on applying low-rank decomposition for compressing Transformer networks[1] and LSTM[2,3]. 

Question:

The empirical observation that weight matrices in BERT models are not low-rank is interesting. Do you have an explanation on why this is the case? 

Does DRONE require to pre-decide a fixed batch size to apply the closed-form solution as in Equation 3? How does the speedup ratio look like when the batch size is 1? 

Minor:
[3] applies low rank decomposition for LSTMs, not CNN.

[1] Mao et. al. "LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression", https://arxiv.org/pdf/2004.04124.pdf, April 2020

[2] Winata et. al. "On the Effectiveness of Low-Rank Matrix Factorization for LSTM Model Compression",  https://arxiv.org/abs/1908.09982, 2019

[3] Yang et. al. "Fast LSTM Inference by Dynamic Decomposition on Cloud Systems", https://ieeexplore.ieee.org/document/8970702/, ICDM, 2019

review_title: Official Blind Review #2

Meta-review: This paper proposes a method for compressing weight matrices in large scale pre-trained NLP encoders (like BERT) through low-rank decompositions of both fully connected and self-attention layers. The method is used to compress and speedup pre-trained models. Experiments measure timing on a single CPU thread and demonstrate speedups with small loss of accuracy. Reviewers noted the that goal of this paper is potentially impactful. Some reviewers viewed the resulting loss in accuracy as marginal, while others viewed it as more substantial -- a potential downside. Reviewers also raised concerns about the methodology used to measure inference speedup, a critical measure of success. Specifically, timing experiments were done only on a single CPU thread -- while most practical scenarios would almost certainly rely on GPUs -- as a result, positive experimental results are less impactful. Authors updated the paper to include GPU timing experiments, which did show speedups -- though only marginal speedups over the baseline, TinyBERT. Further, reviewers pointed out that there are several other relevant baselines on compression approaches that are not compared with, and that further analysis should be done on the timing/accruacy tradeoff of baseline methods. Finally, reviewers felt that the contribution of the proposed method relative to other approaches that also attempt to compress transformers is not clearly outlined. Weighing these concerns, I agree with reviewers that the paper is not ready for acceptance in its current form. 
Decision: Reject