Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The work targets an interesting direction of improving the efficiency of Transformers by reducing the sequence length. The main contributions of the work are (1) proposing LengthDrop as the way to achieve length reduction; (2) utilizing techniques developed in NAS, namely one-shot NAS, to enable proper training and allow adaptive drop ratio search after training. All these ideas are very reasonable and interesting. Empirically, the authors show that the proposed method is able to match or even outperform BERT-base model with 1/3 - 1/2 FLOPs during inference (not training).

The first concern I have is that the source of the gain is unclear in two aspects:
(1) The proposed model is finetuned for longer (5 + 5 epochs) compared to the 3 epochs of the original BERT. However, the baseline numbers are still based on 3 epochs. This could be one of the reasons why the proposed model even outperform the original BERT.
(2) With the "Inplace Distillation" in play, the obtained model is effectively a distilled model. This adds another layer of complication to judge how much the gain/loss comes from distillation and the length reduction.

Secondly, authors do not mention much about the training (finetuning + ES) cost compared to the standard BERT or Power-BERT. In many real-world cases, this cost is also non-trivial. This may be part of the reason why only 3 datasets are considered in this paper. 
 
review_title: Interesting direction, reasonable and interesting techniques, limited experiments and unclear source of gain

final_score: rating: 4: Ok but not good enough - rejection
review_content: This work introduces a method, called LengthDrop, to train a Length-Adaptive Transformer that supports adaptive model architecture based on different latency constraints. In order to make the model robust to variable input lengths, the method stochastically reduces the length of a sequence at each layer during training. Once the model is trained, the method uses an evolutionary search to find subnetworks that maximize model accuracy under a latency budget. 
 
Pros:
- Accelerating the inference speed of Transformer networks is an important problem.
- The idea of training a length-adaptive Transformer once and using it in different scenarios with different latency constraints is interesting. 
 
Cons:
- The discussion with several state-of-the-art work is lacking.
- The experimental setup is vague, and the evaluation results are inadequate. 
 
The paper looks from an interesting angle to build adaptive Transformers for inference -- reducing the input sequence at each Transformer layer. However, there are a few concerns.
 
First, the paper proposes to use a series of techniques to make LengthDrop work but lacks the ablation studies to show how those techniques help to make Transformer length adaptive. For example, Section 3.1 states that LengthDrop requires LayerDrop[1], which also supports adaptive Transformer by stochastically dropping layers during training for adaptive inference. However, there are no ablation studies or comparison results on LengthDrop vs. LayerDrop in terms of the accuracy-vs-latency trade-off. This raises the question of whether LengthDrop is necessary to obtain the given accuracy-vs-latency or perhaps simpler alternatives such as LayerDrop would be sufficient.
 
In addition to LayerDrop, it appears that the paper also incorporates several other fixes, such as the sandwidth rule and inplace distillation, which are borrowed from prior work.  However, how these fixes contribute to LengthDrop is not clearly explained, and there are no studies nor experimental results to explain how each technique contributes to the final accuracy-vs-latency results. 

Second, the comparison with related work is weak. In particular, LengthDrop is built on top of PoWER-BERT, yet the evaluation does not compare with PoWER-BERT.  Furthermore, the paper compares with DistillBERT, but there are multiple knowledge distillation based work that show better performance than DistillBERT, such as TinyBERT[2]. In terms of adaptive architecture, the evolutionary search of length configurations is similar to the NAS process in the Hardware-aware Transformer[2], which seems to be very related as it also uses evolutionary search to find a specialized sub-network of Transformer models with a latency constraint. The paper briefly mentioned [2], but it is not clear the advantage of this work as compared with [2]. 
 
Third, the paper lacks enough information on the evaluation setups, raising several questions on the reported speedups. For example, it is unclear what's the batch size used in the evaluation. Figure 3(a) shows that reducing FLOPs on GPU does not lead to a reduction of latency for batch size 1, which is the common setting for online inference scenarios as queries come in one-by-one. It is unclear whether input length reduction may actually bring significant latency reduction when the batch size is small (e.g., 1), as the large matrix multiplications have been highly optimized on modern CPU and GPU through efficient kernels (e.g., cuDNN). Even for results on CPU and GPU with batch size >= 16, it is less clear whether the linear correlation between FLOPs and latency is a fact of failing to use highly optimized BLAS libraries, because the paper does not report the details on the hardware, the inference frameworks, and libraries it uses for the experimental results. 

In addition to the batch size and lack of hardware/platform/library information, the experimental setup for training a Length-Adaptive Transformer is also not very clear. For example, it is unclear what's the maximum sequence length is used in training. Whether mixed sequence length is used in training BERT?  What is the sequence length(s) to obtain the results in Table 1? Without associating the actual length reduction ratio, it is hard to evaluate the reported FLOPs reduction.
 
[1] Fan et. al. "Reducing Transformer Depth on Demand with Structured Dropout", https://arxiv.org/abs/1909.11556

[2] Jiao et. al. "TinyBERT: Distilling BERT for Natural Language Understanding", https://arxiv.org/abs/1909.10351
review_title: Official Blind Review #2

final_score: rating: 5: Marginally below acceptance threshold
review_content: ##########################################################################

Summary:

The paper proposed the Length-Adaptive Transformer. The model can be trained once and directly applied to different inference scenarios. To achieve this goal, the author proposed the LengthDrop method, which randomly samples the length at each layer. In addition, the author used the sandwich rule to train the model. At each step, the sandwich rule will train the largest model, the smallest model, and another bunch of randomly sampled models. In the inference phase, the paper proposed to search for the best length configuration that balances the accuracy and latency tradeoff via evolutionary search. Moreover, to generalize the model to token annotation tasks, the author proposed the Drop-and-Restore process, in which the tokens that have been dropped are used again in the final layer. Experiments show that Length-Adaptive Transformer is able to outperfom the baseline models when evaluated at the same latency level.


##########################################################################

Reasons for score: 
 

The paper is well-written and the length-adaptive idea is reasonable. However, I think it still requires more experiments. Also, the training process is not very clear to the reader. It is not so clear how different techniques impact the final performance and the author has not reported the training time. Due to these two reasons, I choose to vote for weak rejection.
 

##########################################################################

Pros: 
 

1. The idea of length-adaptive transformer is novel. Perform on-demand truncation of the hidden lengths has not been well explored.
 

##########################################################################

Cons: 
 

1. The paper needs to conduct experiments on more text classification dataset. Currently, only SST-2 and MNLI are considered. Also, the paper lacks the comparison with "[NeurIPS2020] DynaBERT: Dynamic BERT with Adaptive Widthand Depth", which is also able to balance the latency and accuracy.

2. The training process is quite complicated and involves multiple steps, e.g., Length-Drop, LayerDrop, and Sandwich rule. The author has not explained the relative contribution of each techniques. Also, the author has not reported the training time of the whole pipeline.


##########################################################################

Questions during rebuttal period: 
 

Please address and clarify the cons above 
 

#########################################################################

Typos: 

(1) Section 2.2, Paragraph 2, "more efficient and more accuracy" should be "more efficient and more accurate"
(2) Page 7, under Table 1 and Figure 4, "Lengnth" should be "Length"
review_title: Lacks experiments

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper aims to make inference more efficient for finetuned contextual representation models such as BERT. The authors extend PowerBERT, a method introduced recently to perform  efficient inference by dynamically reducing input tokens as the model goes deeper. The authors address two limitations of PowerBERT: the need to pre-set the required computational budget during training (which makes the model inflexible), and the inability to tackle span level tasks (which require the full sentence at the final layer). The authors present a simple solution to both problems (though one that requires heavy engineering to work, see below), and show promising results compared to several BERT baselines. Overall, the ideas presented in this paper are simple (in a good way), and I would like to see more work that applies similar ideas to gain efficiency and not just accuracy. However, the paper leaves important questions unanswered, and thus I cannot recommend accepting it. First, how expensive is the proposed evolutionary search compared to standard fine-tuning? Second, how does this approach compare to the original PowerBERT model (despite its limitations)? Third, given that the evolutionary search is performed on the validation set, how do the authors evaluate it during model development? 
I am curious to read the authors' response, and could be convinced to increase my score, but currently this paper does not meet the ICLR bar.

Detailed summary:

This paper takes PowerBERT, a recently introduced model for efficient inference, and addresses two of its limitations: the need to retrain the model for each computational budget, and the inability to handle span-level tasks such as QA. The authors present simple solutions to both: for the former, they train a model that randomly selects the number of dropped tokens during training, which makes the model more resilient to different levels of pruning, and perform an evolutionary search process to match the exact pruning levels for a given computational budget. For the latter, they restore the tokens dropped at earlier stages in the last layer, allowing the model to access span indices for tokens dropped earlier if necessary. These solutions are solid and seem to work in practice (although they require quite a bit of engineering to work well, see section 3.1). Nonetheless, their implementation leaves a few important questions unanswered:

1. If I understand correctly, the proposed evolutionary search process needs to be run for any computational budget, which makes it suffer from the same problem as the original PowerBERT model. The authors address this concern in section 3.2, saying that "it only require a single pass through the relatively small validation set for each length configuration", but later say they "repeat this iteration G times" (with G=30) and "evolutionary search converges after about fifteen iterations". This leads to the natural question: how much faster is this procedure compared to finetuning the model, as done in PowerBERT? And even if it is faster, it still doesn't result in a completely flexible model, as each new budget level requires rerunning this process.
2. Another issue with this approach, is that the model is (partially) trained on the development set. This leaves me wondering how it was evaluated during model development. 
3. The authors build on the PowerBERT model, but never compare to it. Although they (supposedly) provide important benefits compared to this model, it is important to understand at what (accuracy and efficiency) cost. 

More comments: 
1. This paper would strongly benefit from a proof-read. Several paragraphs and sentences repeat what was just said (e.g., the paragraph starting with "It is not trivial to find an optimal length" on page 2, or the first sentence on page 7). There are multiple typos (e.g., "Lengnth" on Table 1)  and grammatical errors ("it converges rapidly evolutionary search converges after about fifteen iterations.") all around the paper. Moreover, several issues were unclear to me: 
a. what are sub-models? are these layers with fewer tokens? 
b. how is model distillation incorporated in the solution (last paragraph of section 3.1)?
2. BERT uses word-pieces and not words.


review_title: Simple and intuitive idea, important questions unanswered

Meta-review: The paper attempts to reduce computational cost of Transformer models. In this regard, authors generalizer PoWER-BERT by proposing a variant of dropout that reduces training cost by randomly sampling a fraction of the length of a sequence to use at each layer. Further, a sandwich training method is used which trains a spectrum of randomly sampled model between the largest and the smallest size model. At test time, the best length configuration that balances the accuracy and latency tradeoff via evolutionary search is used. The reviewers found the general idea interesting, but raised a number of concerns. First, proper baselines should be used and related works be discussed. In particular, the method is built on top of Power-BERT, yet it does not directly compare with it, and there was no good response when pointed out by a reviewer. Second, as the paper employs many tricks (some new some from prior work), but does not do any ablation studies to show how each of those contributes to the final accuracy gains. Finally, to showcase benefit compared to prior works in terms of computational cost a proper evaluation methodology and actual speedups for batch size 1 inference should be provided. Thus, an improved evaluation would benefit the paper a lot and paper in its current form is not ready for publication.

Decision: Reject