Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
Authors: Gyuwan Kim
Kyunghyun Cho
Venue:  (2021)
Keywords: transformer
efficiency
anytime prediction
sequence length
evolutionary search

Abstract: Although transformers have achieved impressive accuracies in various tasks in natural language processing, they often come with a prohibitive computational cost, that prevents their use in scenarios with limited computational resources for inference. This need for computational efficiency in inference has been addressed by for instance PoWER-BERT (Goyal et al., 2020) which gradually decreases the length of a sequence as it is passed through layers. These approaches however often assume that the target computational complexity is known in advance at the time of training. This implies that a separate model must be trained for each inference scenario with its distinct computational budget. In this paper, we extend PoWER-BERT to address this issue of inefficiency and redundancy. The proposed extension enables us to train a large-scale transformer, called Length-Adaptive Transformer, once and uses it for various inference scenarios without re-training it. To do so, we train a transformer with LengthDrop, a structural variant of dropout, which stochastically determines the length of a sequence at each layer. We then use a multi-objective evolutionary search to find a length configuration that maximizes the accuracy and minimizes the computational complexity under any given computational budget. Additionally, we significantly extend the applicability of PoWER-BERT beyond sequence-level classification into token-level classification such as span-based question-answering, by introducing the idea of Drop-and-Restore. With Drop-and-Restore, word-vectors are dropped temporarily in intermediate layers and restored at the last layer if necessary. We empirically verify the utility of the proposed approach by demonstrating the superior accuracy-efficiency trade-off under various setups, including SQuAD 1.1, MNLI-m, and SST-2. Upon publication, the code to reproduce our work will be open-sourced.


Under review as a conference paper at ICLR 2021
LENGTH-ADAPTIVE TRANSFORMER: TRAIN ONCE WITH LENGTH DROP, USE ANYTIME WITH SEARCH
Anonymous authors Paper under double-blind review
ABSTRACT
Although transformers have achieved impressive accuracies in various tasks in natural language processing, they often come with a prohibitive computational cost, that prevents their use in scenarios with limited computational resources for inference. This need for computational efficiency in inference has been addressed by for instance POWER- DOWER-BERT (Goyal et al. 2020) which gradually decreases the length of sequence as it is passed through layers. These approaches however often assume that the target computational complexity is known in advance at the time of training. This implies that separate model must be trained for each inference scenario with its distinct computational budget. In this paper, we extend POWER-BERT address this issue of inefficiency and redundancy. The proposed extension enables us to train a large-scale transformer, called Length-Adapiin Transformer, once and uses for various inference scenarios without re-training it. To do so, we train a transformer with LengthDrop, structural variant of dropout, which stochastically determines the length of sequence at each layer. We then use multi-objective evolutionary search to find length configuration that maximizes the accuracy and minimizes the computational complexity under any given computational budget. Additionally, we significantly extend the appli- cability of PoWER-BE -ERT beyond sequence- classification into token-level classification such as span- based ceesstion-asseringg by introducing the idea of word-vectors are dropped temporarily
moop-and-Restore With Drop-and-Restonn in intermediate layers and restored at the last layer if necessary. We empirically verify the utility of the proposed approach by demonstrating the superior accuracy- efficiency trade-off under various setups, including SQuAD 1.1, MNLI-m, and SST-2. Upon publication, the code to reproduce our work will be open-sourced.
INTRODUCTION
Pretrained language models (Peters al. 2018 Devlin et al. 2018 Radford et al. 2019 Yang et al.
2019) have achieved notable improvements in various natural language processing (NLP) tasks. Most of them rely transformers (Vaswani al. (017)) and the number of model parameters ranges from hundreds of millions to billions "Shoeybi et al. 2019; Raffel et al. [2019] Kaplan et al. 2020 Brown et al. 2020). Despite this high accuracy, excessive computational overhead during inference. both in terms of time and memory, has hindered its use in real applications. This level of excessive computation has further raised the concern over energy consumption as well (Schwartz al. 2019
Strubell et al. 2019).
Recent studies have attempted at addressing these concerns regarding large-scale transformers' computational and energy efficiency (see for more extensive discussion.) Among these, we focus on POWER-BERT (Goyal et al. 2020) which progressively reduces sequence length by climinating word-vectors based on the attention values passing layers. JAWER-BERR establishes the superiority of accuracy-time trade-off over earlier approaches (Sanh et al. 2019 Sun et al. 2019)] Michel et al. 2019). It however requires to train separate model for each efficiency constraint. In this paper, we thus develop framework based on PoWER-BERS such that we can train a single model that can be adapted in the inference time meet any given efficiency target.
In order to train a transformer to cope with a diverse set of computational budgets in the inference time, we propose train one while reducing the sequence length with random proportion each

---

Under review as a conference paper at ICLR 2021
layer. We refer this procedure as LeogthDrop which was motivated by the nested dropout (Rippel et al, 2014). We can extract sub-models of shared weights with any length configuration without requiring extra post-processing
additional finetuning.
Once a transformer is trained with the proposed LengthDrop. we search for the length configuration that maximizes the accuracy given computational budget. Because this search is combinatorial and has multiple objectives (accuracy and efficiency), we use an evolutionary search algorithm, which further allows to obtain full Pareto frontier of accuracy-efficiency trade-off of each model.
is not trivial to find an optimal length configuration given the inference-time computational budget, although it is extremely important in order to deploy these large-scale transformers in practice. In this work, we propose to use evolutionary search to find length configuration that maximizes the accuracy within a given computational budget. We can further compute the Pareto frontier of accuracy-efficiency trade-off to obtain a sequence of length configurations with varying efficiency profiles.
DOWER-BERY which forms the foundation of the proposed two-stage procedure, is only applicable to sequence-level classification, because by design it eliminates some of the word vectors at each layer. In other words, it cannot be used for token-level tasks such as span-based question answering (Rajpurkar et al. 20T6), because these tasks require hidden representations the entire input sequence at the final layer. We thus propose to extend (3.3). which eliminates this inherent limitation. Word vectors are dropped and set aside, rather than eliminated, in intermediate layers maintain the saving of computational cost, was with the original Po TWER-BERS These set-aside vectors are then restored at the final hidden layer and provided as input subsequent task-specific layer, which unlike the original POWER-BERT.
-ropannd-eetoore with novel Drop-and Restore process
The main contributions of this work are two-fold. First, we introduce LengthDrop, a structured au-
variant of dropout for training a single Length-AAapptive Transformer model that allows tomatically derive multiple sub-models with different length configurations in the inference time using evolutionary search, without requiring any re-training. Second, we design Drop-and Restore process that makes POWER-BERT applicable beyond classification, which enables PoWER-BE -ERT to be applicable a wider range of NLP tasks such as span-based question answering. We empirically verify Length-Adattiv Transformer works quite well using the variants of BERT on a diverse set of NLP tasks, including SQuAD 1.1 (Rajpurkar et al. 2016) and two sequence-level classification tasks in GLUE benchmark (Wang et al., 2018). Our experiments reveal that the proposed approach grants us a fine-gre ame-grained control of computational efficiency and a superior sccuracy-efficieng trade-off in the inference time, compared to existing approaches.
-oop-and-Restore
2 BACKGROUND: TRANSFORMERS AND POWER-BERT
Before we describe our main approach, we review some of the building blocks in this section. In particular, we review transformers, which are standard backbone used natural language processing these days, and SWER-BERT which was recently proposed an effective way to train a large-scale, but highly efficient transformer for sequence-leve classification.
2.1 TRANSFORMERS AND BERT
transformer is particular neural network that has been designed to work with a variable-length sequence input and is implemented as a stack of self-attention and colle-connected layers (Vaswani) (et al. 2017). Here, we give a brief overview of the transformer which is the basic building block of the proposed approach.
Each token Zt sequence of tokens x = (x,,...,,,N representing input text, is first turned into continuous vector hi R which is the sum of the token and position embedding vectors. This sequence is fed into the first transformer layer which returns another sequence of the same length 2/2 E NxH We repeat this procedure L times, for transformer with L layers, to obtain h = (h (..,,., ..). We refer to each vector in the hidden sequence at each layer as a word vector to emphasize that there exists correspondence between each such vector and one of the input words.
Although the transformer was first introduced for the problem of machine translation, Devlin et al. (2018) demonstrated that the transformer can be trained and used as masked language model.
2

---

Under review as a conference paper at ICLR 2021
More specifically, Devlin al. (2018) showed that the wansformer-basee masked language model, called BERT, learns a universally useful parameter set that can be finetuned for any downstream task including sequence-level and token-level classification.
In the case of sequence-level classification, a softmax classifier is attached to the word vector hf associated with the special token [CLS], and the entire network, including the softmax classifier and BERT, is finetuned. For token-level classification, we use each h, as the final hidden reprsentation of the associated (-th word in the input sequence. This strategy of pretraining followed by finetuning, often referred to as transfer learning, has recently become a dominant approach to classification in natural language processing.
2.2 POWER-BERT
PoWER-BEET BERT keeps only the topmost l word vectors at each layer j by climinating redundant ones based on the significance score, which is the total amount of attention imposed by word on the other words (Goyal et al. 2020). 1 is the hyper-parameter determines how many vectors to keep at layer j. BERT has the same model parameters with BERT, but the extraction layers interspersed after the self-attention layer in every transformer blocks (Vaswani et al., (017).
POWER-BERT reduces inference time successfully, achieving better accuracy-time trade-off than DistiBBERT (Sanh et al. 2019), BERT-PKD (Sun et al. 2019), and Head-Prune (Michel et al. 2019). Despite the original intention of maximizing the inference efficiency with the minimal loss in accuracy, it is possible to set POWER-BERT to be both more efficient and more accurate compared the original BERT, which was observed but largely overlooked by /Goyal et al. (2020).
Training a POWER-BERT model consists of three steps: (1) finetuning, (2) length configuration search, and (3) re- re-training. The finetuning step is just like the standard finetuming step of BERT given a target task. A length configuration is sequence of retention parameters (12....> -,), each of which corresponds to the number of word vectors that are kept at each layer. These retention parameters are learned along with all the other parameters to minimize the original task loss together with an extra term that approximately measures the number of retained word vectors across layers. In the re-training step. HOWER-BERT is finetuned with the length configuration fixed to its learned one.
For each computational budget, we must train a separate model going through all three steps described above. Moreover, the length configuration search step above is only approximate, as it relies on relaxation of retention parameters which are inherently discrete. This leads to the lack of guaranteed correlation between the success of this stage and true run-time. Even worse. it is delicate act to tune the length configuration given a target computational budget, because trade-off is implicitly made via regularization coefficient. Furthermore, POWER-BIE BERTT has an inherent limitation in that it only applies to sequence-level classification because it climinates word vectors in intermediate layers.
3 LENGTH-ADAPTIVE TRANSFORMER
In this section, we explain our proposed framework which results in a transformer that reduces the length of a sequence at each layer with an arbitrary rate. We call such a resulting transformer cength-Adaptiv Transformer. We train randomly samples the number of hidden vectors to be dropped at each layer with the goal of making the final model robust to such drop in the inference time. Once the model trained, we search for the optimal trade-off between accuracy and efficiency using multi-objective evolutionary search, which allows us to use the model for any given computational budget without finetuning nor re- re-training. At the end of this section, we describe Drop-and- Restore process way greatly increase the applicability of POWER-BERT which forms a building block of the proposed framework.
Transformer with LengthDrop which
In short, train Length-Adaptiv Transformer once with LengthDrop and Drop-and- Prop-and-Restore. and use it with an automatically determined length configuration for inference with any target computational budget, both sequence-level and token-level tasks.
3

---

Under review as a conference paper at ICLR 2021
3.1 LENGTHDROP
Earlier approaches to efficient inference with transformers have focused on a scenario where the target computational budget for inference is known in advance Sanh et al. (2019); Goyal al. (2020). This greatly increases the cost of deploying transformers. as it requires us to train separate transformer for each scenario. Instead, we propose to train one model that could be used for diverse set of target computational budgets without retraining.
LeopthDrop randomly generates a length configuration by sequentially sampling a sequence length 4+1 at the (i + 1) layer based on the previous layer's sequence length 4, following the uniform distribution [[[[1 p)4.1.4 1.4,, where ig is set to the length of the input sequence, and p is the LengthDrop probability. This sequential sampling results in a length configuration (41," Jr..) Length-Adaptive Transformer can be thought of as consisting of a full model and many sub-models corresponding to different length configuration, similarly to neural network trained with dropout (Srivastava et al. 2014).
LayerDrop From the perspective of each word vector, the proposed LengthDrop could be thought of as skipping the layers between when it was set aside and the final layer where it was restored. The word vector however does not have any information based on which it can determine whether it would be dropped at any particular layer. In our preliminary experiments, we found that this greatly hinders optimization. We address this issue by using LayerDrop (Fan et al., 2019) which skips each layer of a transformer uniformly at random. The LayerDrop encourages each word vector to be agnostic skipping any number of layers between when it is dropped and when it is restored, just like dropout (Srivastava et al. 2014) prevents hidden neurons from co-adapting with each other by randomly dropping them.
Sandwich Rule and Inplace Distillation We observed that standard supervised training with LengthDropp does not work well in the preliminary experiments. We instead borrow a pair of training techniques developed by Yu & Huang (2019) which are sandwich rule and inplace distillation, for better optimization as well as final generalization. At each update, we update the full model without LengthDrop as usual to minimize the supervised loss function. We simultancously update n, randomly-samie sampled sub-models (which are called sandwiches) and the smallest-pone posssible sub- model, which corresponds to keeping only [(1 - p)4] word vectors at each layer i, using knowledge distillation (Hinton et al., 2015) from the full model. Here, sub-models mean models with length reduction. They are trained to their prediction close to the full model's 's prediction (inplace distillation).
3.2 EVOLUTIONARY SEARCH OF LENGTII CONFIGURATIONS
After training a Aangth-Adptiive Transformer with LengthDrop, we search for appropriate length configurations for possible target computational budgets that will be given at inference time. The length configuration determines the model performance in terms of both accuracy and efficiency. In order to search for the optimal length configuration, we propose to use evolutionary search, similarly to co et al. (2019) and Wang et al. (2020). This procedure is efficient, as it only requires a single pass through the relatively small validation set for each length configuration, unlike re-training for new computational budget which requires multiple passes through significantly larger training set for each budget.
We initialize the population with constant-ratio configurations. Each configuration is created by 4++1 [(1 -r)(;] for each layer with r so that the amount of computation within the initial population is uniformly distributed between those of the smallest and full models. At each itera- tion, we evolve the population to consist only of configurations lie on newly updated efficiency- accuracy Pareto frontier by mutation and cross-over. Mutation alters an original length configuration (I...., ,42) to (i',... .,,, 14,) by sampling I/ from the uniform distribution (1' (++1) with the prob- ability Pm or keeping the original length I = 1, sweeping the layers from i = 1 to i L. Crossover takes two length configurations and averages the lengths at each layer. Both of these operations are performed while ensuring the monotonicity of the lengths over the layers. We repeat this iteration G times, while maintaining Num mutated configurations and n. crossover's 'd configurations. Repeating this procedure pushes the Pareto frontier further to identify the best trade-off between two objectives, efficiency and accuracy, without requiring any continuous relaxation of length configurations nor using a proxy objective function.
4

---

Under review as a conference paper at ICLR 2021
(a) PoWER
(b) Drrp--aaddDeeee Restore
Figure 1: Illustration of (a) word-vector elimination process in PoWER-BERT (Goyal et al. 2020 and (b) aap-anddRRssoon process in imply the output of embedding layer and transformer layers, respectively. Green boxes mean vectors dropped in lower layers and restored at the last layer. Red box is the task-specific layer. Though word-vectors in the middle could be eliminated (or dropped), remaining vectors are left-aligned for the better illustration. In this case, the number of transformer layers is four.
Transformer. Yellow box and blue boxes
3.3 DROP-AND-RESTORE PROCESS
The applicability of the POWER-BERT, based on which our main contribution above was made, is limited sequence-level classification, because eliminates word vectors at each layer. In addition to our main contribution above, we thus propose to extend the BOWER-BERT so that it is applicable to teken-level classification, such as span-based question-answeriig refer as Restrre does not eliminate word vectors at each layer according to the length configuration but instead sets them aside until the final hidden layer. At the final hidden layer, these word vectors are brought back to form the full hidden sequence, as illustrated graphically in Fig.
Our proposal, to which we
EXPERIMENT SETUP
Datasets We test the proposed approach on both sequence-level and token-level tasks, the latter of which could not have been done with the original BOWER-BERT unless for the proposed Drop- and-Restore. We use MNLI-m and SST-2 from GLUE benchmark (Wang et al., 2018). as was done to test PoWER-BEE BERT earlier, for sequence-level classification. We choose them because consistent accuracy scores from standard training on them due to their sufficiently large training set imply that they are reliable to verify our approach. We use SQuAD 1.1 (Rajpurkar al. 2016 for token-level classification.
Evaluation metrics We use the number of floating operations (FLOPs) as main metric to measure the inference efficiency given any length configuration, as it is agnostic to the choice of underlying hardware, unlike other alternatives such as hardware-aware latency (Wang et al., (020)) or energy consumption (Henderson et al.) 2020). We later demonstrate that FLOPs and wall-clock time on GPU and CPU correlate well with the proposed approach. which is not necessarily the case for other approaches, such unstructured weight pruning (Han et al. 2015 See et al. 2016).
Pretrained transformers Since BERT was introduced by Devlin et al. (2018), it has become a standard practice to start from a pretrained (masked) language model and finetune for each downstream task. We follow the same strategy in this paper and test two pretrained transformer- based language models; BART (Devlin et al. 2018) and DistilBERT (Sanh et al. 2019), which allows to demonstrate that the usefulness and applicability of our approach not tied any specific architectural choice, such as the number layers and the maximum length of input sequence. Although we focus BERT-based masked language models here, the proposed approach is readily applicable to any transformer-bast models.
Transformer with LengthDrop probability and LayerDrop probability both set to 0.2. We n. randomly sampled intermediate sub-models in addition to the full model and smallest model for applying the sandwich learning rule.
Learning We train a
We start finetuning the pretrained transformer without Drop-and- ans-eeeorree first. just Goyal et al. (2020) did with POWER-BERT We then continue finetuning it for another five epochs with Drop-and- Restore. This is unlike the recommended three epochs by Devlin al, (2018), , learning progresses
5

---

Under review as a conference paper at ICLR 2021
BERT-Base DistiBERT
Length-Adpative BERT-Base Length-Adpative DistilBERT
Length-Adpative BERT-Base
GPU (T)
GPU (4)
Length-Adpative DistilBERT
GPU
GPU
Ludenso (madratano))
15
86
15
FLOPs (8)
(a) GPU
CPU
CPU
CPU (18)
CPU (84)
82
Lazenc Imsinstnnes
1000
750
80
500
280
10
20
30
log FLOPs (G)
PLOP- (G)
Figure 2: Pareto curves of Fl score and FLOPs on SQuAD 1.1 RRajuurkar
(b) CPU
et 2016). We apply the proposed method to BERT Base (solid lines) and Dist (dotted lines). For each model, we draw three curves using
Figure 3: Correlation between FLOPs and latency with different length configurations.
(1) standard finetuned transformer with constant-rate length reduction, (2) Length-Adaptive Transformer with constant-rate length reduction, and Transformer with length configurations obtained from the evolutionary search.
(3)
slower due to a higher level of stochasticity introduced by LengthDrop and LayerDrop. We use the batch size 32, the learning rate of 5e for SQuAD v1.1 and 2e for MNLI-m and SST, and the maximum sequence length of 384 for SQuAD v1.1 and 128 for MNLI-m and SST.
Search We run up to G = 30 iterations of evolutionary search, using Th 30 mutated configura- tions with mutation probability Pm 0.5 and ne 30 crossover'd configurations, find the Pareto frontier of accuracy and efficiency.
RESULTS AND ANALYSIS
Efffciency- trade-off We SQuAD 1.1 examine the effect of the proposed approach on the efficiency-accuracy trade-off. When the underlying classifier was not trained with LengthDrop. as proposed in this paper, the accuracy drops even more dramatically more word vectors are dropped at each layer. The difference between standard transformer and Length-Adaptiv Transformer is stark in Fig. This verifies the importance of training a transformer way that makes malleable for inferencetiime re-configuration.
When the model was trained with the proposed LengthDrop. we notice the efficacy of the proposed approach of using evolutionary search to find the optimal trade-off between inference efficiency and accuracy. The trade-off curve from the proposed search strategy has a larger area-under-curve (AUC) than when constant-rate length reduction was used to meet a target computational budget. It demonstrates the importance using both LengthDrop and evolutionary search.
We make minor observation that the proposed approach ends up with significantly higher accuracy than DistillBERT when enough computational budget is allowed for inference (log FLOPs 10). This makes our approach desirable in wide array of scenarios, as it does not require any additional pretraining stage, as does DisTiLBERT. With a severe constraint on the computational budget, the proposed approach could be used DistilBERT to significantly improve the efficiency without compromising the accuracy.
Maximizing inference efficiency We consider all three tasks, SQuAD 1.1, MNLI-m and SST-2, and investigate how much efficiency be gained by the proposed approach with minimal sacrifice of

---

Under review as a conference paper at ICLR 2021
Model
SQuAD 1.1
MNLI-m
SST-2
Pretrained
Transformer
Method
F1
FLOPs
Acc
FLOPs
Acc
FLOPs
89.1
Standard
88.5
1.00x
84.4
1.00x
92.8
1.00x
BERT
89.6
0.89x
85.0
0.58x
93.1
0.36x
00.0
Length-Adapen Adaptive
88.7
0.45x
84.4
0.35x
92.8
0.35x
Standard
85.8
1.00x
80.9
1.00x
90.6
1.00x
86.7
DistilBERT
Sength-Adapten
86.3
0.81x
81.5
0.56x
92.0
0.55x
20
30
Length-Adaptive'
85.9
0.59x
81.3
0.54x
91.7
0.54x
Figure 4: Exam- ple of area under Pareto curve as the evolutionary search of lenth configurations proceeds.
Table 1:
Comparison results of standard Transformer and Length- Adaptive Transformer. Among length configurations on the Pareto fron- tier of Length-Adaptin Transformer, we pick two representative points: Length-Adaptive and having the highest accuracy and the accuracy higher than (or equal to) standard Transformer, respectively.
as the most efficient one while
accuracy. First, we look at how much efficiency could be gained without losing on the accuracy. That is, we use the length configuration that maximizes the inference efficiency (i.e., minimize the FLOOss while ensuring that the accuracy is above or same the accuracy of the standard approach without any drop of word vectors. The results are presented in the rows marked with Length-Ade Adaptive' from Table I For example, in the case of BERT Base. the proposed approach reduce FLOPs by more than half across all three tasks.
have observed that the proposed Length-Adapting Transformer generalize better than the standard, base model in some cases. We thus try to maximize both the inference efficiency and accuracy, in order that both maximizes inference efficiency and improves accuracy. We present the results in the rows marked with Length-Adaptive from Table For all cases, Sangt--Adpttive Transformer achieves higher accuracy than a standard transformer does while reducing FLOPs significantly. Although it not apparent from the table, tor MNLI-m and SST-2, the accuracy of the smallest sub-model is already greater than or equal to that of standard transformer.
From Fig.
whether it is possible for the proposed algorithm find length configuration
FLOPs vs. Latency As been discussed recent literature (see, e.g., (Li et al. 2020 Chin et al., 2.20),, FLOOSs is not perfect indicator of the real latency measured in wall-clock time, as the latter is affected by the combination of hardware choice and network architecture. To understand study the relationship between FLOPs, obtained
the real- world impact of the proposed approach, by the proposed procedure, and wall-clock time measured on both CPU and GPU by measuring them while varying length configurations. As shown in Fig. [ FLOPs and latency exhibit near-linear correlation on GPU, when the minibatch size is 2 16, and regardless of the minibatch size, on CPU. In other words, the reduction in FLOPs with the proposed approach directly implies the reduction in wall-clock time.
Convergence search Although the proposed approach is efficient in that requires only one round of training, it needs separate search stage for each target budget. It is important for evolution- ary search to converge quickly in the number of forward sweeps of a validation set. As exemplified in Fig. 4 evolutionary search converges after about fifteen iterations.
6 RELATED WORK
The main purpose of the proposed algorithm is to improve the inference efficiency of a large-scale transformer. This goal has been pursued from various directions, and in this section, we provide a brief overview of these earlier, and some concurrent, attempts in the context of the proposed approach.
Weight pruning Weight pruning (Han et al. 2015) focuses on reducing the number of parameters which directly reflects the memory footprint of model and indirectly correlates with inference speed. However, their actual speed-up in runtime is usually not significant, especially while executing a model with parallel computation using GPU devices (Tang et al. 2018 Li al. 2020).
7

---

Under review as a conference paper at ICLR 2021
Adaptive architecture There three major axes along which computation can be reduced in a neural network; (1) imput size/length, (2) network depth and (3) network width. The proposed approach, based POWER-BERY adaptively reduces the imput length as the input sequence is processed by the transformer layers. In our knowledge, [Goyal et al. (2020) is the first work this direction for transformers. More recently, Junnel-Transfonn
(Dai al. (020) and multi-scale transformer language models (Subramanian et al., 2020) also successfully reduce sequence length in the middle and rescale full length for the final computation. However, their inference complexity is fixed unlike PoWER-BEER BERT because they are not designed for the control of efficiency.
LayerDrop (Fan inspired by Huang et al. (2016). Word-level adaptive depth in [Elbayad et al.) (2019) might seemingly resemble with length reduction, but word vectors reached the maximal layer are used for self-attention
2019) drops random layers during the training to be robust pruning
computation without updating themselves. Escaping network early (Teerapittayanon et al. 2016
Huang et al. (017)) based on the confidence of the prediction (Xin al. 2020 [Schwartz et al. 2020,
Liu et al. 2020 also offers control over accuracy-efficiene trade-off, but it is difficult to tune
threshold for a desired computational budget because of the example-wiss adaptive computation.
DynaBERT (Hou et al. 2020) can run at adaptive width (the number of attention heads and interme- diate hidden dimension) and depth. Hardware-aware Transformers (Wang et al., (220)) construct a design space with arbitrary encoder-decoder attention and heterogeneous layers in terms of different numbers of layers, attention heads, hidden dimension, and embedding dimension.
Structured dropout major innovation we introduce over the existing PoWER-BERT is the use of stochastic, structured regularization to make transformer robust to the choice of length configuration in the inference time. [Ripell et al.] (2014) proposes a nested dropout to learn ordered representations. Similar to LengthDrop, it samples an index form prior distribution and drops all units having larger index than sampled one.
Search There have been series of attempts finding the optimal network configuration by solving a combinatorial optimization problem. In computer vision. Once-for-All (Cai et al.) 2019) use an evolutionary search (Real et al. (209) find a better configuration in dimensions of depth, width, kernel size, and resolution given computational budget. Similarly but differently, our evolutionary search is mutli-objective find length configurations on the Pareto accuracy-efficiency frontier to cope with any possible computational budgets. Moreover, we only change the sequence length of hidden vectors instead of architectural model size like dimensions.
CONCLUSION AND FUTURE WORK
In this work, we propose a new framework for training a transformer once and using for efficient inference under any computational budget. With the help of training with LengthDrop and Drop-and- Restore process followed by the evolutionary search, our proposed Length- Adaptive Transformer allows any given transformer models to be used with any inference-ting computational budget for both sequence-level and token-level classification tasks. Our experiments, on SQuAD 1.1, MNLI-m and SST-2, have revealed that the proposed algorithmic framework significantly pushes a better Pareto frontier on the trade-off between inference efficiency and accuracy. Furthermore, we have observed that the proposed Length-Adaptin Transformer could achieve up 3x speed-up over the standard transformer without sacrificing accuracy, both in terms of FLOPs and wallclock time.
Although our approach finds an optimal length configuration of trained classifier per computational budget, it leaves open a question whether the proposed approach could be further extended to support per-instance length configuration by for instance training small, auxiliary neural network for each computational budget. Yet another aspect we have not investigated this paper is the applicability of the proposed approach to sequence generation, such as machine translation. We leave both of these research directions for the future.
Our approach is effective, of existing language models. We will release our implementation, which is based on HuggingFace's 's Transformers library (Wolf al. 2019). publicly and plan to adapt for broader set of transformer- based models and downstream tasks.
have shown in this paper, and also quite simple to implement on top
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. arXiv preprint arXiv:2005.
2020.
Han Cai, Chuang Gan, and Song Han. Once for all: Train one network and specialize it for efficient deployment. arXiv preprint arXiv: pet:::9998,0911,. 2019.
Ting-W Bung-Wu Chin, Ruizhou Ding, Cha Zhang, and Diana Marculescu. Towards efficient model compres- sion via learned global ranking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1518-1528, 2020.
Zihang Dai, Guokun Lai, Yiming Yang, and Quoc Le. Funnel-transformer, Filtering out sequential redundancy for efficient language processing. arXiv preprint arXiv:2006.032.66, 2020.
Jacob Devlin, Ming-Wor Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0480) 2018.
Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. Depth-adaptivy transformer. arXiv preprint
2019.
Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. arXiv preprint arXiv: (900::9990.111 1556, 2019.
Saurabh Goyal, Anamitra Roy Choudhary, Venkatesan Chakaravarthy, Saurabh ManishRaje, Yogish Sabharwal, and Ashish Verma. Power-bert: Accelerating bert inference for classification tasks. arXiv preprint 2rXvv:2001.089500 2020.
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv: 151.000199 2015.
Peter Henderson, Jieru Hu, Joshua Romoff, Emma Brunskill, Dan Jurafsky, and Joelle Pineau. Towards the systematic reporting of the energy and carbon footprints of machine learning. arXiv preprint arXiv:2002.05653 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0253/ 2015.
Lu Hou, Lifeng Shang, Xin Jiang, and Qun Liu. Dynabert: Dynamic bert with adaptive width and depth. arXiv preprint
2020.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European conference computer vision, pp. 646-661. Springer, 2016.
Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Q Wein- berger. Multi-scale dense networks for resource efficient image classification. arXiv preprint arXiv:1703.09844 2017.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin. Kurt Keutzer, Dan Klein, and Joseph E Gon- zalez. Train large, then compress: Rethinking model size for efficient training and inference of transformers. arXiv preprint arXiv:2002.1 1794, 2020.
Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Haotang Deng, and Qi Ju. Fastbert: a self-distiliing bert with adaptive inference time. arXiv preprint arXiv:2004.02178 2020.
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? In Advances in Neural Information Processing Systems, pp. 1401 14014-14024, 2019.

---

Under review as a conference paper at ICLR 2021
Matthew E Peters, Mark Neumann, Mohit lyyer. Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv: 1e02::00230000 05365, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenA/ Blog, 1(8):9 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. arXiv preprint arXiv: 10683, 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv: /6Xiv:6606.0885 1606. 2016.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pp. 4780-4789, 2019.
Oren Rippel, Michael Gelbart, and Ryan Adams. Learning ordered representations with nested dropout. In International Conference on Machine Learning, pp. 1746-17544 2014.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 1910./9/00/.. 2019.
Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. arXiv:1907.10597 2019.
Green ai. arXiv preprint
Roy Schwartz, Gabi Stanovsky, Swabha Swayamdipta, Jesse Dodge, and Noah A Smith. The right tool for the job: Matching model and instance complexities. arXiv preprint arX14:2004.07453, 2020.
Abigail See, Minh-Thang Luong, and Christopher Manning. Compression of neural machine translation models via pruning. arXiv preprint (00600//0000997744 2016.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:/9099 90..9009000...00000 2019.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, (3(1):19291195)) 2014.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint wwXiv:1906.02244 2019.
Sandeep Subramanian, Ronan Collobert, Mare' Aurelio Ranzato, and Y-Lan Bourcau. Multi-scale transformer language models. arXiv preprint 10X1v:2005.0008/, 2020.
Siqi Sun. Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. arXiv preprint arXiv: 190.00590 2019.
Raphael Tang, Ashutosh Adhikari, and Jimmy Lin. Flops as a direct optimization objective for 2018.
learning sparse neural networks. arXiv preprint
Surat Teerapittayanon, Bradley McDanel, and -isiang-Tsung Kung. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 2464 2469922669... IEEE, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 1804 07461, 2018
10

---

Under review as a conference paper at ICLR 2021
Hanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai, Ligeng Zhu, Chuang Gan, and Song Han. Hat: Hardware-awand awar transformers for efficient natural language processing. arXiv preprint arXiv:2005.14187. 2020.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019.
Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. Deebert: Dynamic early exiting for accelerating bert inference. arXiv preprint arXiv:2004. 12993,, 2020.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoe V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, pp. 5754-5764, 2019.
Jiahui Yu and Thomas S Huang. Universally slimmable networks and improved training techniques. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1803-1811, 2019.


Reviews:
[{"review_id": "yKDcZ1zqZx3", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work introduces a method, called LengthDrop, to train a Length-Adaptive Transformer that supports adaptive model architecture based on different latency constraints. In order to make the model robust to variable input lengths, the method stochastically reduces the length of a sequence at each layer during training. Once the model is trained, the method uses an evolutionary search to find subnetworks that maximize model accuracy under a latency budget. \n \nPros:\n- Accelerating the inference speed of Transformer networks is an important problem.\n- The idea of training a length-adaptive Transformer once and using it in different scenarios with different latency constraints is interesting. \n \nCons:\n- The discussion with several state-of-the-art work is lacking.\n- The experimental setup is vague, and the evaluation results are inadequate. \n \nThe paper looks from an interesting angle to build adaptive Transformers for inference -- reducing the input sequence at each Transformer layer. However, there are a few concerns.\n \nFirst, the paper proposes to use a series of techniques to make LengthDrop work but lacks the ablation studies to show how those techniques help to make Transformer length adaptive. For example, Section 3.1 states that LengthDrop requires LayerDrop[1], which also supports adaptive Transformer by stochastically dropping layers during training for adaptive inference. However, there are no ablation studies or comparison results on LengthDrop vs. LayerDrop in terms of the accuracy-vs-latency trade-off. This raises the question of whether LengthDrop is necessary to obtain the given accuracy-vs-latency or perhaps simpler alternatives such as LayerDrop would be sufficient.\n \nIn addition to LayerDrop, it appears that the paper also incorporates several other fixes, such as the sandwidth rule and inplace distillation, which are borrowed from prior work.  However, how these fixes contribute to LengthDrop is not clearly explained, and there are no studies nor experimental results to explain how each technique contributes to the final accuracy-vs-latency results. \n\nSecond, the comparison with related work is weak. In particular, LengthDrop is built on top of PoWER-BERT, yet the evaluation does not compare with PoWER-BERT.  Furthermore, the paper compares with DistillBERT, but there are multiple knowledge distillation based work that show better performance than DistillBERT, such as TinyBERT[2]. In terms of adaptive architecture, the evolutionary search of length configurations is similar to the NAS process in the Hardware-aware Transformer[2], which seems to be very related as it also uses evolutionary search to find a specialized sub-network of Transformer models with a latency constraint. The paper briefly mentioned [2], but it is not clear the advantage of this work as compared with [2]. \n \nThird, the paper lacks enough information on the evaluation setups, raising several questions on the reported speedups. For example, it is unclear what's the batch size used in the evaluation. Figure 3(a) shows that reducing FLOPs on GPU does not lead to a reduction of latency for batch size 1, which is the common setting for online inference scenarios as queries come in one-by-one. It is unclear whether input length reduction may actually bring significant latency reduction when the batch size is small (e.g., 1), as the large matrix multiplications have been highly optimized on modern CPU and GPU through efficient kernels (e.g., cuDNN). Even for results on CPU and GPU with batch size >= 16, it is less clear whether the linear correlation between FLOPs and latency is a fact of failing to use highly optimized BLAS libraries, because the paper does not report the details on the hardware, the inference frameworks, and libraries it uses for the experimental results. \n\nIn addition to the batch size and lack of hardware/platform/library information, the experimental setup for training a Length-Adaptive Transformer is also not very clear. For example, it is unclear what's the maximum sequence length is used in training. Whether mixed sequence length is used in training BERT?  What is the sequence length(s) to obtain the results in Table 1? Without associating the actual length reduction ratio, it is hard to evaluate the reported FLOPs reduction.\n \n[1] Fan et. al. \"Reducing Transformer Depth on Demand with Structured Dropout\", https://arxiv.org/abs/1909.11556\n\n[2] Jiao et. al. \"TinyBERT: Distilling BERT for Natural Language Understanding\", https://arxiv.org/abs/1909.10351", "extra_scores": {}, "extra_text": {}}, {"review_id": "hfSpA-BICvP", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The work targets an interesting direction of improving the efficiency of Transformers by reducing the sequence length. The main contributions of the work are (1) proposing LengthDrop as the way to achieve length reduction; (2) utilizing techniques developed in NAS, namely one-shot NAS, to enable proper training and allow adaptive drop ratio search after training. All these ideas are very reasonable and interesting. Empirically, the authors show that the proposed method is able to match or even outperform BERT-base model with 1/3 - 1/2 FLOPs during inference (not training).\n\nThe first concern I have is that the source of the gain is unclear in two aspects:\n(1) The proposed model is finetuned for longer (5 + 5 epochs) compared to the 3 epochs of the original BERT. However, the baseline numbers are still based on 3 epochs. This could be one of the reasons why the proposed model even outperform the original BERT.\n(2) With the \"Inplace Distillation\" in play, the obtained model is effectively a distilled model. This adds another layer of complication to judge how much the gain/loss comes from distillation and the length reduction.\n\nSecondly, authors do not mention much about the training (finetuning + ES) cost compared to the standard BERT or Power-BERT. In many real-world cases, this cost is also non-trivial. This may be part of the reason why only 3 datasets are considered in this paper. \n ", "extra_scores": {}, "extra_text": {}}, {"review_id": "edTeUw-WAh", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to make inference more efficient for finetuned contextual representation models such as BERT. The authors extend PowerBERT, a method introduced recently to perform  efficient inference by dynamically reducing input tokens as the model goes deeper. The authors address two limitations of PowerBERT: the need to pre-set the required computational budget during training (which makes the model inflexible), and the inability to tackle span level tasks (which require the full sentence at the final layer). The authors present a simple solution to both problems (though one that requires heavy engineering to work, see below), and show promising results compared to several BERT baselines. Overall, the ideas presented in this paper are simple (in a good way), and I would like to see more work that applies similar ideas to gain efficiency and not just accuracy. However, the paper leaves important questions unanswered, and thus I cannot recommend accepting it. First, how expensive is the proposed evolutionary search compared to standard fine-tuning? Second, how does this approach compare to the original PowerBERT model (despite its limitations)? Third, given that the evolutionary search is performed on the validation set, how do the authors evaluate it during model development? \nI am curious to read the authors' response, and could be convinced to increase my score, but currently this paper does not meet the ICLR bar.\n\nDetailed summary:\n\nThis paper takes PowerBERT, a recently introduced model for efficient inference, and addresses two of its limitations: the need to retrain the model for each computational budget, and the inability to handle span-level tasks such as QA. The authors present simple solutions to both: for the former, they train a model that randomly selects the number of dropped tokens during training, which makes the model more resilient to different levels of pruning, and perform an evolutionary search process to match the exact pruning levels for a given computational budget. For the latter, they restore the tokens dropped at earlier stages in the last layer, allowing the model to access span indices for tokens dropped earlier if necessary. These solutions are solid and seem to work in practice (although they require quite a bit of engineering to work well, see section 3.1). Nonetheless, their implementation leaves a few important questions unanswered:\n\n1. If I understand correctly, the proposed evolutionary search process needs to be run for any computational budget, which makes it suffer from the same problem as the original PowerBERT model. The authors address this concern in section 3.2, saying that \"it only require a single pass through the relatively small validation set for each length configuration\", but later say they \"repeat this iteration G times\" (with G=30) and \"evolutionary search converges after about fifteen iterations\". This leads to the natural question: how much faster is this procedure compared to finetuning the model, as done in PowerBERT? And even if it is faster, it still doesn't result in a completely flexible model, as each new budget level requires rerunning this process.\n2. Another issue with this approach, is that the model is (partially) trained on the development set. This leaves me wondering how it was evaluated during model development. \n3. The authors build on the PowerBERT model, but never compare to it. Although they (supposedly) provide important benefits compared to this model, it is important to understand at what (accuracy and efficiency) cost. \n\nMore comments: \n1. This paper would strongly benefit from a proof-read. Several paragraphs and sentences repeat what was just said (e.g., the paragraph starting with \"It is not trivial to find an optimal length\" on page 2, or the first sentence on page 7). There are multiple typos (e.g., \"Lengnth\" on Table 1)  and grammatical errors (\"it converges rapidly evolutionary search converges after about fifteen iterations.\") all around the paper. Moreover, several issues were unclear to me: \na. what are sub-models? are these layers with fewer tokens? \nb. how is model distillation incorporated in the solution (last paragraph of section 3.1)?\n2. BERT uses word-pieces and not words.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "c8HNRTz3Fa", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThe paper proposed the Length-Adaptive Transformer. The model can be trained once and directly applied to different inference scenarios. To achieve this goal, the author proposed the LengthDrop method, which randomly samples the length at each layer. In addition, the author used the sandwich rule to train the model. At each step, the sandwich rule will train the largest model, the smallest model, and another bunch of randomly sampled models. In the inference phase, the paper proposed to search for the best length configuration that balances the accuracy and latency tradeoff via evolutionary search. Moreover, to generalize the model to token annotation tasks, the author proposed the Drop-and-Restore process, in which the tokens that have been dropped are used again in the final layer. Experiments show that Length-Adaptive Transformer is able to outperfom the baseline models when evaluated at the same latency level.\n\n\n##########################################################################\n\nReasons for score: \n \n\nThe paper is well-written and the length-adaptive idea is reasonable. However, I think it still requires more experiments. Also, the training process is not very clear to the reader. It is not so clear how different techniques impact the final performance and the author has not reported the training time. Due to these two reasons, I choose to vote for weak rejection.\n \n\n##########################################################################\n\nPros: \n \n\n1. The idea of length-adaptive transformer is novel. Perform on-demand truncation of the hidden lengths has not been well explored.\n \n\n##########################################################################\n\nCons: \n \n\n1. The paper needs to conduct experiments on more text classification dataset. Currently, only SST-2 and MNLI are considered. Also, the paper lacks the comparison with \"[NeurIPS2020] DynaBERT: Dynamic BERT with Adaptive Widthand Depth\", which is also able to balance the latency and accuracy.\n\n2. The training process is quite complicated and involves multiple steps, e.g., Length-Drop, LayerDrop, and Sandwich rule. The author has not explained the relative contribution of each techniques. Also, the author has not reported the training time of the whole pipeline.\n\n\n##########################################################################\n\nQuestions during rebuttal period: \n \n\nPlease address and clarify the cons above \n \n\n#########################################################################\n\nTypos: \n\n(1) Section 2.2, Paragraph 2, \"more efficient and more accuracy\" should be \"more efficient and more accurate\"\n(2) Page 7, under Table 1 and Figure 4, \"Lengnth\" should be \"Length\"", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper attempts to reduce computational cost of Transformer models. In this regard, authors generalizer PoWER-BERT by proposing a variant of dropout that reduces training cost by randomly sampling a fraction of the length of a sequence to use at each layer. Further, a sandwich training method is used which trains a spectrum of randomly sampled model between the largest and the smallest size model. At test time, the best length configuration that balances the accuracy and latency tradeoff via evolutionary search is used. The reviewers found the general idea interesting, but raised a number of concerns. First, proper baselines should be used and related works be discussed. In particular, the method is built on top of Power-BERT, yet it does not directly compare with it, and there was no good response when pointed out by a reviewer. Second, as the paper employs many tricks (some new some from prior work), but does not do any ablation studies to show how each of those contributes to the final accuracy gains. Finally, to showcase benefit compared to prior works in terms of computational cost a proper evaluation methodology and actual speedups for batch size 1 inference should be provided. Thus, an improved evaluation would benefit the paper a lot and paper in its current form is not ready for publication.

Decision: Reject