Reducing Transformer Depth on Demand with Structured Dropout
Authors: Angela Fan
Edouard Grave
Armand Joulin
Venue:  (2020)
Keywords: reduction
regularization
pruning
dropout
transformer

Abstract: Overparametrized transformer networks have obtained state of the art results in various natural language processing tasks, such as machine translation, language modeling, and  question answering. These models contain hundreds of millions of parameters, necessitating a large amount of computation	and making them prone to overfitting. In this work, we explore LayerDrop, a form of structured dropout, which has a regularization effect during training and allows for efficient pruning at inference time. In particular, we show that it is possible to select sub-networks of any depth from one large network without having to finetune them and with limited impact on performance. We demonstrate the effectiveness of our	approach by improving the state of the art on machine translation, language modeling, summarization, question answering, and language understanding benchmarks. Moreover, we show that our approach leads to small BERT-like models of higher quality than when training from scratch or using distillation.

REDUCING TRANSFORMER DEPTH ON DEMAND WITH STRUCTURED DROPOUT
Angela Fan
Edouard Grave Facebook Al Research egrave fb. .om
Armand Joulin
Facebook Al Research/LORIA angelafa Eb. com
Facebook Al Research ajoul in Eb. com
ABSTRACT
Overparameterized transformer networks have obtained state of the art results various natural language processing tasks, such as machine translation, language modeling, and question answering. These models contain hundreds of millions of parameters, necessitating a large amount of computation and making them prone to overfitting. In this work, we explore LayerDrop, a form of structured dropout, which has regularization effect during training and allows for efficient pruning at inference time. In particular, we show that is possible to select sub- nub-networks any depth from one large network without having finetune them and with lim- ited impact on performance. We demonstrate the effectiveness of our approach by improving the state of the on machine translation, language modeling, summa- rization, question answering, and language understanding benchmarks. Moreover, we show that our approach leads small BERT-like models of higher quality compared to training from scratch or using distillation.
INTRODUCTION
Transformer architectures (Vaswani et al. 2017 have become the dominant architecture in natural language processing, with otate-of-the-art performance across variety of tasks, including machine translation (Vaswani et al.) 2017 Ott et al., 2018), language modeling (Dai et al. 2019 Baevski & Auli 2018) and sentence representation (Devlin et al.) 2018 Yang al.] 2019)) Each of its lay- ers contains millions of parameters accessed during the forward pass, making it computationally demanding in terms of memory and latency during both training and inference. In an ideal situ- ation, we would be able to extract sub-networks this over- parameterized network, for any given memory or latency constraint, while maintaining good performance. In contrast, standard pruning distillation methods follow strategy that often includes a finetuning or retraining step, and the process must be repeated for each desired depth.
automatically and without finetuning from
In this work, propose novel approach extract any sub-network without post-hoc pruning process from over- seer-parameterized networks. The core of our method is to sample small sub-networks from the larger model during training by randomly dropping model weights as in Dropout (Hinton et al. 2012) or DropConnect (Wan et al. (2013). This has the advantage making the network robust subsequent pruning. If well-chosen groups of weights are dropped simultaneously, the resulting small sub-networks can be very efficient. In particular, we drop entire layers to extract shallow models inference time. Previous work (Huang al. 2016) has shown that dropping layers during training can regularize and reduce the training time of very deep convolutional networks. In contrast, we focus on pruming. As illustrated in Figure [ an advantage of our layer dropping technique, or LayerDrop, is that from one single deep model, we can extract shallow sub-networks of any desired depth on demand at inference time.
variety of competitive benchmarks, namely WMT14 English- German for machine translation, WikiText-103 (Merity et al., 2016) for language modeling, CNN- Dailymail (Hermann et al.) 2015) for abstractive summarization, ELI5 (aan et al. 2017) for long form question answering, and several natural language understanding tasks (Wang et al. 2019al for sentence representation. Our approach achieves state of the art most of these benchmarks sult of the regularization effect, which stabilizes the training of larger and deeper networks. We also show that prune Transformer architectures to much smaller models while maintaining com-
We validate our findings

---

Pruned to
Train 9 Layer Model
Pruned to
Train Layer Model
Pruned to
Train 3 Layer Model
On Demand Depth Selection
Train One Full Network
Train Separate
Networks
TRAIN+ + TEST TIME
Docrcasing Model Size
TEST TIME TRAIN TIME
Figure I: LayerDrop (right) randomly drops layers at training time. At test time, this allows for sub-network selection to any desired depth as the network has been trained to be robust to pruning. In contrast to standard approaches that must re-train new model from scratch for each model size (left), our method trains only one network from which multiple shallow models can be extracted.
petitive performance, outperforming specific model reduction strategies dedicated to BERT (Devlin et al. 2018; Sanh 2019) as well as training smaller models from scratch. Overall, applying Layer- Drop to Transformer networks provides the following key advantages:
LayerDrop regularizes very deep Transformers and stabilizes their training, leading state- of-the-art art performance across a variety of benchmarks.
Small and efficient models of any depth can be extracted automatically at test time from single large pre-trained model, without the need for finetuning.
LayerDrop is as simple to implement as dropout.
2 RELATED WORK
Our approach is form of Dropout (rivasstvaa et al. 2014) applied to model weights instead of activations, as in DropConnect (Wan et al. 2013). Different from DropConnect, we drop groups of weights to induce group redundancy to create models suited for pruning to shallow, efficient models inference time. [ooee al.) (2018) propose a targeted Dropout and DropConnect, where they learn the drop rate of the weights to match targeted pruning scheme. Instead, we adapt the masks to the structures that we are interested pruning. Closer to our work, the Stochastic Depth approach of Huang et al. 2016) drops layers randomly during training. As opposed to our work, they are interested in accelerating the training of very deep ResNets (He et al., 2016), so their dropping schedule is adapted to this goal. Concurrently to this work, |ham et al. (2019) applied Stochastic
Depth to train very deep Transformers for speech and show the benefits of its regularization effect.
method is a form of structured pruning (Liu al. 2018h). As opposed to weight pruning (LeCun et al.] 1990). structured pruning removes coherent groups of weights to preserve the original structure of the network. Structured pruning has been used in some NLP applications, such as machine translation (See et al. 2016). text classification (Joulin et al., (016)) and language modeling (Murray & Chiang 2015). However, it has been more widely adopted in computer vision and applied to convolutional network to remove filters (Li et al. 2016 Wen et al., 2016), channels (He et al., 2017). or residual blocks (Huang et al. 2018 Huang Wang 2018).
More generally,
Similar to Mittal et al. 2018), we take advantage of the plasticity of neural networks to learn models that are resilient to random pruning or skipping connections Wang et al. (2018); Wu et al.
( (2018) Liu
ct al. (2018a), rather than learning the pruning itself. We refer the reader to Liu et al. (2018b) for an exhaustive study of these approaches and their evaluation in the context of convolutional networks.
Reducing the memory footprint of Transformer architectures and BERT in particular is an active subject of research. Several works have compressed BERT as a post- sest-rocessing step using different forms of distillation (Ture et al. 2019: Tang et al. 2019: Shulga 2019 Sanh 2019). Similarly, various papers have shown evidence that Transformers are over parameterized, especially that most self-attention heads can be dropped at test time (Michel al. 2019 Voita et al. 2019) Different
2

---

from these, our models are trained to be resilient to pruning, which significantly reduces the perfor- mance drop induced by test time pruming. Others have proposed trainable adaptive mechanisms to control their memory footprint (Jemntte al. 2016 SSukhbaatar al. [019 Correia et al. 2019). These approaches are complementary to ours and should benefit from each other.
3
METHOD
In this section, we briefly introduce the Transformer, then describe our Structured Dropout technique and its application to layers. We also discuss several inference time pruning strategies.
3.1 THE TRANSFORMER ARCHITECTURE
We succinctly review the Transformer architecture and refer the reader to Vaswani et al. (2017) for additional details. A Transformer is a stack of layers composed of two sub-layers: multi-head self-attention followed by a feedforward sub-layer. The multi-head self-attention sub-layer consists of multiple attention heads applied parallel. Each attention head takes matrix X where each row represents an element of the input sequence and updates their representations by gathering information from their context using an Attention mechanism (Bahdanau et al., 2014):
Y Softmax (X (X" K(QX + P)))X,
where K, V, Q and P are matrices of parameters. The outputs of the heads are then concatenated along the time step into a sequence of vectors.
The second sub-layer then applies a fully connected feedforward network to each element of this U ReLU (Vx), where V and U are matrices of parameters.
sequence independently, FFN(x) Each sub-layer is followed by a AddNorm operation that is residual connection (He et al. 2016) and layer normalization (Ba et al. 2016).
3.2 TRAINING TRANSFORMERS WITH RANDOM STRUCTURED PRUNING
We present regularization approach that makes Transformers robust to subsequent structured prun- ing at inference time. We focus in particular on the case where the targeted structure is a layer.
3.2.1 RANDOMLY DROPPING STRUCTURES AT TRAINING TIME
Regularizing networks be robust to pruning can be achieved by randomly removing weights dur- ing its training as in DropConnect (Wan et al. 2013). In this approach, each weight is dropped independently following Bernoulli distribution associated with parameter p > that controls the drop rate. This is equivalent to pointwise multiplication of the weight matrix W with randomly sampled {0, 1} mask matrix M:
W M W.
DropConnect is a form of random unstructured pruning that leads to smaller, but not necessarily more efficient, models. We propose add structure this mechanism target model efficiency.
Random Structured Dropout. The weights of a Transformer network belong to multiple over- lapping structures, such as heads, FFN matrices, or layers. Dropping weights using groups that follow some of these inherent structures potentially leads to significant reduction of the inference time. This is equivalent to constraining the mask M to constant over some predefined groups of weights. More precisely, given a set G of predefined groups of weights, the (0, 1} mask matrix M randomly sampled over groups instead of weights:
Vi, Mli] E 1), and VG G, v(i,)) E G. Mi MD].
This structured dropout formulation is general and can be applied to any overlapping groups of weights, whether heads, FFN matrices, layers. Nonetheless, not all of the structures in Trans- former lead to the same benefits when dropped. For example, dropping attention heads does not usually computed in parallel. For simplicity, we focus dropping lay-
reduce runtime as they ers, and we name this structured pruning, LayerDrop. This inspired by the Stochastic Depth approach of Huang et al. (0166) used train very deep ResNets (He et al. 2015).
3

---

3.2.2 PRUNING AT INFERENCE TIME
Selecting Layers to Prune Training with LayerDrop makes the network more robust to predicting with missing layers. However, LayerDrop does not explicitly provide way to select which groups to prune. We consider several different pruning strategies, described below:
Every Other: A straightforward strategy is to simply drop every other layer. Pruning with rate p means dropping the layers depth d such that d ( 1,.). This strategy is intuitive and leads to balanced networks.
Search Valid: Another possibility is to compute various combinations of layers to form shallower networks using the validation set, then select the best performing for test. This is straightforward but computationally intensive and can lead to overfitting validation.
Data Driven Pruning: Finally, we propose data driven pruning where we learn the drop rate of each layer. Given target drop rate p. we learn an individual drop rate Pa for the layer at depth d such that the average rate over layers is equal to p. More precisely, we parameterize Pa as non-linear function of the activation of its layer and apply softmax. At inference time, we forward only the fixed top-k highest scoring layers based on the softmax output (e.g. chosen layers do not depend on the input features).
In practice, we observe that the Every Other strategy works surprisingly well across many tasks and configurations. Search Valid and Data Driven Pruning only offer marginal gains. Note that do not further finetune any of the pruned networks (see Appendix for analysis of finetuning).
Setting the drop rate for optimal pruning. There is straightforward relationship between the drop rate of groups and the average pruning level that the network should be resilient to. Assuming N groups and fixed drop ratio p. the average number groups used by the network during training is [1 p). As a consequence, to target pruning size of r groups, the optimal drop rate is:
p
N
In practice, we observe that networks are more robust to pruning than their expected ratio but higher pruning rates leads to better performance for smaller models. We use a LayerDrop rate of p = 0.2 for all our experiments, but we recommend p 0.5 to target very small inference time models.
EXPERIMENTAL SETUP
We apply our method to a variety of sequence modeling tasks: neural machine translation, lan- guage modeling, summarization, long form question answering. and various natural language un- derstanding tasks. Our models are implemented in PyTorch using fa i rseq-py (Our et al. 2019 [ Additional implementation and training details with hyperparameter settings are the Appendix.
Neural Machine Translation. We experiment on the WMT English- Gerrmnn machine translation benchmark using the Transformer Big architecture. We use the dataset of 4.5M en-de sentence pairs from WMT16 (Vaswani et al. 2017) for training, newstest2013 for validation, and newstest2014 for test. We optimize the dropout value within the range {0.1, 0.2, 0.5} on the validation set and set the LayerDrop rate 0.2. For generation, penalty to 0.6, and beam size to 8, following the settings suggested in Wu et (2019a). and measure case-sensitive tokenized BLEU, We apply compound splitting, as used in Vaswani et al. (2017).
average the last 10 checkpoints, set the length
Language Modeling, We experiment on the Wikitext- 103 language modeling benchmark (Merity al. 2016) which contains 100M tokens and large vocabulary size of 260K. We adopt the 16 layer Transformer used in Baevski & Auli ( (2018). We set the LayerDrop rate p to 0.2 and tune the standard dropout parameter in (0.1, 0.2, 0.3] on the validation set. We report test set perplexity (PPL).

---

Model
Enc Layers
Dec Layers
BLEU
Transformer Vaswani ct al.
2017)
28.4
Transformer OOt et al. 2018
6
6
29.3
DynamicConv Wu et al.
201 2019a>
6
29.7
Transformer Ott et al. Transformer
2018 8
+ LayerDrop
6
6
29.6
Ott et al.
2018
LayerDrop
12
30.2
Table 1: Results on WMT en-de Machine Translation (newstest2014 test set)
Model
Layers
Params
PPL
Adaptive Inputs (Baevski & Auli 2018
16
247M
18.7
Transformer XL Large DDa al. 2019)
18
257M
18.3
Adaptive Inputs + LayerDrop
16
247M
18.3
Adaptive Inputs + LayerDrop
40
423M
17.7
Table 2: Results on Wikitext-100 103 language modeling benchmark (test set).
We adopt the Transformer base architecture and training schedule from Edunov et al. (2019) and experiment on the CNN-Dailymail
Summarization.
summarization benchmark. The training data contains over 280K full-text news articles paired with multi-sentence summaries (Her- mann et al. 2015 See et al. 2017). We tune generation length in the range {40, 50, 60) and use 3-gram blocking. We set the LayerDrop rate p to 0.2. We evaluate using ROUGE (Lin 2004).
Long Form Question Answering. We consider the Long Form Question Answering Dataset ELI5 Fan et al. (2019), which consists of 272K question answer pairs from the subreddit Explain Like I I'm Five along with extracted supporting documents from web search. We follow the Transformer Big architecture and training procedure Fa al. (2019). We generate long answers using beam search with beam size and apply 3-gram blocking (Fan et al. 2017). We evaluate with ROUGE.
Sentence representation Pre-training. We train base and large BERT (Devlin et al. 2018) mod- els following the open-source implementation of al. (2019). We use datasets: Bookscor- pus + Wiki from Liu et al. (2019) and the larger combination of Bookscorpus + OpenWebTTex 2019). We evaluate the pretrained models on various natural language understanding tasks. Specifically, we evaluate accuracy on MRPC (Dolan & Brocket 2005). QNLI (Rajpurkar al. 2016). MNLI (Williams et al. 2018), and SST2 (Socher al. 2013).
CC-News Stories (Liu
5
RESULTS
5.1
LRYPRDROP AS A REGULARIZER
Language Modeling. In Table we show the impact of LayerDrop on the performance of Trans- former network trained in the setting of Adaptive Inputs (Baevski & Auli, 2018). Adding LayerDrop 16 layer Transformer improves the performance by 0.4 perplexity, matching the state-of-the-art results of TrLLsformer-XL. Our 40 layer Transformer with LayerDrop further improves the state of the art by 0.6 points. Very deep Transformers are typically hard to train because of instability and memory usage, and they prone to overfitting regularizes the network, reduces the memory usage, and increases training stability as fewer layers active each forward pass. These results confirm that this type of approach be used to efficiently train very deep networks, as shown in Huang et al. 2016) for convolutional networks.
small dataset like Wikitext-1033 103. LayerDrop
Sequence to sequence modeling. Similarly, as shown Table I and Table 3 applying Layer- Drop to Transformers text generation tasks such neural machine translation, summarization, and long form question answering also boosts performance for all tasks. In these experiments, we take the Transformer architectures that state-the-art and train them with LayerDrop. In
5

---

Model
Enc
Dec
ROUGE-1
ROUGE-2
ROUGE-L
Abstractive Summarization
Transformer (Edunov et al. 2019
6
40.1
17.6
36.8
Transformer + Drop
40.5
17.9
37.1
Transformer LayerDrop
8
41.1
18.1
37.5
Long Form Question Answering Transformer Multitask Fan et al. 2019
6
28.9
5.4
23.1
Transformer Multitask + LayerDrop
29.4
5.5
23.4
Table 3: Results for CNN-Dailymail Summarization and ELI5 QA (test set).
Data
Layers
Model
MNLI-m
MRPC
QNLI
SST2
Books Wiki
24
RoBERTa
89.0
90.2
93.9
95.3
24
RoBERTa LayerDrop
89.2
90.2
94.2
95.4
+ more data
24
RoBERTa
90.2
90.9
94.7
96.4
24
RoBERTa LayerDrop
90.1
91.0
94.7
96.8
48
RoBERTa LayerDrop
90.4
90.9
94.8
96.9
Table 4: Results on Various NLU Tasks for RoBERTa Large trained for 500K updates (dev set),
ral machine translation on newstest2014, our 12 encoder layer Transformer model with LayerDrop further improves the state of the art, reaching 30.2 BLEU. In comparison, standard Transformer trained without LayerDrop diverges with 12 encoder layers. This is a known problem, and tech- niques such improved initialization could be used to maintain stability Jlnczys-Dowmunt 2019) Zhang et al. 2019; Wang et al.) 2019b] Wu et al., 2019b), but are out the scope of this work. Similar results
summarization.
Bi-Directional Pre-training. In second set of experiments, look the impact of LayerDrop on pre-training for sentence representation models and subsequent finetuning on multiple natural language understanding tasks. We compare our models
variant of BERT for sentence represen- tations, called RoBERTa (Liu et al. 2019), and analyze the results of finetuning for data adaptation on MNLI, MRPC, QNLI, and SST2. We apply LayerDrop during both pre-training and finetuning.
We compare the performance of the large architecture on the BooksCorpustWins Wiki dataset used BERT. We analyze the performance training on the additional data used in RoBERTa as well as pre- pre-training for even longer. Comparing fixed model size and training data, LayerDrop can improve the performance of RoBERTa on several tasks. LayerDrop can further be used both enable and stabilize the training (Huang et al.) 2016) of models double the size for even stronger performance.
5.2 PRUNING TRANSFORMER LAYERS TO ON-DEMAND DEPTH WITH LAYER DROP
Pruning Generation Tasks. In Figure 2 investigate the impact of the number of pruned de- coder layers on the performance of a Transformer for language modeling, neural machine transla- tion, and summarization. We compare three different settings: standard Transformer models trained without LayerDrop but subsequently pruned, standard Transformer models trained from scratch to each desired depth, and lastly our approach: pruning layers of Transformer trained with Layer- Drop. Our model is trained once with the maximum number of layers and then pruned to the desired depth, without any finetuning in the shallower configuration. Our approach outperforms small mod- els trained from scratch, showing that LayerDrop leads to more accurate small models at whole range of depths. Further, training with LayerDrop does not incur the computational cost of retrain- ing model for each desired depth. For completeness, dropping layers of deep Transformer trained without LayerDrop performs poorly as it was not trained to be robust to missing layers.
Pruning BERT-like Models. In Table 7 (left), we compare pruning Transformers trained with LayerDrop different approaches used to create smaller, shallower models. We compare to BERT

---

Language Modeling
Machine Translation
Summarization
35
22
30
40
Perplexity
21
BLEU
25
ROUGE
39
20
20
19
10
36
18
5
35
10
20
30
40
50
0
20
40
60
0
20
40
60
Percentage of layers pruned
Percentage of layers pruned
Percentage of layers pruned
Buseline
Trained from scratch
LayerDrop
Figure 2: Performance as a function of Pruning on various generation tasks (test set), compared to training smaller models from scratch and pruning Transformer baseline trained without LayerDrop. Pruning networks with LayerDrop performs strongly compared to these alternatives.
85
84
MNLI
SST2
83
93
Accuracy
82
Accuracy
92
6 Layers (50% Pruned)
81
91
RoBERTa
82.3
92.1
80
+ LayerDrop
82.9
92.5
90
79
+ more data
84.1
93.2
78
89
77
88
3 Layers (75% Pruned)
75
75
RoBERTa
78.1
90.3
Pereentage of layers pruned
Percentage layers pruned
LayerDrop
78.6
90.5
more data
82.2
92.0
BERT trained from scratch
LayerDrop DistiBBERT
A RoBERTa trained from scratch
Figure 3: (left) Performance as a function of Pruning on MNLI and SST2 compared to BERT and RoBERTa trained from scratch and DistiBBERT. Pruning one network trained with LayerDrop (blue) outperforms alternatives that require a new network for each point. (right) Performance when Training on More Data shows even stronger results on MNLI and SST2 for pruned models.
base and RoBERTa base trained from scratch with 6 and 3 layers as well as recent work on dis- tillation, called DistilBERT (Sanh 2019). We analyze both BERT and RoBERTa models as the vocabulary not the same due to differences subword tokenization, which affects performance.
DistilBERT occasionally performs worse than BERT of the same size trained from scratch, which confirms the findings of Liu et al. (2018b) about the performance of pruned models compared to training small models from scratch. Our approach, however, obtains results better than BERT and RoBERTa trained from scratch. Further, our method does not need any post-processing: we sim- ply prune every other layer of our RoBERTa model that has been pre-trained with LayerDrop and finetune the small models on each of the downstream tasks, following standard procedure. When training with additional data, shown in Table Taige even stronger performance can be achieved.
ABLATION STUDIES
Comparison Structured Dropout Flgute contrasts various forms of structured dropout: dropping attention heads, FFN matrices, and entire Transformer layers. Dropping heads alone is worse than dropping entire sub-layers or layers. It also offers no advantage in terms of running time attention heads are computed in parallel for computational efficiency. We observe large differences between dropping sub-layers and layers, possibly because we are working with relatively shallow networks. In theory, dropping sub-layers should perform better and expect this be the

---

Structured Dropout
Sub-sclection Techniques
19
28
29.3
Perplexity
18.6
18.5
18.4
Perplexity
26
26.1
18.2
18.2
24
18.0
22
20
19.7
19.6
19.5
17
16
Half Baseline Head Sublayerllead+ Layer FFN
Ilead+ Layer
Last 8
First
Every other
Search for 8
Data driven
Sublayer
Figure 4: (left) Impact of Various Structured Dropouts on Wikitext-103 Valid. Dropping Lay- ers is straightforward and has strong performance. (right) Comparison of Pruning Strategies on Wikitext- 103 Valid. Marginal gains can be achieved, but dropping every other layer hard beat.
0.1
22
21.0
Train drop rate-0.1
Train drop rate
0.2
20.5
Perplexity
Parplexity
O Train drop rate-0.4
20.0
0.4
21
O Train drop rate=0.5
0.5
19.5
19.0
20
18.5
0
3
5
7
8
10
11
12
13
14
15
0.0
0.1
0.2
0.3
0.4
0.5
0.6
Layer number
Percentage of pruned layers
Figure Relative Importance of Specific Layers. (Wikitext-103 Valid) The full network is pruned into various layer sub-network configurations, and the av- erage perplexity pruning layer n is displayed above.
Figure 6: Effect of Train LayerDrop on Inference-time Pruning. (Wikitext-103 Valid) Training with larger LayerDrop is beneficial for significant pruning.
case with very deep Transformers. We experiment with overlapping structured groups, such as heads layers and heads + sub-layers and find that the beneficial effect can be advantagcously combined. We focus on layers for simplicity, as dropping more structures introduces more parameters to tune.
Comparison of Various Pruning Strategies. Figure (right) contrasts various approaches to sub-selecting model layers at inference time.
The predominant method used in this paper, the straightforward strategy of selecting every other layer, is tough to beat. We find only marginal improvement can be gained by searching over the validation set for the best of 8 layers use and by learning which layers drop. In contrast, dropping chunks consecutive layers is harmful. Namely, removing the first half or last half of model particularly harmful, as the model does not have the ability to process the input or project to the full vocabulary to predict the subsequent word.
Not all layers are equally important. In an experiment on Wikitext-103. we pruned selections of layers at random. Figure [ displays the perplexity when that layer is removed, averaging results from 20 pruned model per layer. The input and output layers of network are the most important, as they process the input and project to the output vocabulary.
Choosing which Layers to Prune.
Relationship between LayerDrop at Training Time and Pruning at Inference Time. Figure displays the relationship between the training time LayerDrop and the performance of pruned network at test time. If significant depth reduction desired, training with larger LayerDrop is beneficial this equalizes the train and test time settings. An analysis for BERT is in the Appendix.
8

---

7 CONCLUSION
Structured dropout regularizes neural networks to be more robust to applying structured pruning at inference time. We focus on the setting where structures are layers, enabling pruning of shallow and efficient models of any desired depth. In variety of text generation and pre-training tasks, we show that LayerDrop enables and stabilizes the training of substantially deeper networks and simultaneously allows for the extraction of models of various depths with strong performance.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1607.06450 2016.
Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. arXiv preprint 1rXiv:109.108833 2018.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409 2014.
Gonçalo M Correia, Vlad Niculae, and André FT Martins. Adaptively sparse transformers. arXiv preprint veXiiv:199.0000
2019.
Zihang Dai, Zhilin Yang. Yiming Yang. William W Cohen, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-ke xl: Attentive language models beyond fixed-length context. arXiv preprint arXiv: 2019.
Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Pruc. of ICML, 2017.
Jacob Devlin, Ming-Wos Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04480 04805, 2018.
William B Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the International Workshop Paraphrasing, 2005.
Sergey Edunov, Alexci Baevski, and Michael Auli. Pre-trained language model representations for language generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4052 4059, 2019.
Angela Fan, David Grangier, and Michael Auli. Controllable abstractive summarization. arXiv, abs/1711.05217, 2017.
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. Eli5: Long form question answering. arXiv preprint arXiv:1907.09190 2019.
Aidan N Gomez, Ivan Zhang, Kevin Swersky, Yarin Gal, and Geoffrey E Hinton. Targeted dropout. 2018.
Edouard Grave, Armand Joulin, Moustapha Cisse, David Grangier, and Herve Jegou. Efficient 2016.
softmax approximation for gpus. arXiv,
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. Proc. of CVPR, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEFE conference on computer vision and patlern recognition, pp- 770-778, 2016.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. Proceedings of the IEEE International Conference on Computer Vision, pp. 1389-1397, 2017.

---

Karl Moritz Hermann, Tomás Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Proc. of NIPS, 2015.
Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdi- nov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580, 2012.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. European conference on computer vision, pp. 646-661. Springer, 2016.
Gao Huang, Shichen Liu, Laurens Van der Maaten, and Kilian Q Weinberger. Condensenet: An efficient densenet using learned group convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2752-2761, 2018.
Zehao Huang and Naiyan Wang. Data-driven sparse structure selection for deep neural networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 304-320, 2018.
Yacine Jernite, Edouard Grave, Armand Joulin, and Tomas Mikolov. Variable computation in recur- rent neural networks. arXiv preprint 1rXiv:1611.061880 2016.
Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. Fasttext, zip: Compressing text classífication models. arXiv preprint arXiv:1612.0366: 2016.
Marcin Junczys-Dowmune Microsoft translator at wmt 2019: Towards large-scale document-leve neural machine translation. arXiv preprint arXiv:1907.06170 2019.
Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291, 2019.
Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems, pp. 598-605, 1990.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.000 08710, 2016.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Workshop on Text Sunmarization Branches Out, 2004.
Liyuan Liu, Xiang Ren, Jingbo Shang, Jian Peng, and Jiawei Han. Efficient contextualized rep- resentation: Language model pruning for sequence labeling. arXiv preprint (844::8804.08271 2a.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: (90.:1997.11100 1169,, 2019.
Zhuang Liu, Mingjee Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. arXiv preprint 1rx.v:1810.055200, 2018b.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.0333 03988, 2016.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer Sentinel Mixture Models. arXiv, abs/1609.078433, 2016.
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? arXiv preprint 1wXXiv1905.106510 2019.
Deepak Mittal, Shweta Bhardwaj. Mitesh M Khapra, and Balaraman Ravindran. Recovering from random pruning: On the plasticity of deep convolutional neural networks. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 848-857. IEEF, 2018.
10

---

Kenton Murray and David Chiang. Auto-sizing neural networks: With applications to n-gram lan- guage models. arXiv preprint arXiv: 2015.
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation.
In Proc. of WMT, 2018.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-IILT 2019: Demonstrations, 2019.
Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. How to construct deep recurrent neural networks. In Proceedings of the Second International Conference on Learning Representations (ICLR 2014), 2014.
Ngoc-Quan Pham, Thai-Son Nguyen, Jan Nichues, Markus Muller, and Alex Waibel, Very deep self- attention networks for end-to-end speech recognition. arXiv preprint arXiv: /3377, 2019.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners. 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of EMNLP, pp. 2383-2392. Association for Computational Linguisties, 2016.
Victor Sanh. Smaller, faster, cheaper, lighter: Introducing distilbert, distilled version of bert.
1bert ----88333333333 2019.
Abigail See, Minh-Thang Luong, and Christopher D Manning. Compression of neural machine translation models via pruning. arXiv preprint arXiv: 09274, 2016.
Abigail See, Peter J Liu, and Christopher D Manning. Get to the point: Summarization with pointer- generator networks. Proc. of ACL,, 2017.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint 1iiv:1508.0790 2015.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proc. of ACI. 2016.
Dima Shulga. Distilling bert how to achieve bert performance using logistic regression. towards- datascience.com. 2019.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of EMNLP, pp. 1631-1642, 2013.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. JMLR, 1999 2014.
Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span transformers. arXiv preprint arXiv:1905.07700 2019.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance initial- ization and momentum in deep learning. In International conference on machine learning, pp-
1139-1147, 2013.
Raphael Tang, Yao Lu, Linging Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. Distilling task- specific knowledge from bert into simple neural networks. arXiv preprint arXiv: 1366, 2019.
Iulia Ture, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better: The impact student initialization on knowledge distillation. arXiv preprint 190.v 2019.
11

---

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. -19555998-99 2017.
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov, Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint arXiv:1905.09418. 2019.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In International conference on machine learning, pp. 1058-1066, 2013.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. 2019a. In the Proceedings of ICLR.
Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F Wong, and Lidia S Chao. Learning deep transformer models for machine translation. arXiv preprint #rXiv:/906.0/787 2019b.
Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E Gonzalez. Skipnet: namic routing in convolutional networks. In Proceedings of the European Conference on Com- puter Vision (ECCV), pp. 409 424, 2018.
Learning dy-
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in neural information processing systems, pp. 2074-2082, 2016.
Adina Williams, Nikita Nangia, and Samuel R. Bowman. broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of NAACI-HILT, 2018.
Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. In International Conference on Learning Representations, 2019a. URL RtRLhhtts:://arttt
www.oogg////o..o.. abrgg/bbs/900 9011 10430
Lijun Wu, Yiren Wang, Yingce Xia, Fei Tian, Fei Gao, Tao Qin, Jianhuang Lai, and Tie- Yan Liu. Depth growing for neural machine translation. arXiv preprint arXiv:1907.01968 2019b.
Zuxuan WU, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S Davis, Kristen Grauman, and Rogerio Feris. Blockdrop: Dynamic inference paths residual networks. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pp. 8817-8826, 2018.
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoe V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906. 08237, 2019.
Hongyi Zhang, Yann N Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. arXiv preprint arXiv: 1901.0990 2019.
12

---

A APPENDIX
A.1 ADDITIONAL IMPLEMENTATION DETAILS
A.1.1 NEURAL MACHINE TRANSLATION
WMT en-de: We model a 32K joint byte-pair encoding. We train using the cosine (Loshchilov & Hutter (016) learning rate schedule from Wu et al. (2019a) with label smoothing 0.1. vocabu- lary (Sennrich et al. 2015). We train on 8 GPU for total training time 66k seconds.
IWSTT de-en: The dataset consists of 160K training pairs, fully lowercased. We model 10K joint BPE vocabulary and generate with beam size 4. We do not average checkpoints. Following Wu et al.) (0119)), we use the Transformer base architecture with 6 encoder layers and 6 decoder layers. As the dataset is small, we decrease the overall model size and instead use the following parameters: FFN size 1024, hidden dimension 512, and attention heads. We train on GPU.
Pruning: We apply the Every Other Layer strategy to the decoder and do not finetune.
A.1.2 LANGUAGE MODELING
Training: To handle the large vocabulary of Wikitext- 103, we follow Dauphin et al. (2017) and Baevski & Auli (2018) in using adaptive softmax (Grave al. 2016) and adaptive input for com- putational efficiency. For both input and output embeddings, we use dimension size 1024 and three adaptive bands: 20K, 40K, and 200K. We use cosine learning rate schedule (Baevski & Auli 2018 Loshchilov & Hutter, 2016) and train with Nesterov's accelerated gradient (Sutskever et al., 2013). We set the momentum 0.99 and renormalize gradients if the norm exceeds 0.1 (Pascanu et al.) 2014). During training, we partition the data into blocks of contiguous tokens that ignore document boundaries. At test time, we respect sentence boundaries. train on 8 GPU for total training time of 216k seconds.
Pruning: We apply the Every Other Layer strategy and do not finetune.
A.1.3 SUMMARIZATION
Data: We use the full text (non-anonymized) version of CNN-Dailymail introduced by [See et al. (017). Following Fan et al. (2017). we truncate articles to 400 tokens and model joint byte-pair vocabulary of 32K types (Sennrich et al. 2016).
Training: We train using Adam with cosine learning rate schedule, warming up for 10K steps. We optimize dropout the range {0.2, 0.3) on the validation set and set LayerDrop to 0.2. We train on 1 GPU.
Pruning:
We apply the Every Other Layer strategy to the decoder and do not finetune.
A.1.4 LONG FORM QUESTION ANSWERING
Training: We compare to the full multi-task setting of Fan et al. (2019), where data augmentation and multi-tasking is done at training time to increase the data available. We train on 8 GPU.
Generation:
We set the minimum length 150 tokens and the maximum length 200.
A.1.5 BI-DTRECTIONAL PRE-TRAINTS TRAININGG
The base architecture is a layer model with embedding size 768 and FFN size 3072. The large architecture consists of 24 layers with embedding size 1024 and FFN size 4096. For both settings, we follow Liu et al./ (2019) in using the subword tokenization scheme from Radford et al. (2019), which uses bytes subword units. This eliminates unknown tokens. Note this produces a different vocabulary size than BERT (Devlin et al., 2018), meaning models of the same depth do not have the same number of parameters. We train with large batches of size 8192 and maintain this batch size using gradient accumulation. We do not use next sentence prediction (Lample & Conneau 2019.. We optimize with Adam with polynomial decay learning rate schedule. For
Training:
13

---

Hyperparameter
Base
Large
Number of Layers
12
24
Hidden Size
768
1024
FFN Size
3072
4096
Attention Heads
12
16
LayerDrop
0.2
0.2
Warmup Steps
24k
30k
Peak Learning Rate
6e-4
4e-4
Batch Size
8192
8192
Table 5: Hyperparameters for RoBERTa Pretraining
Model
BLEU
Transformer (Wu et al. 2019a
34.4
Dynamic Conv (Wu et al. 2019a)
35.2
Transformer + LayerDrop
34.5
Table 6: BLEU for IWSLT (test set).
BERT-Base, we use 32 GPU (total training time 171k seconds) and for BERT-Large, we use 128 GPU. For the RoBERTa data setting with more data, we use 512 GPU to train BERT-Large.
During finetuning, we hyperparameter search over three learning rate options (le-5, 2e-5, 3e-5) and batchsize (16 or 32 sentences). The other parameters are set following Liu et al. (2019). We do single task finetuning, meaning we only tune on the data provided for the given natural language understanding task. We do not perform ensembling. When finetuning models trained with LayerDrop, we apply LayerDrop during finetuning time as well.
Finetuning:
Training smaller models: We train the and 3 layer RoBERTa models following the same settings, but using the smaller number of layers and without LayerDrop. We finetune with the same sweep parameters. The 6 and 3 layer BERT model results taken from Devlin et al. 2018).
Training larger models: We train the 48 layer RoBERTa model with 0.5 LayerDrop so only 24 layers average active during forward pass.
Pruning: When pruning RoBERTa models, we use the Every Other Layer strategy and finetune without LayerDrop for the smaller models.
A.2 ADDITIONAL RESULTS
IWSLT Table displays results on the IWSLT de-en dataset. We see small improvement, likely as the network is small and already has large quantity of regularization with dropout, attention dropout, and weight decay. The Transformer is not the state of the art architecture, and there remains a large gap between the Transformer and the DynamicCony model proposed by Wu et al. (2019a).
Pruning BERT Models The numerical values corresponding to the pruned and layer RoBERTa + LayerDrop models are shown in Table
A.3 ADDITIONAL ANALYSIS
Impact of LayerDrop training time. Figure shows the increase in training speed when training with increasingly large quantities of LayerDrop. The words per second were computed on 8 V100 GPUs with 32GB of memory, without floating point 16, for 16 layer model trained on Wikitext- 103. Assuming fixed layer size, LayerDrop removes layers at training time randomly, which increases the training speed almost 2x if dropping half the number of layers.
14

---

Model
Dataset
Layers
MNLI-m
MRPC
QNLI
SST-2
BERT Books + Wiki
BERT
6
81.9
84.8
91.3
Distil BERT (anhh 2019
Books + Wiki
6
81.6
82.4
85.5
92.7
RoBERTa
Books + Wiki 6
82.3
82.5
89.7
92.1
RoBERTa Lrye
Books + Wiki 6
82.9
85.3
89.4
92.5
RoBERTa + LayerDrop
+ more data
6
84.1
86.1
89.5
93.2
Books + Wiki 3
77.9
79.8
88.4
RoBERTa
Books + Wiki 3
78.1
79.4
86.2
90.3
RoBERTa + LayerDrop
Books + Wiki 3
78.6
75.1
86.0
90.5
RoBERTa + LayerDrop
+ more data
3
82.2
79.4
88.6
92.0
Table 7: Comparison between BERT base with and without distillation with RoBERTa base trained with LayerDrop. Our models are pruned before finetuning on each individual task. The numbers from BERT are taken from Devlin al. (2018).
Training Speed
Model
Valid PPL
Words per sec.
60
Pruned w/ LayerDrop
20.78
+ Finetune
20.56
40
Table 8: Impact of additional finetuning 16 layer language model pruned to layers.
20
0.1
0.2
0.3
0.4
0.5
Figure 7: Effect of LayerDrop on Training
Time
MNLI
SST2
QNLI
85
94.0
92
84
93.5
91
Accuracy
83
Accuracy
93.0
Accuracy
90
92.5
81
92.0
88
80
91.5
87
79
91.0
86
78
90.5
85
50
75
50
75
50
75
Percentage of pruned layers
Percentage of pruned layers
Percentage of pruned layers
Train drop rate
Train drop rate
Train drop rate-0.5
Figure 8: Effect of Train LayerDrop Inference-time Pruning MNLI, SST2, and QNLI
BERT: Relationship between LayerDrop at Training Time and Pruning Inference Time Similar to the analysis Language Modeling, find that training with larger quantities of Layer- Drop allows for more aggressive pruning at inference time on various natural language generation tasks. However, these tasks involve finetuning step the downstream tasks after pre-training, the effect is less straightforward. Results are shown in Figure X
Impact of Finetuning. LayerDrop allows models to be pruned to the desired depth at test time. Apart from finetuning for data adaptation on the GLUE tasks, we do not finetune the performance of our smaller models any of the other tasks consider in this work. As shown in Table S we found that finetuning the pruned models only results in marginal improvement. Further, the finetuning parameters were dependent the depth of the model at test time and difficult optimize.
15

---

LryerDrop
Dropout
Valid PPL
Model
Valid PPL.
0.5
0.1
19.03
Adaptive Input
18.4
0.5
0.2
19.22
Random LayerDrop 0.2
18.2
0.5
0.3
19.31
Linear LryerDrop to 0.3
18.6
0.5
0.4
19.62
Linear LayerDrop 0.5
18.5
0.5
0.5
19.95
Linear LayerDrop to 0.8
18.9
Table Performance Varying Dropout with Fixed LayerDrop on a 16 layer language model trained on Wikitext- 103 (Valid).
Table 10: Random Linear Decay Layer- Drop on 16 layer language model trained on Wikitext- 103 (Valid). * result is from Baevski & Auli 2018
Structured Dropout
Valid PPL
Half FFN
29.6
Baseline
28.3
Head
28.1
Sublayer
19.9
Head + Sublayer
19.8
Layer
19.7
Head + Layer
19.7
Table 11: Performance Varying Structured
Dropout and Pruning to 8 layer language model trained on Wikitext- (Valid). Prun- ing is done by removing every other layer to half the model size.
Effect of Varying Standard Dropout. LayerDrop adds strong regularization effect to neural network training. We examine the importance of tuning the standard dropout parameter when train- ing with LayerDrop. In Table g we show the performance when LayerDrop fixed and standard Dropout is varied. We see that when training with LayerDrop, the quantity of standard Dropout can be reduced.
LayerDrop Schedule: Random Linear. We investigate the random structured dropping of layers compared to the linear decay schedule proposed in Huang et al. (0066) in Table I0 We find that the linear decay schedule does provide performance improvement compared random dropping, which is more straightforward to implement.
Impact of Types of Structured Dropout when Pruning. F contrasts the performance of various forms of structured dropout, such dropping attention heads, sub-layers of Transformers such as attention or FFN, portions FFN matrics, and entire Transformer layers. It examines these results in the setting of evaluating the full depth model on language modeling and shows that in general, different types of structured dropout can improve performance.
In Table II examine the effect of varying training time structured dropout with performance when pruning. We show that the trend shown in Figure 4 is consistent with inference-tima prun- ing performance, particularly that Half FFN dropout performs slightly worse, but other forms of structured dropout are beneficial.
16


Reviews:
[{"review_id": "S1xLPNk3YH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work explored the effect of LayerDrop training in efficient pruning at inference time. The authors showed that it is possible to have comparable performance from sub-networks of smaller depth selected from one large network without additional finetuning. More encouraging is that the sub-networks are able to perform better than the same network trained from scratch or learned based on distillation.\n\nBesides the promising results, I think the authors could make the presentation more coherent. Since the title is about \"reducing transformer depth on demand\", the focus is on pruning the network to meet inference requirements. But the authors spent a lot of space showing improved results on many tasks, which are mainly from learning a larger network or with additional data compared to the baselines. Then some of the results shown in the appendix, especially the ones referenced in the main text, could be brought into the main part.\n\nOn the other hand, I do not think it is adequate to argue the proposed method is a \"novel approach to train over-parameterized networks\". As the authors acknowledged, the layer dropping technique has been proposed in (Huang et al., 2016). Even though the authors extended this to different components of the network, the main focus is on layer dropping which is exactly the one proposed in (Huang et al., 2016). Actually, two layer dropping schedules were introduced in (Huang et al., 2016). One is the uniform dropping which is adopted in this work, the other is the linear decay dropping which is shown to achieve better performance (Huang et al., 2016). Even though more involved, it is interesting to see how the linear decay dropping works in terms of pruning.\n\nIt is intriguing to see that simple dropping method as every other could perform comparably to exhaustive search as shown in Figure 4 (right). Is this an artifact of the used dropping masks in training or something intrinsic to the method? The Data Driven Pruning approach, in a way, has the same flavor as the recently proposed dynamic inference methods [1,2] reducing the inference on a per-input basis. That is, different inference complexity will be given different inputs based on the inferred difficulty. The proposed method, on the other hand, assigns the same inference complexity to all the inputs but tries to learn strong sub-networks. It is worth mentioning these works and compare the differences.\n\n[1] Z. Wu, T. Nagarajan, A. Kumar, S. Rennie, L.S. Davis, K. Grauman, and R. Feris. BlockDrop: Dynamic inference paths in residual networks. CVPR 2018.\n[2] X. Wang, F. Yu, Z.-Y. Dou, T. Darrell, and J.E. Gonzalez. SkipNet: Learning dynamic routing in convolutional networks. ECCV 2018.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "Hkg3glcHtr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents LayerDrop, a simple method for dropping groups of weights (typically layers) jointly. Despite its simplicity (which is actually a big plus), the method seems to improve performance quite consistently on a range of NLP tasks. Moreover, it allows the authors to train very deep networks, that are very hard to train otherwise (according to the authors). For me the most exciting thing about this approach is that this training regime allows to prune the trained network at test time *without finetuning*, effectively getting a smaller, more efficient network for free. This is a great benefit compared to existing approaches that require retraining a smaller network for each costume size. While the method isn't really applicable to any size, and largely depends on the dropout rate the full model was trained on, I imagine it could serve as a starting point for other researchers to develop more flexible extensions that would allow for any size of network to be pruned at test time. I think this is a very strong submission and strongly advocate accepting it to ICLR.\n\nQuestions and comments:\n\n1. The main thing missing for me is some more analysis on the runtime/energetic savings (e.g., in terms of FLOPs) of the proposed method. The authors argue (3.2.1) that approaches such as DropConnect are not necessarily more efficient, but do not analyze the efficiency of their pruned networks apart from the size of the pruned network. \n\n2. Similarly, details about the experiments are also somewhat lacking:\na. how many GPUs were used to train the models? the authors mention 8 v100 in A.3, but I am not sure if this was the setup for all experiments. \nb. Figure 7, which shows that LayerDrop also improves training speed, is very interesting and should be part of the main text in my opinion. Was this trend consistent for all experiments?\nc. Similarly, presenting the total running time of the models (and not just words per second) would be helpful for reproducibility. \nd. Finally, reporting dev and not only test results (e.g., in tables 1 and 2) would also facilitate future reproducibility efforts.\n\n3. Did the authors use a regular dropout? If I understand correctly, in A.1.3, the authors mention tuning the dropout rate between {0.2,0.3}. Was this done for all tasks? and was it done for the baseline models as well? Using dropout in the baseline model with a similar proportion as LayerDrop seems like an important baseline, and in particular it would be interesting to see whether the deep experiments (e.g., 40 layers on WT103) that are hard to train without LayerDrop could converge with regular dropout.\n\nMinor:\n- 3.2: \"We present *an* regularization approach ...\" (should be \"a\")\n- Table 2 is referred to before table 1, it might be clearer to switch them.\n- In figure 4, it wasn't clear to me why \"Layer\" on the lefthand side is much better than \"Every other\" on the righthand side. Aren't these the same model variant?\n- Missing year for paper \"Language models are unsupervised multitask learners\".\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BklKzkFpYS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a method, LayerDrop, for pruning layers in Transformer based models. The goal is to explore the stochastic depth of transformer models during training in order to do efficient layer pruning at inference time. The key idea is simple and easy to understand: randomly dropping transformer layers during training to make the model robust to subsequent pruning. The authors perform empirical studies on several sequence modeling task to conclude that the proposed approach allows efficient pruning of deeper models into shallow ones without fine-tuning on downstream tasks. There are also empirical experiments done to demonstrate that the proposed approach outperforms recent model pruning techniques such as DistillBERT under comparable configurations. \n\nStrengths:\n+ The technique seems to be simple to apply yet powerful and promising.\n+ Strong results from the pruned networks without fine-tuning on downstream tasks. \n+ Good ablation studies that help establish the connection to other pruning strategies and the internal of LayerDrop. \n\nWeaknesses:\n- Stochastic depth has demonstrated a lot of significance for training in prior work. Although the end goal here (for pruning) is slightly different, the novelty is a little incremental. \n\nOverall, the paper is a good contribution given the current great interest of transformer-based models. The motivation is quite clear, and the writing is easy to follow. It is also a sensible approach given the strong regularization effect of stochastic depth. \n\nQuestion: \nSimilar to Pham et al.'s work on applying stochastic depth to train very deep transformers for speech, do you expect LayerDrop to be helpful for training very deep transformer-based models for NLP tasks assuming memory is not a big constraint? \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents Layerdrop, which is a method for structured dropout which allows you to train one model, and then prune to a desired depth at test time. This is a simple method which is exciting because you can get a smaller, more efficient model at test time for free, as it does not need fine tuning. They show strong results on machine translation, language modelling and a couple of other NLP benchmarks. The reviews are consistently positive, with significant author and reviewer discussion. This is clearly an approach which merits attention, and should be included in ICLR.
Decision: Accept (Poster)