Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper proposes a method, LayerDrop, for pruning layers in Transformer based models. The goal is to explore the stochastic depth of transformer models during training in order to do efficient layer pruning at inference time. The key idea is simple and easy to understand: randomly dropping transformer layers during training to make the model robust to subsequent pruning. The authors perform empirical studies on several sequence modeling task to conclude that the proposed approach allows efficient pruning of deeper models into shallow ones without fine-tuning on downstream tasks. There are also empirical experiments done to demonstrate that the proposed approach outperforms recent model pruning techniques such as DistillBERT under comparable configurations. 

Strengths:
+ The technique seems to be simple to apply yet powerful and promising.
+ Strong results from the pruned networks without fine-tuning on downstream tasks. 
+ Good ablation studies that help establish the connection to other pruning strategies and the internal of LayerDrop. 

Weaknesses:
- Stochastic depth has demonstrated a lot of significance for training in prior work. Although the end goal here (for pruning) is slightly different, the novelty is a little incremental. 

Overall, the paper is a good contribution given the current great interest of transformer-based models. The motivation is quite clear, and the writing is easy to follow. It is also a sensible approach given the strong regularization effect of stochastic depth. 

Question: 
Similar to Pham et al.'s work on applying stochastic depth to train very deep transformers for speech, do you expect LayerDrop to be helpful for training very deep transformer-based models for NLP tasks assuming memory is not a big constraint? 

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This work explored the effect of LayerDrop training in efficient pruning at inference time. The authors showed that it is possible to have comparable performance from sub-networks of smaller depth selected from one large network without additional finetuning. More encouraging is that the sub-networks are able to perform better than the same network trained from scratch or learned based on distillation.

Besides the promising results, I think the authors could make the presentation more coherent. Since the title is about "reducing transformer depth on demand", the focus is on pruning the network to meet inference requirements. But the authors spent a lot of space showing improved results on many tasks, which are mainly from learning a larger network or with additional data compared to the baselines. Then some of the results shown in the appendix, especially the ones referenced in the main text, could be brought into the main part.

On the other hand, I do not think it is adequate to argue the proposed method is a "novel approach to train over-parameterized networks". As the authors acknowledged, the layer dropping technique has been proposed in (Huang et al., 2016). Even though the authors extended this to different components of the network, the main focus is on layer dropping which is exactly the one proposed in (Huang et al., 2016). Actually, two layer dropping schedules were introduced in (Huang et al., 2016). One is the uniform dropping which is adopted in this work, the other is the linear decay dropping which is shown to achieve better performance (Huang et al., 2016). Even though more involved, it is interesting to see how the linear decay dropping works in terms of pruning.

It is intriguing to see that simple dropping method as every other could perform comparably to exhaustive search as shown in Figure 4 (right). Is this an artifact of the used dropping masks in training or something intrinsic to the method? The Data Driven Pruning approach, in a way, has the same flavor as the recently proposed dynamic inference methods [1,2] reducing the inference on a per-input basis. That is, different inference complexity will be given different inputs based on the inferred difficulty. The proposed method, on the other hand, assigns the same inference complexity to all the inputs but tries to learn strong sub-networks. It is worth mentioning these works and compare the differences.

[1] Z. Wu, T. Nagarajan, A. Kumar, S. Rennie, L.S. Davis, K. Grauman, and R. Feris. BlockDrop: Dynamic inference paths in residual networks. CVPR 2018.
[2] X. Wang, F. Yu, Z.-Y. Dou, T. Darrell, and J.E. Gonzalez. SkipNet: Learning dynamic routing in convolutional networks. ECCV 2018.
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: This paper presents LayerDrop, a simple method for dropping groups of weights (typically layers) jointly. Despite its simplicity (which is actually a big plus), the method seems to improve performance quite consistently on a range of NLP tasks. Moreover, it allows the authors to train very deep networks, that are very hard to train otherwise (according to the authors). For me the most exciting thing about this approach is that this training regime allows to prune the trained network at test time *without finetuning*, effectively getting a smaller, more efficient network for free. This is a great benefit compared to existing approaches that require retraining a smaller network for each costume size. While the method isn't really applicable to any size, and largely depends on the dropout rate the full model was trained on, I imagine it could serve as a starting point for other researchers to develop more flexible extensions that would allow for any size of network to be pruned at test time. I think this is a very strong submission and strongly advocate accepting it to ICLR.

Questions and comments:

1. The main thing missing for me is some more analysis on the runtime/energetic savings (e.g., in terms of FLOPs) of the proposed method. The authors argue (3.2.1) that approaches such as DropConnect are not necessarily more efficient, but do not analyze the efficiency of their pruned networks apart from the size of the pruned network. 

2. Similarly, details about the experiments are also somewhat lacking:
a. how many GPUs were used to train the models? the authors mention 8 v100 in A.3, but I am not sure if this was the setup for all experiments. 
b. Figure 7, which shows that LayerDrop also improves training speed, is very interesting and should be part of the main text in my opinion. Was this trend consistent for all experiments?
c. Similarly, presenting the total running time of the models (and not just words per second) would be helpful for reproducibility. 
d. Finally, reporting dev and not only test results (e.g., in tables 1 and 2) would also facilitate future reproducibility efforts.

3. Did the authors use a regular dropout? If I understand correctly, in A.1.3, the authors mention tuning the dropout rate between {0.2,0.3}. Was this done for all tasks? and was it done for the baseline models as well? Using dropout in the baseline model with a similar proportion as LayerDrop seems like an important baseline, and in particular it would be interesting to see whether the deep experiments (e.g., 40 layers on WT103) that are hard to train without LayerDrop could converge with regular dropout.

Minor:
- 3.2: "We present *an* regularization approach ..." (should be "a")
- Table 2 is referred to before table 1, it might be clearer to switch them.
- In figure 4, it wasn't clear to me why "Layer" on the lefthand side is much better than "Every other" on the righthand side. Aren't these the same model variant?
- Missing year for paper "Language models are unsupervised multitask learners".

review_title: Official Blind Review #2

Meta-review: This paper presents Layerdrop, which is a method for structured dropout which allows you to train one model, and then prune to a desired depth at test time. This is a simple method which is exciting because you can get a smaller, more efficient model at test time for free, as it does not need fine tuning. They show strong results on machine translation, language modelling and a couple of other NLP benchmarks. The reviews are consistently positive, with significant author and reviewer discussion. This is clearly an approach which merits attention, and should be included in ICLR.
Decision: Accept (Poster)