Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work introduces a method to accelerate the training speed of deep learning models. The key idea is to start training with shared weights and unroll the shared weights in the middle of the training. The authors report that this strategy accelerates the convergence speed. The paper introduces heuristics on when to stop weight sharing and how many layers to share weights. It further provides an analysis via the view of deep linear models on why weight sharing helps improve the convergence speed. In the evaluation, the paper evaluates their approach against the training of BERT, and shows that their method can obtain comparable and sometimes even better accuracy on downstream tasks while with 50% faster training speed.
 
Strengths:
+ The paper aims to address an important problem in large model training: slow training speed.
+ The paper proposes an easy-to-implement approach to accelerate the convergence speed of BERT-like models. 
 
Weakness:
- The technical contribution seems rather incremental. The main difference between this work and the prior work [1], which also train Transformers via shared weights,  seems to be switching from sharing weights to unsharring weights in the middle of the training.
- Important references are missing, making it not clear the advantage of this work as compared with existing approaches that accelerate the training of Transformer networks. 
The comparison with existing work is inadequate. Important ablation studies are needed. 
 
Comments:
 
Prior work [1] uses weight sharing to train a smaller Transformer model to obtain similar accuracy.  However, weight sharing does not improve training speed per batch, because the training still needs to perform the same amount of forward and backward computation for each batch. Training may actually be slowed down, since the model may need to train with more batches to reach the same accuracy. This paper aims to speed up the training process by switching from shared to unshared weights in the middle of the training, and it observes faster convergence -- achieving similar downstream accuracy with less number of pretraining samples. This is an interesting empirical observation and can potentially become useful in practice. However, there are some major concerns about the paper.
 
It is still unclear whether this faster convergence comes from switching from sharing to unsharring weights or is an effect of the model or hyperparameter changes. First, the stop condition (e.g., the switching threshold) cannot be known as a prior. Therefore, it is controlled with an additional hyperparameter. From the text, it is unclear how this hyperparameter has been chosen or will be chosen when training a new model. It would be better to test the sensitivity of the hyperparameter on another model such as GPT-2 to verify the effectiveness of the proposed method.
 
Second, the paper adopts Pre-LN in its evaluation (as briefly mentioned in Section 5.1). However, from the text description, it seems it employs the original BERT as the baseline ("We first show experiment results English Wikipedia and BookCorpus for pretraining as in the original BERT paper"). As Pre-LN has been studied in several prior work [2,3] and has been demonstrated to also have the effect of accelerating the convergence speed, it is unclear whether the observed speedup in this paper is an effect of PreLN or weight sharing/unsharing. An ablation study with BERT-PreLN is needed if not already included.
 
Third, the analysis on deep linear models appears to be over-simplified, where important characteristics of the DNNs such as non-linear activations and residual branches are not represented, making it difficult to connect it with the actual observations in practice.
 
Finally, importance reference [4] is missing. [4] starts with a shallow BERT and progressively stacks Transformer blocks to accelerate training speed, which is in some sense similar to the proposed technique, which starts with shallow BERT with shared weights and switches to full-length BERT in the middle of the training. The paper might need to highlight the difference between this work and [4]. 
 
Several places in the paper are vague or inconsistent: 

1. The paper claims that it uses the same BERT implementation and training procedure as the one used by Devlin et. al.. However, the accuracy reported in Table 2 seems to be consistently lower than what was reported in the original BERT paper. For example, QQP in the original paper reaches 72.1, whereas this paper reports 71.4, QNLI was 92.7 in the BERT paper and 91.7 in this paper. If we use the original BERT reported results, the proposed technique seems to incur low accuracy on most tested datasets. Some clarification on the accuracy results is needed. 

2. The paper claims that "it sounds natural for us to expect that ALBERT's performance will be improved if we stop its weight sharing at some point of training. The optimal models are supposed to not be far from weight sharing." However, this explanation actually creates some confusion. First, the paper does not provide an analysis of how the weight distribution between the weights trained with and without weight sharing, so it is unclear what "not be far" means. Second, what does it mean by "optimal model"?  Does it refer to models trained not through weight sharing? If so, prior work [5] identified that model weights and gradients at different layers can exhibit very different characteristics, which seems to contradict the argument that "The optimal models are supposed to not be far from weight sharing". 

3. I find it challenging to claim the proposed technique generic for models with repeatedly layers, whereas only BERT is evaluated in the experiments.

4. The paper says "This means that the weight sharing stage should not be too long", but it is unclear how long is considered as not too long. 
 
[1] Lan et. al. "ALBERT: A Lite BERT for Self-supervised Learning of Language Representations", https://arxiv.org/abs/1909.11942

[2] Xiong et. al. "On Layer Normalization in the Transformer Architecture", https://arxiv.org/abs/2002.04745

[3] Shoeybi et. al. "Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism", https://arxiv.org/abs/1909.08053

[4] Gong et. al. "Efficient Training of BERT by Progressively Stacking", http://proceedings.mlr.press/v97/gong19a/gong19a.pdf

[5] You et. al. "Large Batch Optimization for Deep Learning: Training BERT in 76 minutes", https://arxiv.org/abs/1904.00962
review_title: Official Blind Review #3

final_score: rating: 4: Ok but not good enough - rejection
review_content: ##########################################################################

Summary:

The paper proposed a new way for training models that stack the same basic block for multiple times -- share the weights first and then untie the weights. The author tried to provide theoretical insights on why it can be better. Also, ablation study shows that the proposed algorithm has marginal improvement over the baseline.

##########################################################################

Reasons for score: 


The improvement over the baseline method, which just trains BERT for more steps, is quite marginal. In addition, after the weights are untied, the training process becomes exactly the same as that in BERT. It's highly likely that both methods can have similar performance after being trained for a large number of steps. Also, I do not think the theoretical results on a deep linear network can help explain the  phenomena we see in BERT training. BERT uses a much more complicated building block that involves layer normalization, attention and non-linearity. Due to these reasons, I vote for rejection.

##########################################################################

Pros: 
 

1. The idea of first train the model with shared weights and then untie the weights is interesting.
 

##########################################################################

Cons: 
 

1. The improvement over the baseline method is not very substantial. Basically, it is possible that both methods perform similarly after being trained for a longer number of steps (e.g., 1M, 1.5M).

2. I'm not convinced by the theoretical analysis. Building block in BERT is much more complicated that the building block discussed in Section 3.

3. The criteria of changing from sharing to unsharing is largely heuristic.


##########################################################################

Questions during rebuttal period: 
 

Please address and clarify the cons above 
 

#########################################################################

Typos: 

(1) Page 3, "To be best of" should be "To the best of"

review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper provides a simple way to speed up training deep networks which contain repeated structures, such as the transformer module, by sharing the weights first and then unsharing. Three methods are shown to stop weight sharing: 1) stop weight sharing at a predefined point; 2) stop weight sharing at a point using an adaptive way, in which unsharing will be triggered when the majority of the gradient correlations between layers is below a threshold; 3) gradually untie weights between layers by splitting groups based on the gradient correlations. The authors did experiments on BERT large for SQuAD and the GLUE benchmark, and showed that their approach outperform the baseline at the same iteration of 500k, and is on par w/ the baseline training at 1M iterations.

Basically I think this approach could be a practical way to train deep networks w/ repeated layers. It's like between BERT large and ALBERT, the results are not that surprising. The main contribution of this work is that it provides the empirical results of this training method on several tasks.

My questions regarding to this paper are:

1. In Section 3, the authors provide analysis of weighting sharing can achieve better convergence rate and for good weight sharing each layer's update needs to well correlate with its gradients for deep linear network. How about any theoretical insights for networks w/ activation functions? And How could different activation functions matter (ReLU, Swish et al)? And will these activation functions affect the parameters of when to stop weight sharing?

2. For untie weights between adjacent layers, SWE-A and SWE-M, what's the motivation of setting a unified threshold for all layers? Basically I'm wondering what's the gradient correlation between adjacent layers of top layers and bottom layers, are the correlations on the same level? Will we get better accuracy if we do not limit to a unified threshold? Say, for the top 12 layers, we set the threshold to X and for the bottom 12 layers we set the threshold to Y. Need further information on this.

3. In Table 2, why is SWE-A missing?

4. For Table 3, I think the authors need to add the experiments when \tau is smaller than 50k. Basically the experiments will show that \tau should be carefully designed, an arbitrary number may not work, we still need to train some iterations w/ weight sharing for the good model performance.

5. This paper may need to cite some other papers discussing some insights of weight sharing, such as  DEEPER INSIGHTS INTO WEIGHT SHARING IN NEURAL ARCHITECTURE SEARCH (https://arxiv.org/pdf/2001.01431.pdf).
review_title: Maybe a practical way to train deep networks w/ repeated layers

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a simple yet effective method to train transformer-like networks for NLP tasks. Weight sharing idea is not new, it has been investigated in ALBERT as mentioned in the paper and CV areas (see refs [1],[2]).  Though novelty of this paper is not strong enough, their experimental results are adequate to show the effectiveness of proposed method.  The writing and description are quite clear, but you need to clarify the definition of 'iters' you used in  SWE exps, does it mean iters_of_weight_share + iters_of_wight_unshare?

To show the advantage of weight sharing, the authors take deep linear model as an example to show that with weight sharing, layers far away from output will not suffer from gradient vanishing or explosion, which is true for deep LINEAR models. But for deep learning based models, non-linear operations is required, which still leads to gradient vanishing or explosion even with weight sharing. For example, recurrent neural network is a typical kind of weight-sharing network and its weight is shared through time. Even with single hidden layer, RNN still suffers from gradient vanishing or explosion, and that is why LSTM is proposed in 1990s. So in my opinion, you'd better take a TRUE deep learning model as an example to analyze the effectiveness of weight sharing during training. For example, you can try to analyze deep feed forward neural networks with ReLU activation, which can be treated as a deep piece-wise linear model.

[1] J. Wang and X. Hu, “Gated recurrent convolution neural network for ocr,” in NIPS, 2017.
[2] M. Liang and X. Hu, “Recurrent convolutional neural network for object recognition,” in CVPR, 2015, pp. 3367–3375
review_title: Simple yet effective method, deeper analysis is needed

Meta-review: 
The paper proposed a new way for training models that stack the same basic block for multiple times -- share the weights first and then untie the weights. Ablation study shows that the proposed algorithm has marginal improvement over the baseline. The authors also provide some theoretical justifications to how the proposed idea works.
The proposed idea is straightforward and intuitive. Weight sharing has been used in previous works, and what’s new in this paper is to unshare the weights in the middle (with a heuristic rule). The hope is that by doing so, one can achieve a better tradeoff between speedup and accuracy. However the experimental supports are somehow weak and incomplete. For example, in order to show the real speedup, one should provide the full training curve (until convergence) under different settings, instead of just showing one data point (at 500K). It is very common that one can get some speedup at 500K, but the speedup totally disappears after another 500K steps. 
Furthermore, the theoretical analysis is conducted in a simplified setting, and it is not very clear whether it can be used to explain what really happened during BERT training.
The reviewers conducted some lengthy discussions after the author rebuttal was available. As a final consensus, we think that there are still concerns on the paper, which makes us hesitate to give an ACCEPT recommendation.

Decision: Reject