Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper investigates the problem of accelerating the large-scale transformer model's training. The idea is to initialize the large-scale model with the transformed parameters of the small-scale model. Compared to existing methods, the proposed method can expand the network in both width and depth and the Kronecker factorization is used to reduce the computational cost. The experimental results in both BERT and Vision Transformer demonstrate that LEGO can make the model converge faster.
strength_and_weaknesses: Pros:
* The first work to combine both width- and depth-growth operators together and thus expand the model in both width and depth.
* LEPO shows good capacity in both the linear layer and the attention layer.
* The empirical results show that LEPO can greatly improve training efficiency.

Questions:
* The reported FLOPs and Wall Time of LEPO are counted during training. Would it be more reasonable to also consider the calculation of the LEGO operator before the transformer training? 
* As claimed in the experiment part, the authors claim that they investigate the performance of LEGO when transferring the model to other downstream tasks. Do the authors compare LEGO with some adapter-based methods or visual prompt tuning methods?
* Some typos can be fixed. For example, the reported results of LEGO show it can reduce 40.5% wall time when transferring BERT-Small to BERT-Base, however in Figure 2(b) the number is 40.7%.
clarity,_quality,_novelty_and_reproducibility: Overall, the paper is well-written and easy to follow. The novelty of the method seems incremental, but the empirical results are good. Sufficient details (e.g., experimental setup) are described in the paper such that an expert should be able to reproduce the main results
summary_of_the_review: Please refer to the 'Clarity, Quality, Novelty And Reproducibility' part above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method named LEGO to efficiently train a transformer model by initializing it with a pretrained smaller transformer. The method expands the pretrained transformer model on width and depth by learning linear maps on the parameters. Experiments on various language and vision tasks show that LEGO can obviously reduce the training time compared to training from scratch.
strength_and_weaknesses: Strengths:
1. The idea of this paper is easy to follow. By learning the expansion operators with task loss, the pretrained smaller transformer can be naturally mapped to a larger network, which could obtain a significantly higher accuracy at the beginning of training compared to training from scratch.

2. Extensive experiments on various tasks validate the efficacy of the proposed method. For example, in Table 1, LEGO outperforms Scratch in most cases with 40.5% training time reduced.

Weaknesses:
1. The method requires a trained smaller transformer to initialize the target network, but I'm not sure whether the training cost is still low if we take the training time of the smaller network into account.
2. It seems that LEGO only supports expanding the width of a linear transformation or growing it into a sequence of linear transformations, while two transformations connected by a non-linear activation function can not be initialized by one transformation. This may be a limitation of this work.
3. The improvements compared to existing methods such as StackBERT are marginal. In Figure 2, StackBert has lower log perplexity curves when FLOPs > 4e18 or wall time > 40 hrs.

clarity,_quality,_novelty_and_reproducibility: This paper is well presented, but the novelty is sort of limited. 
summary_of_the_review: Overall, I think this paper is a borderline paper. My major concerns of this paper are the limited improvements compared to the existing methods and the generalizability of the method.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a method for initializing a larger model by learning a linearly map using the parameters of a small model to obtain the parameters of a larger one, thus reducing the training cost of the larger model by using the existing small model.
Kronecker factorization is used to reduce the computational parameters to an acceptable level, making it possible to learn a linearly map as mentioned above.
The proposed method is applicable not only to multilayer neural network structures but also to transformers, and experiments have been conducted on various transformer-based models with good results.
strength_and_weaknesses: Strength
1.This paper uses a subtle factorization which substantially reduces the number of computational parameters, making it feasible to learn such a linearly map. This idea is very good, simple but useful, and also has some explainability.
2.Clearly illustrates how to extend the parameters in depth and width from a theoretical point of view, and with some justification.
3.The paper is well written.
Weakness
1.The training process of LEGO method is not described in detail in the paper, although illustrated in the appendix, I suppose it should be described in detail in the main paper in Sec.3 or another place. And the optimization process is not mentioned either. As I understand it, the parameters of LEGO are trained for a specific number of steps together with the parameters of the larger model, and then the training of LEGO is stopped to train only the larger model. Is it right? 
2. Compared to training from sketch, I wonder how about the performance in downstream task of model which use LEGO method initialized. The performance of downstream task is what we all concern, and it’s important. I hope you can add some relevant experiments.
3.Only one Vision Transformer model has experimented, more vision models should be considered.

clarity,_quality,_novelty_and_reproducibility: Simple but it’s a good try. Overall this paper is of good quality, with detailed formulas and clear diagrams, and a certain degree of originality.
summary_of_the_review: The idea and method in this paper is novel and subtle, makes it easy to learn a linearly map to grow model. However, the specific training process of LEGO is not highlighted in the text, and the experimental part is not sufficient, such as the detailed evaluation of the performance of downstream tasks.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents a learning strategy to grow smaller pre-trained networks into larger networks with the hope that model growth can save compute compared to a network that is trained from scratch. While mostly in the literature, growth strategies are based on heuristics, e.g., copying the layers of the network for expanding depth, or block diagonal expansion for increasing width, learning-based model growth is not extensively studied. In this work, the authors initialize the parameters of the larger network as a linear transformation of the parameters of the smaller network. This linear transformation is learned through data using a small number of gradient steps (typically ~100) – a cost which is marginal compared to the training of the larger network. The authors propose to decompose the linear transformation into depth- and width-operators, and use structured sparse matrices for these operators. For enforcing structures and sparsity, layer-wise transformations are proposed using kronecker factorization. Through experiments on BERT, RoBERTA, GPT2 and ViT, the authors show that their proposed growth strategy saves up to ~40% of compute without sacrificing the upstream and downstream performance of Transformer models across the domains. 
strength_and_weaknesses: **Strengths**

1. The paper is written well and easy to follow. 
2. Given the prevalence of large (foundation) models, and the fact that compute is perhaps the most important constraint while training these models, the paper tackles a very good problem in a timely manner. 
3. The experiments aptly make the points of the main paper. 

**Weaknesses/ Questions/ Suggestions**

1. **Performance of the base model**: I would encourage the authors to report the performance of the base (small) models, from where the larger models are trained, in the plots. It would help us clarify a) how quickly the larger model recovers the performance of the base model, b) how much it improves upon the base model. 

2. I would encourage the authors to modify the introduction/ story and emphasize the fact that their proposed method allows one to train a family of Transformer models, where new and larger models can utilize the information in the previous and smaller models to save compute, which is not possible when training from scratch. In the absence of this messaging, one could fairly ask, when comparing model growth with training from scratch, the compute budget of training the smaller models should also be included in the plots. 

3. **Scalability to larger models**: The authors have reported performance on relatively smaller architectures (up to 774 million parameters) compared to modern Transformer models. I wonder if the proposed method would also result in equivalent gains on these larger models. I see that in the conclusion the authors already concede this point but, nevertheless, I would like to know the authors' take on that. 

clarity,_quality,_novelty_and_reproducibility: The paper is written clearly and the work is novel. Please refer to the strengths and weaknesses section above for details. 
summary_of_the_review: Please refer to the strengths and weaknesses section above for details.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper is to accelerate the training of large-scale Transformer models by initializing their parameters with the parameters of the small-scale models. Also the authors expand the pretrained transformer model's width and depth by learning linear maps of the parameters. Extensive experiments have been done to support the effectiveness of the proposed algorithm. 
justification_for_why_not_higher_score: The overall idea of this paper is interesting. But actually, the idea could be heuristic and natural to improve the training of large-scale models. Importantly, the proposed algorithm needs the parameters of small-scale models as the 'warming up', which somehow is a practical constraint. A spotlight presentation would be good already to give this paper enough exposure to the relevant research community or audience. 
justification_for_why_not_lower_score: Almost all reviewers think this paper is well-written and easy to follow. The authors focus on the interesting problem addressing the constraints during the training of large-scale Transformer models. The novelty of the method seems incremental, but the empirical results are good. The authors also successfully evaluated the model's to very large models during the rebuttal time window, which is very nice.
Decision: Accept: notable-top-25%