Structured-Initialization Learning

Authors: Deyuan Liu, Peng Sun, Xufeng Li, Tao Lin
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: efficient learning

Abstract:
The emergence of large language models (LLMs) has revolutionized natural language processing, but their development and deployment face significant challenges in computational resources and environmental sustainability.
Traditional self-supervised learning (SSL) paradigms requiring extensive computational infrastructure and exhibiting slow convergence rates, leading to increased energy consumption and longer training durations.
While existing model fine-tuning techniques such as Low-Rank Adaptation (LoRA) are resource-intensive and fail to facilitate swift knowledge updates when integrating a mount of new data in model version iteration.
To mitigate these challenges, we introduce Sail, a novel method for accelerating the training of neural network models by leveraging knowledge from (publicly available) pre-trained models.
Our approach comprises two key components: (1) a parameter transformation technique that adjusts the dimensions of pre-trained model parameters to match the target architecture, and (2) a proximal parameter integration and retraining strategy that efficiently combines transformed parameters to initialize new models.
We formalize the concept of Proximal Parameter and provide theoretical guarantees for its convergence advantages.
Our approach achieves substantial reductions in training time and computational resources while maintaining or improving model performance on downstream tasks.
These results indicate that Sail provides a promising direction for the more efficient and accessible development of the deep learning community.
Our code will be made publicly available.

Decision: reject
Reviewer scores: 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1730341746892

thread_type:
official_review

<thread_object>
creator: reviewer_wx7e

<content>
summary:
This paper proposed a novel way to combine the weights of multiple pre-trained LLMs into a smaller model. The problem to be addressed is important, as it offers opportunity to train a smaller specialized model faster, rather than train from scratch. 

The paper proposed a set of theorems support it's claims, which offers theoretical guarantees.

strengths:
- This paper supports its claims with both theoretical and empirical results. The proposed theorems are proved in the appendix, which looks correct (without in-depth review)

- This paper address an important problem of LLM reusing, which lacks efficient solutions before. The proposed method seems to be better than naive averaging weights.

- This paper marks different theorems and labels in color, which helps readers to quickly locate and navigate.

weaknesses:
- The field of knowledge distillation should be discussed in the related works section, which is highly related.

- The use of colored block seems to be too extensive, which is not common in research papers. Though not an outstanding weakness point.

questions:
- Why is there a spike at $\gamma=0.5$? 

- Are you assuming all other network elements are the same, e.g. activation functions, vocabulary size?

- Better to add discussion about knowledge distillation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730492231602

thread_type:
official_review

<thread_object>
creator: reviewer_1vma

<content>
summary:
This paper proposes Structured-Initialization Learning (SAIL), a method to accelerate training for large models by reusing parameters from pre-trained models. The approach includes transforming parameters to fit the target model and integrating them to form a more optimal starting point for training, reducing the need for random initialization.

strengths:
1. The paper provides a solid theoretical analysis, effectively demonstrating how the proposed Proximal Parameter initialization leads to faster convergence. The authors present well-structured convergence theorems, lending strong support to the efficacy of SAIL in reducing training time and improving efficiency.
2. The method is tested on both NLP and computer vision tasks, demonstrating applicability across different domains and model architectures.

weaknesses:
1. **Limited Novelty in Leveraging Pre-trained Models for Initialization** The proposed method of reusing parameters from pre-trained models to accelerate the training of new models is similar to existing work [1-2].
2. The motivation and system design of Figure 1 claims to use a pre-trained model such as LLM, however, the actual experiments are conducted by training the model from scratch on small-scale datasets in a controlled setup. It would be good to actually use the pre-trained models in hugging faces to create the SAIL. 
3. The explanation of how the parameter transformation is conducted is not clear enough, the authors mentioned the random projection and learnable methods, but a detailed investigation and experiments on which method is better are not included.


**Reference**

[1] Initializing Models with Larger Ones ICLR 2024
[2] Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization

questions:
1. Could the author provide the training curves of the NLP experiments as shown in the CV experiments, it helps to verify if the proved fast convergence is also applicable to transformer training.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730708483072

thread_type:
official_review

<thread_object>
creator: reviewer_yrjd

<content>
summary:
This paper presents SAIL, a method to accelerate training by leveraging knowledge from pre-trained models using a structured initialization approach. It introduces a parameter transformation technique to adapt pre-trained model dimensions to the target architecture, alongside a proximal parameter integration strategy. Theoretical guarantees show the benefits of using transformed pre-trained parameters for faster convergence compared to random initialization as well as the guidance on obtaining the optimal parameters in the integration strategy. Experimental results across NLP and computer vision tasks confirm that SAIL reduces training time while improving model performance, supporting the efficacy and broad applicability of structured initialization for efficient large model training.

strengths:
1. This paper offers strong motivation for addressing the challenges of efficient model initialization by harnessing pre-trained models, making a compelling case for its approach.
2. Rigorous theoretical analysis and effective visualizations are used throughout, solidly supporting the paper’s claims and enhancing interpretability.
3. Extensive experiments demonstrate that SAIL significantly outperforms random initialization, highlighting its effectiveness in reducing training time and improving model performance.

weaknesses:
1. A major concern is the limited comparison with related methods. This paper’s approach aligns closely with areas like model reuse/expansion and model merging, both mentioned in the related work. Model expansion combined with proximal parameter integration or parameter transformation with model merging could potentially address the problem posed here. A comparison with methods from these areas would provide a more comprehensive evaluation of SAIL’s efficiency.
2. The paper lacks discussion on the gap between its linear model theory and real-world application, which is crucial to understanding SAIL's limitations. For instance, it would be beneficial to clarify practical computation of $\gamma^*$ and address situations where the required data isn’t available—a common issue with open-source models that often lack access to original training data.

questions:
1. How do the authors justify the claim in lines 268-269 based on Theorem 2?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732288962679

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
# General Response

We have conducted extensive experiments to evaluate the performance of our proposed method, **SAIL**, compared to various baseline methods. Due to space constraints, we provide a summary here and include detailed settings and results in **Appendix I in page of  34**.

**Comparison Methods:**

- **Training from Scratch:** Models initialized randomly and trained on the target tasks.
- **Existing Methods:** We compared SAIL with methods such as **LIGO** (a weight transformation method) and **Model Soup** (a linear model merging method).

**Benchmarks:**

We evaluated our approach on a diverse set of NLP benchmarks to assess its performance across various linguistic and reasoning tasks:

- **PIQA:** Assesses physical commonsense reasoning.
- **HellaSwag:** Challenges models with complex multiple-choice questions requiring robust inference.
- **Winogrande:** Focuses on pronoun resolution for contextual understanding.
- **SciQ:** Tests comprehension of scientific texts.
- **ARC-Easy:** Presents grade-school level science questions for factual knowledge application.
- **COPA:** Measures causal reasoning by selecting plausible alternatives.

**Overview of Results:**

- **Table 1:** Compares the accuracy of models trained from scratch versus models initialized with SAIL across various NLP benchmarks.
- **Table 2:** Compares SAIL with LIGO, highlighting the effectiveness of our parameter transformation method.
- **Table 3:** Presents a comparison between Uniform Soup, Greedy Soup, and SAIL, demonstrating that SAIL's γ* yields better performance.

*Note: The best performance for each dataset is highlighted in bold.*

---

Table 1: Training from Scratch vs. SAIL (Accuracy) 

| Dataset | Training from Scratch (%) | SAIL (Ours) (%) |
| --- | --- | --- |
| PIQA | 51.96 | **61.92** |
| HellaSwag | 24.87 | **34.48** |
| Winogrande | 51.14 | **52.96** |
| SciQ | 22.10 | **70.30** |
| ARC-Easy | 27.54 | **42.63** |
| COPA | 58.00 | **63.00** |

Table 2:  LIGO vs. SAIL (Accuracy)

| Dataset | LIGO (%) | SAIL (Ours) (%) |
| --- | --- | --- |
| PIQA | 52.29 | **61.92** |
| HellaSwag | 25.33 | **34.48** |
| Winogrande | 50.20 | **52.96** |
| SciQ | 23.90 | **70.30** |
| ARC-Easy | 29.65 | **42.63** |
| COPA | 55.00 | **63.00** |

Table 3: Uniform Soup vs. Greedy Soup vs. SAIL (Accuracy)

| Dataset | Uniform Soup (%) | Greedy Soup (%) | SAIL (Ours) (%) |
| --- | --- | --- | --- |
| PIQA | 54.80 | 57.73 | **61.92** |
| HellaSwag | 25.25 | 27.65 | **34.48** |
| Winogrande | 50.51 | 52.09 | **52.96** |
| SciQ | 51.90 | 59.70 | **70.30** |
| ARC-Easy | 27.19 | 34.91 | **42.63** |
| COPA | 57.00 | 51.00 | **63.00** |

*Note: The best performance for each dataset is highlighted in bold. For perplexity, lower is better (indicated by ↓).*
</content>
</thread_object>

</thread>
<thread>
creation date:
1732525500061

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers,

Based on your valuable feedback, we have expanded our experiments to enhance the training efficiency of diffusion models.

Specifically, building on the theoretical foundations of our Structured-Initialization Learning (SAIL) approach, we have discovered that a slightly modified version of SAIL can effectively **leverage representation models (e.g., DINOv2 [1]) to significantly accelerate the training of the SiT diffusion model [2]**, a state-of-the-art generative model.

For benchmarking, we conducted experiments comparing our method to the recently proposed state-of-the-art diffusion model acceleration method, REPA [3]. This method employs a pre-trained representation model DINOv2 to accelerate diffusion model training by $17.5\times$ through representation alignment, garnering significant attention in the generative model community. Detailed experimental setups are provided in Appendix J on page 39.

**Table**: Comparison of the performance of our SAIL with the baseline SiT-B/2 model and SiT-B/2 augmented with REPA across various training iterations for ImageNet $256 \times 256$ generation.

| Model | #Params | Iter. | FID↓ | IS↑ | Prec.↑ |
| --- | --- | --- | --- | --- | --- |
| SiT-B/2 | 130M | 400K | 33.0 | 43.7 | 0.53 |
| REPA  | 130M | 50K | 78.2 | 17.1 | 0.33 |
| SAIL(ours) | 130M | 50K | **67.6** | **20.5** | **0.34** |
| REPA  | 130M | 100K | 49.5 | 27.5 | 0.46 |
| SAIL(ours) | 130M | 100K | **35.9** | **45.1** | **0.53** |
| REPA | 130M | 200K | 33.2 | 43.7 | 0.54 |
| SAIL(ours) | 130M | 200K | **19.78** | **81.9** | **0.64** |
| REPA | 130M | 400K | 24.4 | 59.9 | 0.59 |
| SAIL(ours) | 130M | 400K | **12.16** | **119.4** | **0.70** |

The experimental results in the table demonstrate that our SAIL significantly outperforms both REPA and SiT.

[1] Oquab, Maxime, et al. "Dinov2: Learning robust visual features without supervision." *arXiv preprint arXiv:2304.07193* (2023).

[2] Ma, Nanye, et al. "Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers." *arXiv preprint arXiv:2401.08740* (2024).

[3] Yu, Sihyun, et al. "Representation alignment for generation: Training diffusion transformers is easier than you think." *arXiv preprint arXiv:2410.06940* (2024).
</content>
</thread_object>

</thread>


Metareview:
metareview:
This submission discusses the resource challenges of developing and deploying LLMs. To address this, they introduce Sail, a method that accelerates training by leveraging publicly available pre-trained models. Sail combines a parameter transformation technique for aligning pre-trained model parameters with target architectures and a proximal parameter integration strategy for efficient model initialization, significantly reducing training time and resource usage while maintaining or improving performance on downstream tasks. However, most experiments are conducted on small-scale benchmarks. It is very hard to argue that the nanoGPT is a LLM. We see some mismatch between the claim and the experimental validation.

Most reviewers acknowledge the motivation, the importance of the targeted problem, and the theoretical analysis. However, the reviewer like the most positive reviewer seems mainly attracted by the LLM story ("This paper addresses an important problem of LLM reusing, which lacks efficient solutions before. The proposed method seems to be better than naive averaging weights.") and does not have in-depth analyses. One of the reviewers argued about the limited innovation. In our view, the major limitation of this submission lies in the insufficient large-scale experimental validation and the mismatch between claimed and validated. The models that "have revolutionized natural language processing" are ones with at least billion-level parameters.

Meanwhile, more comparisons with merging literature/methods (e.g., https://github.com/arcee-ai/mergekit) could greatly strengthen this submission, since merging serves as a key component in the proposed framework. 

Lastly, in the current shape, both the abstract and introduction emphasize LLMs. However, more than 50% of experiments and findings are on tool vision datasets like CIFAR. Reorganizing and reshaping the overall story and turning down the tune of LLMs could improve this submission.