Param$\Delta$ for Direct Mixing: Post-Train Large Language Model At Zero Cost

Authors: Sheng Cao, Mingrui Wu, Karthik Prasad, Yuandong Tian, Zechun Liu
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: large language model, post-training, transfer learning, model merging, weights averaging, artificial intelligence

Abstract:
The post-training phase of large language models is essential for enhancing capabilities such as instruction-following, reasoning, and alignment with human preferences. However, it demands extensive high-quality data and poses risks like overfitting, alongside significant computational costs due to repeated post-training and evaluation after each base model update. This paper introduces Param$\Delta$, a novel method that streamlines post-training by transferring knowledge from an existing post-trained model to a newly updated base model with \textbf{zero} additional training. By computing the difference between post-trained model weights ($\Theta_\text{post}$) and base model weights ($\Theta_\text{base}$), and adding this to the updated base model ($\Theta_\text{base}'$), we define Param$\Delta$ Model as: $\Theta_{\text{Param}\Delta} = \Theta_\text{post} - \Theta_\text{base} + \Theta_\text{base}'$. This approach surprisingly equips the new base model with post-trained capabilities, achieving performance comparable to direct post-training. We did analysis on LLama3, Llama3.1, Qwen, and DeepSeek-distilled models. Results indicate Param$\Delta$ Model effectively replicates traditional post-training. For example, the Param$\Delta$ Model obtained from 70B Llama3-inst, Llama3-base, Llama3.1-base models attains approximately 95\% of Llama3.1-inst model's performance on average. Param$\Delta$ brings a new perspective on how to fully leverage models in the open-weight community, where checkpoints for base and instruct models are readily available and frequently updated, by providing a cost-free framework to accelerate the iterative cycle of model development.

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1729503156061

thread_type:
official_review

<thread_object>
creator: reviewer_dt6b

<content>
summary:
This paper proposes to use the change of model parameters for representing the knowledge learned by LLMs. The core idea is to perform a weight averaging operation for pre-trained and post-trained model parameters. It is discovered that such weight averaging leads to comparable results.

strengths:
- The idea itself is interesting;

- The method is straightforward, easy to follow.

- The results provide some insights on how to transfer pre-trained LLMs to a new task/domain: if we have a pretrained LLM $f$, and two other checkpoints, one ($g_1$) is pretrained, the other ($g_2$) is post-trained on the new domain, then we can adapt f to this new domain by $f + (g_2 - g_1)$.

weaknesses:
- The method itself is simple, but the presentation needs substantial improvements. Now the presentation makes the paper seem complicated. The core ideas are clear and easy to follow, but the writing is confusing with so many long subscriptions in equations. For example, $\theta_{model_i-pretrain}, \theta_{post-train-llama3.1-8b}$ are redundant expressions, making readers more confusing.
  
- Moreover, the figures are so small. The x and y labels are hard to see. It is highly recommended that the authors improve the representation of equations, and provide a straightforward illustration of their method by figures. This is also an effect of too long subscriptions.

- The empirical improvements are marginal (Figs 1, 2, Tabs 1, 2). The current results fail to provide useful insights or surprising ovservations. It is recommended that the authors show some scenarios where existing post-trained models cannot achieve very good results, yet the proposed method easily outperform them with simple parameter fusing.

questions:
- To my understanding, the technical novelty is limited. Parameter fusing is performed via valinna operations. Simplicity is a strength, but the technical novelty is lacking, given that the obtained results are not very promising (the improvements are marginal). Although I'm not an expert on LLMs, it could be easily found that the method requires naive addition/subtractions on the whole model parameters. Therefore, I cannot accurately assess the value of the proposed parameter fusing approach. It is recommended that the authors elaborate on how their approach differs from or improves upon existing parameter fusion techniques in the context of LLMs.

- What if $f$ and $g_1$ are pre-trained on different domains (the notations is from the strength part)? Does the method assume that both of them have already be pre-trained on a variety of data, and share some common knowledge?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730638658081

thread_type:
official_review

<thread_object>
creator: reviewer_uxgg

<content>
summary:
The paper introduces a novel approach, parameter fusing, which simplifies knowledge and capability transfer in large language models (LLMs) by integrating parameter deltas—the differences between instruct-tuned and base model checkpoints—into a new base model. This technique allows LLMs to incorporate specialized skills or domain-specific knowledge without the need for resource-intensive post-training phases. Parameter Fusing is grounded in the observation that performance improvements correlate with a concave relationship to changes in parameters, suggesting diminishing returns as models approach an optimal performance plateau. This relationship was validated through comprehensive experiments, showing that parameter fusion not only matches but can sometimes enhance the effects of traditional post-training. By leveraging open models, such as Meta’s Llama, this method enables efficient and flexible customization of LLMs, significantly reducing costs and time associated with conventional fine-tuning while ensuring adaptability for diverse applications.

strengths:
This work builds on prior research in parameter aggregation but offers fresh insights and significant contributions. Notably, it presents an intriguing hypothesis that links performance gains to parameter changes—a relationship convincingly supported by experimental results. Beyond its theoretical contributions, the paper demonstrates a practical application for its proposed Parameter Fusing approach: when LLMs require continual pretraining to acquire specialized skills or domain-specific knowledge, Parameter Fusing offers a resource-efficient alternative to traditional post-training. The experimental outcomes are promising, validating the method's effectiveness. Overall, this paper introduces a novel perspective on post-pretraining, with potential for wide-reaching applications in future research. It is poised to make a meaningful impact on the LLM research community.

weaknesses:
My major concern is that there lacks a quantitative evaluation to evaluate if the new knowledge in a continual pretrained model will be preserved in the fused model. In the current experiments, this validation is achieved by showing merely one example in Table 4. More concrete results should be provided in the main experiment section.

questions:
When fusing parameters from different checkpoints, is there any criteria that can be used to select the most effective parameter deltas?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730857443839

thread_type:
official_review

<thread_object>
creator: reviewer_faxs

<content>
summary:
This paper introduces a novel post-training approach termed "Parameters Fusing" designed to simplify the transfer of knowledge and capabilities in large language models (LLMs) during the post-training phase. Traditional post-training requires extensive high-quality data and significant resource consumption. This research innovatively achieves the effects of the post-training phase by merging parameter deltas from existing instruct-tuned models with a newly pre-trained base model, thereby enhancing instruction-following capabilities and domain-specific knowledge without conventional post-training.

strengths:
1.	Innovation: The "Parameters Fusing" approach leverages parameter deltas to achieve post-training effects, representing an innovative advancement over traditional methods which requires high-quality training data.
2.	Cost effectiveness: This method significantly reduces post-training costs, making model customization more economical and efficient.
3.	Flexibility: Parameter delta operations allow freedom within homologous models, enabling fine-tuning across characteristics like coding ability and tool usage.
4.	Experiments: Experimental results show that fused models perform excellently across benchmarks, approaching or even exceeding traditional post-trained models, validating the method's effectiveness.

weaknesses:
1.	Potential Performance Limitations: In some benchmarks, fused models slightly underperform compared to traditional post-trained models, indicating potential limitations in transfer efficiency.
2.	Experimental Transparency: Certain experimental details, particularly criteria for choosing different parameter delta combinations and the implementation process, are insufficiently detailed, potentially affecting reproducibility.
3.	Lack of Adaptive Delta Selection: The method relies on manual tuning of delta combinations, which increases costs and limits flexibility. An adaptive mechanism for delta selection would enhance efficiency and usability.

questions:
Please refer to the weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1731116614430

thread_type:
official_review

<thread_object>
creator: reviewer_omkj

<content>
summary:
The paper introduces an innovative approach for post-training large language models (LLMs) through "Parameters Fusing," a method that fuses model parameters from instruct-tuned checkpoints into a newly pre-trained model. The goal is to replicate post-training effects without the extensive time and resource costs typically required. By leveraging parameter deltas, the authors enable the efficient transfer of domain-specific knowledge and model capabilities, showcasing the model's ability to maintain or enhance performance across multiple benchmarks. Experiments validate that fusing models can rival or even exceed the effectiveness of traditional post-trained models.

strengths:
- The paper clearly explains the challenges of post-training and the need for efficient knowledge transfer, establishing a strong foundation for the introduction of Parameters Fusing.
- The "Parameters Fusing" approach is a creative and resource-efficient alternative to conventional post-training, presenting a valuable technique for the efficient transfer of knowledge in LLMs.
- The paper includes rigorous experiments across multiple benchmarks, which provide clear empirical support for the proposed method's performance and efficiency.
- By using open-weight models like Llama, the authors demonstrate an adaptable approach that can be widely applied across different models and domains.
- The paper offers a well-structured theoretical grounding, discussing the relationships among model parameters, training steps, and knowledge acquisition.

weaknesses:
- The study could benefit from comparisons with other parameter-efficient methods in addition to traditional post-training, such as adapter-based or LoRA methods, to contextualize its performance and efficiency.
-  It is unclear if Parameters Fusing will perform as effectively on larger models. Expanding the analysis to address scalability and potential limitations in diverse applications would strengthen the paper.
- While the paper focuses on Llama models, it does not fully address whether the approach is model-agnostic or if any adjustments would be necessary for different architectures.
- The approach may introduce a risk of overfitting in highly specialized domains. Including an analysis of model generalizability when exposed to new or unseen tasks would improve the robustness of the findings.
- Although Parameters Fusing is efficient, there is limited discussion about interpretability and potential risks (e.g., model degradation) when applying delta parameters from various sources.

questions:
- There is minimal discussion on the risks of model degradation when fusing parameters from multiple sources, especially when domain mismatches or conflicting knowledge bases are involved. Investigating and reporting any observed performance declines, conflicts in fused knowledge, or mitigation strategies would strengthen the paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732134743276

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal

comment:
# Rebuttal

We extend our sincere gratitude to the reviewers for their insightful feedback. We are pleased to note that the reviewers recognize and appreciate our paper's focus on addressing the critical and impactful challenges of post-training and knowledge transfer in LLMs in a resource-efficient manner [**omkj, fAXs, UXgg, dT6B**]. The proposed method is acknowledged for its innovation and impact [**omkj, fAXs, UXgg**], adaptability [**omkj, fAXs**], and clarity [**dT6B**], supported by a comprehensive theoretical foundation [**omkj, UXgg**] and rigorous empirical results [**omkj, fAXs, UXgg**]. Overall, the work demonstrates applicability across various models [**omkj**] and potential for wide-reaching applications [**UXgg**], making a significant impact on the LLM research community [**UXgg**].

&ensp;

## Highlights and Common Questions
We summarize several highlights and common feedback as below.

&ensp;

### Our novelty mainly comes from the below three parts
1. Our work provides a creative and resource-efficient framework for bypassing the post-training process [**omkj, fAXs, UXgg**]. Specifically, while nowadays people conduct expensive instruction fine-tuning during the post-training phase, our method suggests that the entire post-training process can be replaced by parameter delta fusion. This innovation has the potential to **revolutionize current perspectives on LLM post-training** and pave the way for new directions in open-weight model research.
2. Our work offers fresh insights and significant contributions [**UXgg**] on top of prior research. Earlier work on parameter fusion has been limited to simple domains with low-dimensional knowledge or straightforward tasks such as classification and text generation. Our focus is on tackling the high-dimensional knowledge base of LLMs, which has not been thoroughly explored before.
3. We provide a theoretical foundation for our method, linking the parameter delta to the amount of training received and knowledge acquired during the training process [**omkj, UXgg**]. Additionally, we offer a quantitative evaluation of the efficiency ($\beta$) and effectiveness ($R^2$) of our method in Section 3.4, illustrating that our fused model can achieve knowledge transfer efficiency ranging from 97.9% to 99.9% as of traditional post-training, and more than 99.1% of effectiveness.

&ensp;

### Comments on experiments details
1. Add more specific experimental implementation details [**fAXs**] and absence of a quantitative summary on the domain performance while only giving model response examples [**UXgg**]: We have included the experiment details in section 3.2.1, 3.3.1 in **teal** color, and a quantitative summary (with its evaluation details) in Section 3.3.1 in **orange** color.
2. Ask of parameter delta selection criteria [**fAXs, UXgg**]: The primary objective of the paper is to demonstrate that the entire post-training process can be bypassed by directly integrating parameter deltas from pre-existing checkpoints. Therefore, we leave an automatic selection of the parameter deltas for future work.

&ensp;

### Comments on insufficient presentation 
1. Unclear notation, overly lengthy subscripts, figures being too small [**dT6B**]:
We apologize for insufficient presentations in the paper and have done the following in **blue** color to improve readability and presentation:
   - Included a notation table in Section 2.3
   - Reduced all the lengthy subscripts and updated the notation across the paper (either in equations and in tables, figures)
   - Enlarge all the figures and their labels
   - Added a main figure in the introduction section (Section 1) to illustrate our method in a straightforward manner
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces an approach for post-training large language models (LLMs) through Parameters Fusing, a method that fuses model parameters from instruct-tuned checkpoints into a newly pre-trained model. The paper received four positive ratings, including one 8. The reviewers praised the clear motivation, valuable technique, good experiments, well-structured presentation, etc. There is a agreement here among reviewers that this paper is ready for publication. Some concerns are raised by reviewers, like method novelty and insufficient presentation, the author well addressed these concerns in rebuttal. Given the acceptance recommendations from all reviewers, and the technical contribution of the paper, the AC recommend this paper for acceptance.