DARE the Extreme: Revisiting Delta-Parameter Pruning For Fine-Tuned Models

Authors: Wenlong Deng, Yize Zhao, Vala Vakilian, Minghui Chen, Xiaoxiao Li, Christos Thrampoulidis
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: Delta parameter pruning, Efficiency, Large Language Models

Abstract:
Storing open-source fine-tuned models separately introduces redundancy and increases response times in applications utilizing multiple models. Delta-parameter pruning (DPP), particularly the random drop and rescale (DARE) method proposed by Yu et al., addresses this by pruning the majority of delta parameters—the differences between fine-tuned and pre-trained model weights—while typically maintaining minimal performance loss. However, DARE fails when either the pruning rate or the magnitude of the delta parameters is large. We highlight two key reasons for this failure: (1) an excessively large rescaling factor as pruning rates increase, and (2) high mean and variance in the delta parameters. To push DARE’s limits, we introduce DAREx (DARE the eXtreme), which features two algorithmic improvements: (1) DAREx-q, a rescaling factor modification that significantly boosts performance at high pruning rates (e.g., > 30% on COLA and SST2 for encoder models, with even greater gains in decoder models), and (2) DAREx-L2, which combines DARE with AdamR, an in-training method that applies appropriate delta regularization before DPP. We also demonstrate that DAREx-q can be seamlessly combined with vanilla parameter-efficient fine-tuning techniques like LoRA and can facilitate structural DPP. Additionally, we revisit the application of importance-based pruning techniques within DPP, demonstrating that they outperform random-based methods when delta parameters are large. Through this comprehensive study, we develop a pipeline for selecting the most appropriate DPP method under various practical scenarios.

Decision: accept (spotlight)
Reviewer scores: 6, 8, 8, 8

Reviews and discussion:
<thread>
creation date:
1730675605011

thread_type:
official_review

<thread_object>
creator: reviewer_3h3b

<content>
summary:
The authors revisit delta-parameter pruning (DPP) for fine-tuned models, focusing on the limitations of the existing Random Drop and Rescale (DARE) method when dealing with high pruning rates or large delta parameters. They identify two key issues causing DARE's performance degradation: the excessively large rescaling factor at high pruning rates and the high mean and variance in the delta parameters. To address these challenges, they propose two algorithmic improvements:

1. **DARq**: A modification of DARE that adjusts the rescaling factor, leading to significant performance gains at high pruning rates.
2. **AdamR**: An in-training regularization technique that incorporates delta (L2) regularization during fine-tuning to control the mean and variance of delta parameters before applying DPP.

They demonstrate that their methods outperform the original DARE, especially at extreme pruning rates, and can be combined with parameter-efficient fine-tuning techniques like LoRA. Additionally, they show that importance-based pruning methods can outperform random-based methods when delta parameters are large.

The findings of this work seem very high utility and are of substantial increase in performance in edge cases where the original methods failed very strongly. A very good step forward for DPP methods.

strengths:
- **Comprehensive Analysis**: The paper provides a thorough analysis of the limitations of the existing DARE method, identifying the root causes of its failure at high pruning rates.
- **Innovative Solutions**: The proposed methods, DARq and AdamR, are well-motivated and offer practical improvements over existing techniques.
- **Empirical Validation**: Extensive experiments on both encoder and decoder models across various datasets showcase the effectiveness of the proposed methods, with significant performance gains.
- **Flexibility and Compatibility**: The authors demonstrate that DARq can be combined with existing fine-tuning techniques like LoRA and can be used for structural pruning, increasing its practical utility.
- **Revisiting Importance-Based Pruning**: By exploring the application of importance-based pruning methods within DPP, they offer valuable insights into scenarios where these methods may be more effective.
- **Clear Recommendations**: The paper provides a practical pipeline for selecting the appropriate DPP method under different scenarios, which is valuable for practitioners.

weaknesses:
- **Complexity of Methods**: While the proposed methods are effective, they introduce additional complexity, such as tuning the rescaling factor \( q \) in DARq, which may require validation data or additional computations.
- **Limited Theoretical Depth**: The theoretical analysis, while helpful, could be further deepened to provide more rigorous guarantees or insights into why the proposed methods work.
- **Applicability Scope**: The methods are primarily validated on language models and NLP tasks; it would strengthen the paper to demonstrate their applicability to other domains or types of neural networks.
- **Clarity in Presentation**: Some sections, particularly the methodology and theoretical analysis, could benefit from clearer explanations and more intuitive descriptions to make them accessible to a broader audience.
- **Dependence on Regularization**: The AdamR method relies on in-training modifications and specific regularization strategies, which may not always be feasible or may interact unpredictably with other training dynamics.

questions:
1. **Scalability and Applicability to Other Domains**: While your methods have shown impressive results on language models, have you considered how DARq and AdamR might scale to even larger models or different architectures beyond NLP? Specifically, do you anticipate any challenges or necessary modifications when applying these techniques to other domains such as computer vision or speech recognition?

2. **Automated Selection of Rescaling Factor in DARq**: The selection of the rescaling factor \( q \) in DARq appears to be crucial for maintaining performance after pruning. Have you explored automated methods for selecting \( q \) without relying on a validation set or manual tuning? For instance, could analyzing the statistics of the delta parameters during pruning provide a way to determine an optimal \( q \) more systematically?

3. **Impact of AdamR on Training Dynamics**: Introducing delta regularization through AdamR alters the fine-tuning process. How does this affect the convergence, stability, and overall training dynamics compared to standard fine-tuning methods? Are there any trade-offs, such as increased training time or complexity, that practitioners should be aware of when integrating AdamR into their training pipelines?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730685394538

thread_type:
official_review

<thread_object>
creator: reviewer_xpsx

<content>
summary:
The paper addresses limitations in the random-rescaling Delta-parameter pruning method (DARE), which struggles with high pruning rates or large mean/variance in delta parameters. To tackle this, the authors propose two new approaches: (1) DARq, a post-training rescaling modification strategy, and (2) AdamR, a DDP-aware fine-tuning method. They validate the effectiveness of these methods through experiments on various model architectures and datasets.

strengths:
* This work provides a clear analysis of why DARE struggles with high pruning rates and large delta parameter mean/variance cases. Additionally, it includes a theoretical proof of DARq’s effectiveness as a modified rescaling strategy.
* The paper presents comprehensive experiments on both encoder- and decoder-based language models across multiple datasets. DARq, with various rescaling variations, consistently outperforms other state-of-the-art methods across models and datasets, demonstrating its robustness. Moreover, DARq can be combined with parameter-efficient tuning approaches, such as LoRA, broadening its applicability.
* The AdamR fine-tuning method achieves promising results, effectively maintaining performance even at high pruning rates.

weaknesses:
* This paper only compares their method with DARE, but there are lots of other methods proposed in the neural network pruning community. I wonder how this method compares to some existing pruning methods. For example, measuring the importance of weights in the original model and pruning the delta parameters based on that.

questions:
* In Table 3, could you elaborate on why global rescaling outperforms local rescaling?
* In Table 5, after a 0.99 pruning rate, the pruned model seems to be able to achieve better performance than the original and AdamR-L2 fine-tuned models. For instance, in the RoBERTa-COLA cell, it reaches 59.01 at a 0.99 pruning rate with a standard deviation of 1.26, indicating that it can outperform the unpruned model (59.27) in optimal test runs. Does this suggest the potential for further pruning? Could you experiment with even higher pruning rates to assess the method's limits?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730710348545

thread_type:
official_review

<thread_object>
creator: reviewer_cuxb

<content>
summary:
This work makes progress over DARE to obtain Prunable Delta Paramters at higher pruning rates. Towards this, the authors propose DARq, that modifies, the rescaling factor based on the insight that it is more prudent to look at output layer perturbations when pruning DPs as opposed to worrying about the expected change in delta parameters. The authors also propose to use AdamR, different from AdamW since it uses regularisation based on gradient norm, to obtain DPs that are more prunable than those obtained by AdamW. The authors also experiment with importance based pruning methods to show that they can obtain better result than random pruning. They also show that DARq works with structural pruning, LoRA.

strengths:
S1. The problem considered is quite interesting and not well-addressed in the literature

S2. The paper is well written

S3. Empirical results evidently show that DARq is performs quite well. The other insights provided are valuable too.

weaknesses:
W1. This paper seems quite exploratory. The authors do not conclude by providing a single method that they recommend to use for obtaining DPPs.

W2. Conclusions derived from Theorem 3.1 seem misleading. The authors provide an upper bound as opposed to a lower bound. On line 229, authors claim "from Thm. 3.1 as p increases, the absolute difference in outputs $|h^{diff}_{i} |$ grows at a rate of $\mathcal{O}((1 − p)^{-\frac{1}{2}})$, which is incorrect. But rather, $|h^{diff}_{i} |$ grows at most by $\mathcal{O}((1 − p)^{-\frac{1}{2}})$. It's not clear how tight this bound is to precisely make subsequent analysis hold true.



Typos:
Line 137: powerful

questions:
Please refer to the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730726765608

thread_type:
official_review

<thread_object>
creator: reviewer_s9bb

<content>
summary:
The paper proposes improved algorithms for Delta-Parameter Pruning (DPP), notably DARq and AdamR, which achieve substantial performance gains at high pruning rates.   It begins with a comprehensive analysis of DARE, an existing DPP approach, and leverages insights from this analysis to develop two enhanced algorithms. Furthermore, the proposed DARq method can be integrated with a parameter-efficient fine-tuning strategy to further minimize delta parameters. Additionally, the paper provides guidance on applying DPP across various scenarios.

strengths:
1 The  paper provides a detailed analysis of DARE about the failure cases 

2 base on the observation of DARE, they proposed two improved methods

3 their present the effectiveness of their method through theoretical and experimental points.

4 good presentation for reading 

5 consider different scenarios

weaknesses:
see questions

questions:
1 could you explain why you present a different range of regulation strength instead of a unity strength range to better understand the trend of the scale of dp in the Figure 6 ?

2 how do you conduct the experiments on the different regulation strengths in Figure 4? do they keep all other hyperparameters the same?

3 have you try lora + adamr-l2/adamr-l1?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732275109814

thread_type:
official_comment

<thread_object>
creator: area_chair_sb4k

<content>
title:
Discussion

comment:
Dear reviewers,

The authors have responded to your reviews. 

Until November 26th @ 2359 (AOE time) reviewers and authors can freely exchange responses, so if there any clarifications you require from the authors, now is the time to seek them!

Best,

AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper analyses the "DARE" delta-parameter pruning method, and its failure mechanisms. It then proposes two algorithmic improvements in the form of "DARq" and "AdamR". Reviewers appreciated the motivation and the analysis of DARE, as well as strong, comprehensive experimental results. Criticisms were fairly minor, but included a lack of comparison to other pruning methods, issues with the theory, and a lack of validation outside of language domains (points that were later addressed by the authors). The authors did well with their responses, leading to two reviewers increasing their scores. With entirely positive reviews, solid experiments, and a lack of major weaknesses, I believe this paper should be accepted.