SaRA: High-Efficient Diffusion Model Fine-tuning with Progressive Sparse Low-Rank Adaptation

Authors: Teng Hu, Jiangning Zhang, Ran Yi, Hongrui Huang, Yabiao Wang, Lizhuang Ma
Primary area: generative models
Keywords: Diffusion Model, Fine-tuning

Abstract:
The development of diffusion models has led to significant progress in image and video generation tasks, with pre-trained models like the Stable Diffusion series playing a crucial role.
However, a key challenge remains in downstream task applications: how to effectively and efficiently adapt pre-trained diffusion models to new tasks.
Inspired by model pruning which lightens large pre-trained models by removing unimportant parameters, we propose a novel model fine-tuning method to make full use of these ineffective parameters and enable the pre-trained model with new task-specified capabilities.
In this work, we first investigate the importance of parameters in pre-trained diffusion models and discover that parameters with the smallest absolute values do not contribute to the generation process due to training instabilities.
Based on this observation, we propose a fine-tuning method termed SaRA that re-utilizes these temporarily ineffective parameters, equating to optimizing a sparse weight matrix to learn the task-specific knowledge.
To mitigate potential overfitting, we propose a nuclear-norm-based low-rank sparse training scheme for efficient fine-tuning.
Furthermore, we design a new progressive parameter adjustment strategy to make full use of the finetuned parameters.
Finally, we propose a novel unstructural backpropagation strategy, which significantly reduces memory costs during fine-tuning.
Our method enhances the generative capabilities of pre-trained models in downstream applications and outperforms existing fine-tuning methods in maintaining model's generalization ability.  Source code is available at https://sjtuplayer.github.io/projects/SaRA.

Decision: accept (poster)
Reviewer scores: 6, 5, 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730479410553

thread_type:
official_review

<thread_object>
creator: reviewer_gtzk

<content>
summary:
This paper presents SaRA, a method for fine-tuning pre-trained diffusion models that introduces progressive Sparse low-Rank Adaptation (SaRA) to enhance efficiency and reduce memory costs in adapting diffusion models to new tasks. The proposed method leverages parameters with low absolute values, presumed to have limited initial impact on the model’s performance, making them suitable for fine-tuning. SaRA combines sparse parameter updates with a nuclear norm-based low-rank constraint to mitigate overfitting. It also introduces a progressive parameter adjustment and unstructured backpropagation strategy, aimed at further memory efficiency. Extensive experiments demonstrate SaRA's superiority over traditional fine-tuning methods on image generation and customization tasks.

strengths:
The paper clearly explains the intuition in parameter efficiency by proposing the use of low absolute value parameters for adaptive updates, effectively avoiding overfitting through a nuclear norm constraint.  The progressive parameter adjustment strategy positively contributes to the stability and convergence of model performance, while the unstructured backpropagation method effectively reduces memory costs, making SaRA a practical solution in resource-constrained environments. Additionally, the extensive experiments cover various tasks, such as image generation and customization, thoroughly validating the advantages of the SaRA method in balancing the preservation of model priors and the learning of task-specific knowledge.

weaknesses:
My main concern centers on the assumption underlying this approach—that parameters with the smallest absolute values are inherently ineffective. This seems more empirical than rigorously substantiated. While small absolute values can indeed correlate with lower impact in certain contexts, particularly in pruning, their effectiveness actually depends on the model architecture and specific task. Small value parameters may exert less direct influence on the output, but they are not intrinsically ineffective; their impact can vary depending on training dynamics, model structure, and optimization objectives.

Some minor weaknesses include:
* The generalizability of the 'adapting small-value parameters' strategy across different architectures is crucial to ensure broader applicability. This paper only investigates the phenomenon of the pre-trained Stable diffusion models. In this sense, I'm worrying about whether it can be applied across different network architectures or frameworks.
* In the caption of Figure 3, weight distributions are claimed to be Gaussian without further clarification, which seems empirical rather than solid.
* The choice of threshold for selecting low-absolute-value parameters could be better justified. If this choice is sensitive, it could limit SaRA's robustness and generalizability across different diffusion models and tasks. An ablation study on this threshold choice would strengthen the claims.
* While the experiments show SaRA's success, the paper could benefit from an analysis of its limitations. For instance, discussing cases where SaRA may not perform as well, such as tasks requiring extensive re-training of high-impact parameters, would improve the comprehensiveness of the evaluation.

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730483126646

thread_type:
official_review

<thread_object>
creator: reviewer_bh95

<content>
summary:
The paper proposes a PEFT method for diffusion models, which progressively trains selectively chosen parameters with small, inefficient values. To prevent memory waste, it avoids storing the gradient of all parameters and instead stores only the chosen parameters in separate nodes, which are then replaced after training.

strengths:
- Unlike methods like LoRA that require additional parameters, this approach selects parameters within the existing model for fine-tuning, minimizing additional memory usage.
- Existing selective PEFT methods continuously update masks, which requires storing gradients for all parameters, making them inefficient. In contrast, this paper’s method progressively trains only the fixed parameters at each stage, storing only certain gradients, making it more memory-efficient.
- The paper conducts extensive comparison experiments with various versions of Stable Diffusion (SD) and different sizes of fine-tuning parameters.

weaknesses:
- Although the paper differentiates itself from selective fine-tuning, I think this method still appears to be a form of selective fine-tuning. The memory efficiency improvement seems to stem from implementing a separate node for gradient storage and selectively fine-tuning only those nodes, rather than from an inherent algorithmic difference.
- Selecting parameters based on a specific threshold seems not a new concept. For example, the related work PaFi is described that it also trains based on absolute values.
- The reasoning behind inefficient parameters becoming efficient during training due to the randomness of the training process is unclear. Since the initial weights are set randomly, many values are likely to change through training. There is no strong basis to assume a correlation with the initial values. Even parameters with initially small values are expected to converge toward the average distribution, making Figure 3 somewhat self-evident.
- Although selecting a mask based on a threshold is computationally efficient, this method is relatively naive compared to other SFT methods that dynamically choose parameters during fine-tuning, which may lead to lower performance. Although it was compared with LT-SFT, there is a lack of comparison with other SFT methods.
- In the table, it is unclear if this method offers a significant performance improvement. While the FID scores are mostly favorable, the CLIP scores appear to be higher for LoRA. Additionally, the L_rank removal in the ablation study does not lead to a significant performance change.

questions:
- Would similar results be achievable if other SFT methods stored the selected parameters in leaf nodes and updated them as implemented here?
- Could you provide more explanation regarding the impact of training process randomness on initially inefficient parameters becoming efficient?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730661506585

thread_type:
official_review

<thread_object>
creator: reviewer_gku7

<content>
summary:
The paper presents a new method for fine-tuning diffusion models by training only low-value parameters making them effective on a new task. Additionally, a nuclear norm is used to prevent overfitting, and efficient selective backpropagation and progressive parameter adjustment reduce the memory and time requirements during training. The results show SaRA is on par or better than other fine-tuning methods in terms of FID, CLIP score, and qualitative assessment.

strengths:
1. The paper is very well written and easy to follow. It has a good flow of information. Every claim is carefully explained and proven by experiments or analysis.
2. The novelty of the method is good. The idea of fine-tuning only ineffective parameters was explored before but combined with nuclear norm regularization, novel approach to backpropagation of a sparse matrices, and adaptive fine-tuning, creates a valuable addition to the field.
3. The experiments are extensive, both comparisons and ablation study.
4. Qualitative results suggest an improvement over other methods.
5. Quantitative results show the model behaves better or similar to other methods. I can see it becoming one of the methods of choice depending on one's needs.

weaknesses:
1. Minor - as mentioned in Strengths 5., the results are not showing overall superiority over other methods.

questions:
1. The authors should consider splitting Figures 2 b) and 5 into more subplots. In my opinion, they are too cluttered and it takes too much time to read them.
2. In Table 1, the use of "optimal" and "sub-optimal" is not correct, e.g. optimal FID is 0. "Best" and "second best" or something similar should be used.
3. Could the authors provide more qualitative results for the same prompts (Appendix C) with different seeds to see the diversity of the samples?
4. It would be good to include some of the visual results in the main text.
5. Can authors elaborate on how CLIP score is related to overfitting?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730695082198

thread_type:
official_review

<thread_object>
creator: reviewer_efnm

<content>
summary:
To leverage the unimportant parameters concept in model pruning, this paper proposes a model fine-tuning method for diffusion model by reusing these ineffective parameters. The authors find that such ineffective parameters with small absolute values are random and dynamically change over finetuning. Based on this observation, they further design some efficient strategies for tuning these parameters.

strengths:
1. This paper verifies that the ineffective parameter concept also applies for diffusion model, i.e., the parameters with small absolute values are not important for the generation. 
2. This paper adopts a series of strategies for the efficient tuning of these ineffective parameters.
3. The effectiveness of this method is verified on the stable diffusion series. It demonstrates better performance over the baselines.

weaknesses:
1. The main limitation is that this paper poorly extends the concept in static model to the finetuning area, which is a dynamic model. Specifically, the ineffective concept works in model pruning, which is a given fixed model. The ineffective parameters in such static model can be discarded or reused. However, in this paper, the model parameters dynamically change during finetuning, and unimportant parameters also change. Thus, a right way to extend such unimportant parameters is to study their dynamics over change, instead of simple reuse.  Simple reuse may have several issues, for example, smaller optimal parameter search space in the finetuning case.
2. Another concern may lie in how to merge multi tasks’ parameters, and their merging performance. The multi-task parameter merging is a good property of LORA. It is encouraged to explain and verify this.

questions:
How to understand the better performance after setting some parameters to 0 in Section 3.1? This is actually interesting and may be useful for understanding the behavior of diffusion model.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720702467

thread_type:
official_review

<thread_object>
creator: reviewer_rosu

<content>
summary:
The paper presents a novel approach for fine-tuning pre-trained diffusion models called SaRA for visual content generation. The method builds on a key insight: parameters with the smallest absolute values in diffusion models contribute minimally to generation due to training instabilities, allowing for their selective reuse. SaRA enhances these low-impact parameters by applying a sparse weight matrix that learns task-specific knowledge while retaining the model’s generalization abilities. To avoid overfitting, the authors introduce a nuclear-norm-based low-rank training scheme. Additionally, SaRA includes a progressive parameter adjustment strategy and an unstructured backpropagation approach to efficiently manage memory use during fine-tuning.


Note: the supplementary materials contain the author's username and IP address (Supplementary Material/code/.idea/deployment.xml)

strengths:
1. The method visualizations (Figures 1 and 4) provide a step-by-step comparison of the proposed approach with previous techniques that helps the reader to easily understand the nuances of SaRA.
2. The authors show that the method could be applied to different diffusion models denoisers architectures U-Net (SD1.5 and 2.0) and Diffusion Transformer (SD3.0).
3. The design of the method allows a plug-and-play experience for the users that is highly beneficial for practical adoption.
4. The authors demonstrate the capabilities of the proposed approach on various widely-used by the open source community datasets.

weaknesses:
1. The authors introduce a novel Visual-Linguistic Harmony Index (VLHI) metric; however, it's described only in the appendix.
2. No comparison with the recent SOTA PEFT techniques (e.g., DoRA [Liu et al. 2024] that is available for different models on mentioned in the paper CIVITAI).
3. No ablation on scaling the trained SaRA weights for the inference (as the lora_scale parameter controls the influence of LoRA weights) or mentioning it in the limitations.
4. The authors say that for the FID computation they sampled 5K images from the source and generated data; however, BarbieCore dataset has only 315 images which is definitely not enough for the proper FID evaluation. The details about the sizes of the used datasets should be in the paper.
5.  CLIP L/14 used by authors is trained to provide overall image captions and could miss the details. The visual language models-based evaluations used in T2I-CompBench++[Huang et al. 2023] could be more accurate.
6. The authors skip the most popular Stable Diffusion XL 1.0 version, whereas they include 2.0.
7. Typographical mistakes such as:
*) Table 1: wrong column 2&3 names
*) Figure 1: addictive-> additive

questions:
see the weaknesses section
</content>
</thread_object>

</thread>
<thread>
creation date:
1731522201292

thread_type:
public_comment

<thread_object>
creator: ~kartikeya_bhardwaj1

<content>
title:
Related Work: Sparse High Rank Adapters (Neurips 2024)

comment:
Dear Authors,

Thanks for the interesting work SARA. It is important for the research community to focus on sparse finetuning techniques that are as memory-efficient as LoRA so we really appreciate your contributions. We recently published a highly related work at Neurips 2024 "Sparse High Rank Adapters" (SHiRA) which also presents memory benefits using a PEFT implementation compared to LoRA and analyzes multi-adapter fusion properties for sparse finetuning (see https://openreview.net/forum?id=6hY60tkiEK, older preprint: https://arxiv.org/abs/2406.13175). Can you please discuss our _concurrent_ work in your related work section? 

Another important related work is SpIEL. https://arxiv.org/abs/2401.16405.

It would be nice to see qualitative differences between these recent works (SARA, SHiRA, and SpIEL).

All the best!
</content>
</thread_object>

</thread>
<thread>
creation date:
1732289329000

thread_type:
public_comment

<thread_object>
creator: ~shitong_shao1

<content>
title:
LISA and SaRA [1/3]

comment:
Thank you for your interesting work! I believe the two key issues encountered when fine-tuning a diffusion model are: 

1.  The excessive memory overhead required by the optimizer to store the first and second-order moments in both AdamW and AdamW8bit.

2.  Additional memory demands arising from models like DPO and Consistency Model, which require the introduction of an additional EMA update. In other words, the algorithm proposed by the authors is not an algorithm specifically designed around the diffusion model.

Additionally, this algorithm is quite similar to LISA [1]. I personally experimented with LISA on a diffusion model, and the results were quite good, though it imposed a significant memory burden. Interestingly, I found that removing the first and second-order moments for parameters that do not need updating did not affect performance and significantly reduced the memory footprint.

[1] LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning, NeurIPS 2024.

I present the implementation of LISA in diffusion model here, everyone can try this:

```python
import torch
import numpy as np
import gc

class LISADiffusion:
    def __init__(self, model, other_model=None, rate=None, dtype=torch.float16, 
                 grad_aware=False, accelerator=None):
        self.model = model
        self.other_model = other_model
        self.dtype = dtype
        self.grad_aware = grad_aware
        if self.other_model is not None:
            for param in self.other_model.parameters():
                param.requires_grad = True
        self.rate = rate
        self.accelerator = accelerator
        self.last_epoch = 0
        num_sort = []
        name_sort = []
        total_num = 0.
        lisa_target_name = self.get_better_parameters_name
        for i, (name, param) in enumerate(list(model.named_parameters())[1:-1]):
            num_sort.append([i+1, param.numel()])
            total_num += param.numel()
            name_sort.append(name)
        self.allowed_index = [j[0] for j in sorted(num_sort,key=lambda x:x[1],reverse=False)]
        self.probability = [1. for j in range(len(self.allowed_index))]
        # used_num = 0.
        for i, j in enumerate(self.allowed_index):
            if any([(module_key in name_sort[j-1]) for module_key in lisa_target_name]):
                self.probability[j-1] = (0.9995 ** i)
            else:
                self.probability[j-1] = 0
        p_sum = sum(self.probability)
        self.probability = [j/p_sum for j in self.probability]
        self.initialize()

    def freeze_all_layers(self, model):
        for param in model.parameters():
            param.requires_grad = False

    def random_activate_layers(self, model, p):
        activate_number = int((len(list(model.parameters()))-2) * p)
        index = np.random.choice(range(1,len(list(model.parameters()))-1,1), activate_number, replace=False, p=self.probability)
        count = 0
        for param in model.parameters():
            if count == 0 or count == len(list(model.parameters()))-1:

                param.requires_grad = True
                param.data = param.data.to(dtype=torch.float32)
            elif count in index:
                param.requires_grad = True
                param.data = param.data.to(dtype=torch.float32)        
            else:
                param.requires_grad = False
                param.data = param.data.to(dtype=self.dtype)
            count += 1
        count_number = 0
        pos_count_number = 0
        for param in model.parameters():
            if param.requires_grad:
                pos_count_number += param.numel()
            count_number += param.numel()
        if torch.distributed.get_rank() == 0:
            print("Total Trainable Parameters:", round(pos_count_number*100/count_number,2),"%")

    def lisa(self, model, p=0.25):
        self.freeze_all_layers(model)
        self.random_activate_layers(model, p)
```
</content>
</thread_object>

</thread>
<thread>
creation date:
1732385648230

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary Of Changes

comment:
We sincerely appreciate all Reviewers and the Area Chair for their time and efforts for our paper. Following the valuable suggestions and insights provided in the reviews, we have carefully revised the manuscript with changes highlighted in **blue**. Specifically, we have made several major changes in the following aspects: 

1. We have compared our SaRA with more state-of-the-art methods, including **DoRA [1]**, and **DiffPruning [2]**, which are the representative RFT and SFT methods. The results further validate the effectiveness of our method. (Sec. C in Appendix)

2. We have conducted more comparison experiments on **Stable Diffusion XL 1.0 [3]**, which demonstrates the robustness of our method across different backbones. (Sec. C in Appendix)

3. We use an additional metric from **T2I-CompBench++ [4]** to evaluate the fine-grained alignment between the output image and the given text, where the results still show our model's superior ability to keep the model priors. (Sec. C in Appendix)

4. We have studied the **scale weight for SaRA parameters** and find that as the weight increases, the generated images will contain more task-specified features. (Sec. F in Appendix)
5. We add more studies on **merging SaRA parameters** learned from different datasets, enabling a more flexible application of our SaRA. (Sec. G in Appendix)
6. We conduct additional **ablation studies on the threshold** and find that SaRA with a bigger threshold can learn more task-specified knowledge. And with the help of our low-rank loss, SaRA will not overfit as the full-parameter fune-tuning methods. (Sec. H in Appendix)
7. We have added **more visualization results** of our method to show the generation diversity of our SaRA. (Sec. C in Appendix)
8. We have added a **limitation section** in the end of the Suppl to study the potential limitations of our methods, which gives a more comprehensive analysis of our SaRA. (Sec. M in Appendix)
9. We have corrected the mentioned typos and carefully checked our paper again. Moreover, we have also changed the figures as suggested.


**Reference**

[1] Liu S, Wang C Y, Yin H, et al. DoRA: Weight-Decomposed Low-Rank Adaptation[C]. ICML, 2024

[2] Guo D, Rush A M, Kim Y. Parameter-efficient transfer learning with diff pruning[J]. arXiv preprint arXiv, 2020.

[3] Podell D, English Z, Lacey K, et al. SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis[C]. ICLR, 2024.

[4] Huang K, Sun K, Xie E, et al. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation[J]. NeurIPS, 2023.

Detailed point-by-point responses are listed for each **Reviewer** below.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732502663172

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Please let us know whether we address all the issues (SaRA #229)

comment:
Dear Reviewers,

Thank you for your valuable comments on our paper.

We have submitted our responses to your comments and included additional results in the revised version of the manuscript. Please let us know if you have any further questions or require additional clarifications so that we can address them during the discussion period. We hope that after we have addressed all the issues, you will consider raising the score of our submission.

Thank you for your time and consideration.

Best regards,

Authors of SaRA #229
</content>
</thread_object>

</thread>
<thread>
creation date:
1732519527067

thread_type:
official_comment

<thread_object>
creator: area_chair_wbbm

<content>
title:
Please check the authors' responses

comment:
Dear reviewers, 

Could you please check the authors' responses, and post your message for discussion or changed scores?

best,

AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper works on the fine-tuning of diffusion models for new tasks. The basic idea is to identify the importance of parameters in the pre-trained diffusion model as the parameters with smallest absolute values. Then the proposed method fine-tunes these ineffective parameters using sparse weight matrix, with nuclear-norm-based low-rank regularization in fine-tuning. The proposed approach was applied to the downstream fine-tuning tasks on five styles, showing the better performance in FID, CLIP and VLHI metrics, compared with the LoRA, Adaptformer, LT-SFT.  This proposed approach tackles an important task for fine-tuning the diffusion models based on parameter selection and optimization. The reviewers commented on the strength in the experimental results and comparisons, and the fundamental idea of storing certain gradients and memory-efficiency, etc.