SaRA: High-Efficient Diffusion Model Fine-tuning with Progressive Sparse Low-Rank Adaptation
Authors: Teng Hu
Jiangning Zhang
Ran Yi
Hongrui Huang
Yabiao Wang
Lizhuang Ma
Venue: ICLR 2025 Poster (2025)
Keywords: Diffusion Model
Fine-tuning
Primary area: generative models

Abstract: The development of diffusion models has led to significant progress in image and video generation tasks, with pre-trained models like the Stable Diffusion series playing a crucial role.
However, a key challenge remains in downstream task applications: how to effectively and efficiently adapt pre-trained diffusion models to new tasks.
Inspired by model pruning which lightens large pre-trained models by removing unimportant parameters, we propose a novel model fine-tuning method to make full use of these ineffective parameters and enable the pre-trained model with new task-specified capabilities.
In this work, we first investigate the importance of parameters in pre-trained diffusion models and discover that parameters with the smallest absolute values do not contribute to the generation process due to training instabilities.
Based on this observation, we propose a fine-tuning method termed SaRA that re-utilizes these temporarily ineffective parameters, equating to optimizing a sparse weight matrix to learn the task-specific knowledge.
To mitigate potential overfitting, we propose a nuclear-norm-based low-rank sparse training scheme for efficient fine-tuning.
Furthermore, we design a new progressive parameter adjustment strategy to make full use of the finetuned parameters.
Finally, we propose a novel unstructural backpropagation strategy, which significantly reduces memory costs during fine-tuning.
Our method enhances the generative capabilities of pre-trained models in downstream applications and outperforms existing fine-tuning methods in maintaining model's generalization ability.  Source code is available at https://sjtuplayer.github.io/projects/SaRA.

Published as a conference paper at ICLR 2025
HIGH-EFFICIENT DIFFUSION MODEL FINE-TUNING WITH PROGRESSIVE SPARSE LOW-RANK ADAPTATION
SaRA
Teng Hul Jiangning Zhang"; Ran Yi Hongrui Huang', Yabiao Wang", Lizhuang Ma
Shanghai Jiao Tong University Youtu Lab, Tencent
(hu-teng, ranyi, 1 zma j.sjju. edu. cn 2C222120 68stu. hit edu. [vt (vtzhang, caseywang]@tencenss
ABSTRACT
The development of diffusion models has led to significant progress in image and video generation tasks, with pre-trained models like the Stable Diffusion series playing a crucial role. However, key challenge remains in downstream task ap- plications: how to effectively and efficiently adapt pre-trained diffusion models new tasks. Inspired by model pruning which lightens large pre-trained models by removing unimportant parameters,
propose SaRA, a novel model fine-tuning method with progressive Sparse low-Rank Adaptation to make full use of these ineffective parameters and enable the pre-trained model with new task-specified capabilities. In this work, we first investigate the importance of parameters pre-trained diffusion models and discover that parameters with the smallest ab- solute values do not contribute to the generation process due to training instabil- ities. Based on this observation, we propose fine-tuning method termed SaRA that re-utilizes these temporarily ineffective parameters. equating to optimizing a sparse weight matrix learn the task-specific knowledge. To mitigate potential overfitting, we propose a muclear-norm-based low-rank sparse training scheme for efficient fine-tuning. Furthermore, we design new progressive parameter adjust- ment strategy make full use of the finetuned parameters. Finally, propose novel unstructural backpropagation strategy, which significantly reduces memory costs during fine-tuning. Our method enhances the generative capabilities of pre- trained models in downstream applications and outperforms existing fine-tuning methods in maintaining model's generalization ability. Source code available /ttts::/sjjupp
github. io/ii/////o Lo/projects /AA
INTRODUCTION
In recent years, with the development of diffusion models (Ho al. 2020 Rombach et al. 2022
tasks such as image generation (Van Le et al. 2023 Zhang et al. 2023a), video generation Guo et al. 2023 Baatmann al. made significant advancements. Pre-trained diffusion models, particularly the Stable Diffusion se- ries (Rombach et al. 2022). have played crucial role in these developments, including image customization ( Van Le et al. 2023), image editing (Kawar et al. 2023), and controllable genera- tion (Zhang et al., 2023a] Mou et al. 2024). Additionally, by leveraging prior information from the image domain, diffusion models have been extended to tasks such video (Guo et al.) [023] Blattmann et al.] 2023) and 3D generation (Poole et al. 20222] Sun et al. 2023). As these appli- cations continue to evolve, core issue emerges: how to effectively and efficiently fine-tune the foundational pre-treee trainedide diffusion models and apply them to new tasks.
2023). and 31) generation (Poole al. 2022 Sun et al. 2023 have
Existing fine-tuning methods (Han al. 2024 Pan et al. 2024 Ansell al.) 2024 Sung et al.
2021 Fang et al. (024) can be categorized into three categories (Fig. 1b: 1) Additive fine-tuning (AFT) methods (Chen et al. 2022), which introduce additional modules fine-tune the model,
"Equal Contribution
Corresponding Author

---

Published as a conference paper at ICLR 2025
Output
Octput
Output
Output
Combine
From global to local
Extro
inference latency
Meege
A
Memory
officiant
LoRA Module
Adapper
Fre solnod weighte
Predipined
Train Portol
Weights
Paramelars
Input
Input
Input
Input
Addlive Fine-tuning
(b) Reparameterized Fine-tuning
Selective Fine-luning
(d) (Ours)
Frozen parameters without gradient
Frozen parameters with gradient
e
Trainable perameters
Figure 1:
The reparameterized fine-tuning methods (b) address the additional inference latency in-
troduced by additive fine-tuning methods (a) through reparameterizing the pre-trained weights from a global view. Selective fine-tuning methods (c) improve upon global parameter updates by em- ploying sparse updates, which better preserve the model prior by freezing most of the pre-trained parameters. Our SaRA (d) further enhances (c) by significantly reducing memory costs and achiev- ing superior performance in both adaptation capability and prior preservation.
such as adapter-based tuning (Ye et al. 2023 Mou et al. 2024). However, these methods require additional modules and parameters, which has changed the source model, and also introduced ad- ditional latency during the inference stage. 2) Reparameterized fine-tuning (RFT) methods (Hu al., 202T; Zhang et al. 2023b), which primarily utilize low-rank matrices learn new informa- tion and merge the learned parameters with the pre-trained one, but it still suffers from the risk of overfitting, since all parameters are adjusted by the low-rank matrices globally. Moreover, the choice of rank and the specific layers to which LoRA is applied requires a tailored design for each model. 3) Selective-based fine-tuning (SFT) methods (Guo et al. [220] et al. (2021), which select subset of the model's existing parameters for tone-tunigg. However, the complex parame- ter selection process and high memory cost restrict their application diffusion models. Overall, both AFT and RFT methods require model-specific designs, e.g., exploration of which layers to ap- ply Adapters or LoRAs within the model, and the hidden dimension or rank needs to be adjusted according the specific tasks. The SFT method introduces considerable latency, suffers from hy- perparameter sensitivity in parameter selection, and also performs poorly in terms of effectiveness and training efficiency. Therefore, pressing question arises: Can we design universal method that is model-agnostic, does not require hyperparameter searching, inherently avoids overfitting, and simultaneously achieves high-efficiency slug-and-play model fine-tuning?
Inspired by a theory in model pruning, which posits that within a trained model, there exist pa- rameters with relatively small absolute values that have negligible impact on the model's s output, an intuitive idea is: whether we can find way to leverage these ineffective parameters to make them effective again, and enhance the model's generative capabilities. To achieve this goal, the target "ineffective" parameters we seek must possess two properties: 1) temporary ineffectiveness: the parameters themselves have minimal impact on the current model's output; 2) potential effective- ness: the parameters not redundant due to the model structure, but have certain ability to learn new knowledge (if handled properly, they can be effective again). We first conducted an analysis on the influence of small parameters in pre-trained diffusion models the model outputs, and found that the smallest 10% (even 20%) of parameters by absolute values did not contribute much to the generative process (Fig. [ see Sec. [31]). Furthermore, we examined the potential effectiveness of these parameters and discovered that their ineffectiveness is not inherent (extrinsic) to the model's nature, but rather due the instability of the training process (see Sec. 3.2). Specifically, the ran- domness in the training process causes some parameters to approach zero by the end of training. This observation inspired us to rationally utilize these temporally ineffective parameters to make them effective again and fine-tune pre-trained generative models.
Therefore, we propose SaRA, a novel fine-tuning method for pre-trained diffusion models that trains the parameters with relatively small absolute values. We first identify the "temporally ineffective, potentially effective" parameters as paramters smaller than a threshold in the pre-trained weights. We then efficiently fine-tune these parameters in the pre-trained weights by sparse matrices while preserving prior knowledge. To mitigate the risk of overfitting due to the potential high rank of sparse matrices, we propose low-rank sparse training scheme, which employs nuclear norm- based low-rank loss to constrain the rank of the learned sparse matrices, achieving efficient fine- tuning of diffusion models. In addition, recognizing that some parameters may not be fully utilized
2

---

Published as a conference paper at ICLR 2025
during the fine-tuning process, we propose progressive parameter adjustment strategy. which introduces a second stage to reselect parameters below the pre-defined threshold and train them, en- suring that almost all parameters contribute effectively. Finally, different from the typical selective PEFT methods that retain the gradient of the entire parameter matrices and require high memory cost (the same as full-parameter fine-tuning), we propose an unstructural backpropagation strat- egy with smaller memory cost. In this strategy, we only retain the gradients for the parameters to be updated, and automatically discard the gradients for other parameters during the backpropagation process. This results in a memory -effrrr-efficient selective PEFT method, which also advances the devel- opment of future selective PEFT techniques. Compared to previous tinettuning methods (Hu et al.] 2021 Valipour et al. 2022 Hayou et al.) 2024), our SaRA is capable of effectively enhancing the generative capabilities of the pre-trained model itself, and it also demonstrates the best ability for model adaptation and prior preservation in different downstream tasks.
Contributions of this paper can be summarized in the following four aspects: importance of the parameters in pre-trained diffusion models, revealing the temporal ineffectiveness and potential effectiveness of the parameters with the smallest absolute weight, which motivates us to make full use of these parameters. 2) We propose SaRA, a novel efficient fine-tuning method based on progressive sparse low-rank adaptation, enabling the model to learn new knowledge without influencing the original generalization ability. 3) We propose unstructural backpropagation, which resolves the high memory consumption problem of selective PEFT methods and surpasses LoRA in memory efficiency (save more than 40% GPU memory than LoRA and selective PEFT methods). 3) We efficiently encapsulated and implemented our method in single line of code modification, which significantly reduces the coding overhead associated with fine-tuning pre-trained models.
1eWe investigate the
2 RELATED WORKS
2.1 DIFFUSION MODELS
Diffusion models (Ho et al. 2020 Rombach et al. 2022) have demonstrated significant advantages in image generative tasks. Text-to-image models, represented by Stable Diffusion (Rombach al.) 2022]. have diversified into various applications. However, their large parameter sizes somewhat limit the feasibility of full fine-tuning to adapt to specific new tasks. Methods such as Control- Net (Zhang et al., 2023a), T2I-Adapter (Mou et al. 2024), and IP-Adapter (Ye et al.
2023) achieve
controlled generation under different conditions by adding external networks diftusion models. Additionally, models like LoRA (Hu et al., (02T) and DreamBooth (Ruiz et al. (023) enhance the
original diffusion models through fine-tuning, enabling them to generate content in new domains and
concepts. Furthermore, some video generation models (Guo et al.] 2023) Blattmann et al.] 2023) are built on diffusion models to achieve video generations and employ Lora and adapters to accomplish controllable video generations.
2.2 PARAMETER- MODEL FINE-TUNING
Addictive Parameter Fine-tuning (AFT), AFT introduces additional modules to the model while keeping the pre-trained backbone fixed. Serial Adapter (Houlsby et al., 2019) enhances the Trans- former block by adding new modules after the self-attention layer and FFN layer. AdapterFu- sion PPeeifeer et al. 2020) streamlines this by inserting adapter layers only after the FFN layers to boost computational efficiency. Parallel adapters. including Adaptformer (Chen et al., 2022). CoDA (Lei al.) 2023), and KronA (Edalati et al.] (022). reorganize the traditionally sequential adapter layers into a parallel side-network, optimizing both performance and efficiency. To further enhance adapter performance and generalization, multi-task learning strategies like AdaMix (Wang al. 2022). and Hyperformer (Mahabadi et al. ) have also been developed.
Reparameterized Parameter Fine-tuning (RFT). An early work (Aghajanyan et al., 2020) has verified the presence of low intrinsic dimensionality in pre-trained models. LoRA (Hu et al. 102 proposes to use a low-rank matrix to learn new feature representations. To address the issue of selecting the appropriate rank, DyLoRA (Valipour al. (002)) employs dynamic and search-free approach to obtain the optimal rank. AdaLoRA (Zhang et al. 2023b) decomposes the trainable low- rank matrix using singular value decomposition (SVD) and implements dynamic rank adjustment
3

---

Published as a conference paper at ICLR 2025
(o] Frequency- Frequency-Paraadde Parameter Value Difluioon Model
(b) and CLIP on Threshold
1310
Prosuee
1325
AF10x115
921-1F0
131.
$35-10
(11 Stubre Difision 1.4
(2) Stakki Dilizion
31.4
212
502. CUP Scoe
Scoe
ACUP Scorects
575 Scor
035
Pouer (4) Scotle DiPusion 3.0
(31 Stabie Difusion 2.0
Piotrolnod modal
Throshold
Figure 2: (a) Weight distributions of the pre- rre-traied parameters in Stable Diffusion (SD) 1.4, 1.5, 2.0, and 3.0, which are all similar a Gaussian distribution, therefore large number of parameters are around 0. (b) The performance (FID and CLIP Score ) of SD Models when the parameters in the pre-trained models with absolute values smaller than a certain threshold are set to
by pruning singular values. Furthermore, numerous subsequent methods YYang al. 2023 et al. 2023 Hayou et al. 2024) have aimed to enhance the performance of LoRA.
Ding
Selective Parameter Fine-tuning (SFT). Selective parameter finetuning (Han et al. 2024 methods finetume selected subset of the parameters in the pre-trained model. Diffpruning (Guo et al.) 2020) fine-tunes specific parameters by learning a mask matrix, constraining its size through differen- tiable LO norm. PaFi (Liao et al.) 2023) selects the parameters with the smallest absolute values for learning, while LTSFT (Ansell et al. 2021), grounded in the Lottery Ticket Hypothesis (Frankle & Carbin 2018), selects the parameters that change the most during fine-tuning. SHiRA (Bhard- waj et al. 2024) proposes a sparse high-rank adaption method to improve the adaptation ability. Essentially, these methods all learn a sparse mask matrix fine-tune the pre-trained models.
Overall, both AFT and RFT methods require model-specific designs, e.g., determinnng which layers apply Adapter LoRA, and adjusting the hidden dimension rank according specific tasks. Additionally, the SFT method introduces significant latency, exhibits hyperparameter sensitivity in parameter selection, and underperforms in terms of effectiveness and training efficiency, In contrast, our SaRA is model-agnostic, which eliminates the need for layer selection, and can effectively fine- tune the pre-trained model while reducing training costs in both time and memory.
3 THE POTENTIAL EFFECTIVENESS OF THE INEFFECTIVE PARAMETERS
3.1 INEFFECTIVE PARAMETERS IN STABLE DIFFUSION MODELS
Based on the theorem model pruning proposed in (Liang et al. 2021). which regards the parame- ters with the smallest absolute values "ineffective" parameters, we investigated the effectiveness of these parameters in pre-trained stable diffusion models (version 1.4, 1.5, 2.0, and 3.0). We set parameters with absolute values below a certain threshold 0g (from 10-3 10 10-5) to zero, and evalu- ated the performance of the regularized models on generative tasks with CLIP Score (Radford et al.) 202T) and Fréchet Inception Distance (FID) (Heusel et al. 2017).
The results are shown in Fig 2()). We observed that within a certain threshold range 0, E (0, 10-3), with the small parameters set O, the generative ability of the SD models is minimally affected. And in some cases, the regularized model with "ineffective" parameters set to 0 even outperforms the original model (i.e., no parameters are set with 0, = 0). Specifically, SD1.4 and SD1.5 show better FID scores than the original model when thresholds are in the range of 0; E [5 x 10-4, 10-4,10 10-3], and SD2.0 and SD3.0 exhibit superior FID scores at a threshold 0, = 10 These results show that parameters with the smallest absolute values have a limited impact on the generative process, and in some cases, they may even slightly impair the model's generative ability.
3.2 UNSTABLE TRAINING PROCESS CONTRIBUTES TO USELESS PARAMETERS
Sec. 3.1 demonstrated that parameters with smaller absolute values have minimal impact on the generative capability of diffusion models. A natural question arises: are these currently ineffective
4

---

Published as a conference paper at ICLR 2025
parameters caused by the model structure and inherently redundant, the training process and can become effective again? it the former case, the structural design of the model prevents these parameters from learning effective information, then these parameters are redundant and unlikely to be useful in subsequent training processes. While if it is the latter case, these parameters are potentially effective when leveraged rationally in the subsequent training. Therefore, we further investigated the reasons behind the ineffectiveness of these parameters, and found that the ineffectiveness is due the randomness of the optimization process, rather than an inherent inability caused by model structure.
they caused by
Specifically, we employed a Stable Diffusion model pre-trained on the FFHQ dataset (Karras et al. 2019), whose parameter matrices are denoted as Po- We recorded the parameters in the pre-trained model with absolute values below the 1% threshold 0, by a parameter mask M, where Par Po O M denotes the initially below-threshold parameters (1% of all parameters), and PI-M Po (1 M) denotes the initially thove-threshole parameters (99% of all parameters), satisfying:
Training process
1.00%
Comwagge
0.80%
Percentage
([tt i dij Pl IF
0.00%
iile full [e]
0.40%
Elll 9 Pua!]
0.20%
0.00%
2000
4000
6000
8000
10000
Iterations
|p| < 0,VV E PMs:
Figure 3: The changes of parameters whose absolute values are bewlow the 1% thesh- old 0, during full-parameter fine-tuning. The blue and yellow curves show the proportions of parameters originated from both the ini- tially below-threshold PM and the initially above-threshold PI-M.
(1)
|pl 0, PI-M-
Then, we continue training this pre-trained model on the FFHQ dataset, and observe the changes of its parameters P during the training process. During this fine-tuning stage, we recorded the "source" of parameters whose absolute values are below the threshold 0c((lp) < 0, i.e. whether they are initially worowshhesholdddco or initially thooe-threshodd And we found that these parameters originated from both the initially below-threshol. threshold Par and the initially bhove--hreehool PI-M-
The proportions of these two groups and how they change during the finetuning are shown in Fig. 3 As the training progressed, the proportion of Par remaining below 8 gradually decreased from 100% to 1% (blue curve decreases from 1.00% to 0.01%); while 1% of the initially above-threshold PI-M eventually fell below O, (yellow curve raises from 0.00% to 0.99%). The results indicate that initially ineffective parameters PM caused by the randomness of the training process, mostly become effective over time (only 1% remaining below threshold). Conversely, some initially effec- tive parameters become ineffective as training continues. This pheonomenon demonstrates that the ineffectiveness of parameters is not inherent to model structure. but rather result of the stochastic nature of the training process, which causes some parameters to fall below the threshold 0, at the last training step coincidentally, making them temporarily ineffective. As the training continues, most of these parameters regain effectiveness, proving their potentially effectiveness, which motivates us leverage these temporarily ineffective parameters to fine-tune the pre-trained model.
PROGRESSIVE SPARSE LOW-RANK MODEL ADAPTATION
Inspired by the potential effectiveness of parameters with the smallest absolute values, as discussed in Sec. we propose SaRA, novel parameter-efficient fine-tuning method designed to fully utilize these temporarily ineffective parameters. Specifically, we first identify the ineffective parameters in the pre-trained parameters Po by computing sparse mask M = Po < O,, where 0, is a threshold and the sparse mask only selects a small portion from all parameters. We then use this sparse mask to update the initially ineffective parameters P M, while keeping the initially effective parameters P & (1 M) frozen. This approach enables the pre-trained model to acquire new capabilities for downstream tasks (through the learnable P 3 M) while preserving prior information (through the fixed O (1 - M)). To avoid the problem of overfitting caused by strong representation ability due to the potential high rank of the learnable sparse matrix P o M, we propose nuclear norm-based low-rank loss to mitigate overfitting (Sec. 4.2 In addition, we propose a progressive parameter adjustment strategy to further make full use of the ineffective parameters by progressively reselecting them (Sec. 4.3) Finally, we propose an unstructured backpropagation strategy, which significantly reduces memory costs and can be applied to enhance all selective PEFT methods.
5

---

Published as a conference paper at ICLR 2025
4.1 FINE-TUNING ON THE POTENTIAL EFFECTIVE PARAMETERS
In Sec. 3. we have demonstrated that parameters with small absolute values are ineffective in the generative process of diffusion models, and this ineffectiveness is not due to the model's architec- ture but rather the stochastic nature of the optimization process. Therefore, we propose SaRA, which fine-tunes these temporarily ineffective parameters to adapt the pre-trained diffusion model down- stream tasks, enabling it to learn new knowledge while preserving its original generative capability. Specifically, we first obtain a mask M for the initial parameter set Po. which satisfies:
pl f., Vp E Po O M,
where M is sparse matrix, since the theshold 0g is set low and only selects a small portion from all the parameters. We then use this sparse mask update the initially ineffective parameters PM POM while keeping the initially effective parameters (0 frozen. During training, for the gradient V of the parameters, we use the pre-defined sparse mask M to retain the gradients we need and update the corresponding parameters PM P ④ M by:
PMM VP @ M + 0 @ M),
Prew P X VPM-
In this way, we can focus on training the ineffective parameters while keeping the other parameters unchanged, ensuring the original generation ability of the pre-trained model is preserved, while learning new knowledge by the parameters PM-
4.2 NUCLEAR NORM-BANE BASED LOW-RANK CONSTRAINT
The sparse parameter matrices PM can sometimes have a high rank, resulting in strong representa- tional capabilities that may lead to overfitting during the training process of downstream tasks. To mitigate this issue, we introduce nuclear norm-based low-rank constraint on the sparse matrix to prevent the rank from becoming excessively high during the training process.
A direct way to apply low-rank constraint is to minimize the rank of the sparse parameter matrix Pank(P) as a constraint. However, directly minimizing the rank function is computationally in- tractable due to its non-convex nature. Therefore,
nuclear norm to estimate its rank:
Parll = Σ oo (PM), where o; are the singular values of Pas-
(4)
To compute the nuclear norm | Bhll ll.. we employ the singular value decomposition (SVD) of the matrix PM = UEVT, where U and V are orthogonal matrices, and Σ is a diagonal matrix con- taining the singular values o, (Pr)). The subgradient of the nuclear norm PM has been derived by (Watson, 1992). Based on this derivation of the nuclear norm gradient, we can ensure that gradi- ent descent methods can be employed to incorporate nuclear norm-based low-rank constraints into the training process, thereby achieving our nuclear norm- beane low-rank constrained loss:
Lrank ||PMll+
Σ (PM))
(5)
4.3 PROGRESSIVE PARAMETER ADJUSTMENT
As discussed in Sec. 3.2 and Fig. when continuing training the pre-trained model, the initially ineffective parameters gradually become above threshold and effective, with only 1% of initially ineffective parameters remaining below threshold eventually. However, the speed at which inef- fective parameters become effective (the slope of blue curve in Fig. B varies during the finetuning process. In the early stage of the finetuning process (e.g., the first 2.5k iterations), a large portion (over 80%) of initially ineffective parameters quickly become effective, with a small part (less than 20%) remaining below threshold. However, the speed slows down in the later stage of finetuning: from 2.5k to 8k iterations, the small portion of remaining below-threshon parameters jumps out of the theshold very slowly. However, the finetuning iterations are typically limited (e.g., few thou- sands), in which case the slow speed in the later finetuning stage can cause problems: the remaining telow-threshen ineffective parameters may not be trained to be effective and fully utilized.
To address this issue, we propose a progressive parameter adjustment strategy. To alleviate the slow speed of ineffective parameters becoming effective in the later stage, we reselect the ineffective

---

Published as a conference paper at ICLR 2025
X1+3
X
Xx=
X
X
Lor
Mucki
Upanne
Perssweer
LoRA
Selective PEFT
(c) Ours
Foraard -...e Sardwardd
Intermediine Varkatble
Punction
Paomeeer grad
Primerater without gracken!
Figure 4: Visualization of our unstructural backpropagation. a) LoRA stores additional interme- diate variable X in each LoRA layer, and b) selective PEFT methods store the gradients for the whole parameters matrices, causing waste of memory and computation resources. c) In contrast, our Unstructural Backpropagation method extracts the trainable parameters. sets them as indepen- dent leaf nodes, and only retains gradients for them, which largely reduces the memory cost.
parameters that remain below threshold (about 15%-20% of initial ineffective parameters) after the early finetuning stage and focus on optimizing these remaining below-threshold parameters in the subsequent finetuning stage. Compared to finetuning without this reselecting operation, this strategy can quickly make remaining ineffective parameters effective again in the later finetuning stage.
Specifically, we introduce parameter readjustment phase. After the early finetuning stage (we set the first half of the total iterations as the early finetuning stage, egg, 2,500 iterations when there are 5,000 finetuning iterations) on the initially selected below-threshold parameters po Plearg we reselect parameters from P that remain below the predefined threshold as new trainable parameters Pieern (which a subset of po Plearn and typically has 15%-20% of Piearn 's parameters). Then in the subsequent finetuning stage, we only optimize this subset of initial ineffective parameters, and keep other parameters of Plearn frozen. By focusing on optimizing the small subset remaining below- threshold parameters, this strategy greatly improves the speed of ineffective parameters jumping out of the threshold in the later finetuning stage, thereby enhancing the model's s adaptation capability. In our experiments, found that under the same number of finetuning iterations, models without learn remained ineffective after finctuning, while models with the progressive strategy only had 2% of Po learn that still ineffective. The results indicate this strategy significantly improves the performance of our method during the fine-tuning process.
the progressive strategy had 15% of po
4.4 UNSTRUCTURAL BACKPROPAGATION
Currently, both the LoRA-based methods (the same for the adapter-based methods) and selective PEFT meth- ods cause a heavy burden on the computation resources: 1) For the LoRA- bbsed methods, since the LoRA mod- ule is additional to the original model, there is no need to store the gradients of the model parameters, but they still require additional memory costs to store the inter- mediate variables in the LoRA module, which is shown in Fig. (a). 2) And for the selective PEFT methods, persistent issue is that they require the same more computational resources (especially GPU mem- ory) as aull-parameter fine-tuning. Although they only finetune a subset of the model's parameters for fine- tuning, they retain the gradients of the entire parame- ter matrices P, because the mainstream deep learning libraries (such PyTorch and TensorFlow) only sup- port gradient backpropagation and updates for the en- tire parameter matrices. Consequently, previous selec- tive PEFT methods had to perform gradient backpropagation on the entire parameter matrices P. and then use pre-computed mask matrices M to mask out the gradients unnecessary parameters by VPM M O VP, and perform an overall parameter update by Prew = P AV PM (visual- ized in Fig. (b)). This approach necessitates storing the gradients of all model parameters and the additional mask matrices, leading to greater computational resource demands than full- hall-parameter fine-tuning. This clearly contradicts the "efficient" requirements of PEFT and limits the practical applications of such methods.
SaRA (Ouns)
Memory (GB)
LoKA
-52%Memory
40
8.208
Memory
SaRA (Ours)
Selective PEPT
LoRA
Time (1000))
44%%Timo
15
10
Batch
Figure 5: Computation cost on memory and time of different PEFT methods.
7

---

Published as a conference paper at ICLR 2025
Imageher ditaset
Fretune
doracet
52
Aul disture
LofA
13
330
Out
225
Acorfarne
23.0-
Dan
Patra bnd Node
0 65.6
120
25.4
352
42
Figure 6: Quantitative comparison among different PEFT methods on Backbone Fine-tuning on ImageNet, FFHQ, and CelebA-HQ datasets. Our method achieves the best FID scores, indicating our method effectively improves the performance of the pre-trained models the main task.
To address this issue, we propose Unstructural Backpropagation (shown in Fig. (c)), which supports efficient gradient backpropagation and updates for unstructured parameters. Different from previous selective PEFT methods that require retaining the gradient for the whole parameter ma- trices, our Unstructural Backpropagation only needs to retain gradients for the selected subset of below-threshold parameters PM. Specifically, we first store the mask matrices M corresponding to each layer's parameters that need to be trained] In the computational graph, we deviate from the traditional approach of setting model parameters as leaf nodes. Instead, we extract the trainable M lo and set them as independent leaf nodes, where [:] denotes element-wise indexing of the matrix. Lee. shown in Fig. (c), we extract the subset of train- able parameters and combine them into a separate parameter vector, and only retain gradients for this vector. Then, during the forward pass, we define an Unstructural Mapping function U (-) to update the model parameters P by:
parameters Ptearn P[M]
[ [[[M]
Plearns = = M).
P (M, Plearn, M), where
(6)
P[1 M]
And the updated model paramaters P will then participate in the training process. During back- propagation, define the Unstructural Backpropagation function UB) propagate the gradients from the model parameters to the trainable parameters by:
V Ptearn UBVVP, M) P[M].
In this way, during backpropagation, the gradients on the model parameters VP will be automati- cally cleared, since is no longer leaf node, and only the gradients on the learnable parameters Pleearn are stored, which significantly reduces the GPU memory during the training process. No- tably, unstructual backpropagation is not limited to our method but can be employed in other SFT methods like LT-SFT Ansell et al. (2021) which can advance the development of future SFT fields.
5 EXPERIMENTS
To validate the effectiveness of our method, we conduct experiments on various tasks, including backbone fine-tuning, downstream dataset fine-tuning, image customization, and controllable video generation (appendix). We compare our method with three state-of-the-at parameter efficient fine- tunining methods: LoRA (Hu et al. 2021). Adaptformer (Chen et al.) 2022), and LT-SFT (Ansell ct al. 2021); along with the full-parameter fine-tuning method. We evaluate the generation models by three metrics: 1) Fréchet Inception Distance (FID) (Heusel et al.) (017). 2) CLIP Score, and 3) Visual-Linguistic Harmony Index (VLHI), which balances FID and CLIP Score by:
CLIP, min([CLIP,) VLHI = max((FID,) (FID max (CLIP] min (CLP())
max((FID,))
FID,
5.1 BACKBONE FINR-TUNING
Different from the previous paramcter-efficient fine-tuning methods that mainly aim to fine-tune the pre-trained model to downstream tasks, our model enables the pre-trained model to make full use of the parameters. In other words, our finetuning method can improve the performance of prained models the main task (the original task it is trained on), by optimizing the initially ineffective
Since the mask M is of boolean type, it does consume significant GPU memory.
8

---

Published as a conference paper at ICLR 2025
Tucdees
Pazza:
Middd
Berkeeberr
Coleryak
Shreefie
Prpeñties
CUPP
LakA
15.05
28.99
1.35
117.0
38.22
1.66
1$1.0
27.47
1.20
136.11
22.34
1.30
196.36
26.99
1.38
15074
21.66
1A5
S0M
177777
165.04
157.03
3350
14.10
0.51
1/00
13..9
136.1
33:13
3531
0.45
D106
171.7
15.03
15:11
36.3
6:41
1.5
19:70
136.0
27.11
36.1
583
15
151.5
15..5
26.00
25.35
135
186
150.5
70/66
25.36
0.04
1:14
5x (0hmn))
146.54
25.00
172
12.22
152
25.21
1.40
131.5
22.54
1.48
148.6
23.40
133
15456
1.59
LokA
15.05
1251
112
176.0
27.0
152
13.35
11.00
15..0
27.05
1.40
105.11
501 15
27M
All
195.22
29.08
23.70
0.59
1.36
133.35
12.85
35.00
23.33
1.19
191.1
12.15
23.55
35.33
6.45
0.5
146.77
130155
36.0
033
16547
15/10
26.95
36.90
1.30
1.30
155.4
15024
25.99
25.99
0.42
1.21
3xMA (Ohun))
155.85
20.33
146
106.09
136.4
20.33
1.58
12556
14542
26.40
1.29
136.0
La9
LoRA
13.55
166
107.55
154.91
27.34
122
1551
15385
26.35
1.40
101.3
1.44
SM
29.31
1.1
131.5
25.11
15.35
0.34
14501
171.5
26.55
0.54
136.0
1.11
12-577
34.23
1/8
127.11
312-47
35.90
0.68
155.00
17641
27.34
1.00
165.22
26,77
0.27
SxRA (Ount)
23.29
1,64
118.3
174.0
27.84
1.13
134,8
15724
36.97
1:33
147.0
27.75
1.44
27.37
140
130222
21.84
136.97
13.09
0.0
1200
134.6
24.44
1.00
133.00
1.30
LakA
195.4
1.64
136.22
25.00
1.55
192.0
27.39
1.25
18.15
27.55
1:18
26.99
1.89
159.1
2645
1:31
3194
Adgiberar
151.57
154.00
38.75
25.13
1.50
1.75
13571
135.22
20.40
27.35
1.50
111
17644
185.25
$3.50
$7.35
0.50
122
18894
142.0
27.06
27.56
2.4
10512
175.50
26.55
24.99
0.75
1.5
10033
16343
21/15
21/13
0.37
1.23
SaNA (Qurd)
19272
25:12
1:51
13505
25.39
1.56
191.0
21.34
1.68
19677
26.91
16660
26.71
1.0
15020
21/6
1.59
LOKA
15132
12.11
1.55
12:
35.34
3100
(919)
27.12
1.3
147/6
27.00
10010
26.77
1.13
155.0
26.66
1.53
2114
Adgeterar
199.29
155.09
38.42
29:29
1047
1.70
141.90
135.35
23.47
27.92
030
19417
19457
33.54
37.33
0.66
1.0
155.51
157.31
27.56
37.87
2.00
175.50
17466
27.12
26.99
0.76
1.54
16500
16548
27:25
27.86
0.39
1.33
SARA (Ound)
18437
22.23
1,50
13.25
25.23
1.60
16667
27.39
3.79
14h22)
27.01
2.59
16540
26.71
1.50
155.00
20/15
1.99
E.RA
163477
38.01
1.30
132.35
36.33
126
15:.5
$7.65
1.54
15:.0
37.41
1.8
16000
56.11
1.35
156m
27.09
1.41
350
Adgethear
125877
363.55
175.45
33.53
29.74
0.23
1463
143.1
137.54
35.55
27.09
D.66
D/6
186-4
391.1
37:45
37.39
0.70
3.15
101.77
100.33
37.37
27.90
576
1.00
13007
186.0
27.42
36.75
0.71
1.00
114.23
166.08
27:05
27.42
0.32
1.33
SARA (Ound)
155.7
18.59
1,41
13.92
27.37
1.15
17475
27.00
1.40
15508
27.29
1.00
106.00
26.30
1.33
15850
25:45
1.53
TVll Batraw
193.97
34.30
1.20
11.19
34313
1.70
142:15
35.81
6.16
137.11
36.11
1.33
105.00
55.95
1.00
155.31
77.16
0.43
LaRA
23:27
1.2
133.9
20.35
1.29
197:30
27.34
1.26
140.38
26.33
19000
26.90
1.35
15840
27:45
1.32
31M
Adattoemaz
284.9
20.33
23.49
0.29
1.42
12675
135.99
25:23
21322
0.10
133
19635
352/4
27:34
27.39
6.14
6.50
28616
19677
2681
27.32
230
19811
197:26
26.00
26.31
0.35
1.59
16273
19410
21:4
21:26
0.35
SARA (Dunn)
120.3
18.33
1.25
126.55
38.45
1.50
17647
37.63
1.68
137.92
36.17
1.35
1966
27.00
1.65
15:11
200
1.20
EIRA
33:35
1.3
13112
1.48
187.14
27.55
3.62
14..9
36.6
174.2
35.35
1.50
1565
27.25
SD:30
21M
Adatberas
147.7
30:30
1,40
100
135.5
15:31
38.31
10:00
0.19
175.01
381.75
27.30
6:16
3.54
1500
195/8
36.50
15
177.0
174.1
26.30
06.11
0.11
0.99
171-4
16077
27.99
77:39
0.3
SaRA (Duee))
35531
18.43
1.20
15.99
38.14
27.31
1.78
19115
36.55
16h.5
27.81
1.18
147.5)
27.66
LaRA
351.03
33:31
13417
38.55
174.6
37:33
3.48
172.6
35.85
1.33
18641
37.87
Adytteras
368.1
33.50
1.00
127.55
27.89
354.99
37.70
3.08
1550
37.85
26.81
1.09
16:31
5000
156.36
1.55
13.91
27.68
0.25
181.4
22.30
155.5
23.30
19525
3651
8.49
161
71A1
SARA (Ounn))
18.60
1/
34.91
22.03
1.24
152.0
22.02
27.30
1.31
2514
103
1330
175%
3051
1:00
1903
Table 1: Comparison with different parameter-efficient fine-tuning methods on Stable Diffusion 1.5, 2.0, and 3.0. For most of the conditions, our model achieves the best FID and VLHI score, indicating that our model learns domain-speciffh knowledge successfully while keeping the prior information well. Bold and underline represent the best and second best results, respectively.
Prantt
Catmel
LORA
Adaptareer
LT4FFT
FulFirrou
Rutunati numelaa fertaing poker who neking pooerdl fer
siring nuterr fattolly poklig
fott-nng
rell' who
Figure 7: Comparison of the generated images between different PEFT methods.
parameters to be effective and thus increasing the number of effective parameters. Therefore, apart downstream tasks like traditional PEFT methods, we first apply our method to the main task of the pre-trained model, continuing to tune the backbone on the original train- ing dataset, in order to explore whether method can enhance the base model's performance. Specifically, we employ the pre-trained Stable Diffusion models on ImageNet (Deng et al.) 2009). FFHQ (Karras al. 2019), and CelebA-HQ (Karras et al. 2017) datasets, and fine-tune them on these pre-trained datasets for 10K iterations. We compare our method with full-parameter finctun- ing, LoRA, AdaptFormer, and LT-SFT by computing the FID metric between 5K generated data and 5K randomly sampled data from the source dataset. The results are shown in Fig. 6 which demon- strates that our method achieves the best FID scores, indicating our method effectively improves the performance of the pre-trained models on the main task.
from experimenting
5.2 MODEL FINE-TUNING ON DOWNSTREAM TASKS
Downstream Dataset Fine-tuning, In this experiment, we choose 5 widely-used datasets from CIVITA with 5 different styles to conduct the fine-tuning experiments, which are Barbie Style,

---

Published as a conference paper at ICLR 2025
Cyberpunk Style, Elementfire Style, Expedition Style and Hornify Style. To comprehensively com- pare PEFT methods, we conduct three sets of experiments for each PEFT method on Stable Diffusion 1.5, 2.0, and 3.0. with selected trainable parameter sizes of 50M, 20M, and 5M. We compute the FID score, CLIP score for the generated data, along with VIHII which measures both the style (FID) and generalization (CLIP score). The quantitative results are shown in Tab. [] from which we can draw the following conclusions: 1) Our model can always achieve the best VLHI on average across five datasets, indicating that our model can preserve the prior information in the pre-trained model well (a good CLIP score), while learning as much task task-specific knowledge as possible (a good FID), outperforming all the other PEFT methods and full-finetune method; 2) As the number of learnable parameter increases, our model can learn more task task-specific knowledge (better FID), but may lose part of the prior information (lower CLIP score): 3) For Stable Diffusions 1.5 and 2.0, our model achieves the best FID and usually the second best CLIP score on average across five datasets, and under different parameter numbers; while for Stable Diffusion 3.0, which has much more parame- ters than SD 1.5 and 2.0, our model achieves the best CLIP score and usually the second best FID on average across five datasets. The results indicate that for a larger pre-trained model, more learn- able parameters are needed to learn the mask-specific knowledge well. Moreover, we provide some qualitative comparisons in Fig. 7 which shows the superior generation quality of our method (See appendix for more details.).
5.3 ABLATION STUDIES
We conduct ablation studies to validate the ef- fectiveness of our proposed modules: the pro- gressive parameter adjustment (PPA), and the low-rank constrained loss (Crank). Then, further assess the effectiveness of training pa- rameters with the smallest absolute values, by comparing different werameter-selettion strate- gies, including selecting the largest parameters and random parameters. We conduct downstream dataset finetuning experiments using the Expedi- tion dataset comparing six ablated models: 1) model without PPA and Lrank: 2) model with PPA but without Crank, 3) model with Lrank but without PPA, 4) model with both PPA and Lrank (Ours), 5) model fine-tuned with the largest absolute values parameters, and 6) model fine-tuned with randomly selected parameters. The quantitative metric results are presented in Tab. 2 /) The model without both the PPA and Lrank results in a poor FID and low CLIP score. 2) Introduc- ing PPA improves the FID but decreases the CLIP score, indicating its effectiveness in learning task-specific knowledge. 3) Incorporating Crank helps achieve a better CLIP score, but results in worse FID, indicating its effectivenes in better preserving the model prior knowledge, but with loss of task-specific information. 4) Regarding parameter-sele selection strategies, fine-tuning the largest absolute values parameters yields a relatively good FID but the worst CLIP score, suggesting that fine-tuning the most effective parameters severely disrupts the model's prior knowledge and leads to worse content-text consistency. 5) Moreover, fine-tuning randomly selected parameters results in both poor FID and CLIP scores, indicating randomly selecting parameters to finetune is unable to learn task-specific knowledge and preserve the model's prior. 6) In contrast, our model achieves the best VLHI, validating its effectiveness in both fitting capability and prior preservation. More analysis of the hyperparameters is presented in the appendix.
Method
FID
CLIP Score 1
VLHI 1
wlo. PPA &Crask PPA, wlo. Lrank C.asé, w/o. PPA
134.75
27.01
1.16
130.95
26,66
1.56
135.31
27.12
PPA &Crank (Ours)
131,56
27.34
1.79
Tuning Largest Parameters
130.55
25.42
1,00
Tuning Random Parameters
133.57
26.58
0.97
Table 2: Ablation studies on six ablated models.
6 CONCLUSION
In this paper, we propose SaRA, novel parameter-efficient fine-tuning method, which makes full use of the ineffective parameters with the smallest absolute values in the pre-trained model. We propose nuclear bose- low-rank loss to constrain the rank of the learned sparse matrices, thereby avoiding model overfitting. Moreover, we design a progressive parameter adjustment strat- egy, which can further improve the effectiveness of the fine-tuned parameters. Finally, we propose a novel unstructural backpropagation method, largely saving the memory cost during parameter fine-tuning, which can also reduce the memory costs for other selective PEFT methods. Extensive experiments demonstrate the effectiveness of our method, which achieves the best fitting capability while keeping the prior information of the pre-trained model well.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by National Natural Science Foundation of China (No. Shanghai Sailing Program (22YF1420300), Young Elite Scientists Sponsorship Program by CAST (2022QNRC001), the Fundamental Research Funds for the Central Universities (YG2023QNB17, YG2024QNA44), Beijing Natural Science Foundation (L222117).
REFERENCES
Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the ef- fectiveness of language model fine-tuning. arXiv preprint arXiv:2012.13255, 2020.
Alan Ansell, Edoardo Maria Ponti, Anna Korhonen, and Ivan Vulié. Composable sparse fine-tuning for cross-lingual transfer. arXiv preprint arXiv:21 2021.
Alan Ansell, Ivan Vulić, Hannah Sterz, Anna Korhonen, and Edoardo M Ponti. Scaling sparse fine-tuning to large language models. arXiv preprint arXiv:2401. 16405, 2024.
Kartikeya Bhardwaj, Nilesh Prasad Pandey, Sweta Priyadarshi, Viswanath Ganapathy, Rafael Esteves, Shreya Kadambi, Shubhankar Borse, Paul Whatmough, Risheck Garrepalli, Mart Van Baalen, et al. Sparse high rank adapters. In NIPS, 2024.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhor, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion mod- els. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 22563-22575 2023.
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 35:16664-16678, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. Sparse low-rank adaptation of pre-trained language models. arXiv preprint arXiv:2311. 1.ii::211.111990 16666 2023.
Ali Edalati, Marzich Tahaci, Ivan Kobyzev, Vahid Partovi Nia, James J Clark, and Mehdi Rezagholizadeh. Krona: Parameter efficient tuning with kronecker adapter. arXiv preprint arXiv:2212. 10650, 2022.
Zhengyi Fang, Yue Wang, Ran Yi, and Lizhuang Ma. Dropout mixture low-rank adaptation for visual parameters-efficient fine-tuning. In European Conference on Computer Vision. Springer, 2024.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: 1003:000330333 2018.
Demi Guo, Alexander M Rush, and Yoon Kim. Parfmeter-efficient transfer learning with diff prun- ing. arXiv preprini arXiv:2012.07466; 2020.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023.
Zeyu Han, Chao Gao, Jinyang Liu, Sai Qian Zhang, et al. Parameter-efffiient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403. /4608, 2024.
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lorat: Efficient low rank adaptation of large models. arXiv preprint 202. 2024.
11

---

Published as a conference paper at ICLR 2025
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficient transfer learning for nlp. In International conference on machine learning, pp. 2790-2799. PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 12Xiv:2/06009688 2021.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint arXiv: 10196, 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 4401-4410, 2019.
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pultern Recognition, pp. 6007-6017, 2023.
Tao Lei, Junwen Bai, Siddhartha Brahma, Joshua Ainslie, Kenton Lee, Yanqi Zhou, Nan Du Vincent Zhao, Yuexin Wu, Bo Li, et al. Conditional adapters: Parameter-officient transfer learning with fast inference. Advances in Neural Information Processing Systems, 36:8152-8172, 2023.
Tailin Liang, John Glossner, Lei Wang, Shaobo Shi, and Xiaotong Zhang, Pruming and quantization for deep neural network acceleration: A survey. Neurocomputingg 461:370- 403, 2021.
Baohao Liao, Yan Meng, and Christof Monz. Parameter-efffiient fine-tuning without introducing new latency. arXiv preprint arXiv:2305. 16742, 2023.
Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter- efficient multi-task fine-tuning for transformers via shared hypernetworks. arXiv preprint arXiv:2/06.0445 04489, 2021.
Chong Mou, Xintao Wang. Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 4296- 4304, 2024.
Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. Lisa: Lay- erwise importance sampling for memory-eficiintt large language model fine-tuning. Advances in Neural Information Processing Systems, 2024.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapter- fusion: Non-destructive task composition for transfer learning, arXiv preprint arXiv:2005.00247 00247, 2020.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. /4988, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763, PMLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
12

---

Published as a conference paper at ICLR 2025
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuming text-to- eext-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 22500- 22510, 2023.
Jingxiang Sun, Bo Zhang, Ruizhi Shao, Lizhen Wang, Wen Liu, Zhenda Xie, and Yebin Liu. Dreamcraft3d: Hierarchical 3d generation with bootstrapped diffusion prior. arXiv preprint arXiv:2310.16818, 2023.
Yi-Lin Sung, Varun Nair, and Colin A Raffel. Training neural networks with fixed sparse masks. Advances in Neural Information Processing Systems, 34:24193-24205, 2021.
Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. Dylora: Parameter effi cient tuning of pre-trained models using dynamic search-free low-rank adaptation. arXiv preprint arXiv:2210.07558, 2022.
Thanh Van Le, Hao Phung, Thuan Hoang Nguyen, Quan Dao, Ngoc N Tran, and Anh Tran. Anti- dreambooth: Protecting users from personalized text-to-image synthesis. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2116-21221 2023.
Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. Adamix: Mixture-of-adaptation for parameter-efficienl model tuning. arXiv preprint 2tlv:2205./24100 2022.
G Alistair Watson. Characterization of the subdifferential of some matrix norms. Linear Algebra Appl, 170(1):33-45, 1992.
Adam X Yang, Maxime Robeyns, Xi Wang, and Laurence Aitchison. Bayesian low-rank adaptation for large language models. arXiv preprint arXiv:2308. 13/11, 2023.
Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.06722 2023.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836-3847, 2023a.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In International Con- ference on Learning Representations. Openreview, 2023b.
13


Reviews:
[{"review_id": "z3j0zHGYd3", "reviewer": "Reviewer_RosU", "summary": "The paper presents a novel approach for fine-tuning pre-trained diffusion models called SaRA for visual content generation. The method builds on a key insight: parameters with the smallest absolute values in diffusion models contribute minimally to generation due to training instabilities, allowing for their selective reuse. SaRA enhances these low-impact parameters by applying a sparse weight matrix that learns task-specific knowledge while retaining the model’s generalization abilities. To avoid overfitting, the authors introduce a nuclear-norm-based low-rank training scheme. Additionally, SaRA includes a progressive parameter adjustment strategy and an unstructured backpropagation approach to efficiently manage memory use during fine-tuning.\n\n\nNote: the supplementary materials contain the author's username and IP address (Supplementary Material/code/.idea/deployment.xml)", "questions": "see the weaknesses section", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method visualizations (Figures 1 and 4) provide a step-by-step comparison of the proposed approach with previous techniques that helps the reader to easily understand the nuances of SaRA.\n2. The authors show that the method could be applied to different diffusion models denoisers architectures U-Net (SD1.5 and 2.0) and Diffusion Transformer (SD3.0).\n3. The design of the method allows a plug-and-play experience for the users that is highly beneficial for practical adoption.\n4. The authors demonstrate the capabilities of the proposed approach on various widely-used by the open source community datasets.", "weaknesses": "1. The authors introduce a novel Visual-Linguistic Harmony Index (VLHI) metric; however, it's described only in the appendix.\n2. No comparison with the recent SOTA PEFT techniques (e.g., DoRA [Liu et al. 2024] that is available for different models on mentioned in the paper CIVITAI).\n3. No ablation on scaling the trained SaRA weights for the inference (as the lora_scale parameter controls the influence of LoRA weights) or mentioning it in the limitations.\n4. The authors say that for the FID computation they sampled 5K images from the source and generated data; however, BarbieCore dataset has only 315 images which is definitely not enough for the proper FID evaluation. The details about the sizes of the used datasets should be in the paper.\n5.  CLIP L/14 used by authors is trained to provide overall image captions and could miss the details. The visual language models-based evaluations used in T2I-CompBench++[Huang et al. 2023] could be more accurate.\n6. The authors skip the most popular Stable Diffusion XL 1.0 version, whereas they include 2.0.\n7. Typographical mistakes such as:\n*) Table 1: wrong column 2&3 names\n*) Figure 1: addictive-> additive", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZHN78pT9jt", "reviewer": "Reviewer_EfNM", "summary": "To leverage the unimportant parameters concept in model pruning, this paper proposes a model fine-tuning method for diffusion model by reusing these ineffective parameters. The authors find that such ineffective parameters with small absolute values are random and dynamically change over finetuning. Based on this observation, they further design some efficient strategies for tuning these parameters.", "questions": "How to understand the better performance after setting some parameters to 0 in Section 3.1? This is actually interesting and may be useful for understanding the behavior of diffusion model.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This paper verifies that the ineffective parameter concept also applies for diffusion model, i.e., the parameters with small absolute values are not important for the generation. \n2. This paper adopts a series of strategies for the efficient tuning of these ineffective parameters.\n3. The effectiveness of this method is verified on the stable diffusion series. It demonstrates better performance over the baselines.", "weaknesses": "1. The main limitation is that this paper poorly extends the concept in static model to the finetuning area, which is a dynamic model. Specifically, the ineffective concept works in model pruning, which is a given fixed model. The ineffective parameters in such static model can be discarded or reused. However, in this paper, the model parameters dynamically change during finetuning, and unimportant parameters also change. Thus, a right way to extend such unimportant parameters is to study their dynamics over change, instead of simple reuse.  Simple reuse may have several issues, for example, smaller optimal parameter search space in the finetuning case.\n2. Another concern may lie in how to merge multi tasks’ parameters, and their merging performance. The multi-task parameter merging is a good property of LORA. It is encouraged to explain and verify this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QAd0sRlgGZ", "reviewer": "Reviewer_GkU7", "summary": "The paper presents a new method for fine-tuning diffusion models by training only low-value parameters making them effective on a new task. Additionally, a nuclear norm is used to prevent overfitting, and efficient selective backpropagation and progressive parameter adjustment reduce the memory and time requirements during training. The results show SaRA is on par or better than other fine-tuning methods in terms of FID, CLIP score, and qualitative assessment.", "questions": "1. The authors should consider splitting Figures 2 b) and 5 into more subplots. In my opinion, they are too cluttered and it takes too much time to read them.\n2. In Table 1, the use of \"optimal\" and \"sub-optimal\" is not correct, e.g. optimal FID is 0. \"Best\" and \"second best\" or something similar should be used.\n3. Could the authors provide more qualitative results for the same prompts (Appendix C) with different seeds to see the diversity of the samples?\n4. It would be good to include some of the visual results in the main text.\n5. Can authors elaborate on how CLIP score is related to overfitting?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper is very well written and easy to follow. It has a good flow of information. Every claim is carefully explained and proven by experiments or analysis.\n2. The novelty of the method is good. The idea of fine-tuning only ineffective parameters was explored before but combined with nuclear norm regularization, novel approach to backpropagation of a sparse matrices, and adaptive fine-tuning, creates a valuable addition to the field.\n3. The experiments are extensive, both comparisons and ablation study.\n4. Qualitative results suggest an improvement over other methods.\n5. Quantitative results show the model behaves better or similar to other methods. I can see it becoming one of the methods of choice depending on one's needs.", "weaknesses": "1. Minor - as mentioned in Strengths 5., the results are not showing overall superiority over other methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SefRzQjBIP", "reviewer": "Reviewer_bh95", "summary": "The paper proposes a PEFT method for diffusion models, which progressively trains selectively chosen parameters with small, inefficient values. To prevent memory waste, it avoids storing the gradient of all parameters and instead stores only the chosen parameters in separate nodes, which are then replaced after training.", "questions": "- Would similar results be achievable if other SFT methods stored the selected parameters in leaf nodes and updated them as implemented here?\n- Could you provide more explanation regarding the impact of training process randomness on initially inefficient parameters becoming efficient?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Unlike methods like LoRA that require additional parameters, this approach selects parameters within the existing model for fine-tuning, minimizing additional memory usage.\n- Existing selective PEFT methods continuously update masks, which requires storing gradients for all parameters, making them inefficient. In contrast, this paper’s method progressively trains only the fixed parameters at each stage, storing only certain gradients, making it more memory-efficient.\n- The paper conducts extensive comparison experiments with various versions of Stable Diffusion (SD) and different sizes of fine-tuning parameters.", "weaknesses": "- Although the paper differentiates itself from selective fine-tuning, I think this method still appears to be a form of selective fine-tuning. The memory efficiency improvement seems to stem from implementing a separate node for gradient storage and selectively fine-tuning only those nodes, rather than from an inherent algorithmic difference.\n- Selecting parameters based on a specific threshold seems not a new concept. For example, the related work PaFi is described that it also trains based on absolute values.\n- The reasoning behind inefficient parameters becoming efficient during training due to the randomness of the training process is unclear. Since the initial weights are set randomly, many values are likely to change through training. There is no strong basis to assume a correlation with the initial values. Even parameters with initially small values are expected to converge toward the average distribution, making Figure 3 somewhat self-evident.\n- Although selecting a mask based on a threshold is computationally efficient, this method is relatively naive compared to other SFT methods that dynamically choose parameters during fine-tuning, which may lead to lower performance. Although it was compared with LT-SFT, there is a lack of comparison with other SFT methods.\n- In the table, it is unclear if this method offers a significant performance improvement. While the FID scores are mostly favorable, the CLIP scores appear to be higher for LoRA. Additionally, the L_rank removal in the ablation study does not lead to a significant performance change.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BbrHlbFD9G", "reviewer": "Reviewer_GTZk", "summary": "This paper presents SaRA, a method for fine-tuning pre-trained diffusion models that introduces progressive Sparse low-Rank Adaptation (SaRA) to enhance efficiency and reduce memory costs in adapting diffusion models to new tasks. The proposed method leverages parameters with low absolute values, presumed to have limited initial impact on the model’s performance, making them suitable for fine-tuning. SaRA combines sparse parameter updates with a nuclear norm-based low-rank constraint to mitigate overfitting. It also introduces a progressive parameter adjustment and unstructured backpropagation strategy, aimed at further memory efficiency. Extensive experiments demonstrate SaRA's superiority over traditional fine-tuning methods on image generation and customization tasks.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper clearly explains the intuition in parameter efficiency by proposing the use of low absolute value parameters for adaptive updates, effectively avoiding overfitting through a nuclear norm constraint.  The progressive parameter adjustment strategy positively contributes to the stability and convergence of model performance, while the unstructured backpropagation method effectively reduces memory costs, making SaRA a practical solution in resource-constrained environments. Additionally, the extensive experiments cover various tasks, such as image generation and customization, thoroughly validating the advantages of the SaRA method in balancing the preservation of model priors and the learning of task-specific knowledge.", "weaknesses": "My main concern centers on the assumption underlying this approach—that parameters with the smallest absolute values are inherently ineffective. This seems more empirical than rigorously substantiated. While small absolute values can indeed correlate with lower impact in certain contexts, particularly in pruning, their effectiveness actually depends on the model architecture and specific task. Small value parameters may exert less direct influence on the output, but they are not intrinsically ineffective; their impact can vary depending on training dynamics, model structure, and optimization objectives.\n\nSome minor weaknesses include:\n* The generalizability of the 'adapting small-value parameters' strategy across different architectures is crucial to ensure broader applicability. This paper only investigates the phenomenon of the pre-trained Stable diffusion models. In this sense, I'm worrying about whether it can be applied across different network architectures or frameworks.\n* In the caption of Figure 3, weight distributions are claimed to be Gaussian without further clarification, which seems empirical rather than solid.\n* The choice of threshold for selecting low-absolute-value parameters could be better justified. If this choice is sensitive, it could limit SaRA's robustness and generalizability across different diffusion models and tasks. An ablation study on this threshold choice would strengthen the claims.\n* While the experiments show SaRA's success, the paper could benefit from an analysis of its limitations. For instance, discussing cases where SaRA may not perform as well, such as tasks requiring extensive re-training of high-impact parameters, would improve the comprehensiveness of the evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)