Large Convolutional Model Tuning via Filter Subspace
Authors: Wei Chen
Zichen Miao
Qiang Qiu
Venue: ICLR 2025 Poster (2025)
Keywords: Efficient Fine-tuning
Filter Decomposition
Filter Subspace
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Efficient fine-tuning methods are critical to address the high computational and parameter complexity while adapting large pre-trained models to downstream tasks.
Our study is inspired by prior research that represents each convolution filter as a linear combination of a small set of filter subspace elements, referred to as filter atoms. In this paper, we propose to fine-tune pre-trained models by adjusting only filter atoms, which are responsible for spatial-only convolution, while preserving spatially-invariant channel combination knowledge in atom coefficients.
In this way, we bring a new filter subspace view for model tuning. 
Furthermore, each filter atom can be recursively decomposed as a combination of another set of atoms, which naturally expands the number of tunable parameters in the filter subspace.
By only adapting filter atoms constructed by a small number of parameters, while maintaining the rest of model parameters constant, the proposed approach is highly parameter-efficient. It effectively preserves the capabilities of pre-trained models and prevents overfitting to downstream tasks. 
Extensive experiments show that such a simple scheme surpasses previous tuning baselines for both discriminate and generative tasks.

Published as a conference paper at ICLR 2025
LARGE CONVOLUTIONAL MODEL TUNING VIA FILTER SUBSPACE
Wei Chen, Zichen Miao, Qiang Qiu
Purdue University, IN, USA
(chen2732, miaoz, qqiu)epurdue.edd
ABSTRACT
Efficient fine-tuning methods are critical to address the high computational and pa- rameter complexity while adapting large pre-trained models to downstream tasks. Our study is inspired by prior research that represents each convolution filter a linear combination of a small set of filter subspace elements, referred to as fil- ter atoms. In this paper, we propose fine-tune pre-trained models by adjusting only filter atoms, which are responsible for spatial-only convolution, while pre- serving spatially-invariant channel combination knowledge in atom coefficients. In this way, we bring new filter subspace view for model tuning. Furthermore, each filter atom can be recursively decomposed as a combination of another set of atoms, which naturally expands the number of tunable parameters in the filter subspace. By only adapting filter atoms constructed by a small number of pa- rameters, while maintaining the rest of model parameters constant, the proposed approach is highly parameter-efffcient. pre-trained models and prevents overfitting to downstream tasks. Extensive periments show that such a simple scheme surpasses previous tuning baselines for both discriminate and generative tasks]
effectively preserves the capabilities
INTRODUCTION
Large models have demonstrated exceptional performance across diverse domains and tasks (Brown et al. 2020
Dosovitskiy et al. 2020 He et al. [016 Kirillov et al. 2023 Radford et al.) Rombach
jtt al. 2022 Silver et al. 2016 Touvron et al. 2023] Vaswani et al. 2017), attributing to their capability to effectively represent complex patterns and relationships (Khan et al.] (020) by pre-
training on massive datasets (Russakovsky et al. 2015 Raffel et al. 2020; Zhu et al.) 2015). A common strategy to adapt these large models for specific downstream tasks is fine-tuning them with full parameters. But this method presents two main challenges: (1) Adjusting vast number of parameters for particular target tasks is computationally intensive; (2) The limited availability of target data increases the risk of overfitting (Lian et al.) 2022).
To address these challenges, researchers have developed parameter-efficien methods (Chen al. 2022 Hu et al. 12021] Jia et al., (2022] Shen et al. [2021] YEH et al. 2023 Zaken et al. 2022) by fine- tuning the pre-trained models with only a minimal number of parameters. Among these methods, LoRA (Hu et al. 2021) fine-tunes models without altering the model architecture, becoming notably popular for its efficacy. However, LoRA still risks overfitting when fine-tuned on limited data and compromising the generalization capability of large models. For instance, Figure [ illustrates that with only 5 training samples, LoRA tends to produce images that closely resemble the training data, compromising the ability for diverse image generation, compared with pre-trained models.
Motivation. To preserve the capabilities of pre-trained models when fine-tuning them on the downstream tasks, one prominent approach in continual learning (Parisi al.) 2019 Rusu et al. 2016 Yoon et al., (219)) is to formulate convolution filters in ConvNets as a linear combination of filter atoms (Li al. 2019 Papyan et al. 2017 Qiu et al. 2018a) and fine-tuning only filter atoms (Miao et al. 2021a Zhai et al.; 2021). I. Specifically, filters in each convolutional layer are
decomposed over small set of filter subspace elements, referred to as filter atoms, responsible for
Our code is available htt.:e//ggtthbb...c//eeeeeeeeeee
finetune

---

Published as a conference paper at ICLR 2025
reo-trained
Ingut Images
LoRA
Fidelity
Diversity
Tex2lmaae Alignment
Pre-traineed
Pre-trained
Pre-trained
LoRA
LoRA
LoRA
Ours
Ours
Ours
Figure 1: Compared with LoRA (Hu et al. 2021 method updates small set of parameters and mitigates the risk of overfitting the target concept. In this example task learning the concept (castle) from 5 input images, we only require fine-tuning 0.75M parameters, stark reduction from the 22.67M parameters required by LoRA. Furthermore, the model fine-tuned by our method captures the target concept while ensuring a rich diversity and strong alignment with input text prompts. It demonstrates that approach preserves the generalization capability of large models. The text prompts used generate images from left right "The (castle) stands against backdrop snow-capped mountains". (castle) surrounded by lush, vibrant forest", "The (castle) overlooks a screne lake", and "The (castle) in the autumn season with colorful foliage".
spatial-only convolution. Each convolutional layer is now constructed as linear combinations of filter atoms using decomposition coefficients, referred to atom coefficients, responsible for the spatially invariant channel combination. Hypothesizing variations across tasks can be reduced by bridging spatial discrepancies
images, we propose to calibrate the pre-trained model by solely fine-tuning the spatial-only filter atoms while preserving the spatially- -intarian channel weights, i.e., atom coefficients.
In work, we demonstrate that fine-tuning large model via filter atoms is substantially effec- tive and parameter-efficient as filter atoms are responsible for spatial-only convolution and usually comprise only few hundred parameters. This strategy is in harmony with task subspace modeling principles, which suggest that task parameters occupy a low-dimensional subspace, allowing tasks to be represented combinations of latent basis tasks (Evgeniou & Pontil 2007 Kumar & Daume III] 2012 Maurer et al. 2013 Romera- maintaining fixed atom coefficients, i.e. spatially-invarianl invariant channel mixing weights, plays a crucial role in preserving the generalization capability of pre-trained large models.
et al., 2013] [203]] & Yang 2021). We also discover that
With large number of parameters fixed, fine-tuning only tiny set of parameters in filter atoms is potentially challenging to adapt to more complex tasks. We further demonstrate a simple yet effective way expand the tunable parameters in filter subspace, without any modification on atom coefficients, by decomposing each filter atom over another set of filter atoms. This process provides an overcomplete set of filter atoms expands the tunable parameter space, all while still requiring fewer parameters than LoRA. Additionally, we provide a simple technique extend this method to
linear layers, ensuring alignment with the characteristics in prior literature (Chen et al. 2023 Cheng
et al. 2021b Li al. 2019 Miao et al. 2021a b Papyan et al. 2017: Qiu et al. 2018a 8a Wang al.
2021a Wang 2023
Our method is illuttatedd in Figure 2
We demonstrate the effectiveness of our approach both discriminative and generative tasks with ResNet50 (He et al., 2016). ConvNeXt (Liu et al., 2022) and Stable Diffusion (Rombach et al. 2022
We summarize contributions follows,
We propose method by adapting only filter subspace elements (filter atoms), with few hundred parameters, to achieve significantly parameter-efficient fine-tuning.
We observe that maintaining fixed atom coefficients plays crucial role in preserving the generalization capability of large models.
We further demonstrate simple way expand the number of tunable parameters in fil- ter subspace by recursively decomposing each filter atom over another set of filter atoms, which extends the parameter space for tuning.
We conduct extensive experiments demonstrating the efficacy of our approach on discrim- inative and generative tasks for fine-tuning large models.
2

---

Published as a conference paper at ICLR 2025
2 PRELIMINARY
2.1 LOW-RANK ADAPTATION FOR FINE-TUNING
LoRA (Hu et al. 2021 introduces trainable low-rank matrices into layers approximate weight updates. For a pre-trained weight matrix W E Re'xe xe, LoRA represents the weight update with low-rank decomposition
W WWAW- W down W up
where W Wown E R'' and Wup up E Xx updates, d' and are the size of inputs and outputs. is usually much smaller than c' and c. A learnable scalar hyperparameter a scales the weight updates, which leads to W + a a W Wup.
xe are tunable parameters, is the intrinsic rank of weight
2.2 SPARSE CODING AND MATRIX FACTORIZATION
Sparse coding attempts find the representation of input w with respect dictionary of atoms [d with the fewest number of coefficients a', i.e., w = Ei=1 a d₁. The objective of a sparse coding problem can be written as
min and .d
Vlw Σ
4 di + > |a
where I|-I|ı the Lj norm, and is Lagrange multiplier. It can be further expressed in a tensor form,
min
F a DII: A| D.D
where F IR'd' xec''' xk is the input tensor, a E Re' xexm is the tensor of coefficients and D mmx xk: is a tensor of basis. Several algorithms have been developed to solve D. such as fast iterative soft-thresholding algorithm (FISTA) (Beck Teboulle, 2009). matching pursuit (Mallat & Zhang, 1993). and least absolute shrinkage and selection operator (LASSO) (Santosa & Symes) 1986). We can interpret the first term in (T) as finding a suitable tensor factorization for J, while the second term serves penalty enforcing sparsity in the representation of F.
3 METHODS
In this section, we decompose convolution filters over a small set of filter subspace elements, referred as fitler atoms. This formulation enables a new model tuning method via filter subspace by solely adjusting filter atoms.
3.1 FORMULATION OF FILTER DECOMPOSITION
Our approach involves decomposing each convolutional layer F into two standard convolutional layers: filter atom layer D that models filter subspace and an atom coefficient layer a with 1 x filters that represent combination rules of filter atoms, as displayed in Figure (a). This formulation written
F=xxD,
where is the tensor product.
F E e xexkxk contains e' c filters (Fi) H where and c are the numbers of input and output channels, & is the kernel size. D E Rmxxk is set of m filter atoms, i.e., D {d, )m where d, k Each filter Fiù is linear combination of filter atoms, calculated by Fi En a adddd d,, a haid represents each element in a Re' xexm
Considering input features X E IRC xw' and output features Z cx where ht w/ and h x w represent the dimension of input features and output features. The convolution process now can be comprehended the following two steps:
The filter subspace is span of filter atoms D.
3

---

Published as a conference paper at ICLR 2025
D Bunkek
R
T
AF
AF
Input Features ge xh xw'
Output Features
( kmm)xxx
R
AF a AD
(b)
Figure 2: A filter subspace view of model tuning. (a) Each convolutional layer FF is constructed as linear combinations of filter subspace elements, i.e., filter atoms D, using decomposition coeffi- cients, ie.. atom coefficients a. Hypothesizing variations across tasks can be reduced by bridging spatial discrepancies the images, we propose calibrate the pre-trained model by solely fine- tuning the spatial-only D while keeping the spatially-ing spatially-invariant invariant channel weights a fixed. (b) The convolution operation represented in two stages: At the spatial-only convolution stage, each filter atom D is adapting to the target task, and then at the cross-channel mixing stage, Z' are combined into using fixed atom coefficients a, which is obtained from the pre-trained model. More details provided in Section 3
Spatial-only Convolution with D. Each channel of the input features X convolves with each filter atom separately to produce intermediate features
Z' = D X,
where Z' ( xm)xxxw the convolution operation.
For each input feature channel X' and filter atom d: R we have Z = X' dd, where Z Rhx X E R xw" are one feature channel in Z', X, and d; is one filter atom in D = [d:
This process leads to the generation of m distinct intermediate output channels for each input chan- nel, which is illustrated in Figure 2 (b). In this step, filter atoms focus only on handling the spatial information of input features, and cross-channel mixing is postponed to the next step.
Cross-channel Mixing with a. Subsequently, atom coefficients weigh and linearly combine the intermediate features to produce output features
= Z'
Each output feature channel z E R is linearly combined from c' x m intermediate feature channels z E gax with the coefficient {a- im Imom which is, Z
2 2i=1 qi.....h.
Ji=l,i=1
Here, add represents each element atom coefficients
gdd xcxxm
The spatially invariant channel weights, atom coefficients a, serve as operators for channel mixing, functioning distinct combination rules that construct the output features from the elemental feature maps generated by the filter atoms. During the model tuning, a is obtained from the pre-trained model and remains unchanged, while only filter atoms D adapt to the target task.
Summary. The two-step convolution operation explains different functionalities of filter atoms D and atom coefficients a, which is, D only contribute to spatial convolution and a only perform cross-channel mixing. In practice, the convolution operation is still performed as one layer, without generating intermediate features, to avoid memory cost. In the fine-tuning process, solely adjust D. which contains a small set of number parameters, & xxk & c' xc, thereby facilitating parameter- efficient fine-tuning.
3.2 OVERCOMPLETE FILTER ATOMS
The parameters of filter atoms are extremely small compared with overall model parameters. For instance, the filter atoms constitute mere 0.004% of the total parameters in ResNet50 (He al.)
4

---

Published as a conference paper at ICLR 2025
( Mi
R sa)o0s
B KOOK
a
B
(cl
Input Features X F '''k'www" 2 B moxo,)show
Output Features
Dj RR
z' g (c)))))s 2. Bexbow
(a)
(b)
Figure 3: (a) The process of constructing overcomplete filter atoms: One filter atom in D is presented as a linear combination of my filter atoms in Dj. (b) The convolution operation is represented in three stages: At the spatial-only convolution stage, set of overcomplete filter atoms D₁ adapting to the target task and generates m m feature channels by convolving with each input channel. At the intra-channel mixing stage, every group of mi features linearly combined by coefficients / with no cross-channel mixing. At the cross-channel mixing stage, Z' are combined into Z using fixed atom coefficients a, which is obtained from the pre-trained model.
2016). To fully explore the potential of filter subspace fine-tuning, show next simple way to construct a set of overcomplete filter atoms by recursively applying the above decomposition to each filter atom, to expand the parameter space for fine-tuning needed.
Specifically, each filter atom d, can be further decomposed over mi number of filter atoms in 1
Dr.: = {d] Lm with its corresponding coefficients 3 [ , 3 E R" ie.. d = E3 d 1.1- Therefore, D1 E ((mmm)) xkxk contains m my filter atoms, leading to overcomplete set. The recursive decomposition process illustrated Figure
Considering input X E Ig''"k and output Z gexhxw, convolution using the overcomplete filter atoms be comprehended three steps. While the spatial-only convolution with D and cross-channel mixing with a steps are the same as before, it linearly combines the features channels corresponding to my decomposed filter atoms in intra-channel mixing step.
Intra-channel Mixing with J. After convolving X and D1 to get Z E R(e'mmxm)xxee the intermediate feature map Z' p( is obtained by linearly combining feature channels within the corresponding input channel via coefficients 3.
Z' = B x Z'
Each feature channel Z' calculated by Z' En 3' where 3&d is one element in B, and Zzd'J e g is one feature channel in Zj. This phase exclusively blends the feature channels within the same input channel, avoiding any mixing across different channels. The whole process illustrated Figure To adapt the model to downstream tasks, we fine-tune both ( and D,, which contain more tunable parameters than D.
This method can be readily adapted to X convolutional and linear layers as well. We start by
applying Kronecker decomposition
linear layer W A & B, where A R B Kk W E Re'xe is the Kronecker product. This approach has been explored by recent literature on fine-tuning (Edalati et al. 2022 YEH et al. 2023). The idea is to represent a matrix multiple blocks, where each block comes from the same basis B but varied by different constant
Aij. We then introduce m. pairs of A and B to reconstruct W, W >n A, X B, such that each block in W is represented as linear combination of all B,,
3.3 DECOMPOSITION
LINEAR LAYERS
3 As the size of filters is x k. m = 2 independent filter atoms is complete since every filter can be linearly combined by these filter atoms. As the number of filter atoms is larger than k², it becomes overcomplete. Overcompleteness can potentially bring more stable fine-tuning, and in our expand the number of parameters for adapting downstream tasks.
5

---

Published as a conference paper at ICLR 2025
2
ae E Be
We ve-irite the above formulation as the no- tation with coefficients and atoms such that Ore A and D. (Bi There- fore, W is constructed from me atoms in D, E R. xk= by the combination using coeffi-
AE RERE
B Babie
B
De E panniss
U2
cients Cee E ER Xm which is, W ac De. This formulation is mathematically equal to the decomposition filters as filter atoms and atom coefficients. Similarly, the x 1 con- volutional layer Fe F. xexlx1 can be for- mulated as a, and De with the same process,
W RC
4. (A
D. = (B
Figure 4: Formulate the linear layer W com- D
bination of atoms, W 0c
= X De.
(3)
The procedure of formulating linear layers coefficients and atoms illustrated in Figure During the model tuning, is obtained from the pre-trained model and remains unchanged, while only filter atoms De adapt to the target task.
3.4 PARAMETER EFFICIENT FINE-TUNING
Building on our filter subspace formulation, we investigate various parameter-efficient fine-tuning strategies for tuning the pre-trained model. For convolutional filters F, we directly sparse coding (1) to decompose them as atom coefficients and filter atoms a x D. For linear layers or x convolution layers, we first formulate them Fe (3) and then decompose them coefficients dee and atoms D. using (T). For tinettunig the models, we only adapt D or D.. while keeping a fixed. When more parameters are needed, we represent D as 3 and D incorporate overcomplete filter atoms. We initialize D2 by simply repeating each filter atom in D for times and initialize every element B3 with the value 1/m1- This enables us to fine-tune both B3 and D
In this study, we investigate three strategies that offer range of tunable parameters from small set slightly larger number: fine-tuning (1) only D, (2) D and D and (3) B, D, and D Downstream tasks are modeled by their updates AD, AB, AD, and AD,. In the Appendix B we present an analysis of the decomposition complexity and a comparison with baseline methods.
number of parameters
4 EXPERIMENTS
In this section, we begin with studying the effectiveness of our method across various configura- tions to determine the most suitable application scenario for each configuration. Subsequently, demonstrate that fine-tuning only filter atoms requires far fewer parameters while preserving the capacity of pre-trained models, compared with baseline methods the contexts of discriminative and generative tasks.
4.1 EXPERIMENTAL SETTINGS
Datasets. Our experimental evaluations are mainly conducted on the Visual Task Adaptation Benchmark (VTAB) (Zhai al., 2019), which contains distinct visual recognition tasks sourced from 16 datasets. As a subset of VTAB, VTAB-1k comprises mere 1, 0000 labeled training exam- ples each dataset.
Models. For the validation experiment, we choose ResNer50 (He et al. 2016) pre-trained on ImageNet-IK. For discriminative tasks, we choose the convolution-bass model, ConvNeXi-B (Liu al. 2022) pre-trained on ImageNet-21K K as the initialization for fine-tuning. For generative tasks, we choose Stable Diffusion (Rombach et al. 2022) which contains a downsampling- factor factor au- text encoder for the diffusion model. The model is pre-trained on the LAION dataset (Schuhmann et al., (022). which contains over billion image-text pairs. More details of pre-trained models listed in the Appendix A
toencoder with 860M UNet and CLIP VIT-L/14

---

Published as a conference paper at ICLR 2025
Table 1: Comparison among different configurations of our method.
Fine-tune D
Lincar Probe
LoRA
Full Finctuning
en 12
Accumey
55.4
78.6
83.3
66.25
66.86
68.68 0.21
Parum. (M) 4
0.2
2
25.6
0.21
0.21
Fine tune (9 and D;
Fine tune
D and D.
FT 99 D. and D. My 4, Ar
184
my
A. 2
Ne
Accuracy
78.69
78.7
78.9
75.03
79.83
8188
Param. (M) 4
0.82
1.2
0.23
0.62
2.1
4.2 VALIDATION EXPERIMENTS
In this section, we study the performance of our approach across various configurations.
Implementation details. We employ ResNet50 (He et al. 2016) pre-trained on ImageNet- 1K (Russakovsky et al. 2013) and fine-tune on CIFAR-10 110
(Krizhevsky Hinton 2009) for
50 epochs. We utilize the Adam optimizer (Kingma& & Ba 2015) with a learning rate of 0.001 and a weight decay of x 10-4 4 and a batch size 256.
We compare different configures of our methods with the following experimental settings and o, Oi are always fixed: (1) When fine-tuning D, we experiment with different numbers of filter atoms m in the range of [6, 9, 12). (2) When fine-tuning D and D... we choose (kg, ke) {(2, 2), (4, 4) }. and the number of filter atoms is m = 9. (3) When fine-tuning D and B, we choose m = 9, and my in the range of [3, 4, 5]. (4) When fine-tuning D.. D1 and G, we choose (Kes ki)) = (2.
9, m 3 and
The effectiveness of filter subspace. The results of all configurations are displayed in Table T with m = 9 as an example, and m = 6, 12 are detailed in the Appendix C (1) While fine-tuning D
with fixed a. our approach requires a negligible increase in the number parameters, amounting
to only 0.5% additional parameters, but it achieves an almost 20% improvement in accuracy when compared to linear probing. (2) Fine-tuning B and D further improves accuracy from 66.8% to 78.7% compared to fine-tuning D alone. It validates that overcomplete filter atoms provide more capacities for model tuning. (3) Fine-tuning D and D. results significant improvements com- pared to fine-tuning D alone. For example, with k. = 2, fine-tuning 0.02M additional parameters improves accuracy from 66.86% to 75.03%. (4) Fine-tuning D D1, and ,3 achieves the highest accuracy but it also requires the greatest number of parameters among all of our configurations.
Fine-tuning the filter atoms D or atoms D. enhances accuracy while keeping the parameter increase minimal, making highly appropriate for scenarios where the number of pa- rameters is a critical consideration. However, increasing the number of atoms does not easily yield further accuracy improvements. To achieve additional accuracy gains, incorporating the set of over- complete filter atoms D and their coefficients B can be effective, but at the cost of an increase in parameters.
Discussion.
4.3 GENERATIVE TASKS
Baselines. We compare our method to 8 baseline fine-tuning approaches: (i) Full fine-tuning, which entails updating all model parameters during the fine-tuning process; (ii) LoRA (Hu et al.) 202T), involving the introduction of a low-rank structure of accumulated gradient update by decom- posing it up- ap-projection and down pwoje-rooection.. (iii) LoHa (YEH et al. 2023) utilizes the Hadamard product across two sets of low-rank decompositions to elevate the rank of the resultant matrix and reduce the approximation error. (iv) LoKr (YEH al. 2023) introduces the Kronecker product for matrix decomposition to reduce the tunable parameters. (v) BitFit (Zaken et al.) (022) fine-tunes the
bias term of each layer. (vi) DiffFit (Xie al. 2023) fine-tunes the bias term. as well as the layer norm and the scale factor of each layer. (vii) OFT and COFT (Qiu et al. 2023) adapts the diagonal blocks of weight matrices to achieve orthogonal fine-tuning.
Implementation details. We fine-tune Stable Diffusion using the AdamW optimizer (Loshchilov) & Hutter 2018) with a learning rate of x 10-6 6 for full parameter fine-tuning, and x 10 for all other parameter-efficien fine-tuning methods. For few-shot learning, adopt 30 different customized concepts from Dreambooth (Ruiz et al., (023) and fine-tune the model on 4 ~ 9 images.
We utilize 25 different text prompts following (Qiu et al. 2023 Ruiz al. 2023) and generate
7

---

Published as a conference paper at ICLR 2025
(a) Input Images
(b) No Fine-tuning
(c) BitFit
(d) Ours, only
(e) LoRA
(f) LoHHa
(g) Full Fine-tuning
(h) Ours.
Figure 5: Fine-tune Stable Diffusion (Rombach et al. 2022 to learn the concept (castle) from (a) and generate images using text prompt: "A peacock in front of the (castle)". (b) Images generated by the pre-trained model, without any fine-tuning, align well with the text prompt but fail to match the target concept. (d) When only fine-tuning the filter atoms D, our approach achieves good alignment with the text prompt and the target concept. (h) However, fine-tuning the spatially-invarian channel weights, Le., atom coefficients a, results in generated images only align with the target concept, compromising the capability of the pre-trained model of producing diverse images that align with the text prompt. This issue is also observed in (e-g).
Table 2: Evaluate different approaches in learning the customized concept.
Pre-trained
Full
LoRA
LoHta
LoKr
OFT
COFT
DiffFi
BitFit
C,
C₁
C
C₁
Fidelity
0.144
0.711
0.697
0.693
0.693
0.656
0.652
0.622
0.571
0.594
0.652
0.707
0,716
Diversity 1
10.08
T21 Alignment
0.332
0.234
0.232
0.216
0.238
0.267
0.264
0.268
0.277
0.301
0.279
0.236
0.201
Param. (M)
22.67
8.47
1.06
11.75
11.75
0.34
0.75
587
images with shape of 512 x 512 for each concept using these prompts. We follow the method in YEH et al. (2023) and fine-tune the model for 1000 steps, and evaluate the fidelity, diversity, and text-to-image alignment of generated images.
We assess the fidelity as the average cosine similarity between DINOv2 embeddings (Oquab et al. 2023) of the generated and dataset images. The diversity of generated images is measured by the Vendi score (Friedman Dieng 2022) calculated with the DINOy2 embeddings. The alignment between generated images and corresponding prompts is measured via average cosine similarity in the CLIP feature space (Radford et al. 2021)
For the VTAB dataset, fine-tune /3, D1, and D. of the model for 5000 steps with the image shape of 256 x 256 and utilize Frechet Inception Distance (FID) (Heusel et al.) 2017) as quantitative metric for evaluation. To calculate FID, we generate 20,000 images from each checkpoint and compare them with images from the corresponding dataset. In cases where the dataset contains more than 20,000 images, we sample 20,000 images for comparison. See the Appendix for details of experimental settings.
Few-shot generative transfer learning. We report the evaluation of all methods Table 2 Our method with different configurations is denoted as C₁ C4. (1) Ci and C2 represent fine-tuning D and D. of the model with (m, ke) E {(6, 4), (9. 16)} (2) Css fine-tunes B. D2 and D with (m, kn, m,) = (9. 16,3), 3). (3) C4 is similar to C2 but it fine-tunes together with D and D
Compared with other baseline methods, our method with configuration C, C3 generate diverse images that are well-aligned with the text prompt. It means they preserve the capability of the pre-
8

---

Published as a conference paper at ICLR 2025
Table 3: FIDs (lower the better) of image generation models on VTAB benchmark with Stable Diffusion pre-trained on LAION.
Sprcialized
Potch Camebon
Calrchott
C3FAR
Rosbe45
88pmees
SOaINOR
Parana (M)
DTD
Man
113.3
45.5
212.
258.5
196.2
144.2
271.3
225.0
373.9
266.7
179.1
suning
39.0
33.8
42.9
107.6
144.5
54.3
69.0
61.3
800
LoRA
Ha al. 2001
36,8
42.4
40.6
45.3
2088
120.9
17.7
1012
81.9
61.5
69.5
33.6
32,1
66.4
69.4
21.6
57.5
22.67
Lotta
33.8
00.5
41.
41.5
33
140.9
17.3
111.
73.7
50.4
65.3
12.1
37.9
04.5
73.3
65.7
58.4
33.86
Lakr
TELT aia 2023
37.7
43.5
43.2
30.6
41.1
144.2
21.4
95.4
20.5
85.5
79.3
33.9
64.8
64.1
79.4
74.4
61.8
2.12
Oues
38.6
33.4
43.9
38.5
38.6
96.8
17.2
109.7
64.1
52.4
68.6
18.7
41.9
69.5
73.2
76.5
54.7
1.31
trained model while learning the new concept. However, C1 fine-tunes the atom coefficients a, resulting in the model overfitting to the target concept, which is reflected in notably high fidelity score, while compromising the model's ability to generate images aligned with the text prompt. This observation suggests that maintaining the spatially invariant channel weights a helps prevent overfitting when fine-tuning pre-trained models to downstream tasks. Figure [ examples of learning the concept "castle" from CustomConccept101 dataset (Kumari et al. 2023).
illustrates visual
Methods like LoRA (Hu et al., 2021) or full fine-tuning potentially update these a. thus, they lead lower diversity and text-to-imaee tomiioeee------o alignment in generated images. contrast, BitFit (Zaken al. 2022) and DiffFit (Xie et al. (022)) mostly fine-tune the bias, leaving a fixed, thus, they have higher diversity and text-to-image alignment than LoRA. However, they also keep the spatial operation D unchanged, resulting in lower fidelity score compared with Cz. More results can be found in Appendix
Performance comparisons on generative transfer learning. We report FIDs of models trained and evaluated on VTAB tasks in Table 3 In contrast to full parameter fine-tuning and LoRA, our ap- proach attains the lowest FID scores (54.7 v.s. 57.5) while employing the least number of fine-tuning parameters (1. 11. v.s. 22. 22.67M) Despite fine-tuning only 0.13% of the total model parameters, our method effectively tailors pre- pre-trained Stable Diffusion to align it with the desired target distribution.
4.4 DISCRIMINATIVE TASKS
In this section, we apply our method to the discriminative task, namely the classification VTAB- 1k (Zhai et al. 2019). We compare our method to baseline fine-tuning approaches: (i) Full fine- tuning, (ii) Linear probing, (iii) BitFit (Zaken et al. 2022). and (iv) LoRA (Hu et al. 202T.
Implementation details. Images are resized to 224 x 224, following the default settings in VTAB (Zhai et al., (201)). We employ the AdamW (Loshchilov & Hutter (018) optimizer fine- tune models for T00 epochs. The cosine decay strategy is adopted for the learning rate schedule, and the linear warm-up is used in the first 10 epochs. In this experiment, we fine-tune D and D. while keeping a and ac fixed, as this configuration delivers adequate accuracy without increasing parameters.
Performance comparisons on few-shot transfer learning. We compare the performance of our approach and other baseline methods, and the results on VTAB-1k 1k are shown in Table In these tables, the bold font shows the best accuracy of all methods and the underlined font shows the sec- ond best accuracy. Our method outperforms other parameter-efficient fine-tuning methods and even outperforms full fine-tuning. Specifically, our method obtains 6% improvement in accuracy com- pared to LoRA on the VTAB-Ik benchmark while utilizing significantly fewer trainable parameters (0.45M v.s. 17.4M). The Appendix also includes the experimental results for ViT-B/16.
RELATED WORKS
Pre-training and Fine-tuning. The standard practice of pre-training and fine-tuning (He et al. 2016 Huang et al.) 2017; Tan & Le 2019: Xie et al. 2017) entails models initially undergoing pre-training datasets such as ImageNet-22 2 BookCorpus, and Common Crawl (Russakovsky et al., 2013] Raffel et al. 2020 Zhu et al. 2013). Subsequently, these models are fine-tuned enhance their convergence and performance specific tasks (He et al. 2019). In the realm

---

Published as a conference paper at ICLR 2025
Table 4: Performance comparisons on the VTAB-IK benchmark with ConvNeXt models pre-trained on ImageNet-21K.
Speciallod
Stractand
Camelgon
SmHINORRN ORROO
SSMINORSSO
Caitechh III
GBAR00
Clesshoonn
Zeann (M)
Sun397
Restect
DMILab
KITTI
Full lice outing
94.9
66.2
13.6
59.5
50.8
82.6
37.7
86.6
85.]
35.0
73.6
73.3
61.3
52.1
83.0
96.8
61.1
32.7
38.8
72.14
$7.67
Lince Proking
92.3
65.8
26.8
09.3
02.7
46
35.4
18.4
0.11
BaFii
Zakener dl 2023
0.24
Lo8.
Dnes
94.8
71.7
76.9
99.6
93.1
67.1
57.5
85.1
98.6
87.6
748
30
62.8
90.3
M2.7
89.4
60.4
31.2
29
73.59
0.45
of parameter-efficient fine-tuning (Zhou et al. 2022). various approaches have been proposed. LoRA (Hu et al. (02T)) fine-tunes lower-rank matrices at each layer to represent weight updates. The adapter (Houlsby et al. 2019 approach inserts small modules between layers and reduces pa- rameters by only tuning these adapters (Chen et al. [022] Karimi Mahabadi et al., 2021
10211Li [i & Liang, 2021 Zaken al., 2022). Visual prompt tuning (VPT) (Jia et al., 2022; Sohn et al. (023) has intro- duced limited number of learnable parameters for optimization while keeping the backbone frozen.
SSF (Lian et al. 2022) proposes scaling and shifting deep features extracted by a pre-trained model.
Model Architectures. Compared with transformer-based models (Dosovitskiy et al. 2020
al. 2021; Touvron al., 2021 Yu et al. 2022), convolution has been used for long time as the main module to extract the image features in computer vision tasks. With an inductive prior, convolution-based models require fewer training images and computation resources achieve good
generalization. Convolution-banl based architectures have been largely studied (He et al. [016]] Liu
al. 2022 [Simonyan Zisserman 2015) and have found multiple applications, such as feature
extracting (Razavi et al. 2019), image generation (Karras et al.] 2020; [Song et al. 2021), super-
resoluton (Wang et al.) (020), and et cetera. Numerous studies explore the integration of convolu- tional techniques with vision transformers to enhance their performance (Guo al.] 2022 Raghu
ctt al., 202I). cerramtter-efficiin fine-tuning in downstream tasks is crucial and requires further examinations when utilizing pre-trained large-scale convolution-based models.
Discriminative and Generative Tasks. Discriminative and generative tasks are fundamental in machine learning. Discriminative models (Hao et al. 2020 He et al. [2016; Padilla et al.) 2020; Zou 2023) are designed to distinguish between data instances, while generative models (Karras Razavi et al. 2019 Song et al. 2021 Wang et al. (020) are employed to create new data Discriminative models have been applied to image classifications (He et al. 2016 Liu
jet al.
ee al. instances.
2020
cet al. 2022 Simonyan & Zisserman, 2015), object detection (Padilla et al. 2020 Zou et al. 2023),
and semantic segmentation (Hao et al.) (020)). Generative models have been extensively studied for image synthesis, including variational autoencoder (Kingma al.) 202 Razavi et al. 2019 Vahdat
& Kautz| 2020) Van Den Oord et al., (017). diffusion (Dhariwal & Nichol| 2021 Kim et al. 2024
Rombach et al., 2022 Song et al. 2021), and autoregressive models (Parmar al.] 2018 Van den
Oord et al. 2016,] Van Den Oord er al. 2016). In this study, our primary focus is on implementing
parameter-efiicte fine-tuning techniques for two tasks: image classification using ConvNeXt (Liu
ct al. 2022 and image synthesis employing Stable Diffusion (Rombach et al. 2022).
6 CONCLUSION
In this work, we proposed the parameter-efficiem fine- tine-tuning method for large convolutional models by formulating the convolutional layers over the filter subspace. Fine-tuning filter atoms composed of a small number of parameters and keeping the atom coefficients unchanged, is notably efficient in terms of parameters. It successfully maintains the capabilities of pre-trained models while avoiding overfitting to downstream tasks. We then formulate a simple yet effective way to achieve an over- complete filter subspace by decomposing each filter atom over another set of filter atoms, thereby expanding the parameter space available for fine-tuning as needed. Our approach has demonstrated effectiveness in different configurations on both discriminate and generative tasks.
Limitations. Our method, which concentrates on tuning models within the filter subspace, is par- ticularly advantageous for ConvNets. While it can be naturally extended to linear layers through appropriate mathematical formulations, the full potential of our approach when applied to linear layers remains under-explored.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Amir Beck and Mare Teboulle. A fast iterative shrinkage-turesholding algorithm for linear inverse problems. SIAM journal on imaging sciences, pp. 183-202, 2009.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 2020.
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 2022.
Wei Chen, Zichen Miao, and Qiang Qiu. Inner product-based neural network similarity. Advances in Neural Information Processing Systems, 2023.
Xiuyuan Cheng, Zichen Miao, and Qiang Qiu. Graph convolution with low-rank learnable local filters. In International Conference Learning Representations, 2021a.
Xiuyuan Cheng, Zichen Miao, and Qiang Qiu. Graph convolution with low-rank learnable local filters. In International Conference Learning Representations, 2021b.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbor,, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition at scale. International Confer- ence on Learning Representations, 2020.
Ali Edalati, Marzich Tahaci, Ivan Kobyzev, Vahid Partovi Nia, James Clark, and Mehdi Rezagholizadeh. Krona: Parameter efficient tuning with kronecker adapter. arXiv preprint arXiv:2212.10650. 2022.
An Eygeniou and Massimiliano Pontil. Multi-task feature learning. Advances in neural information processing systems, 2007.
Dan Friedman and Adji Bousso Dieng. The vendi score: A diversity evaluation metric for machine learning. arXiv preprint arXiv:2210.0244 2022.
Jacob Gildenblat and contributors. Pytorch library for cam methods. htttps://github com/ jacobgi1/pytorciz http://yttooch-rr----eeeeee grad- 2021.
Jianyuan Guo, Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang, and Chang Xu. Cmt: Convolutional neural networks meet vision transformers. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, 2022.
Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact parameter space for diffusion fine-tuning. Proceedings of the IEFF/CVF Interna- tional Conference on Computer Vision, pp. 7323-7334, 2023.
Shijie Hao, Yuan Zhou, and Yanrong Guo. A brief survey on semantic segmentation with deep learning. Neurocomputing, pp. 302-321, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. In Proceedings of the IEFF/CVF International Conference on Computer Vision, 2019.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 2017.

---

Published as a conference paper at ICLR 2025
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, 2019.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. International Conference on Learning Representations, 2021.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinherger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, 2017.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference Computer Vision, 2022.
Shibo Jie and Zhi-Hong Deng. Fact: Factor-tuning for lightweight adaptation on vision transformer. In Proceedings of the AAAI conference on artificial intelligence, 2023.
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems, 2021.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, 2020.
Asifullah Khan, Anabia Sohail, Umme Zahoora, and Aqsa Saeed Qureshi. A survey of the recent
architectures of deep convolutional neural networks. Artificial intelligence review, 2020.
Taewook Kim, Wei Chen, and Qiang Qiu. Learning customize text-to-image diffusion in diverse context. arXiv preprint arXiv:2410. 20.224100000 2024.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Ad- vances in neural information processing systems, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXhv:2304.0266 004.04433. 2023.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers features from tiny images. PhD thesis, University of Toronto, 2009.
Abhishek Kumar and Hal Daume III. Learning task grouping and overlap in multi-task learning. International Conference on Machine Learning, 2012.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, 2023.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the Association for Computational Linguistics, 2021.
Yawei Li, Shuhang Gu, Lue Van Gool, and Radu Timofie. Learning filter basis for convolutional neural network compression. In Proceedings of the IEEE/CVF International Conference on Com- puter Vision, 2019.
Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Scaling & shifting your features: A new baseline for efficient model tuning. Advances in Neural Information Processing Systems, 2022.
12

---

Published as a conference paper at ICLR 2025
Tianci Liu, Ruirui Li, Haoyu Wang, Yunzhe Qi, Hui Liu, Xianfeng Tang, Tianqi Zheng, Qingyu Yin, Monica Xiao Cheng, Jun Huan, et al. Unlocking efficient, scalable, and continual knowledge editing with basis-level representation fine-tuning, In The Thirteenth International Conference Learning Representations.
Tianci Liu, Haoxiang Jiang, Tianze Wang, Ran Xu, Yue Yu, Linjun Zhang, Tuo Zhao, and Haoyu Wang. Roserag: Robust serreval-augmentee generation with small-scale Ilms via margin-aware preference optimization. arXiv preprint arXiv:2502. 10993, 2025.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, 2021.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Confer- ence on Learning Reprexentations, 2018.
Stéphane G Mallat and Zhifeng Zhang. Matching pursuits with time-frequency dictionaries. IEEE Transactions on signal processing, pp. 5397-3415 1993.
Andreas Maurer, Massi Pontil, and Bernardino Comera-Paredess Sparse coding for multitask and transfer learning. In International conference on machine learning, 2013.
Zichen Miao, Ze Wang, Wei Chen, and Qiang Qiu. Continual learning with filter atom swapping. In International Conference Learning Representations, 2021a.
Zichen Miao, Ze Wang, Xiuyuan Cheng, and Qiang Qiu. Spatiotemporal joint filter decomposition in 3d convolutional neural networks. Advances in Neural Information Processing Systems, 2021b.
Zichen Miao, Jiang Wang. Ze Wang, Zhengyuan Yang. Lijuan Wang. Qiang Qiu, and Zicheng Liu. Training diffusion models towards diverse image generation with reinforcement learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
10844-1053,, 2024a.
Zichen Miao, Zhengyuan Yang, Kevin Lin, Ze Wang, Zicheng Liu, Lijuan Wang, and Qiang Qiu. Tuning timestep-distilee diffusion model using pairwise sample optimization. arXiv preprint TrXiv:2410.03190, 2024b.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXhv:2304.071933 2023.
Rafael Padilla, Sergio L Netto, and Eduardo AB Da Silva. A survey on performance metrics for object-detection algorithms. In 2020 international conference on systems, signals and image processing (/WSSIP), 2020.
Vardan Papyan, Yaniv Romano, and Michael Elad. Convolutional neural networks analyzed via convolutional sparse coding. The Journal of Machine Learning Research, 18:2887-2938, 2017.
Genman Parisi, Ronald Kemker, Jose Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 2019.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International conference on machine learning, 2018.
Gaurav Patel, Christopher Sandino, Behrooz Mahasseni, Ellen Zipipi, Erdrin Azemi, Ali Moin, and Juri Minxha. Efficient source-free time-series adaptation via parameter subspace disentanglement. arXiv preprint 14. 2024.
Qiang Qiu, Xiuyuan Cheng, Guillermo Sapiro, and Robert Calderbank. DCFNet:
Deep neural net- work with decomposed convolutional filters. In International Conference on Machine Learning, 2018a.
13

---

Published as a conference paper at ICLR 2025
Qiang Qiu, Xiuyuan Cheng, Guillermo Sapiro, et al. Defnet: Deep neural network with decomposed convolutional filters. In International Conference on Machine Learning, pp. 4198 4207. PMLR, 2018b.
Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. Advances in Neural Information Processing Systems, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 2020.
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers like convolutional neural networks? Advances in Neural Information Processing Systems, 2021.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- on computer vision and pattern recognition, 2022.
Bernardino Pareee,, Hane Aung, Nadia Stanchi-Berthouee and Massimiliano Pontil. Mul- tilinear multitask learning. In International Conference Machine Learning, 2013.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yacl Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2023.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 2015.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. NIPS Deep Learning Symposium, 2016.
Fadil Santosa and William Symes. Linear inversion of band-limited reflection seismograms. SIAM journal scientific and statistical computing, pp. 1307-1330, 1986.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 2022.
Zhiqiang Shen, Zechun Liu, Jie Qin, Marios Savvides, and Kwang-Ting Cheng. Partial is better than all: revisiting fine-tuning strategy for few-shot learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 2016.
14

---

Published as a conference paper at ICLR 2025
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
Kihyuk Sohn, Huiwen Chang, José Lezama, Luisa Polania, Han Zhang, Yuan Hao, Irfan Essa, and Lu Jiang. Visual prompt tuning for generative transfer learning. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score- based diffusion models. Advances in Neural Information Processing Systems, 2021.
Yizhi Song, Liu He, Zhifei Zhang, Soo Ye Kim, He Zhang, Wei Xiong, Zhe Lin, Brian Price, Scott Cohen, Jianming Zhang, et al. beline-b--align: Reference-guided artifacts refinement through semantic alignment. arXiv preprint wtx.xx:2422.000000 2024a.
Yizhi Song, Zhifei Zhang, Zhe Lin, Scott Cohen, Brian Price, Jianming Zhang, Soo Ye Kim, He Zhang, Wei Xiong, and Daniel Aliaga. Imprint: Generative object compositing by learning preesrit-preserviing representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8048-8058, 2024b.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International conference on machine learning, 2019.
Gemma Canet Tarrés, Zhe Lin, Zhifei Zhang, Jianming Zhang, Yizhi Song, Dan Ruta, Andrew Gilbert, John Collomosse, and Soo Ye Kim. Thinking outside the bbox: Unconstrained generative object compositing. arXiv preprint wrXiv:2409.04555 2024.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, 2021.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:230..1397,, 2023.
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. Advances in neural information processing systems, 2020.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Con- ditional image generation with pixelenn decoders. Advances in neural information processing systems, 2016.
Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International conference on machine learning. 2016.
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 2017.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. Cub-200-2011. Technical Report CNS-TR-201 1-001, California Institute of Technology, 2011.
Haoyu Wang, Tianci Liu, Ruirui Li, Monica Cheng, Tuo Zhao, and Jing Gao. Roselora: Row and column-wise sparse low-rank adaptation of pre-trained language model for knowledge editing and fine-tuning, arXiv preprint arXiv:2406.10 10777. 2024.
Ze Wang. Efficient Adaptation of Deep Vision Models. PhD thesis, Purdue University, 2023.
Ze Wang, Seunghyun Hwang, Zichen Miao, and Qiang Qiu. Image generation using continuous filter atoms. Advances in Neural Information Processing Systems, 2021a.
Ze Wang, Zichen Miao, Jun Hu, and Qiang Qiu. Adaptive convolutions with per- pixel dynamic filter atom. In Proceedings of the IEEF/CVF International Conference on Computer Vision, 2021b.
15

---

Published as a conference paper at ICLR 2025
Zhihao Wang, Jian Chen, and Steven CH Hoi. Deep learning for image super-resolution: A survey. IEEE transactions on pattern analysis and machine intelligence, 2020.
Enze Xie, Lewei Yao, Han Shi, Zhili Liu, Daquan Zhou, Zhaoqiang Liu, Jiawei Li, and Zhenguo
Li. Difffit: Unlocking transferability of large diffusion models via simple parameter-efficient fine-tuning, arXiv preprint arXiv:2304.0666 2023.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEÉ conference on computer vision and pattern recognition, 2017.
Zhexiao Xiong, Wei Xiong, Jing Shi, He Zhang, Yizhi Song, and Nathan Jacobs. Groundinghooth: Grounding text-to-image customization. arXiv preprint arXiv:2409.08520, 2024.
SHIH- YING YEH, Yu-Guan Hsich, Zhidong Gao, Bernard BW Yang, Giyeong Oh, and Yanmin Gong, Navigating text-to-image customization: From lycoris fine-tuning model evaluation. In International Conference on Learning Representations, 2023.
Jachong Yoon, Saehoon Kim, Eunho Yang, and Sung Ju Hwang. Scalable and order-robust con- tinual learning with additive parameter decomposition. In International Conference on Learning Representations, 2019.
Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer actually what you need for vision. In Proceedings the the IEFF/CVF conference on computer vision and pattern recognition, 2022.
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. Bitfit: Simple parameter-effcient fine-tuning for transformer-base based masked monguage-modes. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics, 2022.
Mengyao Zhai, Lei Chen, and Greg Mori. liger-lifelonggan: Scalable lifelong learning for image conditioned generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucie, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint
048677 2019.
Yu Zhang and Qiang Yang. survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering, 2021.
Yibo Zhong, Haoxiang Jiang, Lincan Li, Ryumei Nakada, Tianci Liu, Linjun Zhang, Huaxiu Yao, and Haoyu Wang. Neat: Nonlinear parameter-efficient adaptation of pre-trained models. arXiv preprint arXiv:2410001877 2024.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, 2022.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference computer vision, 2015.
Zhengxia Zou, Keyan Chen, Zhenwei Shi, Yuhong Guo, and Jieping Ye. Object detection in 20 years: survey, Proceedings of the IEEE, 2023.
16

---

Published as a conference paper at ICLR 2025
A DETAILS OF EXPERIMENTS
A.1 DETAILS OF TASKS AND DATASETS
VTAB Dataset. VTAB dataset uniquely challenging and well-suited for the evaluation of parameter-effiient tuning methods in the context of few-shot knowledge transfer. VTAB-1K encom- passes a diverse range of image domains, including natural, structured, and specialized categories such medical or satellite imagery. The tasks span various objectives, comprising object and scene recognition, distance classification, and counting. Consequently, VTAB-1k emerges as a highly valuable resource catering to the needs of both discriminative and generative transfer learning tasks.
In 3 we provide information on 19 tasks of the VTAB dataset, including the number classes and the number of images in each data split of VTAB. Images in the VTAB benchmark encompass three distinct domains: (1) Natural images captured using standard cameras, (2) Specialized images captured using non-standard cameras like those in remote sensing and medical applications, and (3) Structured images generated through simulation environments. VTAB-IK is a subset of VTAB. contains only 1000 training and validation samples, which are designed for few-shot transfer learning.
Dreambooth Dataset. The DreamBooth dataset Ruiz et al. (2023) focuses on fine-tuning large pre-trained text-to-image diffusion models for personalized subject-driven image generation. This dataset supports the development and evaluation of methods, which enable the generation of novel, photorealistic images of a specific subject in diverse contexts based on few reference images.
Discriminative and Generative Fine-tuning Tasks. As there emergent needs to customize the pre-trained foundation models for downstream tasks, large corpus of fine-tuning methods have been proposed for both both discriminative and generative tasks. Among them, Ruiz et al. (2023): Miao et al.] (2024a b); Xiong et al. (2024); Tarrés et al.] (2024); [Song et al. (2024b a) have focused on tuning pre-trained image diffusion models for personalized generation, diversity, compositional generation, and human preference. While Hu et al.] (202T); Chen et al. (2022); Karimi Mahabadi
(202I) propose to tune propose tune vision transformers for downstream discriminative tasks.
Parameter Efficient Fine-tuning. To efficiently adapt large models with limited resources. LoRA (Hu al. 2021) decomposes the weight matrix along the channel dimension into two low- rank matrices, which induce much lower costs in fine-tuning compared with tuning the original full-rank weight. Our filter composition method takes inspiration from (Qiu al. 2018b), where the multidimensional weights have been decomposed between the spatial and channel dimensions. which has shown effectiivness continual learning (Miao 2025 Liu et al.). video representation learning (Miao et al. (021b), graph learning (Cheng et al, 2021a), and generative tasks (Wang et al. 2021a b). There are some other works on fine-tuning the SVD decomposition of weights(Han et al. 2003). Kronecker decomposition (Patel et al. 2024).
2021a] Chen et al. 2023 Liu al.
sparsity (Wang et al. 2024),, non-linearity (Zhong et al. 2024), or fine-tunig bias parameters/ Xie al. 2023).
A.2 EXPERIMENTAL SETTINGS
Baseline Methods. We compare our method to 7 PEFT approaches: LoRA (Hu et al. 202 involving the introduction of low-rank structure of accumulated gradient update by decomposing it as up-projection and down- projecttotttionn 4T (ii) LoHa (YEH et al. 2023) utilizes the Hadamard product across two sets of low-rank decompositions to elevate the rank of the resultant matrix and reduce the approximation error. (iii) LoKr (YEH et al.
2023) introduces the Kronecker product for matrix
decomposition to reduce the tunable parameters
(iv) BitFit (akeen et al. 2022) fine-tunes the bias term of each layer. (v) DifiFit (Xie et al. 2023) fine-tunes the bias term, as well as the layer norm
*LoRA implementation; httppss/aitibb.cc/mirrrroo/////s
LooHa and LoKr implementation: http::///thhb..co////t/e/// uue wea f
17

---

Published as a conference paper at ICLR 2025
Table Information of VTAB dataset.
Dataset
classes
train
val
test
all
Caltech-101
102
2754
306
6084
9144
CIFAR-100
100
45000
5000
10000
60000
Clevr (object distance)
6
63000
7000
15000
85000
Clevr (count)
8
63000
7000
15000
85000
Diabetic Retinopathy
5
35126
10906
42670
88702
LMLaa
6
65550
22628
22735
110913
Dsprites (x position)
16
589824
73728
73728
737280
Dsprites (orientation)
16
589824
73728
73728
737280
DTD
47
1880
1880
1880
5640
EuroSAT
10
16200
5400
5400
27000
Flowers102
102
1020
6149
8189
Kitti
4
6347
423
711
7481
Patch Camelyon
2
262144
32768
32768
327680
Pet
37
2944
736
3669
7349
Resise45
45
18900
6300
6300
31500
Smallnorb (azimuth)
18
24300
12150
12150
48600
Smallnorb (elevation)
9
24300
12150
12150
SUN3977
397
76128
10875
21750
108753
SVHN
10
65931
7326
26032
99289
and the scale factor of each layer (vi) OFT and COFT of weight matrices to achieve orthogonal fine-tuning
Qiu et al.
2023) adapts the diagonal blocks
A.2.1 GENERATIVE TASKS
Stable diffusion checkpoints. The pre-trained checkpoint we choose for Stable Diffusion
sabble-iiffusoon--144 which can be found https::/ ottpes//lugggigfaac.ooo////////
stable-diffusion
Text prompts for the few-shot generative task. We adapt the text prompts from YEH et al.
2023 to generate images for Figure [ Additionally, use text prompts from Dreambooth (Ruz et al. 2023 to generate the images and get evaluation results in Table
Text prompts for the full generative task. We use specific text prompts to train the Stable Diffu- sion or generate the images. We list the example prompts for each dataset as follows:
Caltech- 101: This is a picture accordion.
CIFAR-100: This a picture of apple.
Clevr: This a picture from CLEVR dataset.
Diabetic Retinopathy: This is retina image with no diabetic retinopathy.
DMLab: This is picture from DMLab dataset.
Dsprites: This is picture from dSprites dataset.
DTD: This is picture of banded texture.
EuroSAT: This is satellite picture of annual crop.
1lowers This is picture of pink primrose.
Kitti: This is a picture from KITTI dataset.
Patch Camelyon: This a histopathologic scans without tumor.
Pet: This a picture of Abyssinian cat.
Resisc45: This is remote sensing picture of airplane.
DDiffitt and BitFit implementation:
"OFT and COFT implementation from PEFT library httpa//ttttt///gghh......... wtwp:/ggggggaaaeaeeeeee
18

---

Published as a conference paper at ICLR 2025
Table 6: Number of parameters of different PEFT methods.
Conv.
Param.
Attn.
Param.
Original
Cckk
3, 686, 400
4c²
1, 1,63, 400
LoRA LoHa
ckr + ckr
30, 720
Scr
40, 960
kr + 2ckr
61.440
16cr
81,920
Lokr
d'k - ck + 7 dokk/r
3, 904
8c + 47
5,378
OFT
460, 800
4c2 /r 4c
207, 360
Ours (D or D.)
mk2
81
4mk2
576
Ours (+/3)
mm1k k2 + c'mm1
17, 523
4mk
576
Smallnorb: This is picture from SmallNORB dataset.
SUN397: This picture of abbey.
SVHN: This is picture of street view house number O.
B ADDITIONAL ANALYSIS
Computational Time. The decomposition process using the ISTA algorithm for convolutional atoms and atom coefficients takes about second for each layer and 20 seconds for the whole model, with the code implemented on GPU. This time is negligible compared to the training duration, which approximately minutes.
Additionally, we only need to perform sparse coding once for each pre-trained model. The decom- posed coefficients can then be reused across all fine-tuning tasks, further reducing the computational cost.
Computational Cost. We estimate the computation cost terms of FLOPs for solving the sparse coding problem: min //I aD + A||a|1, where we aim to obtain atom coefficients and atoms from the pre-trained weights W. Here a e d xm D E R= W E R xe, d' and are the numbers of input and output channels, l is the kernel size, is the number of filter atoms. Suppose ISTA requires K iterations, the FLOPSs required for this algorithm is K K(4d cm + c'c + 6mk²))
In comparison, given the input data X Bxe' with batch size B, the FLOPSs required for one linear layer Z WX b, where W ge'xe x is 6Béc 4Bc + d'c which includes 2Bc'c 2Bc (forward pass), 4Bc'c e + Bbc (backward pass) and c'c + (update parameters).
Suppose we have c' the decomposition is approximately 9.7 MFLOPs, while the computational cost of one linear layer 101 MFL.Ps.
512, 4, 64, m 9, with one iteration the computational cost of
Number of Parameters. We estimate the parameter numbers of different PEFT methods by con- sidering two types layers as examples: convolutional layers with dimensions (c', k. k), and tention layers with parameters W, g. WA. W., W., which have dimensions (c, c). Table lists the 640,
PEFT fine-tuning methods along with their corresponding parameter counts. Suppose A 3. the hyper-parameter for other approach is 8, the hyper-parameters for our method are k. = 4, m = 9, mi 3.
In Table "Ours (D or D.)" refers to our method with tuning filter atoms D and atoms in the linear layer De. while "Ours (+3)" indicates that, in addition to tuning filter atoms, we also incorporate overcomplete filter atoms and their coefficients B. Compared to other approaches, requires the least number of parameters. To determine the parameter counts reported the paper, enumerate all the model parameters and sum those that require gradients.
method
19

---

Published as a conference paper at ICLR 2025
Liver probe
Parai. (M)
Param (M]
(a)
m 12
Figure 6:
The relations between accuracy and number of fine-tuning parameters, with different numbers of filter atoms (m = and m 12).
Performance comparisons the VTAB-Ik benchmark with ConvNeXT models pre-trained on ImageNet-2IK.
Table 7:
CIFAR- R-0
Params. (M)
ImageNet-Ik
Params. (M)
Full Fine-tuning Linear Probe
94.1
87.7
85.8
88.9
88.6
0.1
84.7
1.0
LoRA (Hu ct al. 2021
89.2
20.1
84.8
21.0
Ours
91.8
0.3
84.9
1.2
C ADDITIONAL EXPERIMENTAL RESULTS
C.I VALIDATION EXPERIMENTS
We compare different configures of our methods with the following experimental settings and a, ar are always fixed: (1) When fine-tuning D, experiment with different numbers of filter atoms
m in the range of [6, 9, 12). (2) When fine-tuning D and D.. we choose (Ke, k.) E {(2, 2), (4, 4)); and the number of filter atoms is
9. (3) When fine-tuning D₁ and B, we choose m = and in the range of [3, 4, 5]. (4) When fine-tuning D., D and B, we choose m = 9,, m_ = 3 and
(K,, ke) (2, 2). We provide additional experiments with m 6, in Figure As we increase
m from 6 to 12, the accuracy improves from 66.86% to 68.68%
C.2 ADDITIONAL EXPERIMENTS OF DISCRIMINATIVE TASKS
Full Dataset Fine-tuning. For CIFAR-100 and ImageNet-IK, we follow the fine-tuning setting of ConvNeXt in (Lian et al. 2022). We employ the Adam (Loshchilov & Hutter (018) optimizer to fine-tune models for 100 epochs for CIFAR-100, and 30 epochs for ImageNet- 1K. The cosine decay strategy is adopted for the learning rate schedule, and the linear warm-up is used in the first 10 epochs for CIFAR- 100 and 5 epochs for ImageNet-IK.
We compare the performance of our approach with other baseline methods, and the results on CIFAR-100 and ImageNet-IK tuning achieves the highest accuracy, outperforming the parameter-efficient fine-tuning methods. One possible reason is both datasets have sufficient data to prevent over-fitting of the model. Our method achieves higher accuracy than LoRA while requiring only a small number of parameters (1.2M v.s. 21M). contrast, the VTAB-11 benchmark, the amount of data not very large (e.g., only 1,000 training images), which might cause wer-fitting of the model for the full fine-tuning.
shown in Table 7 With full dataset fine-tuning, the full fine-
Few-shot Results of VIT.
We also present the results of VIT in the Table Compared to SSF (Lan al. (022), FacT (Tie & Deng, 2023), and Adapter (Houlsby et al., 2019). our method achieves higher average accuracy while keeping the number of tuned parameters minimal.
C.3 RESULTS OF FEW-SHOT GENERATIVE TASKS
We provide experimental results of few-shot generative learning learned concepts "castle" and "canal" in Table. | and I0, In this experiment, we also include LoRA, LoHa, and LoKr with different configurations.
20

---

Published as a conference paper at ICLR 2025
Table 8: Performance comparisons on the VTAB-IK benchmark with ViT-B/16 models pre-trained on ImageNet-21K.
Specialiod
Cuncepos
SHaINORRB
Calsea 101
Punuss (M)
Etowess
SVRK
Sua397
Bunb
KITTI
anting prohing
86.9
$7.4
46.7
25.
2011
Adspter
Roglds <
20/9
86.1
74,1
63.2
97.7
81.0
34.6
50.8
76.3
85.0
73.1
70.5
45.7
37.4
31.2
53.2
30.3
25.4
13.8
22.1
55.82
0.27
FacT
9066
70.6
70.8
99
60.
18.
54.1
64.8
86.1
88.5
75.7
82.6
68.1
08.
M1.7
868
47.4
33.2
43H
73.23
0.11
SSF
2004
926
69.0
75.1
59.4
91.
90.2
52.9
97.4
95.9
87
755
75.9
62.3
53.3
50.6
77.3
54.9
29.5
37.9
73.1
0.24
Table 9: Evaluate different approaches in learning the concept (castle).
Lotta
6.A,
Fidelity
0.73
0.73
0.71
0.67
0.7
0.73
0.65
0.57
0.64
0:64
0.52
0.72
Diversity
3.51
3.51
4.85
5.39
4.96
4.27
6.08
10.38
16.8
16,82
8:97
8.53
T21 Aligrment
0.21
0.21
0.23
0.25
0.23
0.23
0.25
0.26
0.25
0.31
0.28
0.27
Piramn (XT)
3386
847
22.77
3.87
1.42
172
176
0.35
0.4
0.73
2.50
The generated images of different fine-tuning methods are shown in Figure and
C.4 VISUALIZATION OF GENERATED IMAGES
9 We visualize images generated by the models trained on each of VTAB tasks from Figure to
Figure 24
C.5 GRAD-CAM
To understand the underlying reason for the effectiveness of our approach on convolution-based models, we employ Grad-CAM (Gildenblat & contributors, (02T)) on the first block of ResNet50, which are fine-tuned on the CUB dataset (Wah et al. 2011)) using the same experimental setting as above. For our method, we compare the experiment setting with m 9, which means filter atoms AD and the setting with (m, mi) = (9, 4), which means 36 AD,
Based on the Grad-CAM visualization in Figure 25 our method exhibits larger active regions com- pared with LoRA. This observation indicates that our approach benefits from preserving the spatial structure of convolutional layers. When utilizing AD,, which expands the number of filter atoms, observe more active regions in the Grad-CAM heatmap. This suggests that the introduction of extra filter atoms potentially captures a wider range of feature maps.
We provide more heatmap visualizations of Grad-CAM from the first block of ResNet50 in Fig-
ure 26
Table 10: Evaluate different approaches learning the concept (canal).
Lott
LoKi
DifFF:
Batti
Eddlity
Deversity
12.00
24.72
0.35
(M)
21

---

Published as a conference paper at ICLR 2025
(a) Full Fine-tuning
(b) LoRA
(c) LoHa
(d) LoKr
Figure 7: Images sampled from Stable Diffusion RRombach et al. 2022 checkpoints fine-tuned with different approaches. The text prompts used to generate images from top to bottom are: "The (castle) stands against a backdrop snow-capped mountains", "A (castle) surrounded by lush, vibrant forest", "A peacock in front of the (castle)", and 'The (castle) overlooks serene lake, where a family of geese swims".
22

---

Published as a conference paper at ICLR 2025
(a) Pre-trained
(b) DifIFit
(c) BitFit
(d) Ours
Figure 8: Images sampled from Stable Diffusion (Rombach 2022 checkpoints fine-tuned with different approaches. The text prompts used to generate images from top to bottom are: "The (castle) stands against a backdrop snow-capped mountains", "A (castle) surrounded by lush, vibrant forest", "A peacock in front of the (castle)", and 'The (castle) overlooks serene lake, where a family of geese swims".
M
(a) Original
(b) LoRA
(c) Ours
(d) Full tuning
Figure 9: Images sampled from Stable Diffusion checkpoints fine-tuned on the Caltech-101.
23

---

Published as a conference paper at ICLR 2025
(a) Original
(b) LoRA
(c) Ours
(d) Full tuning
Figure 10: Images sampled from Stable Diffusion checkpoints fine-tuned on the CIFAR-100.
TO
(a) Original
(b) LoRA
(c) Ours
(d) Full tuning
Figure 11: Images sampled from Stable Diffusion checkpoints fine-tuned on the SUN397.
93
3
21
9
2(8
6
55
37
18
59
6
66
50
13
7733
33
2
22
8
3
6 10
17 9
5
7
6
400
10
03
9 6
8
89
367
8
6
969
56
9
14
(a) Original
(b) LoRA
(c) Ours
(d) Full tuning
Figure 12: Images sampled from Stable Diffusion checkpoints fine-tuned on the SVHN.
(a) Original
(b) LoRA
(c) Ours
(d) Full tuning
Figure 13: Images sampled from Stable Diffusion checkpoints fine-tuned on the Flowers 102.
24

---

Published as a conference paper at ICLR 2025
(a) Original
(b) LoRA
(c) Ours
(d) Full tuning
Figure 14: Images sampled from Stable Diffusion checkpoints fine-tuned on the Pets.
(a) Original
(b) LoRA
(c) Ours
(d) Full tuning
Figure 15: Images sampled from Stable Diffusion checkpoints fine-tuned the DTD.
(a) Original
(b) LoRA
(c) Ours
(d) Full tuning
Figure 16: Images sampled from Stable Diffusion checkpoints fine-tuned on the ST.
(a) Original
(b) LoRA
(c) Ours
(d) Full tuning
Figure 17: Images sampled from Stable Diffusion checkpoints fine-tuned on the Resisc45.
25

---

Published as a conference paper at ICLR 2025
Original
(b) LoRA
Ours
(d) Full tuning
Figure 18: Images sampled from Stable Diffusion checkpoints fine-tuned on the Patch Camelyon.
|I
Original
(b) LoRA
Ours
(d) Full tuning
Figure 19: Images sampled from Stable Diffusion checkpoints fine-tuned on the Diabetic Retinopa- thy.
(a) Original
(b) LoRA
(c) Ours
(d) Full tuning
Figure 20: Images sampled from Stable Diffusion checkpoints fine-tuned on the Kitti.
(a) Original
(b) LoRA
(c) Ours
(d) Full tuning
Figure 21: Images sampled from Stable Diffusion checkpoints fine-tuned on the Smallnorb.
26

---

Published as a conference paper at ICLR 2025
(a) Original
(h) LoRA
(c) Ours
(d) Full tuning
Figure 22: Images sampled from Stable Diffusion checkpoints fine-tuned the Dsprites.
TUSH
Original
(b) LoRA
Ours
(d) Full tuning
Figure 23: Images sampled from Stable Diffusion checkpoints fine-tuned on the CLEVR.
B
(a) Original
(b) LoRA
(c) Ours
(d) Full tuning
Figure 24: Images sampled from Stable Diffusion checkpoints fine-tuned on the DMLab.
27

---

Published as a conference paper at ICLR 2025
repo
Oup
day
Irput
No Fite-turing
Lo
Ouro
Ouro D1
(a)
(b)
Input
No Finettriing
LofA
Ours D
Ours D1
Input No Fine-tuning
LoH4
Dura
Dura D1
d
Figure 25: The Grad-CAM heatmap comparisons between our method and LoRA reveal that our ap- proach exhibits larger active regions. The heatmap is generated from the first block of ResNet50 (He al. 2016) utilizing the CUB dataset (Wah et al. 2011). Fine-tuning the model with AD, involves additional filter atoms, which leads to larger active regions in the heatmap compared to fine-tuning AD only. (a) The Grad-CAM from the first block of ResNet50. (b-d) The Grad-CAM from the 2-4 blocks of ResNet50.
28

---

Published as a conference paper at ICLR 2025
nout
No Fine-funing
LoRA
Ours
Ours D1
Figure 26: Additional Grad-CAM heatmap comparisons between our method and LoRA from the first block of ResNet50.
29


Reviews:
[{"review_id": "xAs75awy45", "reviewer": "Reviewer_MS4b", "summary": "This paper presents a new way to decompose convolutional layers and experimented a new way to fine-tune large models with those layers by adjusting a small number of parameters based on the decomposition. In particular, the observation that maintaining fixed atom coefficients leads to better results is showed based on the experimental results. Experimental results were compared with other PEFT methods such as LoRA and LoHa and showed interesting results in the provided examples.", "questions": "Parameter fine-tuning often involves one large pre-trained model and many small tasks. Multiple LoRA's can be plug-in to one model even though there could be conflicts, to solve that scenario. How could this method achieve that?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper presents an interesting parameter decomposition method to split parameters in large convolutional models. \n2. In some situations as shared in the paper, the proposed method can achieve comparable or better results by fine-tuning an even smaller amount of parameters.", "weaknesses": "While the proposed decomposition and fine-tuning method is different, this method adjusts parameters in the big model. Comparatively, LoRA serves as a plug-in, which reduces the chance to hurt the capacity of pre-train models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9mIJmBYcn1", "reviewer": "Reviewer_GeoC", "summary": "This paper proposes to fine-tune large pre-trained models over the filter subspace by only adjusting filter atoms and keeping atom coefficients unchanged for parameter-efficient fine-tuning. To adapt to more complex tasks, the number of tunable parameters in filter subspace is increased to construct an overcomplete set of filter atoms by recursively decomposing each filter atom over another set of filter atoms. Experiments on multiple CNN network architectures across discriminative and generative tasks show the effectiveness of the proposed method.", "questions": "See the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.Clear motivation to fine-tune the spatial-only filter atoms for PEFT.\n2.An interesting idea is to use the overcomplete filter atoms to improve performance.\n3.Comprehensive experiments to evaluate the effectiveness of the proposed method.", "weaknesses": "1. Spatial-only convolution and cross-channel mixing are similar to group convolution and point-wise convolution. What is the difference when using group convolution and point-wise convolution as filter atoms and coefficients?\n\n2. The authors mainly consider the parameter usage by only fine-tuning filter atoms. I think memory usage and computation are important for PEFT, which should be discussed in this paper for further evaluating the effectiveness of the proposed method. In addition, how to obtain the total parameters of fine-tuning across different networks should be analyzed to improve the readability\n3.There are multiple important hyper-parameters (e.g., $m, m_1, k_c$), which significantly affect the final performance. How to set these hyper-parameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pLJql2Dn05", "reviewer": "Reviewer_gvwM", "summary": "This work proposes a PEFT technique for convolution by decomposing the convolutional kernel into spatial and channel components and only fine-tuning the spatial components. Furthermore, the authors introduce a second-order decomposition technique to allow for the training of more parameters. The author validate the effectiveness of this method on various backbone models, such as ResNet50, ConvNeXt, and Stable Diffusion.", "questions": "- Besides comparing the number of parameters, what is the GPU memory footprint during fine-tuning for the proposed method? Considering that there is already work indicating that PEFT methods are generally not memory-efficient. \n\n- The idea of decomposing the convolutional kernel and only fine-tuning filter atoms is interesting. However, the experiments in this paper on various tasks do not solid to support the effectiveness of the method. It is necessary to further increase the comparison methods and improve experimental settings. Condidering all the factors, I tend to give a rating of below the acceptance level.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The idea of decomposing the convolutional kernel and only fine-tuning the spatial convolution part is interesting, providing options for fine-tuning convolution layers. \n\n- The explanation of the methods and the mathematical formulas are clear.", "weaknesses": "- The paper requires additional sparse coding at the start of training to decompose convolutional atoms and coefficient atoms. Due to the need to solve optimization problems, I express concern about its efficiency. The computational cost and time delay associated with this part need to be provided. \n\n- The benchmarks compared in Tables 1 and Table 4 are not up-to-date. LoRA was proposed in 2021, but it is now 2024. To my knowledge, a series of related tasks have been continuously proposed in discriminative tasks in recnet years, such as SSF, FacT, Adapter, Compactor, BinaryAdapter, etc. The authors are encouraged to include the latest methods to demostrate the effectiveness of the proposed method. \n\n- The evaluation metrics for the generation task seem non-standard. It appears that the authors only compared results under one theme image, i.e., the castle. As far as I know, exsting common experimental setups for evaluating subject-driven generation tasks 750 prompt-image pairs, such as in OFT. The experimental setup in this paper only take one subject image, makeing it difficult to prove the effectiveness of the method, especially considering the inherent randomness of diffusion. In addition, I also suggest adding OFT and COFT in the compared methods, which are important and widely used baselines in diffusion model fine-tuning, and are included in the HuggingFace's PEFT library.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)