Large Convolutional Model Tuning via Filter Subspace

## Abstract

Abstract Efficient fine-tuning methods are critical to address the high computational and parameter complexity while adapting large pre-trained models to downstream tasks.
Our study is inspired by prior research that represents each convolution filter as a linear combination of a small set of filter subspace elements, referred to as filter atoms . In this paper, we propose to fine-tune pre-trained models by adjusting only filter atoms, which are responsible for spatial-only convolution, while preserving spatially-invariant channel combination knowledge in atom coefficients.
In this way, we bring a new filter subspace view for model tuning.
Furthermore, each filter atom can be recursively decomposed as a combination of another set of atoms, which naturally expands the number of tunable parameters in the filter subspace.
By only adapting filter atoms constructed by a small number of parameters, while maintaining the rest of model parameters constant, the proposed approach is highly parameter-efficient. It effectively preserves the capabilities of pre-trained models and prevents overfitting to downstream tasks.
Extensive experiments show that such a simple scheme surpasses previous tuning baselines for both discriminate and generative tasks.

###### Keywords:

## 1 Introduction

Large models have demonstrated exceptional performance across diverse domains and tasks [, , , , , , , , ], attributing to their capability to effectively represent complex patterns and relationships [] by pre-training on massive datasets [, , ].
A common strategy to adapt these large models for specific downstream tasks is fine-tuning them with full parameters. But this method presents two main challenges: (1) Adjusting a vast number of parameters for particular target tasks is computationally intensive; (2) The limited availability of target data increases the risk of overfitting [].

To address these challenges, researchers have developed parameter-efficient methods [, , , , , ] by fine-tuning
the pre-trained models with only a minimal number of parameters.
Among these methods, LoRA [] fine-tunes models without altering the model architecture, becoming notably popular for its simplicity and efficacy.
However, LoRA still risks overfitting when fine-tuned on limited data and compromising the generalization capability of large models. For instance, Figure [1](#S1.F1) illustrates that with only 5 training samples, LoRA tends to produce images that closely resemble the training data, compromising the ability for diverse image generation, compared with pre-trained models.

Figure: Figure 1: Compared with LoRA [], our method updates a small set of parameters and mitigates the risk of overfitting to the target concept. In this example task of learning the concept <castle> from 5 input images, we only require fine-tuning 0.75M parameters, a stark reduction from the 22.67M parameters required by LoRA. Furthermore, the model fine-tuned by our method captures the target concept while ensuring a rich diversity and strong alignment with input text prompts. It demonstrates that our approach preserves the generalization capability of large models. The text prompts used to generate images from left to right are: “The <castle> stands against a backdrop of snow-capped mountains”, “A <castle> surrounded by a lush, vibrant forest”, “The <castle> overlooks a serene lake”, and “The <castle> in the autumn season with colorful foliage”.
Refer to caption: /html/2403.00269/assets/x1.png

#### 1.0.1 Motivation.

To preserve the capabilities of pre-trained models when fine-tuning them on the downstream tasks, one prominent approach in continual learning [, , ] is to formulate convolution filters in ConvNets as a linear combination of filter atoms [, , ] and fine-tuning only filter atoms [, ].
Specifically, filters in each convolutional layer are decomposed over a small set of filter subspace elements, referred to as *filter atoms*, responsible for *spatial-only convolution*. Each convolutional layer is now constructed as linear combinations of filter atoms using decomposition coefficients, referred to as *atom coefficients*, responsible for the *spatially invariant channel combination*.
Hypothesizing variations across tasks can be reduced by bridging spatial discrepancies in the images, we propose to calibrate the pre-trained model by solely fine-tuning the spatial-only filter atoms while preserving the spatially-invariant channel weights, i.e., atom coefficients.

In our work, we demonstrate that fine-tuning a large model via filter atoms is substantially effective and parameter-efficient, as filter atoms are responsible for spatial-only convolution and usually comprise only a few hundred parameters.
This strategy is in harmony with task subspace modeling principles, which suggest that task parameters occupy a low-dimensional subspace, allowing tasks to be represented as combinations of latent basis tasks [, , , , ].
We also discover that maintaining fixed atom coefficients, i.e., spatially-invariant channel mixing weights, plays a crucial role in preserving the generalization capability of pre-trained large models.

Figure: (a)
Refer to caption: /html/2403.00269/assets/x2.png

With a large number of parameters fixed, fine-tuning only a tiny set of parameters in filter atoms is potentially challenging to adapt to more complex tasks.
We further demonstrate a simple yet effective way to expand the tunable parameters in filter subspace, without any modification on atom coefficients, by decomposing each filter atom over another set of filter atoms. This process provides an *overcomplete* set of filter atoms and expands the tunable parameter space, all while still requiring fewer parameters than LoRA.
Additionally, we provide a simple technique to extend this method to linear layers, ensuring alignment with the characteristics in prior literature [, , , ].
The illustration of our method is displayed in Figure [2](#S1.F2).

We demonstrate the effectiveness of our approach on both discriminative and generative tasks with ResNet50 [], ConvNeXt [] and Stable Diffusion [].
We summarize our contributions as follows,

- •
We propose a method by adapting only filter subspace elements (filter atoms), with a few hundred parameters, to achieve significantly parameter-efficient fine-tuning.
- •
We observe that maintaining fixed atom coefficients plays a crucial role in preserving the generalization capability of large models.
- •
We further demonstrate a simple way to expand the number of tunable parameters in filter subspace by recursively decomposing each filter atom over another set of filter atoms, which extends the parameter space for tuning.
- •
We conduct extensive experiments demonstrating the efficacy of our approach on discriminative and generative tasks for fine-tuning large models.

## 2 Preliminary

### 2.1 Low-rank Adaptation for Fine-tuning

LoRA [] introduces trainable low-rank matrices into layers to approximate weight updates. For a pre-trained weight matrix $\mathbf{W}\in\mathbb{R}^{d^{\prime}\times d}$, LoRA represents the weight update with a low-rank decomposition

$$ $\mathbf{W}+\Delta\mathbf{W}=\mathbf{W}+\mathbf{W}_{down}\mathbf{W}_{up},$ $$

where $\mathbf{W}_{down}\in\mathbb{R}^{d^{\prime}\times r}$ and $\mathbf{W}_{up}\in\mathbb{R}^{r\times d}$ are tunable parameters, $r$ is the intrinsic rank of weight updates, $d^{\prime}$ and $d$ are the size of inputs and outputs. $r$ is usually much smaller than $d^{\prime}$ and $d$. A learnable scalar hyperparameter $\alpha$ scales the weight updates, which leads to $\mathbf{W}+\alpha\mathbf{W}_{down}\mathbf{W}_{up}$.

### 2.2 Sparse Coding and Matrix Factorization

Sparse coding attempts to find the representation of input $\mathbf{w}$ with respect to a dictionary of $m$ atoms $\{\mathbf{d}_{l}\}_{l=1}^{m}$ with the fewest number of coefficients $\alpha^{l}$, i.e., $\mathbf{w}=\sum_{l=1}^{m}\alpha^{l}\mathbf{d}_{l}$. The objective of a sparse coding problem can be written as

$$ $\min_{\alpha^{i,l},\mathbf{d}_{l}}\sum_{j}\lVert\mathbf{w}^{i}-\sum_{i}\alpha^{i,l}\mathbf{d}_{l}\rVert^{2}_{2}+\lambda\sum_{i}\lVert\alpha^{i,l}\rVert_{1},$ $$

where $\lVert\cdot\rVert_{1}$ is the $L_{1}$ norm, and $\lambda$ is a Lagrange multiplier.

It can be further expressed in a tensor form,

$$ $\min_{\boldsymbol{\alpha},\mathbf{D}}\lVert\mathcal{F}-\boldsymbol{\alpha}\times\mathbf{D}\rVert^{2}_{F}+\lambda\lVert\boldsymbol{\alpha}\rVert_{1,1},$ (1) $$

where $\mathcal{F}\in\mathbb{R}^{c^{\prime}\times c\times k^{\prime}\times k}$ is the input tensor, $\boldsymbol{\alpha}\in\mathbb{R}^{c^{\prime}\times c\times m}$ is the tensor of coefficients and $\mathbf{D}\in\mathbb{R}^{m\times k^{\prime}\times k}$ is a tensor of basis.
A number of algorithms have been developed to solve ([1](#S2.E1)), such as fast iterative soft-thresholding algorithm (FISTA) [], matching pursuit [], and least absolute shrinkage and selection operator (LASSO) [].

We can interpret the first term in ([1](#S2.E1)) as finding a suitable tensor factorization for $\mathcal{F}$, while the second term serves as a penalty enforcing sparsity in the representation of $\mathcal{F}$.

## 3 Methods

In this section, we decompose convolution filters over a small set of filter subspace elements, referred to as *fitler atoms*. This formulation enables a new model tuning method via filter subspace by solely adjusting filter atoms.

### 3.1 Formulation of Filter Decomposition

Our approach involves decomposing each convolutional layer $\mathcal{F}$ into two standard convolutional layers: a filter atom layer $\mathbf{D}$ that models filter subspace(^1^11The filter subspace is a span of $m$ filter atoms $\mathbf{D}$.), and an atom coefficient layer $\boldsymbol{\alpha}$ with $1\times 1$ filters that represent combination rules of filter atoms, as displayed in Figure [2](#S1.F2) (a).
This formulation is written as

$$ $\mathcal{F}=\boldsymbol{\alpha}\times\mathbf{D},$ (2) $$

where $\times$ is the tensor product.

$\mathcal{F}\in\mathbb{R}^{c^{\prime}\times c\times k\times k}$ contains $c^{\prime}\times c$ filters $\{\mathcal{F}^{i,j}\}_{i=1,j=1}^{i=c^{\prime},j=c}$, where $c^{\prime}$ and $c$ are the numbers of input and output channels, $k$ is the kernel size. $\mathbf{D}\in\mathbb{R}^{m\times k\times k}$ is a set of $m$ filter atoms, i.e., $\mathbf{D}=\{\mathbf{d}_{l}\}_{l=1}^{m}$, where $\mathbf{d}_{l}\in\mathbb{R}^{k\times k}$. Each filter $\mathcal{F}^{i,j}$ is a linear combination of $m$ filter atoms, calculated by $\mathcal{F}^{i,j}=\sum_{l=1}^{m}\alpha^{i,j,l}\mathbf{d}_{l}$, $\alpha^{i,j,l}$ represents each element in $\boldsymbol{\alpha}\in\mathbb{R}^{c^{\prime}\times c\times m}$.

Considering input features $\mathbf{X}\in\mathbb{R}^{c^{\prime}\times h^{\prime}\times w^{\prime}}$ and output features $\mathbf{Z}\in\mathbb{R}^{c\times h\times w}$, where $h^{\prime}\times w^{\prime}$ and $h\times w$ represent the dimension of input features and output features. The convolution process now can be comprehended as the following two steps:

#### 3.1.1 ∙ ∙ \bullet Spatial-only Convolution with 𝐃 𝐃 \mathbf{D} .

Each channel of the input features $\mathbf{X}$ convolves with each filter atom separately to produce intermediate features

$$ $\mathbf{Z}^{\prime}=\mathbf{D}*\mathbf{X},$ $$

where $\mathbf{Z}^{\prime}\in\mathbb{R}^{(c^{\prime}\times m)\times h\times w}$, $*$ is the convolution operation.

For each input feature channel $\mathbf{X}^{i}$ and filter atom $\mathbf{d}_{l}\in\mathbb{R}^{k\times k}$, we have $\mathbf{Z}^{\prime i,l}=\mathbf{X}^{i}*\mathbf{d}_{l}$, where $\mathbf{Z}^{\prime i,l}\in\mathbb{R}^{h\times w},\mathbf{X}^{i}\in\mathbb{R}^{h^{\prime}\times w^{\prime}}$ are one feature channel in $\mathbf{Z}^{\prime},\mathbf{X}$, and $\mathbf{d}_{l}$ is one filter atom in $\mathbf{D}=\{\mathbf{d}_{l}\}_{l=1}^{m}$.

This process leads to the generation of $m$ distinct intermediate output channels for each input channel, which is illustrated in Figure [2](#S1.F2) (b). In this step, filter atoms focus only on handling the spatial information of input features, and cross-channel mixing is postponed to the next step.

#### 3.1.2 ∙ ∙ \bullet Cross-channel Mixing with 𝜶 𝜶 \boldsymbol{\alpha} .

Subsequently, atom coefficients weigh and linearly combine the intermediate features to produce output features

$$ $\mathbf{Z}=\boldsymbol{\alpha}\times\mathbf{Z}^{\prime}.$ $$

Each output feature channel $\mathbf{Z}^{j}\in\mathbb{R}^{h\times w}$ is linearly combined from $c^{\prime}\times m$ intermediate feature channels $\mathbf{Z}^{\prime i,l}\in\mathbb{R}^{h\times w}$ with the coefficient $\{\alpha^{i,j,l}\}_{i=1,l=1}^{i=c^{\prime},l=m}$,
which is, $\mathbf{Z}^{j}=\sum_{i=1}^{c^{\prime}}\sum_{l=1}^{m}\alpha^{i,j,l}\cdot\mathbf{Z}^{\prime i,l}$. Here, $\alpha^{i,j,l}$ represents each element in atom coefficients $\boldsymbol{\alpha}\in\mathbb{R}^{c^{\prime}\times c\times m}$.

The spatially invariant channel weights, atom coefficients $\boldsymbol{\alpha}$, serve as operators for channel mixing, functioning as distinct combination rules that construct the output features from the elemental feature maps generated by the filter atoms. During the model tuning, $\boldsymbol{\alpha}$ is obtained from the pre-trained model and remains unchanged, while only filter atoms $\mathbf{D}$ adapt to the target task.

#### 3.1.3 Summary.

The two-step convolution operation explains different functionalities of filter atoms $\mathbf{D}$ and atom coefficients $\boldsymbol{\alpha}$, which is, $\mathbf{D}$ only contribute to spatial convolution and $\boldsymbol{\alpha}$ only perform cross-channel mixing.
In practice, the convolution operation is still performed as one layer, without generating intermediate features, to avoid memory cost. In the fine-tuning process, we solely adjust $\mathbf{D}$, which contains a small set of number of parameters, $k\times k\ll c^{\prime}\times c$, thereby facilitating parameter-efficient fine-tuning.

### 3.2 Overcomplete Filter Atoms

The parameters of filter atoms are extremely small compared with overall model parameters. For instance, the filter atoms constitute a mere $0.004\%$ of the total parameters in ResNet50 []. To fully explore the potential of filter subspace fine-tuning, we show next a simple way to construct a set of overcomplete(^2^22 As the size of filters is $k\times k$, $m=k^{2}$ independent filter atoms is *complete* since every filter can be linearly combined by these filter atoms. As the number of filter atoms is larger than $k^{2}$, it becomes *overcomplete*. Overcompleteness can potentially bring a more stable fine-tuning, and in our case, expand the number of parameters for adapting to downstream tasks.) filter atoms by recursively applying the above decomposition to each filter atom, to expand the parameter space for fine-tuning as needed.

Specifically, each filter atom $\mathbf{d}_{l}$ can be further decomposed over $m_{1}$ number of filter atoms in $\mathbf{D}_{1,l}=\{\mathbf{d}_{1,l}^{j}\}_{j=1}^{m_{1}}$ with its corresponding coefficients $\boldsymbol{\beta}^{i}=\begin{bmatrix}\beta^{i,1},\cdots,\beta^{i,m_{1}}\end{bmatrix}\in\mathbb{R}^{m_{1}}$,
i.e., $\mathbf{d}_{l}=\sum_{j=1}^{m_{1}}\beta^{i,j}\mathbf{d}_{1,l}^{j}$.
Therefore, $\mathbf{D}_{1}\in\mathbb{R}^{(m\cdot m_{1})\times k\times k}$ contains $m\cdot m_{1}$ filter atoms, leading to a overcomplete set. The recursive decomposition process is illustrated in Figure [3](#S3.F3) (a).

Considering input $\mathbf{X}\in\mathbb{R}^{c^{\prime}\times h^{\prime}\times w^{\prime}}$ and output $\mathbf{Z}\in\mathbb{R}^{c\times h\times w}$, convolution using the overcomplete filter atoms can be comprehended as three steps. While the spatial-only convolution with $\mathbf{D}$ and cross-channel mixing with $\boldsymbol{\alpha}$ steps are the same as before, it linearly combines the features channels corresponding to $m_{1}$ decomposed filter atoms in *intra-channel mixing* step.

Figure: (a)
Refer to caption: /html/2403.00269/assets/x4.png

#### 3.2.1 ∙ ∙ \bullet Intra-channel Mixing with 𝜷 𝜷 \boldsymbol{\beta} .

After convolving $\mathbf{X}$ and $\mathbf{D}_{1}$ to get $\mathbf{Z}^{\prime}_{1}\in\mathbb{R}^{(c^{\prime}\cdot m\times m_{1})\times h\times w}$, the intermediate feature map $\mathbf{Z}^{\prime}\in\mathbb{R}^{(c^{\prime}\cdot m)\times h\times w}$ is obtained by linearly combining feature channels within the corresponding input channel via coefficients $\boldsymbol{\beta}$,

$$ $\mathbf{Z}^{\prime}=\boldsymbol{\beta}\times\mathbf{Z}^{\prime}_{1}.$ $$

Each feature channel $\mathbf{Z}^{{}^{\prime}i}$ is calculated by $\mathbf{Z}^{{}^{\prime}i}=\sum_{j=1}^{m_{1}}\beta^{i,j}\mathbf{Z}_{1}^{{}^{\prime}i,j}$, where $\beta^{i,j}$ is one element in $\boldsymbol{\beta}$, and $\mathbf{Z}_{1}^{{}^{\prime}i,j}\in\mathbb{R}^{h\times w}$ is one feature channel in $\mathbf{Z}^{\prime}_{1}$.
This phase exclusively blends the feature channels within the same input channel, avoiding any mixing across different channels.
The whole process is illustrated in Figure [3](#S3.F3) (b).
To adapt the model to downstream tasks, we fine-tune both $\boldsymbol{\beta}$ and $\mathbf{D}_{1}$, which contain more tunable parameters than $\mathbf{D}$.

### 3.3 Decomposition of Linear Layers

Figure: Figure 4: Formulate the linear layer $\mathbf{W}$ as a combination of atoms, $\mathbf{W}=\boldsymbol{\alpha}_{c}\times\mathbf{D}_{c}$.
Refer to caption: /html/2403.00269/assets/x6.png

This method can be readily adapted to $1\times 1$ convolutional and linear layers as well.
We start by applying Kronecker decomposition on a linear layer $\mathbf{W}=\mathbf{A}\otimes\mathbf{B}$, where $\mathbf{A}\in\mathbb{R}^{\frac{c^{\prime}}{k^{\prime}_{c}}\times\frac{c}{k_{c}}}$, $\mathbf{B}\in\mathbb{R}^{k^{\prime}_{c}\times k_{c}}$, $\mathbf{W}\in\mathbb{R}^{c^{\prime}\times c}$, $\otimes$ is the Kronecker product. This approach has been explored by recent literature on fine-tuning [, ]. The idea is to represent a matrix as multiple blocks, where each block comes from the same basis $\mathbf{B}$ but varied by different constant $A_{ij}$.
We then introduce $m_{c}$ pairs of $\mathbf{A}$ and $\mathbf{B}$ to reconstruct $\mathbf{W}$, $\mathbf{W}=\sum_{i=1}^{m_{c}}\mathbf{A}_{i}\otimes\mathbf{B}_{i}$, such that each block in $\mathbf{W}$ is represented as a linear combination of all $\mathbf{B}_{i}$.

We re-write the above formulation as the notation with coefficients and atoms such that $\boldsymbol{\alpha}_{c}=\{\mathbf{A}_{i}\}_{i=1}^{m_{c}}$ and $\mathbf{D}_{c}=\{\mathbf{B}_{i}\}_{i=1}^{m_{c}}$. Therefore, $\mathbf{W}$ is constructed from $m_{c}$ atoms in $\mathbf{D}_{c}\in\mathbb{R}^{m_{c}\times k^{\prime}_{c}\times k_{c}}$ by the combination using coefficients $\boldsymbol{\alpha}_{c}\in\mathbb{R}^{\frac{c^{\prime}}{k^{\prime}_{c}}\times\frac{c}{k_{c}}\times m_{c}}$, which is, $\mathbf{W}=\boldsymbol{\alpha}_{c}\times\mathbf{D}_{c}$. This formulation is mathematically equal to the decomposition filters as filter atoms and atom coefficients. Similarly, the $1\times 1$ convolutional layer $\mathcal{F}_{c}\in\mathbb{R}^{c^{\prime}\times c\times 1\times 1}$ can be formulated as $\boldsymbol{\alpha}_{c}$ and $\mathbf{D}_{c}$ with the same process,

$$ $\mathcal{F}_{c}=\boldsymbol{\alpha}_{c}\times\mathbf{D}_{c}.$ (3) $$

The procedure of formulating linear layers as coefficients and atoms is illustrated in Figure [4](#S3.F4).
During the model tuning, $\boldsymbol{\alpha}_{c}$ is obtained from the pre-trained model and remains unchanged, while only filter atoms $\mathbf{D}_{c}$ adapt to the target task.

### 3.4 Parameter Efficient Fine-tuning

Building on our filter subspace formulation, we investigate various parameter-efficient fine-tuning strategies for tuning the pre-trained model.
For convolutional filters $\mathcal{F}$, we directly use sparse coding ([1](#S2.E1)) to decompose them as atom coefficients and filter atoms $\boldsymbol{\alpha}\times\mathbf{D}$. For linear layers or $1\times 1$ convolution layers, we first formulate them as $\mathcal{F}_{c}$ ([3](#S3.E3)), and then decompose them as coefficients $\boldsymbol{\alpha}_{c}$ and atoms $\mathbf{D}_{c}$ using ([1](#S2.E1)). For fine-tuning the models, we only adapt $\mathbf{D}$ or $\mathbf{D}_{c}$, while keeping $\boldsymbol{\alpha}$ or $\boldsymbol{\alpha}_{c}$ fixed.

When more parameters are needed, we represent $\mathbf{D}$ as $\boldsymbol{\beta}$ and $\mathbf{D}_{1}$ to incorporate overcomplete filter atoms. We initialize $\mathbf{D}_{1}$ by simply repeating each filter atom in $\mathbf{D}$ for $m_{1}$ times and initialize every element in $\boldsymbol{\beta}$ with the value $1/m_{1}$. This enables us to fine-tune both $\boldsymbol{\beta}$ and $\mathbf{D}_{1}$.

In this study, we investigate three strategies that offer a range of tunable parameters from a small set to a slightly larger number: fine-tuning (1) only $\mathbf{D}$, (2) $\mathbf{D}$ and $\mathbf{D}_{c}$, and (3) $\boldsymbol{\beta}$, $\mathbf{D}_{1}$ and $\mathbf{D}_{c}$. Downstream tasks are modeled by their updates $\Delta\mathbf{D}$, $\Delta\boldsymbol{\beta}$, $\Delta\mathbf{D}_{1}$ and $\Delta\mathbf{D}_{c}$.

## 4 Experiments

In this section, we begin with studying the effectiveness of our method across various configurations to determine the most suitable application scenario for each configuration.
Subsequently, we demonstrate that fine-tuning only filter atoms requires far fewer parameters while preserving the capacity of pre-trained models, compared with baseline methods in the contexts of discriminative and generative tasks.

### 4.1 Experimental Settings

#### 4.1.1 Datasets.

Our experimental evaluations are mainly conducted on the Visual Task Adaptation Benchmark (VTAB) [], which contains 19 distinct visual recognition tasks sourced from 16 datasets. As a subset of VTAB, VTAB-1k comprises a mere $1,000$ labeled training examples in each dataset.

#### 4.1.2 Models.

For the validation experiment, we choose ResNet50 [] pre-trained on ImageNet-1K. For discriminative tasks, we choose the convolution-based model, ConvNeXt-B [] pre-trained on ImageNet-21K as the initialization for fine-tuning.
For generative tasks, we choose Stable Diffusion [] which contains a downsampling-factor 8 autoencoder with an 860M UNet and CLIP ViT-L/14 as text encoder for the diffusion model. The model is pre-trained on the LAION dataset [], which contains over 5 billion image-text pairs. More details of pre-trained models are listed in the Appendix [7](#S7).

### 4.2 Validation Experiments

In this section, we study the performance of our approach across various configurations.

#### 4.2.1 Implementation details.

We employ ResNet50 [] pre-trained on ImageNet-1K [] and fine-tune it on CIFAR-100 [] for 50 epochs. We utilize the Adam optimizer [] with a learning rate of 0.001 and a weight decay of $1\times 10^{-4}$ and a batch size of 256.

We compare different configures of our methods with the following experimental settings and $\boldsymbol{\alpha}$, $\boldsymbol{\alpha}_{c}$ are always fixed: (1) When fine-tuning $\mathbf{D}$, we experiment with different numbers of filter atoms $m$ in the range of $[6,9,12]$. (2) When fine-tuning $\mathbf{D}$ and $\mathbf{D}_{c}$, we choose $(k^{\prime}_{c},k_{c})\in\{(2,2),(4,4)\}$, and the number of filter atoms is $m=9$. (3) When fine-tuning $\mathbf{D}_{1}$ and $\boldsymbol{\beta}$, we choose $m=9$, and $m_{1}$ in the range of $[3,4,5]$. (4) When fine-tuning $\mathbf{D}_{c}$, $\mathbf{D}_{1}$ and $\boldsymbol{\beta}$, we choose $m=9$, $m_{1}=3$ and $(k^{\prime}_{c},k_{c})=(2,2)$.

**Table 1: Comparison among different configurations of our method.**
|  |  |  |  | Fine-tune $\mathbf{D}$ |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Linear Probe | LoRA | Full Finetuning | $m=6$ | $m=9$ | $m=12$ |
| Accuracy $\uparrow$ | 55.4 | 78.6 | 83.3 | 66.25 | 66.86 | 68.68 |
| Param. (M) $\downarrow$ | 0.2 | 2 | 25.6 | 0.21 | 0.21 | 0.21 |
|  | Fine-tune $\boldsymbol{\beta}$ and $\mathbf{D}_{1}$ | Fine-tune $\mathbf{D}$ and $\mathbf{D}_{c}$ | FT $\boldsymbol{\beta}$, $\mathbf{D}_{1}$ and $\mathbf{D}_{c}$ |  |  |  |
|  | $m_{1}=3$ | $m_{1}=4$ | $m_{1}=5$ | $k_{c}=2$ | $k_{c}=4$ | $m_{1}=4$, $k_{c}=4$ |
| Accuracy $\uparrow$ | 78.69 | 78.7 | 78.9 | 75.03 | 79.83 | 81.8 |
| Param. (M) $\downarrow$ | 0.82 | 1 | 1.2 | 0.23 | 0.62 | 2.1 |

#### 4.2.2 The effectiveness of filter subspace.

The results of all configurations are displayed in Table [1](#S4.T1), with $m=9$ as an example, and $m=6,12$ are detailed in the Appendix [8](#S8).
(1) While fine-tuning $\mathbf{D}$ with fixed $\boldsymbol{\alpha}$, our approach requires a negligible increase in the number of parameters, amounting to only $0.5\%$ additional parameters, but it achieves an almost $20\%$ improvement in accuracy when compared to linear probing.
(2) Fine-tuning $\boldsymbol{\beta}$ and $\mathbf{D}_{1}$ further improves accuracy from $66.8\%$ to $78.7\%$ compared to fine-tuning $\mathbf{D}$ alone. It validates that overcomplete filter atoms provide more capacities for model tuning.
(3) Fine-tuning $\mathbf{D}$ and $\mathbf{D}_{c}$ results in significant improvements compared to fine-tuning $\mathbf{D}$ alone. For example, with $k_{c}=2$, fine-tuning $0.02$M additional parameters improves accuracy from $66.86\%$ to $75.03\%$.
(4) Fine-tuning $\mathbf{D}_{c}$, $\mathbf{D}_{1}$, and $\boldsymbol{\beta}$ achieves the highest accuracy but it also requires the greatest number of parameters among all of our configurations.

#### 4.2.3 Discussion.

Fine-tuning the filter atoms $\mathbf{D}$ or atoms $\mathbf{D}_{c}$ enhances accuracy while keeping the parameter increase minimal, making it highly appropriate for scenarios where the number of parameters is a critical consideration. However, increasing the number of atoms does not easily yield further accuracy improvements. To achieve additional accuracy gains, incorporating the set of overcomplete filter atoms $\mathbf{D}_{1}$ and their coefficients $\boldsymbol{\beta}$ can be effective, but at the cost of an increase in parameters.

### 4.3 Generative Tasks

In this section, we apply our method to the generative task by evaluating the generative samples from VTAB dataset [].

#### 4.3.1 Baselines.

We compare our method to 6 baseline fine-tuning approaches: (i) Full fine-tuning, which entails updating all model parameters during the fine-tuning process; (ii) LoRA [], involving the introduction of a low-rank structure of accumulated gradient update by decomposing it as up-projection and down-projection. (iii) LoHa [] utilizes the Hadamard product across two sets of low-rank decompositions to elevate the rank of the resultant matrix and reduce the approximation error. (iv) LoKr [] introduces the Kronecker product for matrix decomposition to reduce the tunable parameters. (v) BitFit [] fine-tunes the bias term of each layer. (vi) DiffFit [] fine-tunes the bias term, as well as the layer norm and the scale factor of each layer.

#### 4.3.2 Implementation details.

We fine-tune Stable Diffusion using the AdamW optimizer [] with a learning rate of $5\times 10^{-6}$ for full parameter fine-tuning, and $1\times 10^{-3}$ for all other parameter-efficient fine-tuning methods.
For few-shot learning, we fine-tune Stable Diffusion to learn the concept “castle” from the CustomConcept101 dataset [] with the image shape of $512\times 512$. We follow the method in Yeh et al. [] and fine-tune the model for 1000 steps, and evaluate the fidelity, diversity, and text-to-image alignment of generated images.

We assess the fidelity as the average cosine similarity between DINOv2 embeddings [] of the generated and dataset images. The diversity of generated images is measured by the Vendi score [] calculated with the DINOv2 embeddings. The alignment between generated images and corresponding prompts is measured via average cosine similarity in the CLIP feature space [].

For the VTAB dataset, we fine-tune $\boldsymbol{\beta}$, $\mathbf{D}_{1}$, and $\mathbf{D}_{c}$ of the model for 5000 steps with the image shape of $256\times 256$ and utilize Frechet Inception Distance (FID) [] as a quantitative metric for evaluation. To calculate FID, we generate 20,000 images from each checkpoint and compare them with images from the corresponding dataset. In cases where the dataset contains more than 20,000 images, we sample 20,000 images for comparison. See the Appendix [7](#S7) for details of experimental settings.

Figure: (a) Input Images
Refer to caption: /html/2403.00269/assets/x7.jpg

**Table 2: Evaluate different approaches in learning the concept <castle>.**
|  | Pre-trained | Full FT | LoRA [] | LoHa [] | LoKr [] | DiffFit [] | BitFit [] | $\mathbf{C}_{1}$ | $\mathbf{C}_{2}$ | $\mathbf{C}_{3}$ | $\mathbf{C}_{4}$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Fidelity $\uparrow$ | 0.15 | 0.74 | 0.71 | 0.73 | 0.73 | 0.57 | 0.44 | 0.44 | 0.62 | 0.72 | 0.77 |
| Diversity $\uparrow$ | 32.56 | 4.29 | 4.85 | 3.51 | 4.27 | 10.38 | 16.8 | 16.82 | 8.97 | 8.53 | 2.56 |
| T2I Alignment $\uparrow$ | 0.32 | 0.24 | 0.23 | 0.21 | 0.23 | 0.26 | 0.25 | 0.31 | 0.28 | 0.27 | 0.19 |
| Param. (M) $\downarrow$ | - | 860 | 22.67 | 8.47 | 1.06 | 0.58 | 0.34 | 0.05 | 0.75 | 2.39 | 587 |

#### 4.3.3 Few-shot generative transfer learning.

We report the evaluation of all methods in Table [2](#S4.T2). Our method with different configurations is denoted as $C_{1}-C_{4}$.
(1) $C_{1}$ and $C_{2}$ represent fine-tuning $\mathbf{D}$ and $\mathbf{D}_{c}$ of the model with $(m,k_{c})\in\{(6,4),(9,16)\}$.
(2) $C_{3}$ fine-tunes $\boldsymbol{\beta}$, $\mathbf{D}_{1}$ and $\mathbf{D}_{c}$, with $(m,k_{c},m_{1})=(9,16,3)$.
(3) $C_{4}$ is similar to $C_{2}$ but it fine-tunes $\boldsymbol{\alpha}$ together with $\mathbf{D}$ and $\mathbf{D}_{c}$.

Compared with other baseline methods, our method with configuration $C_{1}-C_{3}$ generate diverse images that are well-aligned with the text prompt. It means they preserve the capability of the pre-trained model while learning the new concept.
However, $C_{4}$ fine-tunes the atom coefficients $\boldsymbol{\alpha}$, resulting in the model overfitting to the target concept, which is reflected in a notably high fidelity score, while compromising the model’s ability to generate images aligned with the text prompt. This observation suggests that maintaining the spatially invariant channel weights $\boldsymbol{\alpha}$ helps prevent overfitting when fine-tuning pre-trained models to downstream tasks. The visual examples can be found in Figure [1](#S1.F1) and [5](#S4.F5).

Methods like LoRA [] or full fine-tuning potentially update these $\boldsymbol{\alpha}$, thus, they lead to lower diversity and text-to-image alignment in generated images.
In contrast, BitFit [] and DiffFit [] mostly fine-tune the bias, leaving $\boldsymbol{\alpha}$ fixed, thus, they have a higher diversity and text-to-image alignment than LoRA. However, they also keep the spatial operation $\mathbf{D}$ unchanged, resulting in a lower fidelity score compared with $C_{2}$.
More results can be found in Appendix [8](#S8).

**Table 3: FIDs (lower the better) of image generation models on VTAB benchmark with Stable Diffusion pre-trained on LAION.**
|  | Natural | Specialized | Structured |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Caltech101 | CIFAR100 | DTD | Flowers102 | Pets | SVHN | Sun397 | Patch Camelyon | EuroSAT | Resisc45 | Retinopathy | Clevr | DMLab | KITTI | dSprites | SmallNORB | Mean | Params. (M) |
| No fine-tuning | 52.0 | 113.3 | 74.5 | 45.5 | 117.1 | 212.2 | 28.8 | 258.8 | 186.2 | 144.2 | 307.7 | 271.3 | 225.0 | 288.5 | 373.9 | 266.7 | 179.1 | - |
| Full fine-tuning | 39.0 | 33.8 | 46 | 42.9 | 32.6 | 93.2 | 17.0 | 107.6 | 144.5 | 54.3 | 69.0 | 25.8 | 30.3 | 92.9 | 75.8 | 75.6 | 61.3 | 860 |
| LoRA [] | 36.8 | 42.4 | 40.6 | 45.3 | 30.8 | 120.9 | 17.7 | 101.2 | 81.9 | 61.6 | 69.5 | 33.6 | 32.1 | 64.4 | 69.4 | 71.6 | 57.5 | 22.67 |
| LoHa [] | 33.8 | 40.5 | 41.6 | 41.5 | 33 | 140.9 | 17.3 | 111.4 | 73.7 | 60.4 | 66.5 | 32.1 | 37.9 | 64.5 | 73.3 | 65.7 | 58.4 | 33.86 |
| LoKr [] | 37.7 | 43.5 | 43.2 | 50.6 | 41.1 | 144.2 | 21.4 | 95.4 | 70.5 | 65.5 | 79.3 | 33.9 | 44.8 | 64.1 | 79.4 | 74.4 | 61.8 | 2.12 |
| Ours | 38.6 | 32.4 | 43.9 | 38.5 | 30.6 | 96.8 | 17.2 | 109.7 | 64.1 | 62.4 | 60.6 | 18.7 | 41.9 | 69.5 | 73.2 | 76.5 | 54.7 | 1.11 |

#### 4.3.4 Performance comparisons on generative transfer learning.

We report FIDs of models trained and evaluated on VTAB tasks in Table [3](#S4.T3). In contrast to full parameter fine-tuning and LoRA, our approach attains the lowest FID scores ($54.7$ v.s. $57.5$) while employing the least number of fine-tuning parameters ($1.11$M v.s. $22.67$M). Despite fine-tuning only $0.13\%$ of the total model parameters, our method effectively tailors pre-trained Stable Diffusion to align it with the desired target distribution.

### 4.4 Discriminative Tasks

In this section, we apply our method to the discriminative task, namely the classification on VTAB-1k [].
We compare our method to 4 baseline fine-tuning approaches: (i) Full fine-tuning, (ii) Linear probing, (iii) BitFit [], and (iv) LoRA [].

#### 4.4.1 Implementation details.

Images are resized to $224\times 224$, following the default settings in VTAB []. We employ the AdamW [] optimizer to fine-tune models for 100 epochs. The cosine decay strategy is adopted for the learning rate schedule, and the linear warm-up is used in the first 10 epochs.

In this experiment, we fine-tune $\mathbf{D}$ and $\mathbf{D}_{c}$ while keeping $\boldsymbol{\alpha}$ and $\boldsymbol{\alpha}_{c}$ fixed, as this configuration delivers adequate accuracy without increasing parameters.

**Table 4: Performance comparisons on the VTAB-1k benchmark with ConvNeXt models pre-trained on ImageNet-21K.**
|  | Natural | Specialized | Structured |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Caltech101 | CIFAR100 | DTD | Flowers102 | Pets | SVHN | Sun397 | Patch Camelyon | EuroSAT | Resisc45 | Retinopathy | Clevr/count | Clevr/distance | DMLab | KITTI | dSprites/loc | dSprites/ori | SmallNORB/azi | SmallNORB/ele | Mean | Params. (M) |
| Full fine-tuning | 94.9 | 64.2 | 73.6 | 99.5 | 90.8 | 89.6 | 37.7 | 86.6 | 85.1 | 85.9 | 73.6 | 73.3 | 61.3 | 52.1 | 83.1 | 86.8 | 61.1 | 32.7 | 38.8 | 72.14 | 87.67 |
| Linear Probing | 92.3 | 65.8 | 76.8 | 99.3 | 92.7 | 50.5 | 55.8 | 84.0 | 92.7 | 82.5 | 74.7 | 46.1 | 38.5 | 41.1 | 66.3 | 24.2 | 35.4 | 18.4 | 26.0 | 61.21 | 0.11 |
| BitFit [] | 94.5 | 71.6 | 76.7 | 99.4 | 93.0 | 85.7 | 57.2 | 86.4 | 94.0 | 86.4 | 74.3 | 67.8 | 57.2 | 49.8 | 80.5 | 77.7 | 59.1 | 30.4 | 22.0 | 71.77 | 0.24 |
| LoRA [] | 94.3 | 51.7 | 61.4 | 88.1 | 69.8 | 91.2 | 38.1 | 74.5 | 91.9 | 81.4 | 73.6 | 60.8 | 62 | 50.3 | 80.1 | 96.3 | 56.3 | 39.3 | 21.9 | 67.53 | 17.4 |
| Ours | 94.8 | 71.7 | 76.9 | 99.6 | 93.1 | 87.1 | 57.5 | 85.1 | 94.6 | 86.4 | 74.8 | 70.9 | 62.8 | 50.3 | 82.7 | 89.4 | 60.4 | 31.2 | 29 | 73.59 | 0.45 |
|  | Generalization Error $\downarrow$ |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| BitFit | 0.29 | 1.04 | 0.82 | 0.062 | 0.21 | 0.47 | 2.52 | 0.39 | 0.12 | 0.38 | 0.83 | 0.46 | 0.61 | 1.29 | 1.02 | 0.34 | 0.57 | 4.08 | 1.99 | 0.98 | 0.24 |
| LoRA | 0.23 | 1.22 | 0.93 | 0.028 | 0.24 | 0.4 | 2.53 | 0.72 | 0.16 | 0.41 | 2.35 | 0.94 | 1.09 | 1.56 | 0.93 | 0.25 | 1.17 | 2.56 | 2.39 | 1.06 | 17.4 |
| Ours | 0.24 | 0.95 | 0.96 | 0.029 | 0.22 | 0.43 | 2.52 | 0.54 | 0.14 | 0.42 | 1.64 | 0.5 | 1.08 | 1.84 | 1.01 | 0.16 | 0.99 | 2.37 | 2.01 | 0.94 | 0.45 |

#### 4.4.2 Performance comparisons on few-shot transfer learning.

We compare the performance of our approach and other baseline methods, and the results on VTAB-1k are shown in Table [4](#S4.T4). In these tables, the bold font shows the best accuracy of all methods and the underlined font shows the second best accuracy. Our method outperforms other parameter-efficient fine-tuning methods and even outperforms full fine-tuning. Specifically, our method obtains $6\%$ improvement in accuracy compared to LoRA on the VTAB-1k benchmark while utilizing significantly fewer trainable parameters ($0.45$M v.s. $17.4$M).

The generalization error is measured by the discrepancy between the training loss and the test loss, with the findings illustrated in Table [4](#S4.T4) and Figure [7](#S8.F7) from Appendix [8](#S8).
Our technique, which employs fixed atom coefficients, leads to a comparatively lower generalization error. It means our tuning method better preserves the generalization ability of the pre-trained model.

## 5 Related Works

### 5.1 Pre-training and Fine-tuning

The standard practice of pre-training and fine-tuning [, , , ] entails models initially undergoing pre-training on datasets such as ImageNet-21K, BookCorpus, and Common Crawl [, , ]. Subsequently, these models are fine-tuned to enhance their convergence and performance on specific tasks [].

In the realm of parameter-efficient fine-tuning [], various approaches have been proposed. LoRA [] fine-tunes lower-rank matrices at each layer to represent weight updates.
The adapter [] approach inserts small modules between layers and reduces parameters by only tuning these adapters [, , , ].
Visual prompt tuning (VPT) [, ] has introduced a limited number of learnable parameters for optimization while keeping the backbone frozen.
SSF [] proposes scaling and shifting deep features extracted by a pre-trained model.

### 5.2 Model Architectures

Compared with transformer-based models [, , , ], convolution has been used for a long time as the main module to extract the image features in computer vision tasks. With an inductive prior, convolution-based models require fewer training images and computation resources to achieve good generalization. Convolution-based architectures have been largely studied [, , ] and have found multiple applications, such as feature extracting [], image generation [, ], super-resoluton [], and et cetera. Numerous studies explore the integration of convolutional techniques with vision transformers to enhance their performance [, ].
Parameter-efficient fine-tuning in downstream tasks is crucial and requires further examinations when utilizing pre-trained large-scale convolution-based models.

### 5.3 Discriminative and Generative Tasks

Discriminative and generative tasks represent fundamental concepts in machine learning. Discriminative models [, , , ] are designed to distinguish between various data instances, while generative models [, , , ] are employed to create new data instances. Discriminative models have been applied to image classifications [, , ], object detection [, ], and semantic segmentation []. Generative models have been extensively studied for image synthesis, including variational autoencoder [, , , ], diffusion [, , ], and autoregressive models [, , ].

In this study, our primary focus is on implementing parameter-efficient fine-tuning techniques for two tasks: image classification using ConvNeXt [] and image synthesis employing Stable Diffusion [].

## 6 Conclusion

In this work, we proposed the parameter-efficient fine-tuning method for large convolutional models by formulating the convolutional layers over the filter subspace.
Fine-tuning filter atoms composed of a small number of parameters and keeping the atom coefficients unchanged, is notably efficient in terms of parameters. It successfully maintains the capabilities of pre-trained models while avoiding overfitting to downstream tasks.
We then formulate a simple yet effective way to achieve an overcomplete filter subspace by decomposing each filter atom over another set of filter atoms, thereby expanding the parameter space available for fine-tuning as needed. Our approach has demonstrated effectiveness in different configurations on both discriminate and generative tasks.

#### 6.0.1 Limitations.

Our method, which concentrates on tuning models within the filter subspace, is particularly advantageous for ConvNets. While it can be naturally extended to linear layers through appropriate mathematical formulations, the full potential of our approach when applied to linear layers remains under-explored.

## 7 Details of Experiments

### 7.1 Details of Datasets

VTAB dataset is uniquely challenging and well-suited for the evaluation of parameter-efficient tuning methods in the context of few-shot knowledge transfer. VTAB-1k encompasses a diverse range of image domains, including natural, structured, and specialized categories such as medical or satellite imagery. The tasks span various objectives, comprising object and scene recognition, distance classification, and counting. Consequently, VTAB-1k emerges as a highly valuable resource catering to the needs of both discriminative and generative transfer learning tasks.

In Table [5](#S7.T5), we provide information on 19 tasks of the VTAB dataset, including the number of classes and the number of images in each data split of VTAB. Images in the VTAB benchmark encompass three distinct domains: (1) Natural images captured using standard cameras, (2) Specialized images captured using non-standard cameras like those in remote sensing and medical applications, and (3) Structured images generated through simulation environments.

VTAB-1k is a subset of VTAB. It contains only 1000 training and validation samples, which are designed for few-shot transfer learning.

**Table 5: Information of VTAB dataset.**
| Dataset | classes | train | val | test | all |
| --- | --- | --- | --- | --- | --- |
| Caltech-101 | 102 | 2754 | 306 | 6084 | 9144 |
| CIFAR-100 | 100 | 45000 | 5000 | 10000 | 60000 |
| Clevr (object distance) | 6 | 63000 | 7000 | 15000 | 85000 |
| Clevr (count) | 8 | 63000 | 7000 | 15000 | 85000 |
| Diabetic Retinopathy | 5 | 35126 | 10906 | 42670 | 88702 |
| DMLab | 6 | 65550 | 22628 | 22735 | 110913 |
| Dsprites (x position) | 16 | 589824 | 73728 | 73728 | 737280 |
| Dsprites (orientation) | 16 | 589824 | 73728 | 73728 | 737280 |
| DTD | 47 | 1880 | 1880 | 1880 | 5640 |
| EuroSAT | 10 | 16200 | 5400 | 5400 | 27000 |
| Flowers102 | 102 | 1020 | 1020 | 6149 | 8189 |
| Kitti | 4 | 6347 | 423 | 711 | 7481 |
| Patch Camelyon | 2 | 262144 | 32768 | 32768 | 327680 |
| Pet | 37 | 2944 | 736 | 3669 | 7349 |
| Resisc45 | 45 | 18900 | 6300 | 6300 | 31500 |
| Smallnorb (azimuth) | 18 | 24300 | 12150 | 12150 | 48600 |
| Smallnorb (elevation) | 9 | 24300 | 12150 | 12150 | 48600 |
| SUN397 | 397 | 76128 | 10875 | 21750 | 108753 |
| SVHN | 10 | 65931 | 7326 | 26032 | 99289 |

### 7.2 Experimental Settings

#### 7.2.1 LoRA Implementation

We adopt the LoRA implementation from [https://github.com/microsoft/LoRA](https://github.com/microsoft/LoRA).

#### 7.2.2 LoHa and LoKr Implementation

We adopt the LoHa and LoKr implementation from [https://github.com/KohakuBlueleaf/LyCORIS](https://github.com/KohakuBlueleaf/LyCORIS).

#### 7.2.3 DiffFit and BitFit Implementation

We adopt the DiffFit and BitFit implementation from [https://github.com/mkshing/DiffFit-pytorch](https://github.com/mkshing/DiffFit-pytorch).

#### 7.2.4 Generative Tasks

##### Stable diffusion checkpoints.

The pre-trained checkpoint we choose for Stable Diffusion is stable-diffusion-v1-4, which can be found at [https://huggingface.co/CompVis/stable-diffusion](https://huggingface.co/CompVis/stable-diffusion).

##### Text prompts for the few-shot generative task.

We use specific text prompts to train the Stable Diffusion or generate the images. We list the example prompts for each dataset as follows:

- •
photo of a <castle>.
- •
The <castle> stands against a backdrop of snow-capped mountains.
- •
A <castle> surrounded by a lush, vibrant forest.
- •
The <castle> overlooks a serene lake.
- •
The <castle> in the autumn season with colorful foliage.
- •
The <castle> on a rocky cliff, with crashing waves below.
- •
The <castle> guarded by mythical elves.
- •
A <castle> surrounded by a field of grazing sheep.
- •
A peacock in front of the <castle>.
- •
The <castle> overlooks a serene lake, where a family of geese swims.
- •
<castle>, oil painting ghibli inspired.
- •
<castle> painting by artist claude monet.
- •
<castle> digital painting 3d render geometric style.
- •
Georgia O’Keeffe style <castle> painting.
- •
a watercolor painting of the <castle>.
- •
The <castle> is surrounded by an otherworldly landscape, with glowing mushrooms and mystical creatures.
- •
The <castle>, made of crystal, shimmers in the sunlight.
- •
The <castle>, steampunk aesthetic, adorned with gears and metallic accents.
- •
The <castle> atop a mystical floating island.
- •
Top view of the <castle>.

##### Text prompts for the full generative task.

We use specific text prompts to train the Stable Diffusion or generate the images. We list the example prompts for each dataset as follows:

- •
Caltech-101: This is a picture of accordion.
- •
CIFAR-100: This is a picture of apple.
- •
Clevr: This is a picture from CLEVR dataset.
- •
Diabetic Retinopathy: This is a retina image with no diabetic retinopathy.
- •
DMLab: This is a picture from DMLab dataset.
- •
Dsprites: This is a picture from dSprites dataset.
- •
DTD: This is a picture of banded texture.
- •
EuroSAT: This is a satellite picture of annual crop.
- •
Flowers102: This is a picture of pink primrose.
- •
Kitti: This is a picture from KITTI dataset.
- •
Patch Camelyon: This is a histopathologic scans without tumor.
- •
Pet: This is a picture of Abyssinian cat.
- •
Resisc45: This is a remote sensing picture of airplane.
- •
Smallnorb: This is a picture from SmallNORB dataset.
- •
SUN397: This is a picture of abbey.
- •
SVHN: This is a picture of street view house number 0.

## 8 Additional Experimental Results

### 8.1 Validation Experiments

We provide additional experiments with $m=6,12$ in Figure [6](#S8.F6). As we increase $m$ from 6 to 12, the accuracy improves from $66.86\%$ to $68.68\%$.

Figure: (a) $m=6$
Refer to caption: /html/2403.00269/assets/x15.png

### 8.2 Additional Experiments of Discriminative Tasks

#### 8.2.1 Performance Comparisons on Full Dataset Fine-tuning.

##### Implementation details.

For CIFAR-100 and ImageNet-1K, we follow the fine-tuning setting of ConvNeXt in []. We employ the AdamW [] optimizer to fine-tune models for 100 epochs for CIFAR-100, and 30 epochs for ImageNet-1K. The cosine decay strategy is adopted for the learning rate schedule, and the linear warm-up is used in the first 10 epochs for CIFAR-100 and 5 epochs for ImageNet-1K.

We compare the performance of our approach with other baseline methods, and the results on CIFAR-100 and ImageNet-1K are shown in Table [6](#S8.T6). With full dataset fine-tuning, the full fine-tuning achieves the highest accuracy, outperforming the parameter-efficient fine-tuning methods. One possible reason is both datasets have sufficient data to prevent over-fitting of the model. Our method achieves a higher accuracy than LoRA while requiring only a small number of parameters ($1.2$M v.s. $21$M).
In contrast, in the VTAB-1k benchmark, the amount of data is not very large (e.g., only 1,000 training images), which might cause over-fitting of the model for the full fine-tuning.

**Table 6: Performance comparisons on the VTAB-1k benchmark with ConvNeXT models pre-trained on ImageNet-21K.**
|  | CIFAR-100 | Params. (M) | ImageNet-1k | Params. (M) |
| --- | --- | --- | --- | --- |
| Full Fine-tuning | 94.1 | 87.7 | 85.8 | 88.9 |
| Linear Probe | 88.6 | 0.1 | 84.7 | 1.0 |
| LoRA [] | 89.2 | 20.1 | 84.8 | 21.0 |
| Ours | 91.8 | 0.3 | 84.9 | 1.2 |

#### 8.2.2 Visualization of Generalization Error.

To delve deeper into how various fine-tuning methods impact the generalization capabilities of pre-trained models, we illustrate in Figure [7](#S8.F7) the generalization error for a discriminative task trained on the CIFAR-100 and Diabetic Retinopathy datasets, in relation to the number of fine-tuned parameters.

Figure: (a)
Refer to caption: /html/2403.00269/assets/x17.png

### 8.3 Results of Few-shot Generative Tasks

We provide more experimental results of few-shot generative learning in Table. [7](#S8.T7) and [8](#S8.T8). In this experiment, we also include LoRA, LoHa, and LoKr with different configurations.

The generated images of different fine-tuning methods are shown in Figure [8](#S8.F8) and [9](#S8.F9).

Figure: (a) Full Fine-tuning
Refer to caption: /html/2403.00269/assets/x19.jpg

Figure: (a) Pre-trained
Refer to caption: /html/2403.00269/assets/x23.jpg

**Table 7: Evaluate different approaches in learning the concept <castle>.**
|  | LoHa | LoRA | LoKr | DiffFit | BitFit | Ours |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | $d=16$ | $d=4$ | $r=16$ | $r=4$ | $r=1$ | $f=16$ | $f=4$ |  |  | $m=6,k_{c}=4$ | $m=9,k_{c}=8$ | $m=9,k_{c}=8,m_{1}=3$ |
| Fidelity | 0.73 | 0.73 | 0.71 | 0.67 | 0.7 | 0.73 | 0.65 | 0.57 | 0.44 | 0.44 | 0.62 | 0.72 |
| Diversity | 3.51 | 3.51 | 4.85 | 5.39 | 4.96 | 4.27 | 6.98 | 10.38 | 16.8 | 16.82 | 8.97 | 8.53 |
| T2I Alignment | 0.21 | 0.21 | 0.23 | 0.23 | 0.23 | 0.23 | 0.25 | 0.26 | 0.25 | 0.31 | 0.28 | 0.27 |
| Param. (M) | 33.86 | 8.47 | 22.67 | 5.67 | 1.42 | 1.12 | 1.06 | 0.58 | 0.34 | 0.05 | 0.75 | 2.39 |

**Table 8: Evaluate different approaches in learning the concept <canal>.**
|  | LoHa | LoRA | LoKr | DiffFit | BitFit | Ours |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | $d=16$ | $d=4$ | $r=16$ | $r=4$ | $r=1$ | $f=16$ | $f=4$ |  |  | $m=6,k_{c}=4$ | $m=9,k_{c}=8$ | $m=9,k_{c}=8,m_{1}=3$ |
| Fidelity | 0.52 | 0.47 | 0.39 | 0.38 | 0.37 | 0.36 | 0.38 | 0.31 | 0.33 | 0.16 | 0.29 | 0.39 |
| Diversity | 6.29 | 12.49 | 15.03 | 15.71 | 16.18 | 18.47 | 19.53 | 26.48 | 21.11 | 38.63 | 24.72 | 24.92 |
| T2I Alignment | 0.15 | 0.18 | 0.19 | 0.20 | 0.20 | 0.22 | 0.21 | 0.24 | 0.23 | 0.29 | 0.25 | 0.26 |
| Param. (M) | 33.86 | 8.47 | 22.67 | 5.67 | 1.42 | 1.12 | 1.06 | 0.58 | 0.34 | 0.05 | 0.75 | 2.39 |

### 8.4 Visualization of Generated Images

We visualize images generated by the models trained on each of VTAB tasks from Figure [10](#S8.F10) to Figure [25](#S8.F25).

Figure: (a) Original
Refer to caption: /html/2403.00269/assets/x27.jpg

Figure: (a) Original
Refer to caption: /html/2403.00269/assets/x31.jpg

Figure: (a) Original
Refer to caption: /html/2403.00269/assets/x35.jpg

Figure: (a) Original
Refer to caption: /html/2403.00269/assets/x39.jpg

Figure: (a) Original
Refer to caption: /html/2403.00269/assets/x43.jpg

Figure: (a) Original
Refer to caption: /html/2403.00269/assets/x47.jpg

Figure: (a) Original
Refer to caption: /html/2403.00269/assets/x51.jpg

Figure: (a) Original
Refer to caption: /html/2403.00269/assets/x55.jpg

Figure: (a) Original
Refer to caption: /html/2403.00269/assets/x59.jpg

Figure: (a) Original
Refer to caption: /html/2403.00269/assets/x63.jpg

Figure: (a) Original
Refer to caption: /html/2403.00269/assets/x67.jpg

Figure: (a) Original
Refer to caption: /html/2403.00269/assets/x71.jpg

Figure: (a) Original
Refer to caption: /html/2403.00269/assets/x75.jpg

Figure: (a) Original
Refer to caption: /html/2403.00269/assets/x79.jpg

Figure: (a) Original
Refer to caption: /html/2403.00269/assets/x83.jpg

Figure: (a) Original
Refer to caption: /html/2403.00269/assets/x87.jpg

Figure: (a)
Refer to caption: /html/2403.00269/assets/x91.png

### 8.5 Grad-CAM

To understand the underlying reason for the effectiveness of our approach on convolution-based models, we employ Grad-CAM [] on the first block of ResNet50, which are fine-tuned on the CUB dataset [] using the same experimental setting as above. For our method, we compare the experiment setting with $m=9$, which means 9 filter atoms $\Delta\mathbf{D}$ and the setting with $(m,m_{1})=(9,4)$, which means $36$ $\Delta\mathbf{D}_{1}$.

Based on the Grad-CAM visualization in Figure [26](#S8.F26), our method exhibits larger active regions compared with LoRA. This observation indicates that our approach benefits from preserving the spatial structure of convolutional layers. When utilizing $\Delta\mathbf{D}_{1}$, which expands the number of filter atoms, we observe more active regions in the Grad-CAM heatmap. This suggests that the introduction of extra filter atoms potentially captures a wider range of feature maps.

We provide more heatmap visualizations of Grad-CAM from the first block of ResNet50 in Figure [27](#S8.F27).

Figure: (a)
Refer to caption: /html/2403.00269/assets/x95.jpg

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Parameter fine-tuning often involves one large pre-trained model and many small tasks. Multiple LoRA's can be plug-in to one model even though there could be conflicts, to solve that scenario. How could this method achieve that?
soundness: 3
strengths: 1. This paper presents an interesting parameter decomposition method to split parameters in large convolutional models. 
2. In some situations as shared in the paper, the proposed method can achieve comparable or better results by fine-tuning an even smaller amount of parameters.
summary: This paper presents a new way to decompose convolutional layers and experimented a new way to fine-tune large models with those layers by adjusting a small number of parameters based on the decomposition. In particular, the observation that maintaining fixed atom coefficients leads to better results is showed based on the experimental results. Experimental results were compared with other PEFT methods such as LoRA and LoHa and showed interesting results in the provided examples.
weaknesses: While the proposed decomposition and fine-tuning method is different, this method adjusts parameters in the big model. Comparatively, LoRA serves as a plug-in, which reduces the chance to hurt the capacity of pre-train models.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: See the weaknesses.
soundness: 2
strengths: 1.Clear motivation to fine-tune the spatial-only filter atoms for PEFT.
2.An interesting idea is to use the overcomplete filter atoms to improve performance.
3.Comprehensive experiments to evaluate the effectiveness of the proposed method.
summary: This paper proposes to fine-tune large pre-trained models over the filter subspace by only adjusting filter atoms and keeping atom coefficients unchanged for parameter-efficient fine-tuning. To adapt to more complex tasks, the number of tunable parameters in filter subspace is increased to construct an overcomplete set of filter atoms by recursively decomposing each filter atom over another set of filter atoms. Experiments on multiple CNN network architectures across discriminative and generative tasks show the effectiveness of the proposed method.
weaknesses: 1. Spatial-only convolution and cross-channel mixing are similar to group convolution and point-wise convolution. What is the difference when using group convolution and point-wise convolution as filter atoms and coefficients?

2. The authors mainly consider the parameter usage by only fine-tuning filter atoms. I think memory usage and computation are important for PEFT, which should be discussed in this paper for further evaluating the effectiveness of the proposed method. In addition, how to obtain the total parameters of fine-tuning across different networks should be analyzed to improve the readability
3.There are multiple important hyper-parameters (e.g., $m, m_1, k_c$), which significantly affect the final performance. How to set these hyper-parameters.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: - Besides comparing the number of parameters, what is the GPU memory footprint during fine-tuning for the proposed method? Considering that there is already work indicating that PEFT methods are generally not memory-efficient. 

- The idea of decomposing the convolutional kernel and only fine-tuning filter atoms is interesting. However, the experiments in this paper on various tasks do not solid to support the effectiveness of the method. It is necessary to further increase the comparison methods and improve experimental settings. Condidering all the factors, I tend to give a rating of below the acceptance level.
soundness: 2
strengths: - The idea of decomposing the convolutional kernel and only fine-tuning the spatial convolution part is interesting, providing options for fine-tuning convolution layers. 

- The explanation of the methods and the mathematical formulas are clear.
summary: This work proposes a PEFT technique for convolution by decomposing the convolutional kernel into spatial and channel components and only fine-tuning the spatial components. Furthermore, the authors introduce a second-order decomposition technique to allow for the training of more parameters. The author validate the effectiveness of this method on various backbone models, such as ResNet50, ConvNeXt, and Stable Diffusion.
weaknesses: - The paper requires additional sparse coding at the start of training to decompose convolutional atoms and coefficient atoms. Due to the need to solve optimization problems, I express concern about its efficiency. The computational cost and time delay associated with this part need to be provided. 

- The benchmarks compared in Tables 1 and Table 4 are not up-to-date. LoRA was proposed in 2021, but it is now 2024. To my knowledge, a series of related tasks have been continuously proposed in discriminative tasks in recnet years, such as SSF, FacT, Adapter, Compactor, BinaryAdapter, etc. The authors are encouraged to include the latest methods to demostrate the effectiveness of the proposed method. 

- The evaluation metrics for the generation task seem non-standard. It appears that the authors only compared results under one theme image, i.e., the castle. As far as I know, exsting common experimental setups for evaluating subject-driven generation tasks 750 prompt-image pairs, such as in OFT. The experimental setup in this paper only take one subject image, makeing it difficult to prove the effectiveness of the method, especially considering the inherent randomness of diffusion. In addition, I also suggest adding OFT and COFT in the compared methods, which are important and widely used baselines in diffusion model fine-tuning, and are included in the HuggingFace's PEFT library.

Meta-review: The paper presents the idea of filter subspace tuning, i.e. to fine-tune convolutional neural networks by only adapting spatial filter atoms while leaving the linear combination across channels frozen. The paper presents a clear motivation for the idea, which allows parameter efficient fine-tuning of large models and can achieve results comparable to full fine-tuning as demonstrated on discriminative and generative tasks. After the rebuttal, all reviewers agree on the benefit of the approach.

Decision: Accept (Poster)