Parameter-Efficient Fine-Tuning via Circular Convolution

## Abstract

Abstract Low-Rank Adaptation (LoRA) has gained popularity for fine-tuning large foundation models, leveraging low-rank matrices 𝐀 𝐀 \mathbf{A} bold_A and 𝐁 𝐁 \mathbf{B} bold_B to represent weight changes ( i.e., Δ ⁢ 𝐖 = 𝐁𝐀 Δ 𝐖 𝐁𝐀 \Delta\mathbf{W}=\mathbf{B}\mathbf{A} roman_Δ bold_W = bold_BA ). This method reduces trainable parameters and mitigates heavy memory consumption associated with full delta matrices by sequentially multiplying 𝐀 𝐀 \mathbf{A} bold_A and 𝐁 𝐁 \mathbf{B} bold_B with the activation. Despite its success, the intrinsic low-rank characteristic may limit its performance. Although several variants have been proposed to address this issue, they often overlook the crucial computational and memory efficiency brought by LoRA. In this paper, we propose C ir c ular C onvolution A daptation (C 3 A), which not only achieves high-rank adaptation with enhanced performance but also excels in both computational power and memory utilization. Extensive experiments demonstrate that C 3 A consistently outperforms LoRA and its variants across various fine-tuning tasks.

## 1 Introduction

Figure: Figure 1: Relative comparison of C^3A and baselines on RoBERTa-Base. The Pearson Correlation Coefficient (PCC) is evaluated on STS-B and the Matthew’s Correlation Coefficient (MCC) on CoLA. Accuracies across SST-2, MRPC, QNLI, and RTE are averaged and reported as Acc. #Param denotes the count of learnable parameters (excluding the head), while Memory indicates the peak GPU memory usage during training. Metrics highlighted in blue represent values where higher is better, and those in red indicate values where lower is better. Refer to Table 2 for absolute statistics.
Refer to caption: x10.png

In recent years, Large Foundation Models (LFMs) have surged in popularity due to their remarkable performance across various tasks in natural language processing (NLP) Brown et al. (); Touvron et al. (), computer vision (CV) Radford et al. (); Kirillov et al. (), and other domains Li et al. (). Characterized by their vast number of parameters and substantial computational demands, these models have set new benchmarks in accuracy and versatility. However, their size and complexity pose significant challenges for efficient fine-tuning, particularly in resource-constrained environments Malladi et al. (); Zhang et al. (). To address this issue, Parameter-efficient fine-tuning (PEFT) techniques Mangrulkar et al. (), such as Low-Rank Adaptation (LoRA) Hu et al. (), have emerged as outstanding practical solutions.

LoRA reduces the number of trainable parameters by employing low-rank matrices to approximate weight changes, making fine-tuning more feasible and accessible without compromising the model’s performance. Concretely, LoRA can be expressed in math:

$$ $\displaystyle\mathbf{Wx}=(\mathbf{W}_{0}+\Delta\mathbf{W})\mathbf{x}=\mathbf{W }_{0}\mathbf{x}+\mathbf{B(Ax)},$ $$

where $\mathbf{W},\mathbf{W}_{0},\Delta\mathbf{W}\in\mathbb{R}^{d_{1}\times d_{2}}$ are weight matrices, $\mathbf{B}\in\mathbb{R}^{d_{1}\times r},\mathbf{A}\in\mathbb{R}^{r\times d_{2}}$ are low-rank matrices to construct $\Delta\mathbf{W}$ and $\mathbf{x}\in\mathbb{R}^{d_{2}}$ is the activations. The number of trainable parameters is $r(d_{1}+d_{2})$, prompting the choice $r\ll\min(d_{1},d_{2})$ (e.g., $r=8$ for $d_{1}=d_{2}=1024$) for achieving high parameter efficiency. However, as discussed in Zeng & Lee (), LoRA’s capacity to represent a target model is strictly limited by $r$. To address this trade-off between performance and efficiency, Kopiczko et al. () proposed Vector Random Matrix Adaptation (VeRA), which achieves competitive performance with significantly fewer trainable parameters through fixed random-matrix projections. Despite its low parameter count, VeRA requires substantial computational power and memory due to the large size of the random matrices used for projection. This raises the following open research question in the context of PEFT:

How can high-rank adaptation be achieved without incurring significant costs in terms of time complexity, memory usage, and trainable parameters?

As discussed in Dosovitskiy et al. (), dense linear layers lack inductive biases, which poses challenge on the optimization process and therefore limits transformers’ ability to compete with Convolutional Neural Networks (CNNs) when data is scarce. Given the constraints of a small training set for the downstream task, we posit that incorporating a stronger inductive bias could improve the adaptation performance. In light of this, we propose Circular Convolution Adaptation (C^3A), featuring the circular convolution operator Bamieh ().

Figure: Figure 2: Overview of LoRA (A) and our C^3A (B,C) method. In LoRA, only low-rank matrices $\mathbf{A}$ and $\mathbf{B}$ are trained and the delta weight is represented by their product (i.e., $\Delta\mathbf{W}=\mathbf{BA}$). The total trainable parameter number is $r(d_{1}+d_{2})$, which is assosiated with the rank of the delta weight. In C^3A, circular convolution kernels $\Delta\mathbf{w}$ are tuned to adapt to the downstream task and the delta weight is represented by the (block-)circular matrix they construct (i.e., $\Delta\mathbf{W}=\mathcal{C}_{\mathrm{(blk)}}(\Delta\mathbf{w})$). The total trainable parameter count is $\frac{d_{1}d_{2}}{b}$, which disentangles with the rank of the delta weight. Here, $b$ is the block size of the block-circular matrix and it should be a common divisor (CD) of $d_{1}$ and $d_{2}$.
Refer to caption: x11.png

Circular convolution has been famous in signal processing Li et al. () and cryptography Dworkin et al. () for its distinguished efficiency and compactness. This operator can be equivalently represented as multiplication by a circulant matrix, which offers rank flexibility that is unconstrained by the number of trainable parameters. Moreover, leveraging the Fast Fourier Transform (FFT), C^3A ensures better time and memory efficiency than directly multiplying the circulant matrix Bamieh (), rendering it competitive with LoRA from the efficiency aspect. In a word, circular convolution is a promising option for escaping the rank curse of LoRA at minimal cost. We summarize our contribution as follows:

① We introduce C^3A, a novel approach for PEFT using convolution. This method leverages the circular convolution operation and its equivalent circulant matrix to provide a flexible rank, which is free of linear constraint by the number of trainable parameters, for the delta matrix.

② Leveraging the elegant diagonalization of the circulant matrix, we implement both the forward pass and backpropagation using FFT. With the incorporation of FFT, the computation and memory efficiency of C^3A excels. C^3A strikes a unique balance between performance an efficiency.

③ To offer greater flexibility in controlling the number of trainable parameters, we extend C^3A by incorporating block-circular convolution, which results in block-circulant matrices. This extension allows C^3A to achieve fully customizable parameter counts as well as adaptable rank configurations.

④ We validate C^3A through comprehensive fine-tuning experiments across diverse tasks including natural language understanding, instruction tuning and image classification. Experiments demonstrate C^3A’s outstanding accuracy and memory merits compared to existing state-of-the-art methods.

## 2 Related Work

### 2.1 Parameter-Efficient Fine-Tuning

Research on Parameter-Efficient Fine-Tuning (PEFT) has generally progressed along three main directions. The first direction involves partially updating the pre-trained neural network (e.g., the body Oh et al. () or the biases Zaken et al. ()). Traditional methods relied on hand-crafted heuristics Han et al. (); Oh et al. () to identify which parameters are crucial and should be fine-tuned. More advanced approaches employ optimization techniques Guo et al. (); Xu et al. (); Fu et al. (). For example, Guo et al. () reformulated such a discrete optimization problem into a continuous one by employing Bernoulli masks and the Gumbel-softmax approximation.

The second direction emerged to maintain the integrity of the pre-trained model while enabling a high degree of parameter sharing through adapter-based methods He et al. (); Rebuffi et al. (); Rücklé et al. (); Liu et al. (). These works focus on integrating additional modules, termed adapters, to fit the downstream task, effectively decoupling the pre-trained model parameters from those specific to the downstream task. Prompt Tuning Brown et al. (); Gao et al. (); Chen et al. (); Zhang et al. () and Prefix Tuning Li & Liang () also fall into this category, despite ignoring potential semantic meanings.

The final direction is characterized by delta-weight-based methods, such as Low-Rank Adaptation (LoRA) Hu et al. () and Orthogonal Fine-tuning (OFT) Qiu et al. (). These methods bridge the gap between the pre-trained model and the downstream task by adaptive delta weights, which are stored separately while used in combination with the pre-trained weights. This unique design enables disentanglement of the pretrained and downstream-specific weights. Namely, it achieves parameter sharing and preserves the ability to integrate without additional inference cost. LoRA models the delta-weights by an additive matrix while OFT does it by a multiplicative one. To further improve either parameter efficiency or performance, many variants has been proposed for both of the methods Kopiczko et al. (); Liu et al. (; ); Yuan et al. (). However, these methods can hardly achieve high parameter efficiency and performance without incurring heavy computation and memory usage.

### 2.2 Circular Convolution

Circular convolution has been extensively studied in signal processing Rabiner et al. (); McGillem & Cooper (); Li et al. () and cryptography Dworkin et al. (); Gong et al. (). Owing to its computational advantages, circular convolution has also been explored in machine learning for generating long embeddings of high-dimensional data Yu et al. () and compressing heavily parameterized layers Cheng et al. (); Ding et al. (). Remarkably, it achieves these efficiencies without significant performance degradation, which makes it a promising technique for fine-tuning applications.

Despite its success in small neural networks such as LeNet Cheng et al. (), circular convolution has not demonstrated lossless performance in modern large foundational models (LFMs) or even in their base architecture, the transformer. This limitation may be attributed to the conflict between its high intrinsic bias (i.e., the circulant pattern) and the vast amount of data required for training LFMs. Conversely, when fine-tuning LFMs, it is often impractical to collect as much data as needed for training from scratch. In such scenarios, the intrinsic bias of circular convolution could potentially serve as a regularization mechanism, thereby benefiting the optimization process of fine-tuning.

## 3 Method

In this section, we present C^3A (see an overview in Figure 2), a novel PEFT approach based on the circular convolution. C^3A follows LoRA’s setting of learning an additive linear operation over the original dense linear transformation. However, instead of using low-rank decomposition and the matrix multiplication operator, C^3A resorts to circular convolution as this additive linear operation. Section 3.1 introduces the notations we use. Section 3.2 discusses the circular convolution operator, its equivalent circulant matrix, and its calculation in the frequency domain. Section 3.3 details an efficient method for backpropagation. Section 3.4 describes block-wise convolution for controlling the number of trainable parameters. Finally, Section 3.5 analyzes the computational complexity.

### 3.1 Notations

The adapted weight matrix, the original weight matrix, and the delta matrix are denoted by $\mathbf{W}$, $\mathbf{W}_{0}$, and $\Delta\mathbf{W}$, respectively ($\mathbf{W},\mathbf{W}_{0},\Delta\mathbf{W}\in\mathbb{R}^{d_{1}\times d_{2}}$). The activation vector of the previous layer is denoted by $\mathbf{x}\in\mathbb{R}^{d_{2}}$. The post-transformation vector is $\mathbf{z}$, where $\mathbf{z}=\mathbf{W}\mathbf{x}\in\mathbb{R}^{d_{1}}$, and the incremental part is denoted by $\Delta\mathbf{z}$, where $\Delta\mathbf{z}=\Delta\mathbf{W}\mathbf{x}\in\mathbb{R}^{d_{1}}$. The matrices $\mathbf{A}$ and $\mathbf{B}$ are low-rank matrices introduced by LoRA to represent $\Delta\mathbf{W}$, with $r$ being their rank. $r_{v}$ specifies the rank of the random projection matrix used in VeRA. The circular convolution kernel of C^3A is denoted by $\Delta\mathbf{w}$ and the circular convolution operator by $\star$. The loss function is represented by $\mathcal{L}$. The Fast Fourier Transform and its inverse are denoted by $\mathrm{FFT}$ and $\mathrm{iFFT}$, respectively. The Hadamard product is denoted by $\circ$.

### 3.2 Circular Convolution

Firstly, for simplicity, we assume $d_{1}=d_{2}=d$ and $\Delta\mathbf{w}\in\mathbb{R}^{d}$. The circular convolution operator is defined as $\Delta\mathbf{z}=\Delta\mathbf{w}\star\mathbf{x}=\mathcal{C}(\Delta\mathbf{w})
\mathbf{x}$, where $\mathcal{C}(\cdot)$ is a function which takes a vector and outputs the corresponding circulant matrix. Concretely, the first row of $\mathcal{C}(\Delta\mathbf{w})$ is $\Delta\mathbf{w}$ and the following rows are equal to the row above them periodically shifted to the right by one element. In math,

$$ $\displaystyle\mathcal{C}(\Delta\mathbf{w})=\begin{bmatrix}\Delta w_{1}&\Delta w _{2}&\cdots&\Delta w_{d-1}&\Delta w_{d}\\ \Delta w_{d}&\Delta w_{1}&\cdots&\Delta w_{d-2}&\Delta w_{d-1}\\ \cdots&\cdots&\cdots&\cdots&\cdots\\ \Delta w_{3}&\Delta w_{4}&\cdots&\Delta w_{1}&\Delta w_{2}\\ \Delta w_{2}&\Delta w_{3}&\cdots&\Delta w_{d}&\Delta w_{1}\\ \end{bmatrix}.$ $$

Theoretically, the rank of $\mathcal{C}(\Delta\mathbf{w})$ is given by $d-\mathrm{Deg}(\mathrm{gcd}(f(x),x^{d}-1))$ Ingleton (), where $\mathrm{Deg}(\cdot)$ denotes the degree of a polynomial, $f(x)$ is the polynomial associated with $\Delta\mathbf{w}$ (i.e., $f(x)=\sum_{i=1}^{d}\Delta w_{i}x^{i-1}$), and $\mathrm{gcd}(\cdot)$ represents the greatest common divisor. Consequently, the theoretical upper bound on the rank of $\mathcal{C}(\Delta\mathbf{w})$ is $d$. By learning $\Delta\mathbf{w}$ in the $\mathbb{R}^{n}$ oracle, C^3A automatically achieves dynamic rank selection, which is not linearly constrained by the number of learnable parameters, unlike LoRA.

To achieve high efficiency, we leverage the beautiful circulant structure of $\mathcal{C}(\Delta\mathbf{w})$, which makes it diagonalizable by the Fourier basis ($\mathbf{F}$) . In math, it can be described as $\mathcal{C}(\Delta\mathbf{w})=\mathbf{F}\frac{\Lambda}{d}\mathbf{F}^{-1}$ Golub & Van Loan (), where $\Lambda$ is its eigenvalues and can be calculated by a Fourier transform of the first row (i.e., $\Lambda=\mathrm{diag}(\mathbf{F}\Delta\mathbf{w})$). Therefore, we can calculate $\Delta\mathbf{w}\star\mathbf{x}$ as

$$ $\displaystyle\begin{split}\Delta\mathbf{w}\star\mathbf{x}&=\mathbf{F}\mathrm{ diag}(\frac{\mathbf{F}\Delta\mathbf{w}}{d})\mathbf{F}^{-1}\mathbf{x}\\ &=\mathrm{FFT}(\mathrm{FFT}(\Delta\mathbf{w})\circ\mathrm{iFFT}(\mathbf{x})). \end{split}$ (1) $$

### 3.3 Backpropagation

To implement backpropagation efficiently, we require the analytical derivatives of the loss function $\mathcal{L}$, with respect to $\Delta\mathbf{w}$ and $\mathbf{x}$. Utilizing the chain rule, these derivatives can be expressed as:

$$ $\displaystyle\frac{\partial\mathcal{L}}{\partial\mathbf{x}}=\frac{\partial \Delta\mathbf{z}}{\partial\mathbf{x}}\frac{\partial\mathcal{L}}{\partial\Delta \mathbf{z}},\qquad\frac{\partial\mathcal{L}}{\partial\Delta\mathbf{w}}=\frac{ \partial\Delta\mathbf{z}}{\partial\Delta\mathbf{w}}\frac{\partial\mathcal{L}}{ \partial\Delta\mathbf{z}}.$ (2) $$

Since $\Delta\mathbf{z}=\mathcal{C}(\Delta\mathbf{w})\mathbf{x}$, it follows that $\frac{\partial\Delta\mathbf{z}}{\partial\mathbf{x}}=\mathcal{C}(\Delta\mathbf{
w})$. For $\frac{\partial\Delta\mathbf{z}}{\partial\Delta\mathbf{w}}$, we note that the circular convolution operation is commutative (i.e., $\mathcal{C}(\Delta\mathbf{w})\mathbf{x}=\mathcal{C}(\mathbf{x})\Delta\mathbf{w}$), thus $\frac{\partial\Delta\mathbf{z}}{\partial\Delta\mathbf{w}}=\mathcal{C}(\mathbf{
x})$. Substituting these results into Equation 2, we obtain:

$$ $\displaystyle\frac{\partial\mathcal{L}}{\partial\mathbf{x}}=\mathcal{C}(\Delta \mathbf{w})\frac{\partial\mathcal{L}}{\partial\Delta\mathbf{z}},\qquad\frac{ \partial\mathcal{L}}{\partial\Delta\mathbf{w}}=\mathcal{C}(\mathbf{x})\frac{ \partial\mathcal{L}}{\partial\Delta\mathbf{z}}.$ $$

These expressions can also be interpreted as circular convolutions:

$$ $\displaystyle\frac{\partial\mathcal{L}}{\partial\mathbf{x}}=\Delta\mathbf{w} \star\frac{\partial\mathcal{L}}{\partial\Delta\mathbf{z}},\qquad\frac{\partial \mathcal{L}}{\partial\Delta\mathbf{w}}=\mathbf{x}\star\frac{\partial\mathcal{L }}{\partial\Delta\mathbf{z}}.$ $$

By explicitly implementing this derivative calculation using Equation 1, backpropagation can also enjoy the computational efficiency provided by the FFT algorithm.

### 3.4 Block-Circular Convolution

Despite the elegance and efficiency of the circular convolution operator, it has 2 fundamental limitations due to the requirement that the convolution kernel must be the same size as the activation vector: ❶ It cannot be applied to non-square weight matrices. ❷ The number of learnable parameters cannot be altered. The first limitation restricts its application in scenarios such as fine-tuning a LLaMA3-8B model, where the weight matrix sizes include $4096\times 1024$. The second limitation reduces the flexibility of C^3A, making it challenging to handle complex downstream tasks that require a larger number of learnable parameters. To address these limitations, we resort to block-circular convolution. By splitting the activation vector $\mathbf{x}$ and the post-transformation vector $\mathbf{z}$ into blocks of the same size, we can assign unique convolution kernels to each pair of these blocks. To be concrete,

$$ $\displaystyle\mathbf{x}=\begin{bmatrix}\mathbf{x}_{1}&\mathbf{x}_{2}&\cdots& \mathbf{x}_{\frac{d_{2}}{b}}\end{bmatrix},\qquad\Delta\mathbf{z}=\begin{ bmatrix}\Delta\mathbf{z}_{1}&\Delta\mathbf{z}_{2}&\cdots&\Delta\mathbf{z}_{ \frac{d_{1}}{b}}\end{bmatrix},$ $$

where $b$ is the block size and $b$ need to be a common divisor of $d_{1}$ and $d_{2}$. We will need $\frac{d_{1}d_{2}}{b^{2}}$ convolution kernels to densely connect these blocks, which can be expressed in math as

$$ $\displaystyle\Delta\mathbf{z}_{i}=\sum_{j=1}^{\frac{d_{2}}{b}}\Delta\mathbf{w} _{ij}\star\mathbf{x}_{j},i\in\{1,2,\cdots,\frac{d_{1}}{b}\}.$ $$

This calculation can be represented by a block-circular matrix:

$$ $\displaystyle\Delta\mathbf{z}=\mathcal{C}_{\mathrm{blk}}(\Delta\mathbf{w}) \mathbf{x},\qquad\mathcal{C}_{\mathrm{blk}}(\Delta\mathbf{w})=\begin{bmatrix} \mathcal{C}(\Delta\mathbf{w}_{11})&\mathcal{C}(\Delta\mathbf{w}_{12})&\cdots& \mathcal{C}(\Delta\mathbf{w}_{1\frac{d_{2}}{b}})\\ \mathcal{C}(\Delta\mathbf{w}_{21})&\mathcal{C}(\Delta\mathbf{w}_{22})&\cdots& \mathcal{C}(\Delta\mathbf{w}_{2\frac{d_{2}}{b}})\\ \cdots&\cdots&\cdots&\cdots\\ \mathcal{C}(\Delta\mathbf{w}_{\frac{d_{1}}{b}1})&\mathcal{C}(\Delta\mathbf{w}_ {\frac{d_{1}}{b}2})&\cdots&\mathcal{C}(\Delta\mathbf{w}_{\frac{d_{1}}{b}\frac{ d_{2}}{b}})\end{bmatrix}.$ $$

In this case, $\Delta\mathbf{w}_{ij}\in\mathbb{R}^{b}$ and we have $\frac{d_{1}d_{2}}{b^{2}}b=\frac{d_{1}d_{2}}{b}$ learnable parameters. In fact, $b$ acts as a hyperparameter controlling the number of learnable parameters, just like $r$ for LoRA. However, we highlight that while $r$ controls the rank of the delta matrix and learnable parameter number at the same time, $b$ only affects the latter. Disentangling these attributes enables higher adaptation flexibility and potential better results.

### 3.5 Complexity Analysis

We compare the time complexity and space complexity of LoRA, VeRA and C^3A in Table 1. Detailed analysis follows in this section.

#### 3.5.1 Time Complexity

LoRA incorporates low-rank matrices $\mathbf{A}$ and $\mathbf{B}$, which are sequentially multiplied with the activation vector, resulting in a time complexity of $\mathcal{O}(r(d_{1}+d_{2}))$. Typically, $r\ll\min(d_{1},d_{2})$. Conversely, VeRA, despite its high-rank nature and minimal number of trainable parameters, incurs a prohibitive time complexity of $\mathcal{O}(r_{v}(d_{1}+d_{2}))$, where $r_{v}$ can exceed $\max(d_{1},d_{2})$. Therefore, striking a balance between high rank and computational efficiency is never a low-hanging fruit.

Thanks to the $\mathcal{O}(n\log n)$ complexity of the FFT algorithm used in Equation 1, C^3A achieves a time complexity of $\mathcal{O}((d_{1}+d_{2})\log{b}+\frac{d_{1}d_{2}}{b})$. The first term is the time complexity for FFT, and the second term is for aggregation. In practical scenarios, $b$ is chosen as the greatest common divisor of $d_{1}$ and $d_{2}$ to achieve a high compression ratio. Given that, C^3A is comparable to LoRA in terms of time complexity.

**Table 1: Time and space complexity comparison of LoRA, VeRA and C^3A. We split the space complexity into Parameter number and Other auxiliary tensors to help better understand the differences. We highlight that in practice, to achieve similar performance, $\frac{\max(d_{1},d_{2})}{b}\leq r\ll r_{v}$.**
| Method | Time | # Param | # Other | # Total |
| --- | --- | --- | --- | --- |
| LoRA | $\mathcal{O}(r(d_{1}+d_{2}))$ | $r(d_{1}+d_{2})$ | 0 | $r(d_{1}+d_{2})$ |
| VeRA | $\mathcal{O}(r_{v}(d_{1}+d_{2}))$ | $r_{v}+d_{1}$ | $r_{v}(d_{1}+d_{2})$ | $r_{v}(d_{1}+d_{2})+r_{v}+d_{1}$ |
| C^3A | $\mathcal{O}((d_{1}+d_{2})\log{b}+\frac{d_{1}d_{2}}{b})$ | $\frac{d_{1}d_{2}}{b}$ | $b$ | $\frac{d_{1}d_{2}}{b}+b$ |

#### 3.5.2 Space Complexity

We analyze the space complexity of LoRA, VeRA, and C^3A during training. The differences among these methods primarily arise from the trainable parameters and the auxiliary tensors required for the forward pass and backpropagation. LoRA does not rely on auxiliary tensors, while VeRA necessitates 2 random projection matrices, with a total size of $r_{v}(d_{1}+d_{2})$. Since $r_{v}$ is by no means negligible, the memory consumption of VeRA would be significantly larger than that of LoRA.

In terms of C^3A, the only additional auxiliary tensor would be of size $b\leq\min(d_{1},d_{2})$, which is reserved by the FFT algorithm. Given a proper $b$, which is often close to the greatest common divisor of $d_{1}$ and $d_{2}$, the space complexity of C^3A excels. Additionally, we observe that when applying FFT, input blocks and output blocks are independent within their own group. Therefore, parallelization can be employed to enable the flexibility of trading space for time.

## 4 Experiment

In this section, we firstly conduct experiments on a synthetic dataset to show the superior expressiveness of C^3A over LoRA. After that, we comprehensively evaluate C^3A in both natural language processing (NLP) and computer vision (CV).
For NLP, we first demonstrate the effectiveness of C^3A using the GLUE benchmark with RoBERTa-Base and RoBERTa-Large models. Additionally, we apply C^3A to fine-tuning the LLaMA family models, which stands as a representative of prevalent large language models (LLMs).
For CV, we examine the most representative classification task using Vision Transformers (ViTs) on a wide range of datasets.

### 4.1 Synthetic Data

Settings.
We distribute 8 points evenly on a 2D plane as cluster centers and randomly sample 30 points from 8 corresponding multivariate Gaussian distributions. A 3-layer MLP is then used to classify these point clusters. To compare the expressiveness of 2 types of layers, we replace the middle layer with either a low-rank layer or a circulant layer, ensuring that both layers have the same number of trainable parameters for a fair comparison.

Figure: Figure 3: Expressiveness test on synthetic data. The left figure shows the synthetic data used for the experiment, while the right figure illustrates the training accuracy curves of a 3-layer MLP, incorporating C^3A, LoRA, and standard linear layers, respectively.
Refer to caption: x12.png

Results.
The results are presented in Figure 3. We observe that LoRAr=1 struggles with this straightforward classification task. In contrast, C^3Ab=128/2, despite using the same number of parameters, achieves perfect classification, comparable to a standard linear layer. This demonstrates the high expressiveness of C^3A given the same parameter budget.

### 4.2 Natural Language Understanding

Baselines.
We compare our C^3A with several representative PEFT methods, including BitFit Zaken et al. (), (IA)^3 Liu et al. (), LoRA Hu et al. (), VeRA Kopiczko et al. (), and BOFT Liu et al. ().
BitFit selectively fine-tunes existing parameters, specifically the biases. (IA)^3 is the state-of-the-art method that adds extra adapters. LoRA is a widely known PEFT method that employs low-rank decomposition to compress additive delta matrices. VeRA is a recent approach that focuses on further reducing trainable parameters of LoRA while preserving a high rank. BOFT is another innovative method in PEFT research, compressing multiplicative delta matrices using orthogonal decomposition and butterfly factorization.

**Table 2: Performance of different PEFT methods on the GLUE benchmark. We fine-tune pre-trained RoBERTa-Base and -Large models on 6 datasets. We report the Matthew’s Correlation Coefficient (MCC) for CoLA, Pearson Correlation Coefficient (PCC) for STS-B, and accuracy (Acc.) for all the remaining tasks. For each metric, a higher score indicates better performance. “Avg.” denotes the average score of each method across all datasets. The best results for each dataset are highlighted in bold. # Trainable parameters does not include the classification head since each method uses a head of the same size. Memory Cost is measured on fixed length (i.e., $256$) data with a batchsize of $64$.**
|  | Method | # Trainable<br>Parameters | Memory<br>Cost (GB) | SST-2 | MRPC | CoLA | QNLI | RTE | STS-B | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Base | Full | 124M | 17.19 | 94.01$\pm$0.39 | 87.10$\pm$0.79 | 62.00$\pm$1.16 | 92.40$\pm$0.28 | 77.33$\pm$2.68 | 90.70$\pm$0.14 | 83.92 |
| BitFit | 102K | 12.60 | 93.30$\pm$0.30 | 85.80$\pm$0.21 | 59.21$\pm$1.74 | 91.96$\pm$0.18 | 73.07$\pm$1.34 | 90.18$\pm$0.17 | 82.25 |  |
| (IA)^3 | 111K | 19.86 | 92.98$\pm$0.34 | 85.86$\pm$0.59 | 60.49$\pm$1.09 | 91.56$\pm$0.17 | 69.10$\pm$1.18 | 90.06$\pm$0.21 | 81.67 |  |
| LoRAr=8 | 295K | 13.75 | 94.50$\pm$0.41 | 85.68$\pm$0.74 | 60.95$\pm$1.57 | 92.54$\pm$0.20 | 76.68$\pm$1.42 | 89.76$\pm$0.39 | 83.35 |  |
| VeRAr=1024 | 43K | 15.51 | 93.97$\pm$0.17 | 86.23$\pm$0.41 | 62.24$\pm$1.91 | 91.85$\pm$0.17 | 75.74$\pm$1.56 | 90.27$\pm$0.25 | 83.38 |  |
| BOFT${}^{m=2}_{b=8}$ | 166K | 13.11 | 93.23$\pm$0.50 | 84.37$\pm$0.54 | 59.50$\pm$1.25 | 91.69$\pm$0.12 | 74.22$\pm$0.84 | 89.63$\pm$0.37 | 82.11 |  |
| C^3Ab=768/1 | 18K | 12.83 | 93.42$\pm$0.26 | 86.33$\pm$0.32 | 61.83$\pm$0.96 | 91.83$\pm$0.04 | 76.17$\pm$1.39 | 90.46$\pm$0.29 | 83.34 |  |
| C^3Ab=768/6 | 111K | 12.72 | 94.20$\pm$0.16 | 86.67$\pm$0.54 | 62.48$\pm$1.20 | 92.32$\pm$0.25 | 77.18$\pm$1.41 | 90.16$\pm$0.42 | 83.84 |  |
| Large | BitFit | 271K | 30.65 | 95.09$\pm$0.27 | 88.10$\pm$0.76 | 65.40$\pm$0.76 | 94.06$\pm$0.14 | 82.60$\pm$1.15 | 91.73$\pm$0.20 | 86.16 |
| (IA)^3 | 295K | 48.81 | 95.32$\pm$0.20 | 87.06$\pm$0.57 | 66.52$\pm$1.10 | 94.18$\pm$0.15 | 84.33$\pm$2.38 | 91.58$\pm$0.39 | 86.50 |  |
| LoRAr=8 | 786K | 34.12 | 95.53$\pm$0.35 | 86.12$\pm$3.22 | 65.16$\pm$0.76 | 93.73$\pm$0.30 | 83.75$\pm$0.51 | 91.46$\pm$0.21 | 85.96 |  |
| VeRAr=256 | 61K | 34.16 | 95.83$\pm$0.43 | 87.72$\pm$0.55 | 63.66$\pm$1.45 | 94.11$\pm$0.20 | 83.03$\pm$1.65 | 91.12$\pm$0.37 | 85.91 |  |
| BOFT${}^{m=2}_{b=8}$ | 442K | 33.08 | 95.76$\pm$0.41 | 88.28$\pm$0.33 | 64.72$\pm$2.37 | 93.89$\pm$0.14 | 82.82$\pm$1.40 | 91.03$\pm$0.32 | 86.08 |  |
| C^3Ab=1024/1 | 49K | 31.83 | 95.78$\pm$0.05 | 88.02$\pm$0.62 | 66.59$\pm$1.20 | 94.22$\pm$0.25 | 82.89$\pm$0.67 | 91.86$\pm$0.14 | 86.56 |  |
| C^3Ab=1024/8 | 393K | 31.79 | 95.78$\pm$0.15 | 88.09$\pm$0.47 | 67.18$\pm$1.92 | 94.26$\pm$0.19 | 84.62$\pm$1.36 | 91.81$\pm$0.36 | 86.96 |  |

Settings.
We evaluate our proposed C^3A on the General Language Understanding Evaluation (GLUE) benchmark Wang et al. (), which encompasses a wide range of natural language understanding (NLU) tasks, including single-sentence classification, similarity and paraphrase, and natural language inference. To enhance practicality, we split these datasets following the train-validation-test approach. The best-performing model is selected based on validation set performance across the fine-tuning epochs, and the reported performance corresponds to its performance on the test set. For this evaluation, we fine-tune the pre-trained RoBERTa-Base and RoBERTa-Large models Liu et al. (). For the unique hyperparameters of different baselines, we adopt the values suggested in the original papers (e.g., VeRA’s $r$ and BOFT’s $b$ and $m$). The number of trainable parameters excludes the classification head, as each method uses one of the same size. The shared hyperparameters (i.e., the learning rate for classification head and for other trainable parameters, separately) are found by hyperparameter search. For the memory cost, to ensure fairness and consistency, we fix the length of input data to 256 tokens and use a batchsize of 64.

Results.
Results are summarized in Table 2. In general, C^3Ab=768/1 and C^3Ab=1024/1 achieves better or on-par performance compared to baseline methods with extremely small trainable parameters. When a larger number of trainable parameters is allowed, C^3Ab=768/6 and C^3Ab=1024/8 significantly outperform baselines. In addition, compared to (IA)^3, LoRA, VeRA, and BOFT, despite training a similar number of or more parameters, C^3A stands out for its excellent memory efficiency. The only method with better memory efficiency is BitFit, which can be seen as an upper bound since it does not introduce any new parameters. Furthermore, nearly all circulant delta matrices discovered by C^3A are full rank, indicating maximal capacity Zeng & Lee () and providing theoretical support for the impressive results.

### 4.3 Instruction Tuning

Settings.
We evaluate C^3A against LoRA by fine-tuning LLaMA2-7B/13B and LLaMA3-8B on 4 standard instruction tuning datasets: GSM8k Cobbe et al. (), MATH Yu et al. (), ViGGO Juraska et al. (), and SQL Zhong et al. (), following the setting outlined in Nikdan et al. ().
For the SQL dataset, we preprocess by selecting 12.5% of the dataset, followed by an 4:1 train-test split.
To ensure a fair comparison, we maintain LoRA parameters with $r=32$ and a dropout rate of 0.05, while exploring various $\alpha$ values ranging from 32 to 128.
All experiments are conducted for a standard single pass (epoch) over the dataset.

Results.
In Table 3, we summarize our main
experiments.
It can be observed that C^3A generally surpasses LoRA across both LLaMA2-7B/13B and LLaMA3-8B.
Notably, C^3A only utilizes less than half of the parameter count compared to LoRA.
Overall, these results clearly highlight the effectiveness of C^3A.

**Table 3: Comparison of C^3A and LoRA on fine-tuning LLaMA2 and LLaMA3 models in terms of accuracy and trainable parameters. The best results for each dataset are highlighted in bold. “Avg.” denotes the average accuracy of each method across all datasets.**
| Model | Method | # Trainable<br>Parameters | GSM8k | MATH | ViGGO | SQL | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| LLaMA2-7B | LoRAr=32 | 16.8M | 27.22 | 4.10 | 92.24 | 65.39 | 47.24 |
| C^3Ab=4096/32 | 8.4M | 29.34 | 4.25 | 94.03 | 66.21 | 48.46 |  |
| LLaMA2-13B | LoRAr=32 | 26.2M | 42.68 | 6.70 | 93.19 | 67.53 | 52.53 |
| C^3Ab=5120/32 | 13.1M | 42.00 | 6.65 | 95.57 | 67.28 | 52.88 |  |
| LLaMA3-8B | LoRAr=32 | 13.6M | 56.57 | 18.95 | 91.78 | 68.19 | 58.87 |
| C^3Ab=4096/32 | 5.2M | 57.54 | 19.75 | 93.07 | 69.47 | 59.96 |  |

### 4.4 Image Classification

Settings.
We focus on the task of image classification utilizing the Vision Transformer (ViT) models. Specifically, we employ both the Base and Large variants of this prevalent computer vision foundation model, as introduced by Dosovitskiy et al. (). These ViT models are pre-trained on the extensive ImageNet-21K dataset Ridnik et al. (). For the fine-tuning phase, we utilize a diverse set of datasets including Pets, Cars, DTD, EuroSAT, FGVC and RESISC.

**Table 4: Fine-tuning results with ViT-Base and ViT-Large models on various image classification datasets. The models are fine-tuned for 10 epochs, and the best-performing model, based on validation set accuracy, is selected. The reported accuracy corresponds to the performance on the test set. The best results for each dataset are highlighted in bold. “Avg.” denotes the average accuracy of each method across all datasets.**
|  | Method | # Trainable<br>Parameters | Pets | Cars | DTD | EuroSAT | FGVC | RESISC | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Base | Head | - | 90.28$\pm$0.43 | 25.76$\pm$0.28 | 69.77$\pm$0.67 | 88.72$\pm$0.13 | 17.44$\pm$0.43 | 74.22$\pm$0.10 | 61.03 |
| Full | 85.8M | 92.82$\pm$0.54 | 85.10$\pm$0.21 | 80.11$\pm$0.56 | 99.11$\pm$0.07 | 61.60$\pm$1.00 | 96.00$\pm$0.23 | 85.79 |  |
| LoRAr=16 | 590K | 93.76$\pm$0.44 | 78.04$\pm$0.33 | 78.56$\pm$0.62 | 98.84$\pm$0.08 | 56.64$\pm$0.55 | 94.66$\pm$0.17 | 83.42 |  |
| C^3Ab=768/12 | 221K | 93.88$\pm$0.22 | 79.05$\pm$0.35 | 80.57$\pm$0.53 | 98.88$\pm$0.07 | 54.31$\pm$0.79 | 94.54$\pm$0.23 | 83.54 |  |
| Large | Head | - | 91.11$\pm$0.30 | 37.91$\pm$0.27 | 73.33$\pm$0.26 | 92.64$\pm$0.08 | 24.62$\pm$0.24 | 82.02$\pm$0.11 | 66.94 |
| Full | 303M | 94.30$\pm$0.31 | 88.15$\pm$0.50 | 80.18$\pm$0.66 | 99.06$\pm$0.10 | 67.38$\pm$1.06 | 96.08$\pm$0.20 | 87.53 |  |
| LoRAr=16 | 1.57M | 94.62$\pm$0.47 | 86.11$\pm$0.42 | 80.09$\pm$0.42 | 98.99$\pm$0.03 | 63.64$\pm$0.83 | 95.94$\pm$0.21 | 86.56 |  |
| C^3Ab=1024/16 | 786K | 94.48$\pm$0.30 | 84.94$\pm$0.39 | 82.62$\pm$0.52 | 98.75$\pm$0.19 | 63.80$\pm$0.37 | 95.52$\pm$0.16 | 86.69 |  |

Results.
Table 4 presents a summary of the results obtained from 6 image classification datasets using the ViT Base and Large models. The LoRA and C^3A methods both show substantial performance improvements over Head Tuning, highlighting their effectiveness in the image classification domain. Notably, our method achieves comparable performance to LoRA while utilizing only half of the parameter count.

## 5 Conclusion

In this paper, we introduce C^3A, an innovative approach to Parameter-Efficient Fine-Tuning (PEFT). Unlike LoRA, which utilizes low-rank decomposition, C^3A employs circular convolution and its associated circulant matrix to model the delta weight matrix. This approach aims to decouple the rank of the delta weight matrix from the number of learnable parameters, allowing for high-rank adaptation while maintaining a limited parameter size. By leveraging the Fast Fourier Transform (FFT) in both the forward and backward passes, C^3A achieves significant computational and memory efficiency. Consequently, C^3A presents itself as a compelling alternative to LoRA for model fine-tuning.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. I will appreciate additional analysis on how initialization affects model convergence speed and accuracy across varied tasks. Specifically, experiments with detailed metric comparisons for convergence time under different initializations would strengthen this claim.

2. The method lacks experimental validation on tasks involving significantly higher dimensions or complexity, where circular convolution may face generalization challenges. We encourage the authors to include experiments on larger-scale datasets or more complex tasks to confirm C3A's adaptability and performance.

3. How different block sizes may impact both parameter efficiency and performance across tasks? We request the authors to provide guidelines for block size selection.

4. Aside from the existing comparison with baseline methods LoRA and VeRA, we suggest authors to Include more recent PEFT methods as baselines, as that will provide a more comprehensive understanding of C3A's standing relative to broader industry benchmarks.

5. Please elaborates the limitations of the proposed Block-Circular Convolution.
soundness: 3
strengths: 1. High-Rank Adaptation with Low Parameters: C3A achieves high-rank adaptation without increasing the number of trainable parameters. By leveraging circular convolution, it avoids the limitations of low-rank adaptation in methods like LoRA, maintaining expressive power with fewer resources.
﻿
2. Efficient Use of FFT for Computational Gains. The use of Fast Fourier Transform (FFT) in both forward and backward propagation enhances computational speed and reduces complexity, achieving a time complexity comparable to LoRA but with better efficiency in memory use.
﻿
3. Memory Savings with Block-Circular Convolution. By implementing block-circular convolution, C3A allows for flexible control of trainable parameters. This adaptability to block size enables parameter tuning based on task requirements, optimizing both memory and computational resources.
﻿
4. Comprehensive Benchmark Performance. C3A demonstrates robustness performance across multiple tasks, including GLUE benchmarks for NLP and Vision Transformer (ViT) for image classification, showing significant improvements in accuracy with fewer parameters.
﻿
5. Adaptable Parameter Count with Minimal Performance Trade-Offs. C3A allows for adjustable parameter configurations through its block-circular design, offering a range of parameter counts without significant sacrifices in performance, providing versatility across tasks.
summary: This work introduces Circular Convolution Adaptation (C3A) as an advanced parameter-efficient fine-tuning (PEFT) technique, addressing limitations of existing methods like Low-Rank Adaptation. C3A leverages the circular convolution operator, enabling high-rank adaptation without a proportional increase in trainable parameters. Utilizing Fast Fourier Transform (FFT) operations, C3A achieves efficient memory and computational performance while retaining expressive power. The method effectively reduces memory requirements and computational load compared to high-parameter alternatives like VeRA. A notable strength of C3A is its flexibility in adjusting the number of trainable parameters through block-circular convolution, which adapts to diverse downstream tasks. Experimental results validate C3A's superior accuracy and memory efficiency across various tasks.
weaknesses: 1. Although C3A claims robustness to initialization methods, the paper does not thoroughly analyze how different initializations may impact convergence speed or model performance across varying tasks. Additional experiments on initialization sensitivity could further validate this robustness claim and clarify any subtle effects on fine-tuning stability.
﻿
2. While the method demonstrates efficiency on moderate NLP and CV benchmarks, it lacks evaluation on tasks with substantially higher dimensions or complexity, where circular convolution’s rank flexibility may not be as effective. Testing C3A on such high-dimensional data would strengthen claims about its scalability.
﻿
3. Inadequate Justification for Block Size Selection: The paper introduces block-circular convolution with adaptable block sizes but provides minimal guidance on how to select optimal block sizes for specific tasks. This omission makes it challenging for readers to understand the trade-offs and practical considerations in choosing block sizes to balance accuracy and parameter efficiency.
﻿
6. The proposed method relies on block-circular convolution for flexibility, but the paper does not clearly address its limitations, such as potential rank deficiencies or performance degradation in non-square matrices. Expanding on these limitations and detailing their implications would provide a more balanced view of C3A’s adaptability across different architectures.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Given that the results don’t dominate existing methods in many scenarios, what would be the ideal scenario/environment/conditions in which the proposed method is most effective?
2. Is there an ablation study for the block size (b)? I understand that it is constrained because of the divisibility, but I believe it is important to see some insights about its impact on the result.
3. Can you present more insights/analysis about the decision and the impact of using FFT (e.g., some ablation for the FFT part on its own)?
soundness: 3
strengths: 1. Providing an interesting approach for low-rank adaptation that relies on FFT instead of the typical residual matrix to try to address existing limitations.
2. The extension to block-circular convolution allows for better control over parameter numbers (and makes the method applicable to non-square matrix weights) without compromising rank.
3. Providing detailed practical implementation and foundation for the different components (e.g., circular convolution)
summary: The paper proposes a new parameter-efficient fine-tuning (PEFT) method (denoted C³A) that leverages circulant matrices and Fast Fourier Transform (FFT) to achieve high-rank adaptations while maintaining computational efficiency. The proposed method allows independent control of the rank and number of trainable parameters through block-circular convolution. The authors evaluate their method across various NLP and vision tasks, including NLP, comparing the performance of C³A to existing PEFT.
weaknesses: 1. While using circular convolution and FFT in PEFT is interesting, the improvement over LoRA and similar methods may be seen as incremental, with modest/marginal gains rather than a significant improvement over existing PEFT methodologies.
2. The method's reliance on FFT may introduce implementation complexity, particularly in environments with limited support for these operations, which could restrict the practical applicability of the method.
3. Many of the presented results don’t show clear gains or dominance, which questions the reliability of the presented method, which is the main weakness of the paper.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 4
questions: 1. Can you compare with the state-of-the-art FourierFT method (https://icml.cc/virtual/2024/poster/33821), which is also a Fourier domain based method? 
2. Can you provide memory cost details in Table 3 and Table 4, as you did for Table 2?
3. Need a breakdown of the memory cost of proposed method, since in Table 2, more parameters somehow results in less GPU memory for both base and large models. 
4. Initialization test for large FFT size setting like LLaMA for section 4.5
5. FFT operations can easily result in nan/inf values in low precision computation. Does the proposed method actually run in half precision (as LLMs are stored) or full precision?
soundness: 2
strengths: 1. Experimental results are very promising, and evaluated models cover both the base and large version of RoBERTa and ViT, also including  LLaMA-2 and LLaMA-3. 
2. There is also synthetic data experiment which is simple and easy to demonstrate the effectiveness of proposed method against LoRA and full fine-tuning.
summary: This paper borrows the block circulant method from existing work and applies to LLMs. The proposed method is about using block circulant matrix for training which has been already published in the referenced work (Ding et al., 2017). However, the entire method writing (section 3) does not acknowledge/cite the work at all and instead present as if authors created block circulant based training. As a result, the major contribution narrows down to the empirical verification of block circulant training method on different large models.
weaknesses: 1. No technical novelty, because the proposed method is borrowed from the referenced work (Ding et al., 2017)
2. Lack of analysis regarding the impact of block size setting, e.g., smaller block size gets more parameters but smaller GPU cost (Table 2)? Why more parameters result in smaller GPU cost? According to theoretical space analysis (section 3.5.2), with d1=d2=768, the formulation cannot explain the observation. 
3. FFT based solution is naturally sensitive to intialization, as a small change in frequence domain can be easily spreaded to all time domain. The experimental design in section 4.5 should include results on LLaMA models, as the FFT size is large in LLaMA models. The hypothesis is inclined to be that intialization for FFT based method is sensitive in the case of large models or large FFT size settings.

Meta-review: (a) Summary

The paper proposes to use circular convolution as an alternative to the low-rank AB matrices in LoRA/PEFT. The method is called Circular Convolution Adaptation (C3A). It maintains higher rank than LoRA, with more efficiency benefits. It also outperforms LoRA variants.

(b) Strengths

Consistent performance improvement; higher-rank adaptation than LoRA; improved efficiency

(c) Weaknesses

Limited experiments with large models; more complexity (including initialization methods) than other LoRA variants (hence potential limited usage); not applicable to non-square matrix; marginal improvement in most cases

(d) Reasons for decision

Not enough support from reviewers; higher complexity outweighs the marginal benefits; lack of experiments on large models.

Decision: Reject