Parameter-Efficient Fine-Tuning via Circular Convolution
Authors: Aochuan Chen, Jiashun Cheng, Zijing Liu, Ziqi Gao, Fugee Tsung, Yu Li, Jia Li
Accepted at: acl25

Parameter-Efficient Fine-Tuning via Circular Convolution

Aochuan Chen*†, Jiashun Cheng*†, Zijing Liuidea, Ziqi Gao†, Fugee Tsung†, Yu Liidea, Jia Li†‡
†The Hong Kong University of Science and Technology (Guangzhou)
The Hong Kong University of Science and Technology
ideaInternational Digital Economy Academy
jialee@hkust-gz.edu.cn

Abstract

Low-Rank Adaptation (LoRA) has gained popularity for fine-tuning large foundation models, leveraging low-rank matrices \( \mathbf{A} \) and \( \mathbf{B} \) to represent weight changes (\emph{i.e.}, \( \Delta \mathbf{W} = \mathbf{B} \mathbf{A} \)). This method reduces trainable parameters and mitigates heavy memory consumption associated with full delta matrices by sequentially multiplying \( \mathbf{A} \) and \( \mathbf{B} \) with the activation. Despite its success, the intrinsic low-rank characteristic may limit its performance. Although several variants have been proposed to address this issue, they often overlook the crucial computational and memory efficiency brought by LoRA. In this paper, we propose Circular Convolution Adaptation (C\(^3\)A), which not only achieves high-rank adaptation with enhanced performance but also excels in both computational power and memory utilization. Extensive experiments demonstrate that C\(^3\)A consistently outperforms LoRA and its variants across various fine-tuning tasks.

1 Introduction

In recent years, Large Foundation Models (LFMs) have witnessed a pronounced ascendance in both scholarly and practical realms, attributable to their exceptional efficacy across diverse tasks in natural language processing (NLP) (Brown et al., 2020; Touvron et al., 2023), computer vision (CV) (Radford et al., 2021; Kirillov et al., 2023), and other domains (Li et al., 2024a,b, 2025). Distinguished by an extensive parameter count and significant computational requisites, these models have established unprecedented benchmarks in both accuracy and versatility. Nonetheless, their considerable size and intricate structure present formidable obstacles for efficient fine-tuning, especially within resource-constrained environments (Malladi et al., 2023; Zhang et al., 2024b). To mitigate these challenges, parameter-efficient fine-tuning (PEFT) techniques (Mangrulkar et al., 2022), exemplified by Low-Rank Adaptation (LoRA) (Hu et al., 2021), have emerged as highly effective solutions.

LoRA reduces the number of trainable parameters by leveraging low-rank matrices to approximate alterations in weights, thereby facilitating fine-tuning without degrading the model’s efficacy. Specifically, LoRA can be articulated mathematically as follows:

\[
\mathbf{W} \mathbf{x} = (\mathbf{W}_0 + \Delta \mathbf{W}) \mathbf{x} = \mathbf{W}_0 \mathbf{x} + \mathbf{B}(\mathbf{A} \mathbf{x}),
\]

where \( \mathbf{W}, \mathbf{W}_0, \Delta \mathbf{W} \in \mathbb{R}^{d_1 \times d_2} \) are weight matrices, \( \mathbf{B} \in \mathbb{R}^{d_1 \times r}, \mathbf{A} \in \mathbb{R}^{r \times d_2} \) are low-rank matrices formulated to construct \( \Delta \mathbf{W} \), and \( \mathbf{x} \in \mathbb{R}^{d_2} \) are the activations. The number of trainable parameters is \( r(d_1 + d_2) \), thereby motivating the selection of \( r \ll \min(d_1, d_2) \) (\emph{e.g.}, \( r = 8 \) for \( d_1 = d_2 = 1024 \)) to attain elevated parameter efficiency. Nonethe-

![Performance comparison of C^3A and other methods relative to LoRA on LLaMA-8B. Higher values are better across all metrics. See Table 3 and Table 4 for more statistics.](page_1012_370_367_367.png)

Figure 1: Performance comparison of C\(^3\)A and other methods relative to LoRA on LLaMA-8B. Higher values are better across all metrics. See Table 3 and Table 4 for more statistics.
less, as elaborated by Zeng and Lee (2023), the potential of LoRA to encapsulate a target model is inherently constrained by \( r \). In an effort to reconcile the dichotomy between performance and efficiency, Kopiczko et al. (2023) introduced Vector Random Matrix Adaptation (VeRA). VeRA attains comparable performance with a markedly reduced count of trainable parameters via fixed random-matrix projections. However, despite its minimal parameter count, VeRA demands considerable computational resources and memory capacity due to the extensive nature of the random matrices employed for projection. As depicted in Figure 1, other representative works share the same resource problem. This precipitates the following open research question within the scope of PEFT:

Beyond low parameter counts, how to achieve high-rank adaptation without incurring significant costs of time and memory?

To address this question, we introduce Circular Convolution Adaptation (C\(^3\)A), which incorporates the circular convolution operator (Bamieh, 2018). Circular convolution has garnered significant attention in both signal processing (Li et al., 2020) and cryptography (Dworkin et al., 2001) due to its exceptional efficiency and compactness. This operator can be equivalently expressed as multiplication by a circulant matrix, providing rank flexibility that is independent of the number of trainable parameters. Furthermore, by employing the Fast Fourier Transform (FFT), C\(^3\)A achieves superior time and memory efficiency compared to the direct multiplication (Bamieh, 2018), which makes it competitive with LoRA in terms of efficiency.

In addition, as explicated by Dosovitskiy et al. (2020), dense linear layers exhibit a deficiency of inductive biases, engendering a complex optimization landscape. Consequently, this hampers the effectiveness of transformers in comparison to Convolutional Neural Networks (CNNs) under conditions of limited data availability. Within the framework of a constrained training dataset for the downstream task, we postulate that a robust inductive bias could potentially augment adaption performance. The circular pattern in C\(^3\)A serves precisely as such an inductive bias.

In summary, circular convolution presents a promising solution for circumventing the rank limitations of LoRA at minimal costs. Our contributions can be summarized as follows:

① We introduce C\(^3\)A, a novel approach for PEFT. This method leverages the circular convolution operation and its equivalent circulant matrix to provide a flexible rank, which is free of linear constraint by the number of trainable parameters.

② Leveraging the elegant diagonalization of the circulant matrix, we implement both the forward pass and backpropagation using FFT. With the incorporation of FFT, the computation and memory efficiency of C\(^3\)A excels. C\(^3\)A strikes a unique balance between performance an efficiency.

③ To offer greater flexibility in controlling the number of trainable parameters, we extend C\(^3\)A by incorporating block-circular convolution, which results in block-circulant matrices. This extension allows C\(^3\)A to achieve fully customizable parameter counts as well as adaptable rank configurations.

④ We validate C\(^3\)A through comprehensive fine-tuning experiments across diverse tasks, which demonstrate C\(^3\)A’s outstanding accuracy and memory merits compared to existing methods.

2 Related Work

2.1 Parameter-Efficient Fine-Tuning

Research on PEFT has generally progressed along three main directions. The first direction involves partially updating the pre-trained neural network (e.g., the layer norm (Basu et al., 2024) or the biases (Zaken et al., 2021)). Traditional methods relied on hand-crafted heuristics (Raghu et al., 2019) to identify which parameters are crucial and should be fine-tuned. More advanced approaches employ optimization techniques (Guo et al., 2020; Xu et al., 2021; Fu et al., 2023). For example, Guo et al. (2020) reformulated such a discrete optimization problem into a continuous one by employing Bernoulli masks and the Gumbel-softmax approximation (Jang et al., 2016).

The second direction emerged to maintain the integrity of the pre-trained model while enabling a high degree of parameter sharing through adapter-based methods (He et al., 2021; Rebuffi et al., 2017; Rücklé et al., 2020; Liu et al., 2022; Lian et al., 2022). These works focus on integrating additional modules, termed adapters, to fit the downstream task, effectively decoupling the pre-trained model parameters from those specific to the downstream task. Prompt Tuning (Brown et al., 2020; Gao et al., 2020; Chen et al., 2023; Zhang et al., 2024a) and Prefix Tuning (Li and Liang, 2021; Jia et al., 2022) also fall into this category, despite ignoring potential semantic meanings.
The final direction is characterized by delta-weight-based methods, such as Low-Rank Adaptation (LoRA) (Hu et al., 2021) and Orthogonal Fine-tuning (OFT) (Qiu et al., 2023). These methods bridge the gap between the pre-trained model and the downstream task by adaptive delta weights, which are stored separately while used in combination with the pre-trained weights. This unique design enables disentanglement of the pretrained and downstream-specific weights. Namely, it achieves parameter sharing and preserves the ability to integrate without additional inference cost. LoRA models the delta-weights by an additive matrix while OFT does it by a multiplicative one. To further improve either parameter efficiency or performance, many variants has been proposed for both of the methods (Kopiczko et al., 2023; Liu et al., 2024b, 2023; Yuan et al., 2024; Hayou et al., 2024b). However, these methods can hardly achieve high parameter efficiency and performance without incurring heavy computation and memory usage.

2.2 Circular Convolution

Circular convolution has been extensively studied in signal processing (Rabiner et al., 1978; McGillem and Cooper, 1984; Li et al., 2020) and cryptography (Dworkin et al., 2001; Gong et al., 2024). Owing to its computational advantages, circular convolution has also been explored in machine learning for generating long embeddings of high-dimensional data (Yu et al., 2014) and compressing heavily parameterized layers (Cheng et al., 2015; Ding et al., 2017). Remarkably, it achieves these efficiencies without significant performance degradation, which makes it a promising technique for fine-tuning applications.

Despite its success in small neural networks such as LeNet (Cheng et al., 2015), circular convolution has not demonstrated lossless performance in modern large foundational models (LFMs) or even in their base architecture, the transformer. This limitation may be attributed to the conflict between its high intrinsic bias (\( i.e. \), the circulant pattern) and the vast amount of data required for training LFMs. Conversely, when fine-tuning LFMs, it is often impractical to collect as much data as needed for training from scratch. In such scenarios, the intrinsic bias of circular convolution could potentially serve as a regularization mechanism, thereby benefiting the optimization process of fine-tuning.

3 Method

In this section, we present C\(^3\)A (see an overview in Figure 2), a novel PEFT approach based on the circular convolution. C\(^3\)A follows LoRA’s setting of learning an additive linear operation over the original dense linear transformation. However, instead of using low-rank decomposition and the matrix multiplication operator, C\(^3\)A resorts to circular convolution as this additive linear operation.

3.1 Notations

The adapted weight matrix, the original weight matrix, and the delta matrix are denoted by \( \mathbf{W}, \mathbf{W}_0, \) and \( \Delta \mathbf{W} \), respectively (\( \mathbf{W}, \mathbf{W}_0, \Delta \mathbf{W} \in \mathbb{R}^{d_1 \times d_2} \)). The activation vector of the previous layer is denoted by \( \mathbf{x} \in \mathbb{R}^{d_2} \). The post-transformation vector is \( \mathbf{z} \), where \( \mathbf{z} = \mathbf{W} \mathbf{x} \in \mathbb{R}^{d_1} \), and the incremental part is denoted by \( \Delta \mathbf{z} \), where \( \Delta \mathbf{z} = \Delta \mathbf{W} \mathbf{x} \in \mathbb{R}^{d_1} \). The matrices \( \mathbf{A} \) and \( \mathbf{B} \) are low-rank matrices introduced by LoRA to represent \( \Delta \mathbf{W} \), with \( r \) being their rank. \( r_v \) specifies the rank of the random projection matrix used in VeRA. The circular convolution kernel of C\(^3\)A is denoted by \( \Delta \mathbf{w} \) and the circular convolution operator by \( * \). The loss function is represented by \( \mathcal{L} \). The Fast Fourier Transform and its inverse are denoted by FFT and iFFT. The Hadamard product is denoted by \( \circ \).

3.2 Circular Convolution

Firstly, for simplicity, we assume \( d_1 = d_2 = d \) and \( \Delta \mathbf{w} \in \mathbb{R}^d \). The circular convolution operator is defined as \( \Delta \mathbf{z} = \Delta \mathbf{w} * \mathbf{x} = \mathcal{C}(\Delta \mathbf{w}) \mathbf{x} \), where \( \mathcal{C}(\cdot) \) is a function which takes a vector and outputs the corresponding circulant matrix. Concretely, the first row of \( \mathcal{C}(\Delta \mathbf{w}) \) is \( \Delta \mathbf{w} \) and the following rows are equal to the row above them periodically shifted to the right by one element. In math,

\[
\mathcal{C}(\Delta \mathbf{w}) = \begin{bmatrix}
\Delta w_1 & \Delta w_2 & \cdots & \Delta w_{d-1} & \Delta w_d \\
\Delta w_d & \Delta w_1 & \cdots & \Delta w_{d-2} & \Delta w_{d-1} \\
\cdots & \cdots & \cdots & \cdots & \cdots \\
\Delta w_3 & \Delta w_4 & \cdots & \Delta w_1 & \Delta w_2 \\
\Delta w_2 & \Delta w_3 & \cdots & \Delta w_d & \Delta w_1
\end{bmatrix}
\]

Theoretically, the rank of \( \mathcal{C}(\Delta \mathbf{w}) \) is given by \( d - \mathrm{Deg}(\gcd(f(x), x^d - 1)) \) (Ingleton, 1956), where \( \mathrm{Deg}(\cdot) \) denotes the degree of a polynomial, \( f(x) \) is the polynomial associated with \( \Delta \mathbf{w} \) (\( i.e. \), \( f(x) = \sum_{i=1}^d \Delta w_i x^{i-1} \)), and \( \gcd(\cdot) \) represents the greatest common divisor. Consequently, the theoretical upper bound on the rank of \( \mathcal{C}(\Delta \mathbf{w}) \) is \( d \). By learning \( \Delta \mathbf{w} \) in the \( \mathbb{R}^n \) oracle, C\(^3\)A automatically achieves dynamic rank selection, which is
Figure 2: Overview of LoRA (A) and our C^3A (B,C) method. In LoRA, only low-rank matrices A and B are trained and the delta weight is represented by their product (i.e., \( \Delta W = BA \)). The total trainable parameter number is \( r(d_1 + d_2) \), which is associated with the rank of the delta weight. In C^3A, circular convolution kernels \( \Delta w \) are tuned to adapt to the downstream task and the delta weight is represented by the (block-)circular matrix they construct (i.e., \( \Delta W = \mathcal{C}_{(blk)}(\Delta w) \)). The total trainable parameter count is \( \frac{d_1 d_2}{b} \), which disentangles with the rank of the delta weight. Here, b is the block size and it should be a common divisor (CD) of \( d_1 \) and \( d_2 \).

not linearly constrained by the number of learnable parameters, unlike LoRA.

To achieve high efficiency, enlightened by Ding et al. (2017), we leverage the beautiful circulant structure of \( \mathcal{C}(\Delta w) \), which makes it diagonalizable by the Fourier basis (F). In math, it can be described as \( \mathcal{C}(\Delta w) = F \Lambda F^{-1} \) (Golub and Van Loan, 1996), where \( \Lambda \) is its eigenvalues and can be calculated by a Fourier transform of the first row (i.e., \( \Lambda = \mathrm{diag}(F\Delta w) \)). Therefore, we can calculate \( \Delta w * x \) as

\[
\Delta w * x = F \mathrm{diag}\left(\frac{F \Delta w}{d}\right) F^{-1} x \\
= \mathrm{FFT}(\mathrm{FFT}(\Delta w) \circ \mathrm{iFFT}(x)).
\]

(1)

3.3 Backpropagation

To effectuate backpropagation with optimal efficiency, it is imperative to obtain the analytical derivatives of the loss function \( \mathcal{L} \) with respect to \( \Delta w \) and \( x \). Following the approach outlined in Ding et al. (2017), we aim to explain backpropagation using simpler language. By applying the chain rule, these derivatives are delineated as follows:

\[
\frac{\partial \mathcal{L}}{\partial x} = \frac{\partial \Delta z}{\partial x} \frac{\partial \mathcal{L}}{\partial \Delta z}, \quad \frac{\partial \mathcal{L}}{\partial \Delta w} = \frac{\partial \Delta z}{\partial \Delta w} \frac{\partial \mathcal{L}}{\partial \Delta z}.
\]

(2)

Given that \( \Delta z = \mathcal{C}(\Delta w)x \), it logically follows that \( \frac{\partial \Delta z}{\partial x} = \mathcal{C}(\Delta w) \). Concerning \( \frac{\partial \Delta z}{\partial \Delta w} \), we observe the commutative property of the circular convolution operation (i.e., \( \mathcal{C}(\Delta w)x = \mathcal{C}(x)\Delta w \)), which implies \( \frac{\partial \Delta z}{\partial \Delta w} = \mathcal{C}(x) \). Substituting these findings into Equation 2, we derive:

\[
\frac{\partial \mathcal{L}}{\partial x} = \mathcal{C}(\Delta w) \frac{\partial \mathcal{L}}{\partial \Delta z}, \qquad \frac{\partial \mathcal{L}}{\partial \Delta w} = \mathcal{C}(x) \frac{\partial \mathcal{L}}{\partial \Delta z}.
\]

These expressions can also be interpreted as circular convolutions:

\[
\frac{\partial \mathcal{L}}{\partial x} = \Delta w * \frac{\partial \mathcal{L}}{\partial \Delta z}; \qquad \frac{\partial \mathcal{L}}{\partial \Delta w} = x * \frac{\partial \mathcal{L}}{\partial \Delta z}.
\]

By meticulously executing this derivative computation in accordance with Equation 1, backpropagation can harness the computational efficacy facilitated by the FFT algorithm.

3.4 Block-Circular Convolution

Notwithstanding the elegance and efficiency of the circular convolution operator, it is subject to two fundamental limitations stemming from the constraint that the convolution kernel must match the dimensions of the activation vector: ① It is inapplicable to non-square weight matrices. ② The count of learnable parameters remains fixed. The first restriction hampers its applicability in scenarios such as fine-tuning a LLaMA3-8B model, where the weight matrix dimensions include 4096 × 1024. The second constraint diminishes the adaptability of C^3A, presenting challenges in addressing complex downstream tasks that necessitate a greater number of learnable parameters. To mitigate these limitations, we employ block-circular convolution (Ding et al., 2017). By partitioning the activation vector \( x \) and the post-transformation vector \( \Delta z \)
into blocks of identical size, unique convolution kernels can be allocated to each pair of these blocks. Specifically,

\[
\mathbf{x} = \begin{bmatrix}
x_1 & x_2 & \cdots & x_{\frac{d_1}{b}}
\end{bmatrix}
\]
\[
\Delta \mathbf{z} = \begin{bmatrix}
\Delta z_1 & \Delta z_2 & \cdots & \Delta z_{\frac{d_1}{b}}
\end{bmatrix},
\]

where \( b \) is the block size and \( b \) need to be a common divisor of \( d_1 \) and \( d_2 \). We will need \( \frac{d_1 d_2}{b^2} \) convolution kernels to densely connect these blocks, which can be expressed in math as

\[
\Delta z_i = \sum_{j=1}^{\frac{d_2}{b}} \Delta w_{ij} * x_j, i \in \{1, 2, \ldots, \frac{d_1}{b}\}.
\]

This calculation can be represented by a block-circular matrix:

\[
\Delta \mathbf{z} = C_{blk}(\Delta \mathbf{w}) \mathbf{x}
\]
\[
C_{blk}(\Delta \mathbf{w}) = \begin{bmatrix}
C(\Delta w_{11}) & \cdots & C(\Delta w_{1 \frac{d_2}{b}}) \\
C(\Delta w_{21}) & \cdots & C(\Delta w_{2 \frac{d_2}{b}}) \\
\cdots & \cdots & \cdots \\
C(\Delta w_{\frac{d_1}{b} 1}) & \cdots & C(\Delta w_{\frac{d_1}{b} \frac{d_2}{b}})
\end{bmatrix}.
\]

We refer our readers to Algorithm A1 in Appendix A for a Pytorch implementation. In this context, \( \Delta w_{ij} \in \mathbb{R}^b \), and it follows that \( \frac{d_1 d_2}{b^2} = \frac{d_1 d_2}{b} \) represents the number of learnable parameters. Notably, the parameter \( b \) serves as a hyperparameter modulating the quantity of learnable parameters, analogous to the role of \( r \) in LoRA. It is imperative to distinguish, however, that whereas \( r \) simultaneously governs the rank of the delta matrix and the number of learnable parameters, \( b \) exclusively influences the latter. This disentanglement of matrix rank and parameter count facilitates greater adaptability and potentially yields superior outcomes.

3.5 Complexity Analysis

We compare the time complexity and space complexity of LoRA, VeRA and C$^3$A in Table 1. Detailed analysis follows in this section.

3.5.1 Time Complexity

LoRA integrates low-rank matrices \( \mathbf{A} \) and \( \mathbf{B} \), which are successively multiplied with the activation vector, resulting in a computational complexity of \( \mathcal{O}(r(d_1 + d_2)) \). Generally, \( r \ll \min(d_1, d_2) \). In contrast, VeRA, despite its high-rank structure and relatively few trainable parameters, suffers from a prohibitive computational complexity of \( \mathcal{O}(r_v(d_1 + d_2)) \), where \( r_v \) can exceed \( \max(d_1, d_2) \). Consequently, striking an optimal balance between high rank and computational efficiency remains an elusive task.

On GPUs, the cuFFT backend automatically parallelizes FFT operations along the axes not being transformed, with the degree of parallelism \( p \) determined by the available resources. Thanks to the \( \mathcal{O}(n \log n) \) complexity of the FFT algorithm used in Equation 1, C$^3$A achieves a time complexity of \( \mathcal{O}(\frac{(d_1 + d_2)}{p} \log b + \frac{d_1 d_2}{b}) \). The first term is the time complexity for FFT and the second term is for aggregation. In practical scenarios, \( b \) is chosen as the greatest common divisor of \( d_1 \) and \( d_2 \) to achieve a high compression ratio. Given that, C$^3$A is comparable to LoRA in time complexity.

<table>
  <tr>
    <th>Method</th>
    <th>Time</th>
    <th colspan="2">Space</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th># Param</th>
    <th># Other</th>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>\( \mathcal{O}(r(d_1 + d_2)) \)</td>
    <td>\( r(d_1 + d_2) \)</td>
    <td>0</td>
  </tr>
  <tr>
    <td>VeRA</td>
    <td>\( \mathcal{O}(r_v(d_1 + d_2)) \)</td>
    <td>\( r_v + d_1 \)</td>
    <td>\( r_v(d_1 + d_2) \)</td>
  </tr>
  <tr>
    <td>C$^3$A</td>
    <td>\( \mathcal{O}(\frac{d_1 + d_2}{p} \log b + \frac{d_1 d_2}{b}) \)</td>
    <td>\( \frac{d_1 d_2}{b} \)</td>
    <td>\( pb \)</td>
  </tr>
</table>

Table 1: Time and space complexity comparison of LoRA, VeRA and C$^3$A. We split the space complexity into Parameter number and Other auxiliary tensors to help better understand the differences. We highlight that in practice, to achieve similar performance, \( \frac{\max(d_1, d_2)}{b} \leq r \ll r_v \).

3.5.2 Space Complexity

We analyze the space complexity of LoRA, VeRA, and C$^3$A during training. The differences among these methods primarily arise from the trainable parameters and the auxiliary tensors required for the forward pass and backpropagation. LoRA does not rely on auxiliary tensors, while VeRA necessitates 2 random projection matrices, with a total size of \( r_v(d_1 + d_2) \). Since \( r_v \) is by no means negligible, the memory usage of VeRA is significantly larger than that of LoRA.

In terms of C$^3$A, the only additional auxiliary tensor would be of size \( pb \leq \min(d_1, d_2) \), which is reserved by the FFT algorithm. By selecting an appropriate \( b \), which is often close to the greatest common divisor of \( d_1 \) and \( d_2 \), the space complexity of C$^3$A is optimized. Furthermore, because \( p \) scales with the available resources, the algorithm inherently manages dynamic memory consumption without additional effort.

4 Experiment

Baselines. We compare our C$^3$A with several representative PEFT methods, including BitFit (Zaken
<table>
  <tr>
    <th>Methods</th>
    <th># Params</th>
    <th>Mem</th>
    <th>SST-2</th>
    <th>MRPC</th>
    <th>CoLA</th>
    <th>QNLI</th>
    <th>RTE</th>
    <th>STS-B</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>Full</td>
    <td>124M</td>
    <td>17.19G</td>
    <td>94.01±0.39</td>
    <td><b>87.10±0.79</b></td>
    <td>62.00±1.16</td>
    <td>92.40±0.28</td>
    <td><b>77.33±2.68</b></td>
    <td><b>90.70±0.14</b></td>
    <td><b>83.92</b></td>
  </tr>
  <tr>
    <td>BitFit</td>
    <td>0.102M</td>
    <td>12.60G</td>
    <td>93.30±0.30</td>
    <td>85.80±0.21</td>
    <td>59.21±1.74</td>
    <td>91.96±0.18</td>
    <td>73.07±1.34</td>
    <td>90.18±0.17</td>
    <td>82.25</td>
  </tr>
  <tr>
    <td>(IA)<sup>3</sup></td>
    <td>0.111M</td>
    <td>19.86G</td>
    <td>92.98±0.34</td>
    <td>85.86±0.59</td>
    <td>60.49±1.09</td>
    <td>91.56±0.17</td>
    <td>69.10±1.18</td>
    <td>90.06±0.21</td>
    <td>81.67</td>
  </tr>
  <tr>
    <td>LoRA<sub>r=8</sub></td>
    <td>0.295M</td>
    <td>13.75G</td>
    <td><b>94.50±0.41</b></td>
    <td>85.68±0.74</td>
    <td>60.95±1.57</td>
    <td><b>92.54±0.20</b></td>
    <td>76.68±1.42</td>
    <td>89.76±0.39</td>
    <td>83.35</td>
  </tr>
  <tr>
    <td>VeRA<sub>r=1024</sub></td>
    <td>0.043M</td>
    <td>15.51G</td>
    <td>93.97±0.17</td>
    <td>86.23±0.41</td>
    <td>62.24±1.91</td>
    <td>91.85±0.17</td>
    <td>75.74±1.56</td>
    <td>90.27±0.25</td>
    <td>83.38</td>
  </tr>
  <tr>
    <td>BOFT<sub>b=8</sub></td>
    <td>0.166M</td>
    <td>14.11G</td>
    <td>93.23±0.50</td>
    <td>84.37±0.54</td>
    <td>59.50±1.25</td>
    <td>91.69±0.12</td>
    <td>74.22±0.84</td>
    <td>89.63±0.37</td>
    <td>82.11</td>
  </tr>
  <tr>
    <td>C<sup>3</sup>A<sub>b=768/1</sub></td>
    <td>0.018M</td>
    <td>12.83G</td>
    <td>93.42±0.26</td>
    <td>86.33±0.32</td>
    <td>61.83±0.96</td>
    <td>91.83±0.04</td>
    <td>76.17±1.39</td>
    <td>90.46±0.29</td>
    <td>83.34</td>
  </tr>
  <tr>
    <td>C<sup>3</sup>A<sub>b=768/6</sub></td>
    <td>0.111M</td>
    <td>12.72G</td>
    <td>94.20±0.16</td>
    <td>86.67±0.54</td>
    <td><b>62.48±1.20</b></td>
    <td>92.32±0.25</td>
    <td>77.18±1.41</td>
    <td>90.16±0.42</td>
    <td>83.84</td>
  </tr>
  <tr>
    <td colspan="10">BASE</td>
  </tr>
  <tr>
    <td>Full</td>
    <td>354M</td>
    <td>43.40G</td>
    <td>95.75±0.45</td>
    <td><b>88.35±0.64</b></td>
    <td>64.87±1.25</td>
    <td>92.40±0.28</td>
    <td>84.48±1.14</td>
    <td>91.65±0.14</td>
    <td>86.25</td>
  </tr>
  <tr>
    <td>BitFit</td>
    <td>0.271M</td>
    <td>30.65G</td>
    <td>95.09±0.27</td>
    <td>88.10±0.76</td>
    <td>65.40±0.76</td>
    <td>94.06±0.14</td>
    <td>82.60±1.15</td>
    <td>91.73±0.20</td>
    <td>86.16</td>
  </tr>
  <tr>
    <td>(IA)<sup>3</sup></td>
    <td>0.295M</td>
    <td>48.81G</td>
    <td>95.32±0.20</td>
    <td>87.06±0.57</td>
    <td>66.52±1.10</td>
    <td>94.18±0.15</td>
    <td>84.33±2.38</td>
    <td>91.58±0.39</td>
    <td>86.50</td>
  </tr>
  <tr>
    <td>LoRA<sub>r=8</sub></td>
    <td>0.786M</td>
    <td>34.12G</td>
    <td>95.53±0.35</td>
    <td>86.12±0.86</td>
    <td>65.16±0.76</td>
    <td>93.73±0.30</td>
    <td>83.75±0.51</td>
    <td>91.46±0.21</td>
    <td>85.96</td>
  </tr>
  <tr>
    <td>VeRA<sub>r=256</sub></td>
    <td>0.061M</td>
    <td>34.16G</td>
    <td><b>95.83±0.43</b></td>
    <td>87.72±0.55</td>
    <td>63.66±1.45</td>
    <td>94.11±0.20</td>
    <td>83.03±1.65</td>
    <td>91.12±0.37</td>
    <td>85.91</td>
  </tr>
  <tr>
    <td>BOFT<sub>b=8</sub></td>
    <td>0.442M</td>
    <td>34.98G</td>
    <td>95.76±0.41</td>
    <td>88.28±0.33</td>
    <td>64.72±2.37</td>
    <td>93.89±0.14</td>
    <td>82.82±1.40</td>
    <td>91.03±0.32</td>
    <td>86.08</td>
  </tr>
  <tr>
    <td>C<sup>3</sup>A<sub>b=1024/1</sub></td>
    <td>0.049M</td>
    <td>31.83G</td>
    <td>95.78±0.08</td>
    <td>88.02±0.62</td>
    <td>66.59±1.20</td>
    <td>94.22±0.25</td>
    <td>82.89±0.67</td>
    <td><b>91.86±0.14</b></td>
    <td>86.56</td>
  </tr>
  <tr>
    <td>C<sup>3</sup>A<sub>b=1024/8</sub></td>
    <td>0.393M</td>
    <td>31.79G</td>
    <td>95.78±0.15</td>
    <td>88.09±0.47</td>
    <td><b>67.18±1.92</b></td>
    <td><b>94.26±0.19</b></td>
    <td><b>84.62±1.36</b></td>
    <td>91.81±0.36</td>
    <td><b>86.96</b></td>
  </tr>
  <tr>
    <td colspan="10">LARGE</td>
  </tr>
</table>

Table 2: Performance of different PEFT methods on the GLUE benchmark. We fine-tune pre-trained RoBERTa-Base and -Large models on 6 datasets. We report the Matthew’s Correlation Coefficient (MCC) for CoLA, Pearson Correlation Coefficient (PCC) for STS-B, and accuracy (Acc.) for all the remaining tasks. For each metric, a higher score indicates better performance. “Avg.” denotes the average score of each method across all datasets. The best results for each dataset are highlighted in bold. # Params does not include the classification head since each method uses a head of the same size. Memory cost (Mem) is measured on fixed length (\( i.e., 256 \)) data with a batchsize of 64.

et al., 2021), (IA)<sup>3</sup> (Liu et al., 2022), LoRA (Hu et al., 2021), VeRA (Kopiczko et al., 2023), BOFT (Liu et al., 2023) and DoRA (Liu et al., 2024b). BitFit fine-tunes biases. (IA)<sup>3</sup> is a SOTA method adding adapters. LoRA uses low-rank decomposition to compress additive delta matrices. VeRA reduces LoRA’s trainable parameters while maintaining a high rank. BOFT compresses multiplicative delta matrices via orthogonal decomposition and butterfly factorization. DoRA separately learns magnitude and direction of delta matrices.

4.1 GLUE Benchmark

Settings. We assess our C<sup>3</sup>A on the GLUE benchmark (Wang et al., 2018) covering various tasks like single-sentence classification, similarity, paraphrase, and inference. See Table A3 in Appendix C for more details. Datasets are split into train, validation, and test sets. Models are chosen based on validation performance and evaluated on the test set. We fine-tune RoBERTa-Base and RoBERTa-Large models (Liu et al., 2019). Unique hyperparameters are from original papers (\( e.g., \) for VeRA’s \( r \) and BOFT’s \( b \) and \( m \)). Count of trainable parameters (# Params) exclude the classification head, which is uniform across methods. Shared hyperparameters (\( i.e., \) learning rates for the classification head and other parameters) are determined by hyperparameter search. Regarding the memory test, input data is fixed to 256 tokens with a batch size of 64 for consistency.

Results. Results are presented in Table 2. Overall, C<sup>3</sup>A<sub>b=768/1</sub> and C<sup>3</sup>A<sub>b=1024/1</sub> achieve superior or comparable performance to baseline methods, despite using an exceptionally small number of trainable parameters. As the number of trainable parameters increases, models like C<sup>3</sup>A<sub>b=768/6</sub> and C<sup>3</sup>A<sub>b=1024/8</sub> significantly outperform the baselines. Moreover, compared to (IA)<sup>3</sup>, LoRA, VeRA, and BOFT, C<sup>3</sup>A distinguishes itself with remarkable memory efficiency. The only method with better memory efficiency is BitFit, which serves as an upper bound since it does not introduce new parameters. Furthermore, most of the delta matrices identified by C<sup>3</sup>A are of full rank, indicating maximum capacity (Zeng and Lee, 2023) and providing a theoretical basis for outstanding performance.

4.2 Instruction Fine-tuning

Commonsense Reasoning. Our training utilizes Commonsense170K (Hu et al., 2023), aggregating multiple-choice questions from BoolQ (Clark et al., 2019), PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), HellaSwag (Zellers et al., 2019), Winogrande (Sakaguchi et al., 2021), ARC-e, ARC-c (Clark et al., 2018), and OBQA (Mihaylov et al., 2018). Evaluation follows Hu et al. (2023) and Liu et al. (2024b), using greedy search and first-
<table>
  <tr>
    <th>Methods</th>
    <th>Params (%)</th>
    <th>Mem</th>
    <th>BoolQ</th>
    <th>PIQA</th>
    <th>SIQA</th>
    <th>HellaS.</th>
    <th>WinoG.</th>
    <th>ARC-e</th>
    <th>ARC-c</th>
    <th>OBQA</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>ChatGPT</td>
    <td>-</td>
    <td>73.1</td>
    <td>85.4</td>
    <td>68.5</td>
    <td>78.5</td>
    <td>66.1</td>
    <td>89.8</td>
    <td>79.9</td>
    <td>74.8</td>
    <td>77.0</td>
  </tr>
  <tr>
    <td>LoRA<sub>=32</sub></td>
    <td>0.83</td>
    <td>41.71G</td>
    <td>71.0</td>
    <td>81.4</td>
    <td>79.6</td>
    <td>87.4</td>
    <td>83.2</td>
    <td>82.6</td>
    <td>67.5</td>
    <td>81.5</td>
    <td>79.3</td>
  </tr>
  <tr>
    <td>VeRA<sub>=16384</sub></td>
    <td>0.05</td>
    <td>57.88G</td>
    <td>68.9<sup>↑</sup></td>
    <td>81.0<sup>↑</sup></td>
    <td>77.4<sup>↑</sup></td>
    <td>87.5<sup>↑</sup></td>
    <td>81.3<sup>↑</sup></td>
    <td>81.1<sup>↑</sup></td>
    <td>62.8<sup>↓</sup></td>
    <td>79.9<sup>↑</sup></td>
    <td>77.5<sup>↑</sup></td>
  </tr>
  <tr>
    <td>BoFT<sub>=8</sub></td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>DoRA<sub>=32</sub></td>
    <td>0.84</td>
    <td>55.24G</td>
    <td>72.3<sup>↑</sup></td>
    <td>84.2<sup>↑</sup></td>
    <td>78.7<sup>↑</sup></td>
    <td>88.5<sup>↑</sup></td>
    <td>84.0<sup>↑</sup></td>
    <td>80.7<sup>↑</sup></td>
    <td>69.1<sup>↑</sup></td>
    <td>83.7<sup>↑</sup></td>
    <td>80.2<sup>↑</sup></td>
  </tr>
  <tr>
    <td>C<sup>3</sup>A<sub>=4096/32</sub></td>
    <td>0.35</td>
    <td>44.89G</td>
    <td>73.4<sup>↑</sup></td>
    <td>83.4<sup>↑</sup></td>
    <td>82.1<sup>↑</sup></td>
    <td>88.3<sup>↑</sup></td>
    <td>84.9<sup>↑</sup></td>
    <td>86.6<sup>↑</sup></td>
    <td>66.7<sup>↓</sup></td>
    <td>82.5<sup>↑</sup></td>
    <td>81.0<sup>↑</sup></td>
  </tr>
  <tr>
    <td>LLaMA2-7B</td>
    <td>-</td>
    <td>70.7</td>
    <td>51.18G</td>
    <td>73.8</td>
    <td>88.2</td>
    <td>80.4</td>
    <td>94.0</td>
    <td>85.5</td>
    <td>87.5</td>
    <td>78.1</td>
    <td>84.0</td>
    <td>83.9</td>
  </tr>
  <tr>
    <td>VeRA<sub>=16384</sub></td>
    <td>0.04</td>
    <td>66.03G</td>
    <td>72.2<sup>↑</sup></td>
    <td>87.5<sup>↑</sup></td>
    <td>80.0<sup>↑</sup></td>
    <td>94.3<sup>↑</sup></td>
    <td>83.9<sup>↑</sup></td>
    <td>84.5<sup>↑</sup></td>
    <td>75.6<sup>↓</sup></td>
    <td>82.4<sup>↑</sup></td>
    <td>82.5<sup>↑</sup></td>
  </tr>
  <tr>
    <td>BoFT<sub>=8</sub></td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>DoRA<sub>=32</sub></td>
    <td>0.71</td>
    <td>63.37G</td>
    <td>75.3<sup>↑</sup></td>
    <td>88.9<sup>↑</sup></td>
    <td>82.2<sup>↑</sup></td>
    <td>96.7<sup>↑</sup></td>
    <td>86.0<sup>↑</sup></td>
    <td>89.0<sup>↑</sup></td>
    <td>79.8<sup>↑</sup></td>
    <td>84.9<sup>↑</sup></td>
    <td>85.3<sup>↑</sup></td>
  </tr>
  <tr>
    <td>C<sup>3</sup>A<sub>=4096/32</sub></td>
    <td>0.26</td>
    <td>56.08G</td>
    <td>76.9<sup>↑</sup></td>
    <td>91.4<sup>↑</sup></td>
    <td>82.1<sup>↑</sup></td>
    <td>94.9<sup>↑</sup></td>
    <td>86.9<sup>↑</sup></td>
    <td>89.6<sup>↑</sup></td>
    <td>79.4<sup>↑</sup></td>
    <td>86.1<sup>↑</sup></td>
    <td>85.9<sup>↑</sup></td>
  </tr>
</table>

Table 3: Comparison of various methods on the LLaMA2-7B and LLaMA3-8B models across eight commonsense reasoning datasets. For each language model, the best result on each dataset is shown in bold. The symbols \( \uparrow \) and \( \downarrow \) indicate relative improvement and decrease, respectively, compared to the LoRA method. Experiments are conducted on a single H800 GPU with 80GB HBM.

<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th rowspan="2">Params (%)</th>
    <th colspan="2">Math</th>
    <th rowspan="2">Avg.</th>
    <th colspan="2">Code</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>GSM8K</th>
    <th>MATH</th>
    <th>HumanEval</th>
    <th>HumanEval+</th>
    <th>MBPP</th>
    <th>MBPP+</th>
  </tr>
  <tr>
    <td>Zero-Shot</td>
    <td>-</td>
    <td>7.3</td>
    <td>1.1</td>
    <td>4.2</td>
    <td>11.0</td>
    <td>9.8</td>
    <td>30.2</td>
    <td>24.1</td>
    <td>18.8</td>
  </tr>
  <tr>
    <td>LoRA<sub>=32</sub></td>
    <td>0.83</td>
    <td>60.5</td>
    <td>11.7</td>
    <td>36.1</td>
    <td>31.7</td>
    <td>28.0</td>
    <td>35.4</td>
    <td>30.4</td>
    <td>31.4</td>
  </tr>
  <tr>
    <td>VeRA<sub>=16384</sub></td>
    <td>0.05</td>
    <td>58.5<sup>↑</sup></td>
    <td>9.4<sup>↑</sup></td>
    <td>34.0<sup>↑</sup></td>
    <td>28.9<sup>↑</sup></td>
    <td>24.6<sup>↑</sup></td>
    <td>35.1<sup>↑</sup></td>
    <td>29.6<sup>↑</sup></td>
    <td>29.5<sup>↑</sup></td>
  </tr>
  <tr>
    <td>DoRA<sub>=32</sub></td>
    <td>0.84</td>
    <td>62.3<sup>↑</sup></td>
    <td>12.4<sup>↑</sup></td>
    <td>37.3<sup>↑</sup></td>
    <td>33.2<sup>↑</sup></td>
    <td>28.8<sup>↑</sup></td>
    <td>36.8<sup>↑</sup></td>
    <td>31.8<sup>↑</sup></td>
    <td>32.7<sup>↑</sup></td>
  </tr>
  <tr>
    <td>C<sup>3</sup>A<sub>=4096/32</sub></td>
    <td>0.35</td>
    <td>61.5<sup>↑</sup></td>
    <td>13.0<sup>↑</sup></td>
    <td>37.2<sup>↑</sup></td>
    <td>33.8<sup>↑</sup></td>
    <td>29.0<sup>↑</sup></td>
    <td>36.5<sup>↑</sup></td>
    <td>31.9<sup>↑</sup></td>
    <td>32.8<sup>↑</sup></td>
  </tr>
  <tr>
    <td>Zero-Shot</td>
    <td>-</td>
    <td>33.1</td>
    <td>5.3</td>
    <td>19.2</td>
    <td>33.5</td>
    <td>29.3</td>
    <td>61.4</td>
    <td>51.6</td>
    <td>44.0</td>
  </tr>
  <tr>
    <td>LoRA<sub>=32</sub></td>
    <td>0.70</td>
    <td>77.2</td>
    <td>28.2</td>
    <td>52.7</td>
    <td>57.9</td>
    <td>52.4</td>
    <td>64.8</td>
    <td>55.3</td>
    <td>57.6</td>
  </tr>
  <tr>
    <td>VeRA<sub>=16384</sub></td>
    <td>0.04</td>
    <td>76.0<sup>↑</sup></td>
    <td>26.7<sup>↑</sup></td>
    <td>51.4<sup>↑</sup></td>
    <td>56.3<sup>↑</sup></td>
    <td>50.8<sup>↑</sup></td>
    <td>64.0<sup>↑</sup></td>
    <td>53.6<sup>↑</sup></td>
    <td>56.2<sup>↑</sup></td>
  </tr>
  <tr>
    <td>DoRA<sub>=32</sub></td>
    <td>0.71</td>
    <td>77.2<sup>↑</sup></td>
    <td>30.0<sup>↑</sup></td>
    <td>53.6<sup>↑</sup></td>
    <td>59.1<sup>↑</sup></td>
    <td>52.9<sup>↑</sup></td>
    <td>65.7<sup>↑</sup></td>
    <td>55.8<sup>↑</sup></td>
    <td>58.4<sup>↑</sup></td>
  </tr>
  <tr>
    <td>C<sup>3</sup>A<sub>=4096/32</sub></td>
    <td>0.26</td>
    <td>78.4<sup>↑</sup></td>
    <td>29.9<sup>↑</sup></td>
    <td>54.1<sup>↑</sup></td>
    <td>59.5<sup>↑</sup></td>
    <td>53.8<sup>↑</sup></td>
    <td>65.4<sup>↑</sup></td>
    <td>56.1<sup>↑</sup></td>
    <td>58.7<sup>↑</sup></td>
  </tr>
</table>

Table 4: Comparison of various methods on the LLaMA2-7B and LLaMA3-8B models across math reasoning and code generation datasets. For each language model, the best result on each dataset is shown in bold. The symbols \( \uparrow \) and \( \downarrow \) indicate relative improvement and decrease, respectively, compared to the LoRA method. Experiments are conducted on a single H800 GPU with 80GB HBM.

keyword appearance for answer determination.

Mathematical Reasoning. We use Meta-MathQA (Yu et al., 2023), containing 395K QA pairs from GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2020). Following Yu et al. (2023), we evaluate using greedy search on test sets, requiring chain-of-thought reasoning (Wei et al., 2022).

Code Generation. Training uses Magicoder-Evol-Instruct-110k (Wei et al., 2024), a decontaminated subset of WizardCoder (Luo et al., 2024). Evaluation on HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), and their Plus variants follows EvalPlus (Liu et al., 2024a), reporting Pass@1 metrics.

Results. In Table 3 and Table 4, our principal experimental observations are summarized. The C<sup>3</sup>A framework consistently surpasses LoRA and other baselines within the LLaMA series, with particular efficacy demonstrated in the most recent model, LLaMA3-8B. Noteworthy is the significant enhancement in the efficacy of LLaMA3-8B as a foundational model following the implementation of more sophisticated post-training techniques. This underscores the criticality of optimizing the fine-tuning protocols for this advanced model. It is also remarkable that C<sup>3</sup>A achieves such results while employing less than half the parameter count of LoRA. Furthermore, C<sup>3</sup>A achieves high rank adaptation and delivers optimal performance while maintaining remarkably low memory consumption. In contrast, VeRA and DoRA require substantially greater memory resources, and BoFT results in an OOM condition for a single H800, thereby underscoring the practicality and efficiency of C<sup>3</sup>A. Taken together, the findings robustly underscore the superior efficacy of the C<sup>3</sup>A methodology. We
refer readers to Appendix D for examples of model answers after different tuning methods.

4.3 Ablation Study

Initialization. We investigated initialization effects in C^3A compared to LoRA, which is known for initialization sensitivity due to its A and B matrices (Hayou et al., 2024a). We evaluated multiple initialization strategies across five distinct tasks. Each initialization method was tested with 5 independent runs per task, producing a total of 20 runs per task. As shown in Figure 3, testing zero, Gaussian, Kaiming uniform, and Xavier uniform initializations for C^3A’s convolution kernels revealed performance variations mostly within standard deviations, demonstrating C^3A’s robustness to initialization choices.

Figure 3: Violin plots for runs on different tasks. Within each violin plot, a white bar indicates the range of average performance for each initialization strategy. It is clear that the choice of initialization does not affect the final result more than the intrinsic stochasticity.

Expressiveness. We demonstrate C^3A’s expressiveness on a synthetic dataset by placing 8 cluster centers on a 2D plane and sampling 30 points from their Gaussian distributions. A 3-layer MLP is used to classify these clusters. To compare the expressiveness of LoRA and C^3A, we replace the middle layer with either a low-rank layer or a circulant layer, ensuring that both layers have the same number of trainable parameters for a fair comparison.

The results are presented in Figure 4. We observe that LoRA_{r=1} struggles with this simple classification task. In contrast, C^3A_{b=128/2}, despite using the same number of parameters, achieves a perfect classification, comparable to a standard linear layer. This demonstrates the high expressiveness of C^3A given the same parameter budget.

Scaling. To investigate the scaling effect of C^3A, we examine it from both data and model perspectives, following the setup in He et al. (2025). From a data perspective, we focus on the MATH dataset and vary the training data used. As shown in Figure 5, we observe that C^3A’s performance improves faster than LoRA as more data is added, suggesting it is more effective at leveraging additional data. From the model perspective, we evaluate C^3A in small models (LLaMA3-8B, Mistral-7B) and large models (LLaMA3-70B, Mistral-8x7B). Results show that C^3A outperforms the LoRA baseline in both settings, indicating that its benefits generalize to different model scales. Experiments on various data quantities and model sizes showcase C^3A’s scalability and robustness, making it a promising approach for adapting language models.

![Training curves on the synthetic dataset.](page_682_98_670_312.png)

Figure 4: Training curves on the synthetic dataset. We refer our readers to Figure A1 in Appendix E for a visualization of the synthetic dataset.

![Data and model scaling of C^3A compared to LoRA.](page_682_420_670_312.png)

Figure 5: Data and model scaling of C^3A compared to LoRA. LM specifies LLaMA3 and MT specifies Mistral.

This manuscript introduces C^3A, a novel PEFT method. Unlike LoRA’s low-rank decomposition, C^3A utilizes circular convolution and circulant matrices to construct the delta weight matrix. This approach allows independent control over the delta weight matrix’s rank and the number of trainable parameters, enabling high-rank adaptation with limited parameter size. By employing FFT in forward and backward propagation, C^3A achieves significant computational and memory efficiency, presenting a compelling alternative to LoRA for model fine-tuning.
6 Limitations

Despite C^3A’s efficiency and effectiveness within limited parameter budgets, it has some intrinsic drawbacks. First, due to the requirement of \( b \in \gcd(d_1, d_2) \), C^3A may not be flexible enough to tackle all transformer architectures. Besides, the integration of FFT in C^3A necessitates specialized kernel support for optimal performance. While this support is well-developed and readily available for NVIDIA GPUs, it may not be as mature or optimized for edge devices with different hardware architectures. This can potentially hinder the deployment and efficiency of C^3A on resource-constrained edge platforms.

References

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.

Bassam Bamieh. 2018. Discovering transforms: A tutorial on circulant matrices, circular convolution, and the discrete fourier transform. arXiv preprint arXiv:1805.05533.

Samyadeep Basu, Shell Hu, Daniela Massiceti, and Soheil Feizi. 2024. Strong baselines for parameter-efficient few-shot fine-tuning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 11024–11031.

Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.

Aochuan Chen, Yuguang Yao, Pin-Yu Chen, Yihua Zhang, and Sijia Liu. 2023. Understanding and improving visual prompting: A label-mapping perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19133–19143.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.

Gong Cheng, Junwei Han, and Xiaoqiang Lu. 2017. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105(10):1865–1883.

Yu Cheng, Felix X Yu, Rogerio S Feris, Sanjiv Kumar, Alok Choudhary, and Shi-Fu Chang. 2015. An exploration of parameter redundancy in deep networks with circulant projections. In Proceedings of the IEEE international conference on computer vision, pages 2857–2865.

Mircea Cimpoi, Subhransu Maji, Jasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. 2014. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606–3613.

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.

Caiwen Ding, Siyu Liao, Yanzhi Wang, Zhe Li, Ning Liu, Youwei Zhuo, Chao Wang, Xuehai Qian, Yu Bai, Geng Yuan, Xiaolong Ma, Yipeng Zhang, Jian Tang, Qinru Qiu, Xue Lin, and Bo Yuan. 2017. Cirnn: accelerating and compressing deep neural networks using block-circulant weight matrices. In Proceedings of the 50th Annual IEEE/ACM International Symposium on Microarchitecture, MICRO-50 ’17, page 395–408, New York, NY, USA. Association for Computing Machinery.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.

Morris Dworkin, Elaine Barker, James Nechvatal, James Foti, Lawrence Bassham, E. Roback, and James Dray. 2001. Advanced encryption standard (aes).

Zihao Fu, Haoran Yang, Anthony Man-Cho So, Wai Lam, Lidong Bing, and Nigel Collier. 2023. On the effectiveness of parameter-efficient fine-tuning. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 12799–12807.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2020. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723.

Gene H. Golub and Charles F. Van Loan. 1996. Matrix computations (3rd ed.). Johns Hopkins University Press, USA.

Yanwei Gong, Xiaolin Chang, Jelena Mišić, Vojislav B Mišić, Jianhua Wang, and Haoran Zhu. 2024. Practical solutions in fully homomorphic encryption: a survey analyzing existing acceleration methods. Cybersecurity, 7(1):5.

Demi Guo, Alexander M Rush, and Yoon Kim. 2020. Parameter-efficient transfer learning with diff pruning. arXiv preprint arXiv:2012.07463.

Soufiane Hayou, Nikhil Ghosh, and Bin Yu. 2024a. The impact of initialization on lora finetuning dynamics. arXiv preprint arXiv:2406.08447.

Soufiane Hayou, Nikhil Ghosh, and Bin Yu. 2024b. Lora+: Efficient low rank adaptation of large models. arXiv preprint arXiv:2402.12354.

Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2021. Towards a unified view of parameter-efficient transfer learning. arXiv preprint arXiv:2110.04366.

Zhiwei He, Zhaopeng Tu, Xing Wang, Xingyu Chen, Zhijie Wang, Jiahao Xu, Tian Liang, Wenxiang Jiao, Zhihuang Zhang, and Rui Wang. 2025. RaSA: Rank-sharing low-rank adaptation. In The Thirteenth International Conference on Learning Representations.

Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. 2019. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217–2226.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.

Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka-Wei Lee. 2023. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2304.01933.

A. W. Ingleton. 1956. The rank of circulant matrices. Journal of the London Mathematical Society, s1-31(4):445–460.

Eric Jang, Shixiang Gu, and Ben Poole. 2016. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144.

Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. 2022. Visual prompt tuning. In European Conference on Computer Vision, pages 709–727. Springer.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. 2023. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4026.

Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki Markus Asano. 2023. Vera: Vector-based random matrix adaptation. arXiv preprint arXiv:2310.11454.

Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 2013. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554–561.

Changli Li, Hon Keung Kwan, and Xinxin Qin. 2020. Revisiting linear convolution, circular convolution and their related methods. 2020 13th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), pages 1124–1131.

Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190.

Yuhan Li, Peisong Wang, Zhixun Li, Jeffrey Xu Yu, and Jia Li. 2024a. Zerog: Investigating cross-dataset zero-shot transferability in graphs. arXiv preprint arXiv:2402.11235.

Yuhan Li, Peisong Wang, Xiao Zhu, Aochuan Chen, Haiyun Jiang, Deng Cai, Victor W Chan, and Jia Li. 2024b. Gilbench: A comprehensive benchmark for graph with large language models. Advances in Neural Information Processing Systems, 37:42349–42368.

Yuhan Li, Xinni Zhang, Linhao Luo, Heng Chang, Yuxiang Ren, Irwin King, and Jia Li. 2025. G-refer: Graph retrieval-augmented large language model for explainable recommendation. In Proceedings of the ACM on Web Conference 2025, pages 240–251.

Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang. 2022. Scaling & shifting your features: A new baseline for efficient model tuning. Advances in Neural Information Processing Systems, 35:109–123.

Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohita, Tenghao Huang, Mohit Bansal, and Colin A Raffel. 2022. Few-shot parameter-efficient fine-tuning
is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950–1965.

Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024a. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36.

Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024b. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353.

Weiyang Liu, Zeju Qiu, Yao Feng, Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng, Zhen Liu, Juyeon Heo, Songyou Peng, et al. 2023. Parameter-efficient orthogonal finetuning via butterfly factorization. arXiv preprint arXiv:2311.06243.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.

Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2024. Wizardcoder: Empowering code large language models with evol-instruct. In The Twelfth International Conference on Learning Representations.

Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. 2013. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151.

Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. 2023. Fine-tuning language models with just forward passes. Advances in Neural Information Processing Systems, 36:53038–53075.

Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. 2022. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/peft.

Clare D. McGillem and George R. Cooper. 1984. Continuous and discrete signal and system analysis.

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789.

Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. 2012. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498–3505. IEEE.

Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. 2023. Controlling text-to-image diffusion by orthogonal finetuning. Advances in Neural Information Processing Systems, 36:79320–79362.

L. R. Rabiner, B. Gold, and C. K. Yuen. 1978. Theory and application of digital signal processing. IEEE Transactions on Systems, Man, and Cybernetics, 8(2):146–146.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR.

Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. 2019. Rapid learning or feature reuse? towards understanding the effectiveness of maml. arXiv preprint arXiv:1909.09157.

Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. 2017. Learning multiple visual domains with residual adapters. Advances in neural information processing systems, 30.

Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. 2021. Imagenet-21k pretraining for the masses. arXiv preprint arXiv:2104.10972.

Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. 2020. Adapterdrop: On the efficiency of adapters in transformers. arXiv preprint arXiv:2010.11918.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106.

Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019. Socialiq: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728.

Hugo Touvron, Thibaut Lavril, Gautier Izard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.
Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2024. Magicoder: Empowering code generation with oss-instruct. In Forty-first International Conference on Machine Learning.

Runxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang, Songfang Huang, and Fei Huang. 2021. Raise a child in large language model: Towards effective and generalizable fine-tuning. arXiv preprint arXiv:2109.05687.

Felix Yu, Sanjiv Kumar, Yunchao Gong, and Shih-Fu Chang. 2014. Circulant binary embedding. In International conference on machine learning, pages 946–954. PMLR.

Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhengguo Li, Adrian Weller, and Weiyang Liu. 2023. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284.

Shen Yuan, Haotian Liu, and Hongteng Xu. 2024. Bridging the gap between low-rank and orthogonal adaptation via householder reflection adaptation. arXiv preprint arXiv:2405.17484.

Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. 2021. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106.10199.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830.

Yuchen Zeng and Kangwook Lee. 2023. The expressive power of low-rank adaptation. arXiv preprint arXiv:2310.17513.

Yihua Zhang, Hongkang Li, Yuguang Yao, Aochuan Chen, Shuai Zhang, Pin-Yu Chen, Meng Wang, and Sijia Liu. 2024a. Visual prompting reimagined: The power of activation prompts.

Yushun Zhang, Congliang Chen, Ziniu Li, Tian Ding, Chenwei Wu, Yinyu Ye, Zhi-Quan Luo, and Ruoyu Sun. 2024b. Adam-mini: Use fewer learning rates to gain more. arXiv preprint arXiv:2406.16793.
Appendix

A  Implementations

Algorithm A1 Block-Circular Convolution PyTorch Implementation

import torch
from torch.autograd import Function
from torch.fft import fft, ifft

class BlockCircularConvolution(Function):
    @staticmethod
    def forward(ctx, x, w):
        m, n, b = w.shape
        x = x.reshape(*x.shape[:-1], n, b)
        ctx.save_for_backward(x, w)
        x = torch.einsum(
            "...nb,mnb->...mb", ifft(x), fft(w)
        )
        x = fft(x).real
        x = x.reshape(*x.shape[:-2], -1)
        return x

    @staticmethod
    def backward(ctx, grad_output):
        x, w = ctx.saved_tensors
        m, n, b = w.shape
        grad_output = grad_output.reshape(
            *grad_output.shape[:-1], m, b
        )
        grad_output_fft = fft(grad_output)
        x_grad = fft(torch.einsum(
            "...mb,mnb->...nb",
            grad_output_fft, ifft(w)
        )).real
        x_grad = x_grad.reshape(
            *x_grad.shape[:-2], -1
        )
        w_grad = fft(torch.einsum(
            "...mb,...mb->mnb",
            grad_output_fft, ifft(x)
        )).real
        return x_grad, w_grad

We present the PyTorch implementation of Block-Circular Convolution in Algorithm A1. Furthermore, due to the inefficiency of directly assigning entries (as shown in Equation 4), we derive an alternative algorithm to compute the \( \Delta W \) more efficiently. Rather than direct assignment, we employ a forward process on the Identity matrix. Mathematically, this can be expressed as

\[
\Delta W = C_{blk}(\Delta w) \\
= C_{blk}(\Delta w) \cdot I_{d_2} \\
= C_{blk}(\Delta w) \cdot [e_1, e_2, \cdots, e_{d_2}] \\
= [C_{blk}(\Delta w)e_1, C_{blk}(\Delta w)e_2, \cdots, C_{blk}(\Delta w)e_{d_2}] \\
= [\Delta w * e_1, \Delta w * e_2, \cdots, \Delta w * e_{d_2}].
\]

Where \( I_{d_2} \in \mathbb{R}^{d_2 \times d_2} \) represents an Identity matrix and \( e_i \) is the \( i \)th column of it. In pytorch, we can efficiently compute the iFFT of \( \{e_i\}_{i=1,2,\cdots,d_2} \) by a column-wise iFFT of \( I_{d_2} \). We present the Pytorch implementation in Algorithm A2 as well.

Algorithm A2 Fast Algorithm of Getting \( \Delta W \)

import torch
from torch.fft import fft, ifft

def get_circulant_fast(w):
    m, n, b = w.shape
    x = torch.eye(n*b)
    x = x.reshape(*x.shape[:-1], n, b)
    x = torch.einsum(
        "...nb,mnb->...mb", ifft(x), fft(w)
    )
    x = fft(x).real.flatten(start_dim=1).T
    return x

B  Image Classification

Settings. In this study, we concentrate on the task of image classification leveraging Vision Transformer (ViT) models. Specifically, we employ both the Base and Large variants of this prominent foundational computer vision model, as delineated by (Dosovitskiy et al., 2020). These ViT models undergo pre-training on the expansive ImageNet-21K dataset (Ridnik et al., 2021). During the fine-tuning phase, we use an eclectic array of datasets encompassing Pets (Parkhi et al., 2012), Cars (Krause et al., 2013), DTD (Cimpoi et al., 2014), EuroSAT (Helber et al., 2019), FGVC (Maji et al., 2013), and RESISC (Cheng et al., 2017). Statistics for these datasets are provided in Table A1.

<table>
  <tr>
    <th>Dataset</th>
    <th>#Train</th>
    <th>#Validation</th>
    <th>#Test</th>
    <th>#Class</th>
  </tr>
  <tr>
    <td>Pets (Parkhi et al., 2012)</td>
    <td>3,312</td>
    <td>368</td>
    <td>3,669</td>
    <td>37</td>
  </tr>
  <tr>
    <td>Cars (Krause et al., 2013)</td>
    <td>7,329</td>
    <td>815</td>
    <td>8,041</td>
    <td>196</td>
  </tr>
  <tr>
    <td>DTD (Cimpoi et al., 2014)</td>
    <td>4,060</td>
    <td>452</td>
    <td>4,128</td>
    <td>47</td>
  </tr>
  <tr>
    <td>EuroSAT (Helber et al., 2019)</td>
    <td>16,200</td>
    <td>5,400</td>
    <td>5,400</td>
    <td>10</td>
  </tr>
  <tr>
    <td>FGVC (Maji et al., 2013)</td>
    <td>3,000</td>
    <td>334</td>
    <td>3,333</td>
    <td>100</td>
  </tr>
  <tr>
    <td>RESISC (Cheng et al., 2017)</td>
    <td>18,900</td>
    <td>6,300</td>
    <td>6,300</td>
    <td>45</td>
  </tr>
</table>

Table A1: Details about the vision datasets.

Results. Table A2 delineates a comprehensive summary of the outcomes derived from six distinct image classification datasets employing the ViT Base and Large models. The LoRA and C^3A techniques exhibit significant enhancements in performance relative to Head Tuning, thereby underscoring their efficacy within the realm of image classification. Remarkably, our methodology demonstrates a performance on par with LoRA while necessitating only half of the parameter count.

C  GLUE Benchmark Details

The General Language Understanding Evaluation (GLUE) benchmark is a collection of nine natural language understanding tasks designed to test a model’s performance on a diverse set of natural
<table>
  <tr>
    <th>Method</th>
    <th># Params</th>
    <th>Pets</th>
    <th>Cars</th>
    <th>DTD</th>
    <th>EuroSAT</th>
    <th>FGVC</th>
    <th>RESISC</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td rowspan="2">BASE</td>
    <td>Head</td>
    <td>-</td>
    <td>90.28±0.43</td>
    <td>25.76±0.28</td>
    <td>69.77±0.67</td>
    <td>88.72±0.13</td>
    <td>17.44±0.43</td>
    <td>74.22±0.10</td>
    <td>61.03</td>
  </tr>
  <tr>
    <td>Full</td>
    <td>85.8M</td>
    <td>92.82±0.54</td>
    <td>85.10±0.21</td>
    <td>80.11±0.56</td>
    <td>99.11±0.07</td>
    <td>61.60±1.00</td>
    <td>96.00±0.23</td>
    <td>85.79</td>
  </tr>
  <tr>
    <td rowspan="2">LARGE</td>
    <td>LoRA<sub>r=16</sub></td>
    <td>0.59M</td>
    <td>93.76±0.44</td>
    <td>78.04±0.33</td>
    <td>78.56±0.62</td>
    <td>98.84±0.08</td>
    <td><b>56.64±0.55</b></td>
    <td><b>94.66±0.17</b></td>
    <td>83.42</td>
  </tr>
  <tr>
    <td>C<sup>3</sup>A<sub>b=768/12</sub></td>
    <td>0.22M</td>
    <td><b>93.88±0.22</b></td>
    <td><b>79.05±0.35</b></td>
    <td><b>80.57±0.53</b></td>
    <td><b>98.88±0.07</b></td>
    <td>54.31±0.79</td>
    <td>94.54±0.23</td>
    <td><b>83.54</b></td>
  </tr>
  <tr>
    <td rowspan="2">LARGE</td>
    <td>Head</td>
    <td>-</td>
    <td>91.11±0.30</td>
    <td>37.91±0.27</td>
    <td>73.33±0.26</td>
    <td>92.64±0.08</td>
    <td>24.62±0.24</td>
    <td>82.02±0.11</td>
    <td>66.94</td>
  </tr>
  <tr>
    <td>Full</td>
    <td>303M</td>
    <td>94.30±0.31</td>
    <td>88.15±0.50</td>
    <td>80.18±0.66</td>
    <td>99.06±0.10</td>
    <td>67.38±1.06</td>
    <td>96.08±0.20</td>
    <td>87.53</td>
  </tr>
  <tr>
    <td rowspan="2">LARGE</td>
    <td>LoRA<sub>r=16</sub></td>
    <td>1.57M</td>
    <td><b>94.62±0.47</b></td>
    <td><b>86.11±0.42</b></td>
    <td>80.09±0.42</td>
    <td><b>98.99±0.03</b></td>
    <td>63.64±0.83</td>
    <td>95.52±0.21</td>
    <td>86.56</td>
  </tr>
  <tr>
    <td>C<sup>3</sup>A<sub>b=1024/16</sub></td>
    <td>0.79M</td>
    <td>94.48±0.30</td>
    <td>84.94±0.39</td>
    <td><b>82.62±0.52</b></td>
    <td>98.75±0.19</td>
    <td><b>63.80±0.37</b></td>
    <td><b>95.94±0.16</b></td>
    <td><b>86.69</b></td>
  </tr>
</table>

Table A2: Fine-tuning results with ViT-Base and ViT-Large models on various image classification datasets. The models are fine-tuned for 10 epochs, and the best-performing model, based on validation set accuracy, is selected. The reported accuracy corresponds to the performance on the test set. The best results between LoRA and C<sup>3</sup>A for each dataset are highlighted in bold. “Avg.” denotes the average accuracy of each method across all datasets.

<table>
  <tr>
    <th>Corpus</th>
    <th>Task</th>
    <th># Train</th>
    <th># Val</th>
    <th># Test</th>
    <th># Labels</th>
    <th>Metrics</th>
    <th>Domain</th>
  </tr>
  <tr>
    <th colspan="8">Single-Sentence Tasks</th>
  </tr>
  <tr>
    <td>CoLA</td>
    <td>Acceptability</td>
    <td>8.55k</td>
    <td>1.04k</td>
    <td>1.06k</td>
    <td>2</td>
    <td>Matthews Corr.</td>
    <td>misc.</td>
  </tr>
  <tr>
    <td>SST-2</td>
    <td>Sentiment</td>
    <td>67.3k</td>
    <td>872</td>
    <td>1.82k</td>
    <td>2</td>
    <td>Accuracy</td>
    <td>Movie reviews</td>
  </tr>
  <tr>
    <th colspan="8">Similarity and Paraphrase Tasks</th>
  </tr>
  <tr>
    <td>MRPC</td>
    <td>Paraphrase</td>
    <td>3.67</td>
    <td>408</td>
    <td>1.73k</td>
    <td>2</td>
    <td>Accuracy/F1</td>
    <td>News</td>
  </tr>
  <tr>
    <td>STS-B</td>
    <td>Sentence similarity</td>
    <td>5.75k</td>
    <td>1.5k</td>
    <td>1.38k</td>
    <td>1</td>
    <td>Pearson/Spearman Corr.</td>
    <td>misc.</td>
  </tr>
  <tr>
    <td>QQP</td>
    <td>Paraphrase</td>
    <td>364k</td>
    <td>40.4k</td>
    <td>391k</td>
    <td>2</td>
    <td>Accuracy/F1</td>
    <td>Social QA</td>
  </tr>
  <tr>
    <th colspan="8">Inference Tasks</th>
  </tr>
  <tr>
    <td>MNLI</td>
    <td>NLI</td>
    <td>393k</td>
    <td>19.65k</td>
    <td>19.65k</td>
    <td>3</td>
    <td>Accuracy</td>
    <td>misc.</td>
  </tr>
  <tr>
    <td>QNLI</td>
    <td>QA/NLI</td>
    <td>105k</td>
    <td>5.46k</td>
    <td>5.46k</td>
    <td>2</td>
    <td>Accuracy</td>
    <td>Wikipedia</td>
  </tr>
  <tr>
    <td>RTE</td>
    <td>NLI</td>
    <td>2.49k</td>
    <td>277</td>
    <td>3k</td>
    <td>2</td>
    <td>Accuracy</td>
    <td>News & Wikipedia</td>
  </tr>
</table>

Table A3: Task descriptions and dataset statistics of the GLUE benchmark (Wang et al., 2018).

language understanding challenges. It was introduced by researchers from New York University, the University of Washington, and DeepMind in a 2018 paper titled “GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding” (Wang et al., 2018).

The nine tasks included in the benchmark are:

1. CoLA (Corpus of Linguistic Acceptability): A binary classification task that tests a model’s ability to determine whether a given sentence is grammatically acceptable.

2. SST-2 (Stanford Sentiment Treebank): A binary sentiment classification task using movie review excerpts.

3. MRPC (Microsoft Research Paraphrase Corpus): A binary classification task to determine whether two sentences are semantically equivalent.

4. STS-B (Semantic Textual Similarity Benchmark): A regression task that scores the semantic similarity of sentence pairs on a scale from 1 to 5.

5. QQP (Quora Question Pairs): A binary classification task to determine whether two questions are semantically equivalent.

6. MNLI (Multi-Genre Natural Language Inference): A ternary classification task that tests a model’s ability to perform textual entailment across multiple genres.

7. QNLI (Question Natural Language Inference): A binary classification task that tests a model’s ability to determine whether a sentence contains the answer to a given question.

8. RTE (Recognizing Textual Entailment): A binary classification task that tests a model’s ability to determine whether a premise entails a hypothesis.
9. WNLI (Winograd Natural Language Inference): A binary classification task that tests a model’s ability to resolve ambiguous pronouns in a sentence.

The GLUE benchmark has become a widely-used standard for evaluating the performance of pre-trained language models and fine-tuned models on a variety of natural language understanding tasks. The benchmark provides a single-number metric, the GLUE score, which is an average of the scores on each individual task. This allows for easy comparison of different models and architectures. We refer readers to Table A3 for more details.

D Instruction Fine-tuning Examples

Example D.1: MATH
Below is an instruction that describes a task. Write a response that appropriately completes the request.
Instruction: Find the number of square units bounded by the x-axis, the y-axis and the line \( x + y = 6 \). Answer the above question. First think step by step and then answer the final number.
LoRA Answer: The line \( x + y = 6 \) intersects the x-axis at \( x = 6 \) and the y-axis at \( y = 6 \). So, the line divides the square with side length 6 into two triangles. The area of each triangle is \( \frac{1}{2} \times 6 \times 6 = 18 \). So, the area of the square is \( 2 \times 18 = 36 \). The final answer is: 36.
C^3A Answer: The line \( x + y = 6 \) intersects the x-axis at (6, 0) and the y-axis at (0, 6). Thus, the line divides the first quadrant into two triangles, one with base 6 and height 6 and the other with base 6 and height 0. The area of the first triangle is \( (6 \times 6)/2 = 18 \) square units, and the area of the second triangle is \( (6 \times 0)/2 = 0 \) square units. Thus, the total area is \( 18 + 0 = 18 \) square units. The final answer is: 18.

![Synthetic dataset used for the expressiveness ablation study.](page_749_153_627_410.png)

Figure A1: Synthetic dataset used for the expressiveness ablation study.

Example D.2: GSM8K
Below is an instruction that describes a task. Write a response that appropriately completes the request.
Instruction: Tim makes a special honey and jam mix. To make 1 jar of the mixture it takes 2 pounds of passion fruit and 1.5 pounds of jackfruit. It also takes 2 pounds of special honey. The passion fruit costs $6 per pound, the jackfruit is $8 per pound, and the honey is $10 per pound. He sells each jar for $50. How much profit does he make selling 10 jars? Answer the above question. First think step by step and then answer the final number.
LoRA Answer: The passion fruit costs \( 6 \times 2 = 12 \). The jackfruit costs \( 8 \times 1.5 = 12 \). The honey costs \( 10 \times 2 = 20 \). So the total cost is \( 12 + 12 + 20 = 44 \). So he makes a profit of \( 50 \times 10 - 44 = [460] \). The final answer is: 460.
C^3A Answer: The passion fruit costs \( 6 \times 2 = 12 \). The jackfruit costs \( 8 \times 1.5 = 12 \). So the fruit costs \( 12 + 12 = 24 \). The honey costs \( 10 \times 2 = 20 \). So the total cost per jar is \( 24 + 20 = 44 \). So he makes a profit of \( 50 - 44 = 6 \) per jar. So he makes a profit of \( 6 \times 10 = [60] \). The final answer is: 60.

E Synthetic Dataset
We visualize the synthetic dataset we use for the expressiveness ablation study in Figure A1.

F Hyperparameters

G Singular Value Comparison
<table>
  <tr>
    <th>Model</th>
    <th>Hyperparameter</th>
    <th>SST-2</th>
    <th>MRPC</th>
    <th>CoLA</th>
    <th>QNLI</th>
    <th>RTE</th>
    <th>STS-B</th>
  </tr>
  <tr>
    <td rowspan="5">Both</td>
    <td>Optimizer</td>
    <td colspan="6">AdamW</td>
  </tr>
  <tr>
    <td>LR Schedule</td>
    <td colspan="6">Linear</td>
  </tr>
  <tr>
    <td>Warmup Ratio</td>
    <td colspan="6">0.06</td>
  </tr>
  <tr>
    <td>C<sup>3</sup>A Initialization</td>
    <td colspan="6">Xavier Uniform</td>
  </tr>
  <tr>
    <td>Max Seq. Len</td>
    <td colspan="6">512</td>
  </tr>
  <tr>
    <td rowspan="4">Base</td>
    <td>Epochs</td>
    <td>40</td><td>80</td><td>80</td><td>40</td><td>80</td><td>80</td>
  </tr>
  <tr>
    <td>Batch Size</td>
    <td>128</td><td>128</td><td>128</td><td>64</td><td>64</td><td>128</td>
  </tr>
  <tr>
    <td>Learning Rate (C<sup>3</sup>A<sub>b=768/6</sub>)</td>
    <td>2E-1</td><td>3E-1</td><td>2E-1</td><td>7E-2</td><td>3E-1</td><td>2E-1</td>
  </tr>
  <tr>
    <td>Learning Rate (Head)</td>
    <td>2E-4</td><td>4E-6</td><td>3E-2</td><td>8E-6</td><td>6E-3</td><td>4E-2</td>
  </tr>
  <tr>
    <td rowspan="4">Large</td>
    <td>Epochs</td>
    <td>10</td><td>80</td><td>70</td><td>30</td><td>60</td><td>40</td>
  </tr>
  <tr>
    <td>Batch Size</td>
    <td>128</td><td>128</td><td>128</td><td>32</td><td>64</td><td>128</td>
  </tr>
  <tr>
    <td>Learning Rate (C<sup>3</sup>A<sub>b=1024/8</sub>)</td>
    <td>9E-2</td><td>3E-1</td><td>2E-1</td><td>7E-2</td><td>5E-2</td><td>2E-1</td>
  </tr>
  <tr>
    <td>Learning Rate (Head)</td>
    <td>2E-4</td><td>5E-6</td><td>3E-3</td><td>8E-6</td><td>3E-3</td><td>5E-4</td>
  </tr>
</table>

Table A4: Hyperparameter setup of C<sup>3</sup>A for the GLUE benchmark.

<table>
  <tr>
    <th>Model</th>
    <th>Hyperparameter</th>
    <th>Pets</th>
    <th>Cars</th>
    <th>DTD</th>
    <th>EuroSAT</th>
    <th>FGVC</th>
    <th>RESISC</th>
  </tr>
  <tr>
    <td rowspan="5">Both</td>
    <td>Optimizer</td>
    <td colspan="6">AdamW</td>
  </tr>
  <tr>
    <td>LR Schedule</td>
    <td colspan="6">None</td>
  </tr>
  <tr>
    <td>C<sup>3</sup>A Initialization</td>
    <td colspan="6">Xavier Uniform</td>
  </tr>
  <tr>
    <td>Epochs</td>
    <td colspan="6">10</td>
  </tr>
  <tr>
    <td>Batch Size</td>
    <td colspan="6">64</td>
  </tr>
  <tr>
    <td rowspan="3">Base</td>
    <td>Learning Rate (C<sup>3</sup>A<sub>b=768/12</sub>)</td>
    <td>4E-1</td><td>4E+0</td><td>2E+0</td><td>2E+0</td><td>7E+0</td><td>2E+0</td>
  </tr>
  <tr>
    <td>Learning Rate (Head)</td>
    <td>1E-2</td><td>1E-2</td><td>2E-2</td><td>8E-3</td><td>1E-2</td><td>2E-2</td>
  </tr>
  <tr>
    <td>Weight Decay</td>
    <td>3E-4</td><td>5E-4</td><td>6E-5</td><td>2E-5</td><td>1E-5</td><td>2E-5</td>
  </tr>
  <tr>
    <td rowspan="3">Large</td>
    <td>Learning Rate (C<sup>3</sup>A<sub>b=1024/16</sub>)</td>
    <td>7E-1</td><td>4E+0</td><td>2E+0</td><td>2E+0</td><td>4E+0</td><td>3E+0</td>
  </tr>
  <tr>
    <td>Learning Rate (Head)</td>
    <td>3E-3</td><td>8E-3</td><td>7E-3</td><td>2E-2</td><td>1E-1</td><td>4E-3</td>
  </tr>
  <tr>
    <td>Weight Decay</td>
    <td>4E-3</td><td>1E-5</td><td>2E-4</td><td>5E-4</td><td>2E-5</td><td>9E-5</td>
  </tr>
</table>

Table A5: Hyperparameter setup of C<sup>3</sup>A for image classification tasks.

<table>
  <tr>
    <th>Hyperparameter</th>
    <th>Common</th>
    <th>Math</th>
    <th>Code</th>
  </tr>
  <tr>
    <td>Optimizer</td>
    <td colspan="3">AdamW</td>
  </tr>
  <tr>
    <td>LR Scheduler</td>
    <td colspan="3">Cosine</td>
  </tr>
  <tr>
    <td>Batch Size</td>
    <td>16</td><td>128</td><td>128</td>
  </tr>
  <tr>
    <td>Warmup Ratio</td>
    <td>0.05</td><td>0.03</td><td>0.03</td>
  </tr>
  <tr>
    <td>Epoch</td>
    <td>3</td><td>2</td><td>2</td>
  </tr>
  <tr>
    <td>Dropout</td>
    <td colspan="3">0.05</td>
  </tr>
  <tr>
    <td>Learning Rate (LLaMA2-7B)</td>
    <td>3E-1</td><td>4E-1</td><td>6E-1</td>
  </tr>
  <tr>
    <td>Learning Rate (LLaMA3-8B)</td>
    <td>3E-1</td><td>5E-1</td><td>6E-1</td>
  </tr>
</table>

Table A6: Hyperparameter setup of C<sup>3</sup>A for instruction tuning.

![Singular values spectra of the weight difference between C^3A and LoRA, Full finetuning.](page_1012_1342_388_246.png)

Figure A2: Singular values spectra of the weight difference between C<sup>3</sup>A and LoRA, Full finetuning.