Correlated Low-Rank Adaptation for ConvNets
Authors: Wu Ran
Weijia Zhang
ShuYang Pang
Qi Zhu
Jinfan Liu
JingSheng Liu
Xin Cao
Qiang Li
Yichao Yan
Chao Ma
Venue: NeurIPS 2025 poster ()
Keywords: Low-rank adaptation
convnets
correlated weight update
Primary area: deep_learning
TL;DR: This paper introduces a correlated low-rank adaptation strategy within each paired adjacent convolution layer for PEFT ConvNets, which demonstrates better convergence and performance than the high-rank tuning approach on downstream vision tasks.

Abstract: Low-Rank Adaptation (LoRA) methods have demonstrated considerable success in achieving parameter-efficient fine-tuning (PEFT) for Transformer-based foundation models. These methods typically fine-tune individual Transformer layers using independent LoRA adaptations. However, directly applying existing LoRA techniques to convolutional networks (ConvNets) yields unsatisfactory results due to the high correlation between the stacked sequential layers of ConvNets. To overcome this challenge, we introduce a novel framework called Correlated Low-Rank Adaptation (CoLoRA), which explicitly utilizes correlated low-rank matrices to model the inter-layer dependencies among convolutional layers. Additionally, to enhance tuning efficiency, we propose a parameter-free filtering method that enlarges the receptive field of LoRA, thus minimizing interference from non-informative local regions. Comprehensive experiments conducted across various mainstream vision tasks, including image classification, semantic segmentation, and object detection, illustrate that CoLoRA significantly advances the state-of-the-art PEFT approaches. Notably, our CoLoRA achieves superior performance with only 5\% of trainable parameters, surpassing full fine-tuning in the image classification task on the VTAB-1k dataset using ConvNeXt-S. Code is available at [https://github.com/VISION-SJTU/CoLoRA](https://github.com/VISION-SJTU/CoLoRA).

Correlated Low-Rank Adaptation for ConvNets
Wu Ran'
Weijia Zhang' Shuyang Pang" Qi Zhu'
Jinfan Liu'
Jingsheng Liu²
Xin Cao² Qiang Lia Yichao Yan' Chao Ma
MoE Key Lab of Artificial Intelligence, Al Institute, Shanghai Jiao Tong University CISDI Information Technology CO., LTD.
(bonjourlemonde weijia. siling,hang, georgezhu, 1jflnjz, yanyichao, chaoma) jstt edu. (shuyang pannang-pann,,,,.. jingsheng. liu, xin.a.cao, qlang. Kciisdd..... com. cn
Abstract
Low-Rank Adaptation (LoRA) methods have demonstrated considerable success in achieving parameter-efficient fine-tuning (PEFT) for Transformer-base foun- dation models. These methods typically fine-tune individual Transformer layers using independent LoRA adaptations. However, directly applying existing LoRA techniques convolutional networks (ConvNets) yields unsatisfactory results due to the high correlation between the stacked sequential layers of ConvNets. To overcome this challenge, we introduce a novel framework called Correlated Low- Rank Adaptation (CoLoRA), which explicitly utilizes correlated low-rank matri- ces to model the inter-layer dependencies among convolutional layers. Addition- ally, to enhance tuning efficiency, we propose parameter-free filtering method that enlarges the receptive field LoRA, thus minimizing interference from non- informative local regions. Comprehensive experiments conducted across various mainstream vision tasks, including image classification, semantic segmentation, and object detection, illustrate that CoLoRA significantly advances the state-of- the-art PEFT approaches. Notably, our CoLoRA achieves superior performance with only 5% of trainable parameters, surpassing full fine-tuning in the image classification task on the VTAB-1k dataset using Ce--Ne Code is available at http://///ttttbb.oom/miinn------
1 Introduction
Transformers [TI. 2 have significantly advanced the development of large-scale pre-trained foun- dation models leverage the capabilities of these pre-trained foundation models, parameter-efficient fine-tuning (PEFT) [6. 7 X IOC. IT] has emerged a prevalent solution. Typically, low-rank adaptation (LoRA) approaches 2 IH, 14, 1.L. [6]] have demonstrated particular promise in Transformer-be based models such as LLaMA [ and ViT [2]. By introducing independent low-rank matrices into each layer, LoRA effectively modulates the global attention maps of Transformers, facilitating efficient adaptation to downstream tasks while significantly reducing the number of trainable parameters.
[S] in both natural language processing (NLP) and computer vision. To fully
Despite its success in Transformers, the application of LoRA to convolutional networks (Con- vNets) remains largely unexplored. ConvNets are often the preferred architecture for various vision tasks [ L8, [9], such as semantic segmentation and object detection, owing to their computational efficiency and robust inductive bias. However, using LoRA adapt pre-trained ConvNets for down- stream vision tasks presents considerable challenges. Unlike Transformers, which utilize inputs with global receptive fields, ConvNets are designed with sequential convolutional layers that have limited receptive fields. Directly applying independent LoRA to each convolutional layer, similar to how it
"CCoresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

20:
200
(a) Rank of weights (pre-trained vs. full-tuned) (b) Rank of weights full (16k vs. 32k iter) (6) Rank of weights: full tunee (144k vs. 160k iter)
Figure 1: Adapting a convolutional network ConvNeXI-B [17]] pretrained on ImageNet to ADE20K [20] requires full-rank tuning of the weight matrix W. The ranks of the weight ma- trix W E Ridd and the weight update matrix AW E Rdddd of all pwconv1 layers in ConvNeXt-B, where d specifies channel dimension. We can that both W and AW almost remain consistent rank d (full-rank) after tuning (a), at the initial (b) or final (c) stages, respectively. Morcover, when concatenating W and AW, the rank is nearly double (2d). As AW resides in totally different parameter space from W, we cannot adapt ConvNets using layer-independent low-rank matrices.
Figure 2: Fine-tuning the pre-trained ConvNext-B | on ADE20K [20]] In order to show the high correlation of adjacent convolutional layers, we compute the mean singular value of their update matrices AW1 (pwconv1) and AW2 (pwconv2) during fine-tuning, Note that their correlation be- comes higher along with the fine-tuning process.
done for Transformers, results in high-rank weight update matrix, complicating the fine-tuning of ConveNets. To illustrate this, we compare the ranks of the weight matrix W and the weight update matrix AW of the ConvNeXt-B [[λ]] model pre-trained on ImageNet-22K and subsequently
adapted to ADE20K. Figure MI illustrates that the rank of the update matrix remains close to full throughout training. Fine-tuning ConvNets with such a high-rank update matrix is unlikely to be efficient. This raises a critical question: Can we overcome the full-rank learning bottleneck in the fine-tuning of ConvNets and develop an effective LoRA method for ConvNets?
ConvNets hierarchically stack convolutional layers to extract features. resulting in a significantly higher similarity between adjacent convolutional layers compared to Transformer layers, as high- lighted by [2I]]. We observe that introducing independent LoRA into each convolutional layer leads to suboptimal adaptation performance, primarily because the correlations between adjacent convo- lutional layers are not accounted for. Therefore, we first examine the relationship between two adjacent linear layers, represented as Y = XW, and Y, Y, According to backpropagation, the gradients of W1 and W2 share a common term, X BY;' which implies that their weight update matrices, AW1 and A 22, are highly correlated. For instance, illustrated in Figure D we ana- lyze the correlation between two adjacent convolution layers, namely pwconvl and pwconv2. Their update matrices, AW, and AW2, show a strong correlation throughout the entire tuning procedure. This correlation has also been explored using the Centered Kernel Alignment (CKA) metric in Ap- pendix DI Thus, is essential to consider the correlations between adjacent convolution layers when fine-tuning on downstream vision tasks.
In contrast to recent research [2 PK [ that focuses on tuning Transformer layers with independent LoRA, our approach explicitly considers the correlation between adjacent convolutional layers using low-rank matrices. We leverage the hierarchical architecture of ConvNets by updating these adjacent convolutional layers together through correlated low-rank matrices. Following the pack-propagation rule applicable to adjacent convolution layers, we introduce an innovative weight update strategy (re- fer to Figure Bic)) that explicitly binds their updates through shared low-rank matrices. This ensures that the weight update strategy effectively inherits the full-tuning behavior of the adjacent layers from gradient perspective. To enhance the tuning efficiency of ConvNets further, we implement filtering- -aaeed scheme that expands the receptive field of the shared low-rank matrices, thereby pre- venting uninformative local regions from dominating the gradients. Based on these strategies, we propose a novel and high-ppefformance method called Correlated Low-RAnk Adaptation (CoLoRA) for ConvNets. Experimental results across the mainstream downstream image classification, seman- tic segmentation, and object detection tasks demonstrate that our CoLoRA significantly surpasses current state-of-the-art PEFT methods. In summary, we present the following contributions:

---

We show that fine-tuning pre-trained ConvNets on downstream tasks is a full-rank learn- ing process. Existing low-rank LoRA methods designed for Transformers cannot achieve satisfactory results for fine-tuning ConvNets.
We propose correlated LoRA for fine-tuning ConvNets. We propose a novel weight update strategy to update adjacent convolution layers together.
We conduct extensive validation on the classification, segmentation, and detection tasks. The proposed CoLoRA for ConvNets performs favorably against the state-of-the-art meth- ods.
2 Related Work
Parameter-effiiien fine-tuning. PEFT [26]] has emerged as practical alternative to full fine- tuning, especially for adapting large-scale pre-trained models to downstream tasks while minimiz- ing computational overhead. Current PEFT methods can be broadly classified into three categories: prompt tuning [26], adapter- based methods [27, 2x]], and LoRA-based approaches [[, 28). Prompt tuning techniques [26, 129 30. KI] introduce learnable visual prompts for tuning pre-trained foun- dation models on downstream vision tasks. Among these, visual prompt tuning (VPT) is recog- nized as a highly parameter-efficient approach, though may struggle with larger domain gaps or more complex downstream vision tasks, such as dense prediction. Recent adapter-based meth- ods [2Z. 32 53,
EEl [55 have gained traction by inserting lightweight bottleneck layers into each Transformer block, and have been extensively studied in both NLP and computer vision. For in- stance, the notable Mona [aei] incorporates multi-scale adapters into the Swin Transformer [GG]] to enhance adaptation abilities image recognition. Further research [67, E8, B9] investigates the po- tential of parallel networks, which adapt deep features using lightweight side modules. Meanwhile, LoRA [6] has gained popularity due its effectiveness and efficiency. By integrating trainable low-rank matrices into each layer of Transformers, LoRA achieves performance comparable to full fine-tuning while significantly reducing the number of trainable parameters. In particular, LoRA effectively modulates the global attention maps of Transformer-based models by injecting indepen- dent low-rank matrices into each layer, which contributes to its flexibility and efficacy tuning. Despite the considerable success of LoRA-bascd approaches in both NLP [ [2] and vision Trans- formers [28], their applicability to ConvNets remains largely uncharted.
25
LoRA and its variants. LoRA [6] approximates the full tuning of pre-trained foundation models by employing layer-independent low-rank matrices. Recent studies have demonstrated that initial- ization schemes, weight update strategies, and rank values significantly influence LoRA's perfor- mance [IZ, 14, IIC, HQ, III 42, 22, zero and random initializations for its low-rank matrices, which inherits a highly random tuning property. Meng et al. [ propose that initializing LoRA with principal components derived from pre-trained weights leads to faster and more effective convergence. Concurrently, LoRA-GA [[4]] seeks align LoRA weight initialization with the gradients from full tuning. terms of weight up- dates, RsLoRA [A]] introduces rank-stable update strategy by reevaluating the impact of scaling factors. Subsequently, Liu al. [II]] break down the weight update matrix into two distinct compo- nents: magnitude and direction. Fan et al. [10] further enhance adaptation by integrating SVD-based initialization with mixture of experts [45,, 26 (MoE). An alternative avenue of exploration involves high-rank LoRA methods, such as SURMs P and HiRA [22], which aim to directly address the limitations imposed by the low-rank assumption through the introduction of higher-rank updates and more sophisticated fusion mechanisms.
23, I15 HK [el The standard approach of Vanilla LoRA utilizes
Most PEFT methods have been developed within the framework of Transformer-basee architectures. which benefit from their modular structure and self-attention mechanisms. However, convolutional networks [177 I8] continue to dominate many practical deployment scenarios due to their efficiency and robust inductive biases [α8]. Recent attempts [97, 32] have merely adapted Transformer based adaptation techniques for ConvNets, neglecting the unique characteristics of convolution, such as its strong correlation in hierarchical feature extraction. Furthermore, the full-rank property of weight updates convolution layers, as demonstrated Figure II poses additional challenges to the low- rank assumption that underlies most existing PEFT methods. Our work addresses this gap by ex- plicitly modeling the correlation between adjacent convolution layers during tuning, providing a principled and efficient PEFT solution specifically designed for ConvNets.
3

---

Pretrained
dk ow;
Pretrained W2 E R
Pretrained
AW; wy'
W2 R
AW:
0%
Aa B2d4
W2 R
Correlated
Yi
Independent
Correlated
Pretrained
dG OW?
Pretrained W; Raodd
Pretrained
HA
W; gase
AW,
(X"
W
Resn
AW, S
AW; B44
Time all parameters!
Full-rank tuning!
Low-rank tuniin!
Input x
Input x
Input
fine-tuning
(c) Comeelaned-LLRAA
Figure 3: An intuitive comparison among full fine-tuning, existing LoRA, and the proposed CoL- oRA. (a) Full fine-tuning requires adjusting all parameters, resulting in substantial computational overhead. (b) Implementing independent full-rank LoRA within convolutional networks leads to a large number of tunable parameters. (c) contrast, our CoLoRA facilitates low-rank tuning by capitalizing on the correlations between adjacent layers.
CoLoRA: Correlated Low-Rank Adaptation for ConvNets
3.1
Preliminaries
The LoRA approach [6] has emerged as a sophisticated and theoreticall--gronneed technique for parameter-effiiiet fine-tuning of large models [S. B3 HS]. In essence, LoRA decomposes the update of pre-trained weight Wo 1d into low-rank matrices, as illustrated in Figure (],), where d and d' represent the input and output feature dimensions, respectively. Mathematically. the update weight with respect to Wo is expressed AW sAB, where A E Rdxr, B E B"xd", r < min (d, d') defines the rank, and serves as a scaling factor.
A useful way to understand LoRA is by comparing the gradient of W in full tuning with the corre- sponding gradient induced by LoRA [Tτ]]. Let gw denote the gradient with respect W during full tuning. We can first compute the gradients with respect to A and B using the following formulas: gA sgw B and gB sAA gw.
=sA''g.. The equivalent gradient of W can then be expressed as:
Ew = sg B+ sAgg = = (gw B" B AAT gw).
Since B is initialized and A randomly initialized (typically using kaiming. uniform [ with independently and identically distributed (i.i.d.) elements (mean 0 and variance o²), the equiva- lent gradient Eq. (I) can be approximated s²ro (for further details, [IO]). ). By selecting Sexpect /rox LoRA tend that there is dilemma choosing a single value of s that can simultaneously align with both the expectation and variance of gw.
effectively match the full tuning scenario the outset. However, con-
Lemma 3.1 If A Rdxr is initialized using the Kaiming uniform method with variance a.A. then we have Var[AAT] = roa where 14 denotes d x d matrix with all elements equal to 1.
The proof is included in Appendix Al Assuming that all elements in gw are i..d. and share distribution of (0, o²), we can calculate the variance of the elements in gw as Var Varlg se Ardoo Consequently, using a scaling factor a will amplify the variance by a factor of +,
1 leading to unstable tuning when the rank small. suitable variance matching scaling factor Vida
derived as Svar
3.2 Decomposing Weight Update in Adjacent Layers
Let us begin by examining two adjacent fully connected layers, which can be viewed straight- forward pair of adjacent 1 x convolutional layers. The relationships are defined as follows:
Y, = XW, +b;
Y2 = Y, + +b2:
(2)

---

where X E R Vxd represents the input with batch size of N and dimensionality of d. The weight matrices are defined as W1 Rdxdd and 2 E IR xd" while b and b denote the corresponding bias terms.
Existing LoRA-based approaches I6, 17, [0] operate under the assumption that the weight matrix W can be updated using low-rank matrices, as illustrated in Figure ((.). When incorporating LoRA updates into Eq. (2), we arrive at independent updates: AW1 81 A1 Br and AW2 A2 B2- However, it is important ing back-propagation. This correlation implies that stacking independent LoRA layers struggles to replicate the behavior of full training and may even undermine the hierarchical feature extraction ca- pabilities of ConvNets. We examine this correlation from the perspective of gradients. In particular, the backward propagation rule applied to Eq. (C)) yields the following relationships:
note that the updates AW1 and 2W2 are inherently correlated dur-
Oc 2W2
àL 0Y.
OL
aL 0Y27
= Y2
= WT x
+
D
bIJN independent
(3)
0Y2
correlated
OC
0L 0Y1
aL
=x
XTT
W.T
(4)
OW,
0Y2
The above equations illustrate that the weight update process in consecutive layers can be decom- posed into two components; a correlated component (the blue part in Eqs. (G3) and (EI) and an independent component. This motivates us to represent the weight updates in adjacent layers in general form:
AW1 SW3 P1, AW2 W P2.
(5)
where S is shared between layers, P1 and Pg are specific updates for each layer.
3.3 Correlated Low-Rank Adaptation
Similar to LoRA, the quantities S, P1, and P2 Eq. (5) can be decomposed into low-rank matrices, leading to the proposed CoLoRA formulation:
AW1 = sA,B.W" + BIA, AW2 = sWTA,B, A,B, + s2A2B2.
(6)
In this formulation, and 82 are layer-specific scaling factors, while s is shared across layers. The matrices A, E _dxrr and B, E IRT. xd" are shared low-rank matrices with rank of r., Conversely, A1 E Rdx,, ; B1 E Rr and A2 E IRd xri, , B2 E R xd' are layer-specific low-rank matrices with rank of ri. Typically, we constrain fold: first, it emulates the correlated updates of gradient descent for sequential layers as described in Eqs. (C) and (+)) Second, by explicitly incorporating pre-trained weights into the tuning update in Eq. (8), it effectively guides the optimization direction compared to vanilla LoRA. Third, Col.- oRA potentially more parameter-efficien than vanilla LoRA. Specifically, LoRA requires a total of r(d 2d'+ +d") parameters, whereas CoLoRA introduces r. r
rs + The advantages of CoLoRA are at least three-
(d 2d' + dd)) parameters, yielding a saving of 2r,d' trainable parameters. For instance, using a ConvNeXt backbone where d' 4d = 4d", CoLoRA achieves reduction of Sr,d parameters (40% when r. d/2).
Scaling factor. For the independent updates outlined in Eq. (6). we find that the Sexpect discussed in section I is effective for both layers. Consequently, we utilize con-trainable scaling factors defined as $1 L/Vri(oA, + TB,) and 82 1/ (riiioaaa + TB,)) However, when it comes to the layer-s update, we notice that Sexpect can lead divergent loss values when fine-tuning on datasets with significant domain gaps, such as Retinopathy sub-task in the 1K [50] benchmark. Therefore, we employ a variance-matcho shared low-rank matrices. Our experiments in image classification, object detection, and semantic segmentation indicate that the configuration of these scaling factors facilitates stable tuning process while ensuring satisfactory performance.
scaling factor svars as defined in section B. for the
B, W and WT A, in Eq. denote alternative low-rank B and A matrices, respectively.
5

---

3.4 Integrating CoLoRA into ConvNets
Our CoLoRA framework is designed to work effectively with every pair of adjacent layers. In this section, we will detail the integration of CoLoRA into standard ConvNets. Specifically, we aim to tackle two critical challenges: (1) the spatial issue of incorporating LoRA into convolutional kernels, which refers to the detrimental gradients that can backpropagate from local irrelevant regions to the LoRA weights, as illustrated in Figure H. and (2) determining the proper locations for applying CoLoRA within a ConvNet, such as vanilla ResNet.
The spatial issue of LoRA in ConvNets. The convolution kernel, denoted W E [gdxd'xkxkrp 1), consists of d input channels and d' output channels and operates on local k x k regions. During the forward pass of a ConvNet, the convolution kernel is applied across each local region in sliding- window fashion. Consequently, the gradients are backpropagated from all local lo x k regions of the output feature to the LoRA weights, as highlighted by the green and red boxes in Figure ever, many of these local regions may contain irrelevant information as underlined in the red boxes in Figure EL Gradients backwarded from such regions can interfere with the adaptation process.
How-
Enlarging the receptive fields of the LoRA weights is a straightforward approach to reduce the im- pact of irrelevant gradients [32]] However, implementing this naively can lead to an excessive num- ber of tunable parameters, scaling as (rrdk²). In this section, we propose refined edge-preserving filtering technique that does not require any training. Given feature map F C
employ channel-wise filtering as follows:
Fech.au
Σ (óh,
ka(öh, bw) Fe,het Fettthhwrtt
Ze.how
where -t)-,-) represents spatial filtering kernel, such as Gaussian kernel e -812 ) with variance o², which can be computed in parallel across all spatial windows Ω. On the other hand, (-).,.) denotes a range kernel, which typi- cally cannot be processed in parallel. Drawing inspiration from the work of [5I]], which uses raised cosine function for (1) Bilateral filtering, we adopt simple range kernel defined as (z, y) = cos(7e( y)) = cos( To2) cos(7ee) + sin (x.x)) sin().9). This formulation allows us to compute spa- tial filtering on c))(T.F) @ F and F) @ F to achieve bigeeprresriiig filtering that expands receptive fields while suppressing noise within features. Further details are avail- able in Appendix Note that Ya is a channel-wise scaling factor designed to constrain Na( Fe
comti gradiert
p aden busetd gredicts
Pre Pa
ongeii
onoprie
Figure 4: Spatial issues arising from applying LoRA to convolution kernels.
1-x/2, m/2). ensuring that the range kernel applies greater weights to similar pixels, which helps preserve distinct edges. Specifically,
calculate "eo as T(max Fi-m F.+ej. where 10 is in- troduced to avoid division by zero. In practice, we apply our filtering method to features compressed by the low-rank matrix A to curtail the computational burden associated with this filtering technique.
Where to apply CoLoRA? We adapt CoL.RA to mimic the weight update behavior of two con- secutive layers. ConvNets such as ResNet-50 [[2]], consist of numerous sequential convolutional layers, and they are constructed from fundamental residual blocks. In this context, we implement our CoLoRA within each residual block by pairing every two adjacent convolutional layers from the bottom up, while using vanilla LoRA on the remaining layer that cannot be paired.
4 Experiments
Setup. We investigate the effectiveness of the proposed CoLoRA on two families of ConvNets: ResNet [152] and the more contemporary ConvNeXt []7]]. For fine-tuning these ConvNets on various downstream tasks, we utilize the official ImageNet-22KK pre-trained ResNet and ConvNeXt model series. Our evaluation encompasses a range of vision tasks, including classification on the large- scale visual adaptation benchmark VTAB-1k [500], object detection on the MS-COCO [ dataset, and segmentation on the ADE20K [20] benchmark. Notably, VTAB-1k comprises 19 tasks that span diverse array of categories, including seven NATURAL, four SPECIALIZED, and eight STRUC-

---

Table 1: Average top-1 accuracy the VTAB-1k benchmark of three runs (See Appendix I| for details on standard deviation). The best results among PEFT methods are in bold.
Calocdiii
Rosteot
Rotinopulyy
Clav--Cam
SNORD- Elee
Barkhome
Methed
*Param
FT
23.5M
89 49 30.82
64.41
39.3 29.35
$4.43
31.00
82.51
65.31
91.83
50.94
73,
45.
27
55.21
45.67
50.1K
41.
90.99
26.54
31
64.50
ResNett
Con--Adpprr (X]
PASSA (0)
1:2M
1.4M
96.
86. 12
08
26.
27. 23
24
04
62.91
60
85.8 85
92.21
89.
88
68.98
82.71
52.67
83. 93
51.31
81
78
59
59
50
88
74
75.
78
48.47
2M
73.8
73
80
46.61
15.94
49.
66.95
98.21
9.40
52
41
50
82
15
23
28
89
73.
99
16.05
16.22
38.21
19.43
55.05
58.68
HiRA (12)
1.2M
K7.00
22.97
63.99
82 19
89.20
12-78
$3.91
07
89 10
45-42
85
40.95
46.09
10
45
16.
71.31
13.53
44.05
56-20
CoLoRA (ours)
1,004
89.12
29.98
62.82
97.51
88.87
32.38
75.31
92.60
92.86
$1.08
74.30
54.04
53.95
42,85
76.42
57.18
79.98
22.58
48.51
63.74
49
05
66.58
83.83
19.95
46.91
16.15
Cow-N
PISSA
Adapeer
2,83
2,000
51.96
49.91
54.04
73.30
73.56
98.11
61.30
92.12
64.91
56.50
CsLoRA
2.6M
TURED tasks. These benchmarks provide a comprehensive study for evaluating the proposed Col.- oRA in comparison state-of-the-art techniques. All experiments are conducted on NVIDIA A800 GPUs using the PyTorh [5_]]] framework, supported by APEX [{>]] for mixed precision training.
We compare CoLoRA with several recent PEFT methods: 1) Conv-Adapter [[2]] which is specif- ically designed for ConvNets and incorporates lightweight adapters into intermediate K x K con- volution layers (K > 1). 2) PiSSA [Z]] which introduces an improved LoRA initialization scheme by decomposing pre- traitrained weights into principal tunable components and frozen residual parts. 3) HiRA [22], the latest high-rank tuning method, which constructs high-rank weight updates from low- rank matrices using the Hadamard product, i.e., AW = W @ (B To ensure a fair comparison, align the number of tunable parameters across different methods by adjusting the ranks of B and A. Specifically, we define the rank as r C/7, following [B2]] where C' represents the channel number and y is a compression factor. Morcover, we adopt existing training-free LoRA merging techniques [24], 56]] to enhance the adaptation ability of CoLoRA. Further details on complexity analysis and training configurations can be found in section B.S. Appendix IB, and Appendix H.
4.1 Evaluation on Vision Adaptation Benchmark
We begin by examining the effectiveness of CoLoRA on the VTAB-1k benchmark, utilizing ResNet-50 and ConvNeXt-S as backbones. We set the compression factor to = 16 and ensure that the ranks of the shared and layer-specific matri- ces are equal, i.e., T. T!. This configuration results in approximately 1.0M trainable parameters for ResNet-50 and 2.6M for ConvNeXI-S, savings 0.2M and 0.4M parameters compared to PiSSA and HiRA, respectively. The average top- 1 accuracy results from three runs are reported in Table MI where CoLoRA demonstrates the highest adaptation perfor- mance among all PEFT methods. When using ResNet-50 as the backbone, CoLoRA achieves average accuracy gains of 8.71%, 5.06%, and 7.54% over Conv-Adapter, PiSSA, and HiRA, respectively. CoLoRA even surpasses full-tuning per- formance with ConvNeXt-S, while recent methods experience significant performance degradation. These results highlight the considerable advantages of mimicking the correlation of weight updates.
Ours
1I8L (Remunnnccinnn
Colande snlo
Inarcw (306
Figure 5: Evolution of segmenta- tion mIoU with the 160k training schedule.
4.2 Comparison on Semantic Segmentation
We opt for ConvNeXt-S and ConvNeXt-B backbones for our evaluations on semantic segmen- tation. Following the mainstream 160k training schedule on ADE20K [20], we utilize the MM- Segmentation [57] framework, employing UPerNet [58] for semantic prediction at a resolution of 512 x 512. The mean Intersection over Union (mloU) metric is calculated using the default single- scale regime. While multi-scale evaluation paradigm can enhance mloU, it also incurs a greater computational burden. We denote the configuration with 7 16 and r, = rı as CoL ooRA and stronger configuration with 7 = 8 and ra = 4r, to reduce parameters as At Further
7

---

Table 2: Quantitative comparison of different PEFT methods on object detection (COCO) and se- mantic segmentation (ADE20K) tasks. The best results are in bold.
Backbone
Method
#Faaram
Objectt Detection (COCO)
(ADF20X)
Segmentation
APa.. 00.55
APa., 80075
APwx
AP..k
mioU
FT
49. SM
70.0
52.5
47.5
43.0
40.87
Com- Adaperr (63] (CNFRW"24) 24)
2.8M
65.7
45.5
41,9
39.3
46.51
ConvNeXt-S
PISSA m (NeaiPS 34)
3.0
65.8
45.3
41.8
39.6
46.50
HIRA ( (ICLR'25)
3.0M
65.3
44.7
41.2
39.1
47.02
Col ARA (ears)
2.6M
66.8
47.9
43.6
40.3
48.28
87.6M
71,4
49.1
46.1
20.99
Mdltte (CMPRW224)
48.75
ConvNeXt-B
(NearPPS22)
[Ca (CCLR'25)
5.3M
67.6
40.8
ARA (ears)
4.6M
69.0
47.0
42.9
CaL.RRA (ours)
5.7M
70.7
52.3
47.5
43.3
50.65
Table 3: Ablation studies on our contributions and designs. "Spatial Conv" denotes convolution layers with kernel size larger than 1, while "Channel Conv" specifies the x convolution layers.
Backhone
Spatial Conv
Channel Conv
Correlated LoRA
Merging LoRA
mloU
LoRA
46.30
ConvNeXi-S
LoRA + Filtering
X
47.04
Filtering
LoRA
LoRA Filtering
LoRA
V
48.00
LoRA + Filtering
LoRA
48.28
details can be found in the Appendix HI A quantitative comparison is presented in Table D where all recent PEFT methods, including Conv-Adapter, PiSSA, and HiRA, experience significant per- formance degradation compared to the full-tuning scenario. Notably, our CoLoRA demonstrates an improvement of 1 1.26 and 0.82 mloU over the previous best methods with ConvNeXt-S and ConvNeXt-B backbones, respectively. Additionally, our CoLoRAi, with slight 1.1M extra param- eters, achieves an mIoU of 50.65, surpassing the current top-performing method by 1.92 mloU, almost comparable to full-tuning. Remarkably, we achieve 51.13 mloU with only 19M tunable pa- rameters (y = 4, = 2/3rt), thereby slightly outperforming full-tuning by an impressive 0.14 mloU gap. More results are available in the Appendix These findings underscore the superior- ity of our CoLoRA and its considerable potential to match or even exceed full-tuning performance. Figure 5 illlustrates the evolution of mloU throughout the entire tuning process of various methods us- ing ConvNeXt-S Our CoLoRA consistently achieves higher mloU during the entire tuning process, indicating superior convergence compared to other methods.
4.3 Experiments Object Detection
In this section, present experiments object detection uti- lizing the standard Faster R-CNN framework [59]. We select ConvNeXt-S and ConvNeXt-B as backbone architectures. All methods are implemented within the MMDetection frame- work [60], adhering to standard Ix schedule over 12 epochs. As in section H.2, we train two variants of the proposed method: CoLoRA and CoLoRAI, The performance metrics include AP APemail, APemaily APhox, and APmusk as detailed in Table El Typ- ically, we adopt single-scale evaluation regime for metric com- putation. From Table E. it is evident that Conv-Adapter, PiSSA, and HiRA experience significant performance degradation com- pared to full tuning. For instance, using ConvNeXI-B as the backbone, the high-rank tuning method, HiRA, shows a decline of 5.6 in AP Even with more effective LoRA initialization, PiSSA fails deliver satisfactory performance. Conversely, in- corporating adapters into convolution layers, specifically Conv- Adapter, yields only marginal improvements over PiSSA and HiRA. In contrast, our CoLoRA results in notable enhancement in performance. Particularly, CoLoRAT achieves 2.6 APbax gain over the pest-performing method, Conv-Adapter, while also reducing the parameter count by 0.6M. These findings underscore the significance of emulating correlated weight updates in fine-tuning ConvNets.
Figure 6: Investigation of the di- mension compression factor.

---

4.4 Extend CoLoRA to Vision Transformer on Fine-grained Classification
We further extend the proposed CoLoRA framework to the Vision Transformer (ViT) architecture
by fine-tuning pre-trained ViT-B model initialized with MAE [3]. The results, summarized in Ta- ble H. present top-1 accuracies on two fine-grained classification benchmarks, CUB-200-2011 [6]]] and FGVC-Aircratt [62]]. As shown, CoLoRA consistently outperforms PiSSA and HiRA, demon- strating superior adaptability within transformer-bes architectures.
4.5 Complexity Analysis
To mitigate the computational overhead intro- duced by the filtering strategy in Eq. (z), we apply forch.jit.scripp optimization and omit gradient computations from the trigonometric kernels in Eq. (I2) to sim- plify the backward pass. A comprehensive com- plexity analysis comparing PiSSA, HiRA, and CoLoRA is presented in Table B. All LoRA- based variants demonstrate reduced training memory consumption and shorter training time compared to full fine-tuning (FT). Notably, the additional cost introduced by the bilateral filtering module is negligible (e.g.. 1.509s 1.527s per iteration). It is also observed that, under mixed- precision training, removing the filtering module may slightly increase GPU memory usage due to precision conversion overhead. During inference, the weight-merging property of LoRA ensures identical runtime across all variants, indicating that the filtering mechanism in CoLoRA does not compromise inference efficiency. Collectively, these observations validate the computational effi- ciency and practical feasibility of the optimized CoLoRA framework.
Table 4: Quantitative results of tuning a ViT- backbone on two fine-grained classification benchmarks, CUB-200-201 and FGVC-Aircrafi.
for operator-level
Datasets
FT
PISSA
HiRA
CoLoRA
CUB-200-2011
79.24
74.70
51.97
75.61
FGVC-Airerafi
79.30
72.40
34.32
72.79
Table 5: Training and inference complexity analysis of the proposed CoLoRA, evaluated on the MS-COCO dataset using a ConvNeXt-S backbone.
CoL.oRA (w/o filtering)
Methods
FT
PiSSA
HiRA
CoLoRRA
Training GPU memory (MiB)
18418
14648
14536
15042
16488
Per-step training time (s)
1.747
1,426
1.501
1.527
1.509
Inference time
0.108
0.108
0.108
0.108
0.108
4.6 Ablation Studies
In this section, we perform ablation studies to validate the effectiveness of our contributions and designs. All experiments are conducted using the ADE20K semantic segmentation benchmark, fea- turing ConvNeXt-S as our backbone. We use ConvNeXt-S with LoRA applied exclusively to the depth convolution layers
baseline, as suggested by recent study [32]]
Does filtering improve performance? We incorporate our filtering method detailed Eq. ( into the baseline. As indicated in Table BL this adjustment increases the mloU from 46.30 to 47.04, yielding gain of 0.74 mloU. This finding demonstrates that expanding the receptive field of LoRA weights positively influences the tuning of ConvNets.
Where should low-rank adaptation be applied? Recently, Conv-Adapter [> proposed tuning solely the spatial convolution layers while leaving the 1 x convolution layers untouched. Notably, adjacent 1 x convolution layers operate as an inverted bottleneck design within the ConvNeXt [[77]] architecture. This design through MobileNetV2 [18] has been widely explored in advanced Con- convolution layers could yield additional performance gains. However, as illustrated in Table E, applying LoRA to these layers results in only a modest 0.1 mloU improvement, which aligns with the results of Conv-Adapter To explore this contradiction further, we conducted full tuning of all 1 x convolution layers in Co-S while keeping the other layers frozen. This configuration achieves an mloU of 50.45, exceeding the performance of full-tuning (49.87 in Table D) Hence, conclude that the x 1 convolution lay-
vNets 10 63). We hypothesize that tuning the 1

---

ers exhibit significant potential for tining. However, this potential cannot be effectively harnessed through anyer-independen LoRA or adapters.
Impact of correlated weight updates. We show that the potential of the inverted bottleneck can be effectively harnessed through the proposed correlated weight update method (as detailed in sec- tion ( As shown in Table C we improve the mean mIoU from 47.14 14 to 48.00 using the correlated LoRA update. We achieve an impressive mloU of 50.65 while only tuning 5.7M parameters, out- performing the CoaveAdapter, which still lags behind full-tuning despite having approximately 50% more tunable parameters (39.40M; refer to [32] for further details).
Impact of Periodically Merging LoRA Weights. Figure W illustrates that tuning pre-trained mod- els for downstream tasks closely resembles nearly full-rank learning process. In this study, leverage the current LoRA merging technique [24]] to enhance the rank of AW. Specifically, we periodically merge the weight updates computed via Eq. (8) into the pre-trained weights,and subse- quently re-initialize the LoRA matrices. Following the suggestions of [24]], we reset the optimizer and implement restart warmup schedule to stabilize the tuning process. This approach yields noticeable improvement of 0.28 mIoU on ConvNeXI-S, as depicted in Table B.
Analysis on various dimension compression factors. We analyze the effect of the compression factor, γ. A smaller value of r generally results in higher-rank weight updates with an increased number of tunable parameters. Figure @ visualizes the ratio of tunable parameters alongside the mloU metric for different y. Notably, full-rank tuning can be achieved with a tunable parameter ratio of 75% (< 1) due the phenomenon of parameter sharing. Empirically, this full-rank tuning yields a segmentation mloU of 49.5, which is comparable to the full-tuning. Furthermore, even at compression factor of y 32, the proposed CoLoRA framework surpasses the performance of Conv-Adapter by margin of 1.67 mIoU, despite utilizing only 1.4M tunable parameters, which is less than half of the parameters for Conv-Adapter. When progressively reducing 7 from 32 we consistently observe an increase in mIoU from 48.2 to 48.8. Moreover, maintaining y within the range of [1, 32] systematically results in enhanced convergence when compared to state-of-the-art methods as shown in Figure E.
Analysis the rank of A. and B.. We explore how perfor- mance metrics vary under different correlation strengths as delin- maintain fixed compres-
Table 6: Effect of r..
eated in Eq. (6)) For experiments, sion factor at = 16 while modifying the ratio rutri- An increase in this ratio signifies a stronger correlation. As demonstrated in Table i. we identify that the exclusive correlation of LoRA updates produces a mean mloU of 47.60. The optimal mIoU is attainable through the combined application of both correlated and layer-specific LoRA updates, this strategy effectively emulates the gradient behavior outlined in Eq. (E))
0
50%
mloU
47.14
48.28
47.60
5 Conclusion
This paper presents CoLoRA, innovative and highly PEFT methodology designed for the adap- tation of pre-trained ConvNets across a variety of downstream applications. Rather than solely incorporating independent LoRA to each layer in ConvNets, the proposed method focuses on em- ulating the correlated weight update behavior between adjacent layers. In addition, we introduce parameter-free filtering strategy aimed at the seamless integration of correlated weight updates within ConvNets. Through extensive experiments encompassing large-scale classification, semantic segmentation, and object detection, we demonstrate that CoLoRA significantly outperforms existing PEFT approaches.
6 Limitations
Despite its effectiveness, CoLoRA still has several limitations. First, this work primarily focuses on adapting pre-trained vision backbones for various downstream vision tasks, leaving its poten- tial on emerging vision-language model-based perception tasks, such as visual question answering and visual grounding unexplored. In addition, this study mainly investigates 2D vision scenarios. Extending A to domains may introduce new challenges and opportunities, further broad- ening the scope and impact of this research.
10

---

Acknowledgementss This work was supported in part by NSFC (62322113, 62376156), Shang- hai Municipal Science and Technology Major Project (2021SHZDZX01022, and the Fundamental Research Funds for the Central Universities.
References
[1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the Advances in Neural Information Processing Systems, pages 1-11, 2017.
[2] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint 1rXiv:2010.11929, 2020.
[3] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 16000-16009, 2022.
[4] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of the International Confer- ence Machine Learning, pages 8748-8763, 2021.
[5] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 222.::::230.1197 2023.
[6] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In Proceedings of the International Conference on Learning Representations, 2022.
[7] Fanxu Meng, Zhaohui Wang, and Muhan Zhang, Pissa: Principal singular values and singu- lar vectors adaptation of large language models. In Proceedings of the Advances in Neural Information Processing Systems, pages 121038-122 1110822 2024.
[8] Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models. arXiv preprint 2r02v:2402.1220 /2354, 2024.
[9] Uijeong Jang, Jason D Lee, and Ernest K Ryu. Lora training in the ntk regime has no spurious local minima. In Proceedings of the International Conference Machine Learning, pages 21306-21328. PMLR, 2024.
[10] Chenghao Fan, Zhenyi Lu, Sichen Liu, Xiaoye Qu, Wei Wei, Chengfeng Gu, and Yu Cheng. Make lora great again: Boosting lora with adaptive singular values and mixture-of-experts optimization alignment. arXiv preprint 1688:2550..66894 2025.
[11] Reece Shuttleworth, Jacob Andreas, Antonio Torralba, and Pratyusha Sharma. Lora vs full fine-tuning: An illusion of equivalence. arXiv preprint arXiv:2410.21223 2024.
[12] Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Cheng-Zhong Xu. Hydralora: An asym- metric lora architecture for efficient fine-tuning. In Proceedings of the Advances in Neural Information Processing Systems, pages 9565 9584, 2024.
[13] Xiaojun Xiao, Sen Shen, Qiming Bao, Hongfei Rong. Kairui Liu, Zhongsheng Wang, and Jiamou Liu. Cora: Optimizing low-rank adaptation with common subspace of large language models. arXiv preprint arXiv:2409.02// 19, 2024.
[14] Shaowen Wang, Linxi Yu, and Jian Li. Lora-ga: Low-rank adaptation with gradient approx- imation. Proceedings of the Advances Neural Information Processing Systems, pages 54905-54931, 2024.
11

---

[15] Paul Albert, Frederic Z Zhang, Hemanth Saratchandran, Cristian Modriguez-Opazo Anton van den Hengel, and Ehsan Abbasnejad. Randlora: Full-rank parameter-efficient fine-tuning of large models. arXiv preprini
2025.
[16] Shubhankar Borse, Shreya Kadambi, Nilesh Pandey, Kartikeya Bhardwaj. Viswanath Ganapa- thy, Sweta Priyadarshi, Risheek Garrepalli, Rafael Esteves, Munawar Hayat, and Fatih Porikli. Foura: Fourier low-rank adaptation. In Proceedings of the Advances in Neural Information Processing Systems, pages 71 71504-71539, 2024.
[17] Zhuang Liu, Hanzi Mao, Chao- Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 11976-1 1986, 2022.
[18] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4510-4520, 2018.
[19] Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In Proceedings of the International Conference Machine Learning, pages 6105- 6114, 2019.
[20] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralb.. Scene parsing through ade20k dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 633-641, 2017.
[21] Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovit- skiy. Do vision transformers see like convolutional neural networks? In Proceedings of the Advances in Neural Information Processing Systems, pages 12116-12128, 2021.
[22] Qiushi Huang, Tom Ko, Zhan Zhuang, Lilian Tang, and Yu Zhang. Hira: crramtter-ffficent hadamard high-rank adaptation for large language models. In Proceedings of the International Conference on Learning Representations, 2025.
[23] Zhuo Chen, Rumen Dangovski, Charlotte Loh, Owen Dugan, Di Luo, and Marin Soljacic. Quanta: Efficient high-rank fine-tuning of Ilms with quantum-informed tensor adaptation. In Proceedings of the Advances in Neural Information Processing Systems, pages 92210-92245, 2024.
[24] Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde, and Anna Rumshisky, Relora: High- rank training through low-rank updates. In Proceedings of the International Conference on Learning Representations, 2024.
[25] Yi Xin, Siqi Luo, Haodi Zhou, Junlong Du, Xiaohong Liu, Yue Fan, Qing Li, and Yuntao Du. Parameter-efficicn fine-tuning for pre-trained vision models: A survey. arXiv preprint 20Xi:22402.02242. 2024.
[26] Menglin Jia, Luming Tang, Bor- Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In Proceedings of the European Conference on Computer Vision. pages 709-727. Springer, 2022.
[27] Dongshuo Yin, Leiyi Hu, Bin Li. and Youqun Zhang. Adapter is you need for tuning visual tasks. arXiv preprint arXiv:2311.15011 2023.
[28] Yuedong Yang, Hung- Yueh Chiang, Guihong Li, Diana Marculescu, and Radu Marculescu. Efficient low-rank backpropagation for vision transformer adaptation. Proceedings of the Advances in Neural Information Processing Systems, pages 14725-14734 2023.
[29] Yun-Yun Tsai, Chengzhi Mao, and Junfeng Yang. Convolutional visual prompt for robust visual perception. In Proceedings of the Advances Neural Information Processing Systems, pages 27897-27921, 2023.
[30] Xing Nie, Bolin Ni, Jianlong Chang, Gaofeng Meng, Chunlei Huo, Shiming Xiang, and Qi Tian. Pro-tuning: Unified prompt tuning for vision tasks. IEEE Transactions on Circuits and Systems for Video Technology, 34(6):4653-4667, 2023.
12

---

[31] Shishuai Hu, Zehui Liao, and Yong Xia. Prosfda: Prompt learning based source- free domain adaptation for medical image segmentation. arXiv preprint arXiv:2211.11514. 2022.
[32] Hao Chen, Ran Tao, Han Zhang, Yidong Wang, Xiang Li, Wei Ye, Jindong Wang, Guosheng Hu, and Marios Savvides. Conv-adapter: Exploring parameter efficient transfer learning for convnets, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1551-1561, 2024.
[33] Dongshuo Yin, Yiran Yang, Zhechao Wang, Hongfeng Yu, Kaiwen Wei, and Xian Sun. 1% vs 100%: Parameter-efficient low rank adapter for dense predictions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 201 1011161622, 2023.
[34] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang. Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. In Proceedings of the Advances in Neural Information Processing Systems, pages 16664-16678, 2022.
[35] Shibo Jie, Zhi-Hong Deng, Shixuan Chen, and Zhijuan Jin. Convolutional bypasses are better vision transformer adapters. In Proceedings of the European Conference on Artificial Intelli- gence, pages 202-209.. 2024.
[36] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceed- ings of the IEEE International Conference on Computer Vision, pages 10012-10022, 2021.
[37] Minghao Fu, Ke Zhu, and Jianxin Wu. Dtl: Disentangled transfer learning for visual recog- nition. In Proceedings of the AAAI Conference Artificial Intelligence, pages 12082-12090, 2024.
[38] Dongshuo Yin, Xueting Han, Bin Li, Hao Feng, and Jing Bai. Parameter-efficient is not suf- ficient: Exploring parameter, memory, and time efficient adapter tuning for dense predictions. In Proceedings of the ACM International Conference on Multimedia, pages 1398-14066 2024.
[39] Xuchai He, Chunyuan Li, Pengchuan Zhang, Jianwei Yang, and Xin Eric Wang. Parameter- efficient model adaptation for vision transformers. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 817-825, 2023.
[40] Damjan Kalajdzievski. A rank stabilization scaling factor for fine-tuning with lora. arXiv preprint arXiv:2312.033733 2023.
[41] Shih- Shih-Yang Liu, Chien-Yi Yi Wang, Hongxu Yin, Pavlo Molchanov. Yu- Yu- fu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposs low-rank adaptation. In Proceedings of the International Conference Machine Learning, pages 1-22, 2024.
[42] Arijit Schanobish, Kumar Avinava Dubey, Krzysztof M Choromanski, Somnath Basu Roy Chowdhury, Decpali Jain, Vikas Sindhwani, and Snigdha Chaturvedi. Structured unrestricted-rank matrices for parameter efficient finetuning. In Proceedings of the Advances 2024.
in Neural Information Processing Systems, pages
[43] Zihan Zhong, Zhiqiang Tang, Tong He, Haoyang Fang, and Chun Yuan. Convolution meets lora: Parameter efficient finetuning for segment anything model. In Proceedings of the Inter- national Conference on Learning Representations, 2024.
[44] Jui-Nan Yen, Si Si, Zhao Meng, Felix Yu, Sai Surya Duvvuri, Inderjit S Dhillon, Cho-Jui Hsieh, and Sanjiv Kumar. Lora done rite: Robust invariant transformation equilibration for lora optimization. arXiv preprint arXiv:2410.20625 2024.
[45] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gates mixture-of-experts layer. In Proceedings of the International Conference on Learning Representations, 2017.
[46] Xiaoye Qu, Daize Dong, Xuyang Hu, Tong Zhu, Weigao Sun, and Yu Cheng. Llama-moe v2: Exploring sparsity of lama from perspective of mixture-of-experts with post-training. arXiv preprint arXiv:2411.15708, 2024.
13

---

[47] Bijay Adhikari, Pratibha Kulung, Jakesh Bohaju, Laxmi Kanta Poudel, Confidence Raymond, Dong Zhang. Udunna C Anazodo, Bishesh Khanal, and Mahesh Shakya. Parameter-efficiee fine-tuning for improved convolutional baseline for brain tumor segmentation in sub-saharan africa adult glioma dataset. arXiv preprint arXiv: 14/00, 2024.
[48] Yuchen Zeng and Kangwook Lee. The expressive power of low-rank adaptation. arXiv preprint arXiv:2310. 17513, 2023.
[49] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Sur- passing human-level performance imagenet classification. In Proceedings of the IEEE In- ternational Conference on Computer Vision, pages 1026-1034, 2015.
[50] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv: 1910004867 2019.
[51] Kunal Narayan Chaudhury, Daniel Sage, and Michael Unser. Fast o(1) bilateral filtering us- ing trigonometric range kernels. IEEE Transactions on Image Processing, 20( 2013233336-3332. 2011.
[52] Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion. pages 770-778, 2016.
[53] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco: Common objects in context. In Pro- ceedings of the European Conference on Computer Vision, pages 740-755, 2014.
[54] A Paszke. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv: 2019.
[55]
http://ggitubbccom/////AAApppp
[56] Wenhan Xia, Chengwei Qin, and Elad Hazan. Chain of lora: Efficient fine-tuning of language models via residual learning. In Proceedings of the ICML Workshop on LLMs and Cognition, pages 1-9.
[57] MMSegmentation Contributors. Mmsegmentatiion Openmmlab semantic segmentation tool- box and benchmark, 2020.
[58] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. Proceedings of the European Conference on Computer Vision, pages 418 434, 2018.
[59] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Proceedings of the Advances in Neural Information Processing Systems, pages 1-9, 2015.
[60] Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint ar Ttt1::19060075555 2019.
[61] Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongie. The caltech- uesd birds-200-2011 dataset. 2011.
[62] Subhransu Maji. Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. arXiv preprint arXiv: /36.5/5//.511110. 2013.
[63] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe V Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceed- ings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 2820-2828. 2019.
14

---

[64] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning. pages 3519-3529. PMIR, 2019.
15

---

NeurIPS Paper Checklist
The checklist designed to encourage best practices for responsible machine learning research, ad- dressing issues of reproducibility, transparency. research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (12 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evalu- is perfectly acceptable answer "[No] provided proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can be found.
ation. While "[Yes] " generally preferable to "[No]
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Check- list".
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our contributions briefly discussed in the Abstract, and then carefully elaborated Section M. Both the Abstract and Section M precisely articulate the scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
fine to include aspirational goals motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
16

---

Justification: Please see Section R for details on the limitations. We also include complex- ity analysis in section 4.S and Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Please refer to Sections
2 and 3 for assumptions and theoretical results for
the proposed correlated low-rank adaptation. Please Section 3.44 for theoretical results on the proposed filtering technique. Sections A and B include details of theoretical proof and derivation.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theo- rems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data provided or not)?
17

---

Answer: [Yes]
Justification: We use the publicly-accessal datasets VTAB-IK [SD], MS-COCO [ and ADE20K [20]] In Section H. we carefully present the settings of our CoLoRA, as well as compared methods. In Section III we provide details model architecture, optimizer, learn- ing rate scheduler, etc. The code is available at CoLORA.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our experiments are based on publicly-accessale datasets VTAB-Ik [[0]] MS-COCO [[53] and ADE20K [20]] Notably, we implement all methods under the public MMMclassification, MM-segmentation, and MM-detection framework. We have provided the detailed configuration of these methods Sections and EI Our code available at
attps://githut com/ htt.a/cc///SSSOOSSTT/OO....
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
18

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:)
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Please Sections BL and HI for experimental settings. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: As stated in Section H.ll we run all methods on VTAB-1k three times. We report the standard deviation concerning the top-1 accuracy in Section I. We do not report the standard deviation results on ADE20K and MS-COCO datasets considering resource limitations. In practice, fine-tuning Convnext-S / ConvNext-B ADE20K requires one NVIDIA A800 GPU and about two days for training. Furthermore, fine-tuning Conv S / ConvNext-B on MS-COCO requires two NVIDIA A800 GPUs and about 2.5 days for training.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
19

---

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please find details in Sections 3 and
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't i make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpp////.aarr....oo////////scse
Answer: [Yes]
Justification:
The paper conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We only use publicly available datasets and models. We believe that there exists societal impact in this work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
20

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e-g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: We only use publicly available datasets and models. Also, we do not foresee any high risk of misuse of this work.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere usage guidelines restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We credited them in appropriate ways. We carefully cite the original papers that produced the datasets in Section !
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should provided.
If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
21

---

Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, cura- tion, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
22

---

The core method development in this research does not involve LLMs as any important, original, or non-standard components. We only use LLM such GPT-4 for grammar checking and expression re-writing to make the paper easily understood.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
23

---

Appendix
A Proof of Lemma 3.1
Suppose all elements in R2 are i..d., initialized with the uniform distribution 4(-0.00 a). Typically, we have B[A,j] 0 and Var B[A = za Notably, we have o = 3a² Denote the matrix multiplication result as T AA"; then we can calculate
Σ E[A (ro..
i i
BT& = IE
A
(8)
else
Therefore, we have E[AAT] the variance of the elements T. For the off-diagonal elements of T, we have
ro Ia. where La denotes the d d identity matrix. Next, we analyze
Var[Tig]
E[A A3 r. s.L,i
(9)
k=1
For the diagonal elements, we have
4 -
Var[Ta]
B[A - 1[
(10)
km
where we can compute E[A[+] obtain the variance of AA as
S0 x dd = faa² fo
Summarizing the above results, we
(ro'
i i
Var[Ty]
(1I)
else.
In this paper, we approximate Var[T] = ror 1a where 1, means a d x d matrix with all elements being The approximation error terms of the Frobenius norm can be formed
Err
[[ar[]]] - 10
25 do
(12)
Note that with kaiming uniform initialization where x 1 1/vd, the approximation error Err xx r² /2/d.. Hence, we have
Err
1
/2
(13)
[ar[]]]
25 d2 25d'
It is evident that Var is independent of the rank and is inversely proportional to the channel dimension d. When d is large (e.g.. 96, 192, 384, 768 for Con-NeXt-B), approximate Var [ z A with a maximum relative error 1/525 x 96) = 0.04%.
B Tuning with Various LoRA Ranks
To provide a more rigorous justification for the high-rank tuning strategy of LoRA, we conduct experiments by fine-tuning ConvNeXt-B using PiSSA under varying ranks. For convolutional net- works, the rank is parameterized by a channel compression factor where r d/q and d denotes the number of channels.
Notably, = corresponds to full-rank adap- tation. It is worth mentioning that LoRA intro- duces paraettaamettintioonnn when r = due the decomposition AW sBA. resulting in more trainable parameters than the original full-rank weight. Quantitative results are sum- marized in Table M Specifically, performance peaks
Table 7: Tuning ConvNeXt-B ADE20K with different ranks PiSSA.
20
#aramss (tunable)
104.8M
52.5M
26.3M
13.2M
5.3M
Param ratio (tunable) mloU
120%
60%
30%
15%
6%
50.1
50.6
49.9
49.1
47.6
while both under- and over- parameterization lead to degradation. Increas-
ing 7 consistently reduces mloU from 50.6 to 47.6. These findings validate that independent LoRA tuning on Con-NeXt-B benefits from high-rank adaptation, whereas performance deteriorates under tight parameter constraints. This analysis will incorporated into the revised manuscript further substantiate our argument.
24

---

(a) CKA among all blocks
(b) CKA among 1wwconvl/wwoonv2222 layers (c) CKA among all blocks in ViT-B/I6
Figure 7: CKA analysis for ConvNeXt-B and VIT-B/16. (a) shows CKA similarities among all 36 blocks. (b) presents CKA similarties among all 36 pwconv1 and pwconv2 layers. (c) examines CKA similarities among all 12 blocks within ViT-B/16.
C More Evidence on Weight Update Correlation in ConyNets
In this section, we provide additional evidence to investigate the correlation between adjacent convo- lution layers. Specifically, we select the ConvNeXt backbone and examine the correlation between the adjacent puconvl and psconv2 layers. We denote their convolution weights as W1 and W2. respectively. To demonstrate that the updates of W1 and W₂ are highly correlated, compute AW by subtracting the old weight Wold from the new weight W new. i..., AW W new Wola- Typically, Wala and Weew correspond to the weights saved from the previous and current epochs, respectively. For ADE20K, model weights are checkpointed every 16K iterations, while for MS-COCO, weights saved after every epoch.
Since both W1 and W₂ exhibit full-rank update behavior, we assess their correlation by computing the mean of the singular values of AW, and AW2 If their mean singular values exhibit similar trends throughout
fine-tuning process, it indicates that the update weight matrices for Wi and W2 are highly correlated.
Figure & presents the mean singular values of AW1 and AW₂ throughout the fine-tuning process on ADE20K. Specifically, we analyze the weights of the pwconvi and pwconv2 layers across all residual blocks in ConvNeXt-B. It can be observed that nearly all adjacent pweonv1 and pwconv2 layers exhibit strong correlation in their update behavior.
Figure El shows the mean singular values of AW, and AW2 throughout the fine-tuning process on MS-COCO. Specifically, we analyze the weights of the pwconv1 and pwconv2 layers across all residual blocks in ConyNeXt-S. It can be observed that nearly all adjacent pwconvi and pwconv2 layers exhibit strong correlation. The sharp drop in mean singular values is attributed to the learning rate scaling in the standard 1 x schedule. The results in Figures N and E collectively demonstrate that adjacent convolution layers exhibit high correlation during fine-tuning on downstream tasks such as semantic segmentation and object detection.
D Centered Kernel Alignment Analysis
In this section, we further investigate the correlation between adjacent layers or blocks within Con- vNets and ViTs. We employ the Centered Kernel Alignment (CKA) metric [64]] to quantify the similarity between specific layer or block pairs. The CKA metric offers the following advantages:
CKA is invariant to orthogonal transformations and scaling.
CKA remains applicable when the compared features have different dimensionalities, whereas metrics such as the Frobenius inner product cannot be used in such cases.
CKA has been widely adopted to analyze the representational behaviors of ConvNets and Vision Transformers (VITs) [2]]].
We first select pre-trained ConvNeXt-B model consisting of 36 convolutional blocks. As shown in Figure (aa), the CKA scores across blocks reveal that long-range block pairs exhibit lower similarity, whereas adjacent blocks consistently show higher similarity. This indicates that nearby
25

---

Migett 3yanen)
Megen Syaneo)
Mirgss sereeco
667
320
K0
041
662
602
601
Branaite
032
A32
Biyeu Zontond
677
0.32
042
300
042
M2
M22
602
640
oayau Sunues
Contect
dryad Sumuect
13
X77
0.3
082
M6-
644
482
032
ram 12paren
116
eamn Hyparedd
516
A34
630
2.1
3.2
6.2
3.7
Cape- Dhomsnn
6279
1800118 D011
Man0r110 3011
2.3
0.3
(2)
2.22
A35
2.2
M45
Figure 8: Correlation between pwconv1 and pwconv2 layers all 36 residual blocks. We fully fine-tune the pre-trained ConyNeXt-B on ADE20K.
convolutional blocks are strongly correlated. Furthermore, we employ CKA to analyze the similarity between adjacent convolutional layers puconv1 and pwconv2 illustrated in Figure (Z(b). The results show that these paired layers exhibit high similarity, suggesting strong correlation between adjacent convolutional layers. Based these findings, we infer that such inter-layer correlations persist during downstream fine-tuning as well.
We extend the CKA to pre-trained ViT-B/16 model and provide the result in Figure ((c). It is obvious that the correlation strength within Transformer blocks are weaker than convolution blocks, suggesting lower correlation strength at the fine-tuning process.
E Details of slltering-based Strategy
We first recall in Section 3.4 that the filtering process can be formulated as
Fehha Zeh.ow (6h,ōw)ER
Σ
kōhh,, (r Fo &o. F.t htch.
(14)
where kig (8h, dw) denotes spatial kernel, whereas ki (ee,hor Fant+ tw represents range kernel in the context of image filtering. Without the range kernel kr, the above Eq. (l4) can be
26

---

036
015
Layen Lywirl
Jagen
Maye
544
0.33
1617
140
027
412
1125
022
dlpec Jumand
cayss Lawied
daps
5303
6.08
000
6009
5:0
827
612
16
13
6
0/5
617
10gh (epper
012
Mayed unord
618
685
6.3
6.8
219
Piges 222
Maper ueen?
518
13
Roat seand
Bost Loaren
-m. naon Lavers
1130 22 soardd
03
Figure 9: Correlation between pwconvi and puconv2 layers in all 36 residual blocks. We fully fine-tune the pre-trained ConvNeXt-S on MS-COCO.
simplified as
Fe.h. Ze (6h,5w)ER
1
Σ
(,hh,, Fchhh
Rs Conv Fc) Zo hww
(15)
where the normalization factor Z, convolution operation on X utilizing a kernel K. Eq. (I55) indicates that a channel- wise filtering on deep features can be expressed depthwise convolution operation, which can be rapidly computed with the help of existing algorithm.
-Z (5h,,, d')) is spatially invariant. Conv (K, X) means
However, with the range kernel (,,),, Eq. (L4) requires double-loop process through all (h, w)
coordinates. this paper, we utilize special range kernel (,,, y) cos(Te( - 9) cos()ar) cos())8)) sin(Too) sin(rey), sin()))) where %o is constant to constrain that 7e(1
27

---

1-m/2, Incorporating this range kernel into Eq. (LE), we derive
Fo,hw Zoihiu (5h,6w)en
Σ
(Fo,b,w o Fo, Foh (ôh, 8w (Fr,
Σ
(öh, ōw) cos YoFo, cOs Yo c. wethh
Zen (ai,E
(ōh, 8w) sin Fe sin Ve Fe f.h Fw Fc. wo
heh,ww (éh,6w)eR
1oos ye @ Conv(kis, COS Y.P F @ FM w + [sin F.F F ④ Conv(ks, sin F ④ F)
(16)
Ze.h.w
where the 3 means the element-wise multiplication. Specifically, the normalization factor is
Zehhi
Σ
Kss (8h, 8w) cos Ye Fe f
(óh,
Σ
(ih, (() sin Yo Fehh, sin Ve Fe
(Eh
|cos Yoo @ Conv(k,, cos YeF) + sin YoF @ Conv sin (r
(17)
Combining Eqs. (IG) and (LT))
[) we obtain the final simplified form of Eq. (144 as follows:
F cos Y Conv(k,, cos YoF F)) + sin %elf @ Conv sin 7 о F). cos F @ Conv(ks, cos F)) ) + sin F.F @ Conv(kg, sin F) +
(18)
where we set c = 10-5 5 to avoid division by zero. As shown in Eq. (LS), only four depthwise convolution operations required to compute Eq. (IE),, eliminating the need for time-consuming for-loops. For implementation, we employ a fixed Gaussian spatial kernel k2 with a window size of 11 and standard deviation of 1.0. Consequently, Eq. (IE) contains no trainable parameters, and gradients need not be computed for this filtering process.
Table 8: Complexity analysis of different LoRA methods. Here, denotes the rank of matrices A and B, #Params represents the number of trainable parameters in LoRA matrices, and FLOPs measure the computational complexity of LoRA modulation during both training and inference phases.
Method
Rank r
#Params
FLOPs (train)
FLOPs (inference)
LoRA
r(d - 2d + d²))
(hond' + rd' + d'd +d") d()) (hood' + d" + d') +d)) (hord' +rdd + 2d [dd")(d=)) (d d²)
haod (d + d") haod (d + d")
PISSA
r(d - 26 + +d")
HiRA
r(d 28 + +d")
hodd (d + d")
CoLoRA (ours) + Fs = r(d + d') 2rid'
(hwrd + rdd 2d' + red d²)
hadd (d + d'))
F
Computational Complexity Analysis
The computational complexity of CoLoRA stems from both the proposed correlated low-rank adap- tation process and the filtering technique.
Complexity of correlated low-rank adaptation. We analyze the complexity of two sequential 1 x convolutional layers as described in Section This analysis can be extended to arbitrary convolution layers. Consider input feature X E Rgww (with spatial dimensions rearranged for simplicity), where d denotes the channel dimension and h, w represent its spatial resolution. The feature is processed by two weight matrices W1 e ddd and W2 E Rd xd" For clarity, we omit the analysis of bias terms they are independent of LoRA. The vanilla LoRA modifies the feature X through
Yi = X(W, + A,
(19)
Yn2 = (W,, + B2),
(20)
28

---

where A1 E Raxe, B1 E R"xd" A2 E R and B2 E Rexd" are low-rank matrices with rank r. while 81 and 82 denote scaling factors. We contend that such layer-independent LoRA formulations fail to capture inter-layer correlations, which are particularly crucial for effectively fine- tuning convolutional networks. To overcome this limitation, we propose CoLoRA, which transforms the feature X through
Y = X(W, + + B + sA.B.W2) Y/2 = (1 + s2 + SWTA, A,),),
(21)
(22)
where we incorporate shared low-rank matrices A, Rdxr. and B3 R'x to explicitly model correlations between adjacent layers. To efficiently compute A, A,B.W2 in Eq. (22)), we first compute
C = B, W2 followed by the product A,C, leveraging the low-rank property ra min d' d").. The computation of W A, follows an analogous procedure. Table 8 compares the computational complexity of LoRA, PiSSA, HiRA, and our proposed CoLoRA
During training, CoLoRA introduces an additional computational cost of r.,d' (d + d") FLOPs com- pared to existing HiRA methods. Note that since r. < min d, d', dd), this overhead is negligi- ble in practice. Furthermore, due to CoLoRA's weight decomposition formulation, we can directly incorporate the weight update matrices into the base weights W, and W2 after training completion. Consequently, CoLoRA introduces no additional computational overhead during inference.
Complexity of filtering technique. As demonstrated in Eq. (18), the filtering operation can be efficiently implemented using four depthwise convolutions. To optimize computational efficiency, we apply the filtering process exclusively to features compressed by the LoRA matrix A. This design
yields an overall computational complexity of (hwwkrr), where & denotes the window size (A: = 11 in our implementation).
G Details of Selected Pre-trained Backbones
In Table E. we present the specifications of the pre-trained ConvNets backbones used in our study,
Table 9: Details of selected pre-trained backbones in this paper.
Backbones
dataset
#arams
Public
ResNet-50
ImageNet-IK IK
23.5M
2048
hrtpa 7et:7//ooaa pysorch. ore/visson /sin//odelr 1 3 (v2)
ComNNcXi-S
Ima
49.5M
768
Rttpe 77 desppubicciiee con/ convneatt convaest, stall 22k 12 pa
ComsNeXi-B
ImageNet222K
87.6M
1024
/ Fba pub 20n/
228
H Training Configurations
We conduct experiments by training ResNet-50, ConvNeXt-S, and ConvNeXt-B models on mul- tiple downstream vision tasks. The detailed training configurations are provided in this sec- tion. The basic experimental configurations employed in our study are summarized in Tables III and [ Notably, our implementation follows the official configurations from https://githut. comm htt.faabbokkesercc/oooo....
Basic configuration. The baseline configuration employs consistent hyperparameters across dif- ferent downstream tasks and PEFT methods (with potential exceptions for MS-COCO, as detailed below). Tables III and D present the complete specifications of baseline hyperparameters and train- ing configurations.
Training configuration on VTAB-Ik. For VTAB-Ik classification tasks, our processing pipeline consists of a Global Average Pooling (GAP) layer, followed by normalization and linear classi- fier head. We optimize the models using standard cross-entropy loss. Each VTAB-1k sub-task is fine-tuned with a batch size of 32 for 100 epochs. In addition to the trainable low-rank matrices, we optimize all bias terms and normalization layers the backbone network. Unlike existing ap- proaches such as Conv-Adapter that perform grid search to identify optimal hyperparameters for individual sub-tasks, we maintain consistent hyperparameter settings across all sub-tasks. The spe- cific configurations for each PEFT method are detailed below:
29

---

Table 10: Basic configuration for ResNet-50.
Hyper-pone paremeerr
Setting
Optimizer
AdamW
Learning rate
le-4
Weight decay
0.05
Layer weight decay
Number of layers is 16. Decay rate is 0.99
Linear warmup
1500 steps with ratio le-6
Mixed precision training
APEX
Table 11: Basic configuration for ConvNeXt (ConvNeXt-S and ConvNeet-B)
Hyper-pun parameer
Setting
Optimizer
AdamW
Learning rate
lc-4
Weight decay
0.05
Stage-wise decay
Number of stages Decay rate 0.9
Linear warmup
1500 steps with ratio 1e-6
Mixed precision training
APEX
Conv-Adapter. We utilize the official Conv-Adapter implementation from https:// github. com/ hwta///hhhhhhooaaaa Haaaub/.co//hhhhhh//Cooowa adapters to spatial convolution layers with kernel sizes greater than 1, we set its channel compression factor y 4 to maintain parameter count comparability with other methods. This configuration results in Conv-Adapter having the highest rank among all compared methods' low-rank matrices.
Adonaaaooooo-aaaeer....aaa As Conv-Adapter exclusively applies trainable
PiSSA. We employ the official PiSSA implementation from https://githu: com/ GraphPKU/PiSSA, PiSSA is applied to all 1 x 1 convolutional layers, as directly reshaping convolution kernels from shape [c. [C_out, c_in, k, k] to [C [C_out, C_in*k"22 for stan- dard LoRA application would compromise the network's inherent inductive bias. In prac- tice, we y to maintain parameter count consistency with other methods. Following established LoRA practices, we use default scaling factor configuration of a/r 2 as recommended recent literature.
HiRA. We adapt the official HiRA implementation from https://githut com/ agsiswiliam/hize for convolutional networks. Following the approach with PiSSA, we restrict HiRA application exclusively to 1 convolutional layers. To maintain parameter count consistency, we set 7 20 and utilize the theoretically derived scaling factors Sexpeet presented in Section 3.1.
Training configuration on ADE20K. For ADE20K segmentation, we employ both a primary de- coder head and auxiliary head. The primary decoder head is optimized using cross-entropy loss (weight = 1.0) and incorporates deep features from four network stages in both ResNet-50 and ConvNeXt architectures. In contrast, the auxiliary head employs only third-stage features and is trained with reduced cross-entropy loss weight of 0.4.
For both full fine-tuning and PEFT approaches, employ the standard 160k iteration schedule. Us- ing a batch size of 16, we perform evaluations at 16k-iteration intervals. addition to the introduced trainable low-rank matrices, optimize all bias terms and normalization layers. The PEFT method configurations for ADE20K segmentation remain identical to those used for VTAB- lk classification.
Training configuration on MS-COCO. We conduct object detection experiments using the Faster R-CNN framework, adopting the configuration from https://github.... com wwt.::iaawawwtiatttaaoaamaaaa orrmer wtw.e//ettiooo/ee//eeeeeeeeeeeee
Our implementation differs from Table
three key aspects: (1) a learning rate of
2 x 10-1 (2) a standard 500-iteration warmup period, and (3) batch size of 16 dis-
tributed across two NVIDIA A800 GPUs. Following the standard training schedule
luttps://githmn http://giihhutbo.ooosssstttttnnr wwwee/mweewwwiiiieeeeeeeeeee otg.a//eeeeeettteeeetemtottttttr

---

Table 12: Average top-1 accuracy the VTAB-IK Ikk NATURAL benchmark across three independent
runs.
The pighest-n PEFT method for each task is highlighted in bold.
Backbone
Method
#Param
Caltech101
CIFAR-100
DTD
Flowens102
Pets
Sun397
SVHN
Average
FT
23.5M
89,49=0.19
30.8240016
60,41
89.35-0.34
84.4320.46
31,0010.17
82.51
67.43:00.00 43:10.29
Conv-Adapter [32]
27.2240.32
82.21
61.97:+0.31
P15SA Ui
85.12-0:20
20 99
02:9570111
63.81 0:91
89-1510000
S4.7170
05:32-0000
04:11
HiRA
1.2M
87.00=0.18 27.9710.07 63,9910.30
82.39-0.26
89.20:1.14
32.28:0.14
53.3120.64
62.31
CoLoRA (ours)
62.82.10.74
88.87:10:45
32.38:10.57
66.57:10.50
FT
49.5k
91,33-0.55
65.60110.32
74.1710.05
98.74=0.10
90.36:10.22
49.02.10.19
92.30.10.11 13
80.22:10.22
ConvNeXi-5 Conv-Adapter
288
90.94-0.45
68-52+0.36
75.37+0440
99.12-0.07
91.13+0.09
49.91+0006
90.35-0.34
80.76+0.26
PISSA
3.0M
90.93=0.12
69:22:::400
75.62:0.10
99,06=0.044
91.29::011
51,9620.29
90.27:00.29
81.19:00.20
HiRA [[A]
3.0M
90.71=0:244
70.851..200 76.30.10.11
99.27-0.00
92.12:10.07
13.56.10.14
86.39.10.10 10
81.31
CoLoRRA (ours)
2.6
91.60-0.1 66.34+00.55 75.00+0.60 98.89
90.43+0.27 49,94+0.33 92.08
60.61++0.31
Table 13: Average top-1 classification accuracy on the VTAB-Ik SPECIALIZED benchmark across
three experimental trials. Top-performing PEFT methods are indicated bold.
Backhone
Method
#Param
Camelyon
EuroSAT
Resise45
Retinopathy|
Average
FT
23.5M
85.31=0.61
91.83++034
80.94+0.11
73.82+0020
02.9
Conv-Adapter [62]]
1.4M
78.59110.28
88.20.10.44 75.2911.23
73.80110.6
78.97
ResNet-50
PISSA [
1.2M
81.59=0.58 90.25=0.55
78.47:0.566
73.82=0.15
81.03=0.46
HiRA
1.2M
79.07110.10
89.10:10.33
75.42-10.11
73.85+10.02
79.36=10.07
CoL.RA (ours)
1.0M
82.60:10.58
92.16:10.20
0..0
74.30-10.19
82.54-10.31
FT
49.5M
87.95:0.45
95.9811.06
85.9810.75
76.83+0.27
86.69=0.38
Conv-Adapter [62]
2.8M
85.75+00.75
94.89+0.16
84.04+0.24
7.230+0.22
84.99+0.34
ConvNeXt-S
PiSSA [
3.0M
85.79++0.07 95.45+0.099
85.56+0.15 75.56+0.26
85.59-0.144
HiRA [22]
3.0M
84.2911.20
94.8510.07
84.9110.06 75.8510.12
84.9710.11
CoLoRA (ours)
2.6M
87.51+0.07 96.00+0.12 85.95+0.15 77.13+0.12 86.65=#0111
master/configs/ base for 12 epochs with learning rate adjustments at the 8th and 11th epochs. The PEFT method configurations for object detection remain consistent with those used for VTAB-1k classification and ADE20K segmentation.
wtw.aee/chhduullee//chhhuuit 1x.py), we train models on MS-COCO
I More Details on VTAB-1k Benchmark
We present comprehensive quantitative results on the VTAB-Ik benchmark, including the average top-1 accuracy and its standard deviation in Tables I22 to [4] All methods evaluated on V-ABB- 1k across three independent runs. Notably, the proposed CoLoRA significantly outperforms existing PEFT methods in the NATURAL, SPECIALIZED, and STRUCTURED categories with ResNet- 50 as the backbone. When using ConvNeXI-S, CoLoRA achieves the highest top-1 accuracy in the SPECIALIZED and STRUCTURED categories, compared state-of-the-art PEFT methods. Furthermore, CoLoRA surpasses full fine-tuning in the STRUCTURED category, highlighting its superior adaptation capability. Consequently, CoLoRA outperforms full fine-tuning on VTAB-Ik with Con-NeXt-S as the backbone.
J Scaling CoLoRA on ADE20K
In this section, we investigate the relationship between the number of tunable parameters and adap- tation performance. Using the ConvNeXt-B backbone, we evaluate different values for 7 and the rank ratio T. rr/(r, + rt). Note that full fine-tuning of ConvNeXt-B on ADE20K achieves 50.99 mloU. The results, presented in Table [T. demonstrate that CoLoRA with merely 19.5M tunable parameters (compared to 87.6M for full fine-tuning) not only matches but surpasses full fine-tuning performance, reaching 51.13 mloU. This remarkable performance highlights CoLoRA's potential for efficiently fine-tuning pre- mr-inraened ConvNets across diverse downstream tasks.
31

---

Table 14: Average top-1 aceuracy across three runs on the VTAB-IK STRUCTURED benchmark. The highest- PEFT method is highlighted in bold.
Backbone
Method
4Param
Clevr-Count
Clevr-Dist
DMLab
dSpr Loe
dSpr-Oni
KTTTI Dist
sNORD-Axim
sNORD Elev
Average
FT
23.5M
43.2710.58
55.2111048
45.6710.14
80.18
41,9010.56
80.59L1111
26.5110.116
48.21_1..5
52 02.70.10.67
ResNet: 50
PSSA
Con--Adgper
12M
14M
48.1100.29
35:4::005
19.1610163
1.9811.55
35 35.40.0.366
382110.46
526221.11
41 5.100.2 15:2910.95
22.29:0.32
69.9511.04
13.70
18.05:0.61
14.72.10.12
38.21:0.79
39.43-0038
37.00.10.67
40.96-0:31
46,0400.72
45.70+0.26
16.94-0.25
71331+075
13.5340.277
44.05-0.3
10. 20-0044-
CeLoRA
11.88
16.91.0.1
ComNeeXioS
Com--Adaper
PSSSA
3.M
01.51:0.54
67.04:0::00
51.35:1049
94447:030
47:00.6.55
2410.46
09.15:0005
06.88:00433
HIRA [
30M
79.04233011
6395110.39
47.33:10.48
79.0022220
56 35.9
82.32.0.77
25.85.5
42.13:0026
59.65:1110
Cal .RA (ours)
2.6M
91.71-0.18
65:57=0.08
54.99+00661
92.44-0.22
02.28-0.88
81:52-0.40
40.71+00.67
48.95-0.40
67.57+0.47
Table 15: Quantitative evaluation of Col CoLoRA scaling effects ConvNeXt-B for ADE20K adapta- tion.
7
16
8
8
4
4
2
/ (rx + nn)
0.5
0.8
0.5
0.8
0.5
0.4
0.9
0.8
#Params
4.6M
5.7M
8.8M
11.1M
17.3M
19.5M
17.7M
21.9M
mloU
49.55
50.65
50.86
50.86
50.96
51.13
50.38
51.06
32


Reviews:
[{"review_id": "ItyUzsl8p4", "reviewer": "Reviewer_wEks", "summary": "The CoLoRA framework proposed in the paper solves the high-rank learning bottleneck in ConvNets fine-tuning and significantly improves the performance of the PEFT method. The paper analyzes the gradient correlation between ConvNets layers in detail and verifies the necessity of correlation modeling through mathematical formulas and experiments.", "questions": "Question：\n1.\tThere are typos in the text. It is recommended to check and modify it.\n2.\tAlthough some limitations are mentioned in the appendix, the main text does not clearly point out the possible limitations of the method. For example:\n3.\tIs CoLoRA applicable to deeper networks (such as ConvNeXt-XL) or more complex tasks (such as video processing).\n4.\tIs the method sensitive to the initialization of low-rank matrices, and its performance under different data distributions.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths：\nThis work shows that CoLoRA outperforms existing methods in multiple mainstream visual tasks (classification, segmentation, and detection). The experimental settings are reasonable, based on public datasets and models, and highly reproducible. Using only about 5% of the trainable parameters, the performance can reach or even exceed that of full fine-tuning, demonstrating the practical application potential of the method.\n\n\nWeaknesses：\nThe paper contains typos that need correction, and although some limitations are discussed in the appendix, the main text does not clearly address potential shortcomings. Considering there are lots of works on efficient finetuning, the proposed method might be somewhat incremental. \n\nOverall, its strengths has higher weight than its weakness.", "was_revised": true, "final_justification": "Given that part of my concerns are addressed, I decide to keep the socre.", "rebuttal": "We sincerely thank the reviewer for the positive score and the recognition of the potential of our proposed method. Below, we carefully address the raised concerns.\n\n[Q1] **There are typos in the text.**\n\nWe gratefully appreciate your careful review and pointing out the typos in our manuscript. We have thoroughly proofread and corrected these issues. For example:\n\n- \"ConveNets\" on page 1 has been corrected to \"ConvNets\".\n- \"Correlated Low RAnk (CoLoRA)\" on page 2 has been revised to \"Correlated Low-Rank Adaptation (CoLoRA)\".\n\n[Q2] **Although some limitations are mentioned in the appendix, the main text does not clearly point out the possible limitations of the method. For example: 3. Is CoLoRA applicable to deeper networks (such as ConvNeXt-XL) or more complex tasks (such as video processing). 4. Is the method sensitive to the initialization of low-rank matrices, and its performance under different data distributions.**\n\n*Limitations*: We acknowledge that, while we briefly mentioned the “correlation strength issue”, the profound limitations were not clearly articulated in the main paper. The **primary limitation** of our proposed CoLoRA method lies in its **computational overhead**. Specifically, the introduction of both shared and layer-specific LoRA modules, as well as the proposed filtering technique, results in increased computational load compared to recent LoRA methods such as PiSSA and HIRA on classification task.\n\nAdditionally, we conducted new experiments using ViT backbones on FGVC datasets. While CoLoRA significantly outperforms PiSSA and HiRA in this setting, it still lags behind full fine-tuning by over 3 top-1 accuracy points, suggesting that adapting CoLoRA to ViTs requires further architectural exploration. **These limitations will be incorporated into the revised manuscript and we will move it to the main text**.\n\n*Applicability to Deeper Networks or Complex Tasks*: We have evaluated CoLoRA on deeper architectures, including **ConvNeXt-XL** and **Vision Transformers (ViT)**, to assess scalability and generalization.\n\n- For ConvNeXt-XL, we conducted semantic segmentation on ADE20K for 64k iterations. The results are:\n\n|      | Full-tuning | PiSSA | HIRA  | CoLoRA |\n| :--: | :---------: | :---: | :---: | :----: |\n| mIoU |    53.21    | 51.77 | 50.61 | 52.41  |\n\nPlease notice that CoLoRA demonstrates superior performance compared to PiSSA and HIRA, as indicated by its higher mIoU metric, reflecting better convergence. Additionally, we fine-tuned ConvNeXt-XL on the MS-COCO detection dataset using the CascadeRCNN framework. In this process, we found that CoLoRA achieves performance comparable to full tuning while utilizing only 2.34% of the tunable parameters after 110,000 iterations, resulting in an impressive mAP of 52.5.\n\n- For ViTs, we tested on CUB-2011 (200 bird species, 5994 train / 5794 test) and FGVC-Aircraft (variant split) (6667 train / 3333 test, 102 categories). We fine-tuned a ViT-B model for 50 epochs using a batch size of 32. Quantitative results are reported below:\n\n|               | Full-tuning | PiSSA | HIRA  | CoLoRA |\n| :-----------: | :---------: | :---: | :---: | :----: |\n|   CUB-2011    |    79.24    | 74.70 | 51.97 | 75.61  |\n| FGVC aircraft |    79.30    | 72.40 | 34.32 | 72.79  |\n\nWhile CoLoRA was initially designed for ConvNets, it still shows better performance than recent PEFT baselines on ViTs. However, the distance to full fine-tuning indicates that **applying CoLoRA to ViTs is still an open area for exploration** and a recognized limitation.\n\n*Sensitivity to Initialization and Data Distribution*. Our experiments show that CoLoRA is robust to various initialization schemes and shifts in data distribution. In the VTAB-1k benchmark, we found that CoLoRA achieves comparable or lower standard deviations in top-1 accuracy compared to other methods, particularly on the SPECIALIZED and STRUCTURED subsets, which are known to experience significant distribution shifts.\n\nIn summary, we evaluated CoLoRA across a diverse range of datasets, including ADE20K, MS-COCO, VTAB-1k, plate_number_1, and FGVC-Aircraft, demonstrating strong performance in various settings."}, {"review_id": "bCusTR8rOe", "reviewer": "Reviewer_fmfm", "summary": "This paper proposes a novel method to fine-tune convolutional neural network taking into account the correlation between adjacent convolutional layers. Authors decompose low-rank matrix into a sum of two low-rank matrices: one of them is shared across the pair of adjacent layers and the second one is layer-specific. Apart from that, authors introduce edge-preserving convolutional filter consisting of 4 convolutions to expand the receptive field of their fine-tuning method.", "questions": "1. Did you try any other similarity metrics (for example Frobenius scalar product or distance between Images) to support the idea of the adjacent layers’ correlation?\n2. How much does applying edge-preserving filter on every step affect fine-tuning time?\n3. Do you iteratively merge low-rank adapters for all methods in your experiments? It seems that there is also no comparison to vanilla LoRA or its version with the iterative merging. Therefore, it is difficult to identify the effect solely from weight sharing.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* Authors consecutively apply their theoretical findings one by one in their experiments showing a gradual increase in quality metrics, supporting usability of each upgrade.\n* This approach introduces an idea that we can exploit the correlation between matrices of adjacent layers and use a shared matrix in their adapters. This idea is quite interesting and could inspire future research.\n\nWeaknesses:\n* The authors argue that ConvNext-B requires a full-rank matrix update via checking the matrix rank of the update after full fine-tuning. However, this does not guarantee that full-rank updates are the only viable option. Imposing structural constraints during training may lead to a different set of parameters. A more rigorous approach would be to evaluate the dependence on rank by fine-tuning the model multiple times with varying ranks and comparing performance metrics across different parameter budgets.\n* The choice of similarity metric for weight update matrices seems unjustified. The proximity of mean singular values for adjacent layers, shown in Figure 2, shows only the relative poximity of Frobenius norms of weight updates, and may not imply real correlation between weight updates. Furthermore, the mean singular value of the weight update primarily depends on the matrix entries magnitude, which, in particular, depends on the learning rate schedule.\n* Transition from (3) and (4) to (5) requires more details and explanations.", "was_revised": true, "final_justification": "The results became more convincing after the rebuttal. Among the drawbacks, there are currently no error bars.", "rebuttal": "We sincerely thank the reviewer for the high-quality review and insightful comments. We are also delighted that the reviewer finds our proposed method interesting and potentially inspiring for future research. Below, we address the raised concerns in detail.\n\n[Q1] **The authors argue that ConvNext-B requires a full-rank matrix update via checking the matrix rank of the update after full fine-tuning. However, this does not guarantee that full-rank updates are the only viable option. Imposing structural constraints during training may lead to a different set of parameters. A more rigorous approach would be to evaluate the dependence on rank by fine-tuning the model multiple times with varying ranks and comparing performance metrics across different parameter budgets.**\n\nWe appreciate your insightful observation. We agree that inspecting the rank of update weights alone is insufficient to conclude that ConvNeXt-B inherently requires full-rank updates. To provide a more rigorous justification, we conducted experiments by tuning ConvNeXt-B using PiSSA under varying ranks. \n\nFor ConvNets, we parameterize the rank via a channel compression factor $\\gamma$, where  $r=d/\\gamma$ and $d$ is the number of channels. Notably, $\\gamma=1$ corresponds to full-rank adaptation. It's worth mentioning that LoRA introduces an over-parameterization when $\\gamma=1$ due to the decomposition $W=sBA$, resulting in more trainable parameters than the original full-rank $W$. Quantitative results are presented below:  \n\n| $\\gamma$            | 1      | 2     | 4     | 8     | 20   |\n| ------------------- | ------ | ----- | ----- | ----- | ---- |\n| tunable params      | 104.8M | 52.5M | 26.3M | 13.2M | 5.3M |\n| tunable param ratio | 120%   | 60%   | 30%   | 15%   | 6%   |\n| mIoU                | 50.1   | 50.6  | 49.9  | 49.1  | 47.6 |\n\nAs shown above, performance peaks at $\\gamma=2$, while both under- and over-parameterization degrade performance. Increasing $\\gamma$ leads to a consistent drop in mIoU from 50.6 to 47.6. This validates that **independent LoRA tuning on ConvNeXt-B requires high-rank adaptation**, and performance deteriorates under tight parameter budgets. We will include this analysis in the revised manuscript to reinforce our argument.\n\n[Q2] **The choice of similarity metric for weight update matrices seems unjustified. The proximity of mean singular values for adjacent layers, shown in Figure 2, shows only the relative poximity of Frobenius norms of weight updates, and may not imply real correlation between weight updates. Furthermore, the mean singular value of the weight update primarily depends on the matrix entries magnitude, which, in particular, depends on the learning rate schedule.** **Did you try any other similarity metrics (for example Frobenius scalar product or distance between Images) to support the idea of the adjacent layers’ correlation?**\n\nWe gratefully appreciate the insightful comment. In the paper, we discussed the correlation between weights of adjacent layers (e.g., $W_1$ and $W_2$) based on shared gradient terms. Specifically, during backpropagation, the term $X^T(\\partial L/\\partial Y_2)$ appears in both gradients $\\Delta W_1$ and $\\Delta W_2$, suggesting inherent correlation.\n\nIn the manuscript, we measured the mean singular value of these update matrices to understand their behavior. However, we agree that this measure is influenced by the magnitude of entries and the learning rate schedule, and does not directly reflect structural correlation.\n\nFollowing your suggestion, we applied the Forbenius scalar product as well as the Centered Kernel Alignment (CKA) metric to explore the correlation between adjacent layers. CKA is invariant to isotropic scaling, orthogonal transformations, and invertible linear mappings, making it a robust metric for capturing correlation between different layers. This provides a more principled analysis of inter-layer correlation, and we will include these results in the revised version.\n\n[Q3] **Transition from (3) and (4) to (5) requires more details and explanations.**\n\nWe gratefully appreciate the constructive suggestion.\n\nStarting from Eqs. (3) and (4):\n\n$$\\mathbf{g}_{W_2}=W_1^TX^T(\\partial L/\\partial Y_2) + [\\mathbf{b}_1,\\cdots, \\mathbf{b}_1]_N (\\partial L/\\partial Y_2),$$\n\n$$\\mathbf{g}_{W_1}=X^T(\\partial L/\\partial Y_2)W^T_2 + \\mathbf{0}.$$\n\nLet us denote the shared term $\\Lambda = X^T(\\partial L/\\partial Y_2)$, and the bias-related term for $\\mathbf{g}_{W_2}$ as $\\Psi$. The gradients can then be rewritten as\n\n$$\\mathbf{g}_{W_2}=W_1^T \\Lambda + \\Psi$$\n\n$$ \\mathbf{g}_{W_1} = \\Lambda W_2^T$$\n\nUnder gradient descent with learning rate $\\eta$ (which is widely employed to understand the behavior of LoRA), we have:\n\n$$\\Delta W_1=SW^T_2 + P_1,\\ \\Delta W_2=W_1^T S + P_2,$$\n\nwhere $S=-\\eta \\Lambda$, $P_2=-\\eta \\Psi$, and $P_1=\\mathbf{0}$. To integrate this behavior to LoRA, we decompose $S$, $P_1$, $P_2$ into learnable low-rank matrices, forming the basis of our proposed CoLoRA.\n\n[Q4] **How much does applying edge-preserving filter on every step affect fine-tuning time?**\n\nWe thank the reviewer for this important and practical question. Below, we provide a comprehensive comparison of GPU memory usage, per-step training time, and inference time across different tuning strategies. \n\nAt inference time, the LoRA weights of PiSSA, HIRA, and CoLoRA are merged into the base model weights. Therefore, the inference time for full-tuning, PiSSA, and HIRA remains identical. The additional inference time of CoLoRA arises solely from the edge-preserving filtering module. To minimize this overhead, we adopt efficient implementations based on `torch.jit.script`.\n\nDuring training, all LoRA-based methods (PiSSA, HIRA, and CoLoRA) introduce additional memory and computation due to the intermediate low-rank matrices. For CoLoRA, the filtering operation and the separate computation of shared and independent LoRA branches lead to further computational overhead. Moreover, **mixed-precision training using APEX introduces memory usage variations depending on bits of parameters**. We provide details below:\n\n*Computation comparison on detection*. We evaluate the time efficiency with ConvNeXt-S backbone on the MS-COCO dataset, where the batch size is 2 with `coco_instance.py` dataset configuration. \n\n|                                            | Full-tuning | PiSSA | HIRA  | CoLoRA | CoLoRA w/o filtering |\n| ------------------------------------------ | ----------- | ----- | ----- | ------ | -------------------- |\n| **Training GPU memory consummation (MiB)** | 18418       | 14648 | 14536 | 15042  | 16488                |\n| **Per step training time (s)**             | 1.747       | 1.426 | 1.501 | 1.527  | 1.509                |\n| **Inference time (s)**                     | 0.108       | 0.108 | 0.108 | 0.108  | 0.108                |\n\n*Computation comparison on classification*. We evaluate the time efficiency with ConvNeXt-S backbone on the FGVC aircraft dataset, where the batch size is 32 and the image resolution is 224x224. \n\n|                                            | Full-tuning | PiSSA  | HIRA   | CoLoRA | CoLoRA w/o filtering |\n| ------------------------------------------ | ----------- | ------ | ------ | ------ | -------------------- |\n| **Training GPU memory consummation (MiB)** | 7247        | 6277   | 6189   | 7243   | 7097                 |\n| **Per batch training time (s)**            | 0.324      | 0.339 | 0.341 | 0.414  | 0.409                |\n| **inference time (s)**                     | 0.042       | 0.042  | 0.042  | 0.053  | 0.042                |\n\nIn summary, our optimized implementation ensures that edge-preserving filtering is not a major bottleneck during training. The slight increase in training time for CoLoRA mainly results from the additional computation of shared and independent LoRA branches that need optimization. **The filtering overhead during inference is minimal for detection task, and moderate for classification**.\n\n[Q5] **Do you iteratively merge low-rank adapters for all methods in your experiments? It seems that there is also no comparison to vanilla LoRA or its version with iterative merging. Therefore, it is difficult to identify the effect solely from weight sharing.**\n\nWe appreciate this observation. In our experiments:\n\n- We do not apply LoRA merging to PiSSA or HiRA by default.\n- HiRA already performs high-rank updates and shows little benefit from merging. In fact, merging often introduces oscillations in fine-grained classification tasks.\n- PiSSA uses SVD-based initialization, making LoRA merging non-trivial. It requires repeated SVD and careful reinitialization of $A$ and $B$. Short-interval LoRA merging may introduce noise into SVD decomposition results, while long-interval LoRA merging diminishes performance.\n\nAdditionally, to ensure a fair comparison in terms of LoRA merging, we conduct experiments on two FGVC benchmark datasets, CUB-2011 and FGVC aircraft, as suggested by other reviewers. We fine-tune a pre-trained ConvNeXt-S model on these datasets with a batch size of 32 for 50 epochs. Specifically, we merge LoRA weights for PiSSA, HIRA, and CoLoRA every 500 steps, followed by a careful optimizer reset and 50 warmup steps. Quantitative results are presented below:\n\n|               | Full-tuning | PiSSA+merging | HIRA+merging | CoLoRA |\n| :-----------: | :---------: | :-----------: | :----------: | :----: |\n|   CUB-2011    |    84.12    |     84.90     |    83.47     | 85.28  |\n| FGVC aircraft |    79.24    |     74.70     |    51.97     | 75.61  |\n\nThese results confirm that **weight sharing, not merging, is the key driver** of CoLoRA’s superior performance.\n\nFinally, for object detection, we found that merging LoRA weights degrades performance, likely due to the multi-branch prediction and the sensitivity of detection heads. Therefore, we disable merging for detection tasks. We will add these results and clarifications to the revised manuscript."}, {"review_id": "d77yd48vqA", "reviewer": "Reviewer_r6JU", "summary": "The paper proposes CoLoRA, a novel Low-Rank Adaptation framework tailored for convolutional networks (ConvNets), addressing the limitations of standard LoRA in highly correlated convolutional layers. Unlike existing LoRA methods that treat layers independently, CoLoRA introduces correlated low-rank matrices to model inter-layer dependencies. To further enhance adaptation efficiency, it incorporates a parameter-free filtering mechanism that expands the receptive field and reduces noise from uninformative regions. Extensive experiments across classification, segmentation, and detection show that CoLoRA consistently outperforms prior PEFT methods. Remarkably, it achieves better performance than full fine-tuning on VTAB-1k using only 5% of trainable parameters.", "questions": "Please refer to the previously mentioned weaknesses. \n\nAdditionally, I would like to understand the degree of correlation between adjacent attention layers in Vision Transformers (ViTs). Is this correlation significantly different from that observed in convolutional networks, thereby making CoLoRA specifically tailored for convolutional architectures?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The proposed Correlated Low-Rank Adaptation (CoLoRA) is both novel and well-motivated, supported by the empirical observation of inter-layer correlations in convolutional networks, as illustrated in Figure 2.\n2. The paper provides a clear and insightful analysis of the spatial limitations of LoRA in ConvNets, and the introduced training-free edge-preserving filtering mechanism (Figure 4) offers a compelling and practical solution.\n3. CoLoRA demonstrates notable performance improvements over state-of-the-art PEFT methods on multiple vision benchmarks, including VTAB-1K, COCO, and ADE20K, with a comparable number of trainable parameters.\n4. The ablation studies presented in Table 3 are thorough and well-designed, effectively validating the contribution of each component within the proposed framework.\n\n\nWeaknesses:\n1. Unlike standard LoRA, CoLoRA applies edge-preserving filtering to the compressed features generated by low-rank projections, which appears to preclude re-parameterization into static weights. As a result, inference-time efficiency may be impacted due to the additional computation. It would be important to quantify this overhead by reporting inference time and computational cost comparisons between CoLoRA and standard LoRA across both classification and detection tasks.\n2. While CoLoRA is evaluated on VTAB-1K, COCO, and ADE20K, many recent visual PEFT methods also benchmark on fine-grained visual classification (FGVC) datasets. A comparison with state-of-the-art methods in this domain would strengthen the generality and competitiveness of the proposed approach.", "was_revised": true, "final_justification": "The authors have addressed most of my concerns through their detailed rebuttal, and I decide to keep my rating as \"Borderline Accept\". Overall, I think this work is above the bar of the acceptance of NeurIPS.", "rebuttal": "We thank the reviewer for the constructive comments and the positive score, especially the recognition of our method as both novel and well-motivated. We address the raised concern in detail below.\n\n[Q1] **Unlike standard LoRA, CoLoRA applies edge-preserving filtering to the compressed features generated by low-rank projections, which appears to preclude re-parameterization into static weights. As a result, inference-time efficiency may be impacted due to the additional computation. It would be important to quantify this overhead by reporting inference time and computational cost comparisons between CoLoRA and standard LoRA across both classification and detection tasks.**\n\nWe acknowledge the reviewer’s concern regarding the inference-time efficiency of CoLoRA. In our design, the LoRA weights can be merged into the pre-trained weights, similar to PiSSA and HIRA. Therefore, **the additional computational cost arises solely from the edge-preserving filtering**.\n\nTo address this overhead, we have implemented several optimizations. These include the use of `torch.jit.script`to accelerate the filtering process. We now provide a detailed comparison of inference time across three representative tasks: classification, detection, and segmentation. All experiments use the ConvNeXt-S backbone.\n\n*Classification task*. We evaluate the inference time with ConvNeXt-S backbone on FGVC aircraft dataset, where image resolution is 224x224. \n\n| Method                                | Inference Time (s) |\n| ------------------------------------- | ------------------ |\n| PiSSA, HIRA, and CoLoRA w/o Filtering | 0.042              |\n| CoLoRA                                | 0.053              |\n\n*Detection task*. We evaluate the inference time under the Faster-RCNN framework with ConvNeXt-S backbone. The results are tabulated below:\n\n| Method                                | Inference Time (s) |\n| ------------------------------------- | ------------------ |\n| PiSSA, HIRA, and CoLoRA w/o Filtering | 0.108              |\n| CoLoRA                                | 0.108              |\n\n*Segmentation task*. We evaluate the inference time with ConvNeXt-S backbone on ADE20K dataset. The results are shown below:\n\n| Method                                | Inference Time (s) |\n| ------------------------------------- | ------------------ |\n| PiSSA, HIRA, and CoLoRA w/o Filtering | 0.250              |\n| CoLoRA                                | 0.260              |\n\n**In summary, for classification tasks, where most of the computational cost is concentrated in the backbone, the filtering module introduces a moderate increase in inference time. In contrast, for detection and segmentation tasks, the additional overhead caused by filtering is comparatively negligible.**\n\n[Q2] **While CoLoRA is evaluated on VTAB-1K, COCO, and ADE20K, many recent visual PEFT methods also benchmark on fine-grained visual classification (FGVC) datasets. A comparison with state-of-the-art methods in this domain would strengthen the generality and competitiveness of the proposed approach.**\n\nWe appreciate this valuable suggestion. To further assess the generality of our method, we conduct experiments on two widely-used FGVC datasets: CUB-2011 and FGVC Aircraft. The CUB-2011 dataset contains 200 bird species, comprising 5,994 training and 5,794 testing images. For the FGVC Aircraft dataset, we adopt the 'variant' split, which includes 6,667 training and 3,333 testing images spanning 102 fine-grained categories.\n\nWe fine-tune a pre-trained ConvNeXt-S backbone on both datasets for 50 epochs using a batch size of 32. The top-1 accuracy results are summarized below:\n\n|               | Full-tuning | PiSSA+merging | HIRA+merging | CoLoRA |\n| :-----------: | :---------: | :-----------: | :----------: | :----: |\n|   CUB-2011    |    84.12    |     84.90     |    83.47     | 85.28  |\n| FGVC aircraft |    83.65    |     79.42     |    53.20     | 80.71  |\n\nCoLoRA consistently outperforms both PiSSA and HIRA across these FGVC benchmarks. On CUB-2011, it achieves a 1.16% improvement over full fine-tuning and a 0.38% gain over PiSSA. On the FGVC Aircraft dataset, CoLoRA surpasses PiSSA by 1.29%. These results further confirm the effectiveness and competitiveness of our method in fine-grained classification tasks. We will include these findings in the revised manuscript to strengthen the generality of our contributions.\n\n[Q3] **Additionally, I would like to understand the degree of correlation between adjacent attention layers in Vision Transformers (ViTs). Is this correlation significantly different from that observed in convolutional networks, thereby making CoLoRA specifically tailored for convolutional architectures?**\n\nWe thank the reviewer for this insightful question. In our current work, CoLoRA is primarily designed for convolutional networks, where the strong correlation between adjacent convolution layers motivates our design. However, exploring the applicability of CoLoRA to Vision Transformers (ViTs) is indeed a valuable direction.\n\nTo investigate this, we fine-tune a pre-trained ViT-B model on two FGVC datasets and analyze the degree of correlation between adjacent modules. In particular, we examine: (1) the correlation between the QKV projection and the output projection in the self-attention block, and (2) the correlation between the two linear layers in the feed-forward network (FFN). We use the **Centered Kernel Alignment (CKA)** metric to measure correlation, as it is invariant to orthogonal transformation and well-suited for comparing representational similarity in ViTs. Our findings suggest that the **correlation between adjacent layers in ViTs is substantially weaker than that observed in ConvNets**. Despite this, we evaluate CoLoRA on ViTs by fine-tuning ViT-B on CUB-2011 and FGVC Aircraft datasets. The results are shown below:\n\n|               | Full-tuning | PiSSA | HIRA  | CoLoRA |\n| :-----------: | :---------: | :---: | :---: | :----: |\n|   CUB-2011    |    79.24    | 74.70 | 51.97 | 75.61  |\n| FGVC aircraft |    79.30    | 72.40 | 34.32 | 72.79  |\n\nAlthough the inter-layer correlation in ViTs is weaker, CoLoRA still outperforms existing PEFT methods such as PiSSA and HIRA on both datasets. However, its performance does not match that of full fine-tuning, suggesting that **the adaptation of CoLoRA to ViTs requires further investigation and architectural refinement**. We consider this an important limitation and will clarify it in the revised manuscript."}, {"review_id": "5bdtNWGsHC", "reviewer": "Reviewer_UVug", "summary": "This paper proposes a correlation low-rank adaptation module for convolutional networks, which is an efficient PEFT strategy. The effectiveness of the approach is validated in classification, detection, and segmentation tasks.", "questions": "I hope the authors will address the issues with the weaknesses, and I will consider revising my score.", "limitations": "1. The only shortcoming of this paper is that its performance is slightly lower than that of SFT, which might be a common issue.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. This paper finds that fine-tuning pre-trained ConvNets on downstream tasks is a full-rank learning process, and existing LoRA methods are ineffective within the ConvNets framework.\n2. This paper proposes a novel weight update strategy to update adjacent convolution layers together.\n3. This paper compares its approach with other PEFT methods on tasks such as object detection, classification, and segmentation, and the results show superior performance.\n\nWeaknesses:\nThe approach only outperforms SFT in classification tasks, while in detection and segmentation tasks, the performance of CoLoRA is not as good as that of SFT. Without considering the resource consumption of the model, the contribution of CoLoRA would be discounted.", "was_revised": true, "final_justification": "I have carefully read the authors’ responses to my and the other reviewers’ comments, and I have decided to keep my rating at ‘borderline accept.’ Overall, I believe this work falls outside the acceptance criteria for NeurIPS.", "rebuttal": "We appreciate your positive rating and recognition of the novelty of our method. Below, we respond to your concern regarding the comparative performance of CoLoRA and SFT.\n\n[Q1] **The approach only outperforms SFT in classification tasks, while in detection and segmentation tasks, the performance of CoLoRA is not as good as that of SFT. Without considering the resource consumption of the model, the contribution of CoLoRA would be discounted.** **The only shortcoming of this paper is that its performance is slightly lower than that of SFT, which might be a common issue.**\n\nWe previously demonstrated in Table 1 that the proposed CoLoRA method outperforms SFT in classification tasks. In this response, we provide further evidence that CoLoRA either surpasses or matches the performance of SFT in classification, segmentation, and detection tasks.\n\n**Classification on VTAB-1k:** As shown in Table 1, CoLoRA using ConvNeXt-S surpasses SFT in terms of mean top-1 accuracy across the 19 VTAB-1k tasks. This illustrates its effectiveness in transfer learning across diverse visual domains.\n\n**Classification on fine-grained CUB-2011:** As suggested also by other reviewers, we evaluated CoLoRA on CUB-2011, a challenging fine-grained benchmark that includes 200 bird categories. We fine-tuned a pre-trained ConvNeXt-S for 50 epochs with a batch size of 32. The results were as follows: CoLoRA achieved 85.28% top-1 accuracy, while SFT (full-tuning) attained 84.12% top-1 accuracy. This represents a gain of 1.16% over SFT, highlighting CoLoRA's effectiveness in fine-grained tasks.\n\n**Segmentation on ADE20K:** Dense prediction tasks, such as segmentation, present significant challenges for PEFT methods. For instance, the official Conv-Adapter, which has 39.4M tunable parameters on ConvNeXt-B, still falls short by 1.8 mIoU compared to SFT. In contrast, **CoLoRA can match or exceed SFT's performance while utilizing significantly fewer parameters in both configurations:**\n\n- Config 1: with a channel compression factor of $\\gamma=4$ and a shared LoRA ratio of 0.4, this configuration results in 19.5M tunable parameters and achieves 51.13 mIoU, slightly outperforming SFT by 0.14 mIoU.\n- Config 2: we set $\\gamma=2$ and the shared LoRA ratio to 0.8, which yields 21.9M tunable parameters. This configuration offers a 51.06 mIoU, marginally surpassing the performance of SFT.\n\nThese results demonstrate that CoLoRA can **match or exceed full-tuning** performance with **50% fewer trainable parameters**.\n\n*Detection on MS-COCO*. Tuning ConvNets to outperform SFT in detection tasks poses a significant challenge. For example, the Conv-Adapter introduces 24.6M parameters on the ConvNeXt-B model but still falls short of SFT by 3.3 mAP points. To investigate the potential of CoLoRA, we fine-tuned a ConvNeXt-XL model within the Cascade R-CNN framework, utilizing only 8.2M tunable parameters. Remarkably, both CoLoRA and SFT achieved identical performance levels of 52.5 mAP. Despite employing significantly fewer parameters, **CoLoRA matches the performance of full fine-tuning**, demonstrating its effectiveness in complex detection pipelines.\n\nIn summary, our extensive experiments across classification (VTAB-1k, CUB-2011), segmentation (ADE20K), and detection (MS-COCO) demonstrate that **CoLoRA can match or exceed SFT** in various tasks, highlighting CoLoRA's practical applicability, scalability, and efficiency for a wide range of vision tasks."}]

Author rebuttal:

Meta-review: This paper proposes CoLoRA, a novel parameter-efficient fine-tuning method designed for convolutional networks that explicitly models inter-layer correlations through shared and layer-specific low-rank matrices, complemented by a parameter-free edge-preserving filtering module. The method demonstrates strong empirical performance, outperforming state-of-the-art PEFT baselines across classification, segmentation, and detection tasks, and even surpassing full fine-tuning on VTAB-1k and fine-grained benchmarks while requiring only 5% of trainable parameters. Reviewers acknowledged the novelty of correlation modeling for ConvNets, the thorough experimental validation, and the potential of this work to inspire future research in efficient adaptation for non-transformer architectures. Concerns were raised about being incremental novelty, computational overhead compared to other LoRA variants, and performance gaps when extending to Vision Transformers, although these were largely addressed through additional experiments and clarifications in the rebuttal. Three reviewers indicated positive assessments. Overall, the strengths in novelty, reproducibility, and breadth of validation outweigh the weaknesses, and as such I recommend the paper for acceptance.
Decision: Accept (poster)