Canonical Rank Adaptation: An Efficient Fine-Tuning Strategy for Vision Transformers
Authors: Lokesh Veeramacheneni
Moritz Wolter
Hilde Kuehne
Juergen Gall
Venue: ICML 2025 poster (2025)
Keywords: CaRA
Canonical Polyadic Decomposition
CPD
Tensor methods
ViT
LoRA
Primary area: deep_learning->other_representation_learning
TL;DR: This paper leverages the Canonical Polyadic Decomposition to define a parameter efficient fine tuning process for vision transformers.

Abstract: Modern methods for fine-tuning a Vision Transformer (ViT) like Low-Rank Adaptation (LoRA) and its variants demonstrate impressive performance. However, these methods ignore the high-dimensional nature of Multi-Head Attention (MHA) weight tensors. To address this limitation, we propose Canonical Rank Adaptation (CaRA). CaRA leverages tensor mathematics, first by tensorising the transformer into two different tensors; one for projection layers in MHA and the other for feed-forward layers. Second, the tensorised formulation is fine-tuned using the low-rank adaptation in Canonical-Polyadic Decomposition (CPD) form. Employing CaRA efficiently minimizes the number of trainable parameters. Experimentally, CaRA outperforms existing Parameter-Efficient Fine-Tuning (PEFT) methods in visual classification  benchmarks such as Visual Task Adaptation Benchmark (VTAB)-1k and Fine-Grained Visual Categorization (FGVC).

Canonical Rank Adaptation: An Efficient Fine-Tuning Strategy for Vision Transformers
Lokesh Veeramacheneni
Moritz Wolter'
Hilde Kuehne 23
Juergen Gall
Abstract
CaRA (ours)
SPT-LORA
RepAdapter
Modern methods for fine-tuning Vision Trans- formers, such as Low-Rank Adaptation (LoRA) and its variants, demonstrate impressive perfor- mance. However, these methods ignore the high-dimensional nature of Multi-Head Attention (MHA) weight tensors. To address this limitation, we propose Canonical Rank Adaptation (CaRA). CaRA leverages tensor mathematics, first by ten- sorising the transformer into two different ten- sors: one for projection layers in MHA and the other for feed-forward layers. Second, the ten- sorised formulation is fine-tuned using the low- rank adaptation in the Canonical-Polyaal composition (CPD) form. Employing CaRA effi- ciently minimises the number of trainable parame- ters. Experimentally, CaRA outperforms existing Parameter-Efficien Fine-Tuning (PEFT) methods in visual classification benchmarks such as the Vi- sual Task Adaptation Benchmark (VTAB)-IK and the Tine-Grained Visual Categorization (FGVC) benchmark.
76
SSF
FocT-TK
NOAH
FacT-TT
Mean Accuracy
75
AdaptFormer
LoRA
74
73
Non-Tensor based
Ver-Deep
72
Tensor based
De-
10
10
1011
10 0.8 10
0.6
10
0.4 10 0.2
#Parameters (M)
Figure I. Mcan accuracy on the VTAB-Ik benchmark versus the number of trainable parameters (log-scaled) for fine-tuning VIT- B/16 pretrained on ImageNet-2Ik. CaRA demonstrates per- formance boost while utilizing only small fraction of trainable parameters.
parameters in the pre-trained network.
A prominent example of PEFT is Low-Rank Adaptation (LoRA) (Hu et al., 2022), which additively fine-tunes the low-rank intrinsic dimension of the pre-trained transformer. It represents the three-dimensional MHA projection layer in two-dimensional space Rddmede xhd, as illustrated in Figure 2. Such fine-tuning in the reduced dimensional space severely limits LoRA's 's representation ability since, by def- inition, two-dimensional representation cannot capture correlations beyond the second axis. While it facilitates simple design and incurs no additional inference costs, a performance gap exists compared to FT (Shuttleworth et al., 2024; Liu al., 2024).
1. Introduction
While Vision Transformer (ViT) architectures demonstrate remarkable performance on wide set of tasks, from classi- fication (Dosovitskiy et al., 2021) to semantic segmentation (Xic et al., 2021), they are data-hungry (Raghu et al., 2021) and pre-trained VITs are required to employ them gen- eral downstream tasks with limited data. This is typically done by full fine-tuning (FT), which is inefficient as net- works increase in scale. To circumvent this issue, Parameter- Efficient Tinnin (PEFT) methods, first introduced by Houlsby et al. (2019), fine-tune only a subset of additional
Universsit of Bonn 'Tuebingen AI Center 'MITIIBM Wat- AI Lab "Lamarr Institute for Machine Learning and Artifi- cial Intelligence. Correspondence to: Lokesh Veeramacheneni <lokki @uni- boni-bonn.de>..
Tensor calculus (Synge & Schild, 1978; De Lathauwer et al., 2000; Kolda & Bader, 2009; Oseledets, 2011) of- fers powerful alternative to defining higher-dimensional rank updates, and recent works in tensor-based low-rank fine-tuning methods (Jie & Deng, 2023; Bershatsky et al., 2024; Yang et al., 2024) have demonstrated significant im- provements over LoRA, with fewer trainable parameters.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Canonical Rank Adaptation: An Efficient Fine-Tuning Strategy for Vision Transformers
classification datasets from two benchmarks. Firstly, we evaluate our approach CaRA on the VTAB-1k benchmark and observe that our approach achieves superior perfor- mance with only a small fraction of trainable parameters. Figure 1 demonstrates the performance-to-paramee of PEFT methods; it highlights CaRA's efficiency. Sec- ondly, we extend the evaluations to five FGVC benchmark datasets. CaRA achieves competitive performance on these datasets. Thirdly, we analyse the performance of CaRA large-scale VIT architecture various image classifi- cation datasets. Finally, we investigate CaRA's 's behaviour through comprehensive ablations. The first ablation focuses on exploring the effect of rank on CaRA. Notably, we found that CaRA maintains robustness towards rank changes with stable parameter growth. In the second ablation, we study the effect of various tensorisation formulations for Wmha and t Subsequently, through investigating series of saliency maps, we highlight the learning patterns of CaRA despite its limited parameters. The source code is available at https: / / //gggthubbb. com/ Baan//BnnBBttee. BaaannyyteeCaRA.. CaRAA.
B 2 gr
A! gaarr
A 6 Rehasi
W& É Relans XinA
W
ratio
LoRA
CaRA (ours)
uuinabble
pectrained
Figure 2. Schematic illustration of fine-tuning only single query projection layer in one transformer block using LoRA on the left and CaRA (ours) on the right. Contrary to LoRA, we represent the low-rank update in CPD form.
For instance, Factor Tuning (FacT) (Jie & Deng, 2023) first tensorise the VIT blocks by stacking all the projection lay- ers into this format R 12Ixdmudee where 12 represents the number of projection layers in the transformer blocks (WQ, WK the number of blocks in the ViT. Later, this work fine-tunes the network using the Tensor- Train and Tucker decomposi- tion format for the low-rank update. Though effective, this formulation still does not account for the high-dimensional nature of projection layers in MHA, as it does not explicitly capture the low-rank update along the head dimension. To this end, we propose a novel method, CaRA, which strate- gically splits the large tensor into two sub-tensors: one for high-dimensional projection layers in MHA (Wmhaa and the other for ford-forward layers (W"
In summary, this paper makes the following contributions:
K WV, W Wup, and Wrlwwn) and denotes
We propose Canonical Rank Adaptation (CaRA), a novel and efficient PEFT method for fine-tuning a Vision Transformer (ViT) that exploits the high- dimensional nature of Multi- Attention (MHA).
2. CaRA proposes novel way of tensorising the trans- former into two individual tensors. One is for MHA and the other is for the feed-forward layer. This ten- sorisation followed by novel low-rank update using the CPD form fine-tune the VIT.
3. CaRA surpasses the performance of various tensor- based methods and LoRA image classification benchmarks with a significantly smaller fraction of parameters. Through a series of ablations, we demon- strate the robustness of CaRA to varying ranks.
The tensor Wmhn is prepared by stacking all W W and WV across [ transformer blocks, re- sulting in four-dimensional tensor R xxxd. Similarly, thth is created by stacking all other feed- forward layers, particularly WO Wup, and Wdoww, into a three-dimensional tensor 129ixdmasx xdmnttt This process of unique tensorisation is graphically represented in Figure 3. The proposed split lets CaRA efficiently represent the low-rank update of MHA layers to their maximum capacity; particularly CaRA allows capturing relations across heads. In addition to this tensorisation, we propose using the CPD format for the low-rank update, as depicted in Figure 2. This choice is motivated by the efficiency and ease of fine-tuning (Lebedev et al., 2015; Phan et al., 2020) provided by the CPD format. While tensor decompositions suffer from the curse of dimensionality (Bershatsky et al., 2024), we observe that CPD is less prone to this behaviour than Tensor-Tral advantages make CPD well-suited for the high-dimensional structure of the proposed tensor.
2. Related Work
2.1. Parameter-Efficien Fine-Tuning (PEFT)
Traditional finetuning involves training the complete ViT network (Dosovitskiy et al., 2021). Recently, PEFT methods have emerged as an attractive approach for fine-tuning large transformer models (Hu et al., 2022: Jia et al., 2022). PEFT provides a significant advantage as it requires only training parts of the network. We broadly arrange the visual PEFT methods into three categories. Firstly, the adapter-based methods introduce additional parameters to the network and fine-tune them. Early works such as Adapter (Houlsby et al., 2019) and Visual Prompt Tuning (VPT) (Jia et al., 2022) established the foundation. Another approach, SSF (Lian et al., 2022), fine-tunes a transformer by learning to scale and shift the individual transformer block features. Recent
and Tucker decompositions. These
We conduct extensive experimentation across diverse vision
2

---

Canonical Rank Adaptation: An Efficient Fine-Tuning Strategy for Vision Transformers
approaches such as (Chen al., 2022: Luo et al., 2023) com- bine multiple PEFT methods to demonstrate performance improvements. While effective, adapter-based methods add additional inference costs.
3.1. Preliminaries
3.1.1. CITONICAL-POLYADIE DECOMPOSITION (CPD)
Canonical-Polyaan Decomposition (CPD) was first pro- posed by Hitchcock (1927; 1928) as an idea of decomposing tensor as finite sum of rank-one tensors. Historically, it was rediscovered with names such as CANDECOMP- PARAFAC, PARAFAC, CAND or Polyadic tensor form (Kolda & Bader, 2009). Following the notation from (Kolda & Bader, 2009), the CPD of a fourth-order tensor T E Ix is mathematically expressed as an outer product of rank-1 vectors
The second style uses Neural Architecture Search to opti- mise the low-rank update. NOAH (Zhang et al., 2024) is an example of performing search over Adapter, LoRA and VPT for each Transformer block. The final type of PEFT methods is based on LoRA. It was first proposed by Hu et al. (2022) for fine-tuning Large Langauge Models (LLMs), and recently adopted for the vision domain (Zhang et al., 2024; Jie & Deng. 2023). LoRA indirectly optimises low-rank decomposition matrices of dense layers, providing a computational advantage during inference as the low-rank weights are merged into the pre-trained weights. Many vari- ants of LoRA such as Vector-based Random Matrix Adapta- tion (VeRA) (Kopiczko et al., 2024), NoLA (Koohpayegani et al., 2024), PiSSA (Meng et al., 2024), and Weight De- composed Low-Rank Adaptation (DoRA) (Liu et al., 2024) are further proposed to reduce the performance gap to FT. SPT-LORA (He et al., 2023) improves LoRA by identify- ing sensitive parameters in LoRA's update. A matrix-based representation for the low-rank updates, such as LoRA and its variants, has demonstrated significant potential in fine- tuning but is limited in expressive power by design. While tensor-based representations can capture multi-dimensional correlation, they remain under-explored mainly in the con- text of PEFT methods. Bershatsky et al. (2024) and Yang et al. (2024) proposed the use of tensor representations for fine-tuning LLMs,, and FacT (Jie & Deng, 2023) uses tensor- based methods for vision classification tasks.
T n (A 1 ((3) ,5(0)
(I)
where denotes the outer product, R is positive integer defining the tensor rank, A E R S() E R J SS3 E RRKER 5(1) E RRLRR are the CP-Factor ma- trices with ff ER (2) E RR 8.3) E RRx1, and sf( E RLxl being the corresponding yath rank one vector in the respective factor matrices. We use this formulation in our proposed approach to represent the low-rank update without normalisation, as this would let us freely scale the X from the factors. CPD represents a tensor T of IJKL elements with only R + R(I + J + L) elements. This low parameter count one of the reasons that CPD is a suitable candidate for the PEFT method.
3.1.2. LOW-RANK ADAPTATION (LoRA)
LoRA (Hu et al., 2022) proposed an efficient fine-tuning of large networks. Contrary to traditional fine-tuning, LoRA updates the weight additively in the low intrinsic dimension. Consider a linear layer with weight W E Baxm the LoRA weight update is then expressed as
The above-mentioned previous works are focused on Tucker and Vensor-Train representations of low-rank updates. An alternative method to represent a tensor low-rank update is the Annonical-Polyadie Decomposition (CPD) (Hitch- cock, 1927: 1928). CPD has been demonstrated to be a stable (Phan et al., 2020) and effective method for train- ing convolutional neural networks (Lebedev et al., 2015; Veeramacheneni et al., 2022). To the best of our knowl- edge, CPD has not been explored as a PEFT method for fine-tuning a VIT. This gap. combined with the limita- tions of matrix-basee representations for fine-tuning high- dimensional MHA layers, motivates us to investigate an effective and novel low-rank update mechanism using CPD.
Wr + +WW
Wx + oBAT,, x,
(2)
where W is a frozen weight matrix during training, B E Rux& and A E Rmx are trainable low-rank matrices, and the scaling hyper-parameter. Hu et al. (2022) ini- tialised matrix A with random Gaussian initialisation and matrix B to zeros so that the weight update oW in Equa- tion 2 is zero at the beginning of the training. This formula- tion is graphically represented in Figure 2. Typically, rank R < min (n, m), and this low-rank representation requires only training (n++mm) R parameters as opposed to nm param- eters in the case of full fine-tuning (FT). Interestingly, oW is added to W during inference, promoting no additional latency and inference costs compared to FT.
3. Methodology
In this section, we start with a brief introduction to Canonical-Polyadic Decomposition (CPD) (Kolda & Bader, 2009) and LoRA (Hu et al., 2022). Then, we introduce CaRA, particularly the process of tensorisation followed by the low-rank update format. Additionally, we derive gradients for our CaRA formulation.
3

---

Canonical Rank Adaptation: An Efficient Fine-Tuning Strategy for Vision Transformers
Classificr
Block-2
Writow
Block--
Concat scrves bhbcks
FEN Wrinon
W
W
W
91
W
W
Loger Norm
w
One transformer block
w
IR 91 stxcoll Modde
W
W
W
W
Comrat acrces
SDPA
We
W
W
W
W
w
w?
Nurm
W E B khxda
W
R xchode hocdla
Isput
Emaeed
Figure 3. Illustration of CaRA's tensorisation process. The figure on the left illustrates the vision transformer with / blocks next to cach other. To make a four-dimensional tensor (bottom), we stack three-dimensional query, key and value projection tensors across blocks. Similarly, we stack the remaining feed-forward layers to create another tensor (top).
3.2. Canonical Rank Adaptation (CaRA)
matrices, i.e., WQ Redaad Xdy WK dh and WV E Riduss X de for individual head in a layer result- ing in
While LoRA exhibits significant advantages compared to fine-tuning, previous work (Bershatsky et al., 2024; Jie & Deng, 2023) demonstrated that tensor methods for low- rank representation are highly efficient. Given the high- dimensional nature of Multi-Head Attention (MHA), it is evident that utilising tensor representations, especially the Canonical Polyadicc Decomposition (CPD) form, for low- rank updates provides compact and expressive approach, all while offering smaller parameter count. This section, presents our novel
= [ W,K W.] 3x X
where da represents the individual head dimension. In the case of ViT, d, = dg = dg and the enclosing square brack- ets denote the stacking operation. Furthermore, we stack the resulting tensor E, Equation across all h heads in specific transformer block j
approach, followed by our CaRA representation of low-rank updates.
Lj = EE, E2, E, Ex-1, En
B ddmata xhxds
(4)
Finally, collect all the corresponding L, from the I blocks of the transformer and stack them to result in five- dimensional tensor
3.2.1. TENSORISATION
We propose novel formulation of tensorising the VIT. It involves creating tensors: one for MHA's projection layers and second for the feed-forward layers. This ten- sorisation will allow us to represent the low-rank update for MHA's projection layers at higher dimensions. Con- trary to the existing works, this formulation will particu- larly allow us to capture any relations across heads in the transformer blocks while being parameter efficient. For the MHA, we stack the query, key and value projection
Wmha LL,, L, L. Li] 1 3xtxdatt xhxds
(5)
Empirically, we observe that combined representation of the first dimensions, and in Equation 5 performs better compared the five-dimensional representation (see Section 5.2). Following this result, represent the tensor Wilha as a four- -in-nimnnsional tensor F. X Xhxd,

---

Canonical Rank Adaptation: An Efficient Fine-Tuning Strategy for Vision Transformers
where ( represent the of CPD factors in matrix form, A (1) E R A Rd A(3) R and
Table 1. Comparison of low-rank updates for various methods such as FacT- FacT-TK, LoRA and CaRA (ours).
A E [ddxR are the four-factor matrices, X E R is the learned scaling factor, (m) represent the rdhh rank column vector in n factor, and o represents the outer product. The fine-tuned weight update in Equation 8, not to be confused with the gradient descent update, is expressed as
Method
Low-rank update
LoRA (Hu et al., 2022)
BAT
FacT-TK (Jie & Deng, 2023)
Ga Po. P3,VvV
4=1t 1 1==1
FacT-TT (Jie & Deng, 2023)
Σ Σ
S S Uv, Vi2
(1-1am1
R
CaRA (ours)
Em X ar
aar or (3) (.
Wmlh +adW
-Wmha
A
(1) ar 0 a
(2) (3) (4)
For the second tensor, we consider the linear layers W E Rdmas dhank and layers from the position-wise ferd-forward network, i.e., Wup E RAmss du and Wrdww E I x dunde Typically, we observe du = 4dmodel in the case of ViT, and subsequently, Wiy and Wdown are represented as three- dimensional tensors with shape IR 1R4Xddaat X Xdmatn, Analogous to the previous tensor. we create a tensor per transformer block j by stacking W0, WuP, and Wdown, resulting in
Similarly, for the stree-dimensione tensor w we get only three factors in CPD format
sw
{A B B(2) BB
R Σ
x AB( o bl2)) o 0b(3)
(10)
F, [WO, W"P, Wdown E R 1x Xdhand
(6)
where B()) E R R, ((2 E R XR B(3 E R Aaoss x
are the factor matrices, A E I² is the learned scaling factor. Finally, we write the fine-tuned weight update for wff
where Wo E I 1xx 1iddoox X is the tensorised version of linear layer O by adding an additional dimension. Finally, we concatenate F, from Equation across blocks of the transformer, resulting in
T + aoW W +
B o b. b
T = |F1, F2, ... F,..,,,,,,,............. FI-1, E R 91 chanee X dlnstel
(7)
This final step completes the tensorisation of the transformer architecture, and this process is graphically presented in Fig- ure 3. Unlike prior work, where all the blocks are aggregated into one single tensor, we decouple the MHA and feed- forward networks into separate tensors. This split particu- larly enables CaRA to represent MHA as a four-dimensional tensor, allowing a richer representation compared to tradi- tional dicenssiinio representations (Bershatsky et al., 2024; Jie & Deng. 2023). Additionally, this allows us rep- resent the low-rank updates along the head basis separately, as depicted in Figure 2. Empirically, we demonstrate in Section 5.2 that this split in an MHA tensor and FFN tensor increases the accuracy compared to a single tensorisation.
Figure 2 graphically illustrates our weight update for one of the query projections in a specific transformer block. Table I presents comparative summary of various tensor-based low-rank updates.
3.3. CaRA Gradients
We fine-tune the proposed formulation using PyTorch auto- grad (Paszke et al., 2017) and the gradient descent technique. This section provides the gradient for our low-rank update concerning individual rank vectors in 5Wmia and 8W as in Equations 9 and 11, respectively. Consider a loss function L; the gradients for the MHA update are defined as
V V L a
(2)
(12)
3.2.2. CARA LOW-RANK REPRESENTATION
V ( L = Al
(13)
With the tensorised network in place, we propose a novel low-rank update representation leveraging the CPD format for fine-tuning the VIT. Using the CPD format for low-rank updates, as depicted in Figure 1, notably offers benefits in terms of parameter efficiency. To represent a low-rank update for the tensor mmhha we define the update in CPD format to be oWmha as
V (2) L = I A2
(14)
V ( L = I A3
(15)
(4) I A
(16)
where AA1 E 333x A E R.dmsta dontal
1 E R
and A4 E Bd xd, are identity matrices. Similarly, gradi-
mha (A4;AU)
A
A()) (4))
Σ
X4
0
a
(8)
5

---

Canonical Rank Adaptation: An Efficient Fine-Tuning Strategy for Vision Transformers
Table VTAB-1k evaluation results with VTT-B/16 backbone on wide range of datasets. Our method results highlighted in grey. The number of parameters is averaged over groupwwise mean values. We indicate both group-wise and overall mean accuracy. We present the best result in bold and the second best as underlined. Standard deviation per dataset over 10 runs presented in Table of the Appendix.
Natural
Specialized
Siructured
arpannn (M)
Cifar 100
Caltech
Hlowr 102
SWHN
Sun399
Camelyon
Resise45
Retinopatty
Cleve
Clev--Ditt Dist
DMLab
KITT-DiS
dSpe-Loe
d
ANORE NORB-AZim Azim
<NORB- Ele
Group Mean
Overall Mean
DDD
Traditional Fine- Tuning
Lincar
63.4
85.0
63.2
97.0
86.3
36.6
$1.0
78.5
87.5
68.6
74.0
34,3
30.6
33.2
55.4
12.5
20.0
9.6
19.2
$7.64
52.94
87.7
64.3
97.2
87.4
38.8
79.7
84.2
73.9
58.6
41.7
65.5
25.7
29.1
68.96
PEPT methods
Adapter bazed
Adapter-256
0.27
74.1
86.1
63.2
97.7
87.0
34.6
50.8
76.3
88.0
73.1
70.5
45.7
37.4
31.2
53.2
30.3
25.4
13.8
22.1
59:95
55.82
VPT-Stallow
0.06
72.7
86.9
62.6
97.5
87.3
74.5
51.2
78.2
92.0
75.6
72.9
50.5
58.6
40.5
67.1
68.7
36.1
20.2
34.1
67.82
64.85
VPT-Deepp
0.53
78.8
90.8
65.8
98.0
58.3
78.1
49.6
81.8
96.1
83.4
68.4
68.5
60.0
46.5
72.8
73.6
47.9
32.9
37,8
71.96
69.43
AdaptFarmer
0.16
70.8
91.2
70.5
99.1
90.9
86.6
54.8
83.0
95.8
84.4
76.3
81.9
64.3
49.3
80.3
76.3
45.7
31.7
41.1
74.75
72.32
SSF
0.34
75.1
99.4
90.2
52.9
87.4
87.4
75.5
53.3
77.3
54,9
29.5
37.9
7/.69
73.10
RepAdapter
0.22
72.4
91.6
71.0
99.2
91.4
90.7
55.1
85.3
95.9
84.6
75.9
82.3
68.0
50.4
79.9
80.4
49.2
38.6
41.0
76.09
73.84
NAS based
NOAH
0.361
69.6
92.7
70.2
99.1
90.4
86.1
53.7
84.4
95.4
83.9
75.8
82,8
68.9
49.9
81.7
81.8
48.3
32,8
44,2
75.5
73.25
LoRA based
LaRA
0.29
67.1
91.4
69.4
98.8
90.4
85.3
54.0
84.9
95.3
84.4
73.6
82.9
69.2
49.8
78.5
75.7
47.1
31.0
44.0
74.60
72.25
FacT-TT
0.04
71.3
89.6
70.7
98.9
91.0
81.8
54.6
85.2
95.5
83.4
75.7
82.0
69.0
49.8
80.0
79.2
48.4
34.2
41.4
75.34
73.04
FacT-TK
70.6
90.6
70.8
99.1
90.7
88.6
54.1
84.8
84.5
75.7
68.2
49.8
80.7
80.8
47.4
33.2
43.0
75.56
73.23
SPT-LORA
0.54
73.5
93.3
72.5
99.3
91.5
87.9
55.5
85.7
96.2
85 9
75.9
84.4
67.6
52.5
82.0
81.0
51.1
30.2
41.3
76.37
74.07
CaRA (ours)
71.8
99.3
90.5
75.4
$1.7
35.3
14.0
76.46
ents for the feed-forward low-rank updates are given by
Appendix B. Similar to LoRA, the weight updates 8W and 6h are added to pretrained weights Wmha and Wifn during inference. This design choice incurs additional latency and inference costs.
V x L = 6(4)
(17)
1 L 31
(18)
6.) L 82
(19)
4. Experiments
6.) L B3 6,
(20)
We evaluate the performance of CaRA on various vision classification datasets and further perform extensive abla- tions to demonstrate its effectiveness and explain our design.
where B E 19 82 E Xdmmo and I B3 E
R dauad x duead are also identity matrices. The proof for the gradients is provided Appendix A. Considering projection layer in MHA, LoRA needs gra- dients for (dmotel + h da. ) parameters, whereas CaRA needs to update only (dmotet + h + d,) R parameters. Math- ematically, CaRA's low-rank update has fewer degrees of freedom, that is, fewer parameters. This constraint forces the network to learn distinct features, and for this purpose, we study the saliency maps of CaRA in Section 5.3.
4.1. Visual Task Adaptation Benchmark (VTAB)
To evaluate the performance of CaRA, we fol- low the experimental setup from (Jia et al., 2022) and benchmark on all VTAB-Ik datasets (Zhai et al., 2019). We use the ImageNet-21k pretrained ViT-B/16 (Dosovitskiy et al., 2021) backbone. Datasets, Metrics and Hyperparameters. We use VTAB-1K, a transfer learning benchmark consisting of 19 diverse vision datasets, arranged into three groups: Natural, Structured and Specialized. The datasets in this benchmark, for example, include CIFAR-100, Resisc45 and KITTI. CaRA is trained on a subset of 1000 samples with an 80-20 split for training and validation, while the original test set is used for evaluation. We report Top-1 accuracy for each dataset in the benchmark for evaluation. We present the hyperparameters, such as rank, in Table 8 of the Appendix, and additional information on the datasets
3.4. CaRA Initialisation
To initialise the above-mentioned factor matrices, we exper- imented with various initialisations presented in Appendix Section B to determine the most effective initialisation. For matrices (1) and (!),, use random normal initiali- sation (Glorot & Bengio, 2010). Matrices A and (2) are initialised to zeros to ensure the corresponding oW is initially zero. The remaining factor matrices (3). (4),
and (3) are initialised as orthogonal matrices (Saxe et al., 2014). More information about A initialisation is given in
6

---

Canonical Rank Adaptation: An Efficient Fine-Tuning Strategy for Vision Transformers
62
61.8
85.9
61.5
Mcan Accuncy
81
Mean Accuracy
Mean Accuracy
61.3
80.66
80.6
85.5
85.3
61
60.5
60.5
80
85
79.5
84.6
60
59.8
CaRAjoun)
FRTT-T CuRAloun)
Fx:T:TT
CaRAioun))
(a) Natural
Special
Structured
Figure 4. Group wise average results on VTAB-IK. While CaRA performs well in Structured and Specialized domains, its performance in Natural datasets is competitive with SPT-LoRA, which requires 6. 6.75 times more trainable parameters than CaRA.
further provided in Section C.2 of the Appendix. Results. Table 2 presents the results of fine-tuning the ViT-B on 19 VTAB-IK datasets. Fine-tuning with CaRA requires only 0.06M parameters, which are averaged across all the datasets. CaRA is comparable in parameters to VPT-shallow and marginally higher than the Tensor- Train representation. This marginal parameter increase relative to FacT-TT arises from using slightly lower rank in the case of FacT-TT. Figure 5 explores the performance of CaRA across varying ranks, demonstrating similar or higher performance compared FacT-TT at lower ranks.
Table 3. Evaluation results on the FGVC benchmark with VIT-B/16 backbone pretrained ImageNet-2Ik. We present the best result in bold and second best as underlined. Our method is highlighted in grey.
AParams CUB-
NABirds Flowers
Stanford
Stanford Mean
(M)
200-
Dogs
Cars
2011
Traditional Fine- Tuning
Linear
85,3
75.9
97.9
86.2
51.3
79.32
FT
85.8
87.3
82.7
98.8
84.5
PEFT methods
VPT-Shallow
0.06
86.7
78.8
98.4
90.7
68.7
84.66
LoRA
0.29
85.6
79.8
98. 9
72.0
84.78
Table 2 establishes that CaRA outperforms existing visual PEFT methods with only a portion of their trainable param- eters. In particular, CaRA achieves state-of-the-ar (SOTA) performance on two datasets. Numbers are presented in bold. Interestingly, we observe that CaRA significantly narrows the gap to SOTA methods on eight datasets. We underlined these numbers in the table. Figure depicts the group-wise average accuracy of VTAB-Ik datasets. CaRA demonstrates significant improvement in comparison isting methods in the case of Special and Structured datasets, whereas for the Natural group, CaRA performs similarly to SPT-LoRA but surpasses LoRA and traditional casor-based methods.
AdaptFormmrr
0.16
84.7
75.2
84.7
83.1
85.12
Adapter
0.16
87.2
84.3
98.5
89.6
68.4
85.60
VPT Deep
88.5
SPT-LoRA
0.54
88.6
83.4
99.5
87.3
90.04
CaRA (ours)
0.08
88.6
86.5
99.4
91.3
86.2
90.46
4.2. Fine-Grained Visual Categorization (FGVC)
Having shown CaRA performance in fine-tuning VTAB-1K 1 further assess the performance on FGVC datasets, using the same ImageNet-211 pretrained ViT-B/16 backbone. Datasets, Metrics and Hyperparameters. FGVC is col- lection five large datascts: CUB-200-200 NABirds, Oxford Flowers, Stanford Dogs and Stanford Cats. More about the dataset statistics. including the training and vali- dation sizes, given in Section C.3 of the Appendix. Unlike VTAB-IK, FGVC utilises the whole training set. We report Top-1 accuracy for evaluation. CaRA is trained with rank 32 across all the datasets. Section C.3 of the Appendix provides more details about hyperparameters.
Overall, in terms of mean accuracy, CaRA outperforms traditional LoRA by 2% and demonstrates higher perfor- mance than existing tensor methods by 1%. Compared to SPT-LoRA, which additionally identifies and trains sensi- tive parameters the LoRA formulation, it achieves similar performance to CaRA at the expense of requiring many more trainable parameters. This result highlights that CaRA further improves the fine-tuning capability of the ViT-B with significantly smaller fraction of trainable parameters. Moreover, as depicted in Figure 4, CaRA performs best the challenging datasets of the Special and Structured group, which mainly cover domains that are highly dissimilar to ImageNet.
Results. Table shows the performance CaRA on FGVC. As observed earlier, CaRA achieves competitive perfor- mance with only small fraction of trainable parameters: 0.08M. Compared to LoRA, VPT-Shallow and other meth- ods, we showease significant improvement and reach per- formance levels comparable to those of SPT-LoRA. CaRA achieves SOTA performance on three out of five datasets, such as CUB-200-2011, NABirds, and Stanford Dogs. In the case of the other datasets, Stanford Cars and Oxford-

---

Canonical Rank Adaptation: An Efficient Fine-Tuning Strategy for Vision Transformers
Table Evaluation results the -arge hackhone pretrained on ImageNet-2IK, We present the best result in bold and the second
UP
CaRA(Ours) FacT-TT PTcT-TKK
#Parameters (M)
107
50
Accurccy
best as underlined. Our method highlighted grey.
#Parms (M)
CI
Food101
Flowers
Resisc45
Mean
Traditional Fine- Tuning
CaRA(Ours) FacT-TT FacT-TK
Linear
79.4
76.5
98,9
67.8
80.65
Pull
303,3
86.8
78.7
98.8
79.0
85.83
156
16 Ranks
30
6
PEFT methods
Ranks
LoRA
0.766
87.0
79.5
99.1
78.3
85.98
0.061
87.5
79.2
99.2
78.6
86.13
PISSA
0.786
87.1
79.6
99.7
78.6
86.24
Figure 5. CaRA's robustness to rank. The left figure depicts the
DoRA
87.9
99.6
80.3
87.60
parameter growth in various tensor formulations with varying rank. The figure on the right illustrates the mean accuracy of various tensor formulations evaluated on the Special group in VTAB-Ik with varying ranks. Best viewed in colour.
CaRA louns)
0.076
89.4
83.7
99.8
82.4
88.77
Flowers, CaRA performs on par with SPT-LoRA and VPT- Deep. Overall, these results, combined with the previous experiments, demonstrate that our proposed tensor- based CaRA significantly outperforms traditional LoRA, adapter and other bessor-based methods in vision classification tasks while utilising only small fraction of trainable parameters.
varying ranks to the powers of two. For comparison, trained the Tucker and Tensor-Train low-rank representa- tions proposed in FacT (Jie & Deng, 2023). Figure 5 on the left depicts the parameter growth with rank increase. We observe that the number of parameters for FacT-TT and FacT-TK grow faster than CaRA. This phenomenon is due to tensors in Tensor-Train and Tucker formulations for low-rank updates. In contrast, the CaRA representation only contains matrices, presented in Table 1. Figure 5 on the right demonstrates the evaluation of FacT-TT, Fac and CaRA with varying ranks. The effect of exponential parameter growth negatively impacts the performance of the Tensor- and Tucker methods, whereas CaRA's per- formance slightly increases with an increase in rank. This result demonstrates the robustness of CaRA rank and its efficient use of parameters.
4.3. Fine-tuning VIT-L
To evaluate the performance of CaRA larger vi- sion transformer, we fine-tune ViT-Large pretrained on ImageNet-2Ik across four datasets.
Datasets, Metrics and Hyperparameters. Following Kopiczko et al. (2024), we fine-tune on CIFARI00, Food101, Flowers and Resisc45 using 10 randomly sampled training examples per class. Evaluation is per- formed the CIFARI00, Food101, and Flowers102 test sets, and the remaining samples for Resisc45. Further implementation and hyperparameter details are provided in Section C.4.
5.2. Dimension Ablation
Results. Table reports the performance of CaRA across all four datasets. As observed in other experiments, CaRA achieves SOTA results using only x 10% of the trainable parameters compared LoRA. Regarding parameter count, CaRA is comparable to VeRA, yet it outperforms VeRA by margin of 2%. Despite the sigmificantly smaller fraction of trainable parameters, these results showcase CaRA's AssAss scaling capability and effectiveness in a few-shot learning regime.
As discussed in Section 3.2.1, formulation allows the low-rank update of Wmiaa to be represented at maximum of five dimensions. In this subsection, we investigate the effect of dimensionality on the performance of various low- rank update dimensions with the Special group of datasets in VTAB-Ik. Table demonstrates the effect of dimen- sionality. Despite slightly smaller trainable parameters in a five-dimensional representation, it is less effective than the four-dimensional format. We hypothesise this effect to the merged QKV projection implementation in the pretrained backbone (Dosovitskiy et al., 2021), contrary to the indi- vidual Q K, and projections as described in (Vaswani et al., 2017). While the three-dimensional representation achieves a similar accuracy the proposed presentation, this improvement comes at the cost of a rise in trainable parameters. Moreover, we trained our model with com- bined tensor of Wmha and fffnn as shown in the last row of Table 5. We observe that the two separate factorisa- tions achieve higher accuracy than the combined represen- tation. Altogether, from this ablation, we learn that the four-dimensional representation yields better performance
5. Ablation Study
For better understanding of our proposed CaRA, per- form comprehensive ablations. We study CaRA's 'ss robust- ness to rank, followed by the effect of dimensionality in tensorisation. Subsequently, we investigate the saliency maps and analyse CaRA's 's computational complexity. All ablations are performed on VTAB-Ik's Special group.
5.1. Robustness to Rank
In this subsection, we explore the effect of rank on the per- formance of CaRA. Accordingly, we trained CaRA with
8

---

Canonical Rank Adaptation: An Efficient Fine-Tuning Strategy for Vision Transformers
rank weights are merged into the corresponding pre-trained weights. We observe that LoRA is most efficient in terms of both training time and memory. We attribute the effi- cient training time of LoRA to Compute Unified Device Architecture (CuDA)-opimmisd matrix multiplications in PyTorch. In contrast, CaRA's code is not optimised, as underlying packages are primarily written in Python. More- over. CaRA's 's multi-dimensional nature results in slightly higher CUDA memory. In general, the memory consump- tion of the methods is very similar. Only DoRA requires much more memory due to the additional weight normal- isation. Despite the slightly higher walltime and memory consumption compared to LoRA and FacT, CaRA demon- strates performance improvements in both ViT base and large architectures.
Table Dimension ablation experiment evaluated on Special group of VTAB-IK, Ik. All experiments are conducted with the rank of 32. Dimensions for W 9 dmadal X dmotel except last line
where it is merged into the MHH tensor.
Tensors
MHA-Dimension
#Params (M)
Accuracy
sD. W
3 I x danoaa X h x da
0.08476
84.682
WDin.D. With
31 x danae x. lid..
0.10758
85.061
w 4D. T (CaRA)
31 x dunde X. la x dy
0.08544
85.134
wmm 3D W
12 x duuddt X dhood
0.05840
83.914
Table CaRA's computational complexity with walltime mea- sured in seconds and VRAM occupied in gigabytes.
Method
LoRA
DoRA
FacT-TT
FacT-TK
CaRA (ours)
Walltime (1)
165.756
204.076
178.283
180,578
206.554
VRAM (1)
20.108
28.065
20.246
20.244
21.374
6. Discussion
and lower trainable parameters. Additionally, the disentan- gled low-rank update for MHA and FFN demonstrates significant improvement in accuracy.
In this work, we have introduced novel and effective tensorisation of a vision transformer, explicitly address- ing the nigh-dimensional nature of Multi-Head Attention (MHA) during fine-tuning. Our proposed Canonical Rank Adaptation (CaRA) exploits this tensorisation alongside the efficiency of the Canonical-Polyadte Decomposition (CPD) form. CaRA consistently outperforms tensor-based and matrix-basedd low-rank update methods across a di- verse range of vision datasets, demonstrating its robustness. Notably, CaRA achieves these results with a significantly smaller number of parameters, fully establishing its place in the visual Crrumeter-Efficieee Furthermore, the CaRA factors can be merged after fine- tuning into the pre-trained weights, ensuring no additional overhead for inference.
5.3. Interpretability
This subsection attempts to understand CaRA's fine-tuning capability. Subsequently, we examine the saliency maps generated from Integrated Gradients (Sundararajan et al., 2017).
Dataset and Model. We use the FGVC-Aireraft dataset (Maji et al., 2013), which specialises in identify- ing the various aireraft families. This dataset is ideal for saliency studies because aircraft are part of the ImageNet dataset, but the ImageNet pre-trained model cannot differ- entiate between aircraft families. Thus, FGVC-Aircraft constitutes an ideal dataset for fine-tuning. The Appendix Section E presents more details on the dataset.
Fining (PEFT) family,
Despite its promising results, our approach has some limita- tions. The training time of our tensor-based implementation is higher compared to the matrix-based approach LoRA. hardware-optimised tensor decomposition, such as (Yang et al., 2022; Kao et al., 2022), might provide more ef- ficient implementation. As part of future work, weight- normalised form of CaRA could be investigated. The benefit of adding weight normalisation to LoRA has been shown in DoRA (Liu et al., 2024) and a similar form exists for CPD (Kolda & Bader, 2009).
Figure 6 in the Appendix depicts the saliency maps of ViT-B fine-tuned with ranks 32 and 16 for various aireraft. Saliency maps from rank 32 show concentrated gradient over specific aircraft parts, while for rank 16, they are more spread out. This difference is likely attributed to signif- icantly lower parameters in the r6 model, resulting in a broader focus. Additionally, as humans, we rely on distinet aircraft features to identify them, like the cockpit bump on the B-747 or the third aft engine on the L-1011. The network also focuses on these features. This highlights CaRA's 's ability to learn distinct and discriminative features, providing effective fine-tuning with limited set of trainable parameters.
Impact Statement
Parameter-Efficien Fine-Tuning (PEFT) methods such as Canonical Rank Adaptation (CaRA) reduce the computa- tional cost, and thus the environmental footprint, for fine- tuning very large transformer architectures, including vision, language and modal models.
5.4. Computational Analysis
In Table 6, we report the training time and GPU memory usage during training for various PEFT methods for fine- tuning ViT-L on CIFAR 100. Note that these methods do not incur additional overhead during inference, as the low-
9

---

Canonical Rank Adaptation: An Efficient Fine-Tuning Strategy for Vision Transformers
Acknowledgements
Dosovitskiy, A., Beyer, L.,, Kolesnikov, A., Weissenborn
D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021.
This research was supported by the Federal Ministry of Ed- ucation and Research (BMBF) under grant no. 011S22094A WEST-AI and 6DHBK 1022 BNTrAInee, and the Deutsche Forschungsgemminsscaff (DFG, German Research Foun- dation) under Germany's ' Excellence Strategy EXC 2070-390732324 (PhenoRob) and project no. 459420781 (for-5361), Prof. Kuehne is supported by the BMBF project STCL-011S22067. We thank our anonymous reviewers for helping us improve this publication. The authors gratefully acknowledge the HPC team at the University of Bonn and RWTH Aachen for compute time on the Bender system and CLAIX-2023 system under project rwth wwth1629, respectively. The authors gratefully acknowledge the Gauss Centre for Supercomputing e.V. (www.gane www.gauss-centre.cm eu) for funding this project by providing computing time through the John von Neumann Institute for Computing (NIC) on the GCS Supercomputer JUWELS Jülich Supercomputing Cen- tre (JSC). They also thank Zahra Ganji for reimplementing VeRA's vision baseline. The authors are solely responsible for the content of this publication.
Fei-Fei, L.., Fergus, R., and Perona, P. Learning generative visual models from few training examples: An incremen- tal bayesian approach tested on 101 object categories. In 2004 Conference on Computer Vision and Pattern Recog- nition Workshop, pp. 178-178, 2004.
Gebru, T., Krause, J., Wang, Y., Chen, D., Deng, J., and Fei-Fei, L. Fine-grained car detection for visual census estimation. Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Geiger, A., Lenz, P., Stiller, C., and Urtasun, R. Vision meets robotics: The KITTI dataset. Int. J. Robotics Res., 12111):121111237, 2013.
References
Glorot, X. and Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In Pro- ceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceed- ings of Machine Learning Research, pp. 249-256. PMLR, 2010.
Beattie, C., Leibo, J. Z., Teplyashin, D., Ward, T., Wain- wright, M., Küttler, H., Lefrancq, A., Green, S., Valdés, V., Sadik, A., et al. Deepmind lab. arXiv preprint arXiv:1612.03801, 2016.
Bershatsky, D., Cherniuk, D., Daulbaev, T., Mikhalev, A., and Oseledets, I. V. LoTR: Low tensor rank weight adap- tation. arXiv preprint arXiv:2402.01376, 2024.
He, H., Cai, J., Zhang, J., Tao, D., and Zhuang, B. Sensitivity- visual parfmeter-efficient tine-tuning. IEFF/CVF International Conference on Computer Vi- sion, ICCV 2023, Paris, France, October 1-6, 2023, pp- 11791-11801, IEEE, 2023.
Bossard, L.,, Guillaumin, M., and Van Gool, L. Food-101 mining discriminative components with random forests. In European Conference Computer Vision, 2014.
Helber, P., Bischke, B., Dengel, A., and Borth, D. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE J. Sel. Top. Appl. Earth Obs Remote. Sens., 12(7):2217-2228 2019.
Chen, S., Ge, C., Tong, Z.,, Wang, J., Song, Y., Wang, J., and Luo, P. Adaptformer: Adapting vision transformers for scalable visual recognition. In Advances in Neural Information Processing Systems, 2022.
Hitchcock, F. L. The expression of tensor or a polyadic as a sum of products. Journal of Mathematics and Physics, ((1-4))16--18, 1927.
Cheng, G., Han, J., and Lu, X. Remote sensing image scene classification: Benchmark and state of the art. Proceed- ings of the IEEE, 105(10)18855-188) 2017.
Hitchcock, F. L. Multiple invariants and generalized rank of a p-way matrix or tensor. Journal of Mathematics and Physics, ((1-4):39-7, 1928.
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In 2014 IEEE Conference on Computer Vision and Pattern Recog- nition, pp. 3606-3613, 2014.
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-effieient transfer learning for NLP. Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2790-2799, PMLR, 2019.
De Lathauwer, L., De Moor, B., and Vandewalle, J. A multilinear singular value decomposition. SIAM Journal on Matrix Analysis and Applications, 21(4):1253-1278, 2000.
10

---

Canonical Rank Adaptation: An Efficient Fine-Tuning Strategy for Vision Transformers
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, I.,, and Chen, W. LoRA: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, 2022.
Lebedev, V., Ganin, Y., Rakhuba, M., Oseledets, I. V., and Lempitsky, V. S. Speeding-up convolutional neural net- works using fine-tuned cp-decomposition. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Representations, 2015.
Jia, M., Tang, L., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., and Lim, S.-N. Visual Prompt Tuning. In European Conference on Computer Vision (ECCV), 2022.
LeCun, Y., Huang, F. J., and Bottou, L. Learning methods for generic object recognition with invariance to pose and lighting. In Proceedings of the 2004 IEEE Computer So- ciery Conference on Computer Vision and Pattern Recog- nition, 2004. CVPR 2004., volume 2, pp. I1-104. IEEE, 2004.
Jie, S. and Deng, Z.-H. FacT: Factor-tuning for lightweight adaptation on vision transformer. In Proceedings of AAAI Conference on Artificial Intelligence (AAAI), 2023.
Johnson, J., Hariharan, B., Van Der Maaten, L.. Fei-Fei, L., Lawrence Zitnick, C., and Girshick, R. Clevr: A diagnostic dataset for compositional language and ele- mentary visual reasoning. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 20022291210, 2017.
Lian, D., Zhou, D., Feng, J., and Wang, X. Scaling & shifting your features: A new baseline for efficient model tuning. In Advances in Neural Information Processing Systems, 2022.
Liu, S.-Y., Wang, C.-Y., Yin, H., Molchanov, P., Wang, Y.-.. F., Cheng, K.-T., and Chen, M.-H. DoRA: Weight- decomposed low-rank adaptation. In Proceedings of the s1 International Conference on Machine Learning, vol- ume 235 of Proceedings of Machine Learning Research, PMLR, 21 27 Jul 2024.
Kaggle and EyePAcs. Kaggle diabetic retinopathy de- tection. http://www..ggl..com/dddaabteete detection/data, 2015.
Kao, C.-C., Hsieh, Y.Y., Chen, C.-H., and Yang. C.-H. Hardware acceleration in large-scale tensor decomposi- tion for neural network compression. In 2022 IEEE 65th International Midwest Symposium on Circuits and Sys- tems (MWSCAS), pp. 1-4. IEEE, 2022.
pp.
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L.., and Stoyanov, V. Roberta: A robustly optimized BERT pretraining approach. arXiv preprint arxivv 2023.
Khosla, A., Jayadevaprakash, N., Yao, B., and Fei-Fei, L. Novel dataset for fine-grained image categorization. In First Workshop on GrainGr Visual Categorization, IEEE Conference on Computer Vision and Pattern Recog- nition, Colorado Springs, CO, June 2011.
Liu, Z., Lin, Y., Cao. Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Kolda, T. G. and Bader, B. W. Tensor decompositions and applications. SIAM review, 51(3):455-500 2009.
Luo, G., Huang, M., Zhou, Y., Sun, X., Jiang, G., Wang. Z.,, and Ji, R. arXiv preprint arXiv:2302.08106, 2023.
Koohpayegani, S. A., I.,, N. K., Nooralinejad, P., Kolouri, S., and Pirsiavash, H. NOLA: compressing lora using linear combination of random basis. In The Twelfth Inter- national Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
Maji, S., Kannala, J., Rahtu, E., Blaschko, M., and Vedaldi, A. Fine-grained visual classification of aircraft. Technical report, 2013.
Mangrulkar, S., Gugger, S., Debut, L.., Belkada, Y., Paul, S., and Bossan, B. PEFT: State-of-the-art parameter- efficient fine-tuning methods. attps::/ee /ggthub. com/ http:/hggnngffcceeeeeee haagfffcceeeefffee ft, 2022.
Kopiezko, D. J., Blankevoort, T., and Asano, Y. M. VeRA: Vector-based Random Matrix Adaptation. In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenRe- view.net, 2024.
Meng. F., Wang, Z., and Zhang, M. PiSSA: Principal sin- gular values and singular vectors adaptation of large lan- guage models. In Advances in Neural Information Pro- cessing Systems, 2024.
Kossaifi, J., Panagakis, Y., Anandkumar, A., and Pantic, M. Tensorly: Tensor learning in python. Journal of Machine Learning Research (JMLR), 20(26), 2019.
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A. Y., et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp. 4. Granada, 2011.
Krizhevsky, A. and Hinton, G. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.

---

Canonical Rank Adaptation: An Efficient Fine- Fine-Tuning Strategy for Vision Transformers
Nilsback, M.-E. and Zisserman, A. Automated flower clas- sification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing. pp. 722-729. IEEE, 2008.
Veeling, B. S., Linmans, J., Winkens, J., Cohen, T., and Welling, M. Rotation equivariant CNNs for digital pathol- ogy. In Medical Image Computing and Computer Assisted Intervention, pp. 210-218. Springer, 2018.
Oseledets, I. V. Tensor-train decomposition. SIAM Journal on Scientific Computing, 33(5):2295-22) 2317, 2011.
Veeramacheneni, L., Wolter, M., Klein, R., and Garcke, J. Canonical convolutional neural networks. In Inter- national Joint Conference on Neural Networks, pp. 1-8. IEEE, 2022.
Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-3505. IEEF, 2012.
Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. The caltech-ucsd birds-200-2011 dataset. Technical Report nology, 2011.
California Institute of Tech-
Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. Automatic differentiation in pytorch. In NIPS Work- shop on Autodiff, 2017.
Wightman, R. Pytorch image models, https://githib: com/ rwi www.:wwigghttt.. wwt.gg/ppaat/tttttttttoo models, 2019.
Phan, A. H., Sobolev, K., Sozykin, K., Ermilov, D., Gusak, J., Tichavský, P., Glukhov, V., Oseledets, I. V., and Ci- chocki, A. Stable low-rank tensor decomposition for com- pression of convolutional neural network. In Computer Vision - ECCV 2020, pp. 522-539. Springer International Publishing. 2020.
Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A. Sun database: Large-scale scene recognition from abbey zoo. In 2010 IEEE Computer Society Conference Computer Vision and Pattern Recognition, pp. 3485- 3492, 2010.
Raghu, M., Unterthiner, T., Kornblith, S., Zhang, C., and Dosovitskiy, A. Do vision transformers see like convolu- tional neural networks? In Advances in Neural Informa- tion Processing Systems, pp. 12116-12128, 2021.
Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, M., and Luo, SegFormer: Simple and efficient design for semantic segmentation with transformers. In Neural Information Processing Systems, 2021.
Saxe, A. M., McClelland, L., and Ganguli, S. Exact solutions the nonlinear dynamics of learning in deep linear neural networks. In 2nd International Conference on Learning Representations, 2014.
Yang, Y., Zhou, J., Wong, N., and Zhang, Z. LORETTA:
Low-rank economic tensor-traie adaptation for ultra-low- parameter fine-tuning of large language models. In Pro- ceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies ( (Volume I: Long Papers), pp. 3161-3176, 2024.
Shuttleworth, R., Andreas, J., Torralba, A., and Sharma, P. LoRA vs Full Fine-tuning: An illusion of equivalence. arXiv preprint arXiv:2410.21228. 2024.
Sundararajan, M., Taly, A., and Yan, Q Axiomatic attribu- tion for deep networks. In Proceedings of the 34th Inter- national Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 3319- 3328. PMLR, 2017.
Yang, Z., Shan, J., and Zhang, Z. Hardware-efficient mixed sexeedprecision cp tensor decomposition. arXiv preprint arXi::2209.0400 2022.
Zhai, X., Puigcerver, J., Kolesnikov, A., Ruyssen, P., Riquelme, C., Lucic, M., Djolonga, J., Pinto, A. S., Neu- mann, M., Dosovitskiy, A., et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv: 1900.04667 2019.
Synge, and Schild, A. Tensor calculus, volume Courier Corporation, 1978.
Van Horn, G., Branson, S., Farrell, R., Haber, S., Barry, J., Ipeirotis, P., Perona, P., and Belongie, S. Building bird recognition app and large scale dataset with citizen scien- tists: The fine print in fine-grained dataset collection. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 595-604, 2015.
Zhang, Y., Zhou, K., and Liu, Z. Neural Prompt Search. IEEE Transactions on Pattern Analysis and Machine In- telligence, pp. 1-14, 2024.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L.,, and Polosukhin, I. Atten- tion is all you need. In Advances in Neural Information Processing Systems, pp. 1999860088 2017.
12

---

Canonical Rank Adaptation: An Efficient Fine-Tuning Strategy for Vision Transformers
Acronyms
CaRA Canonical Rank Adaptation
CPD Canonical-Polyadic Decomposition
CUDA Compute Unified Device Architecture
DoRA Weight Decomposed Low-Rank Adaptation
FacT Factor Tuning
FGVC Fine-Grained Visual Categorization
FT full fine-tuning
GLUE General Language Understanding Evaluation
LLMs Large Langauge Models
LoRA Low-Rank Adaptation
MHA Multi-Head Attention
NLU Natural Language Understanding
PEFT Gtrameter-Effiiien Fine-Tuning
RoBERTa Robustly Optimised BERT Approach
SOTA stale-of-the-art
VeRA Vector-based Random Matrix Adaptation
VIT Vision Transformer
VTAB Visual Task Adaptation Benchmark
A. Gradient Derivation
We derive the gradient for a 3 dimensional (B-xJxX) low-rank CPD update. This can be further generalised to an n-dimensional update. Consider 3D low-rank update
== ow
Σ x 1 s
(21)
where represents the outer product. In case of an individual rank 2, we can omit the summation
F. X { ( (3) The gradient for X is the outer product of rank-one one vectors without X
(22)
V Va s s o
(23)
From here on, we absorb the x into the first factor s for case of derivation. To derive the gradients for other factors, we use the tensor representation from (Kolda & Bader, 2009). The elements of the tensor F, can be written as
F, (1) jr
Vi I...IIj i...J,k 1...K
(24)
kr
Gradient of th element in (1) is given as
jr 8kr
(2) (3)
if t
(25)
0. if i + t.
13

---

Canonical Rank Adaptation: An Efficient Fine-Tuning Strategy for Vision Transformers
The above form is analogous the Kronecker delta. Equation 25 in element form of the tensor, converting CPD form results in
- = o s s,
(3)
(26)
where 7S1 E TRX is an identity matrix resulting from vectorisation of the Kronecker delta. Similarly, for the other two factors, the gradient can be represented as
8, F. 1 o F, 153
(27)
8( 0
(28)
where 752 E RJXJ and J RxK are also identity matrices. Since the gradient for the factors computed from the wisment-wise formulation of tensors, this is generalisable an n-dimensional low-rank update. For an n-dimensional low-rank update, the gradient of the specific factor is replaced by its identity matrix.
B. Effect of Initialisations
In this section, we provide an additional ablation on the effect of various initialisations of CPD factors. Table presents the results of various initialisations such as random normal (Glorot Bengio, 2010) and orthogonal (Saxe et al., 2014). As discussed earlier, we initialise the second factor as zero and evaluate on the SVHN dataset from the VTAB-IK benchmark. We observe that initialising the last two factors in MHA with orthogonal and the first factor by sampling from normal distribution results in the best performance. We hypothesise that the initialisation of the first factor by sampling from Gaussian distribution is crucial because A E Ra is the only factor containing information for each transformer block's s low-rank update. Following this result, we initialise the factors for the feed-forware tensor similarly, with the bias term set to zero
Table Effect of various factor initialisations. represents zero initialisation, O denotes orthogonal initialisation (Saxe al., 2014), N represent random normal initialisation (Glorot & Bengio, 2010). We used the SVHN dataset for evaluation.
A
A
A (3)
A(4)
Accuracy
0
Z
0
O
89.79
O
Z
N
N
89.81
N
Z
N
N
89.95
N
Z
O
O
90.50
C. Impleeentation Details
In this section, provide details about the VTAB- Ik and FGVC benchmarks, well the benchmark (Kopiczko et al., 2024) for fine-tuning ViT-L. For each benchmark, we provide the used hyperparameters.
C.1. Software and Hardware Details
We use PyTorch (Paszke al., 2017). Tensorly (Kossaifi al., 2019) and PyTorch Image Models (Wightman, 2019) for model fine-tuning and evaluation. The pre-trained VIT-B checkpoint is available here!. We fine-tuned the VIT on one Nvidia GA100 GPU for the VTAB-Ik benchmark and one Nvidia H100 GPU for the FGVC benchmark. For evaluation, we use an Nvidia RTX A5000. In the case of language experiments, we use a maximum of 8 Nvidia GA 100 GPUs for fine-tuning and evaluation.
C.2. VTAB-IK
The Visual Task Adaptation Benchmark (VTAB)-1k consists of 19 diverse datasets categorised into Natural, Specialised, and Structured groups. Firstly, the natural group consist of datasets from classical vision problems such as CIFAR- 100 (Krizhevsky & Hinton, 2009), Caltech-101 (Fei-Fei et al., 2004), DTD (Cimpoi al., 2014), Flowers 102vers102 (Nilsback & Zisserman, 2008), Pets (Parkhi et al., 2012), SVHN (Netzer al., 2011), and Sun397 (Xiao et al., 2010). Secondly,
'https://stores" googl googleapis. eapis. comdvl model s/ htt///magnet22111 1kviiitttttk//tB-BBBt 16.
14

---

Canonical Rank Adaptation: An Efficient Fine-Tuning Strategy for Vision Transformers
the Specialised group contains images from special sensors like satellites and mieroscopes. The datasets in this group Resisc45 (Cheng et al., 2017), EuroSAT (Helber et al., 2019), Patch Camelyon (Veeling al., 2018), and Diabetic Retinopathy (Kaggle & EyePAcs, 2015). Finally, the third group includes Clevr (Johnson et al., 2017) (location and orientation), SmallNorb (LeCun et al., 2004) (azimuth and elevation), DMLab (Beattie et al., 2016), and KITTI (Geiger et al., 2013). The Natural group of datasets contains images of domains similar to those ImageNet. Meanwhile, the Structured and Specialised groups contain images from domains dissimilar to ImageNet (Zhai et al., 2019). This diverse mix of domains makes this benchmark suitable for evaluating the fine-tuning characteristics. We initialise X and A by sampling from normal distribution with mean d and standard deviation Ag. Table 8 presents the detailed hyperparameters.
Table Hyperparameter details for the VTAB-IK benchmark using the ViT-Base model. The standard deviation (std) computed over
10 runs.
Hyperparameter
NORB-AAin Aziim
Cifar100
Caltech101
Flower1 102
SVHN
Sun3997
Camelyon
EuroSAT
Resise45
Retinopathy
Clevr-Com Count
Clevr-Dist
DMLab
KITTI-Dist
dSpr-Loc
dSpr- -Hl
SNORB-Ele
DTD
Pots
a
100
0.1
10
100
10
0.1
5
2.5
5
50
100
10
Ag
1.5
0.9
1.0
1.0
1.2
1.0
1.35
1.0
1.08
1.16
1.0
1.0
1.0
1.0
1.0
1.0
1.3
1.0
1.0
Ag
0.1
0.01
0.1
0.02
0.06
0.05
0.06
0.0
0.028
0.03
0.0
0.0
0.0
0.0
0.0
0.0
0.07
0.0
0.0
rank
32
32
16
32
16
32
16
32
32
32
16
32
16
32
32
32
16
32
16
std
0.15
0.31
0.16
0.06
0.18
0.18
0.04
0.31
0.10
0.11
0.12
0.10
0.26
0.37
0.73
0.37
0.58
In Table 2, the results Linear, FT, VPT-Shallow, VPT-Deep, and Adapter-256 are from (Jia et al., 2022). The results for Adaptformer and RepAdapter are from (Luo et al., 2023), and the results for LoRA are from (Jie Deng, 2023). The other results are from their corresponding papers.
C.3. FGVC
The Gane-Grained Visual Categorization (FGVC) benchmark is a collection of five large datasets: CUB-200-2011 (Wah al., 201 1), NABirds (Van Horn al., 2015), Oxford Flowers (Nilsback & Zisserman, 2008), Stanford Dogs (Khosla et al., 2011), and Stanford Cats (Gebru et al., 2017). These five datasets are purely domain-specifie and focus on fine-grained categories. We use the whole training set, unlike the VTAB-1k benchmark. The validation split is done with statistics from (Jia et al., 2022) with seed 0. The class information is as follows: CUB-200-201 I contains 200 classes, NABirds 555 categories, Oxford Flowers 102 classes, Stanford Dogs 120 classes, and Stanford Cars 196 classes. The hyperparameters for the FGVC benchmark are presented in Table 9.
Table Hyperparameter details for the FGVC benchmark using the ViT-Base model.
Hyperparameter CUB-200 SUB-200-2011
NABirds
Oxford Flowers
Stanford Dogs
Stanford Cars
a
0.01
0.01
10
0.001
100
A
1.0
1.0
1.0
0.9
1.0
0.0
0.0
0.02
0.02
0.0
rank
32
32
32
32
32
C.4. Fine-tuning VIT-L
Following experiments from Kopiezko et al. (2024), we fine-tune VIT-L the CIFAR100 (Krizhevsky & Hinton, 2009). Food101 (Bossard et al., 2014), Oxford Flowers (Nilsback & Zisserman, 2008), and Resisc45 (Cheng et al., 2017) datasets. The experimental setup uses 10 training samples per class from the training set and evaluates on the test set. We use the numpy random choice with seed 6 for sampling to ensure reproducibility. Table 10 presents the CaRA hyperparameters for the individual datasets. Regarding this experiment, we also fine- tune other PEFT methods such as DoRA (Liu et al., 2024) and PiSSA (Meng et al., 2024). We use the implementations of these methods from (Mangrulkar et al., 2022). For DoRA
15

---

Canonical Rank Adaptation: An Efficient Fine-Tuning Strategy for Vision Transformers
and PiSSA, we use rank of 8 and a value of 8, and fine-tune for 100 iterations.
Table 10. Hyperparameter details for four image classification datasets using the ViT-Large model. The standard deviation (std) is computed over runs.
Hyperparameter
CIFAR100
Food101
Flowers 102
Resisc45
a 0.01
0.01
0.01
0.01
1.0
1.0
1.0
1.0
X
0.01
0.01
0.0
0.01
rank
32
32
32
32
std
0.16
0.33
0.03
0.19
D. Additional fine-tuning results
We provide additional experiments for fine-tuning Swin Transformer (Liu et al., 2021) and the Robustly Optimised BERT Approach (RoBERTa) (Liu et al., 2023).
D.1. Fine-tuning Swin-B
Swin Transformer (Liu et al., 2021) adds hicrarchical structures to VIT and we demonstrate that CaRA works with hierarchical architectures, which use different embedding dimensions and number of heads for different layers, as well. For the experiments, we use the Swin-Base model pre-trained on ImageNet-21k. The pre-trained model checkpoint is available here?. The Swin-B layers are arranged into four stages, each with varying embedding dimensions dmnddel and number of heads h. The proposed tensorisation in Section 3,2.1 must thus be adapted for each stage individually. Table 11 details each stage with corresponding tensors W milha and W shapes.
Table Tensorising Swin Transformer: Tensor shapes Wmlaa and fi across four stages Swin-B, For each stage, the number layers, embedding dimension (duota), and number of attention heads (4) are also reported.
Stage
# #Layers
dmoddl
h
mmhh
Iin
128
6 x 128 x 32
18 x 128 128
2
256
6 x 256
32
20 256 256
18
512
16
54 x 512 16 32
162 512 512
4
2
1024
32
6 x 1024 X 32 X 32
22 x 1024 1024
Results. We fine-tuned the Swin Transformer with CaRA the VTAB-Ik datasets. The results in Table 12 show that CaRA outperforms existing PEFT methods in terms of accuracy, despite using less trainable parameters than most of the other methods. While SPT-LoRA and FacT-TT perform slightly better on the Natural datasets, CaRA outperforms them on the Special and Structured datasets.
D.2. Natural Language Understanding (NLU)
To evaluate CaRA's 's capabilities for the language domain, we evaluate it on the General Language Understanding Evaluation (GLUE) benchmark using the RoBERTa model. Our experimental setup closely aligns with (Hu et al., 2022). In particular, we apply CaRA only to query and value projection matrices in the MHA module. CaRA trained with rank 64, a set to 100, and all the X's are initialised as ones. The results for FT and non-tensor based PEFT methods are from (Hu al., 2022). For the other methods, we use the results from their corresponding publications.
Results. Table 13 demonstrates the performance of CaRA the GLUE benchmark when fine-tuned with the RoBERTa- Base model. We observe that CaRA outperforms existing tensor-based fine-tuning methods such LoTR (Bershatsky et al., 2024) and LoRETTA adapter (Yang et al., 2024). We observe that tensor-based methods like LoTR and LORETTA lag in
https:: ////t thub. com/ SwwnTrans ndow7_22 224 22k. 22k_ pth
rwt.//eeeeeeeeeeeeeeeee wassee/ddwwloodd//e. /wwi bass__pttchhee_ee
16

---

Canonical Rank Adaptation: An Efficient Fine-Tuning Strategy for Vision Transformers
Table 12. VTAB-Ik evaluation results with the Swin-Base model. Our method results highlighted grey. Mean accuracy and
parameters are averaged over group wise mean values. We present the best result in bold and the second-best is underlined.
#Params(M)
Natural
Special
Structured
Group Mean
FT
86.7
79.1
86.2
59.7
75.00
Linear
73.5
80.8
33.5
62.60
BitFit
0.201
74.2
80.1
42.4
65.57
VPT-Shallow
0.003
79.9
82.5
37.8
66.73
VPT-Deep
0.162
76.8
84.5
53.4
71.57
LoRA
1.023
81.7
87.2
60.1
76.33
SPT-LoRA
0.424
83.1
87.4
60.4
76.97
FacT-TT
0.135
83.1
86.9
62.1
77.37
CaRA (ours)
0.134
82.9
87.7
63.0
77.89
performance compared to LoRA, FT, and Adapter methods. CaRA further closes this performance gap to LoRA and full fine-tuning (FT), with only 0.04% FT parameters.
Table GLUE benehmark cvaluation results with the RoBERTa-Bare model. Following (Hu et al., 2022), we report Matthew's correlation for CoLA, Pcarson correlation for STS-B, and accuracy for other tasks.
#Params (M)
MNLI
QQP
MRPC
SST-2
CoLA
QNLI
RTE
STS-B
Mcan
FT
125
87.6
91.9
90.2
94.8
63.6
92.8
78.7
91.2
86.4
Non-Tensor based
Adapter
0.30
87.1
90.2
88.5
94.2
60.8
93.1
71.5
89.7
84.4
BitFit
0.10
84.7
84.0
92.7
93.7
62.0
91.8
81.5
90.8
85.2
LoRA (r=8)
0.30
87.5
90.8
89.7
95.1
63.4
93.3
86.6
91.5
87.2
Tensor based
LoTR (r=32)
0.074
85.2
87.4
85.9
93.0
60.5
90.0
66.0
88.8
77.2
LoTR (r=88)
0.321
84.7
86.9
88.0
93.3
61.3
92.0
67.0
91.0
79.0
LoRETTApdapter
0.10
85.6
87.2
91.1
94.4
62.7
92.1
78.7
90.3
85.2
CaRA (ours)
0.055
86.5
89.7
90.4
94.6
66.1
92.7
77.3
89.7
85.9
E. Saliency Maps
This section provides the saliency maps using Integrated Gradients, as discussed in Section 5.3. Figure shows the saliency maps for CaRA using ranks 32 and 16 for a few images of the FGVC-Aircraft dataset. This dataset contains aircraft from 70 families, such as Boeing 747, LockheedMartin TriStar L-1011, and Airbus A380. This dataset is ideal because aircraft are a class ImageNet. A pretrained backbone thus able to differentiate an aircraft from other real- world objects, but cannot classify various aircraft families.
17

---

Canonical Rank Adaptation: An Efficient Fine-Tuning Strategy for Vision Transformers
Original
879
379
FONOREEN
CANTAS
Figure 6. Saliency maps depicting the gradient attributions of CaRA fine-tuning ViT-B/16 for aircraft family classification, especially for ranks 32 16.
18


Reviews:
[{"review_id": "3mwlD4EnKD", "reviewer": "Reviewer_Bbm7", "summary": "This paper introduces Canonical Rank Adaptation (CaRA), an efficient fine-tuning strategy for Vision Transformers (ViT). The key finding is that leveraging tensor mathematics can effectively address the high-dimensionality of Multi-Head Attention (MHA), enhancing fine-tuning performance. The main results demonstrate that CaRA outperforms existing methods in visual classification benchmarks such as VTAB-1k and FGVC, while using fewer trainable parameters. The core algorithmic idea is to quantize the Transformer into two tensors, which are used for the MHA projection layer and the feedforward layer respectively, and then fine-tune with low-rank updates in the form of Canonical Polyadic Decomposition (CPD).\n\n## update after rebuttal: \n\nThe rebuttal of the author explained most of the problems. Although this method is novel, its computational weight is not superior, so I think the original score of 3 should be maintained", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful review and for recognising the innovation in our method and finding it meaningful for the vision classification problem. We appreciate your positive feedback. Below are the responses to your review.\n\n***The method has not been fine-tuned and tested on larger models such as ViT-L or ViT-H, so it cannot be proven whether it can maintain high accuracy on these larger models***\n\nFollowing the ViT-L benchmark from VeRA [1], we evaluate our proposed CaRA method against other low-rank fine-tuning methods on four datasets. We use one A100 GPU for fine-tuning. For this experiment, we performed hyperparameter sweeps on lr, scale ($\\alpha$), and various schedulers. For reproducibility, the camera-ready version will include more details of hyperparameters for CaRA, PiSSA, and DoRA, and the code will be available upon acceptance. The rank for LoRA, DoRA, and PiSSA is 8, and the rank for VeRA and CaRA is 256 and 64, respectively.\nNote: The benchmark in [1] only evaluates LoRA and VeRA. We also trained and evaluated PiSSA and DoRA at the request of the other reviewers.\n\nMethod     |ViT-L #Params ($\\downarrow$)|CIFAR100 ($\\uparrow$)|Food101 ($\\uparrow$)|Flowers102 ($\\uparrow$)|Resisc45($\\uparrow$)|Mean ($\\uparrow$)\n|-|-|-|-|-|-|-|\nHead | -  | 79.4|76.5|    98.9     |   67.8     |   80.65\nFull |  303.3M |86.8    |   78.7    |    98.8     |   79.0     |   85.83\nLoRA |  786.4K |87.0    |   79.5    |    99.1     |   78.3     |   85.98\nVeRA | **61.4K** |87.5    |   79.2    |    99.2     |   78.6     |   86.13\nPiSSA |  786.4k |   87.11   |   79.55   |  **99.72**  |   78.55    |   86.24\nDoRA |  860.2K |   87.93   |   81.15   |    99.57    |   80.33    |   87.25\nCaRA (ours) | 75.6K     | **89.36** | **83.65** |    99.63    | **82.43**  | **88.77**\n\nThe table demonstrates that, overall, CaRA significantly outperforms existing baseline methods with only a smaller fraction of trainable parameters ($\\approx 10%$ of LoRA's parameters). In the Flowers102 dataset, PiSSA performs better than CaRA by only a slight margin. Additionally, CaRA achieves state-of-the-art accuracy on the CIFAR100, Food101 and Resisc45 datasets. The accuracy gains combined with CaRA's efficiency demonstrate that CaRA also maintains high accuracy for fine-tuning larger vision transformers.\n\n***When calculating the experimental mean, it is not reasonable to first compute the average accuracy for the three datasets of Natural, Specialized, and Structured, and then calculate the average of these averages.***\n\nWe follow the approach used in prior works [2,3] to maintain consistency with the literature. However, we understand your point and are happy to add one more column to the table with the overall mean. \n\n***This method bears some similarity to the FacT method, and FacT should be introduced in the related work section rather than solely when comparing methods.***\n\nThank you for the suggestion. We introduce and cite FacT in line 123 of the related work section, but we do not explicitly mention the name. We will update the section and introduce FacT by its name in the camera-ready version.\n\n***What is the effect of this method on language models?***\n\nThank you for the suggestion. We would like to emphasise that our study focuses on image classification tasks and we do not make claims regarding language tasks. We already provide additional results for large vision transformers (ViT-L), and we expect that it works for language models as well. Due to the short response time, we cannot provide an evaluation for language models, but we will include them in the paper or supplementary material for the camera-ready version.\n\n***During the fine-tuning process, multiplication turns into a 3-dimensional matrix multiplication. Will the computational load become excessively large?***\n\nAs shown in the table below, CaRA's training time is similar to DoRA and higher than LoRA. However, this increase in computational effort can be mainly attributed to the Python implementation of the Tensorly package. While the implementation makes it easy to use, it is not the most efficient implementation. A more efficient implementation could follow [4]. Nevertheless, the computational load remains small. In terms of memory usage, CaRA performs similarly to FacT.\n\nMethod | Walltime (seconds)($\\downarrow$) | VRAM (GB)($\\downarrow$) |\n|-|-|-|\nLoRA | **165.7560** | **20.1079** |\nDoRA | 204.0761 | 28.0645 |\nFacT-TT | 178.2826  | 20.2464 |\nFacT-TK | 180.5781 | 20.2443 |\nCaRA(ours) | 206.5548 | 21.3740 |\n\nReferences:\n\n[1] Kopiczko, Dawid J., Tijmen Blankevoort, and Yuki M. Asano. \"Vera: Vector-based random matrix adaptation.\" ICLR 2024.\n\n[2] Dosovitskiy, Alexey, et al. \"An image is worth 16x16 words: Transformers for image recognition at scale.\" ICLR 2021.\n\n[3] Jia, Menglin, et al. \"Visual prompt tuning.\" ECCV 2022.\n\n[4] Yang, Zi, Junnan Shan, and Zheng Zhang. \"Hardware-efficient mixed-precision CP tensor decomposition.\" arXiv preprint arXiv:2209.04003 (2022).", "claims_and_evidence": "The proposed claim is supported by derivational proof and citations.", "methods_and_evaluation_criteria": "The proposed method and evaluation criteria are meaningful for visual classification problems. Tensorizing the Transformer and using CPD (Canonical Polyadic Decomposition) for low-rank updates fully consider the high-dimensional characteristics of MHA (Multi-Head Attention), enabling more efficient feature capture.", "theoretical_claims": "The paper provides a detailed explanation of the gradient derivation for CaRA.", "experimental_designs_or_analyses": "1.The method has not been fine-tuned and tested on larger models such as ViT-L or ViT-H, so it cannot be proven whether it can maintain high accuracy on these larger models.\n2.When calculating the experimental mean, it is not reasonable to first compute the average accuracy for the three datasets of Natural, Specialized, and Structured, and then calculate the average of these averages. The correct approach should be to average the results across all 19 datasets. Therefore, the accuracy of the CaRA method should be 74.14%.", "relation_to_broader_scientific_literature": "In terms of tensor representation, existing research on tensor decomposition for fine-tuning was referenced, and based on this, a new tensorization and low-rank update method was proposed, which improves upon the deficiencies of existing methods in dealing with the high-dimensional nature of MHA.", "essential_references_not_discussed": "This method bears some similarity to the FacT method, and FacT should be introduced in the related work section rather than solely when comparing methods.", "other_strengths_and_weaknesses": "Strength:\nThe language is fluent and the article is easy to read.\nThis method is innovative, and the paper provides mathematical derivation for the method.\nWeakness:\nThere are deficiencies in the design of the experiment, which cannot effectively demonstrate the performance of the method.", "other_comments_or_suggestions": "None.", "questions_for_authors": "1. What is the effect of this method on language models?\n2. During the fine-tuning process, multiplication turns into a 3-dimensional matrix multiplication. Will the computational load become excessively large?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GmCLbByPO9", "reviewer": "Reviewer_UspR", "summary": "This paper proposes CaRA, which uses the canonical polyadic decomposition (CPD) to replace the matrix multiplication in LoRA. There are two advantages of using CPD. Firstly, the multi-dimensional formulation can capture the structure of the head-dimension in the projection matrices in multi-head attention (MHA). Secondly, it uses fewer parameters than LoRA for the same rank. The paper also separates the matrices in MHA and those in FFN so that different numbers of decomposition dimensions can be used. CaRA is applied to ViT and tested on VTAB and FGVC benchmarks. The results are slightly better than the best baselines or on par with them.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and for finding our experimentation sound. Below are our responses to the points raised in the review. \n\n***performance baseline marginal***\n\nConsidering SPT-LoRA as the best baseline in both VTAB-1k and FGVC benchmarks, we want to highlight that with only $\\approx 11%$ of SPT-LoRA's parameters, CaRA achieves this performance.\nFurthermore, compared to other tensor-based methods, FacT-TK and FacT-TT, as shown in Figure 4, CaRA consistently achieves $\\approx 1%$ or more accuracy in all three types of VTAB datasets, which is a substantial improvement on these benchmarks. Additionally, the ViT-L experiments presented below further establish CaRA's strong performance and scalability. The gains compared to the other approaches are even larger for ViT-L. Overall, these results demonstrate CaRA's effectiveness in fine-tuning the vision transformer across multiple benchmarks.\n\nMethod|ViT-L #Params ($\\downarrow$)|CIFAR100 ($\\uparrow$)|Food101 ($\\uparrow$)|Flowers102 ($\\uparrow$)|Resisc45($\\uparrow$)|Mean ($\\uparrow$)\n|-|-|-|-|-|-|-|\nHead | -  | 79.4  | 76.5 |98.9|67.8|80.65\nFull |  303.3M | 86.8 |78.7|98.8|79.0|85.83\nLoRA |  786.4K | 87.0|79.5|99.1|78.3|85.98\nVeRA | **61.4K** | 87.5 |79.2|99.2|78.6|86.13\nPiSSA |  786.4k | 87.11|79.55|**99.72**|78.55|86.24\nDoRA |  860.2K | 87.93|81.15|99.57|80.33|87.25\nCaRA (ours) | 75.6K | **89.36** | **83.65** |99.63|**82.43**|**88.77**\n \n***heatmap interpretability***\n\nWe use the integrated gradient maps as a tool to interpret the model's behaviour during fine-tuning, particularly by highlighting the influential image regions the model relies on. \nWhile we acknowledge that these visualisations alone may not provide a complete explanation, they serve as an initial step towards understanding the model's decision-making process.\n\n***comparison to DoRA and PiSSA***\n\nThanks for pointing out. We have included the comparisons with DoRA and PiSSA for ViT-Large training, as detailed in response to Reviewer Bbm7. We will also incorporate these comparisons in the camera-ready version and extend additionally  the discussion in the related work section.\n\n***unclear baseline results in Tab.2***\n\nThe baseline results for LoRA are from FacT [1], where the LoRA rank is set to 8. The results of Adapter and Adaptformer are from RepAdapter [3], and the other results are from their respective papers. For CaRA, we provide the details of the hyperparameters in the supplementary. \nTo enhance clarity, we will update Table 2 to explicitly state the ranks for each method and include the sources of the baseline results.\n\n\n***Tab 4, MHA extra dimension***\n\nTable 4 presents an ablation study on CP-Decomposition across multiple dimensions. When comparing rows 2 and 3 in Table 4, we observe that both the number of parameters decreases and the accuracy increases slightly with d_h as extra dimension. The main gain is in the reduction of parameters. We will clarify this in the camera-ready version.\n\n  \n***NLP domain.***\n\nThank you for the suggestion. We would like to emphasise that our study focuses on image classification tasks and we do not make claims regarding language tasks. We already provide additional results for large vision transformers (ViT-L), and we expect that it works for language models as well. Due to the short response time, we cannot provide an evaluation for language models, but we will include them in the paper or supplementary material for the camera-ready version.\n\n***Why cannot LoRA capture relations across heads?***\n\nLora works with stacks of two-dimensional matrices for the fine-tuning process. By definition, two-dimensional matrix structures and their decompositions model two-dimensional relationships.\nTransformer networks are defined to have the embedding dimension (d_model), the number of heads (n_h) and the dimension of each head (d_h) [1]. If we stack multiple layers, we end up with a four dimensional tensor. Matrix-based structures do not allow us to model tensors directly. Tensor decompositions solve this issue by providing a potentially n-dimensional structure [2]. This paper leverages the Canonical Decomposition, which was designed to handle the tensor data structure that Transformers create naturally.\n\n***Notations to clarify***\n\nThank you for pointing it out, \"[...]\" in equations (3-6) corresponds to stacking, while equation 7 represents concatenations. \nWe followed the exact representation as in [2]. {} represents the set of CP-Decomposition factors in matrix form. We will update the camera-ready version accordingly.\n\nReferences:\n\n1.  Vaswani, Ashish, et al. \"Attention is all you need.\" Advances in neural information processing systems 30 (2017).\n\n2.  Kolda, Tamara G., and Brett W. Bader. \"Tensor decompositions and applications.\" SIAM review 51.3 (2009): 455-500.\n\n3.  Luo, Gen, et al. \"Towards efficient visual adaption via structural re-parameterization.\" arXiv preprint arXiv:2302.08106 (2023).", "claims_and_evidence": "The claim on the CaRA formulation and the parameter efficiency of CaRA is supported. Overall, the performance gain over the baselines is partially supported because the gap between CaRA and the best baselines is marginal.", "methods_and_evaluation_criteria": "Both the method and the evaluation make sense as the paper focuses on parameter-efficient tuning in the vision domain.", "theoretical_claims": "There are no theoretical claims.", "experimental_designs_or_analyses": "The experiments on VTAB and FGVC are sound. The ablation study on changing ranks and dims is also sound. However, the visualization of heatmaps provides limited interpretability and is not convincing.", "relation_to_broader_scientific_literature": "The paper relates to the parameter-efficient fine-tuning of large models. Previous research shows that LoRA is effective, and the idea in this paper provides a similar but alternative formulation using the CP decomposition of tensors.", "essential_references_not_discussed": "The paper did not discuss the improved versions of LoRA, which are highly related. For example, it is helpful to compare with DoRA (ICML'24) and PiSSA (NeurIPS'24).", "other_strengths_and_weaknesses": "Clarity:\n\nIt is unclear how the baseline results in Tab.2 are obtained. Are they trained by the authors or referenced from other papers? Specifically, how much efforts are taken to tune hyper-parameters of baselines. For example, are the learning rate and the alpha value fairly tuned for LoRA? Besides, it would be helpful to readers if the rank of LoRA/CaRA were listed in the table.\n\nSignificance:\n\nFrom the results of Tab.4, it seems that making the heads in MHA an extra dimension does not offer much performance improvement. So, the benefit of exploiting the head dimension is overrated in the introduction.", "other_comments_or_suggestions": "Most LoRA-related papers benchmark on LLMs, such as natural language understanding and natural language generation. It will be more comprehensive if the proposed CaRA is tested on the NLP domain.", "questions_for_authors": "1. L196-199. \"Contrary to the existing works, this formulation ... capture any relations across heads ...\". Why cannot LoRA capture relations across heads?\n2. There are several notations to clarify. Firstly, what does the $[...]$ mean in Equ. (3-7)? concatenation/stacking? Secondly, what does the {$... ; ...$} mean in Equ. (8)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5izu8zaZQP", "reviewer": "Reviewer_dZjR", "summary": "This paper introduces Canonical Rank Adaptation (CaRA), a peft method method specifically designed for ViTs. The core idea of CaRA is to tensorise transformer weights across layers and to directly optimize the stack using a Canonical-Polyadic Decomposition.\nThe authors report minimized trainable parameters and match or outperforms existing PEFT methods with lower parameter counts training a ViT-B/16 on the VTAB-1k and FGVC visual classification benchmarks.\nExperimental results on these benchmarks and ablation studies are presented to support these claims.\n\n## update after rebuttal\n\nThe authors have address most of my concern. \nThe increased wall time and memory of CaRA is a limitation of the method in its current form so I ask the authors to clearly include these findings in the camera ready  \nThe approach remains novel and interesting for future research so I maintain my original score", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review and for recognising CaRA's relevance in PEFT methods. We appreciate your insights on broader evaluation and efficiency comparisons. We address the questions below in detail.\n\n***The proposed evaluation makes sense although I would have expected more experimental results with larger vision architectures (e.g. ViT-L/14 and above).***\n\nWe performed experiments on ViT-L pretrained on ImageNet-21k. Please refer to the Bbm7 reviewer's rebuttal response for the results.\n\n\n***Results for language tasks such as commonsense [1] could also have better supported the evaluation claims.***\n\nThank you for the suggestion. We would like to emphasise that our study focuses on image classification tasks and we do not make claims regarding language tasks. We already provide additional results for large vision transformers (ViT-L), and we expect that it works for language models as well. Due to the short time for the response, we cannot provide an evaluation for language models, but we will include them in the paper or supplementary material for the camera-ready version.\n\n\n***Related work include VeRA [3] or NoLA [4].***\n\nThank you for pointing out these works. We will ensure that VeRA and NoLA are included in the related work section in the camera-ready version. Additionally, we provide comparisons to the VeRA benchmark on ViT-Large in our experiments.\n\n***Figure 1 is no very helpful in understanding the algorithm and should be improved,  Figure 3 is better but there is too much W definitions which makes very loaded.***\n\nThank you for the comment. Figure 1 only presents the performance of CaRA. We assume the comment refers to Figure 2; we will rework Figure 2 to make it more explainable.\nWe appreciate your feedback on Figure 3. We recognise that \"W\" definitions could be integrated directly into the figure. We will reorganise the figure to make it more intuitive and improve the layout for the camera-ready version.\n\n***measure wall clock time and memory***\n\nAs suggested, we present the wall time and VRAM allocated for various fine-tuning methods on ViT-L trained on CIFAR100 for 10 epochs.\nWe observe that LoRA is the most efficient in terms of training time and memory. We attribute this speed to CUDA-optimised matrix multiplications in PyTorch.\nIn contrast, CaRA shows a higher walltime because, just like the Tensorly [3] package we rely on, it is largely written in Python.  While the implementation makes it easy to use, it is not the most efficient implementation. We expect significant improvements in speed if CaRA's operations are optimized. Also, CaRA's multi-dimensional tensor nature results in slightly higher VRAM allocation.\nGiven the added representational capability and the implementation, this behaviour for CaRA is to be expected. In spite of slightly higher wall time and memory, CaRA show notable performance improvements in both ViT-B and ViT-L architectures. Given that fine-tuning CaRA is often a one-time cost, we believe this is a reasonable tradeoff.\nIn the case of FacT methods, we notice that they require lower ranks to match CaRA's parameter count. FacT achieves lower accuracy, with 88.4 (TK) and 87.96 (TT), while CaRA achieve 89.36.\nInterestingly, the DoRA (matrix-based) method exhibits higher memory usage and wall time, which we attribute to extra weight normalisation applied in each forward pass.\nWe are still working on the experiments regarding SPT-LoRA.\n\nMethod| Walltime (seconds)($\\downarrow$)| VRAM (GB)($\\downarrow$) |\n|-|-|-|\nLoRA|**165.7560**|**20.1079**|\nDoRA|204.0761|28.0645|\nFacT-TT|178.2826|20.2464|\nFacT-TK|180.5781|20.2443|\nCaRA(ours)|206.5548|21.3740|\n\n\n***limitations of CaRA and future directions***\n\nThis study currently focuses on vision transformers. We do not present results for other tasks, like language processing, at the moment. We expect that it works for language models as well, and we will include results for language models in the paper or supplementary material of the camera-ready version. Currently, matrix multiplications are more optimized than tensor decompositions. Using a hardware-optimized tensor decomposition will be an interesting future direction. DoRA [2] introduces a weight normalisation. A normalised form of the CP-Decomposition could further boost performance and reduce training time. We will add a discussion of limitations and future directions. \n\nReferences:\n\n[1] Kolda, Tamara G., and Brett W. Bader. \"Tensor decompositions and applications.\" SIAM review 51.3 (2009): 455-500.\n\n[2] Liu, Shih-Yang, et al. \"Dora: Weight-decomposed low-rank adaptation.\" Forty-first International Conference on Machine Learning. 2024.\n\n[3] Jean Kossaifi, Yannis Panagakis, Anima Anandkumar and Maja Pantic, TensorLy: Tensor Learning in Python, Journal of Machine Learning Research (JMLR), 2019, volume 20, number 26.", "claims_and_evidence": "The claims of parameter efficiency and good performance are clearly supported in the experiments.", "methods_and_evaluation_criteria": "The proposed evaluation makes sense although I would have expected more experimental results with larger vision architectures (e.g. ViT-L/14 and above). Results for language tasks such as commonsense [1] could also have better supported the evaluation claims.\n\n[1] Hu, Zhiqiang, et al. \"Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models.\"  EMNLP 2023.", "theoretical_claims": "Gradient derivations are provided but I did not check the correctness in depth.", "experimental_designs_or_analyses": "The proposed evaluation makes sense although I would have expected more experimental results with larger vision architectures (e.g. ViT-L/14 and above). Results for language tasks such as commonsense could also have better supported the evaluation claims.", "relation_to_broader_scientific_literature": "This paper is relevant to literature looking at reducing the number of parameters beyond LoRA's own reduction.\n\nCaRA is relevant in this setting as it provide yet another alternative that appears to be competitive in terms of performance and explores new factorization ideas for PEFT. Setting ranks as global to the whole architecture and not to specific layers is also relevant as it allows for an extension to other rank-adaptive methods such as AdaLoRA [2].\n\n[2] Zhang, Qingru, et al. \"Adalora: Adaptive budget allocation for parameter-efficient fine-tuning.\" ICLR 2023", "essential_references_not_discussed": "A few other algorithms tackle reducing the amount of trainable parameters in LoRA. The contributions are quite orthogonal to the Canonical-Polyadic Decomposition. Examples that could be added to the related work include VeRA [3] or NoLA [4].\n\n[3] Kopiczko, Dawid J., Tijmen Blankevoort, and Yuki M. Asano. \"Vera: Vector-based random matrix adaptation.\" ICLR 2024\n[4] Koohpayegani, Soroush Abbasi, et al. \"Nola: Compressing lora using linear combination of random basis.\" ICLR 2024", "other_strengths_and_weaknesses": "The idea of applying Canonical-Polyadic Decomposition as a parameter efficient algorithm is interesting especially as it considers the weights of the network as a whole.\n\nFigure 1 is no very helpful in understanding the algorithm and should be improved, Figure 3 is better but there is too much $\\mathbf{W}$ definitions which makes very loaded.\n\nThe experimental setting is limited to vision and one (small) network architecture. I would have liked to see experiments with CLIP or on language benchmarks to better substantiate the results and maybe get more insights as to what CaRA does differently to LoRA.\n\nMost importantly there lacks a study of the training time of CaRA compared to alternatives (especially LoRA, SPT-LORA and FacT-TT/TK) in terms of GPU hours or wall-clock time. This would be helpful to understand the practical efficiency of CaRA beyond just parameter count.\n\nThere is no section addressing the limitations of CaRA and recommendations for future work", "other_comments_or_suggestions": "No other comment", "questions_for_authors": "Did the authors perform a wall-time study of training time for CaRA compared to other PEFT ? I would also be interested in whether CaRA requires more VRAM to train that the alternatives.\n\nWhat are limitations of CaRA or future directions for research into PEFT with Canonical-Polyadic Decomposition ?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received three reviews, with overall recommendations leaning toward weak acceptance. Reviewers raised several initial concerns, including: deficiencies in experimental design, unclear baseline results in Table 2, ambiguity around the extra dimension in MHA as shown in Table 4, lack of clarity in Figures 1 and 3, absence of experiments with CLIP or VLLM, insufficient analysis of CaRA’s training time, and limited discussion of CaRA’s potential limitations. The rebuttal addressed many of these issues, at least partially, and some improvements were committed to be included in the final version.

The AC agrees with the reviewers’ assessments and appreciates the paper’s innovative approach and solid mathematical formulation. Additionally, the experimental results, though still with room for improvement, provide important evidence of the method’s effectiveness.

The AC leans toward recommending acceptance, contingent on the authors fully addressing the outstanding concerns and incorporating all committed revisions and clarifications into the camera-ready version.
Decision: Accept (poster)