Efficient Adaptation of Pre-trained Vision Transformer via Householder Transformation
Authors: Wei Dong
Yuan Sun
Yiting Yang
Xing Zhang
Zhijun Lin
Qingsen Yan
Haokui Zhang
Peng Wang
Yang Yang
Heng Tao Shen
Venue: NeurIPS 2024 poster (2024)
Keywords: Parameter-efficient fine-tuning
vision transformer
householder matrix
singular value decomposition
Primary area: optimization_for_deep_networks

Abstract: A common strategy for Parameter-Efficient Fine-Tuning (PEFT) of pre-trained Vision Transformers (ViTs) involves adapting the model to downstream tasks by learning a low-rank adaptation matrix. This matrix is decomposed into a product of down-projection and up-projection matrices, with the bottleneck dimensionality being crucial for reducing the number of learnable parameters, as exemplified by prevalent methods like LoRA and Adapter. However, these low-rank strategies typically employ a fixed bottleneck dimensionality, which limits their flexibility in handling layer-wise variations. To address this limitation, we propose a novel PEFT approach inspired by Singular Value Decomposition (SVD) for representing the adaptation matrix. SVD decomposes a matrix into the product of a left unitary matrix, a diagonal matrix of scaling values, and a right unitary matrix. We utilize Householder transformations to construct orthogonal matrices that efficiently mimic the unitary matrices, requiring only a vector. The diagonal values are learned in a layer-wise manner, allowing them to flexibly capture the unique properties of each layer. This approach enables the generation of adaptation matrices with varying ranks across different layers, providing greater flexibility in adapting pre-trained models. Experiments on standard downstream vision tasks demonstrate that our method achieves promising fine-tuning performance.

Efficient Adaptation of Pre-trained Vision Transformer via Householder Transformation
Wei Dong', Yuan Sun', Yiting Yang', Xing Zhang', Zhijun Lin?, Qingsen Yan?,
Haokui Zhang', Peng Wang", Yang Yang", Hengtao Shen 34
Coollege of Information and Control Engineering, Xi' an University of Architecture and Technology
School Computer Science, Northwestern Polytechnical University Schooo of Computer Science and Engineering, University of Electronic Science and Technology of China School of Computer Science and Technology. Tongji University
Abstract
A common strategy for Craameter-Effiieet Fine- TuninT (PEFT) of pre-trained Vision Transformers (ViTs) involves adapting the model to downstream tasks by learning low-rank adaptation matrix. This matrix is decomposed into product down-projection and up-projection matrices, with the bottleneck dimensionality being crucial for reducing the number of learnable parameters, exemplified by prevalent methods like LoRA and Adapter. However, these low-rank strategies typically employ fixed bottleneck dimensionality, which limits their flexibility in handling miser-wise variations. To address this limitation, we propose a novel PEFT approach inspired by Singular Value Decomposition (SVD) for representing the adaptation matrix. SVD decomposes a matrix into the product a left unitary matrix, diagonal matrix of scaling values, and right unitary matrix. We utilize Householder transformations to construct orthogonal matrices that efficiently mimic the unitary matrices, requiring only vector. The diagonal values are learned in a layer-wise manner, allowing them to flexibly capture the unique properties of each layer. This approach enables the generation of adaptation matrices with varying ranks across different layers, providing greater flexibility in adapting pre-trained models. Experiments on standard downstream vision tasks demonstrate that our method achieves promising fine-tuning performance.
1 Introduction
Earameter-Efficient Fine- Tine-Tuning (PEFT) for pre-trained Vision Transformers (ViTs) aims to adapt these models to downstream tasks by learning a small set of parameters while keeping most or all of the original model parameters frozen. This approach is expected to reduce the cost of fine-tuning and potentially improve the model's generalization performance, particularly when the downstream task involves limited data.
common strategy for adapting the parameters to learn an adaptation matrix that modifies the original matrix through addition multiplication. To reduce the parameter scale of the adaptation matrix, a low-rank strategy is typically employed. This involves decomposing the adaptation matrix into the product of Nown-projection matrix and up- pp-projection matrix, where the bottleneck dimensionality determines the parameter scale. Many prevailing PEFT solutions (T33] follow this approach. However, these methods usually set the bottleneck dimensionality empirically balance adaptation performance and parameter size. The fixed dimensionality, however, lacks the flexibility accommodate variations in layer-wise properties.
"Corresponding author. Email address: www.wa@@otmaiil.commmm com
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

In this work, we propose a novel erffiittr-efficient adaptation method to fine-tune pre-trained ViTs. Our design of the adaptation matrix is inspired by Singular Value Decomposition (SVD), which decomposes a matrix into a product of a left unitary matrix, a diagonal matrix, and right unitary matrix. In SVD, the unitary matrices consist of orthogonal vectors, and the diagonal matrix of singular values essentially determines the rank of the matrix. Inspired by SVD, we propose to use Householder transformations to replace the left and right unitary matrices. Householder transformations maintain orthogonality properties similar to unitary matrices but can be derived simply by a vector, making them parameter efficient. With left and right Householder matrices, we learn the diagonal matrix adaptively for each layer accommodate layer- wise properties. This approach, termed the Householder Transformation- based Adaptor (HTA), enables us to derive the adaptation matrix in a -fffeiiee-e manner while theoretically allowing for varying ranks for the adaptation matrices, thus achieving a better balance between parameter efficiency and adaptation performance.
We conducted experiments on a set of downstream vision classification tasks. The results show that our method can be effectively applied to various ViT versions, achieving promising fine-tuning performance. In summary, the contributions of this work can be summarized as follows:
We approach PEFT from novel angle by viewing the adaptation matrix from the perspective of SVD, which inspires us to propose Householder seafformation-based adaptation strategy that is parameter-efficient
By learning scaling coefficients to compose Householder transformation matrices together into adaptation matrices, our method can theoretically allow varying adaptation matrix ranks to accommodate layer-wise variations.
Experiments on two sets of downstream vision classification tasks reveal our method can achieve an appealing balance between adaptation performance and parameter efficiency.
2 Related Work
2.1 Pre-training and Transfer Learning
As advanced learning strategy, extensive research 7 has demonstrated the wide applicability of transfer learning across various fields. Especially in cases where the target task has limited data, high labeling costs, or poor data quality [8H10], transfer learning significantly enhances model generalization and training efficiency. By pre-training on large-scale datasets and using the obtained parameters initialization for downstream tasks, transfer learning can effectively transfer and apply the knowledge of pre-trained models. In this process, the performance and convergence speed of downstream tasks are highly correlated with the dataset used for pre-training the model. In the field of computer vision, pre-training on large-scale datasets such as ImageNet ([I) has significantly improved the performance of tasks like image classification [22 object detection [16] [7],, and semantic
segmentation [8 19.. Moreover, self-supervised pre-training (20 21) leverages the advantage of not requiring large amounts of labeled data, expanding the data scale and enhancing feature extraction capabilities, thereby further improving the generalization ability and robustness of pre-trained models. However, due to the substantial computational resources required to fully fine-tune the parameters of pre-trained models in downstream tasks, current research has shifted towards exploring more efficient fine-tuning methods.
2.2 Crrameter-Efficiiee Fine- Fine-Tuning (PEFT)
Compared to full fine-tuning, the PEFT methods 22 2 26 aim to reduce the high cost of fine-tuning by freezing the majority of parameters in the pre-trained model and introducing a small number of learnable parameters to adapt to specific downstream tasks.
With the development of large pre-trained models, various PEFT approaches have emerged. Adapter (22) inserts a bottleneck-struetureg adapter layer into the pre-trained model and refines the model by updating only the parameters within the adapter layer. Bias [23 focuses on the fine- tuning of models for specific downstream tasks by meticulously calibrating the bias terms. VPT (24 integrates the concept of prompt learning into visual tasks, thereby facilitating targeted optimization for specific downstream tasks. SSF (25] efficiently fine-tunes the weights in pre-trained models

---

through affine transformations composed of scaling and shifting operations. AdaptFormer [2] ex- plores a parallel adapter solution on ViT for various downstream tasks. FacT 226 decomposes the weights of VIT' into individual three-dimensional tensors and further decomposes the increments into lightweight factors. During fine-tuning phase, these factors are updated and stored, effectively reducing computational overhead.
2.3
LoRA and its variants
As represented by LoRA [[], the core of this type of PEFT method is the utilization of low-rank matrices to approximate weight adjustments during the fine-tuning phase. By employing reparame- terization techniques, these low-rank matrices combined with the existing parameter matrices, thereby circumventing extra inference costs. Ada [27] employs singular value decomposi- tion decompose weight matrices, pruning insignificant singular values to effectively reduce the number of parameters. ARC [3] uses symmetric up- no-down projections to create cross-layer shared bottleneck operations. By learning low-dimensional rescaling coefficients, effectively recombines layer-adaptive adapters, reducing the costs of fine-tuning, FedPara (28) reparameterizes model layers with low-rank matrices and uses the Hadamard product. This approach, unconstrained by low-rank limitations, offers greater capacity and reduces learning costs. RLRR (29] examines mainstream PEFT methods from the perspective of SVD decomposition, exploring the critical balance between preserving generalization in pre-trained models and adapting them to downstream tasks. Our re- search abandons the traditional fixed-rank approach, opting instead for a more flexible adjustment of parameter matrices using small number of learnable parameters.
3 Methodology
In this section, we commence with an introduction of the notations, symbols, and contextual back- ground related to low-rank adaptations and Householder transformation. Then we present the decomposed structure low-rank adaptation and discuss its inherent operating mechanism from the perspective of singular value decomposition. Finally, we propose novel adaptation via Householder transformation. This adaptation primarily aims to construct the Householder unitary matrices via learnable weight vector, thereby trading-off the fully spanned representation space and the affordable parameter size.
3.1 Preliminary knowledge
Low-rank adaptation. Pre-trained ViT models are typically initialized with weights learned from large-scale image datasets, such ImageNet. The pre-training process involves optimizing the model on an unsupervised or supervised pretext task. The resulting pre-trained weights encode rich semantic information that can be transferred one of the most representative methods of fine-tuning, PEFT method achieves excellent results on downstream tasks by merely utilizing a small number of additional learnable parameters to fine-tune the ViT. The most prevalent PEFT is the adaptation method, which can be divided into two categories: LoRA- based and adapter- blasee methods. In general, LoRA-based method is defined as:
wide range of downstream tasks through fine-tuning. As
x ((-1) FT
x (1 (1 (W() + w() W down up + 5
where X (1-1) (w!") $
is the fine-tuning output, Wil) w! w(( will)
any linear weight projection FC1 F in ViT, b is the bias weights, (( E E x and
wil DD' are down- and up-adapting projection matrices across varying layers with the dimensionality D' & D0), The detailed framework of L-RA-based method is shown in Fig. Analogously, adapter-based method is described as:
x ((-1) Act(MHH((X W MHA(U) (MA(A))
(2)
X FT Act((FFNN (1-1) w down
FFN())
W FFN())
with the activation function (-). Multi-Head Attention (MHA), and Feed-Forward Network (FFN) modules in VIT. The detail of adapter-based method is shown in Fig. [ (c) By observing
3

---

Trainable Weights:
Frozen Weights:
Output
Output
Householder Vector
Up
Right Householder
Projection
Up Projection
W eight Matrix
Diagonal Parameters
I-Dimension Bottleneck
Weight Matrix
Down Projection
Down Projection
Left Householder
Householder
Input
Vector
Input
(a) LoRA-based Method
(b) LoRA-based Method with ITT
Output
Output
Householder
Vector
Up Projection
Right
Householder
Up Projection
Diagonal
I-Dimension Bottleneck
ReL U
Parameters
NILL
Down
Down Projection
Left Houscholder
Projection
Houscholder Vector
Input
Input
Adapter-based Method
(d) Adapter-based Method with HTA
Figure 1: Underpinned by (a) LoRA [I] and (c) Adapter [22). we utilize Householder matrix to
construct Householder transformation-aatt based adaptations, involving (b) LoRA-based method with HTA and (d) Adapter-baat based method with HTA.
Eq. and 2). both above-mentions PEFT methods involve a low-rank bottleneck structure, adaptation w down w (!) up- Note that we remove the activation in the low-rank bottleneck because the presence or absence of such activation does not affect the low-rank structure of adaptation.
i.e., W (1)
Householder transformation. Houscholder transformation, or Houscholder reflection, is linear transformation that reflects vector across hyperplane defined by Houscholder vector. It is charac- terized by Houscholder matrix, which is an orthogonal and symmetrical matrix with determinant -1. This transformation, initially proposed by A.S. Householder in 1958 (30], has significant applications numerical linear algebra (31), particularly in QR decomposition 32), where is used to transform matrix into upper triangular or Hessenberg form [33]. Householder transformation be employed to set specific elements of vector to zero while preserving its norm, making it valuable tool for matrix orthogonalization and symmetrization. The Householder matrix is defined as following:
also
=
06
with the identity matrix I and the Householder vector Ø.
3.2 Viewing the adaptation matrix from SVD
Singular Value Decomposition (SVD) offers a profound insight into matrix factorization. It breaks down matrix into three constituent matrices. Viewing the adaptation matrix through the lens of SVD, we represent as:
W adeptation
w-( D W left
(4)
right!

---

where w left E DDD xD() is the left unitary matrix and W right E xDD is the right unitary matrix; and D E x is the diagonal matrix in which the diagonal elements of diagonal matrix are singular values. Unitary matrices w left and w essentially characterize the rotation right
transformations in a linear space. The left unitary matrix W left rotates an arbitrary vector multiplied by the adaptation matrix wi(h addaptation into the space it spans. Then, the vector scaled by the diagonal matrix D Finally, the right unitary matrix w(()
right rotates the vector back the original linear space. Therefore, the SVD decomposition characterizes the transformations of rotation and scaling in the linear space.
When it comes to the fine-tuning strategy, PEFT methods employ ViT the backbone and essentially fine-tune the learnable parameters of unitary matrices w((
left ( x and w right E PR''' D'x
and the learnable singular values of diagonal matrix D R to downstream tasks, implicitly performing the rotation and scaling transformations. Note that the number of non-zero singular values in matrix ( does not exceed its dimensionality D'. However, the fixed bottleneck dimensionality D' empirically set to LoRA- or adapter-based methods is inflexible, thereby without accommodating variations in layer-wise properties. This implies that the linear space spanned by the low-rank adaptation matrix and its corresponding number of non-zero singular values is constrained within low dimensionality D' Increasing the dimensionality D' could effectively enhance the space capacity of the adaptation matrix, thereby improving the performance potential of the fine-tuned VIT model. However, this also further increases the number of parameters in the PEFT method.
3.3 Householder transformation-based adaptation
To address the aforementioned issue, we introduce Householder transformation into the adaptation matrix learning, and propose the Houscholder transformation-basee Adaptor (HTA). Following this
way, HTA facilitates the derivation of the adaptation matrix in a manner that is parameter ceaamtter-ffficinn, while theoretically accommodating the flexibility of varying ranks for the adaptation matrices.
In our approach, we respectively employ the Householder matrices H R (D ( and H(!)
right
R > as substitutes for the left and right unitary matrices (-()) (t)
DDDxD and w())
left
right
DDD(( within the adaptation matrix w
adeptation to form the HTA adaptation matrix Wil)
HTAA
(1) H left
HTA
right (I t (1) T
(5)
DD (I
(1)
right 0 ~00T rrihtt !),
inft lnft
with two learnable parameter vectors jeb E DDD and (())
right E D and learnable diagonal
parameter vector ( E D in the diagonal matrix D E RDD")x(0))
Since the Householder transformation matrix is derived from a single vector, transformation capacity can be limited and sensitive to the vector learned to derive it. To enhance the robustness of the adaptation matrix, we incorporate an additional low-rank adaptation matrix, resulting in the ultimate HTA. Building this design, we can derive the LoRA alternative as follows:
x(-"" x(1-1) (W()) ( ( down up + ()(()
(()
HTA) +
(6)
=x (1-11) (w-())
(!l) (())
down up
+ (I
(11) (T)
left U
left
D
(0) Friht J ((OTT)
) + +5 (0T
where (
down E C and (
down B D unless otherwise stated, The HTA structure the (b)
LoRA alternative is shown in Fig.
5

---

Analogously, we can derive HTA alternative to the adapter-based method (as shown in Fig. (d) follows:
as
1-1) PT
-MHA (X (1) MHA(O
W
MHA(U
+ W M
down
HTA
-MHA(X
MHAU down
MHA
(I
left
(HH((1) MHA(TT
left
DH
MHA
I
MHA T
right
0 right
X FT FFF (1-1 (111) FFN FT =FFN(X (1-1) FFNU
FFN()
W FPN())
down
HTA
FFN
FT
down
[I - Iof left
PPFN(
T
D
PFN
H
(I
right
PFNCT FFN()
rright
By observing Eq. 66 we can that LoRA-based method with HTA could be re-parameterized to the form W param = w
HTA during the model inference stage. And also, the HTA of MHA in Eq. 7 is available due to the fact that the weight matrix w!!) is positioned at the end of MHA. Similarly, the re-parameterization of FFN
rre-mamm
down ( + W
re-parameterization WMHA
re-param ! w!ll
is WFFN = w w!l)
due to the weight matrix w(O at the tail of FFN.
re- paramaaamm
FC2 HTA
Experiments
In this section, we present the experimental settings, comparison to existing solutions, and ablation studies to unveil the key properties of the proposed method.
4.1 Experimental settings
Datasets. We evaluated the effectiveness of method using sets of visual adaptation bench- marks: FGVC and VTAB-IK, involving a total of 24 datasets. The FGVC collection consists of five Fine-Grained Visual Classification (FGVC) datasets: CUB-200-2011, 1, NABirds, Oxford Flowers, Stanford Dogs, and Stanford Cars. These datasets focus on distinguishing between visually similar subcategories within broader category, making the task more challenging and detailed. The VTABB Ik benchmark comprises 19 diverse visual classification tasks, divided into three categories: Natural, which includes images captured by standard cameras: Specialized, which includes images captured by specialized equipment such as remote sensing and medical imaging devices; and Structured, which includes synthesized images from simulated environments, such as object counting and 3D depth prediction. Each VTAB- 1 task includes 1,000 training samples.
Pre-trained backbones. We employ ViT [33 and Swin Transformer [1] as backbone architectures evaluate our proposed approach. To demonstrate the versatility of the proposed HTA model, we uti- lize two variants of ViT: ViT-Base and ViT-Large. These models are pre- pre-trained on the ImageNet21K dataset [IT]. Additionally, to ensure a fair comparison, follow the settings from previous work (29
and conduct separate experiments using VIT backbone enhanced with AugReg (3).
Baselines and existing PEFT methods. In our comparative analysis, we evaluate the performance of our HTA against two baselines and several specified, our HTA follows the design in Eq. (6), with the dimension of the low-rank adaptation matrix set to The two baselines we consider are: (1) Full Fine-tuning: This baseline involves updating all the parameters of the pre-trained model using the training data of the downstream task. (2) Linear Probing: This baseline focuses on learning linear classification head on the
PEFT methods. Unless otherwise
downstream task while keeping the remaining parameters of the pre-trained model frozen. In addition to the baselines, we compare our method with the following state-of-the-an solutions: Adapter [22]. Bias [23] LoRA (1. VPT (24), AdaptFormer FacT 26), ARC and RLRR 229 The results are presented Table T and Table 2

---

Table 1: Performance comparisons the VTAB-1k benchmark with ViT-B/16 models pre-trained on ImageNet-2IK. denotes leveraging the augmented ViT backbone by AugReg (4). The bold font shows the best accuracy of all methods and the underline font shows the second best accuracy.
Natural
Speciallod
Stricted
Mithads
Calechi3
Clern
Paemmo MM
TNK
Meau
Mann
SNORD
Mon
Full tasiss
08.9
67.7
66.3
07.
$6.9
87.4
15.8
75.9
79.7
05.7
66.3
73.9
$5.4
56.3
58.6
4.77
66.5
57.5
46.7
25.7
25-1
47.6
66.5
85:00
Laver pultes
56.3
08.9
17.2
25:1
VPT Stullo 23
72.
85.9
02.6
07.5
57.3
14.5
51.2
76.
79.2
92.0
75.6
72.0
79.7
50.5
59.6
455
67.1
68.7
16.1
30.2
34.1
47.0
66.9
0.11
V9
89
78.
0338
66.1
08.0
58.3
79.1
496
78.5
81.8
96.1
83.4
08.6
82.4
68.5
06.0
45.5
72-
736
41.9
32.9
31.8
550
09.4
0.60
LOKA
91.4
864
96.
340
19.5
95.5
16.4
168
52.
09.1
75.7
443
393
1639
Adasttmmet
@
11
91.2
1103
W1.
1619
06.6
563
91.
13.0
933
1644
76.3
16.9
$3.4
64.1
76.3
65.3
51.1
41.1
X83
12.3
936
1100
8627
85h
26.1
16-3
13.1
124
1011
414
410
734
161
055
RLRR 10
75.6
92.4
723
99.
91.5
595
57A
82.7
56.8
153
15.
15.7
66.2
539
RXT
33.
E37
33.4
436
61.
76.
0.33
76.6
94.3
73.5
993
91.3
86
16.
87.
057
850
737
86B
12.6
631
93.5
51.0
865
13.6
365
413
62.
747
0.23
634
96.5
57.0
546
76.1
RLRIE
06.
91.7
16.3
996
92.
91.
360
K3.
87.8
95.2
191
16.3
K2.3
904
63.3
545
X30
13.3
52.0
41-7
61.5
15.1
1131
92.8
77.6
22.4
35.1
83.7
95.1
S7.
54.2
51.
95.1
53.
41.9
Table 2: Performance comparisons five FGVC datasets with ViT-B/16 models pre-trained on
ImageNet-2IK. * denotes leveraging the augmented ViT backbone by AugReg 33
Methods
Datasets
CUB-200-2011
NABirds
Oxford Flowers
Stauford Dogs
Stanford Cars
Meaan Total
Params. (M)
fine-tuning
82.7
9%.8
89.4
84.5
N8.5
85.98
Lincar probing
85.3
75.9
97.9
86.2
$1.3
79.3
0.18
Adapter
698.
Bias 23
$3.4
84.2
98.8
91.2
79.1
88.4
0.28
Shallo W
$6.7
78.8
98.4
90.7
68.7
84.6
0.25
VPT-Deep
88.5
84.2
90.2
83.6
LoRA
83.3
85.6
99.2
91.0
83.2
89.5
0.44
ARC 3
$8.5
85,3
99.3
91.9
85.7
90.1
0.25
RIRR 29
89.3
84.7
99.5
92.0
87.0
90.4
0.47
HTA
53.8
84.4
99.5
92.3
87.9
90.6
0.36
SSF 25
89.5
M5.7
93.6
89.6
89.2
90.7
0.39
89.5
RLRR* 29
39.8
85.3
99.6
90.0
90.4
91.0
0.47
HTA"
90.5
85.4
99.6
89.3
90.5
91.1
0.36
Implementation details. Following previous work, we employed data augmentation during the training phase. For the FGVC datasets, we processed the images with a random resize crop to 224 x 224 and applied random horizontal flip for data augmentation. For the VTAB-Ik datasets, we directly resized the images 224 224, adhering to the default settings in VTAB-IK. We used the Adam (35) optimizer to fine-tune the models for 100 epochs. The learning rate schedule was managed using the cosine decay strategy. All experiments are conducted using the PyTorch framework (3) on an NVIDIA A800 GPU with 80 GB of memory.
4.2 Experimental comparisons
In this section, we conduct a comprehensive comparison of our method with other state-of- of-the-art approaches using different benchmarks and backbones. We evaluate the classification accuracy of each method across various downstream tasks and examine the number of trainable parameters during the fine-tuning phase.
Comparison with the existing PEFT methods. We conducted comparison of our method with other PEFT methods and baselines using two different benchmarks: FGVC and VTAB-1k. The results are presented in Table D and Table 2 On the VTAB- 1k dataset, our method not only demonstrates strong competitiveness compared to the baselines but also shows advantages over current state-of-thr
art methods. On many of the datasets, our method achieves the best performance with a reasonable parameter count. Compared to the previous state-of-the-art method, RLRR (29). our method achieves superior overall performance while reducing the number of parameters by one-third. When using the AugReg-enhanced model, our lead is further amplified. In Table [ we further compare our method with others on the FGVC benchmark. While our method also achieves appealing performance on this dataset, the advantage is less evident. This is due to the fact that very high performance has already been achieved on this dataset, and the performance improvements have nearly saturated.
7

---

Table Performance comparison on VTAB- using ViT-Large pre-trained on ImageNet-21k the backbone. Detailed results are presented in the Appendix.
Datasets
Natural (7)
Specialized (4)
Structed (8)
Mean Total
Params.(M)
Methods
Full fine-tuning
74.7
83.8
48.1
65.4
303.40
Linear probing
70.9
69.1
25.8
51.5
0.05
Bias 24
70.5
73.8
41.2
58.9
0.32
VPT-Shallow
78.7
79.9
40.6
62.9
0.15
VPT-Deep
82.5
83,9
54.1
70.8
0.49
LoRA [
81,4
85.0
57.3
72.0
0.74
SSF [25
81.9
85.2
59.0
73.0
0.60
ARC
82.3
85.6
57.3
72.5
0.18
RLRR 29
83.9
86.4
61.9
75.2
0.82
HTA
84.1
86.6
62.3
75.4
0.54
Table 4: Performance comparison on VTAB-I k using Swin Transformer pre-trained on ImageNet-21k the backbone. Detailed results presented in the Appendix.
Datasets
Natural
Specialized (4)
Structed (8)
Mean Total
Params.(M)
Methods
Full fine-tuming
79.1
59.7
72.4
86.80
Linear probing
73.5
80.8
33.5
58.2
0.05
ML.P-
24
70.6
80.7
31.2
57.7
4.04
Partial 24
73.1
81.7
35.0
58.9
12.65
Bias 23
74.2
80.1
42.4
62.1
0.25
VPT-Shallow
79.9
82.5
37.8
62.9
0.05
VPT-Deep
76.8
84.5
53.4
67.7
0.22
ARC 3
79.0
86.6
59.9
72.6
0.27
RLRR 29
81.3
86.7
59.0
73.0
0.41
HTA
81.8
86.7
61.3
74.2
0.23
Experiments larger-scale VIT backbone. In addition to using the ViT-B backbone, also employed the ViT-L backbone, which has a deeper block structure, to validate the scalability and generalizability of our method. The comparison results shown Table 3 Our method achieves the best performance among all the compared methods while maintaining a reasonable parameter count. These results demonstrate that our method can effectively adapt models of varying scales and complexities in an efficient manner.
Experiments hierarchical Vision Transformers. To further validate the effectiveness of our method, we tested it on the Swin Transformer [14]. The Swin Transformer is renowned for its hierarchical design, consisting of multiple stages, each with transformer blocks that maintain consistent feature dimensions unique to that stage. As shown in Table 4 our method notably outperforms existing state-of-the-ar methods across various downstream tasks, with overall improvement of I. 1.2% over the previous best performance while using only half of the parameters.
4.3
Ablation studies
To gain deeper insights into the proposed method, we conducted comprehensive ablation studies to elucidate its critical features and carry out pertinent analyses.
Using HTA as alternative to low-rank based adaptation matrix. As mentioned earlier, our proposed HTA model offers more flexible adaptation capacity compared to other low-rank based adaptation matrices.
validate this claim, we replaced the adaptation matrices of LoRA [I]
and Adaptor 22 with HTA. Initially, following FacT (26), LoRA I was originally applied to the {W,r 9. W.) projection matrices the multi-head attention operation of each ViT layer, while Adapter was applied to the forward neural network components layer-wise, described in Eq. 2 For

---

Table 5: Ablation study on using HTA as alternative to the low-rank based adaptation matrices in LoRA and Adapter on VTAB-IK. lk. Following the configurations in FacT [26], LoRA and Adapter are applied to (Wa W.) and FWF,,, W FC2 projection matrices, separately.
Datasets
Natural (7)
Specialized (4)
Structed (8)
Mean Total
Params.(M)
Methods
LoRA (We. W
79.5
84.6
59.8
72.3
0.29
HTA (Wa, W
81.0
84.6
59.6
72.7
0.09
HTA (W., W WECL. WFC2)
81.1
86.3
61.5
73.9
0.28
Adapter (W FCI, Wro2)
79.0
84.1
58.5
71.4
0.16
HTA (WECI. FF22)
81.0
84.9
60.0
73.0
0.05
Natura (7)
Specialized (4)
Stracted (8)
83,0
62.4
0.5
87.2
86.6
0.5
63.5
62.3
0,5
85.8
62.0
Accuncy
0.4
0,4
0,4
80.3
59.1
0,3
0.3
57.5
0.3
(W0 /1ysumnn
0.2
0,2
56.0
0.2
77.0
0.1
78.8
0.1
$3.0
0.1
76.0
77.4
51.5
75.0
0.0
76.0
0.0
50.0
0.0
Dimension
Dimenssoo
Figure 2: Ablation study on the impact of different bottleneck dimensions of adaptive matrices in HTA. The bar chart represents the Top-1 Test Accuracy, the line graph indicates parameters count.
a fair comparison, we also applied HTA separately to (W,, W and [WFc1, WFC2). From the results in Table 5 we observe that our method slightly outperforms LoRA but with significantly fewer parameters. Moreover, our method achieves significant improvement over Adapter still using much fewer parameters. These results indicate that method achieves better trade-off between adaptation performance and parameter efficiency. To further test the effectiveness of our method when using a similar parameter scale LoRA, additionally applied HTA FFN layers. The results show that under the same parameter size, our method exhibits a noticeable improvement over LoRA.
Ablation study the bottleneck dimensionality of additive adaptation matrix in HTA. We conducted ablation experiments to verify the effect incorporating low-rank adaptation matrices in HTA, as well the impact of its bottleneck dimensionality. The results are presented in Fig. 2 From these results, we observe that without the addition of low-rank adaptation, HTA experiences an obvious performance drop. This is due to the fact that while deriving orthogonal matrices using Householder transformations is parameter-efficient their inherent dependence on a single chosen vector makes them insufficient dimensional space. When using a low-rank adaptation matrix with rank 1, HTA shows significant performance boost. This indicates that even with simple low-rank adaptation, HTA can achieve promising trade-off between adaptation performance and parameter efficiency. By incorporating these low-rank matrices, HTA can maintain high performance while being parameter-efficient.
set of general orthogonal bases for representing arbitrary high-
5 Limitations
In this work, we use Householder transformations to construct adaptation matrices in parameter- efficient manner. Although Householder transformation matrices are orthogonal, they cannot serve as general orthogonal bases in high-dimensional spaces due their inherent dependence on a single vector. This limitation may reduce the adaptation capacity of the adaptation matrix composed of two Householder matrices. We address this issue by incorporating a rank-1 adaptation matrix, which may somewhat detract from the elegance of the proposed method. However, it is worth exploring strategies to eliminate the need for the additive adaptation matrix, thereby further enhancing the elegance and efficiency of the HTA method.

---

6 Conclusions
In this work, we proposed novel Parameter-Efficient Fine-Tuning (PEFT) solution. Our method addresses the limitation of fixed bottleneck dimensionality in low-rank based adaptation matrices, which can restrict adaptation flexibility. By viewing the adaptation matrix from the perspective of Singular Value Decomposition (SVD), we use Houscholder transformations to mimic orthogonal bases. These transformations, derived from a single vector, are highly parameter-efficient We adaptively learn diagonal coefficients to flexibly combine two Householder matrices into an adaptation matrix, accommodating layer-wise variations. Theoretically, our method can generate adaptation matrices with varying ranks while maintaining reasonable parameter size, offering a potential alternative to low-rank based adaptations. Experiments on two sets of downstream vision classification tasks demonstrate the effectiveness of our approach.
Acknowledgments and Disclosure of Funding
W. Dong's participation was in part supported by the Natural Science Basic Research Program of Shaanxi (Program No.2024JC- YBMS-464), Major Science and Technology Innovation Project of Xianyang (Program N012002-2DDK2220G-GG000000
References
[1] E. J. Hu, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen et al., "Lora: Low-rank adaptation of large language models," in International Conference on Learning Representations, 2021.
[2] S. Chen, C. Ge, Z. Tong, J. Wang. Y. Song, J. Wang, and P. Luo, "Adaptformer: Adapting vision transformers for scalable visual recognition," Advances in Neural Information Processing Systems, vol. 35, pp. 16 6646666666 678, 2022.
[3] W. Dong, Yan, Z. Lin, and Wang. "Efficient adaptation of large vision transformer via adapter compoinss,,,"" in Thirty-seventh Conference on Neural Information P Processing Systems, 2023.
[4] Zhuang, Z. Qi, K. Duan, D. Xi, Y. Zhu, H. Zhu, H. Xiong, and Q. He, "A comprehensive survey transfer learning," Proceedings of the IEEE, vol. 109, no. 1, pp. 43-76, 2020.
[5] S. J. Pan and Q. Yang, "A survey on transfer learning," IEEE Transactions on knowledge and data engineering, vol. 22, no. 10, pp. 1345-1359, 2009.
[6] M. Iman, H. R. Arabnia, and K. Rasheed, "A review of deep transfer learning and recent advancements," Technologies, vol. 11, no. 2, p. 40, 2023.
[7] W. Ying, Y. Zhang, J. Huang, and Q Yang, "Transfer learning via learning to transfer," in International Conference on Machine Learning. PMLR, 2018, pp. 5085-5094.
[8] Q Yan, D. Gong, and Y. Zhang, "Two-stream convolutional networks for blind image quality assessment," IEEE Transactions Image Processing, vol. 28, no. 5, pp. 2200-2211 1, 2018.
[9] Su. Q. Yan, Y. Zhu, C. Zhang, X. Ge, J. Sun, and Y. Zhang, "Blindly assess image quality in the wild guided by self-adaptive hyper network." in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020.
[10] Y. Zhang, H. Zhang, N. M. Nasrabadi, and T. S. Huang, "Multi-metric learning for multi-sensor fusion based classification," Information Fusion, vol. 14, no. 4, pp. 431-440, 2013.
[11] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, "Imagenet: A large-scale hierar- chical image database," in 2009 IEEE conference on computer vision and pattern recognition, 2009.
[12) M. Tan and Q. Lee "Efficientnetv2: Smaller models and faster training." " International conference machine learning, 2021.
10

---

[13] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer. G. Heigold, S. Gelly et al., "An image is worth 16x16 words: Transformers for image recognition at scale," in International Conference on Learning Representations, 2020.
[14] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, "Swin transformer: Hierarchical vision transformer using shifted windows," " in Proceedings of the IEEE/CVF international conference computer vision, 2021.
[15] S. Yun, S. J. Oh, B. Heo, D. Han, J. Choe, and S. Chun, "Re-labeling imagenet: from single to multi-labels, from global to localized labels." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[16] C.-Y. Wang, A. Bochkovskiy, and H.-Y. M. Liao, "Scaled-yolov4:: Scaling cross stage partial network," in Proceedings of the IEEE/ovf conferencce computer vision and pattern recognition, 2021.
[17] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, "End-to-end object detection with transformers," in European conference on computer vision, 2020.
[18] B. Cheng, Schwing, and A. Kirillov, "Per-pixel classification is not all you need for semantic segmentation," Advances in neural information processing systems, 2021.
[19] A. Kirillov, Y. Wu, K. He. and R. Girshick, "Pointrend: Image segmentation rendering," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020.
[20] K. He. X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, "Masked autoencoders are scalable vision learners," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022.
[21] X. Chen, S. Xie, and K. He, "An empirical study of training self-supervise vision transformers," Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
[22] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. At- tariyan, and S. Gelly, "Parameter-efficient transfer learning for nlp." in International Conference on Machine Learning, 2019.
[23] E. B. Zaken, Y. Goldberg, and S. Ravfogel, "Bitfit: transformer-basee masked inguage-modells in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2022, pp. 1-9.
Simple parameter-efficient fine-tuning for
[24] M. Jia, L. Tang, B.-C. Chen, C. Cardie, S. Belongie, B. Hariharan, and S.-N. Lim, "Visual prompt tuning," in European Conference on Computer Vision, 2022.
[25] D. Lian, D. Zhou, J. Feng, and X. Wang. "Scaling & shifting your features: A new baseline for efficient model tuning," Advances in Neural Information Processing Systems, 2022.
[26] Jie and Z. Deng, "Fact: Factor-tuning for lightweight adaptation on vision transformer," in Proceedings of the AAAI Conference Artificial Intelligence, 2023.
[27] Q Zhang, M. Chen, A. Bukharin, P. He, Y. Cheng, W. Chen, and T. Zhao, "Adaptive budget allocation for parameter-efficient fine-tuning," in The Eleventh International Conference on Learning Representations, 2023.
[28] N. Hyeon- Woo, M. Ye-Bin, and T.-H. Oh, "Fedpara: Low-rank hadamard product for communication-effiiiee federated learning," arXiv preprint arXiv: S.Niv::108.060990 2021.
[29] W. Dong, X. Zhang, B. Chen, D. Yan, Z. Lin, Q. Yan, P. Wang, and Y. Yang, "Low-rank rescaled vision transformer fine-tuning: A residual design approach," Proceedings of the IEEE/CVF International Conference Computer Vision, 2024.
[30] A. Householder, "Unitary triangularization of a nonsymmetric matrix," Journal of the ACM (JACM), vol. 5, no. 4, pp. 339-342, 1958.
[31] L. N. Trefethen and D. Bau, Numerical linear algebra. SIAM, 2022.
11

---

[32] J. G. Francis, "The qr transformation unitary analogue to the transformation part 1," The Computer Journal, vol. no. pp. 265-271, 1961.
[33] R. A. Horn and R. Johnson, Matrix analysis.
Cambridge university press, 2012.
[34] A. Steiner, A. Kolesnikov, X. Zhai, R. Wightman, J. Uszkoreit, and L. Beyer. "How to train your vit? data, augmentation, and regularization in vision transformers," arXiv preprint arXiv:2106. 10270, 2021.
[35] I. Loshchilov and F. Hutter, "Decoupled weight decay regularization," in International Confer- ence on Learning Representations, 2018.
[36] A. Paszke, S. Gross, F. Massa, Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al., "Pytorch: An imperative style, high-performance deep learning library," Advances in neural information processing systems, 2019.
[37] C. Wah, Branson, P. Welinder, P. Perona, and S. Belongie, "The caltech-ucsd birds-200-2011 dataset," 2011.
[38] G. Van Horn, S. Branson, R. Farrell, S. Haber, J. Barry, P. Ipeirotis, P. Perona, and S. Belongie, "Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection," in Proceedings of the IEEE conference on computer vision and paltern recognition, 2015.
[39] M.-E. Nilsback and A. Zisserman, "Automated flower classification over large number of classes," " in 2008 Sixth Indian conference on computer vision, graphics & image processing. 2008.
[40] A. Khosla, N. Jayadevaprakash, B. Yao, and F.-.. Li, "Novel dataset for fine-grained image categorization: Stanford dogs," in Proc. CVPR workshop on fine graing visual categorization (FGVC), vol. 2, no. 1, 2011.
[41] T. Gebru, J. Krause, Y. Wang, D. Chen, J. Deng, and L. Fei-Fei, "Fine- "Fine-grained car detection for visual census estimation," in Proceedings of the AAAI Conference on Artificial Intelligence, 2017.
[42] X. Zhai, J. Puigcerver, A. Kolesnikov, P. Ruyssen, C. Riquelme, M. Lucic, Djolonga, A. S. Pinto, M. Neumann, A. Dosovitskiy et al., "A large-scale study of representation learning with the visual task adaptation benchmark," arXiv preprint 1ixv:1910.0440 2019.
12

---

Efficient Adaptation of Pre-trained Vision Transformer via Householder Transformation
Supplementary Materials
In the supplementary materials involving our work, we demonstrate Detailed dataset statistic, Hyper- parameters our work, Experimental details, and broader impacts, including:
A Detailed dataset statistic
Hyper-parameters our work
Experimental details on larger-scale and hierarchical ViT backbones
Experimental details on ablation study
Broader impacts
Due to the limitation that supplementary materials larger than 100MB cannot be uploaded to the Open- Review website, only the project code as concise supplementary materials is uploaded this web- site. Please reler the anonymous linkhttps: :/drive google. com/ ht///fie//// /////////////// 4/18shhtqM12244 LRICK2IVS1KiF wwt.s/ii/iiiiia/eeieerere obtain the complete code, datasets, and models.
13

---

A Detailed dataset statistic
We provide detailed information about the datasets used in this paper, including the number classes and the sizes of the training, validation, and test sets, in [able| and (VTAB-II) FGVC datasets include CUB-200-2011, NABirds, Oxford Flowers, Stanford Dogs, and Stanford Cars, which used for fine-grained classification tasks of birds, flowers, dogs, and cars, respectively. The VTAB-IK Ik datasets cover natural, specialized, and structured tasks, including natural image datasets such CIFAR- 1-00, Caltech101, DTD, Flowers102. Pets. SVHN, and Sun397: specialized image datasets such as Patch Camelyon, AuroSAT, Resisc45, and Retinopathy; and structured image datasets such Clevr/count, Clevr/distance, LM,, KITTV/ditaacc, dSprites/location, dSprites/orientation, SmallNORRBaziimuth, and SmalllNORB/elevationn. Detailed information about these datasets is presented in the tables.
Table Dataset statistics for FGVC. "** denotes the train/val split datasets following the dataset
setting in VPT 24
Dataset
Description
Classes
Train size
Val size
Test size
CUB 201-00000000 201
37
"ine-grained bird species recognition
200
5,394*
600*
5,794
NABirds 38
Fine-grained bird species recognition
555
21,536*
2,393*
24,633
Oxford Flowers
Fine-grained flower species recognition
102
1,020
1,020
6,149
Stanford Dogs
40
Fine-grained dog species recognition
120
10,800
1,200*
8,580
Stanford Cars
41
Fine-grained classificatio
196
7,329*
815*
8,041
B Hyper-parameters in our work
Table provides summary of the configurations used in our experiments. As discussed in Section 4 we performed a grid search on the validation set of each task to determine the optimal hyperparameters. including learning rate, weight decay, batch size, and dropout rate.
C Experimental details on larger-scale and hierarchical VIT backbones
Table and respectivyy display the comprehensive results of the comparison conducted in Section among Vir-L and Swin-Base models.
D Experimental details on ablation study
We provide further explanation of the ablation experiments in Section 4 In the study on the transferability of HTA, we replaced the low-rank adaptation matrices in LoRA, we used HTA module to replace the bottleneck part of LoRA, while in Adapter, we directly replaced the Adapter with an HTA module. The detailed experimental results are presented in the table In the study of the low-rank adaptation part of HTA, we set its dimensions to 0, 1, 2, and 4, respectively. The results are shown Table
E Broader impacts
Practicality: Our approach differs from traditional methods by employing Householder transforma- tions rather than standard unitary matrices, which can be efficiently derived. This approach boosts the efficiency of parameter usage and significantly cuts down on the number of parameters requiring fine-tuning. With this technique, we manage to achieve high performance while optimizing parameter use. Leveraging large-scale pre-trained models, our HTA method proves to be both highly efficient and practical across diverse applications.
Low Energy Consumption: Our approach enhances the model's computational efficiency by de- creasing the necessary computational parameters, thus reducing energy usage during training. This reduction aids conserving energy and lowering emissions, aligning with global sustainability goals and the push for eco- aco-friendly practices. Moreover, our method not only improves the model's
14

---

Table 2: Dataset statistics for VTAB- 1k
42
Dataset
Description
Classes
Train size
Val size
Test size
CIFAR- 100
100
10,000
Caltech101
102
6,084
DTD
47
1,880
Flowers 102
Natural
102
800/1,000
200
6,149
Pets
37
3,669
SVHN
10
26,032
Sun397
397
21,750
Patch Camelyon
2
32,768
EuroSAT
10
5,400
Resise45
Specialized
45
800/1,000
200
6,300
Retinopathy
42,670
Clevr/count
15,000
Clevr/distance
15,000
DMLab
22,735
KITTI/distance
711
Structured
800/1,000
200
dSprites/location
16
73,728
dSprites/orientttion
16
73,728
SmallNORBBaziiuth
18
12,150
SmallNORBBcelevttion
12,150
Table The implementation details of configurations such as optimizer and hyper- paprr-parameters. We select the best hyper-paane parameters for each download task via using grid search.
Optimizer
Adam [0.2, 0.1, 0.05, 0.01, 0.005, 0.001, 0.0001] [0.05, 0.01, 0.005, 0.001,
Learning Rate
Weight Decay
Batch Size
{64, 32, 16) (0.5, 0.3, 0.2, 0.1, 0)
Adapter Dropout
Learning Rate Schedule
Cosine Decay 100
Training Epochs
Warmup Epochs
10
Table 4: This table is extended from Table 3 in Section 4 and describes the detailed experimental results of the performance comparison on VTAB 1k using ViT-Large pre-trained on ImageNet-21a as the backbone.
Natural
Specializd
Steastd
Mathads
Mesteepatty
Clor--Coent
ANOHH-30 30t
48pm AM
Moaa
MArdHARS
Mone
SNORI
Lavar grllis
72.2
95.4
63.6
07.4
85.8
36.1
$2.5
70.9
76.9
37.3
66.5
45.4
69.
28.2
38.0
34.7
54.0
10.0
14.3
14.6
21,9
35
$1.5
0.65
Adspis
vpresteligg cm
80
34.
67
68.0
85.0
756
11.
767
79
41.4
11.4
71.1
79
41.5
K24
33.
66.3
188
14.1
21.6
36.8
40.6
62.0
0115
Virctee
31.
5010
19.0
96.6
83.9
60.
461
812
47.4
36.9
714
8:49
ABC
98.1
56.1
912
86.6
75.9
15.3
1118
SSB 25
73.5
91.3
700
99.3
91.3
10.6
37.5
85.9
949
15.5
14.
85.2
90.
1910
393
500
77.6
54.0
31.5
35.0
990
13.0
0.60
92.0
92.1
346
4.9
924
92.8
841
95.5
76.5
968
02.4
315
87.1
48A
75.4
1154
performance and efficiency but also promotes environmental sustainability by embracing sustainable development principles.
Ethical Aspects: Our model utilizes the vast capabilities of large-scale pre-trained models for representation and generalization. However, it is trained on datasets that might contain problematic data, such as illegal content or inherent biases, which our model could inadvertently learn. To tackle this challenge, addressing model toxicity becomes critical. Consequently, it's imperative to develop
15

---

Table 5: This table is extended from Table 4 in Section 4 results of the performance comparison on VTAB-Ik using Swin-Base pre-trained on ImageNet-21k the backbone.
and describes the detailed experimental
Natural
Specialord
Stowee
Mithadh
Choo-Doan
STTTI 046
NORRD
0TD
65p
Mean
Lavorgmberg
74.8
T4T
155
3ST
65.0
28.3
MLP4
71.4
1066
16.8
80.7
66.9
03-3
66.5
21.1
14.5
78.8
31.2
57.2
Biss 13
03.7
16.2
90.4
76.1
80.0
4133
34.2
36.1
13.2
31.6
42.
WPLShalow Ea
7240
91.3
773
99.4
$3.4
54.5
1059
9641
05.9
9338
72.1
82.3
40.
63.9
34.1
63.2
25.4
66.5
21.5
26.3
113
623
0.80
00.3
01.4
76.8
67.6
3011
RIRR [S]
66
50.9
78.5
99.3
92.1
163
58.5
11.3
17.1
95.
871
76
167
66.0
57.8
853
86
91.1
963
110
36.0
107
730
0.01
918
87.5
85A
61.3
Table 6: This table is extended from Table 5 in Section configurations from the A paper. In our implementation, the fully connected layers within the bottlenecks replaced with Householder transformations. "(-)" indicates specific configuration information
LoRA and Adapter both follow the
Daltacts
Specallad
CIPAM-10 00
JNOHLL 60
Teal
DTD
SYNH
Monn
Reddon
DN
44.0
IIAAW W.l
71.
91.3
932
51.
18.6
HEALW W.. Ween Wyezl
717
01.1
110
722
16L3
R5A
814
K1.
96.5
86
7622
99.
918
60.1
310
820
860
82.
32.9
46.0
41
139
0.26
822
023
722
125
102
840
817
341
033
485
763
145
127
-11
33.
Table This table extended from Fig. in Section
Datnnes
Specidlred
Parank/N
CH
DED
SVNH
BUNSAT
Moue
Cleve-D0
BMhah
Mon
13.0
90.1
11.
99.3
91
$3.4
90.5
82.3
94.2
82
13.7
83.2
61.
48.
80.0
$1.3
533
28
41.
22.0
0.15
73.0
55.7
82.5
58.0
84.0
90.
855
33.9
0,30
27
14.5
936
12.1
993
91.
58.3
36.1
823
88.0
96.3
85.0
764
866
81.8
62
536
823
848
336
36.
47.2
62.8
75.0
0.44
enhanced mechanisms that can both identify and reduce such biases and unlawful information the datasets.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please see Abstract and Introduction Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
Our work does not involve specific theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
We provide supplementary materials involving statistical datasets, detailed papee-parameters of model settings. And also we provide complete code, datasets and models at the anonymous link https://drive htwdddiiee.oogll..eeeooooooooeee honggllee.oo//ille/////tttttotoo
Justification:
LRI RRICC22NSSIKKFFPHrr2///veie
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
18

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide supplementary materials involving statistical datasets, detailed hyper-parameters of model settings. And also we provide complete code, datasets and models at the anonymous link https://drive.. www.le....o..oooooooooooo.o..... cwwmfilled/eeaaaccccccecoe /ttpe////mm/////////s/////aaa/
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
www.ssoooiiiyyooooooooooilliiiii olicy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips /twpss.ccppblll.... httppppullllippeeeeepepppppppeee htte:/guddeeeeeeeSubmiiiieeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please see our supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: Our work does not involve specific error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Our paper conforms the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please see Appendix Section
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
20

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The license and terms of use explicitly are mentioned and properly respected.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
21

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "j8dSnC19rx", "reviewer": "Reviewer_6gjh", "summary": "The paper presents a new parameter efficient fine tuning (PEFT) technique for vision transformer (ViT) models. The work utilizes the intuition of creating an adaptation matrix for fine-tuning from a popular dimensionality reduction technique named singular value decomposition (SVD) which is named Householder transformation based Adapter (HTA).\n\nHTA can be utilized to create adaptation matrices to accommodate layer wise variations. Extensive empirical evaluations have been performed over various popular vision benchmarks. Code has been made available for reproducibility.", "questions": "Do you think the HTA idea is adaptable to other modalities for PEFT?", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "HTA presented in this work is a new PEFT technique to avoid full fine-tuning for large vision models. The empirical evaluations are extensive and rigorous and HTA is benchmarked against widely popular PEFT techniques. HTA achieves great performance similar to established previous state-of-the-art thresholds with gains in terms of trainable parameter efficiency.\n\nThe readability of the work is great with great visualizations and empirical evaluations have been crisply presented. The writing structure and language are easily digestible for the reader.", "weaknesses": "No weaknesses apart from the limitations mentioned in Section 5 by the authors", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Questions: Do you think the HTA idea is adaptable to other modalities for PEFT?**\n\nTo validate the adaptability of HTA in other modalities, we selected the large model Llama from the NLP domain for further testing. Due to time constraints, we only used the smaller commonsense_15k dataset as the training set and the complete commonsense dataset as the test set. We used LoRA (rank=32) as a baseline to compare its performance with that of HTA. Despite HTA having only a quarter of the parameters of LoRA (rank=32), its performance was still comparable to that of LoRA (rank=32). This indicates that our method is not only applicable to image modalities but also to other modalities such as text.\n\nWe greatly appreciate your suggestion to explore this research direction and will continue to investigate its potential in other modalities in future studies. Thank you very much for your valuable feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AJNq1MNMgC", "reviewer": "Reviewer_HQtR", "summary": "The paper presents a novel method, HTA, for efficiently adapting pre-trained transformers by applying the Householder transformation to a single vector to approximate the SVD for representing the adaptation matrix. By combining this with an additional rank-1 adaptation matrix, HTA achieves a reduction in the number of parameters while maintaining strong performance across various benchmarks, including FGVC and VTAB-1k, and outperforms existing methods in terms of efficiency and performance.", "questions": "1. Fig. 1c has a skip-connection, but Fig. 1d does not. Is it proposed to be like this?\n2. The proposed HTA includes two feature paths, with both the Householder transformations path and the other down-up projection path being rank 1. It remains unclear why the additional down-up projection path would help significantly boost performance.\n3. The results in Fig. 2 with dimension=0 are different from the ones in previous tables. What are the differences in settings between them?\nFor Swin, it would be better to include which specific Swin model is applied.\n4. For Swin, it would be better to include which specific Swin model is applied.", "limitations": "The novelty and significance of HTA are somewhat limited. However, the extensive experimental results provide a robust foundation for the claims made in the paper. Overall, the work is well-executed and makes a valuable contribution to the field, justifying a recommendation for weak acceptance.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Originality:\nThis work incorporates the Householder transformations to construct orthogonal matrices, replacing the previously proposed low-rank adaptation matrix, and avoids the hyperparameter choice of the rank.\n\nQuality:\nThe submission is technically sound with well-supported claims backed by extensive experimental results.\n\nClarity:\nThe paper is well-written and organized, making it easy to follow the methodology and results.\n\nSignificance:\n1. This work addresses a challenging task and advances the state-of-the-art with high performance and reduced computational costs.\n2. It may lead to further development of methodologies based on Householder transformations.", "weaknesses": "Originality:\nUtilizing Householder transformations for matrix decomposition is not a brand-new idea.\n\nQuality:\nThe improved performance of HTA over the state-of-the-art methods on several benchmarks is not very significant (<0.5%).\n\nClarity:\nA scatter plot for accuracy versus the number of parameters tradeoff would give a clearer insight into HTA's performance.\n\nSignificance:\n1. The challenge of solely relying on Householder transformations for adapting pre-trained transformers remains.\n2. The proposed HTA removes the rank choice in previous adaptation methods but introduces a new rank choice in the additional adaptation matrix, although rank 1 might be sufficient for a good performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weaknesses(Originality): Utilizing Householder transformations for matrix decomposition is not a brand-new idea.**\n\nThe Householder transformation is indeed not a new concept in matrix decomposition; rather, it is a commonly used method. However, the core of our approach does not lie in the use of Householder transformations for matrix decomposition, but rather in transforming a one-dimensional Householder vector into a unitary matrix (analogous to the left and right unitary matrices in SVD decomposition) through the Householder transformation. In this way, we construct a rank-tunable matrix using three one-dimensional vectors (two Householder vectors and one singular value vector). This allows us to fine-tune models using rank-tunable matrices, freeing us from the constraints of fixed rank imposed by previous methods due to preset bottleneck dimensions.\n\n**Weaknesses(Quality): The improved performance of HTA over the state-of-the-art methods on several benchmarks is not very significant (<0.5%).**\n\nOur main contribution lies in providing new insights into the popular low-rank-based pre-trained model adaptation strategy, rather than merely improving fine-tuning performance. We aim to address the inherent limitations of low-rank-based adaptation strategies, such as LoRA and Adapter, which rely on a fixed rank of the adaptation matrix by pre-setting the bottleneck dimensionality. Instead, we construct a rank-tunable adaptation matrix by leveraging the idea of the Householder transformation. This method composes the adaptation matrix in a parameter-efficient manner using only three one-dimensional vectors. Our approach offers greater flexibility in constructing the adaptation matrix and provides a better trade-off between parameter count and adaptation capacity.\n\n**Weaknesses(Clarity): A scatter plot for accuracy versus the number of parameters tradeoff would give a clearer insight into HTA's performance.**\n\nThank you for your suggestion. Following your advice, we have created a scatter plot to more clearly illustrate HTA's performance. The scatter plot is presented in Figure 2 in the rebuttal PDF.\n\n**Questions: Fig. 1c has a skip-connection, but Fig. 1d does not. Is it proposed to be like this?**\n\nIn Figure 1d, we adhered to the original Adapter branch configuration, including the skip connection. However, for the sake of simplicity and aesthetics, we omitted the skip connection in the illustration. Following your suggestion, we have revised Figure 1d, and the updated figure is presented in Figure 2 of the rebuttal PDF.\n\n**Questions: The proposed HTA includes two feature paths, with both the Householder transformations path and the other down-up projection path being rank 1. It remains unclear why the additional down-up projection path would help significantly boost performance.**\n\nIn the Householder transformation branch, the left and right unitary matrices are derived from Householder mappings, resulting in orthogonally symmetric unitary matrices. Although the Householder matrix itself possesses mapping capabilities and can indirectly represent rotational characteristics as the Householder vector is continuously optimized during training, its inherent symmetry limits its ability to perfectly replicate arbitrary unitary matrices, imposing certain constraints. To address this, we introduced an additional matrix to break these constraints. Considering the parameter count, we opted for a down-up projection with a dimension of 1 to resolve this issue.\n\n**Questions: The results in Fig. 2 with dimension=0 are different from the ones in previous tables. What are the differences in settings between them?**\n\nWe need to clarify that in the previous tables, we used dimension=1 as our setting, and it can be observed that the results for dimension=1 in Figure 2 are consistent with those in the previous tables. In the ablation study presented in Figure 2, setting dimension to 0 is intended to validate the performance of the Householder transformation branch itself and the contribution of the down-up projection branch. In future versions, we will provide a more precise description of the settings for each component.\n\n**Questions: For Swin, it would be better to include which specific Swin model is applied.**\n\nSpecifically, we used Swin-B as our fine-tuning model, with the pre-trained weights obtained from the official swin_base_patch4_window7_224_22k.pth, which was trained on ImageNet21k. The details are as follows:\n    \"base\" refers to the base model, which strikes a balance between performance and computational complexity.\n    \"Patch4\" indicates that the input image is divided into 4x4 patches for processing.\n    \"window7\" denotes the use of a 7x7 window size for the window attention mechanism during each stage of computation.\n    The input size of 224 refers to the image size of 224x224 pixels used during pre-training.\n\nThank you very much for your thorough review and valuable comments. Your feedback has been instrumental in helping us refine our work, and we greatly appreciate the time and effort you have dedicated to this process. We look forward to further improving our research based on your insights.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XxHHPA3z4P", "reviewer": "Reviewer_YBq9", "summary": "This paper proposes a new PEFT method by integrating Householder transformations, which enables the creation of adaptation matrices that have varying ranks across different layers, thereby offering more flexibility in adjusting pre-trained models.", "questions": "Is the proposed method effective in tasks such as object detection and semantic segmentation?", "limitations": "The novelty, contribution and experimental comparison of this paper all have certain shortcomings.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1: This paper proposes a new PEFT method based on Householder transformations.\n\nS2: This method allows for layer-wise rank variations.\n\nS3: The authors validated the effectiveness of the proposed method on classification tasks.", "weaknesses": "I am not an expert in the field of PEFT, it appears that the authors have proposed a new PEFT scheme. From an application perspective, we desire PEFT methods that can transfer the pre-trained models to downstream tasks with as few parameters as possible. In the experiments conducted by the authors, the proposed method achieved slight improvements with relatively fewer parameters. In fact, all methods utilize a minimal number of parameters.\n\nIn addition, the authors did not make comparisons in more complex tasks, such as object detection and semantic segmentation. Currently, numerous related studies employ a backbone based on the ViT [A,B,C]. Validation solely on classification tasks is somewhat limiting.\n\n\n[A] Exploring Plain Vision Transformer Backbones for Object Detection.\n\n[B] SegViT: Semantic Segmentation with Plain Vision Transformers.\n\n[C] ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We agree that all PEFT methods use a reduced number of parameters, and at this level, further reduction in parameter count has limited significance in practical applications. However, compared to previous low-rank-based PEFT methods such as LoRA and Adapter, our approach addresses a key issue: the fixed-rank limitation of the adaptation matrix due to preset bottleneck dimensionality. By using fewer parameters, we construct a rank-tunable matrix composed of three one-dimensional vectors, which can theoretically adjust its rank from 1 to full rank. This allows it to flexibly replace any low-rank matrix, thereby enhancing its adaptation capacity. This feature provides our method with great potential for application to more complex datasets.\n\nTo further validate the effectiveness of our proposed method in object detection and semantic segmentation tasks, we conducted downstream experiments following the settings of Swin Transformer. For the object detection task, we used the COCO Object Detection (2017 val) dataset, selecting Swin-B as the backbone (with pre-trained weights on ImageNet1k) and using Cascade Mask RCNN as the decoder. The results show that our HTA method (mIoU=47.2) improved by approximately 3% compared to LoRA (mIoU=44.2).\nFor the semantic segmentation task, we used the ADE20K Semantic Segmentation (val) dataset, again selecting Swin-B as the backbone (with pre-trained weights on ImageNet1k) and UperNet as the decoder. The results indicate that our HTA method (mIoU=47.6) outperformed LoRA (mIoU=46.1) by about 1.5%.\nAdditionally, due to time constraints, we were unable to validate all the experiments you mentioned. Instead, we chose to validate using ViTPose, employing the ViTPose-B (simple) model and conducting experiments on the COCO Object Detection (2017 val) dataset. Using full fine-tuning with the MHSA module frozen as the baseline, we added LoRA and HTA to the MHSA module. The experimental results are as follows: baseline (mAP=72.8), LoRA (mAP=73.6), HTA (mAP=74.1). These results demonstrate the effectiveness of our method in downstream tasks and suggest that it could serve as an alternative to LoRA. It is important to note that all experiments, except for the ViTPose baseline, were independently reproduced by us. Apart from the HTA and LoRA modules, all other components were kept consistent to ensure the accuracy of the results.\n\nThank you for your comments and questions. The issues you raised are very helpful for our future work. We will incorporate your suggestions to ensure that our work is useful across multiple fields.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B8cK9auYVh", "reviewer": "Reviewer_PRgH", "summary": "The study introduces a data-driven variant of LoRA-like Parameter-Efficient Fine-Tuning (PEFT) for Vision Transformers (ViTs), which learns the optimal bottleneck dimension (the rank of the fine-tuning weight matrix) instead of preconfiguring it for each layer. The authors propose learning three additional vectors: $v_{\\text{left}}$, $v_{\\text{right}}$, and a vector of diagonal parameters $D$ that define the rank. Inspired by Singular Value Decomposition (SVD), they employ Householder transformations $I-vv^\\intercal$ on $v_{\\text{left}}$, $v_{\\text{right}}$ to construct orthogonal matrices. These learned vectors and parameters $v_{\\text{left}}$, $v_{\\text{right}}$, and $D$ can then be fused into the frozen weights in a LoRA-like manner. Jointly training $v_{\\text{left}}$, $v_{\\text{right}}$, and $D$ alongside an extremely low-rank LoRA branch (r=1) results in improved performance on downstream tasks.", "questions": "- What rank is applied to the LoRA-based baselines in Table 1? It would be beneficial to list these parameters in the appendix.\n\n- What are the settings used in Table 5? Are the LoRA branches also applied in the HTA row?\n\n- Could the authors include a section discussing theoretical and actual memory computation costs, such as FLOPS?\n\n- Is it feasible to extend the method to Large Language Models?", "limitations": "As the author discussed in the limitation section.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper tackles the optimal rank determination problem for LoRA-based methods using a learning-based approach. Despite requiring an additional rank-1 LoRA branch to enhance accuracy, both experiments and the ablation study highlight the effectiveness of the proposed method, surpassing several state-of-the-art baselines.", "weaknesses": "While the proposed method is parameter-efficient, the discussion overlooks the actual training cost and latency. For instance, it would be beneficial to include theoretical or actual memory and computational costs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Questions: What rank is applied to the LoRA-based baselines in Table 1? It would be beneficial to list these parameters in the appendix.**\n\nIn Table 1, the rank for LoRA is set to 8. In subsequent versions, we will follow your suggestion and include the settings for the comparison methods in the appendix.\n\n**Questions: What are the settings used in Table 5? Are the LoRA branches also applied in the HTA row?**\n\nIn Table 5, we followed the settings from FacT, where the bottleneck dimensions for both LoRA and Adapter were set to 8. Specifically, LoRA was applied only to the fully connected layers of Q and V in the MHA module (for a more direct comparison, we applied HTA in the Q and V fully connected layers as well). In our HTA module, we adhered to the previous experimental setup by setting the bottleneck dimension in the LoRA branch to 1. All other settings were kept consistent throughout the experiments.\n\n**Questions: Could the authors include a section discussing theoretical and actual memory computation costs, such as FLOPS?**\n\nSince our method involves applying Householder transformations to vectors and performing inverse SVD decomposition to multiply the left and right unitary matrices with singular values, it requires more computational steps and resources compared to LoRA. As a result, our method incurs slightly higher FLOPs and memory consumption.\n\nWe conducted a simple calculation of FLOPs and memory usage using data of size (16, 3, 224, 224). The results are as follows: HTA (GFLOPs: 748; MEM: 5196M), LoRA (GFLOPs: 566; MEM: 4752M), Adapter (GFLOPs: 565; MEM: 4813M). The increase in FLOPs is primarily due to the Householder transformation, and the inverse SVD decomposition also introduces some additional computational overhead. Additionally, the increase in memory usage is mainly due to the extra gradient maps generated by these additional computational steps. In this experiment, the bottleneck dimensions for both Adapter and LoRA were set to 8. We will also follow your suggestion and include a separate section in future versions to discuss the theoretical and practical computational costs.\n\n**Questions: Is it feasible to extend the method to Large Language Models?**\n\nTo validate the adaptability of HTA on large language models, we selected the Llama model from the NLP domain for further testing. We used the commonsense_15k dataset as the training set and the complete commonsense dataset as the test set. Comparing HTA with LoRA (rank=32) as the baseline, we found that despite HTA having only a quarter of the parameters of LoRA (rank=32), its performance was still comparable to that of LoRA (rank=32). This indicates that our method is also applicable to large language models.\n\nThank you for your review and comments. Your recognition of our work, as well as the identified shortcomings and issues, is extremely helpful for both our current paper and future research. We will carefully consider your suggestions and incorporate them into our manuscript to enhance its overall quality.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate the reviewers and ACs for their thorough and valuable comments on our manuscript and their recognition of our work. We aim to address the shared concerns raised by the reviewers and provide a unified response below. Additionally, we have offered detailed replies to the specific questions posed by each reviewer within separate rebuttal windows.

**Limited performance improvement and whether further reducing the parameter count is practical given that other PEFT methods already have sufficiently small parameter sizes.** **Re:** Our primary contribution lies not merely in improving fine-tuning performance or reducing parameter count, but in offering new insights into popular low-rank-based pre-trained model adaptation strategies. Our goal is to address the inherent limitations of low-rank adaptation strategies, such as LoRA and Adapter, which rely on fixed-rank adaptation matrices with predefined bottleneck dimensions. To overcome this, we have utilized the concept of Householder transformations to develop an adaptable matrix with adjustable rank. This approach requires only three one-dimensional vectors to efficiently construct the adaptation matrix, providing greater flexibility in matrix design and achieving a better balance between parameter size and adaptability.

**Whether HTA is effective in downstream tasks, other modalities, and large language models.** **Re:** We conducted validations across different domains, including common-sense reasoning, object detection, and semantic segmentation, using models like Llama, Swin Transformer, and ViTPose. The experimental results demonstrate that our method often achieves comparable or even superior performance with less data compared to LoRA, effectively positioning it as a superior alternative. This further validates the effectiveness and generalizability of our approach.

Meta-review: The reviewers unanimously voted for acceptance. They appreciated the technical originality, clarity and extensive experiments that support the efficacy of the proposed approach. After carefully reading the reviews and the rebuttal, this meta-reviewer concurs with the reviewers' recommendations.
Decision: Accept (poster)