Efficient Adaptation of Pre-trained Vision Transformer via Householder Transformation

Authors: wei  dong
yuan  sun
yiting  yang
xing  zhang
zhijun  lin
qingsen  yan
haokui  zhang
peng  wang
yang  yang
hengtao  shen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-04 10:18:47

Abstract: A common strategy for Parameter-Efficient Fine-Tuning (PEFT) of pre-trained Vision Transformers (ViTs) involves adapting the model to downstream tasks by learning a low-rank adaptation matrix. This matrix is decomposed into a product of down-projection and up-projection matrices, with the bottleneck dimensionality being crucial for reducing the number of learnable parameters, as exemplified by prevalent methods like LoRA and Adapter. However, these low-rank strategies typically employ a fixed bottleneck dimensionality, which limits their flexibility in handling layer-wise variations. To address this limitation, we propose a novel PEFT approach inspired by Singular Value Decomposition (SVD) for representing the adaptation matrix. SVD decomposes a matrix into the product of a left unitary matrix, a diagonal matrix of scaling values, and a right unitary matrix. We utilize Householder transformations to construct orthogonal matrices that efficiently mimic the unitary matrices, requiring only a vector. The diagonal values are learned in a layer-wise manner, allowing them to flexibly capture the unique properties of each layer. This approach enables the generation of adaptation matrices with varying ranks across different layers, providing greater flexibility in adapting pre-trained models. Experiments on standard downstream vision tasks demonstrate that our method achieves promising fine-tuning performance.

Main paper: 
 introduction: 
 Parameter-Efficient Fine-Tuning (PEFT) for pre-trained Vision Transformers (ViTs) aims to adapt these models to downstream tasks by learning a small set of parameters while keeping most or all of the original model parameters frozen. This approach is expected to reduce the cost of fine-tuning and potentially improve the model's generalization performance, particularly when the downstream task involves limited data.A common strategy for adapting the parameters is to learn an adaptation matrix that modifies the original matrix through addition or multiplication. To reduce the parameter scale of the adaptation matrix, a low-rank strategy is typically employed. This involves decomposing the adaptation matrix into the product of a down-projection matrix and an up-projection matrix, where the bottleneck dimensionality determines the parameter scale. Many prevailing PEFT solutions [1] [2] [3] follow this approach. However, these methods usually set the bottleneck dimensionality empirically to balance adaptation performance and parameter size. The fixed dimensionality, however, lacks the flexibility to accommodate variations in layer-wise properties. through affine transformations composed of scaling and shifting operations. AdaptFormer [2] explores a parallel adapter solution on ViT for various downstream tasks. FacT [26] decomposes the weights of ViT into individual three-dimensional tensors and further decomposes the increments into lightweight factors. During fine-tuning phase, these factors are updated and stored, effectively reducing computational overhead. 
lora and its variants: 
 As represented by LoRA [1] , the core of this type of PEFT method is the utilization of low-rank matrices to approximate weight adjustments during the fine-tuning phase. By employing reparameterization techniques, these low-rank matrices are combined with the existing parameter matrices, thereby circumventing extra inference costs. AdaLoRA [27] employs singular value decomposition to decompose weight matrices, pruning insignificant singular values to effectively reduce the number of parameters. ARC [3] uses symmetric up-down projections to create cross-layer shared bottleneck operations. By learning low-dimensional rescaling coefficients, it effectively recombines layer-adaptive adapters, reducing the costs of fine-tuning. FedPara [28] reparameterizes model layers with low-rank matrices and uses the Hadamard product. This approach, unconstrained by low-rank limitations, offers greater capacity and reduces learning costs. RLRR [29] examines mainstream PEFT methods from the perspective of SVD decomposition, exploring the critical balance between preserving generalization in pre-trained models and adapting them to downstream tasks. Our research abandons the traditional fixed-rank approach, opting instead for a more flexible adjustment of parameter matrices using a small number of learnable parameters.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: Please see Section 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methodology: 
 In this section, we commence with an introduction of the notations, symbols, and contextual background related to low-rank adaptations and Householder transformation. Then we present the decomposed structure of low-rank adaptation and discuss its inherent operating mechanism from the perspective of singular value decomposition. Finally, we propose a novel adaptation via Householder transformation. This adaptation primarily aims to construct the Householder unitary matrices via a learnable weight vector, thereby trading-off the fully spanned representation space and the affordable parameter size.Low-rank adaptation. Pre-trained ViT models are typically initialized with weights learned from large-scale image datasets, such as ImageNet. The pre-training process involves optimizing the model on an unsupervised or supervised pretext task. The resulting pre-trained weights encode rich semantic information that can be transferred to a wide range of downstream tasks through fine-tuning. As one of the most representative methods of fine-tuning, PEFT method achieves excellent results on downstream tasks by merely utilizing a small number of additional learnable parameters to fine-tune the ViT. The most prevalent PEFT is the adaptation method, which can be divided into two categories: LoRA-based and adapter-based methods. In general, LoRA-based method is defined as:EQUATIONwhereX (l-1) FTis the fine-tuning output, W (l) is any linear weight projection{W (l) q , W (l) k , W (l) v , W (l) o , W (l) FC1 , W (l) FC2 } in ViT, ⃗ b (l)is the bias weights,W (l) down ∈ R D (l) ×D ′ and W (l) up ∈ R D ′ ×D (l)are down-and up-adapting projection matrices across varying layers with the dimensionality D ′ ≪ D (l) . The detailed framework of LoRA-based method is shown in Fig. 1 (a) . Analogously, adapter-based method is described as: Eq. ( 1) and (2), both above-mentioned PEFT methods involve a low-rank bottleneck structure, i.e., WEQUATION(l) adaptation = W (l) down W (l)up . Note that we remove the activation in the low-rank bottleneck because the presence or absence of such activation does not affect the low-rank structure of adaptation.Householder transformation. Householder transformation, or Householder reflection, is a linear transformation that reflects a vector across a hyperplane defined by a Householder vector. It is characterized by a Householder matrix, which is an orthogonal and symmetrical matrix with determinant -1. This transformation, initially proposed by A.S. Householder in 1958 [30] , has significant applications in numerical linear algebra [31] , particularly in QR decomposition [32] , where it is used to transform a matrix into an upper triangular or Hessenberg form [33] . Householder transformation can also be employed to set specific elements of a vector to zero while preserving its norm, making it a valuable tool for matrix orthogonalization and symmetrization. The Householder matrix is defined as following:EQUATIONwith the identity matrix I and the Householder vector ⃗ v.Singular Value Decomposition (SVD) offers a profound insight into matrix factorization. It breaks down a matrix into three constituent matrices. Viewing the adaptation matrix through the lens of SVD, we represent it as:EQUATIONwhereW (l) left ∈ R D (l) ×D (l)is the left unitary matrix andW (l) right ∈ R D (l) ×D (l)is the right unitary matrix; and D (l) ∈ R D (l) ×D (l) is the diagonal matrix in which the diagonal elements of a diagonal matrix are singular values. Unitary matrices W adaptation into the space it spans. Then, the vector is scaled by the diagonal matrix D (l) . Finally, the right unitary matrix W (l) right rotates the vector back to the original linear space. Therefore, the SVD decomposition characterizes the transformations of rotation and scaling in the linear space.When it comes to the fine-tuning strategy, PEFT methods employ ViT as the backbone and essentially fine-tune the learnable parameters of unitary matricesW (l) left ∈ R D (l) ×D ′ and W (l) right ∈ R D ′ ×D (l)and the learnable singular values of diagonal matrix D (l) ∈ R D ′ ×D ′ to downstream tasks, implicitly performing the rotation and scaling transformations. Note that the number of non-zero singular values in matrix D (l) does not exceed its dimensionality D ′ . However, the fixed bottleneck dimensionality D ′ empirically set to LoRA-or adapter-based methods is inflexible, thereby without accommodating variations in layer-wise properties. This implies that the linear space spanned by the low-rank adaptation matrix and its corresponding number of non-zero singular values is constrained within a low dimensionality D ′ . Increasing the dimensionality D ′ could effectively enhance the space capacity of the adaptation matrix, thereby improving the performance potential of the fine-tuned ViT model. However, this also further increases the number of parameters in the PEFT method.To address the aforementioned issue, we introduce Householder transformation into the adaptation matrix learning, and propose the Householder Transformation-based Adaptor (HTA). Following this way, HTA facilitates the derivation of the adaptation matrix in a manner that is parameter-efficient, while theoretically accommodating the flexibility of varying ranks for the adaptation matrices.In our approach, we respectively employ the Householder matricesH (l) left ∈ R D (l) ×D (l)andH (l) right ∈ R D (l) ×D (l)as substitutes for the left and right unitary matricesW (l) left ∈ R D (l) ×D ′ and W (l) right ∈ R D ′ ×D (l)within the adaptation matrix W EQUATIONwith two learnable parameter vectors ⃗ v(l) left ∈ R D (l) and ⃗ v (l) right ∈ R D (l)and a learnable diagonalparameter vector ⃗ d (l) ∈ R D (l) in the diagonal matrix D (l) H ∈ R D (l) ×D (l) .Since the Householder transformation matrix is derived from a single vector, its transformation capacity can be limited and sensitive to the vector learned to derive it. To enhance the robustness of the adaptation matrix, we incorporate an additional low-rank adaptation matrix, resulting in the ultimate HTA. Building on this design, we can derive the LoRA alternative as follows:EQUATIONwhereW (l) down ∈ R D (l) ×1 and W (l) down ∈ R 1×D (l), unless otherwise stated. The HTA structure of the LoRA alternative is shown in Fig. 1 (b) .Analogously, we can derive HTA alternative to the adapter-based method (as shown in Fig. 1 (d) ) as follows:EQUATIONBy observing Eq. ( 6), we can see that LoRA-based method with HTA could be re-parameterized to the formW re-param = W (l) down W (l) up + W (l)HTA during the model inference stage. And also, the re-parameterizationW MHA re-param = W (l) o W (l)HTA of MHA in Eq. ( 7) is available due to the fact that the weight matrixW (l)o is positioned at the end of MHA. Similarly, the re-parameterization of FFN isW FFN re-param = W (l) FC2 W (l)HTA due to the weight matrixW (l)FC2 at the tail of FFN.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiments: 
 In this section, we present the experimental settings, comparison to existing solutions, and ablation studies to unveil the key properties of the proposed method.Datasets. We evaluated the effectiveness of our method using two sets of visual adaptation benchmarks: FGVC and VTAB-1k, involving a total of 24 datasets. The FGVC collection consists of five Fine-Grained Visual Classification (FGVC) datasets: CUB-200-2011, NABirds, Oxford Flowers, Stanford Dogs, and Stanford Cars. These datasets focus on distinguishing between visually similar subcategories within a broader category, making the task more challenging and detailed. The VTAB-1k benchmark comprises 19 diverse visual classification tasks, divided into three categories: Natural, which includes images captured by standard cameras; Specialized, which includes images captured by specialized equipment such as remote sensing and medical imaging devices; and Structured, which includes synthesized images from simulated environments, such as object counting and 3D depth prediction. Each VTAB-1k task includes 1,000 training samples.Pre-trained backbones. We employ ViT [13] and Swin Transformer [14] as backbone architectures to evaluate our proposed approach. To demonstrate the versatility of the proposed HTA model, we utilize two variants of ViT: ViT-Base and ViT-Large. These models are pre-trained on the ImageNet21K dataset [11] . Additionally, to ensure a fair comparison, we follow the settings from previous work [29] and conduct separate experiments using a ViT backbone enhanced with AugReg [34] .Baselines and existing PEFT methods. In our comparative analysis, we evaluate the performance of our HTA against two baselines and several state-of-the-art PEFT methods. Unless otherwise specified, our HTA follows the design in Eq. ( 6), with the dimension of the low-rank adaptation matrix set to 1. The two baselines we consider are: (1) Full Fine-tuning: This baseline involves updating all the parameters of the pre-trained model using the training data of the downstream task.(2) Linear Probing: This baseline focuses on learning a linear classification head on the downstream task while keeping the remaining parameters of the pre-trained model frozen. In addition to the baselines, we compare our method with the following state-of-the-art solutions: Adapter [22] , Bias [23] , LoRA [1] , VPT [24] , AdaptFormer [2] , FacT [26] , ARC [3] and RLRR [29] . The results are presented in Table 1 and Table 2 . Implementation details. Following previous work, we employed data augmentation during the training phase. For the FGVC datasets, we processed the images with a random resize crop to 224 × 224 and applied a random horizontal flip for data augmentation. For the VTAB-1k datasets, we directly resized the images to 224 × 224, adhering to the default settings in VTAB-1k. We used the AdamW [35] optimizer to fine-tune the models for 100 epochs. The learning rate schedule was managed using the cosine decay strategy. All experiments are conducted using the PyTorch framework [36] on an NVIDIA A800 GPU with 80 GB of memory.In this section, we conduct a comprehensive comparison of our method with other state-of-the-art approaches using different benchmarks and backbones. We evaluate the classification accuracy of each method across various downstream tasks and examine the number of trainable parameters during the fine-tuning phase.Comparison with the existing PEFT methods. We conducted a comparison of our method with other PEFT methods and baselines using two different benchmarks: FGVC and VTAB-1k. The results are presented in Table 1 and Table 2 . On the VTAB-1k dataset, our method not only demonstrates strong competitiveness compared to the baselines but also shows advantages over current state-of-theart methods. On many of the datasets, our method achieves the best performance with a reasonable parameter count. Compared to the previous state-of-the-art method, RLRR [29] , our method achieves superior overall performance while reducing the number of parameters by one-third. When using the AugReg-enhanced model, our lead is further amplified. In Table 2 , we further compare our method with others on the FGVC benchmark. While our method also achieves appealing performance on this dataset, the advantage is less evident. This is due to the fact that very high performance has already been achieved on this dataset, and the performance improvements have nearly saturated. Experiments on larger-scale ViT backbone. In addition to using the ViT-B backbone, we also employed the ViT-L backbone, which has a deeper block structure, to validate the scalability and generalizability of our method. The comparison results are shown in Table 3 . Our method achieves the best performance among all the compared methods while maintaining a reasonable parameter count. These results demonstrate that our method can effectively adapt models of varying scales and complexities in an efficient manner.Experiments on hierarchical Vision Transformers. To further validate the effectiveness of our method, we tested it on the Swin Transformer [14] . The Swin Transformer is renowned for its hierarchical design, consisting of multiple stages, each with transformer blocks that maintain consistent feature dimensions unique to that stage. As shown in Table 4 , our method notably outperforms existing state-of-the-art methods across various downstream tasks, with an overall improvement of 1.2% over the previous best performance while using only half of the parameters.To gain deeper insights into the proposed method, we conducted comprehensive ablation studies to elucidate its critical features and carry out pertinent analyses.Using HTA as alternative to low-rank based adaptation matrix. As mentioned earlier, our proposed HTA model offers a more flexible adaptation capacity compared to other low-rank based adaptation matrices. To validate this claim, we replaced the adaptation matrices of LoRA [1] and Adaptor [22] with HTA. Initially, following FacT [26] , LoRA [1] was originally applied to the {W q , W v } projection matrices in the multi-head attention operation of each ViT layer, while Adapter was applied to the feed-forward neural network components layer-wise, as described in Eq. ( 2). For a fair comparison, we also applied HTA separately to {W q , W v } and {W FC1 , W FC2 }. From the results in Table 5 , we observe that our method slightly outperforms LoRA but with significantly fewer parameters. Moreover, our method achieves significant improvement over Adapter still using much fewer parameters. These results indicate that our method achieves a better trade-off between adaptation performance and parameter efficiency. To further test the effectiveness of our method when using a similar parameter scale to LoRA, we additionally applied HTA to FFN layers. The results show that under the same parameter size, our method exhibits a noticeable improvement over LoRA.Ablation study on the bottleneck dimensionality of additive adaptation matrix in HTA. We conducted ablation experiments to verify the effect of incorporating low-rank adaptation matrices in HTA, as well as the impact of its bottleneck dimensionality. The results are presented in Fig. 2 . From these results, we observe that without the addition of low-rank adaptation, HTA experiences an obvious performance drop. This is due to the fact that while deriving orthogonal matrices using Householder transformations is parameter-efficient, their inherent dependence on a single chosen vector makes them insufficient as a set of general orthogonal bases for representing arbitrary highdimensional space. When using a low-rank adaptation matrix with rank 1, HTA shows a significant performance boost. This indicates that even with a simple low-rank adaptation, HTA can achieve a promising trade-off between adaptation performance and parameter efficiency. By incorporating these low-rank matrices, HTA can maintain high performance while being parameter-efficient. 
limitations: 
 In this work, we use Householder transformations to construct adaptation matrices in a parameterefficient manner. Although Householder transformation matrices are orthogonal, they cannot serve as general orthogonal bases in high-dimensional spaces due to their inherent dependence on a single vector. This limitation may reduce the adaptation capacity of the adaptation matrix composed of two Householder matrices. We address this issue by incorporating a rank-1 adaptation matrix, which may somewhat detract from the elegance of the proposed method. However, it is worth exploring strategies to eliminate the need for the additive adaptation matrix, thereby further enhancing the elegance and efficiency of the HTA method. 
conclusions: 
 In this work, we proposed a novel Parameter-Efficient Fine-Tuning (PEFT) solution. Our method addresses the limitation of fixed bottleneck dimensionality in low-rank based adaptation matrices, which can restrict adaptation flexibility. By viewing the adaptation matrix from the perspective of Singular Value Decomposition (SVD), we use Householder transformations to mimic orthogonal bases. These transformations, derived from a single vector, are highly parameter-efficient. We adaptively learn diagonal coefficients to flexibly combine two Householder matrices into an adaptation matrix, accommodating layer-wise variations. Theoretically, our method can generate adaptation matrices with varying ranks while maintaining a reasonable parameter size, offering a potential alternative to low-rank based adaptations. Experiments on two sets of downstream vision classification tasks demonstrate the effectiveness of our approach.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Please see our supplementary materials.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [NA] Justification: Our work does not involve specific error bars.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: This paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: The license and terms of use explicitly are mentioned and properly respected. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: We provide detailed information about the datasets used in this paper, including the number of classes and the sizes of the training, validation, and test sets, in Table 3 provides a summary of the configurations used in our experiments. As discussed in Section 4, we performed a grid search on the validation set of each task to determine the optimal hyperparameters, including learning rate, weight decay, batch size, and dropout rate. We provide further explanation of the ablation experiments in Section 4. In the study on the transferability of HTA, we replaced the low-rank adaptation matrices in LoRA, we used an HTA module to replace the bottleneck part of LoRA, while in Adapter, we directly replaced the Adapter with an HTA module. The detailed experimental results are presented in the table 6. In the study of the low-rank adaptation part of HTA, we set its dimensions to 0, 1, 2, and 4, respectively. The results are shown in Table 7 .Practicality: Our approach differs from traditional methods by employing Householder transformations rather than standard unitary matrices, which can be efficiently derived. This approach boosts the efficiency of parameter usage and significantly cuts down on the number of parameters requiring fine-tuning. With this technique, we manage to achieve high performance while optimizing parameter use. Leveraging large-scale pre-trained models, our HTA method proves to be both highly efficient and practical across diverse applications.Low Energy Consumption: Our approach enhances the model's computational efficiency by decreasing the necessary computational parameters, thus reducing energy usage during training. This reduction aids in conserving energy and lowering emissions, aligning with global sustainability goals and the push for eco-friendly practices. Moreover, our method not only improves the model's Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: The reviewers unanimously voted for acceptance. They appreciated the technical originality, clarity and extensive experiments that support the efficacy of the proposed approach. After carefully reading the reviews and the rebuttal, this meta-reviewer concurs with the reviewers' recommendations.

Reviews: review: paper_summary: paper_summary: The paper presents a new parameter efficient fine tuning (PEFT) technique for vision transformer (ViT) models. The work utilizes the intuition of creating an adaptation matrix for fine-tuning from a popular dimensionality reduction technique named singular value decomposition (SVD) which is named Householder transformation based Adapter (HTA).

HTA can be utilized to create adaptation matrices to accommodate layer wise variations. Extensive empirical evaluations have been performed over various popular vision benchmarks. Code has been made available for reproducibility.
strength_weakness: strength_weakness: No weaknesses apart from the limitations mentioned in Section 5 by the authors
questions: questions: Do you think the HTA idea is adaptable to other modalities for PEFT?
limitations: limitations: NA
paper_summary: paper_summary: The paper presents a novel method, HTA, for efficiently adapting pre-trained transformers by applying the Householder transformation to a single vector to approximate the SVD for representing the adaptation matrix. By combining this with an additional rank-1 adaptation matrix, HTA achieves a reduction in the number of parameters while maintaining strong performance across various benchmarks, including FGVC and VTAB-1k, and outperforms existing methods in terms of efficiency and performance.
strength_weakness: strength_weakness: Originality:
Utilizing Householder transformations for matrix decomposition is not a brand-new idea.

Quality:
The improved performance of HTA over the state-of-the-art methods on several benchmarks is not very significant (<0.5%).

Clarity:
A scatter plot for accuracy versus the number of parameters tradeoff would give a clearer insight into HTA's performance.

Significance:
1. The challenge of solely relying on Householder transformations for adapting pre-trained transformers remains.
2. The proposed HTA removes the rank choice in previous adaptation methods but introduces a new rank choice in the additional adaptation matrix, although rank 1 might be sufficient for a good performance.
questions: questions: 1. Fig. 1c has a skip-connection, but Fig. 1d does not. Is it proposed to be like this?
2. The proposed HTA includes two feature paths, with both the Householder transformations path and the other down-up projection path being rank 1. It remains unclear why the additional down-up projection path would help significantly boost performance.
3. The results in Fig. 2 with dimension=0 are different from the ones in previous tables. What are the differences in settings between them?
For Swin, it would be better to include which specific Swin model is applied.
4. For Swin, it would be better to include which specific Swin model is applied.
limitations: limitations: The novelty and significance of HTA are somewhat limited. However, the extensive experimental results provide a robust foundation for the claims made in the paper. Overall, the work is well-executed and makes a valuable contribution to the field, justifying a recommendation for weak acceptance.
paper_summary: paper_summary: This paper proposes a new PEFT method by integrating Householder transformations, which enables the creation of adaptation matrices that have varying ranks across different layers, thereby offering more flexibility in adjusting pre-trained models.
strength_weakness: strength_weakness: I am not an expert in the field of PEFT, it appears that the authors have proposed a new PEFT scheme. From an application perspective, we desire PEFT methods that can transfer the pre-trained models to downstream tasks with as few parameters as possible. In the experiments conducted by the authors, the proposed method achieved slight improvements with relatively fewer parameters. In fact, all methods utilize a minimal number of parameters.

In addition, the authors did not make comparisons in more complex tasks, such as object detection and semantic segmentation. Currently, numerous related studies employ a backbone based on the ViT [A,B,C]. Validation solely on classification tasks is somewhat limiting.


[A] Exploring Plain Vision Transformer Backbones for Object Detection.

[B] SegViT: Semantic Segmentation with Plain Vision Transformers.

[C] ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation
questions: questions: Is the proposed method effective in tasks such as object detection and semantic segmentation?
limitations: limitations: The novelty, contribution and experimental comparison of this paper all have certain shortcomings.
paper_summary: paper_summary: The study introduces a data-driven variant of LoRA-like Parameter-Efficient Fine-Tuning (PEFT) for Vision Transformers (ViTs), which learns the optimal bottleneck dimension (the rank of the fine-tuning weight matrix) instead of preconfiguring it for each layer. The authors propose learning three additional vectors: $v_{\text{left}}$, $v_{\text{right}}$, and a vector of diagonal parameters $D$ that define the rank. Inspired by Singular Value Decomposition (SVD), they employ Householder transformations $I-vv^\intercal$ on $v_{\text{left}}$, $v_{\text{right}}$ to construct orthogonal matrices. These learned vectors and parameters $v_{\text{left}}$, $v_{\text{right}}$, and $D$ can then be fused into the frozen weights in a LoRA-like manner. Jointly training $v_{\text{left}}$, $v_{\text{right}}$, and $D$ alongside an extremely low-rank LoRA branch (r=1) results in improved performance on downstream tasks.
strength_weakness: strength_weakness: While the proposed method is parameter-efficient, the discussion overlooks the actual training cost and latency. For instance, it would be beneficial to include theoretical or actual memory and computational costs.
questions: questions: - What rank is applied to the LoRA-based baselines in Table 1? It would be beneficial to list these parameters in the appendix.

- What are the settings used in Table 5? Are the LoRA branches also applied in the HTA row?

- Could the authors include a section discussing theoretical and actual memory computation costs, such as FLOPS?

- Is it feasible to extend the method to Large Language Models?
limitations: limitations: As the author discussed in the limitation section.

Comments: title: Thank you for your recognition of our work
comment: Thank you for the detailed response. I have updated my vote to accept.
thank you for your response.
Thanks for the authors' responses, and I learned a lot through this paper. I have adjusted my score accordingly, and vote for acceptance of this paper.
Thank you very much for your valuable comments and recognition of our work.
Thank you for the response. The additional experiments demonstrate the generality and effectiveness of the proposed method for language models as well as various tasks in computer vision. The paper addresses the issue of fixed bottleneck dimensionality in the LoRA-based method using Householder Transformation, which, to the best of my knowledge, is novel. While the method eliminates the need for bottleneck dimensionality tuning, my primary concern is that the experiments show comparable (or only marginally improved) performance compared to a fixed-rank LoRA (with a rank of 8) at the cost of increased computational resources. Although the proposed method uses significantly fewer parameters, the vectorized parameters are expanded into Householder matrices during training, which requires additional gradient computations. Based on the generality of the proposed method demonstrated in the additional experiments and the technical contribution of the paper, I will maintain my current rating.
Thank you for your prompt response. In our global response, we highlighted the major points addressing the questions but did not provide the detailed experimental results on common sense reasoning, object detection, and segmentation as mentioned. To clarify the additional experimental results requested, we have provided the tables below. Please note that the detailed experimental settings and illustrations can be found in the response to Reviewer #YBq9.

| Method                             | Backbone          | Tuning Method | Params (M) | Acc  | Task                |
| ---------------------------------- | ----------------- | ------------- | ---------- | ---- | ------------------- |
| LlamaForCausalLM                   | llama             | LoRA (d=32)   | 25.16      | 69.8 | Commonsense Reasoning |
|   LlamaForCausalLM     |    llama    | HTA  (d=6)    | 5.89       | 70.1 | Commonsense Reasoning |

| Method                             | Backbone          | Tuning Method | Params (M) | mIoU | Task                |
| ---------------------------------- | ----------------- | ------------- | ---------- | ---- | ------------------- |
| Swin-Transformer+UperNet           | Swin-Transformer  | LoRA (d=8)     | 1.60       | 46.1 | Semantic Segmentation |
|  Swin-Transformer+UperNet       |  Swin-Transformer    | HTA (d=1)           | 0.46       | 47.6 | Semantic Segmentation |

| Method                             | Backbone          | Tuning Method | Params (M) | mAP  | Task            |
| ---------------------------------- | ----------------- | ------------- | ---------- | ---- | --------------- |
| Swin-Transformer+Cascade Mask RCNN | Swin-Transformer  | LoRA (d=8)    | 1.60       | 44.2 | Object Detection |
|Swin-Transformer+Cascade Mask RCNN  | Swin-Transformer  | HTA (d=1)           | 0.46       | 47.2 | Object Detection |
| ViTPose                            | ViT               | LoRA (d=8)    | 0.59       | 73.6 | Pose Estimation  |
|ViTPose |     ViT           | HTA  (d=1)          | 0.19       | 74.1 | Pose Estimation  |
Thank you for your responses. Could you kindly point out the tables that contain the experiments on common sense reasoning, object detection, and segmentation mentioned in the global response?