Maintaining Structural Integrity in Parameter Spaces for Parameter Efficient Fine-tuning
Authors: Chongjie Si
Xuehui Wang
Xue Yang
Zhengqin Xu
Qingyun Li
Jifeng Dai
Yu Qiao
Xiaokang Yang
Wei Shen
Venue: ICLR 2025 Poster (2025)
Keywords: parameter efficient fine-tuning
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: A PEFT method aiming to preserve the topological structure of N-dimensional parameter space while seeking low-rank representation.

Abstract: Adapting pre-trained foundation models for various downstream tasks has been prevalent in artificial intelligence. Due to the vast number of tasks and high costs, adjusting all parameters becomes unfeasible. To mitigate this, several fine-tuning techniques have been developed to update the pre-trained model weights in a more resource-efficient manner, such as through low-rank adjustments. Yet, almost all of these methods focus on linear weights, neglecting the intricacies of parameter spaces in higher dimensions like 4D. 
Alternatively, some methods can be adapted for high-dimensional parameter space by compressing changes in the original space into two dimensions and then employing low-rank matrix adaptations. However, these approaches destructs the structural integrity of the involved high-dimensional spaces. To tackle the diversity of dimensional spaces across different foundation models and provide a more precise representation of the changes within these spaces, this paper introduces a generalized parameter-efficient fine-tuning framework, designed for various dimensional parameter space. Specifically, our method asserts that changes in each dimensional parameter space are based on a low-rank core space which maintains the consistent topological structure with the original space. It then models the changes through this core space alongside corresponding weights to reconstruct alterations in the original space. It effectively preserves the structural integrity of the change of original N-dimensional parameter space, meanwhile models it via low-rank tensor adaptation. Extensive experiments on computer vision, natural language processing and multi-modal tasks validate the effectiveness of our method.

Published as a conference paper at ICLR 2025
MAINTAINING STRUCTURAL INTEGRITY IN PARAME- TER SPACES FOR PARAMETER EFFICIENT FINE- TUNING
Chongjie Si Xuehui Wang", Xue Yang'. Zhengqin Xu',
Qingyun Li?,
Jifeng Dai Yu Qiao?. Xiaokang Yang', Wei Shen
MoE Key Lab of Artificial Intelligence, Al Institute, Shanghai Jiao Tong University OpenGVLab, Shanghai AI Laboratory
[chongjiesi, wei. scinsheejole 0s ftu.:ttduuuccoo. edu.
ABSTRACT
Adapting pre-trained foundation models for various downstream tasks has been prevalent in artificial intelligence. Due to the vast number of tasks and high costs, adjusting parameters becomes unfeasible. To mitigate this, several fine- tuning techniques have been developed to update the pre-trained model weights in a more resource-efficient manner, such as through low-rank adjustments. Yet, almost all of these methods focus on lincar weights, neglecting the intricacies of parameter spaces in higher dimensions like 4D. Alternatively, some methods can be adapted for nigh-dimensional parameter space by compressing changes in the original space into two dimensions and then employing low-rank matrix adaptations. However, these approaches destructs the structural integrity of the involved high-dimensional spaces. To tackle the diversity of dimensional spaces across different foundation models and provide more precise representation of the changes within these spaces, this paper introduces generalized parameter- efficient fine-tuning framework, designed for various dimensional parameter space. Specifically, our method asserts that changes in each dimensional parameter space are based on low-rank core space which maintains the consistent topological structure with the original space. It then models the changes through this core space alongside corresponding weights to reconstruct alterations the original space. It effectively preserves the structural integrity of the change of original N-dimensional parameter space, meanwhile models it via low-rank tensor adaptation. Extensive experiments computer vision, natural language processing and multi-modal tasks validate the effectiveness of our method.
INTRODUCTION
The recent introduction of foundation models Brown et al. (222)); Kirillov et al. (2023) :: Devlin et al. (2018): Liu al. (2019) has demonstrated unparalleled performance and potential across diverse domains in artificial intelligence. Traditionally, the adaptation of pre-trained models for downstream tasks achieved through fully fine-tuning of all parameters Ma et al. (2024); |Raffel et al. (2020); Qiu al.] (2020). However, as the parameter count of these foundation models escalates, the conventional approach to fully fine-tuning becomes prohibitively expensive in various aspects.
To tackle this challenge, recent works Chen et al. (2024a); Guo et al. (2020); He et al. 22021a):
Hu et al. (2021a) have focused on the concept of sarameter-efffcicee fine-tuning (PEFT), aiming to minimize the number of adjustable parameters required while achieving optimal task performance. These works specifically explore methods to model the incremental update of pre-trained weights in manner that is economical in terms of parameters, without necessitating changes to the model's s architecture Zaken et al.) (2021): Guo et al.] (2000); Hu et al.) (2021a). Among these works, LoRA Hu al. (2021a) emerges pioneering effort, which proposes adopt an additional term with a low-rank structure to the original weight. Specifically, the original weight matrix Wa E IRd: xd2
"Equal contributions.
1 "ooresponniig author.

---

Published as a conference paper at ICLR 2025
2- 2-Dimension Parameter Space
High Dimension Parameter Space
(Linear Layer)
(2D Convolution for example)
Change of Parameler Space
LoRA
adaptation
A
aduptation
reshape
Low-man Matrix
destroy locality
Lincar Core
FLoRA
A
adaptation
adaptation
Convolurion Core
preserve loculity
Weight Matrix
Figure 1: Difference between LoRA and FLORA. LoRA employs low-rank matrix adaptation for each dimensional parameter space. However, for parameter space of convolution layer, the reshaping operation required by LoRA causes adjacent elements within the kernel be separated in the matrix, disrupting the spatial locality inherent in the original convolutional space. Conversely, FLoRA asserts that the alternations of each dimensional parameter space has low-rank consistent topological structure. This framework enables FLoRA to effectively preserve the structural integrity of the original parameter space, such as maintaining the spatial locality in convolutional operations.
space with the
remains frozen, while a learnable low-rank A AW is added Wo. with the form as Wo Wo + AW = Wo + BAT
(I) where E Rd and E xr with r < (d1, d2 Given that significantly smaller than the dimension of Wo. LoRA necessitates the updating of only limited number of trainable parameters, while achieving comparable performances. Subsequent to LoRA, recent studies He et al. (2021a); Bershatsky et al. (2024); Zhang et al. (2022a) endeavor to explore more efficient low-rank matrix adaptation methods concerning the matrix A..
However, have observed intriguing phenomenon: significant portion of existing works is narrowly focused on two-dimensional parameter spaces (i.e., linear layers), neglecting the existence of other dimensional spaces such 2D convolution (4-dimension) layers, etc. But in practice, all layers are linear across various models for downstream tasks. For example, ConvNeXt Liu et al. models predominantly utilizing 2D) convolutional layers. Alternatively, some approaches can be adapted for nigg--dimessionll parameter spaces by directly adopting low-rank matrix adaptation. They models alternations in high-dimensional spaces into two dimensions, while neglecting the structural complexities of the original spaces. For instance, as detailed Sec. [2.1 and Fig. LoRA Hu al.) (2021a) compresses the changes in convolutions, which is four-dimensional parameter space, into two dimensions. It subsequently applies low-rank matrix adaptation to the two-dimension space, intending to represent the alterations of the original four-dimensional parameter space. Yet, as discussed in Sec. 2.2] this adaptation fails to capture the inherent complexity and spatial locality specific to convolution operations. The result is reshaped two-dimensional structure that compromises the integrity of the original parameter space, leading to representation that does not fully encapsulate the change of convolutional space.
(2022) and Stable Diffusion Carreira & Zisserman (2017)
To this end, in this paper, propose low-rank tensor adaptation based method, FLoRA, represented as Fundamental -Ow-Rank Adaptation. Positioned as a superior alternative to LoRA, FLoRA meets the ensuing three properties:
It can identify an appropriate low-rank representation for the changes in various dimensional parameter spaces, without destructing the structural integrity of the original parameter spaces.
It can maintain a consistent formulation across various dimensional parameter spaces.
When applied linear weights, with the same parameter budget, it requires similar training time and resources to that of LoRA, yet yield superior performance.
Specifically, since a much lower rank than the direct rank of parameter space is sufficient to represent the original space Aghajanyan et al. (2020) Li al. (008)) FLoRA asserts that the alternations of
2

---

Published as a conference paper at ICLR 2025
each dimensional parameter space, whether 2D or 4D, have corresponding core space is low-rank and retains the same spatial dimensions (i.e., 2D or 4D) as the original parameter space, suggesting that they share consistent topological structure. FLoRA then models the alternations by using this core space combined with a series of weights to reconstruct alternations in the original parameter space. Thanks to the intrinsic properties in the structure of the core space, FLoRA efficiently maintains the structural integrity of the original parameter space. Extensive experiments are conducted with several pretrained models on computer vision, natural language processing and multi-modal tasks, which validates that regardless of the model, the kind of downstream task, or the dimensionality of the parameter space, FLoRA's performance surpasses that of LoRA and other existing methods.
space. This
The contributions of this paper are as follows:
We propose novel PEFT method, FLoRA. To the best of our knowledge, it the first time that PEFT method has been designed for different dimensional parameter spaces, aiming to preserve their topological structure while seeking low-rank representations.
Extensive experiments on different tasks, include computer vision, natural language process- ing and multi-modal tasks, demonstrates that FLoRA significantly surpasses other baselines, validating the effectiveness of FLoRA.
2 PRELIMINARIES
2.1 Low RANK ADAPTATION
LoRA Hu et al. 2021a) models the incremental update of a pre-trained weight matrix Wn E Rd xd2
by the product of two small matrices A E pd2xxr and B E gdixr, where < {di, d2). For = Wox, the modified forward pass is
h = Wax + AWx = Wox + BA
(2)
Matrix A is initialized with a random Gaussian distribution, and B with zeros, setting the initial AW zero for training. LoRA's application is straightforward the linear layers, while for a convolution layer characterized by weights We E R d x daus xkk with din / dout denoting the dimension of input output respectively and & representing the kernel size, LoRA is adapted based on matrix decomposition:
W = W. + AW We + Reshape B. Wc),
where A, E X and B E kdi, xr_ Here, Reshope(Bi [..]] A. W.) involves altering the dimen- sions B.A. to match those W.. It is obvious that LoRA unfolds the original 4-dimensional parameter space AW into 2-dimensional space, subsequently leveraging the low-rank approxima- tion of this 2-dimensional space to represent the original 4-dimensional construct. As shown later, this low-rank matrix adaptation method will destruct the structural integrity of the convolution layer.
2.2 WHY MATRIX ADAPTATION BREAKS THE STRUCTURAL INTEGRITY OF THE CONVOLUTION?
The changes of a high-dimension tensor such as convolutional parameter space can be modeled based on low-rank matrix adaptation following Eq. (3) However, during the reshaping process, elements that are adjacent within the convolutional kernel are those dispersed across various positions in the original matrix. More specifically, elements that were located across multiple rows or columns of the matrix for reshaping. This shift poses significant challenges in learning spatial correlations among elements positioned disparately. Therefore, such a transformation disrupts the principle of locality inherent in the original convolutional operation, where each output element is determined by a small region of the input.
localized within the convolutional kernel
METHOD
In this section, we first introduce the formulation of FLoRA in an N-dimensional parameter space. Specifically, for pre-trained weight Wo E RR4xx x4xxxxx..xIx with N-dimension, we update the weights
3

---

Published as a conference paper at ICLR 2025
Wo with the changes AW
Wo + Wo + AW G X2 A x ... X A (N)
(4)
without loss of generality. Here G E p2xx/x--xx>> and A(") E g/,x",, where Jn < In- JI.....JJ JN are eyper-parameters, and X denotes the mode-n product between a tensor and a matrix. FLoRA considers the tensor 9 as low-rank
space with the consistent topological structure to the original parameter space, with A (x) representing the weights associated with each dimension. We then scale AW by with being a constant. In the subsequent subsections, we will detail its manifestations in convolution and lincar layers.
3.1 FLORA FOR CONVOLUTION LAYER
Convolution operations in deep learning are characterized by a four-dimensional parameter space, encapsulated weight tensor W Rd Xdoue xkxk, with dan / dout denoting the dimension of input/ output respectively and k representing the kernel size. A pivotal property to consider within W. is the spatial locality, which plays an essential role in the convolution layer's ability to compile and process information across the input matrix. This process is facilitated by the kernel's spatial dimensions (k x k), which determines the scope of the input data scrutinized by each convolution operation. To preserve the attributes of the spatial locality and uphold the convolution parameter space's integrity, FLoRA models the update AW for a convolution layer as
AW = 9 x1 A X2 B X3 C X111 D,
where G P A Xd B Rddutt and C, Rexrs, rı and 72 two hyper-pare parameters which are significantly smaller than {dix, dout). r3 is another that smaller than the kernel size of convolution layer. Given that x 3 convolution is prevalent configuration in convolutional foundation models Woo et al. (2023) Rombach et al. (2022) ) Wang
al. 2023a) F3 is consequently chosen from {1,2}.
The core tensor G in FLoRA can be viewed as a compressed convolution parameter space. In essence, it serves as a space for convolution. This means that in any convolution layer, there exists a convolution core, and what FLoRA aims to do is to determine this convolution core for each convolution space and configure the corresponding weights A, B. C and D for reconstructing that space. Different to low-rank matrix adaptation based methods, FLoRA does not need to alter the structure of the convolution. Alternatively, by learning the convolution core, FLoRA effectively preserves the convolution's 's property of spatial locality.
Furthermore, while preserving or potentially augmenting the representational power of the convolution process, FLoRA achieves remarkable reduction in the number of trainable parameters in comparison to LoRA. Assuming that the rank for both the input and output dimensions is uniform (r) = 122 = r), the parameter requirement for FLoRA is calculated as O(rgr² + r(d.n + dout) + 2r3k) parameters, while LoRA necessitate training at least O(rk (din + dout)) parameters. Given that typically r, << {dins dout), }. therefore, FLoRA exhibits better parameter efficacy than LoRA as the number of the kernel increases.
3.2 FLORA FOR LINEAR LAYER
For linear layer with weight Wo Rd xd2 FLoRA models the the update A AWa as
AW = 9 Ax T
(6)
where G R A Rdix and B R2 Similar to that for convolution layer, the matrix G can be viewed as a core space for the 2-dimension parameter space, with A and B being the corresponding weights reconstruct the alternations in linear space.
EXPERIMENT
4.1 MODELS AND DATASETS
We conduct comprehensive experiments across computer vision (CV), natural language processing (NLP) and multi-modal (MM) tasks.
4

---

Published as a conference paper at ICLR 2025
Specifically, for CV tasks, employ FLoRA to fine-tune the Conv V2-L Woo et al. (2023).
evaluating it on MS COCO Lin et al. (2014) by using Mask R-CNN He et al. (017) implemented in MMDetection Chen et al. (2019). and remote sensing image datasets DOTA Xia et al. 2018 with
Oriented R-CNN Xie et al. (2021) based on MMRotate Zhou et al. (002), and on the ADE20K /houu
et al. 2017) dataset thanks to UperNet OpeaN al. (2018) integrated in MMSegmentation Contributors (2020). We also employ FLoRA to fine-tune large-scale vision foundation model, i.c. ViT--6 Chen et al. ) on ADE20K datasets. Detailed hyper-parameter settings can be find in Table 9 in Appendix.
For NLP tasks, we evaluate the DeBERTa V base He al. (2021b) with FLoRA on the General Language Understanding Evaluation (GLUB) Wang et al. (2018) benchmark, which includes two single-sentence classification, three similarity and paraphrase and four natural language inference datasets. More details on GLUE dataset can be found in Table T0 in the Appendix.
[Liu et al. (2024a), which is composed of language model, Vicuna- 1.5-7B Peng et al. (2023) and vision encoder, CLIP ViT-L/336px RRadford et al. (2021), on visual instruction tuning tasks, which include seven vision- language benchmarks: VQA Goyal et al. (017). GQA Hudson & Manning (011)). VisWiz Gurari ct al. (2018), SQA et al. 2022). VQA Singh et al. (2019). POPE Li et al. 2023 and MMBench
For multi-modal tasks, we employ FLoRA fine-tune
Liu et al. (2023).
Moreover, we fine-tune SDXI Rombach et al. (002)) with FLoRA for generation task in Appendix A.1. LLaMA3-8B AI@ Meta 2024) for commonsense reasoning tasks in Appendix A.2 and ViT-B Dosovitskiy et al. (2021) for VTAB-IK 1K benchmark Zhai et al. (2019) in Appendix A.3
4.2 BASELINES
We compare FLoRA with several state-of-thh-ar methods: fully fine-tuning, BitFit Zaken et al. (2021
HAdapter Houlsby et al. (2019), PAdapter Pfeiffer et al. (2020). AdaLoRA Zhang et al. (2022a). the most representative low-rank method, LoRA|Hu et al. (2021a). and the state-of-the of-the-art art low-rani adaption method, DoRA Liu et al. (2024b). Specifically, HAdapter is strategically placed between the self-attention module and the FFN module, and includes a subsequent residual connection. Conversely, PAdapter introduces more streamlined design, implementing adapters exclusively after the FFN modules and LayerNorm modules. Furthermore, following Zhang et al. (2022a), we apply AdaLoRA, LoRA and DoRA to all weight matrices or tensors. For more details the baselines, please refer to their original papers.
4.3 IMPLEMENTATION DETAILS
We compare FL.RA with other PEFT methods under different parameter budgets. The hidden dimension of Adapters is chosen from (32, 64), the budget of AdaLoRA is set as { {144, 288, 576} and the rank r of LoRA and DoRA selected from (2, 4, 16, 32). Other hyper-parameters are initialized according to their original papers.
Additionally, we simply set r' rı = r2 for FLORA. For CV tasks, when fine-tuning ConvNeXt- V2-L, we set = 2 and s = 4. We adjusted FLoRA's r to align its parameter budget closely with
CLLA
0000 AP
030
5S12
0600
BER:
Bift
HAdapter
Pkdapta
AdalcTA
Lo12
Dotts
FinRa
MWC
BOTA-WA
Figure 2: The normalized performance improvement of FLoRA over other baselines
5

---

Published as a conference paper at ICLR 2025
that of other methods for fair comparison. For different methods, fine-tune all the convolutional layers for ConvNeXt- V2-L (and SDXL). When fine-tuning IntervViT-6B. FLoRA's r is set to {16, 32). 0.04, and we fine-tune all the linear layers for different methods. For NLP tasks, FLoRA's set to (2. 8), and 8 = 0.4. The mean of runs using various random seeds are reported for all the experiments, and all gains have passed the pairwise 1-test with significance of 0.05. We fine-tune q. k, v, up and down matrices for all methods. For MM tasks, we set 128, 1.5 and fine-tune all linear layers. More training details can be found Appendix.
We use publicly available PyTorch Paszke et al. (2019) implementation to execute all the baseline comparisons, and all the experiments are conducted on NVIDIA A100 GPUs. The tensor ç is initialized as zero, while other weight matrices are initialized as random Gaussian. For more training details, please refer to the Appendix.
Table 1: Results with ConvNeXt-Ve V2-L Woo et al. (203)) fine-tuned on different datasets. The best performances are shown in bold. "Base" represents for the pre-trained backbone with frozen weights.
COCO
ADE20K
DOTA
All
Method
Params
mAP
AP
AP7s
APs
APM
AP1
mAce
mloU
mAP
AP
APr
Avg.
Base
37.3
63.3
39.7
27.8
41.0
46.2
59.6
48.5
31.4
61.8
27.3
44.0
Fully FT
196M
52.7
74.3
58.7
38.3
56.9
67.3
64.7
53.1
33.9
59.9
34.0
54.0
BitFit
0.2M
43.1
67.6
47.4
29.5
46.7
55.3
61.2
49.1
34.6
64.2
32.9
48.3
LORA
12.94M
47.4
70.3
53.0
32.4
51.8
61.2
63.6
51.4
18.3
36.2
16.3
45.6
DoRA
13.07M
47.2
69.8
52.7
32.1
51.5
61.4
63.0
50.9
19.6
37.9
17.2
45.8
FLORA
12.77M
48.1
71.1
53.6
33.1
52.3
62.3
64.1
51.9
37.3
65.6
37.7
52.5
LoRA
25.89M
48.0
70.4
53.6
33.0
52.3
62.8
63.9
51.4
20.0
38.3
18.3
46.5
DORA
26.16M
48.1
70.7
53.6
33.1
52.
62.6
64.0
51.9
21.1
39.7
19.1
46.9
FLORA
25.65M
49.2
71.7
54.7
34.3
53.3
63.5
65.0
52.6
38.8
68.4
39.5
53.7
LoRA
51.78M
48.2
70.7
53.7
33.4
52.5
62.7
63.9
51.6
20.4
39.4
19.1
46.9
DoRA
51.95M
44.0
68.2
48.9
29.1
48.2
57.5
64.1
51.9
21.7
40.9
20.3
45.0
FLORA
40.49M
50.4
72.6
56.2
35.4
54.6
64.8
65.1
52.8
39.7
69.0
40.9
54.7
Table 2: Results with Intern VTT-6B
Chen et al.
2023
fine-tuned ADE20K.
Method
Base
Fully FT
BitFit
LoRA
DoRA
FLoRA
LoRA
DoRA
FLoRA
# Params (%)
100
0.13
0.32
0.32
0.32
0.66
0.66
0.66
mAce
64.5
70.4
68.7
69.7
69.5
70.1
70.3
69.6
71.1
mIoU
53.0
58.2
56.1
57.1
57.3
57.7
57.3
57.1
58.0
Avg.
58.8
64.3
62.4
63.4
63.4
63.9
63.8
63.4
64.6
Table 3: Results with DeBERTaV3--an
He et al.
b021 fine-tuned on GLUE datasets.
Method
# Params
MNLI
S8T-2
CoL.A
Mec
QQP 92.40
QNLI
Acc
RTE
Acc
MRPC Acc
STS-B
Corr
Avg.
All
Fully FT
184M
89.90
95.63
69.19
94.03
83.75
89.46
91.60
88. 24
BiFF
0.TM
89.37
94.84
66.96
88.41
92.24
78.80
87.75
91.35
86.21
HAdapter
122M
90.13
95:33.11
68.64
91.21.0
9.11.1
84.88
89.95 21.1
91.48 3
88.19
PAdapter
1.1MM
90.33
95.61
68.774.0
91.40..1
94.29
85.20
89 89.46
91.54_.7
88.32
AdaLoRA
1.33M
90.38
95.874.a
71,45
91.194.2
94.364.3
88.09
90.69-1.8
89.23
LoRA
0.33M
90.03..a
93.92
69.15
90.61
93.37
85.56
90.19..7
90.75
87.95
DoRA
0.41M
90.21
94.38..5
69.33
90.84
93.26
86.94
90.19-1.2
91.34_
88.31
FLORA
90.60 2.2
96.00
70.20-1.11
91.40
94.46
90.93
91.96
LORA
1.33M
89,80
93.69
69.30
69.30-1.1
91.78.11
9I97
83.70
90.68
91.62
$8.17
DoRA
1.41M
89.67
94.61
69.08--14
91. 91.80..1
93.23
87.33-1.
90.68 1.0
91.734.5
88
FLORA
1.33M
90.82 1
96.21
72.05 2.3
91.94.1
94.60
89.53
91.18
92.04.
89.80
4.4
MAIN RESULTS
We compare FLoRA with other baselines under different parameter budgets. Specifically, ConvNeXt- V2-L is based on convolutions, and other large foundation models are dominantly based on linear layers. We evaluate FLoRA's effectiveness high-dimension space for the CV tasks, and linear parameter space for CV and NLP tasks. The results are shown in Tables. 4 and Tables. 7 in Appendix, and the normalized performance are illustrated in Fig.
For CV tasks, FLoRA employed on ConvNeXt- V2-L achieves superior performances compared other baselines. On average, FLoRA outperforms LoRA and DoRA by least 15 % under

---

Published as a conference paper at ICLR 2025
Table Results with LLaVA-1.5-7B
Liu al. (2024a)
fine-tuned on visual instruction tuning.
Method
Params (%)
VQA 22
GQA
VisWiz
SQA
VQA T
POPE
MMBench
Avg.
Fully FT
100
78.5
61.9
50.0
66.8
58.2
85.9
64.3
66.5
LoRA
4.61
79.1
62.9
47.8
68.4
58.2
86.4
66.1
66.9
DoRA
4.63
78.6
62.9
52.2
69.9
57.0
87.2
66.1
67.6
FLoRA
4.63
78.6
62.9
51.7
70.1
57.4
87.6
66.0
67.8
different parameter budgets. With nearly 80% reduction in parameter budget, FLoRA (12.77M) still significantly outperforms LoRA (51.78M), DoRA (51.95M), validating that FLoRA successfully preserves the structural integrity of convolutions. Specifically, FLoRA achieves comparable or even superior performance to fully fine-tuning, while others lag far behind. Additionally, we observe that on tasks with large domain gap from the pre-training data, e.g. remote sensing images (DOTA), LoRA and DoRA performs significantly worse, while FLoRA maintains consistent superiority. This further validates the robustness of FLoRA, even when facing tasks with large domain gaps. Moreover, when FLoRA is employed to fine-tune Intern VīT-6B, FLoRA consistently outperforms the baseline across various parameter budgets. Remarkably, by fine-tuning only 0.66 % of the parameters, FLoRA achieves even better performance than fully fine-tuning.
For NLP tasks, FL.RA achieves better on all datasets under all different parameter budgets. Specifically, under extremely low parameter budget, FLoRA performs better even than the baselines with higher parameter counts. For example, with 0.3M parameters, FLoRA's performances on SST-2, QNLI, RTE, MRPC and STS-B are all better than baselines with larger parameter budgets.
par performance compared with existing approaches
For modai tasks, FLoRA also achieves SOTA performances compared with baselines. Therefore, this point, we can conclude that overall, FLoRA achieves remarkable performances across various tasks, model backbones, and dimensions of parameter spaces.
5 FURTHER ANALYSIS
In this section, we explore the properties of FLoRA when applied to downstream tasks and in comparison with other baselines like LoRA. We earry out series of empirical studies address the following questions: 1) Is the core space truly low-rank? 2) If so, why is FLoRA's low-rank representation better than other methods like LoRA? 3) Does FLoRA require acceptable training costs compared to other methods? 4) Is FLoRA sensitive to the scaling factor? The insights gained from these questions will illuminate the efficacy FLoRA and guide future research.
PL-b
535
LORA
Lonk
713
235
DxtA
3 565
51C
64
Le5A
RV
RaFk
Pork
(a) COCO
(b) CoLA
(c) RTE
(d) MRPC
Figure Fine-tuning with FLoRA under different rank (parameter budgets).
5.1 IS THE CORE SPACE TRULY LOW-RANK?
We present the performance FLoRA under different rank (i.c., parameter budgets), tested on ConvNeXt-V2-L and DeBERTa
The results are illustrated in Fig. 3 It is evident that FLoRA's performance with a low rank comparable to, or even exceeds, that of a higher rank. This finding aligns with observations in Hu et al. (2021a). la). It indicates that there indeed exists core space in different dimensional spaces, and the rank of the core space is relatively small. When the rank set smaller than the rank of the space, the performance of the model is not optimal. Conversely,
7

---

Published as a conference paper at ICLR 2025
when it exceeds this rank, the space is completely covered, which introduces meaningless redundancy and noise. Additionally, we observe that for the convolutional parameter space, the rank of its space is much larger, for the convolutional space has more complex topologiaa structure, necessitating a larger rank to adequately describe it.
5.2
WHY IS FLORA'S LOW-RANK REPRESENTATION BETTER THAN OTHER METHODS?
Although methods like FL.RA and LoRA can represent changes in the original parameter space using low-rank representations, FLORA performs better, indicating that its low-rank representation is superior to other forms of low-rank representations. We will explain the reasons as follows. We also provide comprehensive analysis in Appendix B.1
In linear parameter spaces, the changes modeled by methods in the LoRA derivatives, can also be simply considered as AGB with G being diagonal matrix. However, FLoRA does not impose any constraints on G. i.e. matrix pattern, which offers several advantages: firstly, it broadens the range of parameter adjustments and enhances the flexibility of parameter learning. Secondly, FLoRA removes the constraint on G. Since any matrix patterns may not represent the characteristics of all downstream tasks, enforcing these constraints could potentially degrade performance. as shown in Tables I Moreover, Si et al. (2024b) has also theoretically validates the effectiveness of FLoRA, which is significantly easier reach the global optimum compared to LoRA. This indicates that the low-rank representations of methods like LoRA are inferior to that of FLoRA, thereby yielding inferior performances.
matrix patterns during the learning process
For high- -igeeeiiensional parameter spaces, as previously discussed. other methods compromise the structural integrity the original parameter spaces. contrast, FLoRA preserves their topological structure, leading to superior performance outcomes.
15.0
0.0
Do
Lot4
Trobestts Narm
20.0
7.5
0.3
2.5
0.1
25
Spoch
Epoch
(a) Frobenius Norm
(b) Feature Amplification Factor
Figure 4: Average of the Frobenius norm of AW and the feature amplification factor during training.
To further substantiate the analysis on FLoRA's superiority, we record the average of the Frobenius Lwll
norm of A AW and the feature amplification factor T WV Hu al. (2021a) of all the layers in 1EBERTaV3-bart on CoLAA during training, with FL.RA, DoRA and LORA'S rank 8. Here U and V being the top r left- and right-singular matrices of the SVD decomposition of AW. The feature amplification factor measures how much of task-specific information are amplified by AW.
The results, illustrated in Fig. show that LoRA and DoRA can amplify more task-specific features than FLoRA in the early stages. As discussed before, these two methods have strong constraints on matrix patterns, resulting in a distinct directional learning pattern at the beginning, which contributes their larger initial values. However, such constraints on matrix patterns may not always be suitable for downstream tasks, and since downstream datasets may possess various properties, their amplification factors upon convergence are smaller than that of FL.RA.
Additionally, we found the trend of the Frobenius norm closely aligns with the feature amplification factor, surprisingly. This might indicate that a larger Frobenius norm of values can accommodate more task-specific information, thereby more effectively amplifying the task-specific information in the frozen weights. We also observed that, initially, the Frobenius norms of AW for LoRA and DoRA are also larger than that of FLoRA, for they can quickly capture the information due to the learning of specific matrix patterns. However, upon convergence, they are both smaller than FLoRA's, suggesting that FL A contain more kinds of properties for task-specific information.
8

---

Published as a conference paper at ICLR 2025
5.3 DOES FLORA REQUIRE ACCEPTABLE TRAINING COSTS COMPARED TO OTHER METHODS?
We assess the training costs under various configurations. All training hyper-paramettes, such as batch size and epochs, are kept consistent, with the results presented in Table Clearly, FLoRA is more efficient in terms of training time and memory footprint compared with SOTA method DoRA. which require significantly more time and GPU memories.
Table 5: Training time (minutes/epoch) and GPU usage (GB) of FLORA.
NeN.
DeBERTa 3-base
Method
COCO
ADE20K
Method
MNLI
SST-2
STS-B
Params
Time
GPU
Time
GPU
Params
Time
GPU
Time
GPU
Time
GPU
LoRA
6.39
70
17.19
53
11:94
LoRA
73.57
11.35
6.38
6.83
0.36
6.85
DoRA
6.59 %
100
30.67
67
17.36
DoRA
0.18 %
118.42
16.72
11.26
9.66
0.91
9,66
FLoRA
6,50
77
17.94
60
13.10
FLoRA
79.57
11.35
6.83
6.86
0.56
6.85
LoRA
26.36 %
77
18.11
52
12.79
LoRA
74.29
11.39
6.45
0.88
0.59
6.87
DoRA
26.36 %
111
31.41
69
17.86
DoRA
0.72 %
117.14
16.75
11.79
9.69
0.92
9.69
FLORA
26.06
18.23
58
13:45
FLoRA
78.57
11.39
6.67
6.87
5.4 Is FLORA SENSITIVE TO THE SCALING FACTOR?
We report the sensitivity w.r.t. the scale s on ConvNeXt- V2-L and DeBERTaV3-base and the results are shown in Fig. 5 changing in a reasonable range, which is desirable property in practice.
obvious that the performance of FLoRA quite stable with the scale
0.52
50
100
15
24
0.49
90
25
70
92
93
0.68
80-
70
8.42
0.
16
59
0.05 0.1
0.8
79
0.03 0:05
0.1
0'4
49
0.00 0.05
(a) COCO
(b) CoLA
(c) RTE
(d) STS-B
Figure 5: Sensitivity analysis FLoRA w.r.t scale s.
RELATED WORK
6.1
CHAMMTERR-EFICIIE FINE-TUNING
Methods for Parameter- Efficcient Fine-tuning (PEFT) have been conceived to mitigate the substantial computational costs associated with the fine-tuning of large-scale models. This economization is realized by honing comparatively minute fraction of the overall parameters, selected strategically for adaptation to variety of downstream tasks. Current PEFT techniques can be divided into three distinct categories Liu et al. (2024b); Ding et al. (2023): Adapter-based Houlsby al. (2019); Chen ct al. (2022); Luo et al. (2023); He et al. (202 Mahabadi et al. (2021) Karimi Mahabadi et al. prompt-based Lester Lester
(2021)
al. 22021) Razdaibiedina et al. (2023); Wang et al. (2023b); (Sh &
Lipani 2023); Fischer et al. (2024) and low-rank matrix adaptation based Hu et al. (20212 la) Liu
al. (2024b); Hyeon- Woo et (202T); Qiu al. (2023): Renduchintala et al. (2023); 3 Kopiczko
| (2022a); Si et al. (2024a). The first category of method integrates linear modules either sequentially or concurrently with the existing layer to enhance the performance, and the second class introduces additional soft tokens (prompts) the initial input and concentrate exclusively on refining these trainable vectors. The last type, proposed by LoRA Hu et al. (2021a), adopts low-rank matrix adaptation to model the change of the weight during fine-tuning, and are capable of merging with pre-traines weights.
et al. 2023); YEH et al. (2023); Zhang et al.
However, these methods only focuses linear weights destructing the structural integrity of high dimensional parameter spaces. To this end, we propose a novel method FLoRA to address various dimensional parameter space.

---

Published as a conference paper at ICLR 2025
6.2 TUCKER DECOMPOSITION
When designing FLoRA, our goal to preserve the topological structure of different dimensions of parameter space while maintaining the changes within a low-rank manifold. To achieve this, we use tensor of the same dimensionality as the original parameter space model its changes. By learning the corresponding projection matrices, we align the dimensional sizes of the low-rank space with those of the original space. From this perspective, we found that the form of Tucker decomposition Tucker 1966) naturally aligns with our approach.
Tucker decomposition one of the well-studied algebraic tensor decomposition. Formally, given a tensor X' E x/xxxxXx-xxx where N is the order of the tensor (i.e., the number of dimensions or modes), Tucker decomposition represents X' as a product of core tensor G E R Xxxx-xXxx and matrix along each mode n, A(m) E l xJs,, where Jn can be considered the dimension of the core tensor along mode n. The decomposition can be compactly written as:
X 9 A( X2 A (2) X A (N)
where Xn denotes the or mode-n product between a tensor and a matrix. The core tensor G represents the interactions between different modes, and the matrices A (m) are analogues to principal components within each respective mode. The selection of dimensions J1, J...... JJN allows for balance between desired approximation quality and computational efficiency, tailored to the specific requirements of the task hand.
6.3
OTHER METHODS BASED ON LOW-RANK TENSOR ADAPTATIONS
There are some works like LoTR Bershatsky et al. (024) and SuperLoRA Chen et al. (2024b) also adopt low-rank tensor adaptations, but they are unrelated to FLoRA.
Usage and Impact. Based on low-rank tensor adaptation, FLoRA directly models AW for each weight, maintaining its spatial structure. However, LoTR concatenates A WW across all layers and then applies low-rank tensor adaptation, assuming all layers share same weight matrices, which is a strong assumption that leads to suboptimal performance. SuperLoRA concatenates the vectorized A AW across all Multi-Layer Attention layers, performs various operations, and then redistributes them back to the original dimensions. this process, low-rank tensor adaptation is just one choice. Besides, both LoTR and SuperLoRA disrupt the original parameter space structure although they use low-rank tensor adaptation.
Motivation and Details. FLoRA aims to model AW while preserving the N-dimensional topological structure, meanwhile addressing the limitations of other methods focusing only on linear weights. However, LoTR aims to achieve extreme parameter efficiency, and SuperL.oRA aims to unify and extend LoRA derivatives by setting different hyper-parameters. Both LoTR and SuperLoRA are tailored for linear weights, limiting their applications.
CONCLUSION
In this paper, we propose generalized low-rank tensor adaptation based PEFT method, FLoRA, aiming for N-dimension parameter space. FLoRA asserts that the alternations each dimensional parameter space contain a low-rank core space structurally consistent to the original space. It models updates using this core space alongside corresponding weights reconstruct the original alternation space. By doing so, FLoRA A effectively preserves the structural integrity of the original N-dimensional parameter space while modeling its change through low-rank tensor adaptations. Extensive experiments computer vision, natural language processing and multi-modal domains substantiate the effectiveness of FLoRA.
There still exists some limitations for FLoRA. For specific backbone, FLoRA could achieve stable superior performance on different datasets when the scaling factor s changing within a wide range. While for different backbones such ConvNeXt- V2-L, Intern ViT-6B and DeBERTa V3-baa-- it still needs different scales. Understanding the role of scale different models and designing a unified scale is topic worthy of further investigation.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was supported by NSFC 62322604, NSFC 62176159, NSFC 62401361, and Shanghai Municipal Science and Technology Major Project 2021 SHZDZX0102.
REFERENCES
Armen Aghajanyan, Luke Zettlemoyer and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint arX1::2012.15255,, 2020.
A Meta. Llama model card. 2024. mlob/main/MOD-5 CARD md
1lama/ llama3/
Daniel Bershatsky, Daria Cherniuk, Talgat Daulbaev, and Ivan Oseledets. Lotr: Low tensor rank weight adaptation. arXiv preprint 11Xxi:24002.1.376. 2024.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense natural language. Proceedings of the AAAI conference on artificial intelligence. volume 34, pp. 7432-7439, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2299,6308, 2017.
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: 00000001555 2019.
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 35: 35:16664-16675 2022.
Wei Chen, Zichen Miao, and Qiang Qiu. Parameter-efficient tuning of large convolutional models. arXiv preprint arXiv22433.00269, 2024a.
Xiangyu Chen, Jing Liu, Ye Wang, Matthew Brand, Guanghui Wang, Toshiaki Koike-Akino, et al. Superlora: Parameter-efficient unified adaptation of multi-layer attention modules. arXiv preprint arXiv:2403. 1887, 2024b.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing. Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual- lingus-iin tasks. arXiv preprint arXiv:2312. 14238, 2023.
Christopher Clark, Kenton Lee, Ming-Was Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:190S.1004. 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 183 2018.
MMSegmentation Contributors. MMSegmeenattinn Openmmlab semantic segmentation toolbox and benchmark.
htt.mmmmmmmmmmmmmmmmmmaaaaaa on 2020.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 18/0:/8/0004400 2018.
11

---

Published as a conference paper at ICLR 2025
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-effiiient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220-25) 235, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL http:::t////t/ httpss:///peeeeeiiiii net / Eorum?id YicbF FicbFdNTTy
Mare Fischer, Alexander Bartler, and Bin Yang. Prompt tuning for parameter-efficient medical image segmentation. Medical Image Analysis, 91:103024, 2024.
Yash Goyal, Tejas Khot, Douglas Summers-Stayy Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904-6913, 2017.
Demi Guo, Alexander M Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning. arXiv preprint urXiv:2012.07465 2020.
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 3608-3617, 2018.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Kerg-Kirkpatrick, and Graham Neubig. Towards a unified view of parameteer-efficient transfer learning. arXiv preprint 1rXiv:2110004366 2021a.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2961-29696 2017.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style embedding sharing. arXiv preprint 1
pre-training with 2021b.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pp. 2790-2799, PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Alle Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 20ti::2006006885 2021a.
Xia Hu, Lingyang Chu, Jian Pei, Weiqing Liu, and Jiang Bian. Model complexity of deep learning: A survey. Knowledge and Information Systems, 63:2585-266 2619, 2021b.
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Fe-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka-Wei Lee. LIm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. arXiv preprint 1txxix2334.01993 2023.
Drew A Hudson and Christopher D Manning. Gqa: A dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Nam Hycon- Woo, Moon Ye-Bin, and Tac-Hyun Oh. Fedpara: Low-rank hadamard product for 2021.
communication-efficee federated learning. arXiv preprint
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pp. 709-727. Springer, 2022.
Shibo Jie and Zhi-Hong Deng. Convolutional bypasses are better vision transformer adapters. arXiv preprint XXiv:2207.07033, 2022.
12

---

Published as a conference paper at ICLR 2025
Shibo Jie and Zhi-Hong Deng, Fact: Factor- Factor-tuning tuning for lightweight adaptation vision transformer. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 1060-1068, 2023.
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems, 14:1022-1003 2021.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, al. Segment anything. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pp. 4015-4026, 2023.
Dawid Jan Kopiezko, Tijmen Blankevoort, and Yuki Markus Asano. Vera: Vector-based random matrix adaptation. arXiv preprint arXiv:2310. 1454, 2023.
Brian Lester, Rami AI-Rfou, and Noah Constant, The power of scale for parameter-efffiient prompt tuning. arXiv preprint arXiv:2104.. 086911 2021.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. arXiv preprint arXiv: 1804,8804888 2018.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023.
Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Scaling & shifting your features: A new baseline for efficient model tuning. Advances in Neural Information Processing Systems, 35: 109-123, 2022.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024a.
Shih-Yang Yang Liu, Chien-Ye Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353 2024b.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 1692, 2019.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: your multi-modal model an all-around player? arXiv preprint arXiv:2307.00280
2023.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 11976-11986, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 10110:1711.05101, 2017.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering, Advances in Neural Information Processing Systems, 35:2507-25 -5221, 2022.
Gen Luo, Minglang Huang, Yiyi Zhou, Xiaoshuai Sun, Guannan Jiang, Zhiyu Wang, and Ron- grong Ji. Towards efficient visual adaption via structural re- re-parameterization. arXiv preprint arXiv:2302.nm 08 0806, 2023.
13

---

Published as a conference paper at ICLR 2025
Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. Nature Communications, 15(1):654, 2024.
Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter- efficient multi-task fine-tuning for transformers via shared hypernetworks. arXiv preprint arXiv:2106.04455 04489, 2021.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. arXiv preprint 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4, arXiv preprint arXiv:2304.03277 2023.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé Kyunghyun Cho, and Iryna Gurevych. Adapter- fusion: Non-destructive task composition for transfer learning. arXiv preprint arXiv:2005.00247. 2020.
Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. Pre-trained models for natural language processing: A survey. Science China Technological Sciences, 63(10); 1872-1897, 2020.
Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng. Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. Advances in Neural Information Processing Systems, 36:7933007793660 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1-61 2020.
Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, Jimmy Ba, and Amjad Almahairi. Residual prompt tuning: Improving prompt tuning with residual reparameterization. arXiv preprini 2305:22005 2023.
Adithya Renduchintala, Tugrul Konuk, and Oleksii Kuchaiev. Tied-lora: Enhacing parameter efficiency of lora with weight tying. arXiv preprint arXhv:2311.09570 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Olaf Ronneberger, Philipp
Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assiste intervention MMCCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pp. 234-241. Springer, 2015.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9)999-106 2021.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint 1xXv:9904.009720 2019.
14

---

Published as a conference paper at ICLR 2025
Zhengxiang Shi and Aldo Lipani. Dept: Decomposed prompt tuning for parameter-effiiiet fine- tuning. arXiv preprint arXiv:2309.05173, 2023.
Chongjie Si, Zhiyi Shi, Shifan Zhang, Xiaokang Yang, Hanspeter Pfister, and Wei Shen. Un- leashing the power of task-specific directions in parameter efficient fine-tuning, arXiv preprint arXiv:2409.0r 01035, 2024a.
Chongjie Si, Xiaokang Yang, and Wei Shen. See further for parameter efficient fine-tuning by standing on the shoulders of decomposition. arXiv preprint arXiv:2407.05411 2024b.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Ledyard R Tucker. Some mathematical notes on mhree-mode factor analysis. Psychometrika, 31(3):
279-311, 1966.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 074461 2018.
Hanqing Wang. Yixia Li, Shuo Wang, Guanhua Chen, and Yun Chen. Milora: Harnessing minor singular components for parameter-efficien Ilm finetuning. arXiv preprint arXiv:2406.09044 2024.
Renzhi Wang and Piji Li. Semantic are beacons: A semantic perspective for unveiling parameter- efficient fine-tuning in knowledge learning. arXiv preprint arXiv:2405. 18292, 2024.
Wenhai Wang. Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14408 14419, 2023a.
Yaqing Wang, Jialin Wu, Tanmaya Dabral, Jiageng Zhang, Geoff Brown, Chun-Ta Lu, Frederick Liu, Yi Liang, Bo Pang, Michael Bendersky, et al. Non-intrusive adaptation: Input-centric parameter- efficient fine-tuning for versatile multimodal modeling. arXiv preprint arXiv:2310.12/00 2023b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, al. Chain-of-thougee prompting elicits reasoning in large language models. Advances in neural information processing systems, 15:24824-24887 2022.
Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Convnext v2: Co-designing and scaling convnets with masked autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16133-16142. 2023.
Taiqiang Wa, Jiahao Wang, Zhe Zhao, and Ngai Wong. Mixture-of-subspaces in low-rank adaptation. arXiv preprint arXiv:2406. 1909, 2024.
Gui-Song Xia, Xiang Bai, Jian Ding, Zhen Zhu, Serge Belongie, Jiebo Luo, Mihai Datcu, Marcello Pelillo, and Liangpei Zhang. Dota: A large-scale dataset for object detection aerial images. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3983. 2018.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European conference on computer vision, pp. 418-434, 2018.
Xingxing Xie, Gong Cheng, Jiabao Wang, Xiwen Yao, and Junwei Han. Oriented r-cmn for object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 3520-3529, 2021.
15

---

Published as a conference paper at ICLR 2025
SHIH YING YEH, Yu-Guan Hsich, Zhidong Gao, Beraadd BW Yang. Giyeong Oh, and Yanmin Gong, Navigating -oxt-to- customization: From lycoris innettuiin to model evaluation. In The Twelfth International Conference Learning Representations, 2023.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficien fine-tuning for transformer-based masked language-models. arXiv preprint urXiv:2/06.10119 2021.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv: 1905.07800 2019.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, al. The visual task adaptation benchmark. 2019.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengeheng Hc, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning, The Eleventh International Conference on Learning Representations, 2022a.
Yuanhan Zhang. Kaiyang Zhou, and Ziwei Liu. Neural prompt search. arXiv:2206. 04673, 2022b.
arXiv preprint
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference computer vision and paltern recognition, pp. 633-641, 2017.
Yue Zhou, Xue Yang. Gefan Zhang, Jiabao Wang, Yanyi Liu, Liping Hou, Xue Jiang, Xingzhao Liu, Junchi Yan, Chengqi Lyu, Wenwei Zhang, and Kai Chen. Mmrotate: rotated object detection benchmark using pytorch. In ACM International Conference on Multimedia, 2022.
16

---

Published as a conference paper at ICLR 2025
A MORE EXPERIMENTS
A. GENERATIVE TASKS
We employ FLoRA and LoRA to fine-tune Stable Diffusion RRobbach et (2022) SDXL with the toolkit diffusers supported by HuggingFace using the AdamW optimizer Loshchilov & Hutter (2017). with learning rate of x 10-4 We fine-tune all the convolutional layers of the U-net Ronneberger et al. (0155.. Specifically, BLIP captions dataset. We show some visualization results of FLoRA and LoRA conditioned on the same texts prompts in Fig. 66 The results indicate that FLoRA outperforms LoRA in generating images that adhere more closely to the desired style. This can be attributed to the more sophisticated parameter fine-tuning FLoRA.
fine-tune SDXL learn the style of Pokemon trained Pokemon
LoRA
FLoRA
mouse
dog
a cat
elephant
Figure 6: The generative results fine-tuned by LoRA and FL.RA on the Pokemon BLIP captions dataset. FL.RA follows the Pokemon style more closely.
A.2 COMMONSENSE REASONING TASKS
The commonsense reasoning benchmarks include eight distinct sub-tasks, each paired with a specific dataset: BoolQ Clark et al. (2019), PIQA Bisk et al. (2020), SIQA (Sap et al. (2019). HellaSwag Zellers al. 2019). WinoGrande (Sakauuchh et al. (2021). ARC-e/ARC-C Clark et al. (2018). and OBQA|Mihayloo et al. (2018). Following the approach in Hu et al. (2023). we combine the training datasets from these tasks into a single Commonsensel 770K dataset and evaluate performance on test set of each task.
use FLoRA to fine-tune LLaMA3-8B AIG (2024) on commonsense reasoning task. Addi- tionally, incorporate results from ChatGPT's gpt-3.5-turbo API using zero-shot Chain of Thought reasoning Wei et al. (2022). The apper-parameter settings for FLoRA are detailed in Table
The results are shown in Table We can observe that FLoRA outperforms other state-of-the-art methods on this task, demonstrating its exceptional performance.
A.3 VTAB-IK BENCHMARK
We also evaluate the performance of FLoRA on VTAB-IK Benchmark. VTAB-IK Zhai et al. (2019) includes 19 image classification tasks spanning diverse domains, categorized into three groups: Natural, Specialized, and Structured. These tasks represent wide variety of potential downstream applications, making this benchmark strong indicator of method's transfer learning capability. Each dataset consists of 800 training samples and 200 validation samples. Following prior studies Jia
et al. (2022). Die & Deng (2022 2023), we fine-tune the pre-trained model ViT-B/16 Dosovitskiy
17

---

Published as a conference paper at ICLR 2025
Table 6: Hyper-parameter configurations for commonsense reasoning task.
Hypereparameter
LoRA
AdaL.oRA
DoRA
FLoRA
Rank
16
32
Dropout
0.05
Batch size
16
Epochs
3
Learning rate
3e-5
Target module
k, v, up. down
Table 7: Performance comparison across commonsense reasoning tasks. Average (Avg.) performance computed over all tasks.
Method
Params (%)
BoolQ
PIQA
SIQA
HellaSwag
WinoGrande
ARC-e
ARC-c
OBQA
Avg.
CharGPT
85.4
68.5
78.5
66.1
74.8
LoRA
0.35
72.3
86.7
79.3
93.5
84.8
87.7
75.7
82.8
82.9
AdaLoRA
0.35
75.1
86.4
76.7
75.4
83.3
90.4
79.1
85.0
81.4
DoRA
0.36
88.8
80.3
84.7
90.1
79.1
87.2
FLORA
0.35
74.6
89.7
81.0
85.8
90.5
ct al. 2021) using all 1,000 training and validation samples and evaluate performance on the test set. Consistent with Jia et al. (2022); Lian et al. (2022), we use unnormalized inputs, aligning with the original VTAB paper Zhai et al. (0119).
We compare FLoRA with SOTA methods and some CV tailored methods, including VPT Jia et al. (2022), NOAH Zhang et al. (2022b). Lia et al. (2022). PAdapter Pfeiffer et al. (2020). FACT Jie & Deng 2023), Adaptformer Chen et al. (2022), Compacter Karimi Mahabadi al. (202 LoRA Hu et al. (2021a) and BitFit Zaken et al. 2021). All baselines are evaluated using HP32 precision by default. For PAdapter, Adapttormer, and LoRA, the hidden dimension is set to 8, and FLoRA 2. For Compacter, the number of Kronecker products and the hidden dimensions are set to 4 and 32, respectively. FACT employs the FACT-TT variant with the rank selected from 16, 32 adapt the MHSA blocks. The configurations for other baselines follow their respective original papers.
Clearly, compared to methods specifically designed for computer vision tasks, FLoRA continues to demonstrate significant advantages.
Table Performance comparison across VTAB- benchmarks. Results are grouped into Natural, Specialized, and Structured categories, with average accuracy (%) listed. Avg. denotes the average results over three groups. #P: number of parameters DI: Cifar 100. D2: Caltech D3: DTD. D4: Flower102. D5: Pets. D6: SVHN. D7: Sun397. D8: Camelyon. D9: EuroSAT, D10: Resisc45, D11: Retinopathy. D12: Clevr-Count. D13: Clevr-Dist. D14: DMLab. DIS:KITTI-Dis DI6: dSpr-Loc. D17: dSpr-Ori. D18: sNORB-Azim. D19: sNORB-Fle.
Natural
Specialized
Method
UP (M)
Avg
D1
132
D3
D4
D5
D6
D)
D8
D9 D10
D11 D12
D13
D14
D15
D16
D17
D18
D19
85.8
87.7
97.2
86.9
79.7
73.9
41.7
65.5
46.7
Last-layer Tuning
0.08
57.6
64.4
85.0
63.2
97.0
86.3
36.6
51.0
78.5
87.5
68.5
74.0
34.3
30.6
33.2
55.4
12.5
20.0
9.6
19.2
VPT-Docp
2.03
72.0
78.8
90.8
65.8
98.0
88.3
78.1
49.6
81.8
96.1
60.0
46.5
72.8
73.6
47.9
32.9
37.8
NOAH
1.37
75.5
69.6
92.7
70.2
99.1
90.4
86.1
53.7
84.4
95.4
83.9
75.8
82.8
68.9
49.9
81.7
81.8
48.3
32
44.2
LoRA
1.13
76.4
72.0
91.2
71.6
99.1
91.3
88.9
56.4
74,9
83.7
64.0
52.3
81.2
84.8
53.3
38.1
43,4
SSF
0.78
75.7
69.0
92.6
75
99
91.8
90.2
52.9
87.4
95.9
87.4
75.5
75.9
62.3
53.3
80.6
77.3
54.9
29.5
37.9
PAdapter
79.7
81.7
AdapiFormer
0.56
76.7
73.8
92.3
72.7
99.
91.6
89.1
56.5
87.8
95.5
84.9
75.2
83.1
62.5
52.4
81.7
86.2
55.9
34.4
40.2
Bitii
97.5
59,9
BACT-TT
0.30
76.7
73.4
91.0
72.4
99.2
91.4
90.1
56.6
87.3
94.7
84.5
75.8
K3.0
64.9
51.3
81.4
87.4
53.2
33.5
44.3
VPT-Shallow
0.24
87.3
74.5
58.6
67.1
68.7
36.1
Compacter
0.15
74.2
71.9
89.0
69.7
99.1
90.7
82.7
56.1
86.0
93.5
K2.4
75.3
80.2
63.4
47.4
77.2
78.1
53.5
27.3
39,8
FLORA
76.2
86.2
18

---

Published as a conference paper at ICLR 2025
B DETAILED DISCUSSION ON SEC. 5.2
B.I THEORETICAL INSIGHTTS ON THE EFFECTIVENESS OF FLORA'S REPRESENTATION
Inspired by Si et al. (2024b), we provide the theoretical insights on the effectiveness of FLoRA's low-rank representation here.
Let W E Ruxm be the optimal weight for a specific linear layer with frozen weight W, and AW* W -Wi the optimal change. Generally, the low-rank representation of LoRA's series can be represented as AW AGB, where A E R B E R G E Rexr and < {n, m), and the essential difference of these methods is the matrix pattern of G. In FLoRA, G is arbitrary. For other methods such as AdaLoRA, G is diagonal, and identity for LoRA.
Ideally, we have = A A'W'B Bi, where A and B are the Moore-Penrose Pseudo-inverses of A and B, respectively. If G is arbitrary matrix such as FLoRA, it is obvious that constraints imposed on A and B, since for any choice of A, B, we can always find a G. However, when G is diagonal or identity, A and B are inevitably subject certain constraints, and there may even exist dependencies or interrelations between them.
Thus, achieve the optimal change AW", FLORA imposes constraints on A, B. Other methods, to varying degrees, impose certain constraints on A and B. However, these constraints are not applied during training, which means that these methods like LoRA are less likely to learn the optimal change compared to FLoRA. Further, even if specific matrix patterns are constrained during training, FLoRA's greater degree of freedom and stronger expressive power learning make it much easier to achieve better results Zhang et al. (2022a); Hu et al. (2021b). Additionally, some subsequent works have also demonstrated the effectiveness of FLoRA from different perspectives Wu al. (2024).
B.2
MORE EXPLANATION ON THE EMPIRICAL PHENOMENA
B.2.1 WHAT IS FEATURE AMPLIFICATION FACTOR?
The Feature Amplification Factor a metric introduced by LoRA measure how much task-specific information is amplified by AW. Simply put, it calculates the ratio of the task-specific information learned by the model (represented by the Frobenius norm of A AW) to the amount of information obtained by projecting the pre-trained weights W onto the task-specific directions. Mathematically, the Feature Amplification Factor is defined as |UTWV singular vectors of AW. Here, r is the rank configuration of PEFT methods like LoRA.
|AW|IF where U and V are the top left and right
This metric provides insight into how effectively the model adapts to downstream tasks by quantifying the relative contribution of the learned task-specific updates, and it has been widely adopted in PEFT works to evaluate the amount of spec-sp information learned by the model WWng et al. (0024); Si al. (0024a): Wang & Li (0024).
B.2.2 RELATIONSHIP BETWEEN FEATURE AMPLIFICATION FACTOR AND PERFORMANCE
Indeed, although LoRA does not explicitly state relationship between these factors and task performance, its experimental results show a clear correlation with amplification levels. Moreover, many existing works Wang et al. (2024) Si et al. (2024a); Wang & Li (024) implicitly or explicitly suggest that a larger Feature Amplification Factor correlates with improved model performance. For instance, Si et al. (2024a) directly highlights that amplifying more task-specific information can enhance model performance.
B.2.3 RELATIONSHIP BETWEEN FEATURE AMPLIHICATION FACTOR AND FROBENIUS NORM OF AW
In our experiments, we observed that the final value of the Frobenius norm of AW is positively correlated with the Feature Amplification Factor, meaning that as one increases, the other also tends to increase. This was somewhat unexpected but also intuitively reasonable: given that the Feature Amplification Factor is directly calculated using the Frobenius norm of A W. is logical to assume some intrinsic connection between the two. Here we offer explanation for this phenomenon:
19

---

Published as a conference paper at ICLR 2025
Firstly, the Feature Amplification Factor is defined the ratio of the Frobenius norm of AW, ie.. [ to the amount information the pre-trained weights W projected onto the tpek-specific directions, denoted
|| WV Since the pre-traine pre-trained weight matrix W is frozen during training, the value of | WV r depends solely on U and V, which are two orthonormal matrices derived from AW and represent task-specific directions.
In Si et al. (2024a) the authors highlight that at any step during training, A W effectively captures the task-specific directions. Consequently, from the beginning the end of training, the changes in U and V are minimal (as also validated in Si et al. (2024a)), which implies that ||U WV remains nearly constant. other words. larger Amplification Factor.
does indeed correspond larger Feature
Building on the analysis of the relationship between amplification factors and model performance, is evident that larger Frobenius norm of AW corresponds to better performance. This is surprising yet logical and exciting conclusion. We hypothesized that larger Frobenius norm values of AW allow for encoding more task-specific information, thereby effectively amplifying the task-specific signals in the frozen weights. Upon further reflection, this insight aligns with the findings of Si et al. (2024a) but from different perspective: amplifying task-specific information leads to better task performance. Moreover, directly increasing the Frobenius norm of AW emerges as a more straightforward approach to achieve this amplification.
Therefore, we believe that exploring more effective PEFT methods from this perspective represents highly promising direction for future research.
B.2.4 MORE EXAMPLES
To further support the above analysis, we record the average of the Frobenius norm A AW and the feature amplification factor of all the layers in DeBERTaV3-larre on SST2 during training, with FLoRA, DoRA and LoRA's rank r = 8, and the results are illustrated Fig. 7 Clearly, the results align well with aforementioned analysis, further reinforcing conclusions.
30
5.0
TL:RA
FLdRA
23
DallA
DatA
LDRA
LtR4
20
Feature Amp catttoo
12
Frobectus
10
3.0
10
25
(a) Frobenius Norm
(b) Feature Amplification Factor
Figure Average Frobenius norm W and the feature amplification factor during training.
C DETAILS ON TRAINING
C.1 TRAINING DETAILS FOR CV TASKS
Table 9 presents some customized hyper-paranenn for all experiments FLoRA, LoRA Hu et al. (202 DoRA Liu al. (2024b), BitFit Zaken al. (202I) and FT. Please refer MMDetection MMSegmentation MMRotate for other default parameters we omit in Table 9
Table 9: Hyper-parameter setup for object detection and segmentation.
Dataset
Toolkit
Model
Schedule
LR
BS
Optimizer
Weight Decay
12op
Adaam
DOTA
MMRomate// Joa e
Oriented R. C-NN Aieeri Brlle 2021
12ep
Se-3
16
SGD
le-4
ADE20K
MMSegmeenttioon
Cont boloos X20
UperNet Xiaor 2018
160k
lo-4
16
AdamW
5e-2
20

---

Published as a conference paper at ICLR 2025
C.2 DETAILS OF GLUE DATASET
We present the details of the GLUE dataset, which are shown in Table 10
Table 10: Details of GLUE dataset.
Dataset
Task
# Train
# Dev
# Test
# Label
Metrics
Single Songle-Sentenee Classification
CoLA
Acceptability
8.5k
1k
1k
2
Matthews corr
SST
Sentiment
67k
872
1.8k
2
Accuracy
Pairwise Text Classification
MNLI
NLI
393k
20k
20k
3
Accuracy
RTE
NLI
2.5k
276
3k
2
Accuracy
QQP
Paraphrase
364k
40k
391k
2
Accuracy Fl
MRPC
Paraphrase
3.7k
408
1.7k
2
Accuracy FI
QNLI
QA/LLA
108k
5.7k
5.7k
2
Accuracy
Text Similarity
STS-B
Similarity
7k
1.5k
1.4k
Pearson/ Spearman Corr
C.3 TRAINING DETAILS FOR NLP TASKS
We provide the training details for NLP tasks in Table II
Table 11: Hyper-paramese setup for GLUE benchmark.
Datasct
MNLI
RTE
QNLI
MRPC 10-3
QQP
SST-2 10
CoLA
STS-B
learning rate
x 10
1.2 x 10
1.2 10-3
5 x
10
x
10-3
butch size
32
32
32
#epochs
7
25
C.4 TRAINING DETAILS FOR MULTI-MODAL TASKS
We provide the training details for multi-modal tasks here in Table 12
Table 12: Hyper-parameter configurations on FLoRA for fine-tuning LLaVA-1.5-7B with visual instruction tuning datasets.
Hyper-parameterss
Value
Rank r
128
Dropout Optimizer
0.05
AdamW W
LR
2 x 10-4
LR Scheduler
Cosine decay
Batch size
16
Warmup ratio
0.03
Epochs
Where
Q.K.V.O.U..DDwn.Ga.tt
21


Reviews:
[{"review_id": "NR8q6B1Ean", "reviewer": "Reviewer_UDNJ", "summary": "This paper introduces FLoRA, a parameter-efficient fine-tuning (PEFT) method for adapting pre-trained foundation models across various parameter spaces while preserving the model’s structural integrity. FLoRA minimizes the computational burden of full fine-tuning by maintaining the original parameter organization, allowing efficient task-specific adaptability of different architectures. This approach supports scalable adaptation in large models, optimizing both resource use and performance. Empirical and theoretical evaluations show FLoRA’s robustness, highlighting its promise as an efficient approach for model adaptation.", "questions": "Please see weaknesses above. Below are additional questions:\n* What exactly is the difference between the current method and existing methods, particularly LoTR? The decomposition appears identical since both methods use Tucker decomposition, yet the paper claims that there are differences. It is hard to appreciate the paper without knowing about the differences between this work and past methods.\n* How is the feature amplification factor considered a reliable indicator of task-specific information amplification, and what insights does it provide regarding task-specific performance? Furthermore, what methods are employed to measure task-specific information in the context of this study?\n* What significance does the average Frobenius norm of $\\Delta W$ hold for FLoRA's performance during fine-tuning? Can the magnitude of this norm be directly correlated to the effectiveness of task-specific adaptation? The presented correlation is based solely on a single sample, and there is no clear explanation provided for why this correlation would exist in the first place.\n* How does the feature amplification factor of FLoRA in the DeBERTaV3-base model compare to that of other fine-tuning methods applied to the CoLA dataset? Does FLoRA exhibit a distinct pattern in terms of norm growth or amplification factor throughout the training process?\n* If different versions of the DeBERTa model were evaluated, would the Frobenius norm or amplification factor be affected by changes in model architecture, or are these metrics consistently reliable across various transformer-based models?\n* Without a direct comparison of the Frobenius norm and amplification factor metrics against alternative fine-tuning approaches, such as standard LoRA or full fine-tuning, it is challenging to assert FLoRA's superiority. If these metrics do not demonstrate a clear advantage over other methods, this could weaken the strength of the conclusions drawn in the paper.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The motivation behind maintaining the topological structure of the pre-trained matrices is compelling, providing a strong foundation for the proposed approach. \n\n+ The paper is well-organized and easy to follow, with each section clearly building upon the previous. \n\n+ The analysis section aligns well with the proposed objectives, and the approach to evaluating whether the core space is low-rank adds depth to the analysis.", "weaknesses": "- The work appears to have similarities with existing approaches such as LoTR and LoKR; however, the paper does not adequately address how FLoRA distinguishes itself from these methods. This lack of clarification raises serious concerns about the novelty of the proposed approach.\n- The manuscript does not provide sufficient evidence to demonstrate how the representations in FLoRA surpass those of LoRA and DoRA. It remains unclear how FLoRA broadens parameter adjustments and how this relates to enhancing parameter learning flexibility.\n- The feature amplification factor introduced in the study may lack a direct correlation with actual task performance gains, making it difficult to interpret its significance in relation to improved task-specific outcomes. Currently, the correlation is indicated only using a single dataset and model architecture.\n- The correlation observed between the Frobenius norm and feature amplification could be incidental rather than causal. There is insufficient evidence to support the claim that a larger norm consistently leads to enhanced task-specific performance.\n- The scalability and computational efficiency of the proposed approach may be limited when applied to extremely high-dimensional tensors, potentially affecting its practicality for very large foundation models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IHHmGxTlYX", "reviewer": "Reviewer_B2Z8", "summary": "This paper proposes a new generalized PEFT method, FLoRA, for N-dimension parameter space. FLoRA is based on the Tucker decomposition and uses a low-rank core tensor and N low-rank matrices to reconstruct the original parameter tensors. The introduction of the low-rank core space helps preserve the structural integrity of the original parameters space. Experiments on CV, NLP and multi-modal tasks validate the effectiveness of the proposed method over baselines like LoRA and DLoRA.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The generalization of LoRA on N-dimension parameter space with form of Tucker decomposition is interesting and effective.\n\n2. The authors conduct many experiments on multiple kinds of tasks and validate the effectiveness of the proposed FLoRA method.\n\n3. The paper is well-written and easy to follow.", "weaknesses": "1. In Line 183, the paper mentions that “in any convolution layer, there exists a convolution core”. Is this an assumption or supported by evidences and theories? More discussion should be added to improve the reliability of this claim.\n\n2. How does the learned convolution core help preserve the convolution’s property of spatial locality?\n\n3. It would be better to compare the FLoRA method with some PEFT method designed for CV tasks, such as [1].\n\n4. Can you provide the standard deviations of the main results (Table 1 etc.)?\n\n[1] Jie, S. and Deng, Z.-H. Fact: Factor-tuning for lightweight adaptation on vision transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 1060–1068, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r6e6ciMUo5", "reviewer": "Reviewer_VXNC", "summary": "The paper introduces FLoRA as a new approach to Parameter-Efficient Fine-Tuning (PEFT), focusing on preserving the structural integrity of high-dimensional spaces in large models.  The authors argue that existing PEFT methods, especially those relying on low-rank matrix adaptations, often disrupt the crucial spatial relationships within these high-dimensional spaces, particularly in convolutional layers, leading to suboptimal performance. \n\nFLoRA addresses this issue by introducing a low-rank core space that maintains the original spatial dimensions of the parameter space being adapted. For example, for a convolutional layer, FLoRA employs a 4D core tensor that mirrors the structure of the convolutional kernel. This ensures that spatial locality, crucial for convolution operations, is preserved during fine-tuning. \n\nThe core space is then combined with corresponding weights to reconstruct the changes in the original parameter space. This approach allows FLoRA to achieve high performance while being efficient in terms of trainable parameters. The paper demonstrates FLoRA's effectiveness across various tasks in computer vision, natural language processing, and multi-modal learning, showcasing its superior performance compared to existing PEFT methods.\n\n* **FLoRA outperforms traditional low-rank methods like LoRA, especially in tasks involving convolutional layers.** This is attributed to FLoRA's ability to preserve the spatial structure of high-dimensional parameter spaces.\n* **FLoRA achieves performance comparable to or even exceeding that of full fine-tuning, while using significantly fewer trainable parameters.** This makes it a highly efficient and effective method for adapting large models to downstream tasks.\n* **The paper provides empirical evidence for the existence of a low-rank core space within different dimensional parameter spaces.** This supports FLoRA's central premise and suggests its potential for broader applicability.", "questions": "Already listed in the Weaknesses section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* **Significant Performance Gains:** The paper convincingly demonstrates FLoRA's superior performance over existing PEFT methods on both ConvNeXt (Table 1) and InternViT (Table 2) architectures.  The improvements are particularly noteworthy on ConvNeXt, which relies heavily on convolutional layers, highlighting FLoRA's strength in handling high-dimensional parameter spaces.  The results show that FLoRA not only outperforms other low-rank adaptation methods like LoRA and DoRA by a substantial margin (at least 15% on average across different parameter budgets) but also achieves performance comparable to, and in some cases even better than, full fine-tuning. \n\n* **Parameter Efficiency and Practicality:** The authors emphasize FLoRA's practicality by showcasing its ability to achieve these strong results while significantly reducing the number of trainable parameters and training time. The paper explicitly states that FLoRA can achieve comparable performance to full fine-tuning with an 80% reduction in parameter budget.  Additionally, Table 5 provides evidence of FLoRA's efficiency in terms of both training time and GPU memory usage compared to other methods, particularly DoRA.\n\n* **Insightful Analysis of Low-Rank Representation:** The paper goes beyond simply presenting performance results. It provides a detailed analysis of why FLoRA's low-rank representation is more effective than other methods. Figure 4, which compares the Frobenius norm and feature amplification factor of FLoRA, LoRA, and DoRA during training, offers valuable insights. This analysis reveals that while LoRA and DoRA might initially amplify task-specific features more aggressively due to their constraints on matrix patterns, FLoRA's less constrained approach allows it to capture a broader range of task-specific information, leading to higher amplification factors upon convergence. The paper also suggests a strong correlation between the Frobenius norm of the learned changes (∆W) and the ability to capture task-specific information.\n\n* **Clear and Well-Written:** The paper is well-structured, and the concepts are explained in a clear and concise manner.  This clarity makes it easy for readers to understand the motivation behind FLoRA, its technical details, and the significance of the results. \n\n* **Addressing a Gap in PEFT Research:** The authors clearly identify a gap in existing PEFT research, which primarily focuses on linear layers while neglecting the complexities of high-dimensional parameter spaces. FLoRA is presented as a solution to this problem, demonstrating its novelty and potential impact on the field.", "weaknesses": "*   **Limited Scope of LLM Fine-tuning Experiments:** The paper lacks experiments on LLM fine-tuning (e.g., LLaMA3-8B), despite PEFT methods being commonly used in these scenarios. Evaluating FLoRA's efficacy on LLMs would enhance the paper's practicality and relevance.\n*   **LoRA's Applicability in Vision-Based Models:** The paper focuses on LoRA as a comparison point for FLoRA, but LoRA is not a standard fine-tuning technique for vision-based models like ConvNext and Mask-RCNN. Comparing FLoRA to more prevalent methods like last-layer fine-tuning and prompt tuning would offer a more comprehensive evaluation of its effectiveness.\n*   **Missing DoRA Results on LLaVA-1.5-7B:** The paper doesn't report the performance of DoRA on LLaVA-1.5-7B fine-tuning, despite mentioning a DoRA result of 67.6, which surpasses FLoRA's performance. Including comprehensive comparisons with state-of-the-art methods like DoRA is crucial for establishing FLoRA's superiority.\n*   **Compatibility with Weight-Decomposed Formulations:** The paper does not address whether FLoRA is compatible with weight-decomposed formulations like those proposed by DoRA. Exploring potential integration with other advanced PEFT techniques could reveal further benefits and limitations of FLoRA.\n*   **Lack of Theoretical Grounding for the Core Space:** The paper demonstrates the existence and effectiveness of a low-rank core space empirically but lacks a theoretical framework to explain these findings. Providing a theoretical foundation for the core space's properties would strengthen the paper's claims and provide valuable insights into FLoRA's underlying mechanisms.\n*   **Potential Computational Overhead:** While the paper asserts that FLoRA exhibits better parameter efficiency than LoRA for larger kernel sizes, it doesn't thoroughly analyze the computational costs in diverse settings. A more comprehensive analysis of FLoRA's computational complexity in various scenarios, especially for extremely large models or complex tasks, is recommended.\n*   **Focus Beyond Convolutional Kernels:** While FLoRA demonstrates its effectiveness on convolutional kernels, it should be assessed on other high-dimensional weight matrices.  Given its positioning as a fundamental low-rank adaptation method, a broader range of experiments would validate its general applicability and effectiveness.\n*   **Quantifying the Impact of Structural Integrity:** The paper could benefit from a more in-depth exploration of whether preserving structural integrity truly matters in weight adaptation for computer vision. This could involve experiments or analyses that isolate the impact of structural preservation on performance, further validating FLoRA's core principle.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I4uaC5mZRt", "reviewer": "Reviewer_wiu6", "summary": "This paper presents FLoRA, a novel parameter-efficient fine-tuning framework that addresses the challenge of adapting pre-trained models across diverse parameter space dimensions. Leveraging Tucker decomposition, FLoRA proposes to model parametric changes through a low-rank core space, ostensibly preserving the structural integrity of high-dimensional parameter spaces. The methodology is evaluated across a spectrum of tasks encompassing computer vision, natural language processing, and multimodal learning. The authors report that FLoRA demonstrates superior performance compared to existing methods, notably LoRA, while utilizing fewer trainable parameters. This efficiency is attributed to FLoRA's capacity to maintain the topological structure of the parameter space.", "questions": "See above", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1)\tThe motivation behind this paper is significant. When using LoRA to fine-tune convolutional layers, it either disrupts the original topology or adds too many parameters. FLoRA, with Tucker decomposition, effectively adjusts high-dimensional parameter spaces, aiding the application of parameter-efficient fine-tuning across more models.\n2)\tAlthough FLoRA is motivated by improvements in tuning high-dimensional parameter spaces, the paper still includes many experiments with models that mainly use linear layers. The experimental setup of the paper covering many fields including computer vision, NLP and multimodal tasks, and shows good performance.\n3)\tThe authors compare FLoRA with LoRA and DoRA, using both theoretical reasoning and empirical evidence. They employ metrics like Frobenius norm and feature amplification factor, offering insights into learning patterns. Notably, they explore the correlation between Frobenius norm and task-specific information amplification.", "weaknesses": "1)\tThe performance gains of FLoRA on linear layer-based models appear limited. While the motivation to address topological distortions in high-dimensional parameter spaces is commendable, it potentially constrains FLoRA's applicability across diverse model architectures. Although the analysis utilizing Frobenius norm metrics and unrestricted low-rank subspaces is insightful, I recommend enhancing the paper through: A more rigorous theoretical treatment, exploring deeper mathematical foundations. An expanded experimental paradigm encompassing various model architectures, tasks, and datasets. These enhancements would strengthen the generalizability claims and broaden FLoRA's potential impact in the field of parameter-efficient fine-tuning.\n2)\tThe initialization strategy in FLoRA deserves more exploration. Although the authors use a conservative approach, diverse methods could enhance performance. Advances in LoRA initialization indicate room for improvement. A study comparing various strategies might boost FLoRA's effectiveness and stability across tasks and models. I look forward to the authors expanding research on how initialization affects FLoRA's low-rank subspace, potentially leading to more robust fine-tuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)