Maintaining Structural Integrity in Parameter Spaces for Parameter Efficient Fine-tuning

## Abstract

Abstract Adapting pre-trained foundation models for various downstream tasks has been prevalent in artificial intelligence. Due to the vast number of tasks and high costs, adjusting all parameters becomes unfeasible. To mitigate this, several fine-tuning techniques have been developed to update the pre-trained model weights in a more resource-efficient manner, such as through low-rank adjustments.
Yet, almost all of these methods focus on linear weights, neglecting the intricacies of parameter spaces in higher dimensions like 4D. Alternatively, some methods can be adapted for high-dimensional parameter space by compressing changes in the original space into two dimensions and then employing low-rank matrix decomposition. However, these approaches destructs the structural integrity of the involved high-dimensional spaces.
To tackle the diversity of dimensional spaces across different foundation models and provide a more precise representation of the changes within these spaces, this paper introduces a generalized parameter-efficient fine-tuning framework, FLoRA, designed for various dimensional parameter space.
Specifically, utilizing Tucker decomposition, FLoRA asserts that changes in each dimensional parameter space are based on a low-rank core space which maintains the consistent topological structure with the original space.
It then models the changes through this core space alongside corresponding weights to reconstruct alterations in the original space. FLoRA effectively preserves the structural integrity of the change of original N-dimensional parameter space, meanwhile decomposes it via low-rank tensor decomposition. Extensive experiments on computer vision, natural language processing and multi-modal tasks validate FLoRA’s effectiveness. Codes are available at https://github.com/SJTU-DeepVisionLab/FLoRA .

## 1 Introduction

The recent introduction of foundation models Brown et al. (); Kirillov et al. (); Devlin et al. (); Liu et al. () has demonstrated unparalleled performance and potential across diverse domains in artificial intelligence. Traditionally, the adaptation of pre-trained models for downstream tasks is achieved through fully fine-tuning of all parameters Ma et al. (); Raffel et al. (); Qiu et al. (). However, as the parameter count of these foundation models escalates, the conventional approach to fully fine-tuning becomes prohibitively expensive in various aspects.

Figure: Figure 1: Difference between LoRA and FLoRA. LoRA employs low-rank matrix decomposition for each dimensional parameter space. However, for parameter space of convolution layer, the reshaping operation required by LoRA causes adjacent elements within the kernel to be separated in the reshaped matrix, disrupting the spatial locality inherent in the original convolutional space. Conversely, FLoRA asserts that the alternations of each dimensional parameter space has a low-rank core space with the consistent topological structure. This framework enables FLoRA to effectively preserve the structural integrity of the original parameter space, such as maintaining the spatial locality in convolutional operations.
Refer to caption: x1.png

To tackle this challenge, recent works Chen et al. (); Guo et al. (); He et al. (); Hu et al. () have focused on the concept of parameter-efficient fine-tuning (PEFT), aiming to minimize the number of adjustable parameters required while achieving optimal task performance. These works specifically explore methods to model the incremental update of pre-trained weights in a manner that is economical in terms of parameters, without necessitating changes to the model’s architecture Zaken et al. (); Guo et al. (); Hu et al. (). Among these works, LoRA Hu et al. () emerges as a pioneering effort, which proposes to adopt an additional term with a low-rank structure to the original weight.
Specifically, the original weight matrix $\mathbf{W}_{0}\in\mathbb{R}^{d_{1}\times d_{2}}$ remains frozen, while a learnable low-rank $\Delta\mathbf{W}$ is added to
$\mathbf{W}_{0}$, with the form as

$$ $\mathbf{W}_{0}\rightarrow\mathbf{W}_{0}+\Delta\mathbf{W}=\mathbf{W}_{0}+ \mathbf{B}\mathbf{A}^{\mathsf{T}},$ (1) $$

where $\mathbf{A}\in\mathbb{R}^{d_{2}\times r}$ and $\mathbf{B}\in\mathbb{R}^{d_{1}\times r}$ with $r\ll\{d_{1},d_{2}\}$. Given that $r$ is significantly smaller than the dimension of $\mathbf{W}_{0}$, LoRA necessitates the updating of only a limited number of trainable parameters, while achieving comparable performances. Subsequent to LoRA, recent studies He et al. (); Bershatsky et al. (); Zhang et al. () endeavor to explore more efficient low-rank matrix decomposition methods concerning the matrix $\Delta\mathbf{W}$.

However, we have observed an intriguing phenomenon: a significant portion of existing works is narrowly focused on two-dimensional parameter spaces (i.e., linear layers), neglecting the existence of other dimensional spaces such as 2D convolution (4-dimension) layers, etc.
But in practice, not all layers are linear across various models for downstream tasks. For example, ConvNeXt Liu et al. () and Stable Diffusion Carreira and Zisserman () are two models predominantly utilizing 2D convolutional layers.
Alternatively, some approaches can be adapted for high-dimensional parameter spaces by directly adopting low-rank matrix decomposition.
They transform alternations in high-dimensional spaces into two dimensions, while neglecting the structural complexities of the original spaces.
For instance, as detailed in Sec. 3.1 and Fig. 1, LoRA Hu et al. () compresses the changes in convolutions, which is a four-dimensional parameter space, into two dimensions. It subsequently applies low-rank matrix decomposition to the two-dimension space, , intending to represent the alterations of the original four-dimensional parameter space. Yet, as discussed in Sec. 3.2, this transformation fails to capture the inherent complexity and spatial locality specific to convolution operations. The result is a reshaped two-dimensional structure that compromises the integrity of the original parameter space, leading to a representation that does not fully encapsulate the convolutional alternation parameter space.

To this end, in this paper, we propose a low-rank tensor decomposition based method, FLoRA, represented as Fundamental LOw-Rank Adaptation. Positioned as a superior alternative to LoRAs, FLoRA, based on Tucker decomposition Tucker (), meets the ensuing three properties:

- •
It can identify a appropriate low-rank representation for the changes in various dimensional parameter spaces, without destructing the structural integrity of the original parameter spaces.
- •
It can maintain a consistent formulation across various dimensional parameter spaces.
- •
When applied to linear weights, with the same parameter budget, it requires a similar training time and resources to that of LoRAs, yet yield superior performance.

Specifically, since a much lower rank than the direct rank of parameter space is sufficient to represent the original space Aghajanyan et al. (); Li et al. (), FLoRA asserts that the alternations of each dimensional parameter space, whether 2D or 4D, have a corresponding core space. This core space is low-rank and retains the same spatial dimensions (i.e., 2D or 4D) as the original parameter space, suggesting that they share a consistent topological structure. FLoRA then models the alternations by using this core space combined with a series of weights to reconstruct alternations in the original parameter space. Thanks to the intrinsic properties in the structure of the core space, FLoRA efficiently maintains the structural integrity of the original parameter space. Extensive experiments are conducted with several pretrained models on computer vision, natural language processing and multi-modal tasks, which validates that regardless of the model, the kind of downstream task, or the dimensionality of the parameter space, FLoRA’s performance surpasses that of LoRA and other existing methods.

The contributions of this paper are as follows:

- •
We propose a novel PEFT method, FLoRA. To the best of our knowledge, it is the first time that a PEFT method has been designed for different dimensional parameter spaces, aiming to preserve their topological structure while seeking low-rank representations.
- •
Extensive experiments on different tasks, include computer vision, natural language processing and multi-modal tasks, demonstrates that FLoRA significantly surpasses other baselines, validating the effectiveness of FLoRA.

## 2 Related Work

Methods for Parameter-Efficient Fine-tuning (PEFT) have been conceived to mitigate the substantial computational costs associated with the fine-tuning of large-scale models. This economization is realized by honing a comparatively minute fraction of the overall parameters, selected strategically for adaptation to a variety of downstream tasks. Current PEFT techniques can be divided into three distinct categories Liu et al. (); Ding et al. (): Adapter-based Houlsby et al. (); Chen et al. (); Luo et al. (); He et al. (); Mahabadi et al. (); Karimi Mahabadi et al. (), prompt-based Lester et al. (); Razdaibiedina et al. (); Wang et al. (); Shi and Lipani (); Fischer et al. () and low-rank matrix decomposition-based Hu et al. (); Liu et al. (); Hyeon-Woo et al. (); Qiu et al. (); Renduchintala et al. (); Kopiczko et al. (); YEH et al. (); Zhang et al. (). The first category of method integrates linear modules either sequentially or concurrently with the existing layer to enhance the performance, and the second class introduces additional soft tokens (prompts) to the initial input and concentrate exclusively on refining these trainable vectors. The last type, proposed by LoRA Hu et al. (), adopts low-rank matrix decomposition to model the change of the weight during fine-tuning, and are capable of merging with pre-trained weights.

However, as mentioned in the introduction, these methods only focuses on linear weight or destructing the structural integrity of high dimensional parameter spaces. To this end, in this paper we propose a novel PEFT method, aimed to address various dimensional parameter space.

## 3 Preliminaries

### 3.1 Low Rank Adaptation

LoRA Hu et al. () models the incremental update of a pre-trained weight matrix $\mathbf{W}_{0}\in\mathbb{R}^{d_{1}\times d_{2}}$ by the product of two small matrices $\mathbf{A}\in\mathbb{R}^{d_{2}\times r}$ and $\mathbf{B}\in\mathbb{R}^{d_{1}\times r}$, where $r\ll\{d_{1},d_{2}\}$. For $\mathbf{h}=\mathbf{W}_{0}\mathbf{x}$, the modified forward pass is

$$ $\mathbf{h}=\mathbf{W}_{0}\mathbf{x}+\Delta\mathbf{W}\mathbf{x}=\mathbf{W}_{0} \mathbf{x}+\mathbf{B}\mathbf{A}^{\mathsf{T}}\mathbf{x}.$ (2) $$

Matrix $\mathbf{A}$ is initialized with a random Gaussian distribution, and $\mathbf{B}$ with zeros, setting the initial $\Delta\mathbf{W}$ to zero for training. LoRA’s application is straightforward to the linear layers, while for a convolution layer characterized by weights $\mathcal{W}_{c}\in\mathbf{R}^{d_{in}\times d_{out}\times k\times k}$, with $d_{in}$ / $d_{out}$ denoting the dimension of input / output respectively and $k$ representing the kernel size, LoRA is adapted based on matrix decomposition:

$$ $\mathcal{W}=\mathcal{W}_{c}+\Delta\mathcal{W}=\mathcal{W}_{c}+Reshape(\mathbf{ B}_{c}\mathbf{A}_{c}^{\mathsf{T}},\mathcal{W}_{c}),$ (3) $$

where $\mathbf{A}_{c}\in\mathbb{R}^{kd_{out}\times r}$ and $\mathbf{B}_{c}\in\mathbb{R}^{kd_{in}\times r}$. Here, $Reshape(\mathbf{B}_{c}\mathbf{A}_{c}^{\mathsf{T}},\mathcal{W}_{c})$ involves altering the dimensions of $\mathbf{B}_{c}\mathbf{A}_{c}^{\mathsf{T}}$ to match those of $\mathcal{W}_{c}$. It is obvious that LoRA unfolds the original 4-dimensional parameter space $\Delta\mathcal{W}$ into a 2-dimensional space, subsequently leveraging the low-rank approximation of this 2-dimensional space to represent the original 4-dimensional construct. As shown later, this low-rank matrix decomposition based method will destruct the structural integrity of the convolution layer.

### 3.2 Why Matrix Decomposition Breaks the Structural Integrity of the Convolution?

A high-dimension tensor can be decomposed based on low-rank matrix decomposition following Eq. (3). However, during the reshaping process, elements that were adjacent within the convolutional kernel may become scattered across various positions in the matrix. More specifically, elements that were localized within the convolutional kernel may now span multiple rows or columns of the reshaped matrix. This shift poses significant challenges in learning spatial correlations among elements positioned disparately. Therefore, such a transformation disrupts the principle of locality inherent in the original convolutional operation, where each output element is determined by a small region of the input.

### 3.3 Tucker Decomposition

Tucker decomposition Tucker () is one of the well-studied algebraic tensor decomposition. Formally, given a tensor $\mathcal{X}\in\mathbb{R}^{I_{1}\times I_{2}\times\cdots\times I_{N}}$, where $N$ is the order of the tensor (i.e., the number of dimensions or modes), Tucker decomposition represents $\mathcal{X}$ as a product of a core tensor $\mathcal{G}\in\mathbb{R}^{J_{1}\times J_{2}\times\cdots\times J_{N}}$ and a matrix along each mode $n$, $\mathbf{A}^{(n)}\in\mathbb{R}^{I_{n}\times J_{n}}$, where $J_{n}$ can be considered as the dimension of the core tensor along mode $n$. The decomposition can be compactly written as:

$$ $\mathcal{X}=\mathcal{G}\times_{1}\mathbf{A}^{(1)}\times_{2}\mathbf{A}^{(2)} \times\cdots\times_{N}\mathbf{A}^{(N)},$ (4) $$

where $\times_{n}$ denotes the or mode-$n$ product between a tensor and a matrix.
The core tensor $\mathcal{G}$ represents the interactions between different modes, and the matrices $\mathbf{A}^{(n)}$ are analogues to principal components within each respective mode. The selection of dimensions $J_{1},J_{2},\ldots,J_{N}$ allows for a balance between desired approximation quality and computational efficiency, tailored to the specific requirements of the task at hand.
Tucker decomposition decomposes any dimensional space into a core tensor and a set of matrices related to the dimensions, without altering the original structure. Specifically, these matrices only transform the representation of different dimensions, which means that as long as the core tensor accurately captures the relationships among these dimensions, the property of original parameter space can be preserved through this decomposition.

## 4 Method

In this section, we first introduce the formulation of FLoRA in an N-dimensional parameter space. Specifically, for a pre-trained weight $\mathcal{W}_{0}\in\mathbb{R}^{I_{1}\times I_{2}\times\cdots\times I_{N}}$ with N-dimension, we model its update $\Delta\mathcal{W}$ as

$$ $\mathcal{W}_{0}\rightarrow\mathcal{W}_{0}+s*\Delta\mathcal{W}=\mathcal{W}_{0}+ s*\mathcal{G}\times_{1}\mathbf{A}^{(1)}\times_{2}\mathbf{A}^{(2)}\times\cdots \times_{N}\mathbf{A}^{(N)}$ (5) $$

without loss of generality. Here $\mathcal{G}\in\mathbb{R}^{J_{1}\times J_{2}\times\cdots\times J_{N}}$ and $\mathbf{A}^{(n)}\in\mathbb{R}^{I_{n}\times J_{n}}$, where $J_{n}\ll I_{n}$. FLoRA considers the tensor $\mathcal{G}$ as a low-rank core space with the consistent topological structure to the original parameter space, with $\mathbf{A}^{(n)}$ representing the weights associated with each dimension. We then scale $\Delta\mathcal{W}$ by $s$ with $s$ being a constant. In the subsequent subsections, we will detail its manifestations in convolution and linear layers.

### 4.1 FLoRA for Convolution Layer

Convolution operations in deep learning are characterized by a four-dimensional parameter space, encapsulated in a weight tensor $\mathcal{W}_{c}\in\mathbf{R}^{d_{in}\times d_{out}\times k\times k}$, with $d_{in}$ / $d_{out}$ denoting the dimension of input / output respectively and $k$ representing the kernel size. A pivotal property to consider within $\mathcal{W}_{c}$ is the spatial locality, which plays an essential role in the convolution layer’s ability to compile and process information across the input matrix. This process is facilitated by the kernel’s spatial dimensions ($k\times k$), which determines the scope of the input data scrutinized by each convolution operation. To preserve the attributes of the spatial locality and uphold the convolution parameter space’s integrity, FLoRA models the update $\Delta\mathcal{W}$ for a convolution layer as

$$ $\Delta\mathcal{W}=\mathcal{G}\times_{1}\mathbf{A}\times_{2}\mathbf{B}\times_{3 }\mathbf{C}\times_{4}\mathbf{D},$ (6) $$

where $\mathcal{G}\in\mathbb{R}^{r_{1}\times r_{2}\times r_{3}\times r_{3}}$, $\mathbf{A}\in\mathbb{R}^{d_{in}\times r_{1}}$, $\mathbf{B}\in\mathbb{R}^{d_{out}\times r_{2}}$ and $\mathbf{C},\mathbf{D}\in\mathbb{R}^{k\times r_{3}}$. $r_{1},r_{2}$ are ranks which are significantly smaller than $\{d_{in},d_{out}\}$. $r_{3}$ is a rank smaller than the kernel size $k$ of a convolution layer. Given that $3\times 3$ convolution is a prevalent configuration in convolutional foundation models Woo et al. (); Rombach et al. (); Wang et al. (), $r_{3}$ is consequently set to {1, 2}.

The core tensor $\mathcal{G}$ in FLoRA can be viewed as a compressed convolution parameter space. In essence, it serves as a core space for convolution. This means that in any convolution layer, there exists a convolution core, and what FLoRA aims to do is to determine this convolution core for each convolution space and configure the corresponding weights $\mathbf{A}$, $\mathbf{B}$, $\mathbf{C}$ and $\mathbf{D}$ for reconstructing that space. Different to low-rank matrix decomposition based methods, FLoRA dose not need to alter the structure of the convolution. Alternatively, by learning the convolution core, FLoRA effectively preserves the convolution’s property of spatial locality.

Furthermore, while preserving or potentially augmenting the representational power of the convolution process, FLoRA achieves a remarkable reduction in the number of trainable parameters in comparison to LoRAs. Assuming that the rank for both the input and output dimensions is uniform ($r_{1}=r_{2}=r$), the parameter requirement for FLoRA is calculated as $O(r_{3}^{2}r^{2}+r(d_{in}+d_{out})+2r_{3}k)$ parameters, while LoRAs necessitate training at least $O(rk(d_{in}+d_{out}))$ parameters. Given that typically $r,k\ll\{d_{in},d_{out}\}$, therefore, FLoRA exhibits better parameter efficacy than LoRA as the number of the kernel increases.

### 4.2 FLoRA for Linear Layer

For a linear layer with weight $\mathbf{W}_{0}\in\mathbb{R}^{d_{1}\times d_{2}}$, FLoRA models the the update $\Delta\mathbf{W}$ as

$$ $\Delta\mathbf{W}=\mathcal{G}\times_{1}\mathbf{A}\times_{2}\mathbf{B}=\mathbf{A }\mathbf{G}\mathbf{B}^{\mathsf{T}},$ (7) $$

where $\mathbf{G}\in\mathbb{R}^{r_{1}\times r_{2}}$, $\mathbf{A}\in\mathbb{R}^{d_{1}\times r_{1}}$ and $\mathbf{B}\in\mathbb{R}^{d_{2}\times r_{2}}$.
Similar to that for convolution layer, the core matrix $\mathbf{G}$ can be viewed as a core space for the 2-dimension parameter space, with $\mathbf{A}$ and $\mathbf{B}$ being the corresponding weights to reconstruct the alternations in linear space.

Figure: Figure 2: The normalized performance improvement of FLoRA over other baselines
Refer to caption: x2.png

## 5 Experiment

### 5.1 Models and Datasets

We conduct comprehensive experiments across computer vision (CV), natural language processing (NLP) and multi-modal (MM) tasks.

Specifically, for CV tasks, we employ FLoRA to fine-tune the ConvNeXt-V2-L Woo et al. (), evaluating it on MS COCO Lin et al. () by using Mask R-CNN He et al. () implemented in MMDetection Chen et al. (), and on remote sensing image datasets DOTA Xia et al. () with Oriented R-CNN Xie et al. () based on MMRotate Zhou et al. (), and on the ADE20K Zhou et al. () dataset thanks to UperNet Xiao et al. () integrated in MMSegmentation Contributors ().
We also employ FLoRA to fine-tune large-scale vision foundation model, i.e. InternViT-6B Chen et al. (), on ADE20K datasets. Detailed hyper-parameter settings can be find in Appendix.

For NLP tasks, we evaluate the DeBERTaV3-base He et al. () with FLoRA on the General Language Understanding Evaluation (GLUE) Wang et al. () benchmark, which includes two single-sentence classification, three similarity and paraphrase and four natural language inference datasets. More details on GLUE dataset can be found in the Appendix.

For multi-modal tasks, we employ FLoRA to fine-tune LLaVA-1.5-7B Liu et al. (), which is composed of a language model, Vicuna-1.5-7B Peng et al. () and a vision encoder, CLIP ViT-L/336px Radford et al. (), on visual instruction tuning tasks, which include seven vision-language benchmarks: VQA^v2 Goyal et al. (), GQA Hudson and Manning (), VisWiz Gurari et al. (), SQA Lu et al. (), VQA^T Singh et al. (), POPE Li et al. (), and MMBench Liu et al. ().

Moreover, we have also fine-tune Stable Diffusion Rombach et al. () with FLoRA in Appendix.

### 5.2 Baselines

We compare FLoRA with several state-of-the-art methods: fully fine-tuning, BitFit Zaken et al. (), HAdapter Houlsby et al. (), PAdapter Pfeiffer et al. (), AdaLoRA Zhang et al. (), the most representative low-rank method, LoRA Hu et al. (), and the state-of-the-art low-rank adaption method, DoRA Liu et al. ().
Specifically, HAdapter is strategically placed between the self-attention module and the FFN module, and it includes a subsequent residual connection. Conversely, PAdapter introduces a more streamlined design, implementing adapters exclusively after the FFN modules and LayerNorm modules. Furthermore, following Zhang et al. (), we apply AdaLoRA, LoRA and DoRA to all weight matrices or tensors. For more details on the baselines, please refer to their original papers.

### 5.3 Implementation Details

We compare FLoRA with other PEFT methods under different parameter budgets. The hidden dimension of Adapters is chosen from {8, 16, 32, 64}, the budget of AdaLoRA is chosen from {144, 288,567} and the rank $r$ of LoRA and DoRA is selected from {2, 4, 8, 16, 32}. Other hyper-parameters are initialized according to their original papers.
Furthermore, we simply set $r_{1}=r_{2}$ for FLoRA. The scale $s$ is chosen from {0.04, 0.4, 4} for different foundation models. We use publicly available PyTorch Paszke et al. () implementation to execute all the baseline comparisons, and all the experiments are conducted on NVIDIA A100 GPUs.
For NLP tasks, the mean of 5 runs using various random seeds are reported for all the experiments, and all gains have passed the pairwise $t$-test with a significance of 0.05.
The tensor $\mathcal{G}$ is initialized as zero, while other weight matrices are initialized as random Gaussian. For more training details, please refer to the Appendix.

**Table 1: Results with ConvNeXt-V2-L Woo et al. () fine-tuned on different datasets. The best performances are shown in bold. “Base” represents for the pre-trained backbone with frozen weights.**
| Method | # Params | COCO | ADE20K | DOTA | All |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| mAP | AP50 | AP75 | APS | APM | APL | mAcc | mIoU | mAP | AP50 | AP75 | Avg. |  |  |
| Base | - | 37.3 | 63.3 | 39.7 | 27.8 | 41.0 | 46.2 | 59.6 | 48.5 | 31.4 | 61.8 | 27.3 | 44.0 |
| Fully FT | 196M | 52.7 | 74.3 | 58.7 | 38.3 | 56.9 | 67.3 | 64.7 | 53.1 | 33.9 | 59.9 | 34.0 | 54.0 |
| BitFit | 0.2M | 43.1 | 67.6 | 47.4 | 29.5 | 46.7 | 55.3 | 61.2 | 49.1 | 34.6 | 64.2 | 32.9 | 48.3 |
| LoRA | 12.94M | 47.4 | 70.3 | 53.0 | 32.4 | 51.8 | 61.2 | 63.6 | 51.4 | 18.3 | 36.2 | 16.3 | 45.6 |
| DoRA | 13.07M | 47.2 | 69.8 | 52.7 | 32.1 | 51.5 | 61.4 | 63.0 | 50.9 | 19.6 | 37.9 | 17.2 | 45.8 |
| FLoRA | 12.77M | 48.1 | 71.1 | 53.6 | 33.1 | 52.3 | 62.3 | 64.1 | 51.9 | 37.3 | 65.6 | 37.7 | 52.5 |
| LoRA | 25.89M | 48.0 | 70.4 | 53.6 | 33.0 | 52.3 | 62.8 | 63.9 | 51.4 | 20.0 | 38.3 | 18.3 | 46.5 |
| DoRA | 26.16M | 48.1 | 70.7 | 53.6 | 33.1 | 52.1 | 62.6 | 64.0 | 51.9 | 21.1 | 39.7 | 19.1 | 46.9 |
| FLoRA | 25.65M | 49.2 | 71.7 | 54.7 | 34.3 | 53.3 | 63.5 | 65.0 | 52.6 | 38.8 | 68.4 | 39.5 | 53.7 |
| LoRA | 51.78M | 48.2 | 70.7 | 53.7 | 33.4 | 52.5 | 62.7 | 63.9 | 51.6 | 20.4 | 39.4 | 19.1 | 46.9 |
| DoRA | 51.95M | 44.0 | 68.2 | 48.9 | 29.1 | 48.2 | 57.5 | 64.1 | 51.9 | 21.7 | 40.9 | 20.3 | 45.0 |
| FLoRA | 40.49M | 50.4 | 72.6 | 56.2 | 35.4 | 54.6 | 64.8 | 65.1 | 52.8 | 39.7 | 69.0 | 40.9 | 54.7 |

**Table 2: Results with InternViT-6B Chen et al. () fine-tuned on ADE20K.**
| Method | Base | Fully FT | BitFit | LoRA | DoRA | FLoRA | LoRA | DoRA | FLoRA |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| # Params (%) | - | 100 | 0.15 | 0.32 | 0.32 | 0.32 | 0.66 | 0.66 | 0.66 |
| mAcc | 64.5 | 70.4 | 68.7 | 69.7 | 69.5 | 70.1 | 70.3 | 69.6 | 71.1 |
| mIoU | 53.0 | 58.2 | 56.1 | 57.1 | 57.3 | 57.7 | 57.3 | 57.1 | 58.0 |
| Avg. | 58.8 | 64.3 | 62.4 | 63.4 | 63.4 | 63.9 | 63.8 | 63.4 | 64.6 |

**Table 3: Results with DeBERTaV3-base He et al. () fine-tuned on GLUE datasets.**
| Method | # Params | MNLI | SST-2 | CoLA | QQP | QNLI | RTE | MRPC | STS-B | All |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| m | Acc | Mcc | Acc | Acc | Acc | Acc | Corr | Avg. |  |  |
| Fully FT | 184M | 89.90 | 95.63 | 69.19 | 92.40 | 94.03 | 83.75 | 89.46 | 91.60 | 88.24 |
| BitFit | 0.1M | 89.37 | 94.84 | 66.96 | 88.41 | 92.24 | 78.80 | 87.75 | 91.35 | 86.21 |
| HAdapter | 1.22M | 90.13 | 95.53 | 68.64 | 91.27 | 94.11 | 84.48 | 89.95 | 91.48 | 88.19 |
| PAdapter | 1.18M | 90.33 | 95.61 | 68.77 | 91.40 | 94.29 | 85.20 | 89.46 | 91.54 | 88.32 |
| AdaLoRA | 1.33M | 90.38 | 95.87 | 71.45 | 91.19 | 94.36 | 88.09 | 90.69 | 91.84 | 89.23 |
| LoRA | 0.33M | 90.03 | 93.92 | 69.15 | 90.61 | 93.37 | 85.56 | 90.19 | 90.75 | 87.95 |
| DoRA | 0.41M | 90.21 | 94.38 | 69.33 | 90.84 | 93.26 | 86.94 | 90.19 | 91.34 | 88.31 |
| FLoRA | 0.33M | 90.60 | 96.00 | 70.20 | 91.40 | 94.46 | 88.09 | 90.93 | 91.96 | 89.21 |
| LoRA | 1.33M | 89.80 | 93.69 | 69.30 | 91.78 | 92.97 | 85.70 | 90.68 | 91.62 | 88.17 |
| DoRA | 1.41M | 89.67 | 94.61 | 69.08 | 91.80 | 93.23 | 87.33 | 90.68 | 91.73 | 88.49 |
| FLoRA | 1.33M | 90.82 | 96.21 | 72.05 | 91.94 | 94.60 | 89.53 | 91.18 | 92.04 | 89.80 |

### 5.4 Main Results

We compare FLoRA with other baselines under different parameter budgets. Specifically, ConvNeXt-V2-L is based on convolutions, and other large foundation models are dominantly based on linear layers. We evaluate FLoRA’s effectiveness on high-dimension space for the CV tasks, and on linear parameter space for CV and NLP tasks. The results are shown in Tables. 1-3 and Table in Appendix, and the normalized performance are illustrated in Fig. 2.

For CV tasks, FLoRA employed on ConvNeXt-V2-L achieves superior performances compared to other baselines. On average, FLoRA outperforms LoRA and DoRA by at least 15% under different parameter budgets. With nearly 80% reduction in parameter budget, FLoRA (12.77M) still significantly outperforms LoRA (51.78M), DoRA (51.95M), validating that FLoRA successfully preserves the structural integrity of convolutions.
Specifically, FLoRA achieves comparable or even superior performance to fully fine-tuning, while others lag far behind. Additionally, we observe that on tasks with a large domain gap from the pre-training data, e.g. remote sensing images (DOTA), LoRA and DoRA performs significantly worse, while FLoRA maintains consistent superiority. This further validates the robustness of FLoRA, even when facing tasks with large domain gaps.
Moreover, when FLoRA is employed to fine-tune InternViT-6B, FLoRA consistently outperforms the baseline across various parameter budgets. Remarkably, by fine-tuning only 0.66% of the parameters, FLoRA achieves even better performance than fully fine-tuning.

For NLP tasks, FLoRA achieves better or on par performance compared with existing approaches on all datasets under all different parameter budgets. Specifically, under extreme low parameter budget, FLoRA performs better even than the baselines with higher parameter counts. For example, with 0.3M parameters, FLoRA’s performances on SST-2, QNLI, RTE, MRPC and STS-B are all better than baselines with larger parameter budgets.

For multi-modal tasks, FLoRA also achieves SOTA performances compared with baselines. Therefore, at this point, we can conclude that overall, FLoRA achieves remarkable performances across various tasks, model backbones, and dimensions of parameter spaces.

## 6 Further Analysis

In this section, we explore the properties of FLoRA when applied to downstream tasks and in comparison with other baselines like LoRA. We carry out a series of empirical studies to address the following questions: 1) Is the core space truly low-rank? 2) If so, why is FLoRA’s low-rank representation better than other methods like LoRA? 3) Does FLoRA require acceptable training costs compared to other methods? 4) Is FLoRA sensitive to the scaling factor? The insights gained from these questions will illuminate the efficacy of FLoRA and guide future research.

Figure: (a) COCO
Refer to caption: x3.png

### 6.1 Is the Core Space Truly Low-rank?

We present the performance of FLoRA under different rank (i.e., parameter budgets), tested on ConvNeXt-V2-L and DeBERTaV3-base. The results are illustrated in Fig. 3. It is evident that FLoRA’s performance with a low rank is comparable to, or even exceeds, that of a higher rank. This finding aligns with observations in Hu et al. ().
It indicates that there indeed exists a core space in different dimensional spaces, and the rank of the core space is relatively small. When the rank set is smaller than the rank of the core space, the performance of the model is not optimal. Conversely, when it exceeds this rank, the core space is completely covered, which introduces some meaningless redundancy and noise. Additionally, we observe that for the convolutional parameter space, the rank of its core space is much larger, for the convolutional space has a more complex topological structure, necessitating a larger rank to adequately describe it.

### 6.2 Why is FLoRA’s Low-rank Representation Better than Other Methods?

Although methods like FLoRA and LoRA can represent changes in the original parameter space using low-rank representations, FLoRA performs better, indicating that its low-rank representation is superior to other forms of low-rank representations. We will explain the reasons as follows.

#### 6.2.1 Theoretical Analysis

In linear parameter spaces, the changes modeled by methods in the LoRA derivatives, can also be simply considered as $\mathbf{AGB}$ with $\mathbf{G}$ being a diagonal matrix.
However, FLoRA does not impose any constraints on $\mathbf{G}$, which offers several advantages: firstly, it broadens the range of parameter adjustments and enhances the flexibility of parameter learning.
Secondly, FLoRA removes the strong orthogonality constraint on $\mathbf{G}$. Since orthogonality is not a characteristic of all downstream tasks, enforcing orthogonal constraints during the learning process could potentially degrade performance, as shown in Tables 1-3.
Lastly, since $\mathbf{G}$ is initialized to zero in FLoRA, under certain conditions, $\mathbf{G}$ may also learn to exhibit orthogonal properties. This indicates that the low-rank representations of methods like LoRA are a special case of FLoRA, which possesses a more robust low-rank representation capability, thereby yielding better performance.

For high-dimensional parameter spaces, as previously discussed, other methods compromise the structural integrity of the original parameter spaces. In contrast, FLoRA preserves their topological structure, leading to superior performance outcomes.

Figure: (a) Frobenius Norm
Refer to caption: x7.png

#### 6.2.2 Empirical Analysis

To further substantiate the theoretical analysis on FLoRA’s superiority, we record the average of the Frobenius norm of $\Delta\mathbf{W}$ and the feature amplification factor $\frac{\|\Delta\mathbf{W}\|_{F}}{\|\mathbf{U}^{\mathsf{T}}\mathbf{W}\mathbf{V}^
{\mathsf{T}}\|_{F}}$ Hu et al. () of all the layers in DeBERTaV3-base on CoLA during training.
Here $\mathbf{U}$ and $\mathbf{V}$ being the left- and right-singular matrices of the SVD decomposition of $\Delta\mathbf{W}$. The feature amplification factor measures how much of task-specific information are amplified by $\Delta\mathbf{W}$.

The results, illustrated in Fig. 4, show that LoRA and DoRA can amplify task-specific features more than FLoRA in the early stages. As discussed in Sec. 6.2.1, these two methods exhibit strong orthogonality, resulting in a distinct directional learning pattern at the beginning, which contributes to their larger initial values. However, strong orthogonality may not always be suitable for downstream tasks, and since downstream datasets may possess various properties, their amplification factors upon convergence are smaller than that of FLoRA.

Additionally, we found the trend of the Frobenius norm closely aligns with the feature amplification factor, surprisingly. This might indicate that larger $\Delta\mathbf{W}$ values can accommodate more task-specific information, thereby more effectively amplifying the task-specific information in the frozen weights. We also observed that, initially, the Frobenius norms of $\Delta\mathbf{W}$ for LoRA and DoRA are also larger than that of FLoRA, for they can quickly capture the orthogonal information if the downstream datasets. However, upon convergence, they are both smaller than FLoRA’s, suggesting that FLoRA can contain more kinds of properties for task-specific information.

### 6.3 Does FLoRA Require Acceptable Training Costs Compared to Other Methods?

We assess the training costs under various configurations. All training hyper-parameters, such as batch size and epochs, are kept consistent, with the results presented in Table 4. Clearly, FLoRA is more efficient in terms of training time and memory footprint compared with SOTA method DoRA, which require significantly more time and GPU memories.

**Table 4: Training time (minutes/epoch) and GPU usage (GB) of FLoRA.**
| Method | ConvNeXt-V2-L | Method | DeBERTaV3-base |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| # Params | COCO | ADE20K | # Params | MNLI | SST-2 | STS-B |  |  |  |  |  |  |  |
| Time | GPU | Time | GPU | Time | GPU | Time | GPU | Time | GPU |  |  |  |  |
| LoRA | 6.59 % | 70 | 17.19 | 53 | 11.94 | LoRA | 0.18 % | 73.57 | 11.35 | 6.38 | 6.85 | 0.56 | 6.85 |
| DoRA | 6.59 % | 100 | 30.67 | 67 | 17.36 | DoRA | 118.42 | 16.72 | 11.26 | 9.66 | 0.91 | 9.66 |  |
| FLoRA | 6.50 % | 77 | 17.94 | 60 | 13.10 | FLoRA | 79.57 | 11.35 | 6.83 | 6.86 | 0.56 | 6.85 |  |
| LoRA | 26.36 % | 77 | 18.11 | 52 | 12.79 | LoRA | 0.72 % | 74.29 | 11.39 | 6.45 | 6.88 | 0.59 | 6.87 |
| DoRA | 26.36 % | 111 | 31.41 | 69 | 17.86 | DoRA | 117.14 | 16.75 | 11.79 | 9.69 | 0.92 | 9.69 |  |
| FLoRA | 26.06 % | 74 | 18.23 | 58 | 13.45 | FLoRA | 78.57 | 11.39 | 6.67 | 6.88 | 0.60 | 6.87 |  |

### 6.4 Is FLoRA Sensitive to the Scaling Factor?

We report the sensitivity w.r.t. the scale $s$ on ConvNeXt-V2-L and DeBERTaV3-base, and the results are shown in Fig. 5. It is obvious that the performance of FLoRA is quite stable with the scale changing in a reasonable range, which is a desirable property in practice.

Figure: (a) COCO
Refer to caption: x9.png

## 7 Conclusion and Limitation

In this paper, we propose a generalized low-rank tensor decomposition based PEFT method, FLoRA, aiming for N-dimension parameter space. FLoRA asserts that the alternations in each dimensional parameter space contain a low-rank core space structurally consistent to the original space. It models updates using this core space alongside corresponding weights to reconstruct the original alternation space. By doing so, FLoRA effectively preserves the structural integrity of the original N-dimensional parameter space while decomposing it through low-rank tensor decomposition. Extensive experiments on computer vision, natural language processing and multi-modal domains substantiate the effectiveness of FLoRA.

There still exists some limitations for FLoRA. For a specific backbone, FLoRA could achieve stable superior performance on different datasets when the scaling factor $s$ changing within a wide range. While for different backbones such as ConvNeXt-V2-L, InternViT-6B and DeBERTaV3-base, it still needs different scales. Understanding the role of scale in different models and designing a unified scale is a topic worthy of further investigation.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: Please see weaknesses above. Below are additional questions:
* What exactly is the difference between the current method and existing methods, particularly LoTR? The decomposition appears identical since both methods use Tucker decomposition, yet the paper claims that there are differences. It is hard to appreciate the paper without knowing about the differences between this work and past methods.
* How is the feature amplification factor considered a reliable indicator of task-specific information amplification, and what insights does it provide regarding task-specific performance? Furthermore, what methods are employed to measure task-specific information in the context of this study?
* What significance does the average Frobenius norm of $\Delta W$ hold for FLoRA's performance during fine-tuning? Can the magnitude of this norm be directly correlated to the effectiveness of task-specific adaptation? The presented correlation is based solely on a single sample, and there is no clear explanation provided for why this correlation would exist in the first place.
* How does the feature amplification factor of FLoRA in the DeBERTaV3-base model compare to that of other fine-tuning methods applied to the CoLA dataset? Does FLoRA exhibit a distinct pattern in terms of norm growth or amplification factor throughout the training process?
* If different versions of the DeBERTa model were evaluated, would the Frobenius norm or amplification factor be affected by changes in model architecture, or are these metrics consistently reliable across various transformer-based models?
* Without a direct comparison of the Frobenius norm and amplification factor metrics against alternative fine-tuning approaches, such as standard LoRA or full fine-tuning, it is challenging to assert FLoRA's superiority. If these metrics do not demonstrate a clear advantage over other methods, this could weaken the strength of the conclusions drawn in the paper.
soundness: 3
strengths: + The motivation behind maintaining the topological structure of the pre-trained matrices is compelling, providing a strong foundation for the proposed approach. 

+ The paper is well-organized and easy to follow, with each section clearly building upon the previous. 

+ The analysis section aligns well with the proposed objectives, and the approach to evaluating whether the core space is low-rank adds depth to the analysis.
summary: This paper introduces FLoRA, a parameter-efficient fine-tuning (PEFT) method for adapting pre-trained foundation models across various parameter spaces while preserving the model’s structural integrity. FLoRA minimizes the computational burden of full fine-tuning by maintaining the original parameter organization, allowing efficient task-specific adaptability of different architectures. This approach supports scalable adaptation in large models, optimizing both resource use and performance. Empirical and theoretical evaluations show FLoRA’s robustness, highlighting its promise as an efficient approach for model adaptation.
weaknesses: - The work appears to have similarities with existing approaches such as LoTR and LoKR; however, the paper does not adequately address how FLoRA distinguishes itself from these methods. This lack of clarification raises serious concerns about the novelty of the proposed approach.
- The manuscript does not provide sufficient evidence to demonstrate how the representations in FLoRA surpass those of LoRA and DoRA. It remains unclear how FLoRA broadens parameter adjustments and how this relates to enhancing parameter learning flexibility.
- The feature amplification factor introduced in the study may lack a direct correlation with actual task performance gains, making it difficult to interpret its significance in relation to improved task-specific outcomes. Currently, the correlation is indicated only using a single dataset and model architecture.
- The correlation observed between the Frobenius norm and feature amplification could be incidental rather than causal. There is insufficient evidence to support the claim that a larger norm consistently leads to enhanced task-specific performance.
- The scalability and computational efficiency of the proposed approach may be limited when applied to extremely high-dimensional tensors, potentially affecting its practicality for very large foundation models.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: See weaknesses.
soundness: 3
strengths: 1. The generalization of LoRA on N-dimension parameter space with form of Tucker decomposition is interesting and effective.

2. The authors conduct many experiments on multiple kinds of tasks and validate the effectiveness of the proposed FLoRA method.

3. The paper is well-written and easy to follow.
summary: This paper proposes a new generalized PEFT method, FLoRA, for N-dimension parameter space. FLoRA is based on the Tucker decomposition and uses a low-rank core tensor and N low-rank matrices to reconstruct the original parameter tensors. The introduction of the low-rank core space helps preserve the structural integrity of the original parameters space. Experiments on CV, NLP and multi-modal tasks validate the effectiveness of the proposed method over baselines like LoRA and DLoRA.
weaknesses: 1. In Line 183, the paper mentions that “in any convolution layer, there exists a convolution core”. Is this an assumption or supported by evidences and theories? More discussion should be added to improve the reliability of this claim.

2. How does the learned convolution core help preserve the convolution’s property of spatial locality?

3. It would be better to compare the FLoRA method with some PEFT method designed for CV tasks, such as [1].

4. Can you provide the standard deviations of the main results (Table 1 etc.)?

[1] Jie, S. and Deng, Z.-H. Fact: Factor-tuning for lightweight adaptation on vision transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 1060–1068, 2023.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Already listed in the Weaknesses section.
soundness: 3
strengths: * **Significant Performance Gains:** The paper convincingly demonstrates FLoRA's superior performance over existing PEFT methods on both ConvNeXt (Table 1) and InternViT (Table 2) architectures.  The improvements are particularly noteworthy on ConvNeXt, which relies heavily on convolutional layers, highlighting FLoRA's strength in handling high-dimensional parameter spaces.  The results show that FLoRA not only outperforms other low-rank adaptation methods like LoRA and DoRA by a substantial margin (at least 15% on average across different parameter budgets) but also achieves performance comparable to, and in some cases even better than, full fine-tuning. 

* **Parameter Efficiency and Practicality:** The authors emphasize FLoRA's practicality by showcasing its ability to achieve these strong results while significantly reducing the number of trainable parameters and training time. The paper explicitly states that FLoRA can achieve comparable performance to full fine-tuning with an 80% reduction in parameter budget.  Additionally, Table 5 provides evidence of FLoRA's efficiency in terms of both training time and GPU memory usage compared to other methods, particularly DoRA.

* **Insightful Analysis of Low-Rank Representation:** The paper goes beyond simply presenting performance results. It provides a detailed analysis of why FLoRA's low-rank representation is more effective than other methods. Figure 4, which compares the Frobenius norm and feature amplification factor of FLoRA, LoRA, and DoRA during training, offers valuable insights. This analysis reveals that while LoRA and DoRA might initially amplify task-specific features more aggressively due to their constraints on matrix patterns, FLoRA's less constrained approach allows it to capture a broader range of task-specific information, leading to higher amplification factors upon convergence. The paper also suggests a strong correlation between the Frobenius norm of the learned changes (∆W) and the ability to capture task-specific information.

* **Clear and Well-Written:** The paper is well-structured, and the concepts are explained in a clear and concise manner.  This clarity makes it easy for readers to understand the motivation behind FLoRA, its technical details, and the significance of the results. 

* **Addressing a Gap in PEFT Research:** The authors clearly identify a gap in existing PEFT research, which primarily focuses on linear layers while neglecting the complexities of high-dimensional parameter spaces. FLoRA is presented as a solution to this problem, demonstrating its novelty and potential impact on the field.
summary: The paper introduces FLoRA as a new approach to Parameter-Efficient Fine-Tuning (PEFT), focusing on preserving the structural integrity of high-dimensional spaces in large models.  The authors argue that existing PEFT methods, especially those relying on low-rank matrix adaptations, often disrupt the crucial spatial relationships within these high-dimensional spaces, particularly in convolutional layers, leading to suboptimal performance. 

FLoRA addresses this issue by introducing a low-rank core space that maintains the original spatial dimensions of the parameter space being adapted. For example, for a convolutional layer, FLoRA employs a 4D core tensor that mirrors the structure of the convolutional kernel. This ensures that spatial locality, crucial for convolution operations, is preserved during fine-tuning. 

The core space is then combined with corresponding weights to reconstruct the changes in the original parameter space. This approach allows FLoRA to achieve high performance while being efficient in terms of trainable parameters. The paper demonstrates FLoRA's effectiveness across various tasks in computer vision, natural language processing, and multi-modal learning, showcasing its superior performance compared to existing PEFT methods.

* **FLoRA outperforms traditional low-rank methods like LoRA, especially in tasks involving convolutional layers.** This is attributed to FLoRA's ability to preserve the spatial structure of high-dimensional parameter spaces.
* **FLoRA achieves performance comparable to or even exceeding that of full fine-tuning, while using significantly fewer trainable parameters.** This makes it a highly efficient and effective method for adapting large models to downstream tasks.
* **The paper provides empirical evidence for the existence of a low-rank core space within different dimensional parameter spaces.** This supports FLoRA's central premise and suggests its potential for broader applicability.
weaknesses: *   **Limited Scope of LLM Fine-tuning Experiments:** The paper lacks experiments on LLM fine-tuning (e.g., LLaMA3-8B), despite PEFT methods being commonly used in these scenarios. Evaluating FLoRA's efficacy on LLMs would enhance the paper's practicality and relevance.
*   **LoRA's Applicability in Vision-Based Models:** The paper focuses on LoRA as a comparison point for FLoRA, but LoRA is not a standard fine-tuning technique for vision-based models like ConvNext and Mask-RCNN. Comparing FLoRA to more prevalent methods like last-layer fine-tuning and prompt tuning would offer a more comprehensive evaluation of its effectiveness.
*   **Missing DoRA Results on LLaVA-1.5-7B:** The paper doesn't report the performance of DoRA on LLaVA-1.5-7B fine-tuning, despite mentioning a DoRA result of 67.6, which surpasses FLoRA's performance. Including comprehensive comparisons with state-of-the-art methods like DoRA is crucial for establishing FLoRA's superiority.
*   **Compatibility with Weight-Decomposed Formulations:** The paper does not address whether FLoRA is compatible with weight-decomposed formulations like those proposed by DoRA. Exploring potential integration with other advanced PEFT techniques could reveal further benefits and limitations of FLoRA.
*   **Lack of Theoretical Grounding for the Core Space:** The paper demonstrates the existence and effectiveness of a low-rank core space empirically but lacks a theoretical framework to explain these findings. Providing a theoretical foundation for the core space's properties would strengthen the paper's claims and provide valuable insights into FLoRA's underlying mechanisms.
*   **Potential Computational Overhead:** While the paper asserts that FLoRA exhibits better parameter efficiency than LoRA for larger kernel sizes, it doesn't thoroughly analyze the computational costs in diverse settings. A more comprehensive analysis of FLoRA's computational complexity in various scenarios, especially for extremely large models or complex tasks, is recommended.
*   **Focus Beyond Convolutional Kernels:** While FLoRA demonstrates its effectiveness on convolutional kernels, it should be assessed on other high-dimensional weight matrices.  Given its positioning as a fundamental low-rank adaptation method, a broader range of experiments would validate its general applicability and effectiveness.
*   **Quantifying the Impact of Structural Integrity:** The paper could benefit from a more in-depth exploration of whether preserving structural integrity truly matters in weight adaptation for computer vision. This could involve experiments or analyses that isolate the impact of structural preservation on performance, further validating FLoRA's core principle.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: See above
soundness: 2
strengths: 1)	The motivation behind this paper is significant. When using LoRA to fine-tune convolutional layers, it either disrupts the original topology or adds too many parameters. FLoRA, with Tucker decomposition, effectively adjusts high-dimensional parameter spaces, aiding the application of parameter-efficient fine-tuning across more models.
2)	Although FLoRA is motivated by improvements in tuning high-dimensional parameter spaces, the paper still includes many experiments with models that mainly use linear layers. The experimental setup of the paper covering many fields including computer vision, NLP and multimodal tasks, and shows good performance.
3)	The authors compare FLoRA with LoRA and DoRA, using both theoretical reasoning and empirical evidence. They employ metrics like Frobenius norm and feature amplification factor, offering insights into learning patterns. Notably, they explore the correlation between Frobenius norm and task-specific information amplification.
summary: This paper presents FLoRA, a novel parameter-efficient fine-tuning framework that addresses the challenge of adapting pre-trained models across diverse parameter space dimensions. Leveraging Tucker decomposition, FLoRA proposes to model parametric changes through a low-rank core space, ostensibly preserving the structural integrity of high-dimensional parameter spaces. The methodology is evaluated across a spectrum of tasks encompassing computer vision, natural language processing, and multimodal learning. The authors report that FLoRA demonstrates superior performance compared to existing methods, notably LoRA, while utilizing fewer trainable parameters. This efficiency is attributed to FLoRA's capacity to maintain the topological structure of the parameter space.
weaknesses: 1)	The performance gains of FLoRA on linear layer-based models appear limited. While the motivation to address topological distortions in high-dimensional parameter spaces is commendable, it potentially constrains FLoRA's applicability across diverse model architectures. Although the analysis utilizing Frobenius norm metrics and unrestricted low-rank subspaces is insightful, I recommend enhancing the paper through: A more rigorous theoretical treatment, exploring deeper mathematical foundations. An expanded experimental paradigm encompassing various model architectures, tasks, and datasets. These enhancements would strengthen the generalizability claims and broaden FLoRA's potential impact in the field of parameter-efficient fine-tuning.
2)	The initialization strategy in FLoRA deserves more exploration. Although the authors use a conservative approach, diverse methods could enhance performance. Advances in LoRA initialization indicate room for improvement. A study comparing various strategies might boost FLoRA's effectiveness and stability across tasks and models. I look forward to the authors expanding research on how initialization affects FLoRA's low-rank subspace, potentially leading to more robust fine-tuning.

Meta-review: The paper presents a compelling advancement in PEFT methods through FLoRA, which effectively addresses the challenges of adapting high-dimensional parameter spaces. It provides strong empirical evidence of significant performance gains over existing methods like LoRA and DoRA across diverse tasks, including computer vision, NLP, and multimodal applications, while maintaining parameter efficiency and practicality. During the rebuttal phase, the authors were able to sufficiently address the concerns from the reviewers. Overall, the paper has sufficient contributions and potential impact on the community.

Decision: Accept (Poster)