RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Introduction
Pre-trained language models such as BERT (Devlin et al., 2018 ) , GPT-3 (Brown et al., 2020 ) , and LLaMA2 (Touvron et al., 2023 ) have demonstrated commendable performance on various natural language processing (NLP) tasks. However, their zero-shot performance on many downstream tasks often falls short of expectations. One possible solution is full fine-tuning (FFT) of the model on the downstream dataset. However, the large model parameter size makes this process prohibitively
costly.

To address this challenge, various parameter-efficient fine-tuning (PEFT) methods including low rank adaptation (LoRA) (Hu et al., 2021 ) , adapter tuning (He et al., 2021 ) , and prompt tuning (Lester et al., 2021 ) are proposed. These methods add parameters to the trained model for fine-tuning,
bypassing the need to adjust the weights of the pre-trained model.
In particular, LoRA (Hu et al., 2021 ) and its variants (Zhang et al., 2023 ) add a
trainable low-rank path consisting of down-projection and up-projection matrices to the model, inspired by Aghajanyan et al. ( 2020 ) which showed that such low-rank paths can effectively approximate the trained weight tensors. ELoRA Kopiczko et al. ( 2024 ) extends LoRA by adding trainable feature transformation vectors to the output of each project matrix. They showed that SoTA accuracy can be achieved with the projection matrices frozen after random initialization while keeping the two feature transformation vectors trainable. This approach significantly reduces the number of trainable parameters. However, compared to LoRA, ELoRA incurs higher computation costs due to higher rank needed for the frozen projection
matrices. Fig. 1 illustrates LoRA and ELoRA, contrasting them to our proposed method AFLoRA.

Our contributions. To reduce the trainable parameter count and computation costs of fine-tuning, we present Adaptive Freezing of Low Rank Adaptation (AFLoRA). More specifically, we first investigate the rank needed for the frozen LoRA path in ELoRA and observe that reducing the rank of the frozen projection matrices (PM) causes a drop in fine-tuning performance.

Based on this insight, we present AFLoRA, that starts with a low-rank trainable path that includes projection matrices and feature transformation vectors and train the path for some epochs. We then gradually freeze the projection matrices based on a novel freezing score that acts as a proxy for the trainability requirement of a LoRA tensor. In this way, we not only help alleviate the over-fitting issue but also, improve the computation efficiency. To evaluate the benefit of AFLoRA, we perform extensive evaluations on multiple NLP benchmark datasets and compare accuracy, FLOPs, and training time with
several existing alternatives. In specific, compared to ELoRA we yield 1.86\times and 2.96\times improvement in runtime and FLOPs, respectively, while remain comparable as LoRA on these two metrics. Compared to LoRA we require 9.5\times fewer average trainable parameters to yield similar or improved performance.
Related Works
PEFT (Hu et al., 2021 ; Kundu et al., 2024 ) refers to a collection of methodologies that focus on allowing a small number of parameters to fine-tune to yield good performance on a downstream task. For example, prefix-tuning (Li and Liang, 2021 ) adds trainable prefix tokens to a model’s input or hidden layers while adapter-tuning (Houlsby et al., 2019 ) inserts small neural network layers, known as adapters, within each layer of a pre-trained model. LoRA (Hu et al., 2021 ) , on the other hand, adds low-rank tensors in parallel to the frozen pre-trained weights. AdaLoRA (Zhang et al., 2023 ) allows the rank of the LoRA path to be chosen in an adaptive way. Other variants like SoRA Ding et al. ( 2023 ) and LoSparse Li et al. ( 2023 ) have investigated the impact of sparsity in and alongside the low-rank path, respectively. Recently, efficient low-rank adaptation (ELoRA) (Kopiczko et al., 2024 ) has proposed to keep the LoRA path frozen, while introducing two trainable feature transformation vectors. Thus, this work only studies an extreme scenario of keeping the LoRA path frozen, and, to the best of our knowledge, no work has investigated
the trainability requirement of the projection matrices.
Motivational Case Study
To understand the high rank requirement for the frozen projection metrices in ELoRA, we conduct two sets of fine-tuning on SST-2 and MRPC, with ELoRA having rank ( r ) of 1024 and 4, respectively. As we can see in Fig. 2 , the model with r=4 , yields poorer performance, highlighting the need for high rank for the frozen tensors. This high rank causes ELoRA to potentially be FLOPs inefficient.
Experiments
Models & Datasets. We use the PEFT framework of Mangrulkar et al. ( 2022 ) and evaluate the fine-tuning performance of DeBERTaV3-base (He et al., 2020 ) to fine-tune on our framework on the General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2018 ) . The details of the hyperparameter settings for each dataset are listed in Appendix A.2 .

Performance Comparison. We benchmark the performance with AFLoRA and present comparison with LoRA and its variants. For ELoRA, we reproduce the results at our end while the results for other methods are sourced from Ding et al. ( 2023 ) . As shown in Table 1, AFLoRA can achieve SoTA performance on majority of datasets and on average while requiring similar and 9.5\times fewer average trainable parameters as compared to ELoRA and LoRA, respectively.

Runtime & FLOPs Comparison. Fig. 3 shows the comparison of the normalized average training runtime, normalized FLOPs, and normalized trainable parameters. For AFLoRA, we average the training time, FLOPs, and trainable parameters over six GLUE datasets (except the MNLI and QQP datasets). Note, for LoRA and ELoRA, the trainable parameters and FLOPs remain fixed for all the dataset. We compute their average runtime same way as ours. Compared to ELoRA we can yield up to 1.86\times and 2.96\times runtime and FLOPs improvement while remain comparable with LoRA in these two metrices. Compared to LoRA we yield 9.5\times parameter reduction, while remain comparable with ELoRA. These results clearly demonstrate AFLoRA
as PEFT method that can yield similar parameter efficiency as ELoRA while costing no training overhead in terms of FLOPs or time.

Results with Large Language Models (LLMs). We now demonstrate the AFLoRA fine-tuning performance with two popular LLM variants, namely, LLaMA-7B (Touvron et al., 2023) and BARTLarge (Lewis et al., 2019) on GSM8k complex reasoning and CNN/Daily mail summarizing task,respectively. As demonstrated in Table 2, on GSM8k, AFLoRA yields improved accuracy of 1.09% while requiring 3.15× fewer trainable parameters as compared to that with LoRA. On the CNN/DailyMail Summarizing task (Table 3), AFLoRA requires 1.69× fewer trainable parameters to reach similar or improved rouge score values.

* The original paper has results with the RoBERTa, we generated the results with our implementation on DeBERTaV3 with rank of 1024.

** As the number of trainable parameters is changed during training, we computed this by averaging over the whole training epochs over all datasets.
Ablations and Discussions
We conducted our ablation studies on six GLUE benchmark datasets, omitting QQP and MNLI, the two most computationally demanding datasets.

Do we really need adaptive freezing? We conducted experiments with all the LoRA PMs frozen (same as ELoRA), all the LoRA PMs trainable, and with our adaptive training of LoRA PMs. We use, r=4 for the LoRA path, for all. As we can see in Table 2, keeping the projection matrices trainable yield better average performance compared to keeping them frozen throughout. However, AFLoRA with adaptive freezing yields even better performance than keeping them trainable throughout, potentially highlighting its ability to regularize the fine-tuning against overfitting.

Do we need to keep the PMs trainable for all layer types? There are two major layer types, FFN and the attention layers. We place the PMs in both along with the feature transformation vectors. We then study the necessity of keeping the PMs trainable in these two layer types. Note, here, we keep the vectors trainable for all throughout. As shown in Table 3, keeping the PMs trainable (and then adaptive freezing) in the FFN yields better performance compared to the alternatives. Note we keep the PMs in the attention layers frozen to random values. Interestingly, allowing all PMs to initially train and then adaptively freeze yields poorer performance than allowing them only in MLP. This may hint at the FFN weights to play more important role in fine-tune performance.

Ablation with sensitivity choices. Fig. 4 presents ablation with three sensitivity scores based on three different sensitivity choices, namely, |grad(p)| (adopted in AFLoRA), |p*grad(p)| , and |grad(p)/p| . On average, the freezing score adopted in AFLoRA, consistently yield better accuracy over the other two.

Discussion on Freezing Trend. We use the RTE dataset as a case study, to understand the freezing trend of the PMs across different layers. In specific, we illustrate the specific number of iterations required before freezing each component in Fig. 5 . Interestingly, as can be seen from the figure, analysis reveals that the down-projection matrix parallel the intermediate linear layer require longer training duration prior to being frozen, as compared to the other PMs. This may potentially hint at the low approximation ability of the intermediate layer as compared to the second MLP in the FFN.
Conclusions
In this paper we presented AFLoRA, an adaptive freezing of LoRA adapters that allow near optimal trainability of the LoRA projection matrices and freezes them driven by a "freezing score" after certain fine-tuning steps. Compared to LoRA, AFLoRA can reduce the trainable parameters by up to 9.5\times while yielding 0.85\% average imporved performance as evaluated on GLUE benchmark.
Limitation
In the ablation study with various freezing score metrics, we discovered that alternative scoring methods outperform ours on certain datasets, suggesting possible room for research in refining the freezing scores. This can further improve performance with AFLoRA. Additionally, integration of AFLoRA in the adaptive rank evaluation framework can potentially open a new direction of PEFT that we consider as a future research.
Figure 1: Schematic comparison of LoRA (Hu et al., 2021 ) , ELoRA (Kopiczko et al., 2024 ) , and AFLoRA and their associated advantages and disadvantages in terms of various metrics. r_{L} and r_{V} , represent rank of the low rank path used in LoRA and ELoRA methods, respectively. FT and KU refers to fine-tuned weights and Kaiming uniform initialization function, respectively.; Schematic comparison of LoRA (Hu et al., 2021 ) , ELoRA (Kopiczko et al., 2024 ) , and AFLoRA and their associated advantages and disadvantages in terms of various metrics. r_{L} and r_{V} , represent rank of the low rank path used in LoRA and ELoRA methods, respectively. FT and KU refers to fine-tuned weights and Kaiming uniform initialization function, respectively.
Figure 2: Performance of ELoRA with two different ranks of the frozen projection matrices.; Performance of ELoRA with two different ranks of the frozen projection matrices.
Figure 3: A comparison of various system performance between LoRA, ELoRA, and AFLoRA.; A comparison of various system performance between LoRA, ELoRA, and AFLoRA.
Figure 4: A comparison of performance outcomes utilizing three distinct freezing score methodologies.; A comparison of performance outcomes utilizing three distinct freezing score methodologies.
Figure 5: Visualization of freezing iterations for each layer. ‘out’ and ‘inter’ refer to the second and the first MLP layer of the FFN, respectively. ‘A’ and ‘B’ represent the down-projection and up-projection matrix, respectively. The darker the color, the more iterations the matrix has to go through before freezing.; Visualization of freezing iterations for each layer. ‘out’ and ‘inter’ refer to the second and the first MLP layer of the FFN, respectively. ‘A’ and ‘B’ represent the down-projection and up-projection matrix, respectively. The darker the color, the more iterations the matrix has to go through before freezing.
Comparison of different LoRA variants with DeBERTaV3 on the GLUE benchmark.
Ablation study on the trainability impact of the projection matrices (PM) of the AFLoRA module. We keep the vectors trainable throughout for all.
Ablation study on making the PMs for different layer-types trainable.
Statistics of the GLUE benchmark datasets."Mcc", "Acc", "F1" and "Pear" represent Matthews correlation coefficient, accuracy, the
F1 score and the Pearson correlation coefficient respectively. And "Acc" for MNLI dataset contains the accuracy for the matched and mismatched subset of the datasets.
Hyperparameter setup for all eight datasets in GLUE benchmark
Ablation study on whether freezing the two projection matrices in the same layer simultaneously or independently.