NOLA: Compressing LoRA using Linear Combination of Random Basis
Authors: Soroush Abbasi Koohpayegani
Navaneet K L
Parsa Nooralinejad
Soheil Kolouri
Hamed Pirsiavash
Venue: ICLR 2024 poster (2024)
Keywords: computer vision
natural language processing
natural language generation
Generative AI
GPT
large language models
LoRA
adaptation
PEFT
Transfer Learning
Transformers
ViT
LLaMA
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We introduce NOLA, a novel parameter efficient fine-tuning method for language and vision models with the number of trainable parameters decoupled from the shape of original weight matrices.

Abstract: Fine-tuning Large Language Models (LLMs) and storing them for each downstream task or domain is impractical because of the massive model size (e.g., 350GB in GPT-3).
Current literature, such as LoRA, showcases the potential of low-rank modifications to the original weights of an LLM, enabling efficient adaptation and storage for task-specific models. These methods can reduce the number of parameters needed to fine-tune an LLM by several orders of magnitude. Yet, these methods face two primary limitations: (1) the parameter count is lower-bounded by the rank one decomposition, and (2) the extent of reduction is heavily influenced by both the model architecture and the chosen rank. We introduce NOLA, which overcomes the rank one lower bound present in LoRA. It achieves this by re-parameterizing the low-rank matrices in LoRA using linear combinations of randomly generated matrices (basis) and optimizing the linear mixture coefficients only. This approach allows us to decouple the number of trainable parameters from both the choice of rank and the network architecture. We present adaptation results using GPT-2, LLaMA-2, and ViT in natural language and computer vision tasks. NOLA performs as well as LoRA models with much fewer number of parameters compared to LoRA with rank one, the best compression LoRA can archive. Particularly, on LLaMA-2 70B, our method is almost 20 times more compact than the most compressed LoRA without degradation in accuracy. Our code is available here: https://github.com/UCDvision/NOLA

Published as a conference paper at ICLR 2024
NOLA: COMPRESSING LORA USING LINEAR COMBINATION OF RANDOM BASIS
Soroush Abbasi Koohpayegani 1.1 Parsa Nooralinejad!
K. L Navaneet
Soheil Kolouri
Hamed Pirsiavash
University of California, Davis
Vanderbilt University
ABSTRACT
Fine-tuning Large Language Models (LLMs) and storing them for each downstream task domain is impractical because of the massive model size (e.g., 350GB in GPT-3), Current literature, such as LoRA, showcases the potential of low-rank modifications to the original weights of an LLM, enabling efficient adaptation and storage for task-specific models. These methods can reduce the number of parameters needed to fine-tune an LLM by several orders of magnitude. Yet, these methods face two primary limitations: (1) the parameter count is lower-bounded by the rank one decomposition, and (2) the extent of reduction is heavily influ- enced by both the model architecture and the chosen rank. We introduce NOLA, which overcomes the rank one lower bound present in LoRA. achieves this by re- parameterizing the low-rank matrices in LoRA using linear combinations of randomly generated matrices (basis) and optimizing the linear mixture coefficients only. This approach allows us decouple the number of trainable parameters from both the choice of rank and the network architecture. We present adaptation results using GPT-2, LLaMA-2, and VIT in natural language and computer vision tasks. NOLA performs as well as LoRA models with much fewer number of parameters compared LoRA with rank one, the best compression LoRA can archive. Partic- ularly, on I.aMM-2 70B, our method is almost 20 times more compact than the most compressed LoRA without degradation in accuracy. Our code is available here: http://shbbbbbom///////i//o/////
INTRODUCTION
Large pre-trained neural networks have exhibited remarkable generalization abilities across a diverse range of downstream tasks in both natural language processing and computer vision, achieving unprecedented data efficiency. For instance, large language models have demonstrated the capability for few-shot generalization (Brown et al., 2020) across a variety of tasks, including translation, question-answeringg cloze tasks, and reasoning. Similarly, in DINOv2, (Oquab et al., 2023) showcase how large pre-trained VIT model (Dosovitskiy et al., 2020) with more than 1B parameters yields superior all-purpose visual features for a variety of downstream benchmark tasks at both image and pixel levels. Typically, these pre-trained large models are adapted to downstream tasks through fine-tuning of their parameters. However, fine-tuning and storing the entire set of model parameters for each task incurs a significant storage cost (e.g., 350GB for GPT-3). This challenge has spurred a considerable body of recent works focusing parameter-efficient fine-tuning of large models (Hu et al., 2021; Xu et al., 2023; Dettmers et al., 2023; Chen et al., 2022; Sung et al., 2022b).
Inspired by the low intrinsic dimensionality of prrmtttramttrri networks' optimal parameters (Li et al., 2018; Aghajanyan et al., 2021), (Hu et al., 2021) proposed a seminal hypothesis that the change in weights during model adaptaaion/finetuiing has a low "intrinsic rank", leading to the development of Raw- Adaptation (LoRA). In essence, LoRA enables the indirect training of linear layer in neural network by optimizing the -enk-cccomooittion matrices for the weight change in these layers, resulting in significant reduction in the number of parameters required for adaptation
* Equal Contribution.

---

Published as a conference paper at ICLR 2024
Seed
Pseudo-random Generator
A
B1
B1
Pre- Pre-trained weight (Frozen)
...
X
B
W E m
Rank-decomposition matrices (LoRA)
A R B xR*
Input
Output xT W)
Random Basis (Frozen)
x
X
B
(A; E Bu 11 {B) E R RR*
f
Trainable parameters {ag E R3
Ji=
(B, E R]
Figure 1: Our Method (NOLA): After constraining the rank of AW by decomposing it to A B. we reparametrize A and B to be a linear combination of several random basis matrices. We freeze the basis and W and learn the combination coefficients. To reconstruct the model, we store the coefficients and the seed of the random generator which is a single scalar. NOLA results more compression compared to LoRA and more importantly decouples the compression ratio from the rank and dimensions of W. One can reduce the number of parameters to 4 times smaller than rank=1 of LoRA which not possible with LoRA due rank being an integer number.
(e.g., x0,000 parameter reduction for GPT-3). Notably, LoRA has gained popularity, and various extensions of this method have been proposed since its inception (Xu al., 2023; Dettmers et al., 2023). However, LoRA and its derivatives have three inherent limitations: (1) the parameter count is lower-bounded by the rank quantized since rank is an integer number, and (3) the number of parameters inherently depends on the model's iss architecture, i.e. the dimensions of the linear matrix, and the choice of rank. In this paper, we introduce a method, denoted as NOLA, that offers the same benefits as LoRA while addressing its limitations. NOLA allows one decouple the number of trainable parameters from both the choice of rank and the network architecture, and it breaks the rank-one one decomposition limit of LoRA.
decomposition of linear layers, and (2) the number of parameters is
NOLA is inspired by the recent work by Nooralinejad et al. (2022), titled PRANC. In this work, we reparameterize neural network using linear combination of pseudo-randomly generated weights. Then, we indirectly train the network parameters by optimizing the linear mixture coefficients. This approach results in significant reduction in the total number of parameters needed to represent the network. Unlike PRANC, our focus in NOLA is on reparameterizing the change of neural weights for fine-tuning large models. More eritically, unlike PRANC, NOLA incorporates the invaluable insight from (Hu et al., 2021), which posits that the weight change during fine-tuning is intrinsically low-rank. In essence, we utilize the rank-decomposition presented in LoRA but assemble the cank-decomposition matrices as linear combination of pseudo-random matrices (i.e., the 'basis"). Optimizing the rank-decomposition matrices in NOLA akin determining the linear mixture coefficients for the random matrices. This design allows us to decouple the number of parameters from the shape of the linear layer and also from the rank choice. Furthermore, the low-rank constraints offer substantial advantages in compute and memory footprint over the methodology proposed in PRANC. Figure/ illustrates the fundamental concept NOLA.
Why Fewer Parameters Matter?
We envision future where we must efficiently manage and transition between multiple Large Language Models (LLMMs), each tailored for specific tasks. This vision arises from the necessity for I.LMs customized with private data and/or the concept of crafting universal L.LM that can summon customized LLMs versatile toolbox to tackle diverse tasks (Schick al., 2023). However, currently, customized LLMs demand substantial storage, and the process of switching between them lacks efficiency due to large I/O operations. NOLA offers that can be stored effectively in GPU memory, allowing for on-demand reconstruction directly on the GPU itself when
compact reparameterization solution
task arises.
Note that while storing parameters in CPU memory is cost-effective option, the process of trans- ferring them from CPU to GPU incurs substantial time and power consumption. Moreover, this
2

---

Published as a conference paper at ICLR 2024
data transfer relies on a shared resource (e.g., PCIe bus), which may experience congestion in busy server environments. Therefore, optimizing model compactness to fit several of them within the limited GPU memory proves advantageous in these scenarios. As an example, 1,000 customized GPT-3 models using LoRA need almost 35GB of memory (assuming LoRA compresses it by a factor of 10, 000x), which may not fit in the GPU memory along with the LLM model itself. Hence, compacting it by an additional factor of reduces it to 7GB, which can fit in the GPU memory, leading to very efficient switching between the tasks.
Contributions. Our specific contributions in this paper are: 1) A novel reparameterization for compressing task-specific large language models, denoted as NOLA. 2) NOLA decouples the compression ratio from the rank and dimension of the weight matrix, unlocking higher compression ratios while keeping most benefits of LoRA, including reduced memory and computation training time. 3) NOLA can be further improved by quantizing the coefficients and can be applied to other architectures like CNNs. 4) Applied to PRANC, NOLA speeds it up and reduces its memory footprint.
2 PROPOSED METHOD: NOLA
LoRA, short for Low-Rank Adaptation, is a widely embraced method for customizing a pre-trained model, such GPT, for specific task. Instead of changing all parameters denoted W within given layer, LoRA maintains the original pre-trained parameters W as a constant and learns a residual adjustment AW to fine-tune the model for the new task. The resulting updated layer parameters are then computed as W + AW. The core concept behind LoRA is to minimize the size of AW by constraining its rank. In more formal context, considering W ?=*= and AW R LoRA accomplishes this by reparameterizing AW as the product of two matrices, AW = -Ax B, where R m and B Rex*, with r representing the rank- a hyperparameter. By selecting a relatively small rank (r << min(m, n.)), LoRA efficiently reduces memory usage. This optimization achieved by storing A and B, which exhibit significantly more compact representation than the full AW. The resulting compression ratio is quantified as F(m+nj
Unfortunately, this compression ratio is: (1) tied the shape of the parameters m and n, and hence the model architecture and (2) is upper-bounded by mn m+n i.c., for r = 1.
In this paper, we introduce a novel reparameterization technique for AW that effectively decouples ma
the rank from the compression ratio, allowing for a compression ratio higher than m which corresponds to 1 in the LoRA framework. To achieve this, draw inspiration from PRANC (Nooralinejad et al., 2022) and reparameterize matrices A and to exist within a lower-dimensional space defined by set of randomly generated basis matrices. Formally, we express this as:
A Σ a,
B=
Σ BB j=1
where, A, E Rmxr and B, E RTx" are random matrices generated by a Pseudo Random Number Generator with fixed seed. We subsequently learn A and B linear combinations of these predefined and frozen random matrices. Importantly, the random matrices themselves remain constant, and we optimize only the coefficient vectors and 3. Then:
AW = (,A,)
Σ 3,B,)
3=1
In practical terms, to store AW for a specific task, we only need to retain the seed (a single scalar) and the coefficient vectors and B. Remarkably, this approach allows for a small number of basis matrices (I- + 1) to be chosen, irrespective of the rank of the A x factorization and the shape of AW, thereby enhancing the compression ratio to go beyond 122n.
m
Quantization of coeflicients and 3: We are mainly interested in reducing the storage for a new task, assuming the pre-trained L.M already available. Hence, to further reduce the storage, we quantize the a and 3 coefficients to lower precision (e.g.. 4 bits) while the random basis and the pre-trained LLM weights have standard FP16 floating point precision. Note that one can also quantize A and B matrices in LoRA; however, our method does not force A and B themselves to be of low precision. One can quantize a and 3 after the optimization (post-training quantization) or while optimizing them (quantization-awwa aware training). We expect the latter to perform better. For quantization- aware learning, we use the method in (Rastegari et al., 2016b; Jacob et al., 2018) where use the quantized a and /3 in the forward pass and update the FP16 versions of and B in the
3

---

Published as a conference paper at ICLR 2024
backward pass. Moreover, we use the Srraight-Througe Estimator (STE) trick (Bengio et al., 2013) to estimate the gradient.
A few recent works have shown that it is possible to quantize the weights of LLMs for each task, which reduces both computation and storage. However, these methods are not suitable for a large number of tasks since the quantized LLM is still task-specific and large.
Memory Efficiency:
Note that depending on the number of basis matrices, the random basis may be large, requiring a large memory. Interestingly, generating random matrices in the GPU itself is very fast, so similar to PRANC, at each iteration, we generate chunks of the basis matrices at a time, multiply them by the corresponding coeficents, and discard them. Generating a basis on the fly the inference time can drastically reduce the communication cost between CPU and GPU since a and ß vectors for several tasks can be stored in the GPU memory.
Efficiency of NOLA compared to PRANC: PRANC (Nooralinejad et al., 2022) reshapes the whole model parameters or each layer of it into a long vector and reparameterizes that by linear combination of random vectors. However, as mentioned in (Nooralinejad et al., 2022), this method involves multiplication of the coefficients with the big random matrix twice at each iteration (once in forward and once in backward passes), which is very expensive. For instance, the size of the random matrix for ResNet18 with 1000 coefficients will be almost M x 1 K. NOLA reduces this computation while keeping the same number of parameters by reshaping the long vector to be 2D matrix and constraining its rank. Assuming d2 weights and k random basis, the basis matrix size for PRANC will be kd? while NOLA with rank reduces that to kdr assuming that each component in A x B has & basis matrices to keep the number of parameters equal to PRANC. Then, the total 2 compute for PRANC will be kdd2 + d2 z kdd while for NOLA, it will be kdr + 2dr R kdr. Hence, assuming small rank r, NOLA can reduce the training time of PRANC by large factor due to the reduction of the computation at forward and backward passes. Moreover, in the appendix, we empirically show that NOLA offers a better coverage of the weight space compared to PRANC.
Structure of the parameters: Note that one can apply NOLA to model architectures other than transformer by simply reshaping the weight tensor be a 2D matrix (preferably close to square) and then compressing it. We do this in our ResNet experiments in the Appendix, where the weight matrices are 4D tensors of convolutional filters.
EXPERIMENTS
Here, we evaluate NOLA in transfer learning tasks in both NLP and vision. Moreover, in the appendix, we evaluate NOLA in training from scratch.
3.1 NOLA ON GPT-2:
We adapt the parameters of pre-trained GPT-2 to three different Natural Language Generation (NLG) datasets by finetuning the parameters using NOLA. We use GPT-2-Large and GPT-2-Medium in our experiments. We follow the (Li & Liang, 2021; Hu et al., 2021) for our adaptation setup.
Datasets: We utilize the following datasets for our Natural Language Generation (NLG) task: E2E NLG Challenge (Novikova et al., 2017) serves as a commonly used benchmark for evaluating NLG models. It encompasses of 51,200 samples, distributed as follows: 42,200 for training, 4,600 for validation, and an additional 4,600 for testing. DART (Nan et al., 2020) is yet another significant dataset employed for evaluating text-to-data generation. This dataset is rich with 82, 191 examples drawn from various domains. WebNLG (Gardent et al., 2017) is a text-to-data dataset, boasting 22,000 examples spanning 14 distinct categories. Notably, the WebNLG test set introduces five new categories. prompting us to present results across all categories within this dataset. These datasets collectively provide comprchensive foundation for our NLG evaluation and experimentation.
LoRA: In our experiments, we apply LoRA on both query and value projection layer in each attention block. Since number of parameters is tied to the rank, we adjust the rank to reduce number of parameters. We compare to LoRA with both rank four and rank one.
Moreover, we compared NOLA to few other baselines, including finetuning all parameters, Adapters (Houlsby et al., 2019; Lin et al., 2020b; Pfeiffer et al., 2021; Rücklé et al., 2020), and Prefix-layer tuning (PreLayer) (L.i & Liang, 2021).
Other Baselines:
4

---

Published as a conference paper at ICLR 2024
Table E2E NLG Challenge: We compare NOLA LoRA with different architectures: GPT-2 medium (M) and large (L). QV refers to Query and Value projection, while MLP denotes the MLP layer within transformers. Adapter* and Adapter" are two Adapter baselines reported in (Hu et al., bold. To reduce number of parameters in
2021). The best and second best performing methods LoRA, we use lower rank (LoRA rank=l), We don't t see drop in performance by reducing number of trainable parameters to 20 number of parameters below rank one.
of LoRA with rank in GPT-L. Note that in LoRA, one cannot reduce the
GPT-2 M
Method
Adapted Layers
Adapter Rank
# Trainable Parameters
EZE NLG Challenge
BLEU
NIST
MET
ROUGE-L
CIDEr
Finetune
All Layers
354.920M
68.2
8.62
46.2
71.0
2.47
Adapter
Extra Layers
0.370M
66.3
8.41
45.0
69.8
2.40
Adapter
Extra Layers
11.090M
68.9
46.1
71.3
2.47
Adapter"
Extra Layers
11.090M
67.3
8.50
70.7
2.44
Finetune Top2
Last 2 Layers
25. 25.190M
68.1
8.59
46.0
70.8
2.41
PreLayer
Extra Tokens
0.350M
69.7
8.81
46.1
71.4
2.49
LoRA
QV
4
0.350M
70.4
8.85
46.8
71.8
2.53
LoRA
QV
0.098M
68.7
8.72
45.6
70.5
2.43
NOLA (Ours)
QV
0.350M
70.1
8.80
71.7
2.53
NOLA (Ours)
QV
0.096M
70.0
NOLA (Ours)
MLP
0.096M
70.2
8.79
46.7
71.8
2.51
NOLA (Ours)
QV
0.048M
70.1
8.82
46.4
71.4
2.52
NOLA (Ours)
MLP
0.048M
69.4
8.71
46.5
71.5
2.47
GPT-2 L
Finetune
All Layers
774.030M
68.5
8.78
46.0
69.9
2.45
Adapter'
Extra Layers
0.880M
69.1
2.49
Adapter*
Extra Layers
23.000M
68.9
8.70
46.1
71.3
2.45
PreLayer
Extra Tokens
0.770M
70.3
8.85
46.2
71.7
2.47
LoRA
QV
0.770M
70.4
72.0
2.47
LoRA
QV
0.184M
69.9
2,53
NOLA (Ours)
QV
8
0.144M
70.5
8.85
46.8
71.7
2.54
NOLA (Ours)
MLP
8
0. 0.144M
70.1
8.80
46.5
71.2
2.52
NOLA (Ours)
QV
8
0.072M
69.8
8.80
46.4
71.3
2.51
NOLA (Ours)
MLP
8
0.072M
69.4
8.71
46.6
71.5
2.48
NOLA (Ours)
QV
8
0.036M
70.1
8.80
46.7
71.7
2.53
NOLA (Ours)
MLP
8
0.036M
70.0
8.81
46.4
71.5
2.53
We evaluate NOLA with two different variations: 1. Adapting MLP layers. 2. Adapting query and value projection matrices. Note that, unlike LoRA, we can use any number of parameters while applying NOLA to any weight structure since the number of parameters is not tied the shape of the weight tensor. We allocate an equal number of parameters to A and in each NOLA layer (i.e., k = 1). Using k- 1 1000 results in 0.096M parameters in GPTTM and 0. 0.144M parameters in GPT-L. Also, we use half (k = 500) and quarter (k = = 250) number of parameters per layer to get smaller checkpoints.
NOLA:
Implementation Details: We trained our models using a single NVIDIA RTX 6000 Ada Generation GPU. For all hyperparameters except learning rate, we the evaluation of GPT-2. We train our models for 5 epochs with a learning rate of 0.1 and no weight decay. We use batch size of We use rank of 8 for NOLA in our experiments. Like LoRA, scale A x B with f, where is hyperparameter and is the rank. We use the default value of 1.
values LoRA for training and
We compare LoRA and other baselines in Table and Table 11 in the Appendix. NOLA is on or better compared to other methods with the same number of parameters. In the E2E task, NOLA with 0.036M parameters archives BLEU score of 70.12, which is 20 times more compact compared to LoRA A with rank that has 0.77M parameters and archives BLEU score of 70.4. This NOLA model uses a rank of 8, which does not affect the number of parameters and increases the run time slightly (negligible compared to that of the actual LLM model). Note that goal is to reduce the number of parameters without reducing the accuracy, which shown Tables and 1I. We are not claiming that NOLA improves the accuracy compared to baselines. We show various variants of NOLA (MLP, QV, etc) to emphasize that NOLA is not very sensitive to the choice the variation.
Results:
Training Time and Memory of NOLA: Similar LoRA, in the inference time,
calculate A x B offline and merge with W. Therefore, NOLA does not have any overhead compared to the original model. In training time, NOLA A has small overhead due the multiplication of coefficients
5

---

Published as a conference paper at ICLR 2024
Table 2: Training time and memory: We compare the training memory and running time of NOLA to LoRA. Since generating a random basis on each layer has a small overhead, we can generate random basis once and share the basis across layers save time. This version of NOLA has a similar runtime to LoRA and on-par accuracy to NOLA with non-shared basis.
Model & Method
Random Basis
Training Memory
Training Time (ms/batch)
# Trainable Parameters
E2E NLG Challenge
BLEU
NIST
MET
ROUGE-L
CIDEr
GPT-2 L (LoRA ml)
33.35GB
776
184K
69.89
8.81
46.70
71,64
2.53
GPT-2 L (NOLA QV) GPT-2 L (NOLA QV)
Non-shared
33.37GB
834
144K
70.46
8.85
46.77
71.68
2.54
Shared
33,40GB
779
144K
70.32
8.85
46.74
71.71
2.54
Table 3: Effect of rank in NOLA:
We vary the rank from 1 to 8. Note that we can use the same number of parameters in all ranks since the number of parameters is not tied to the rank in NOLA.
Table 4:
Quantization of coefficients: Post-
training quantization of parameters does not de- grade the performance up to the 4 bit quantization. In quantization- -warr training, NOLA is more ro- bust to quantization compared LoRA.
Train Params
Rank
EZE NLG Challenge MET
BLEU
NIST
ROUGE
CIDEr
Model & Method
Quant Bits
EZH Challenge MET
BLEU
NIST
ROUGEEL
CIDEr
GPT-2 (NOLA QV)
Post Training
Quamtization
96K
70.03
8.82
46.74
71.64
2.51
96K
96K
70.47
69.69
8.86
8.76
46.56
46.71
71.79
71.44
2,51
2.53
(LoRA ml)
GPT-2
16-bit
4-bic
8-bic
69.63
69.91
69.89
8.81
8.75
8.81
46.69
16.32
16.70
71.24
71.75
71.61
2.48
2.53
2.53
69.09
70.15
2.45
3-bic
67.23
96K
70.03
8.82
46.74
71.64
2.51
16-bit
70:46
8.85
46.77
71.68
2.54
48K
70.09
8.82
46.44
71.36
2,52
GPT-2 L
8-bic
70.43
8.84
46.78
71.72
2.54
24K
68.30
8.67
45.13
68.91
2.40
(NOLA QV)
4-hit
71.82
2.52
12K
67.18
8.60
43.99
68.38
2.26
3-bit
65.1 14
8.58
44.38
67.56
2.23
GPT-2
(NOLA
Quantization Aware Truining
144K
8
70.46
8.85
46.77
71.68
2.54
GPT:2 L (LoRA r=1)
3-bit
08
8.86
44:67
68, 76
2.36
144K
2-bit
56.13
4.70
35.38
63.68
1.40
144K
69.
8.78
46.55
71.25
2.51
GPE21 (NOLA QV)
3-bic
70.14
8.82
46.58
71.61
2.53
2.51
2-bit
68.69
8.72
46.06
70.61
2.48
to basis weights. We measure the rumning time and memory footprint of NOLA during training and compare it to LoRA Table 2. Since generating a random basis for each layer adds small overhead, we can share the random basis across all layers and generate and store them only once to improve the running time. We measure time and memory with a batch size of 8. NOLA, with a unique random basis for each layer, is slightly slower than LoRA. However, NOLA with shared random basis has on-par accuracy with the unique random basis and has a similar running time to LoRA.
Ablation Study on the rank of NOLA: Since the number of parameters is decoupled from the rank of the matrix, we can solely evaluate the effect of rank without changing the number of parameters. We report the effect of rank in Table 3. We vary the rank from 1 to 8 and use = 1.0 for ranks 4 and 8, and e = 0.5 for lower ranks. As also noted by (Hu et al., 2021), understanding the effect of rank needs more rigorous study future work.
3.2 NOLA WITH QUANTIZED COEFFICIENTS, & AND 3:
We evaluate the performance of NOLA with rank 4 with quantized a and 3 parameters on the E2E dataset in Table in different setups. First, we do Post Training Quantization (PTQ) by quantizing the parameters to q bits after the training. We observe no significant drop in both LoRA and NOLA in 4 bits PTQ experiments. Second, we evaluate models with Quantization Aware Training (QAT) where we quantize the coefficients in the forward pass and update them in the non-quantized form. In QAT with 3 bits, NOLA has a slight drop of 0.3 points while LoRA has a drop of 2.8 points in the BLEU metric. Note that in NOLA, although a and 3 are quantized, A and B in Eq are not quantized since the basis matrices, A, and By, are qun-tuuntieed. This in contrast to LoRA where A and B are quantized.
3.3 NOLA ON LLAMA-2:
Finetuning with LoRA for larger LLMs is challenging due to compute demand, so we need to resort to QLoRA where the base model is quantized. Our NOLA framework is agnostic to the quantization of base model, so for LLaMA-2 (Touvron et al., 2023) experiments, we use NOLA with base model quantized to 8-bits while the NOLA coefficients still use 16-bit. We fine-tune the pretrained [LLaM--2

---

Published as a conference paper at ICLR 2024
Table 5: Instruction finetuning for quantized LLaMA-2: NOLA fine-tunes LLaMA-2 70B (8-bit) with 0.57M parameters, a 95% reduction compared to rank-one LoRA. We use quantized version of LLaMA-2 for both LoRA and NOLA, so our LoRA baseline equivalent to QLoRA.
LLaMA-2- 7B
LLaMX-2 13B
LLaMA-2
70B
wio Finetuning
LoRA
NOLA
w/o Fineruning
LoRA
NOLA
Fineraning
LoRA
NOLA
Adlapter Rank
16
16
Trainable Parameiers
2:500
(197%)
391M
0.14M
(1963)
12.94M
0.57M (195%)
Train Loss
1.53
0.97
1.05
1.43
0.94
0.95
1.42
0.87
0.90
Val Loss
1.74
1.04
1.01
1.59
0.96
0.97
1.53
0.92
0.90
MMLU Ace
45.3
46.5
46.5
54.8
55.3
55,3
68.9
69.5
69.4
model using 8-bit QLoRA on the Alpaca dataset (Taori et al., 2023), reporting both training and validation loss metries specific to the Alpaca dataset. Additionally, we employ the MMLU (Massively Multitask Language Understanding) benchmark (Hendrycks et al., 2021) to assess performance across a diverse set of language understanding tasks. This benchmark comprises 57 tasks spanning areas such as mathematics, history, computer science, and law. On MMI.U, we focus on 5-shot evaluation (providing examples in the context), following the standard practice.
Implementation Details: We apply LoRA and NOLA across all layers (Query, Key, Value and Output projections and MLP layers) of the transformer on three different model sizes of LLaMA-2: 7B, 13B, and 70B. For our LoRA experiments, we set the rank r = 1 since LoRA paper (its Table 15) shows that rank one model performs as well as higher ranks for larger LLMS (e.g., GPT-3). For NOLA, we use r 16 and adjust the number of optimized parameters for each LLaMA-2 model size using the following settings: & = 1 = 128 for LLaMA-2 7B, & = = 256 for LLaMA-2 13B, and & I = 512 for LLaMA-2 70B. During fine-tuning LoRA NOLA,
adhere to the
hyperparameters reported in QLoRA, (Dettmers et al., 2023). We optimize for one epoch on the Alpaca dataset with a batch size of 256 using four RTX 3090 GPUs. The learning rate is 0.0002 for LoRA and 0.001 for NOLA. Similar to LoRA, we scale A x B with f. where c is a hyperparameter and is the rank. We use 16 for LoRA and c for NOLA.
Results are presented in Table 5. Remarkably, NOLA is capable of fine-tuning LLaMA-2 70B with fewer than 0.6.M parameters, representing an average reduction of parameters by 95% compared to LoRA with rank one.
Results:
3.4 NOLA ON VISION TRANSFORMERS
We perform experiments on the image classification task on ViT-B and ViT-L architecture with both supervised and self-supervised (MAE) initializations.
Implementation details: All pre-trained networks are obtained from Timm library ( (Whitman). All approaches are trained for 50 epochs, and the top-1 accuracy at the final epoch is reported. We use a batch-size of 64 and tune the initial learning rate for each dataset and architecture for all approaches. Since our focus is on finetuning on small datasets, we use 5 and 10 labeled examples per class for finetuning. Since there is high variance in performance due to the small training sets, sample four different sets of training samples per k-shot and three different initializations for LoRA/NOLA and report the mean accuracy and standard deviation. All approaches are trained with cross-entropy loss. Additional details are in the appendix.
Datasets: ImageNet-21k and ImageNet- Ik are used to pretrain the backbone models. We use CIFARIO (Krizhevsky et al., 2014), CIFAR 100 (Krizhevsky et al., 2009), CUB-200-2011 (Welinder al., 2010), Caltech- M0ttchh-11100 (Fei-Fei et al., 2004), Aireraft (Maji et al., 2013), Food101 (Bossard et al., 2014), Pets (Parkhi et al., 2012) and SUN397 (Xiao et al., 2010) datasets for finetuning.
Baselines: We compare NOLA with three baseline approaches: Linear, Full-FT (full fine-tuning) and LoRA (Hu et al., 2021). In Linear, only the final classifier head is optimized, while in Full-FT. the entire backbone network is optimized too. No additional parameters are used during finetuning for either of these approaches. We apply LoRA on Query, Key, and Value projection matrices with rank set 4 for ViT-B to 1 or 4 for VIT-L. In our preliminary experiments. LoRA with rank one sees a big drop in accuracy. This is aligned with our GPT-2 M experiments where smaller models require higher rank. We apply NOLA on the MLP layers and use rank of 4 for ViT-B and for ViT-L. We report the number of trainable parameters for each approach excluding the classifier head parameter count which is common to all approaches. We also report nearest- neiihhbr (1-NN) evaluation for zero-shot classification downstream tasks using ImageNet pretrained features.
7

---

Published as a conference paper at ICLR 2024
Table 6: Results on vision transformers. We finetune ImageNet pre-trained ViT models on multiple small datasets with 5 and 10 training samples. The mean accuracy and standard deviation across 12 runs are reported. The number of train parameters for Linear classifier depends on the number of classes in the dataset (0.01, 0.1, 0.2, 0.1M parameters for CIFAR- 10, CIFAR- CUB and Caltech respectively). We do not count the linear layer in trainable parameters. The best performing method is in bold while all methods within one point of the best are underlined. NOLA outperforms LoRA with comparable parameters across datasets and architectures, particularly in the low training data regime. The performance of NOLA with half or ahe-third of the training parameters is comparable to that of LoRA. Note that LoRA with rank one is the most compact LoRA.
Base Model
TTrain Params
CIFAR-IO
CIFAR-100
CUB. (UBB-200-011
Caltech- 1D1
5
10
10
10
10
Nearest Neighbor
79.6
80.8
52.4
59.2
71.9
78.0
84.1
87.5
VIT-B
Full-FT
Linear
5.3
0
80.8 (1.1)
73.9(6.55)
85.1
87.6 (2.7)
(0.9)
64.5 (0.7)
(1.1)
59.7
72.7 (0.4)
76.6
79.2 (0.2)
87.9 10.8)
85.8 (0.8)
91.1 (0:5)
88.5 (0.4)
LoRA (r=4)
141K
$7.3 (2.3)
93.1 (0.5)
76.3 (0.5)
81.6 (0.9)
75.7 (0.5)
82.4 (0.3)
88.4 (1.1)
90.8 (0.5)
NOLA-MLP
47K
(1.3)
92.2 (0.5)
75.1 (0.6)
81.3 (0.8)
75.5 (0.6)
81.7 (0:4)
(1.2
90.6 (0.5)
Nsarest, Neighbor
18.2
19.8
9.8
13.2
25.3
28.2
40.7
VIT--MAE
Linear
0
(1.9)
(1.4)
15.7 (0.7)
(0.2)
(0.3)
(0.3)
66.9 (1.1)
76.9 (0.6)
Full-FT
5.3M
(4.4)
(3.6
23.0 (11.)(
$1.9
76.4 (2.3)
86.5 (0.5)
LaRA (r=4)
141K
54.7 (1.6)
70.1 (2:2)
39.3 (3.1)
(1.3)
35.2 (1.5)
$4.9 (0.6)
82.4 (0.6)
87.7 (0.5)
NOLA-MIPP
47K
35.1 (2.61
72.1 (2.7)
42.1 (1.4)
53.5 (1.D)
35.8 (1.5)
53.9 (0.6)
88.0 (1.2)
90.6 (0:5)
Nearest Neighbor
88.7
89.9
68.9
74.0
77.4
$2.3
88.4
90.1
Linear
0
(1.8
$8.4(1..)
63.7 (1.3)
70.6 (0.9)
73.7 (0.6)
(0.3)
87.6 (0.9)
89.9 (0.4)
WTLL
Fall-FT
289M
(2.7
90.2 (2.8)
74.0
86.2 (0.6)
73.3
83.9
8
(0.7)
LaRA (r=4)
375K
86.5 (2.0)
93.8 (1.0)
82.9 (0.9)
$7.6 (0.6)
81.2 (0.4)
85.3 (0.3)
89.3 (0.7)
91.3 (0.3)
LaRA (r-1)
94K
(1.3)
92.8 (0.8)
82.2 (0.8)
85.6 (0.9)
80.6 (0.3)
85,2 (0.3)
89.9 (1.0)
91 (0:4)
NOLA
$9.0 (3.61
96.0 (0.5)
(0.9)
(0.6)
80.8 (0.6)
85.2 (0.2)
90.0 (0.7)
91.7 (0.3)
NOLA-MLP
47K
83.9
(1.7)
(1.0)
87.1 (0.6)
80.7 (0.5)
85.0 (0.3)
89.8 (0.8)
91.5 (0:4)
Nearest Neighbor
33.5
39.2
15.2
21.9
16.9
29.2
$7.4
67.6
Lincar
0
(2.3 (2.3)
49.3 (2.6)
22.6 (0,9)
31.3 (0.5)
15.2 (0.3)
(1.44
75.2 (0.5)
83.2
VIT--MAE
Full-FT
289M
(4.5)
(4.0)
(11.)
(16,)
42.2 (2:3)
67.1 (1.1)
87.2 (0.8)
90.8 (0.7)
LoRA (r=4)
375K
(3.0)
82.4 (2.3)
50.2 (6.8)
35.2 (2.9)
60.8 (1.2)
87.0 (0.9)
90.7 (0.4)
LoRA (r=1)
94K
67.7 (3.8)
83.8 (1.2)
50.4 (1.0)
62.5 (0.6)
32.9 (1.8)
56.6
87.0 (0.6)
90.8 (0:4)
NOLA-MLP
94K
(3.8)
(1.4)
(L.1)
63.8 (0.8)
36,9 (5.6)
(1.6
87.4 (0.4)
90,9 (0.5)
NOLA-MEP
47K
69.6 (3.8)
84.8 (1.1)
49.9 (0.8)
62.8 (0.7)
36.1 (0.8) 58.8 (1.2)
87.1 (0.6)
90.9 (0.4)
Table 7: More results on vision tasks. Using ViT-B, NOLA achieves comparable performance as LoRA (r=4) with just one-third the number of parameters on four challenging datasets. The linear layer sizes are: 0.03M, 0.2M. O.IM, 0.3M for Aireraft, Food101, Pets and SUN397 respectively.
Method
Train Params
Aireralt
Food101
Pets
SUN397
Shot
10 Shot
Shot
10 Shot
5 Shat
10:5 Shix
5 Shot
10 Shot
Nearest Neighbor
24.6
27.1
48.4
54.2
82.3
86.2
44.4
51.5
Linear
(2.3)
(2.1)
53.2 (0.3)
61.5 (0.1)
(0.4)
(0.3)
(0.8)
42.7 (0.4)
Full-FT
82M
(2.4)
(2.0)
48.6 (5.1)
(2.7)
(2.7
91.1 (0.5)
(3.0
52.6 (0.3)
LoRA (r=4)
0. 0.14IM
(1.4)
43.8 (1.5)
60.8 (1.6)
73.1 (0.6)
85.5 (0.8)
91.6 (0.5)
51.6
55.6 (0.3)
NOLA-MPP
0.047M
(2.2)
(1.4
64.5 (0.8)
72.6 (0.4)
88.0 (0.6)
92.2(0.3)
50.5
(0.3
Results: Results on finetuning on image classification tasks are presented in Table 6. A naive Nearest Neighbor approach performs competitively on the CUB and Caltech datasets. LoRA and suggsiiiniiaanii
NOLA are significantly better than Lincar and Full-FT on several settings (c.g. CIFAR- 100 shot ViT-B-MAE). This might be due to the overfitting of the Linear and Full-FT approaches on the limited number of train samples. When using a similar number of training parameters, NOLA outperforms LoRA in most of the settings across architectures and datasets. It also achieves comparable performance to LoRA with just half or one-third of the training parameters of LoRA. This is consistent with our observations the NLG tasks. The difference between the two methods is particularly noticeable when the number of training examples is small either in 5 shot setup or when the number of classes is small, as in CIFAR-10. This suggests that the improvement obtained by NOLA could be due to the reduction in the number of training parameters. Both LoRA and NOLA consistently and significantly outperform Linear and Full-FT approachss. NOLA can easily be employed in MLP layers since the number of training parameters is decoupled from the weight matrix dimensions. A similar application of LoRA would require 8x more training parameters due to the large hidden layer dimensionality of the MLP module. We empirically observe that NOLA-MLP slightly outperforms NOLA on attention block (see Table 10 in appendix). We provide results on four additional datasets used for benchmarking transfer learning in Table 7. Aireraft, Food101 and Pets are finegrained datasets while SUN397 is a large dataset with 397 classes. There is a bigger difference in the performance of Nearest Neighbor and Linear approaches on most of these datasets
8

---

Published as a conference paper at ICLR 2024
compared to those in Table 6, suggesting that it is harder to adapt to these datasets. In line with our prior observations in Table 6, NOLA with just one-third the number of parameters performs comparably to LoRA.
4 RELATED WORKS
Vision and Language Transformer Models: Transformer networks, introduced by (Vaswani et al., 2017), emerged as a sequence-to-sequene to-sequence model in Natural Language Processing (NLP). Their success soon extended to the computer vision community, with subsequent works (Dosovitskly et al,, 2021; Touvron et al., 2021) introducing the Vision Transformer (ViT) network as a strong alternative to the Convolutional Neural Network (CNN) backbones. Transformers accept sequence of tokens as input. These tokens can be, for instance, word embeddings in language or image patches in vision. BERT (Devlin et al., 2019) and GPT-2 (Radford et al., 2018) in NLP, and MAE (He et al., 2021) and DINO (Caron et al., 2021) in computer vision train transformer networks via self-supervision on large amounts of unlabeled data. These studies demonstrate that large transformer networks when trained on massive corpora, generalize well to downstream tasks even when finetuning on very few task-specific examples. For example, (Brown et al., 2020) show that GPT-3 with 175B parameters is a good few-shot learner. Lastly, the scaling law presented by (Kaplan et al., 2020) indicates that simultaneous increase in training data and model parameters can lead to significant performance gains and emergent capabiliities previously unavailable to smaller models.
Owing to their unprecedented few-shot generalization perfor- mance, large neural networks, such as foundation models and LLMs have gained immense popularity in recent years. An increasing number of users are customizing these models to adapt them to their specific tasks. However, given the colossal size of these models, fine-tuning and storing the entire set of model parameters (Devlin et al., 2019; Radford et al., 2018) for each task is impractical. This challenge is exacerbated as the number of tasks increases. In addition to storage concerns, the overhead involved in loading tasc-specific models and transferring weights from CPU to GPU often becomes a computational bottleneck in many applications. Parameter Efficient Fine- Tuning (PEFT) approaches aim to address these issues by identifying the minimum number of parameters needed to adapt a large model. Adapters (Houlsby et al., 2019; Rebuffi et al., 2017; Lin et al., 2020b; Mahabadi et al., 2021) are PEFT approaches that achieve adaptation by adding small modules to the intermediate layers of the model. A major drawback of Adapters is the extra latency they introduce in inference. BitFit (Zaken et al., 2021) only adapt bias of the network. Ladder tuning (Sung et al., 2022a) reduce memory footprint in training by avoiding back propaggatonnn through the main backbone. IA3 (Liu et al., 2022) trains extra parameters in the attention module. Another widely adopted PEFT approach is prompt-tuning for L.MSs that involves optimizing a new set of input tokens, or prompts, for each task (Li & Liang, 2021; Lester et al., 2021; Hambardzumyan et al., 2020; Liu et al., 2021). While reminiscent of prompt engineering, the distinction lies in training a specific set of prompt tokens in prompt-tuning which might also increase inference latency.
Parameter Efficient Fine-Tuning:
(Hu et al., 2021) introduced LoRA, demonstrating that a low-rank modification of the original weights is sufficient to adapt LLM to new task. Unlike adapters and prompt-tuning. these low-rank modifications can be integrated into the original weights, thus avoiding additional overhead during inference. However, LoRA has two main limitations: 1) the rank-one decomposition sets lower bound on the parameters needed for fine-tuning, and 2) the number of required parameters contingent upon the architecture and the rank choice. Our work, termed NOLA, addresses these challenges by decoupling the trainable parameters from both the rank choice and the network architecture. Several recent studies have sought to enhance LoRA by quantizing its parameters (Dettmers et al., 2023; Xu et al., 2023; Kwon et al., 2022; Gong et al., 2023), optimizing the design choice of LoRA through neural architecture search (Zhang et al., 2022), or dynamically allocating parameters based on the required rank for each weight matrix (Zhang et al., 2023). Most of these enhancements are also compatible with our proposed method. In fact, we demonstrate that NOLA can be quantized to 4-bit without any performance degradation, thereby emphasizing that the concept of quantization is distinct from and complementary to, NOLA.
Compact Deep Learning Models: A closely related area to PEFT is model compression. Pruning
(Kim et al., 2020; Lin et al., 2020a; Siems et al., 2021; Tiwari et al., 2021; Hayou et al., 2020; Wang et al., 2020; Li et al., 202.) and quantization (Rastegari et al., 2016a; Lee et al., 2021) stand as the principal methods for compressing neural networks. Techniques such as those in (Kusupati et al., 2020; Isik et al., 2022) can achieve a high pruning rate, leading to significant compression.
9

---

Published as a conference paper at ICLR 2024
Acknowledgment: This work partially supported by DARPA under Contract No. HR00112190135 and HR00112290115 and NSF grants 1845216 and 2339898.
REFERENCES
Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the I Ith International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 7319-7328, 2021.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation, 2013. 4
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 - mining discriminative compo-
nents with random forests. In European Conference on Computer Vision, 2014. 7
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. 1, 9
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers, 2021. 9
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 35:16664166678 2022.
Wenlin Chen, James Wilson, Stephen Tyree, Kilian Weinberger, and Yixin Chen. Compressing neural networks with the hashing trick. In International conference machine learning, pp. 2285-2294.
PMLR, 2015. 16
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hier- archical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pp. 248-2555,000 2009. doi: 10. 201CVPPR.20095206844 16
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized Ilms, 2023. 7,9
Jacob Devlin, Ming-Wai Wei Chang. Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 [cs]. May 2019, URL http:://arkee org/ abs 1810. 04805. arXiv: 1810.04805.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 16 words: Transformers for image recognition scale, 2021. 9
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. Computer Vision and Pattern Recognition Workshop, 2004. 7
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. The webnlg challenge: Generating text from rdf data. Proceedings of the 10th International Conference on Natural Language Generation, pp. 124-133, 2017, 4, 17
Zhuocheng Gong, Jiahao Liu, Qifan Wang, Yang Yang, Jingang Wang, Wei Wu, Yunsen Xian, Dongyan Zhao, and Rui Yan. Prequant: A task-agnostic quantization approach for pre-trained language models, 2023. 9
10

---

Published as a conference paper at ICLR 2024
Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. WARP: Word-level Adversarial ReProgramming. arXiv:2101.000 101121000000111/11 [cs], December 2020. URL http:///axxii oww. /abs/ 2101.00124 arXiv: 2101.00121. 9
Soufiane Hayou, Francorancois Ton, Amaud Doucet, and Yee Whye Teh. Robust pruning at initializa- tion. arXiv preprini arXiv:2002.08797 2020. 9
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016. 16
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners, 2021. 9
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021. 7
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruma Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-Efficient Transfer Learning for NLP. arXiv:1902.0075: [cs, stat], June 2019. URL httttp://aaxiv.ooo/// abs / 1 1902 00750.111111 4,
Edward J Hu, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang. Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference Learning Representations, 2021. 1, 2, 4, 7,,
Berivan Isik, Tsachy Weissman, and Albert No. An information-theerrticc justification for model pruning. In International Conference on Artificial Intelligence and Statistics, pp. 3821-3846.
PMLR, 2022. 9
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-nnj inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2704-2713, 2018. 3
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
Woojeong Kim, Suhyun Kim, Mincheol Park, and Geunseok Jeon. Neuron merging: Compensating for pruned neurons. Advances in Neural Information Processing Systems, 33:585-595, 2020. 9
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar- 10 and cifar-100 datasets. URI: https://www. cs. toronto. edu/kriz/cifar himl, 6(1):1, 2009. 7
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The ci dataset. online: http://www. cs.
toronto. edu/kriz/cifar himl, 55(5), 2014. 16
Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade. and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. In Proceedings of the International Conference on Machine Learning, July 2020.
Se Jung Kwon, Jeonghoon Kim. Jeongin Bae, Kang Min Yoo, Jin- JinHwwa Kim, Baeseong Park. Byeongwook Kim, Jung-Wor Woo Ha, Nako Sung, and Dongsoo Lee. Alphatuning: Quantization- aware parameter-effiiint adaptation of large-scale pre-trained language models, arXiv preprint arXiv:2210.03858, 2022. 9
Junghyup Lee, Dohyung Kim, and Bumsub Ham. Network quantization with element-wise gradient scaling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6448-6457, 2021. 9
Brian Lester, Rami Al-Rfou, and Noah Constant. The Power of Scale for Parameter-Efficienn Prompt Tuning. arXiv:2104.0 104.0869 [cs]. April 2021. URL http: arwxiv wwwpeiiww....aaaaee. 08666 arXiv: 2104.08691. 9
11

---

Published as a conference paper at ICLR 2024
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. In International Conference on Learning Representations, 2018.
Xiang Lisa Li and Perey Liang. Prefix-Tunee Tuning: Optimizing Continuous Prompts for Generation. arXiv:2101.0019 [cs], January 2021. URL http: /ttp:::///xxiiv/// org www/abs////aaaaaaaaaa//////aaaaa 2ttp:2111..000 001 90. 4,
Yuchao Li. Shaohui Lin, Jianzhuang Liu, Qixiang Ye, Mengdi Wang, Fei Chao, Fan Yang. Jincheng Ma, Qi Tian, and Rongrong Ji. Towards compact cnns via collaborative compression. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. (438-6447) 2021. 9
Tao Lin, Sebastian U Stich, Luis Barba, Daniil Dmitriev, and Martin Jaggi. Dynamic model pruning with feedback. arXiv preprint urXiv:2006.07253, 2020a.
Zhaojiang Lin, Andrea Madotto, and Pascale Fung. Exploring versatile generative language model via parameter-efficien transfer learning. In Findings of the Association for Computational Linguisties: EMNIIP 2020, pp. 441-459, Online, November 2020b. Association for Computational Linguis- tics. doi: 10018865/112220...0
findings-emnlp.41. URL WtLhttps:/// https///aaaaaatttaaa//// org/2020. findings emnlp 41. 4,
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950-19989 1965,,996,,..1 2022. 9
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. GPT Understands, Too. arXiv:2 10385 [cs], March 2021. URL http: http://axxvvvo.o org..iiii.o......o.ooooo.oo /bs/ 2103. 10385. arXiv: 2103.10385. 9
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers, 2021.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew B. Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint 306iv: 2013. 7
Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, et al. Dart: Open-domain structured data record to text generation. arXiv preprint 2rXiv:2007.0287/ 2020. 4, 17
Parsa Nooralinejad, Ali Abbasi, Soroush Abbasi Koohpayegani, Kossar Pourahmadi Meibodi, Rana Muhammad Shahroz Khan, Soheil Kolouri, and Hamed Pirsiavash. Pranc: Pseudo random networks for compacting deep models. In International Conference on Computer Vision (ICCV), 2022. 2, 3, 4, 16
Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. The e2e dataset: New challenges for end-to-end generation. arXiv preprint arXiv:1706.092. 2017.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa. Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07199, 2023. 1
O. M. Parkhi, A. Vedaldi. A. Zisserman, and C. V. Jawahar. Cats and dogs. In Computer Vision and Pattern Recognition, 2012. 7
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho. and Iryna Gurevych. Adapter- fusion: Non-destructive task composition for transfer learning, 2021. 4
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, al. Improving language understanding by generative pre- treiirain 2018. 9
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks, In European conference on computer vision, pp. 525-542. Springer, 2016a.
12

---

Published as a conference paper at ICLR 2024
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks, 2016b.
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters, arXiv: 1705, 08045 [cs, stat), November 2017. URL http:://axxii.orrre/e arXiv: 1705.08045. 9
abs/1705. 08045,
Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. Adapterdrop: On the efficiency of adapters in transformers, 2020. 4
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint urXiv:2302.0476. 2023.
Julien Niklas Siems, Aaron Klein, Cedric Archambeau, and Maren Mahsereci. Dynamic pruning of neural network via gradient signal-to-none to-isee ratio. In 8th ICML Workshop on Automated Machine Learning (AutoML), 2021.
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Lst: Ladder side-tuning for parameter and memory efficient transfer learning, Advances in Neural Information Processing Systems, 35:12991- 2022a.
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Vl-adapter: Parameter-efficien transfer learning for vision-and-language tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5227-5237, 2022b.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-follovin Ilama model. https::// https://github www/tattuuueeccoccccooooooootott haabb/stanfor__aapaaa
2023.
Rishabh Tiwari, Udbhav Bamba, Arna Chavan, and Deepak K Gupta. Chipnet: Budget-aware pruning with heaviside continuous approximations. arXiv preprint arXiv:2002.07156, 2021. 9
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers and distillation through attention, 2021. 9
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes. Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan. Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Ilama 2: Open foundation and fine-tuned chat models, 2023. 6
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31 International Conference on Neural Information Processing Systems, pp. 6000-6010, 2017. 9
Huan Wang, Can Qin, Yulun Zhang, and Yun Fu. Neural pruning via growing regularization. arXiv preprint arXiv:2012.09243. 2020. 9
Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pictro Perona. Caltech-ucsd birds 200. 2010. 7
Ross Whitman. Timm. line; accessed 28-Sep-2023).
htmp:////.......o.oooooooe/tttie [On-
13

---

Published as a conference paper at ICLR 2024
Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In Computer Vision and Pattern Recognition. 2010. 7
Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhensu Chen, Xiaopeng Zhang, and Qi Tian. Qa-lora: Quantization-aware low-rank adaptation of large language models, 2023. 2,
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficien tne-tunnig for transformer-based masked language-models. arXiv preprint arXiv:2106. 101999 2021.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning, 2023. 9
Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt search, 2022. 9
14

---

Published as a conference paper at ICLR 2024
A
APPENDIX
A.1 MEASURING THE RANK OF POSSIBLE SOLUTIONS IN NOLA VS PRANC:
Choosing n basis vectors (d dimensional each) in PRANC will result in all possible learned matrices living in n dimensional subspace. However, since NOLA with the same number of total parameters (k+1= = n) uses A x B factorization, the possible solutions can live higher dimensional subspace. We do simple experiment by sampling several random cocfficient vectors, reconstructing the AW matrix, reshaping it to be a long (d")-dimensional vector, and measuring the rank of the covariance of samples
how much of the whole space is covered by the samples. The results are shown in Figure for a simple experiment with varying d and n. As expected, NOLA can cover the whole space (full-rank) using small number of parameters compared PRANC. Note that the rank in this analysis is on the covariance of possible random samples of weight matrices and should not be confused with the rank in LoRA or NOLA formulation.
1.0
Percentage Coverage
0.8
NOLA, d=16
0.6
NOLA, d=32
NOLA, d=64
PRANC, d=16
0.4
PRANC, d=32
PRANC. d=64
0.2
0.0
50
100
150
200
250
72
Figure 2: Comparing the rank of samples in the solution subspace for PRANC and NOLA, given the same number of parameters, n. "Percentage Coverage" is the subspace rank divided by the max possible rank (d²²), 1.0 denotes full rank. As expected, the coverage for PRANC increases linearly while it saturates very fast for NOLA.
A.2 NOLA IN TRAINING FROM SCRATCH:
MLP on MNIST (a toy experiment):
We believe NOLA is better reparametrization than PRANC and can achieve local minimums that PRANC cannot achieve. To show this empirically, we perform a very simple experiment where we apply a 2-layer MLP for the MNIST classification task. Since better at reaching the local minimums and not necessarily the generalization, we evaluate the models by the training loss. Also, we intentionally use large number of neurons in the hidden layer to wver-parameterize and increase the number of local minimums.
measure which method is
We use linear layer from 784 features 256 with bias followed by ReLU and a linear layer from 256 to 10 classes. We use all 60K samples for training to report the training loss. For both PRANC and NOLA, we use 32 parameters for each layer. Other hyperparameters are same for both: 200 epochs, 512 batch size with Ir 0.05. We use rank loss of 0.87, while NOLA achieves a training loss of 0.71. This simple experiment empirically supports that NOLA has better representation power. Additionally, PRANC training finishes in 1152 seconds while NOLA finishes 579 seconds. We will leave the theoretical study of this comparison for future work. Moreover, this experiment empirically shows that NOLA is generic method, and its success is not dependent on the architecture of transformers or attention modules.
4 for NOLA. PRANC has a final training
CNN on ImageNet100 and CIFAR-10:
Moreover, to compare the representation power NOLA and PRANC, we train NOLA from scratch image classification task using CNN architecture. For each convolution layer, reshape
15

---

Published as a conference paper at ICLR 2024
all parameters of layer into matrix (close to square shape) and apply NOLA to the matrix. Then, we reshape it to the original shape of the convolution. Additionally, we train LoRA using a similar approach NOLA. We follow a similar setup as (Nooralinejad et al., 2022) for our experiments on image classification.
Datasets and Architectures: We consider two architectures in experiments: ResNet20 with 270K parameters, and ResNet18 (He et al., 2016) with 11M parameters. We train ResNet20 on CIFARIO (Krizhevsky al., 40, and ResNet18 on ImageNet100 (Deng al., 2009).
Results: We report result of ImageNet100 in Table 9, and CIFARIO in Table 8. NOLA outperforms both PRANC and LoRA with a similar number of parameters.
For ImageNet and ResNet18, we use k = I = 2, 000 basis for each of 20 modules, and for the classifier (last linear layer), we used k = | = 10, 000, resulting in total of 100, 000 trainable parameters excluding 9, 600 batchnorm parameters. We use rank 64 for all layers. We train all models using Adam optimizer with a learning rate of 0.001 and batch size of 256 for 200 epochs. For CIFAR-10 10 and ResNet20, we use k = 1 = 250 basis for each convolutional module, and for the linear layer, we use k = =1- 1000 parameters. We use batch size 256, Adam optimizer, and a learning rate of 0.001. We use a single EVIDIA-GeFord
Implementation Details:
RTX 3090 for all experiments.
Training Time Comparison: We measure the training time of NOLA and PRANC on a single NVIDIA-GeForce RTX 3090 GPU and batch size of 256. Note that training time includes both forward and backward passes for each batch. On average, NOLA processes a batch in 228ms while PRANC does the same in 1070ms, so NOLA is 4.6 times faster than PRANC.
Table 8: Training On CIFAR10: Result of our method CIFARIO dataset and ResNet20.
Table 9: Training On ImageNet100: Result of our method on ImageNet- 100 dataset and ResNet18
Method
f Params
Acc.
Method
Params
Acc.
trained model
11,227,812
82.1 %
trained model
269,722
88.92%
HashedNet (Chen et al., 2015)
129,200
52.96%
PRANC LoRA NOLA
12,752
81.5%
PRANC
119,200
61.08%
13,295
81.5%
LoRA NOLA
150,000
63.50%
12,876
82.4%
109,600
64.66%
A.3 ABLATION AND DETAILS OF NOLA ON VISION TRANSFORMERS:
Implementation detail: We consider learning rates of 3, 1e 3 and 5e 4 for LoRA, NOLA and Linear methods and 3, 5,-555555600 3e 3e-3 5 and 1e-5 5 for Full-FT. The best settings is chosen based on the performance on validation set. For creation of k-shot dataset, we randomly sample without replacement from the train set. For each of these sets, we run with three different initializations of the networks. This process is repeated four times and the averaged values are reported.
Comparison between NOLA-QV and NOLA-MLP: We experiment with NOLA layer in both the attention and MLP modules of the vision transformer. We observe that applying NOLA on MLP performs better than that attention block (Table 10). Thus, we use NOLA-MLP as our default setting. Note that the number of trainable parameters remains the same in both versions. Unlike this, applying LoRA MLP block would require significantly higher number of trainable parameters due to the increased dimensions of the weight matrices in MLP compared to those in attention block.
Table 10: Comparison between NOLA in MLP and attention blocks: We observe that NOLA on MLP block is more effective. We choose this our default setting.
Base Model
# Train Params
CIFAR 10
CIFAR 100
CUB 200 2011
Caltech- 101
10
5
10
5
10
NOLA-QV MOLA-MLP
47K
(7.0
(0.7)
74.8 (0.6)
80.4 (0.9)
75.3 (0.4)
81.7 (0.3)
(1.1
90.6 (0.5)
VI--
47K
(1.3)
92.2 (0.5)
75.1 (0.6)
81.3 (0.8)
75.5 (0.6)
81.7 (0.4)
(1.2)
90.6 (0.5)
16

---

Published as a conference paper at ICLR 2024
A.4 RESULTS OF NLG TASK ON DART AND WEBNLG DATASETS:
In Table 11, we report more results similar to Table using GPT-2 M and GPT-2 L on DART (Nan al., 2020) and WebNLG (Gardent al., 2017) datasets.
Table 11: DART and WebNLG Dataset: Similar to Table compare NOLA other methods.
NOLA is on par or better with other methods with the same number of parameters.
GPT-2 M
Method
Adapted
Adapter Rank
# Trainable
DART METT
WebNLG METT
Layers
Parameters
BLEUT
TER
BLEUT
TER]
Finetune
Layers
354.000M
46.2
0,46
46.5
0.53
Adapter
Layers
0.370M
42.4
0.36
0.48
0.38
0.43
Adapter
Layers
Finetune Tiet
Layers
PreLayer
Extra Tokens
0.350M
46.4
0.38
0.46
55.1
0,41
0.40
LoRA
QV
0.350M
47.1
0.39
0.46
54.9
0.41
0.39
LoRA
QV
0.098M
46.4
0.40
0.40
NOLA (Ours)
QV
0.096M
47.0
0.38
0.48
53.9
0.40
0.40
NOLA (Ours)
MLP
0.096M
47.1
0.38
0..4
54.7
0.41
0.40
NOLA (Ours)
QV
0.048M
45.7
NOLA (Ours)
ML.P
0.048M
45.5
0.38
0..49
53.0
0.40
0.40
L
Finetune
All Layers
774.000M
47.0
0.39
0.46
55.5
0.42
0.42
Adapter
Extra Layers
0.880M
45.7
0.38
0.46
56.0
0.41
0.39
Adapter'
Extra Layers
230.000M
47.1
0.39
0.45
57.7
0.43
0.39
Prelayer
Extra Tokens
0.770M
46.7
0.38
0.45
56.3
0.42
0.40
LoRA
QV
0.770M
47.5
0.39
0.45
57.1
0.43
0.38
LoRA
QV
0.184M
47.7
0.39
0.47
55.9
0.42
0.39
NOLA (Ours)
QV
0.144M
47.8
0.39
0.47
0.41
0.39
NOLA (Ours)
MLP
0. 0.144M
47.8
0.39
0.47
56.0
0.42
0.39
NOLA (Ours)
QV
0.072M
46.4
0.38
0.48
55.5
0.41
0.38
NOLA (Ours)
MLP
0.072M
46.8
0.38
0.48
0.41
0.39
17


Reviews:
[{"review_id": "u4QAaagKGM", "reviewer": "Reviewer_P5nc", "summary": "Low Rank Adaptation (LoRA) presents a series of drawbacks, particularly its constrained parameter reduction due to rank-1 matrices, which cannot be further diminished. Additionally, LoRA's parameter count is heavily reliant on the model's architecture. In response to these limitations, this paper suggests an innovative solution by advocating the use of a linear combination of random projections to replace LoRA's update matrix, effectively addressing the issues mentioned earlier. This approach is inspired by the previous paper known as PRANC. Personally, I found the paper to be a valuable source of knowledge and a unique one, appreciating its quirky yet straightforward idea", "questions": "1. GPT2-L and GPT-2M seems to perform the same for LoRA. Is there any explanation on why this is the case?\n2. The presentation of results preceding the training details in Section 3.1 seems to be an inadvertent oversight. To enhance the logical flow of the content, it is advisable to reverse the order, placing the training details before the results.What happens when you increase the number of parameters for NOLA?  - Does it perform better than LoRA. For example results of NOLA with 0.35M parameters\n3. How does NOLA's performance change when the number of parameters is increased? Does it outperform LoRA? For instance, are there any results available for NOLA with 0.35 million parameters?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "**Leveraging Ideas from Other Papers for Enhanced Parameter Efficiency:** This paper skillfully incorporates concepts from existing research to optimize parameter efficiency.\n\n**Achieving Comparable or Superior Performance to NOLA:** This research attains performance levels akin to LoRA while significantly enhancing parameter efficiency.", "weaknesses": "**Poorly presented results**-  The main issue in the presentation of the results lies in their lack of clarity and explanatory depth. Firstly, the results fail to offer any substantial insights into how the method operates, leaving readers without a clear understanding of the underlying mechanisms. Additionally, Tables 1 and 5 are presented as mere lists of numbers without the necessary context or explanation, making it challenging for the audience to derive meaningful conclusions from the data. A critical element that appears to be missing is a discussion of what works better and the reasons behind it, which is crucial for a comprehensive understanding of the findings. To improve the presentation of the main results, it is essential to provide better explanations and context for the data, as well as a deeper analysis of what drives the observed outcomes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dTZUsXHBIk", "reviewer": "Reviewer_ba8s", "summary": "The paper proposes a new approach for fine tuning LLMs for downstream tasks. The key idea is to replace the low rank updates of LoRA with linear combinations of fixed random matrices for which only the coefficients need to be tuned and stored in memory which significantly reduces the storage cost. The authors present experiments in both language and vision tasks where their approach preserve the accuracy of LoRA while reducing the parameter count by half or more.", "questions": "1. Why is the training time of NoLA with shared random basis similar to that of LoRA when the training time of NOLA with a unique random basis is higher? Aren't the number of coefficients being trained, the same in both cases?\n\n2. The ablation study at the end of Section 3.1 appears inconclusive. Is there any takeaway on the effect of varying the rank in NOLA?\n\n3. In Section 3.2 if only $\\alpha$ and $\\beta$ are quantized while A and B are not then won't that be less memory efficient than quantization in LORA?\n\n4. Please highlight the entries in Table 5 with the best performance for a given scenario. Currently there are too many entries, and it is too difficult to figure out which method is better for which case.\n\n5. If each matrix in PRANC has size $d^2$ then why do we need multiple matrices to cover the rank of the original $\\Delta W$ matrix (which also has size $d^2$)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors propose a novel, intuitive, and principled approach to address the problem of task based fine tuning of transformer based models.\n\n2. The proposed approach shows significant reduction in storage overhead without compromising on accuracy across a range of experiments in both language and vision tasks.", "weaknesses": "1. The technical novelty is relatively minor with the overall idea being a combination of prior works PRANC and NOLA. While this seems enough to provide empirical improvement, the approach itself is not that big of an innovation over prior works. \n\n2. While the prior approach PRANC is directly modified by the authors in this work there are no direct comparisons with it in either the language or vision tasks used to evaluate the proposed approach. There is a comparison of training loss in Section 3.4 and a comparison of the rank of possible solutions of the two approaches in Section 3.5 but without a direct comparison of test accuracy it is unclear if this approach is indeed an improvement over the baseline that it directly modifies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RQME12fRwF", "reviewer": "Reviewer_WWfE", "summary": "The paper looked at the problem of memory requirements for Low-Rank Adaptation (LoRA) and proposed NOLA to break the rank one lower bound present in LoRA. The core concept behind NOLA is to reparameterize a neural network using a linear combination of pseudo-randomly generated weights.\n\nThanks to the authors for a more detailed explanation of the motivations for the paper and for some of the latest research supporting them. Therefore, I will increase the rating by 1 point.", "questions": "If users want to use the trained model on different versions of GPUs or software, how to ensure the consistency of the trained model? In such a situation, the same seeds can not generate the same pseudo-random matrices.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper discusses related works in detail and clearly summarizes its own contributions.\n2. The paper performs extensive experiments to compare NOLA and existing PEFT solutions.\n3. NOLA decouples trainable parameters from the choice of rank and the network architecture.", "weaknesses": "1. The work may need more rationales upfront to motivate the problems (i.e. the rank one lower bound present in LoRA). Given that mainstream GPUs have tens of GB of memory, it is reasonable to reduce the memory requirements from tens of GB to tens of MB at the expense of model quality through LoRA, as this can indeed reduce resource consumption and greatly reduce LLM transition overhead during inference. However, I don't think it makes much sense to further reduce memory requirements to several MBs at the expense of model quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ia9u1rp9ce", "reviewer": "Reviewer_CrTj", "summary": "This paper proposes a technique to reduce the number of trainable parameters while fine-tuning large language models and large vision encoders. Their technique involves modeling the weight update matrix as a linear combination of fixed random matrices that are also rank-constrained. The fine-tuning process then involves learning just the coefficients in the linear combination. When the proposed technique is used, any updates to the models requires the communication/ storage of only the coefficients and the random seeds required to generate the codebook matrices (apart from the base weights of the original model of course). \n\nThe authors demonstrate that the proposed technique preserves the performance on the fine-tuning task while achieving large ( up to 1/20 x baseline models) reduction in the number of trainable parameters. This is shown in both language and vision domains. In the language domain, they show it on the NLG challenge dataset. In the vision domain, they show is on CIFAR, CUB and Caltech-101 datasets.", "questions": "1. What is the distribution used to generate the random basis matrices? Did the authors experiments with a few different choices?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well motivated and clearly written. \n- Parameter-efficient fine-tuning is a popular area of research currently and this paper makes a good contribution to this area.\n- The proposed technique is interesting and the results demonstrate that the method preserves performance while achieving a low parameter count \n- The proposed method helps overcome the some of the limitations of methods such as LoRA, as described in the paper", "weaknesses": "I appreciate the results provided in the paper. But I think that some more in-depth evaluation and some more explanation of the current results would add value to the paper. I outline some specifics below.\n\nLanguage experiments: \n- Why does full fine-tuning achieve much lower performance in Table 1?\n- Could the authors also provide the performance of the GPT-M, L models *without any fine-tuning* on the tasks considered? This will give the readers an idea of how much improvement is being achieved. \n\nVision experiments: I find the experiments provided in this section (3.3) a bit weak. The main reason being that the ViT models pre-trianed on imagenet are already pretty powerful. Fine-tuning these models on much smaller and easier datasets such as CIFAR may not be the best way to demonstrate usefulness. In particular, I have the following comments:\n\n1. Can the authors try their techniques on more challenging datasets? \n2. Although the number of parameters for the linear layer baseline depends on the dataset, it would be good to have this information visible in the Table. \n3. As before, can the authors provide the performance of the models considered without any fine-tuning? (0-shot classification on the downstream datasets)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)