Title: Structured Unrestricted-Rank Matrices for Parameter Efficient Finetuning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Low Displacement Rank

Structured Matrices

Transformers

Vision Transformers

Fine-tuning
Abstract: Recent efforts to scale Transformer models have demonstrated rapid progress across a wide range of tasks (Wei at. al 2022). However, fine-tuning these models for downstream tasks is quite expensive due to their large parameter counts. Parameter-efficient fine-tuning (PEFT) approaches have emerged as a viable alternative, allowing us to fine-tune models by updating only a small number of parameters.
 In this work, we propose a general framework for parameter efficient fine-tuning (PEFT), based on *structured unrestricted-rank matrices* (SURM)  which can serve as a drop-in replacement for popular approaches such as Adapters and LoRA. Unlike other methods like LoRA, SURMs give us more flexibility in finding the right balance between compactness and expressiveness. This is achieved by using *low displacement rank matrices* (LDRMs), which hasn't been used in this context before. SURMs remain competitive with baselines, often providing significant quality improvements while using a smaller parameter budget. SURMs achieve: **5**-**7**% accuracy gains on various image classification tasks while replacing low-rank matrices in LoRA and: up to **12x** reduction of the number of parameters in adapters (with virtually no loss in quality) on the GLUE benchmark.

Structured Unrestricted-Rank Matrices for Parameter Efficient Fine-tuning

Abstract

Recent efforts to scale Transformer models have been successful across a wide range of tasks [77]. However, fine-tuning these models for downstream tasks can be expensive, as it requires updating a large number of parameters in the Transformer model. Parameter-efficient fine-tuning (PEFT) approaches have emerged as a viable alternative that allow us to fine-tune models by updating only a small number of parameters. In this work, we propose a general framework for parameter efficient fine-tuning using structured unrestricted-rank matrices (SURM), which can serve as a drop-in replacement for popular approaches such as Adapters and LoRA. Unlike other methods like LoRA, SURMs provides more flexibility in finding the right balance between compactness and expressiveness. This is achieved by using low displacement rank matrices (LDRMs), which has not been used in this context before. SURMs remain competitive with baselines, often providing significant quality improvements while using a smaller parameter budget. SURMs achieve 5 -7 % accuracy gains on various image classification tasks while replacing lowrank matrices in LoRA. It also results in up to 12x reduction of the number of parameters in adapters (with virtually no loss in quality) on the GLUE benchmark.

1 Introduction

In recent years, large-scale Transformer models have demonstrated impressive performance across a wide range of domains, including natural language processing (NLP) [20, 8], vision [36], robotics [7], and even multi-modal settings [81]. For many applications, a single large pre-trained model is adapted for several downstream problems. Fine-tuning , where all the model parameters are updated, is a popular way to adapt a pre-trained model to a new task or domain. However, fine-tuning large models on specific downstream tasks requires significant computational resources and involves a massive memory footprint, as each task necessitates storing its own set of parameters.

Parameter-efficient fine-tuning (PEFT) methods have emerged as the preferred methodology to adapt pre-trained Transformers to different downstream tasks. PEFT methods often achieve performance on par with full fine-tuning while training only a small number of parameters [80, 45]. PEFT techniques involve either training a small subset of the model's parameters [84, 42] or integrating small modular layers while freezing the base model's weights [26, 25]. There are two popular classes of methods to inject additional parameters: (a) using small modular layers inside Transformers called adapter layers [59], and (b) constraining the updates as low-rank matrices ( LoRA ) [26].

Although adapters and LoRA (including their variants) differ architecturally and conceptually, they share a common reliance on low-rank matrices. The success of these methods has been attributed to the low intrinsic dimensionality of the hidden representations in the pre-trained Transformer

Google DeepMind

UNC Chapel Hill

LowRank Kronecker Circulant

Toeplitz

0.60

0.65

0.70

0.75

Number of Parameters

Full Fine-tuning

Attention

Tuning

Transformer

Probing

Adapter

LoRA

KAdaptation

Lin. Probing

LePE

BitFit

LoRA-Fix

LN Tuning

SURM

models [1, 70]. These low-rank methods primarily aim to approximate updates, which, in general, are not low rank. Hence, there's no justification for imposing low-rank constraints on them. Motivated by this insight, we explore alternative classes of matrices-ones that aren't necessarily low rank but are characterized by a linear number of parameters while exhibiting impressive approximations across various matrix classes. We present Fig. 1 as a preview of the motivating results. In Fig. 1 (left), we show that structured matrices (SURM) can approximate any random matrix better than low rank matrices. In Fig. 1 (right), we show that when SURMs are used for parameter efficient fine-tuning it outperforms existing PEFT methods (see more details in Sec. 4).

We propose a novel paradigm of parameter efficient fine-tuning that leverages Structured UnrestrictedRank matrices (or SURMs). SURMs provide similar efficiency gains as previous works in efficient fine-tuning, but their more expressive structure paves the way for quality improvements. In this work, we propose to perform parameter-efficient fine-tuning by parameterizing learnable weights as structured matrices. We focus on the two sub-classes of SURMs: (1) Kronecker product of matrices [3] and (2) low displacement rank matrices (LDRMs) [66, 6, 58, 54, 73]. To summarize, our primary contributions are:

We propose the class of Structured Unrestricted-Rank matrices (SURMs) (Section 3), for parameter efficient fine-tuning of Transformers. SURMs include low-rank matrices used in LoRA, as special cases. To the best of our knowledge, we are the first to apply LDRMs in this context.

We demonstrate strong matrix approximation capabilities inherent in Low Displacement Rank Matrices, with a specific focus on circulant and Toeplitz matrices (Section 4).

We introduce a new class of adapter-layers using SURMs, achieving a 12x reduction in parameters compared to adapters, with virtually no loss in quality on the GLUE benchmark (Section 6).

We achieve 5 -7 %accuracy gains over LoRA on a wide variety of image datasets as well as in low resource setting (VTAB-1k benchmark). In some cases SURMs outperform full fine-tuning, while using only 55 k training parameters (as shown in Fig. 1 (right)).

2 Related Work

With the introduction of BERT [20] and GPT-2 [8], Transformer models trained on general text corpora have revolutionized the field of machine learning (ML). Since then, these models have continued to increase in size, with open-source variants adopting various architectures. Examples include encoder-decoder models such as T5 [64] with up to 20B parameters [67], and a range of auto-regressive decoder models like Llama [69], Pythia [4], Mistral [28], among others, varying in size from a few million to 180B parameters [2]. These models can be easily adapted to downstream tasks by fine-tuning on task-specific data, resulting in state-of-the-art performance across a broad

spectrum of downstream tasks. Due to the computational infeasibility of fine-tuning all the parameters of these models, in-context learning [8] and prompt engineering [11, 22] have emerged as attractive alternatives to adapt models to downstream tasks. However, such adaptation results depend heavily on the design of the input prompt and tend to vary greatly with small perturbations of the prompts [48].

Consequently, many works have proposed various PEFT techniques. One of the earliest methods involves inserting the so-called adapter layers between existing layers in a neural network [25, 59]. An adapter is typically an MLP with input, output, and a smaller middle layer, encoded by two low-rank matrices, making it compact in terms of parameters. An extension of the adapter is Compacter [51], which uses Tucker decomposition to parameterize the adapter layers and weight-sharing to reduce the number of trainable parameters. Various modifications and extensions of the above methods have been proposed [53, 24, 32, 52, 65]. Another popular PEFT technique is differentiable prompt-tuning (DPT), which can be thought of as optimizing special tokens in the prompt [88]. However, these methods are limited by the sequence length of the underlying models. Even though DPT was originally developed for NLP, several works have extended it for computer vision tasks as well [83, 12, 27, 24].

One of the most popular PEFT methods is Low-Rank Adaptation (LoRA) [26], which imposes a low-rank constraint on the weight updates. The main difference between adapters and LoRA is that the learned LoRA weights can be merged with the frozen model weights during inference without adding any latency. Given the popularity of LoRA, there have been many works on extending it to different contexts like long-range modeling [13], multi-tasking [10] or improving its efficiency [19, 71, 46, 37, 31] among many others.

In general, low-rank matrices are studied extensively in various ML applications [57, 87, 44, 61]. The research on low displacement rank matrices (LDRMs) for ML is more narrow [89, 68, 41, 66, 15, 35, 62]. Although Kronecker matrices (a class of LDRMs) have been explored in the context of LDRMs [21, 24, 51], the constituent matrices in the Kronecker product have low rank even in these work. In this work, we use a fixed parameter budget but do not impose any rank-based condition. To the best of our knowledge, we are the first to systematically explore the effectiveness of different structured matrices and introduce LDRMs for parameter-efficient fine-tuning.

The rest of the paper is organized as follows: (a) We introduce the notion of Structured UnrestrictedRank Matrices (SURM) that are used in this work (Section 3), (b) We motivate the usage of SURM by empirically showing the approximation qualities of these matrices (Section 4), (c) We use SURM as drop-in replacement for popular approaches such as Adapters and LoRA (Section 5), (d) We validate our approach across a wide range of vision and NLP tasks (Section 6).

3 Structured Unrestricted-Rank Matrices (SURM)

In this section, we will define the matrices that are used for parameter efficient fine-tuning. First, we define the concept of a structured matrix , which is a generic term for a matrix A ∈ R m × n that can be represented by fewer than mn parameters. These matrices are useful because they reduce both space and time complexity when performing matrix multiplications.

A simple example of a structured matrix is a low rank matrix of the form W = AB /latticetop ∈ R m × n , where A ∈ R m × r , B ∈ R n × r with r /lessmuch min( m,n ) . In this work, our main focus is on those classes of structured matrices that are not restricted to be low-rank, which we refer to as Structured Unrestricted-Rank Matrices (SURM). Next, we present two classes of SURM matrices that we use for parameter efficient fine-tuning.

Low Displacement Rank Matrices . Our first class of SURMs is low displacement rank matrices (LDRMs). A matrix W ∈ C m × n is said to have ( A , B ) -displacement structure if:

where A ∈ C m × m , B ∈ C n × n , F ∈ C m × n and F has low rank r (as compared to min( m,n ) ). We call ∇ A , B the displacement rank operator , parameterized by A and B .

/negationslash

For a given W , there can exist several pairs of ( A , B ) matrices satisfying Eq. 1 that produce a lowrank matrix, F . Some examples of such ( A , B ) pairs include: ( Z , Z ) , ( Z , Z /latticetop ) , ( D x , Z /latticetop ) , ( D x , D y ) (for x = y ). Here Z is a circulant-shift matrix and D z is a diagonal matrix with nonzero entries equal to z . Low displacement rank matrices ( W in Eq. 1) enable fast (sub-quadratic) matrix-vector multiplication and enhance the efficiency of other matrix operations, such as inversion. By selecting

c

0

c n

-

1

…

c

2

c

1

c

1

c

0

c n

-

1

…

c

2

⋮

c

1

c

0

/uni22F1

⋮

c m

-

2

⋮

/uni22F1

/uni22F1

⋮

c m

-

1

c m

-

2

…

…

c m

a

0

a

-

1

…

…

a

-

(

n

-

1)

a

1

a

0

a

-

1

…

⋮

a

2

a

1

a

0

/uni22F1

⋮

⋮

⋮

/uni22F1

/uni22F1

a

-

1

a m

-

1

…

…

a

1

a

0

a

11

B

…

a

1

n

B

⋮

/uni22F1

⋮

a m

1

B

…

a mn

B

(a) Circulant

(c) Kronecker

more complex ( A , B ) pairs, such as those involving general Jordan form matrices , it is possible to consider more unstructured W that still have compact representations and support efficient matrix operations [56, 66]. In this paper, we focus on classic low displacement rank matrices: circulant and toeplitz matrices, which are described below.

Circulant Matrices : A circulant matrix C ∈ C m × n can be parameterized by its first row. The following rows are obtained from the previous one by applying a right circulant shift. A schematic visualization of a circulant matrix is shown in Fig 2 (a) . Since we only need to store the first row, circulant matrices can be trivially encoded in O ( n ) space. They also support fast matrix-vector multiplication in O (( n + m ) log( n + m )) time using Fast Fourier Transform (FFT) [55].

Toeplitz Matrices : A toeplitz matrix T ∈ C m × n is a matrix whose entries are constant along each diagonal. A schematic visualization of a toeplitz matrix is shown in Fig 2 (b) ). They can be parameterized using only their first row and column, allowing them to be encoded in O ( n + m ) space. Similar to circulant matrices, they support fast O (( n + m ) log( n + m )) matrix-vector multiplication via FFT.

Kronecker Product of Matrices . Kronecker products are another class of structured unrestricted rank matrices that have low storage complexity and admit efficient matrix-vector multiplication. These matrices are obtained using a Kronecker product A ⊗ B of two matrices A and B , as shown in Fig 2 (c) . We provide more details about these matrices in Appendix A.3.

4 LDR-SURMs as General Approximators

In this section, we motivate the usage of structured unrestricted-rank matrices (SURMs) for parameterefficient fine-tuning. In general, the parameter updates ∆ W can be arbitrary matrices, and an effective parameterization of ∆ W should be sufficiently expressive to approximate them. Since we use structured update rank matrices (SURMs) to parameterize ∆ W , we demonstrate that SURMs can approximate various classes of matrices. Without loss of generality, in this section, we assume that all our matrices have real entries and that weight matrices are square ( n = m ).

First, we recall the result from [66], which states that a broad class of low displacement rank matrices, as well as linear combinations of Toeplitz (or their inverses) products 2 , can be parameterized as:

where G = [ g 1 , ..., g r ] , H = [ h 1 , ..., h r ] ∈ R n × r , and Z f ( v ) (for any f ∈ R , v ∈ R n ) is defined as:

When f = 1 , Z f ( v ) is a circulant matrix and when f = -1 , we refer to Z f ( v ) as a skew-circulant matrix. Moreover, F can be decomposed as follows: F = GH /latticetop for G = [ g 1 , ..., g r ] , H = [ h 1 , ..., h r ] ∈ R n × r . One can think about rank r of F of controlling how 'structured' W is.

From the above result, we see that W ( G , H ) is the most expressive parameterization among the ones discussed so far. To understand how they fare with SURMs in practice, we evaluate their approximation qualities in two settings: (a) comparing W ( G , H ) with circulant and Toeplitz matrices, and (b) comparing circulant and Toeplitz matrices with low-rank matrices.

4.1 Comparing W ( G , H ) with Circulant and Toeplitz Matrices

We test the approximation capabilities of matrices W ( G , H ) (Eq. 2) and compare it with popular classes of SURMs: circulant and Toeplitz matrices (Section 3). Specifically, we use these structured matrices to approximate three broad classes of matrices: (a) random, (b) near-low-rank, and (c) nearlow-intrinsic-rank. We denote the ground-truth matrix that we try to approximate as M ∈ R 100 × 100 and parameterized structured matrix as A (see more details about the setup in Appendix A.7). For all matrices, we obtain the parameters of A using gradient descent on the loss function: ‖ A -M ‖ 2 F . 3

In Figure 5, we report the relative Frobenius norm error during training for different settings. In Fig 5 (top left), we use W ( G , H ) with different r (rank of F in Eq. 1) is used to approximate random matrices. While the best approximations are achieved for larger values of r (specifically, r = 20 ), it is interesting to note that the final error does not decrease monotonically with increasing r . For the remaining class of matrices M , which are close to low-rank and therefore easier to approximate, we experiment with smaller values of r and report the results in Figure 5 (left column, middle and bottom). In this case, we observe that the three top-performing approximators W ( G , H ) were trained with r = 1 , 2 , 4 . These results indicate that for more structured ground truth matrices (even if they are not necessarily low-rank), LDRMs with a very low rank for the corresponding F

10 0

10 1

10 2

10 3

Steps

10 0

10 1

Training Loss

LoRA

SURM (Circulant)

SURM (Toeplitz)

Full Fine-tuning

are sufficient.

Motivated by the results showing that LDRMs with low r can serve as effective approximators, we use circulant and Toeplitz matrices to approximate near-to-low-rank and low-intrinsic-rank matrices. In the three plots shown in Fig. 5 (right column), we observe that approximations using Toeplitz matrices (using twice as many parameters as circulant matrices) offer negligible gains and are only beneficial in the near-low-rank case. For the low-intrinsic case, circulant matrices outperform Toeplitz ones. Overall, circulant matrices with few parameters achieve strong performance in this setting.

4.2 Comparing Low Rank with Circulant and Toeplitz Matrices

In this section, we focus on the difference in approximation qualities between low rank matrices and the circulant and Toeplitz matrices under a fixed parameter budget. We use the following settings:

55

random

LDR:10

1.10

low-rank(l=5)+epsilon

circulant

LDR:15

50

LDR:20

1.08

45

error

40

error

1.06 -

35

1.04

30

25

1.02

20

1.00

15

0

25

50

75

100

125

150

175

200

0

250

500

750

100012501500

1750

2000

iterations

iterations

7

low-rank(I=20)+epsilon

LDR: 1

1.05

low-rank(l=10) + epsilon

circulant

LDR: 2

LDR: 3

6

LDR: 4

1.04

LDR: 5

5

LDR: 6

LDR: 7

1.03

error

LDR: 8

error

4

1.02

3

1.01

2

1.00

1

0

25

50

75

100

125

150

175

200

0

250

500

750

100012501500

1750

2000

iterations

iterations

low-intrinsic-rank+

LDR: 1

low-intrinsic-rank

circulant

epsilon

LDR: 2

40

LDR: 3

1.8

LDR: 4

LDR: 5

LDR: 6

1.6

30

LDR: 7

LDR: 8

error

error

1.4

20

1.2

10

1.0

0

25

50

75

100

125

150

175

200

0

250

500

750

1000

1250

1500

1750

2000

iterations

iterations

Approximating Symmetric Positive Definite Matrices . We use a PSD matrix M ∈ R 50 × 50 with L 2 -normalized rows in this experiment. We compare the errors to approximate M using circulant, (symmetric) Toeplitz, low-rank matrices, and Kronecker product of two matrices. We use a fixed parameter budget and repeat this experiment 10 times. We report the results in Figure 1 (left). For each of these 10 trials, we observe that the circulant and Toeplitz achieve the lowest error, therefore the best approximation quality (see more details in Appendix A.7).

Fitting a Toy Dataset . We create a synthetic pinwheel dataset with 5 spokes as shown in Figure 9 (left). We fit this dataset using a simple neural network with one hidden layer with a matrix W ∈ R 64 × 64 . In this experiment, we replace W with a rank 1 LoRA, a circulant, a symmetric Toeplitz, and a Toeplitz matrix ( with all parameterizations having the same number of training parameters). In Fig 4), we report the training loss curves of this experiment. We observe that the LoRA layer struggles to fit the data whereas the LDRMs show similar performance to full fine-tuning (Fig 4). These results show the impressive expressive power of these matrices. Therefore, we conclude that LDRMs with particularly low displacement rank serve as good approximators for various matrices. We perform additional experiments and provide more details in Appendix B.

5 Integration of SURMs with PEFT

Motivated by the results from the previous section, we use SURMs as drop-in replacements for various PEFT methods. In this section, we present the integration of SURMs in two popular classes of PEFT methods: LoRA and Adapters.

5.1 Integration of SURMs in LoRA

LoRA [26] uses a low rank matrix to parameterize the weight matrix updates. Formally, given a pre-trained weight matrix W , then the updated matrix is ̂ W = W + α ∆ W , where ∆ W = AB /latticetop and A ∈ R m × r , B ∈ R n × r for r /lessmuch min ( m,n ) and α is a fixed scaling parameter. For efficient training, ∆ W needs to be initialized as a zero-matrix. LoRA performs this by choosing initializing A to be the zero matrix and B to be a random matrix. In this work, we propose to parameterize ∆ W using structured unrestricted rank matrices. Next, we provide the details of parameterizing ∆ W using different SURM matrices (assuming m = n for simplicity).

Circulant Matrices . In this setting, we parameterize the updates as: W = C 1 /circledot C 2 , where C i ∈ R n × n are circulant matrices encoded using a n -dimensional vector, r i ∈ R n . We use Hadamard products ( C 1 /circledot C 2 ) instead of conventional matrix products as it can be computed efficiently. The construction of Hadamard products which is O( n ) is quicker than the process involved in efficient multiplication (which is O ( n log ( n )) . To enable zero-initialization, we initialize C 1 as a zero-vector and C 2 as a random-vector. Additionally, this approach does not compromise the expressiveness of the network, as the result of the Hadamard product is also a circulant matrix.

Toeplitz Matrices . Similar to the previous setting, we use two Toeplitz matrices to parameterize: ∆ W = g ( T 2 , g ( T 1 , x )) , where T 1 , T 2 ∈ are Toeplitz matrices, and g is the operator that allows efficient matrix-vector multiplication with Toeplitz matrices (see Appendix A.3 ). Each Toeplitz matrix T ∈ R n × n is parameterized using an n -dimensional vector r encoding its first row and an n -dimensional vector c encoding its first column ( 2 n -1 total parameters). This formulation leads to the 4 n -2 trainable parameters. To further reduce this number, we constrain the T 1 , T 2 to be symmetric, reducing the total number of trainable parameters to 2 n . To enable zero-initialization, we initialize T 1 as a zero matrix and T 2 is randomly initialized.

Kronecker Product of Matrices . In this setting, we parameterize: ∆ W = A ⊗ B , where A ∈ R r 1 × r 2 , B ∈ R n r 1 × n r 2 . The hyperparameters r 1 , r 2 allow us to control the trainable parameter count and the rank of ∆ W . In contrast to low-rank matrix updates, we can create matrices ∆ W of fairly large ranks while keeping the number of trainable parameters small (see Appendix A.2). To enable zero-initialization, we set A as a zero matrix and B as a random matrix.

CIFAR-10

CIFAR-100

SUN3

9

7

DTD

STL10

Accuracy (%)

Fraction of Training Data

In all the above settings, it is possible to increase the number of training parameters by relaxing the structure of the matrix, ∆ W . This can be performed by introducing more matrices in the product chains, utilizing asymmetric Toeplitz matrices, adjusting the sizes of factors in the Kronecker product, or employing sums of such matrices. Another way to enhance layer expressiveness is by experimenting with combinations of different LDRMs, such as mixing circulant and skew-circulant matrices. A broad class of matrices, including low-rank ones, can be represented as sums of these matrices (see Theorem A.2 and the subsequent discussion).

5.2 Integration of SURMs in Adapters

Adapters [25] are small bottleneck networks into Transformer layers as shown below:

where σ ( · ) is a non-linear activation function, X ∈ R b × s × n represents input to the layer ( b : batch size, s : sequence length), A ∈ R r × n , B ∈ R n × r are low-rank matrices ( r /lessmuch n ) and Y is the output of the layer. For simplicity, layer norms and bias terms are not included in the equation. We use SURMs as a drop-in replacement for matrices A and B . Next, we will provide details of integrating SURMs within the adapter setting. Additional details are provided in Appendix A.5.

Circulant Matrices . In this setting, we apply two circulant matrices C 1 , C 2 (encoded by r 1 , r 2 ), resulting in the adapter block: Y = X + σ ( f ( r 1 /circledot r 2 , X ))+ b , where f is an operator that efficiently computes the matrix multiplication between input X and the circulant matrix encoded by the vector r 1 ◦ r 2 (Appendix A.3). To enable zero-initialization, the vector r 1 is initialized randomly while r 2 and b are initialized as zero vectors.

Toeplitz Matrices . In this setting, we use two symmetric Toeplitz matrices T 1 , T 2 , where T 1 and b is initialized as a zero vector and T 2 is initialized randomly. We then define the adapter layer as: Y = X + σ ( g ( T 1 , g ( T 2 , X ))) + b , where g is an operator that efficiently computes the matrix multiplication between an input X and a Toeplitz matrix.

Number of Parameters

Full Fine-tuning

Attention

Tuning

Transformer

Probing

Adapter

LoRA

KAdaptation

Lin. Probing

LePE

BitFit

LoRA-Fix

LN Tuning

SURM

Number of Parameters

Performance

AA

Parallel

MAM

Serial

LoRA

Prefix

AdaMix

BitFit

AutoPEFT

UniPEFT

Compacter

Full fine-tuning

Adapter

(Pfeiffer)

Adapter

(Houlsby)

SURM

(Adapter)

Number of Parameters

Performance

GLUE

Vision

Kronecker Product of Matrices . In this case, we rewrite Equation 4 as: Y = X + σ ( X ( B ⊗ A ))+ b , where B ⊗ A is the Kronecker product. In this case, B is initialized randomly and A and b are initialized by zeros. In all experiments using SURM-adapters, σ ( · ) is the GeLU non-linearity.

6 Experiments

In this section, we show the effectiveness of our proposed methods in a wide range of vision and NLP tasks through extensive empirical studies.

Image Classification Experiments . We evaluate SURM on several vision datasets: CIFAR10, CIFAR100 [39], SUN397 [79], DTD [16] and STL10 [17]. We experiment using ViT-B/16 [36] & Clip-ViT-B/16 [63] as base models and inject trainable parameters Q , K , V matrices in the LoRA setting. We report the results using ViTbase are presented in Table 1 (left). We observe that SURM consistently outperforms 12 baseline methods (that use up to 10x parameters). On three out of the five tasks, SURMs emerge as the top performers, surpassing LoRA by a margin of up to 5 -7 % while achieving competitive performance on the remaining tasks. We report the results using Clip-ViT in Table 1 (right). In this setting, SURM is among the top two methods across all 5 tasks. SURM also uses fewer trainable parameters, reducing them by 3.65x compared to LoRA and 2.4x compared to LoRA-Fix.

Low Data Regime . We evaluate SURM in low data regime using VTAB-1k datasets [85] and the ViT model. VTAB-1k is a diverse collection of vision datasets with only 1000 training examples. We focus on the NATURAL and SPECIALIZED subsets of VTAB. In Table 2, we observe that SURMs are among the top 2 methods on 10 datasets while being competitive on the remaining tasks.

We evaluate different variants of SURM and train it on a varying fraction of data on 5 datasets using Clip-ViT model. We report the results in Figure 6. We observe that the circulant SURM works best in low data regime. Furthermore, SURM achieves the performance of full fine-tuning trained on the entire dataset with only a small fraction of the data. For more challenging datasets like SUN397, we achieve comparable accuracy using approximately 20% of the training data, while for datasets such as CIFAR10 and STL10, only about 2% is needed.

Large Data Regime . We perform experiments to show that SURM generalizes well in large data regimes. On ImageNet [18] and iNat2021 [72], SURM achieves performance comparable to full fine-tuning while using only 0 . 06 % of the training parameters (see detailed results in Appendix B.2).

NLP Experiments . We extensively evaluate SURM models on the GLUE benchmark [74] using BERTbase [20]. We compare with different adapter baselines and 11 other PEFT techniques. These include full-finetuning, Adapter (Houlsby) and Adapter (Pfeiffer) , among others (we report the corresponding results from [59]). BiTFit results are taken from [84] (except QQP numbers which are obtained from [9]) and the numbers for AA -adapters from [53]. Prefix, Serial, AdaMix, UniPELT,

Parallel, MAM, and AutoPEFT numbers are taken from [90]. The results for the remaining baselines are replicated by us. More experimental details can be found in Appendix B.

For brevity, we summarize the average performance across 8 tasks for SURM-adapters and compare it to 11 baselines, in Fig 7 (right). We observe that SURM achieve much better performance while using a fraction of the parameters (the complete results are reported in Appendix Table 5). We also observe that SURM (integrated into LoRA) outperforms the baseline LoRA, under the same parameter budget. This shows the effectiveness of using structured matrices as a drop-in replacement for low rank matrices used in LoRA. We further analyze the representations learnt by SURMs and LoRAs (Appendix B.4). We find that LoRA learns weights that are very similar to the pre-trained weights whereas SURM is able to explore a larger parameter space (an observation similar to [91]).

Large-scale Experiments . In this setting, we integrate SURMs in LLMs. Specifically, we use matrices of the form W ( G , H ) (as described in Eqn 2) to increase the number of training parameters. We use the experimental setup introduced in [29], where the LLM tries to fit a dataset of UUID pairs using the Llama-2-7B model [69]. This was shown to be a challenging task (UUID prediction is significantly different from the pre-training tasks) that requires higher rank values in LoRA. We report the results in Fig. 8, demonstrating that SURMs is able to fit the data, whereas other methods struggle to do so (see more details in Appendix B.1).

Image Segmentation . Next, we focus on the extremely challenging task of medical image segmentation using Synapse multi-organ segmentation

0

20

40

60

80

100

120

Iterations

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Training Loss

MoRA

LoRA

SURM (w/ Circulant)

SURM (w/

W

(

G

,

H

))

Figure 8: Fitting the UUID dataset using Llama-2-7b. We fit the data using various SURM-based PEFT methods and LoRA.

dataset [82]. Segment-Anything-Model (SAM) [34] is used as the foundation model for this task. We follow [86] and adapt the Q , V in ViTbase image encoder in SAM. Finally, in this low data regime, we use Circulant variant of SURM as it is the best performing variant (Fig. 6). We report the Dice similarity coefficient (DSC) metric for each of the 8 organ segmentations and their average (higher is better). For a fair comparison, we include LoRA with rank 1 , matching the exact parameter count of Circulant. The results are presented in Table 3. We report the baseline performance from [86]. SURMs compare favorably with specialized architectures developed for medical imaging like U-Net, Attention U-Net, Transformer-based U-Net, and the Swin U-Net even though they have significantly higher number of training parameters than our method (see details in Appendix B.1).

7 Conclusion

We introduce structured unrestricted-rank matrices (SURMs) as an alternative to low-rank matrices for the parameter-efficient fine-tuning of large Transformer models. In this setting, structured matrices form the cornerstone of a comprehensive framework, offering a solid base for various parameter efficient fine-tuning methods, such as adapters and LoRA, with enhanced efficiency. SURMs improve the overall effectiveness of PEFT, contributing to its efficient integration into diverse models and domains. Based on extensive numerical experiments and theoretical insights, we conclude that the Circulant variant is our most performing variant (in terms of speed and accuracy).

8 Author Contributions

AS designed the integration of SURM in Adapters and LoRA and ran the GLUE experiments. AD helped in developing the integration and ran all image experiments. KC came up with the idea of using LDRMs in the context of PEFT. SBRC helped in running various large-scale experiments and writing the manuscript. All authors contributed to the writing of this manuscript.

References

Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 7319-7328, Online, August 2021. Association for Computational Linguistics.

Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. The falcon series of open language models. arXiv preprint arXiv:2311.16867 , 2023.

Michele Benzi and Valeria Simoncini. Approximation of functions of large matrices with kronecker structure. Numerische Mathematik , 135(1):1-26, 2017.

Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning , pages 2397-2430. PMLR, 2023.

James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018.

Richard P Brent. Stability of fast algorithms for structured linear systems. Fast reliable algorithms for matrices with structure , pages 103-116, 1999.

Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817 , 2023.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 , 2020.

Jin Cao, Chandana Satya Prakash, and Wael Hamza. Attention fusion: a light yet efficient late fusion mechanism for task adaptation in NLU. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz, editors, Findings of the Association for Computational Linguistics: NAACL 2022 , pages 857-866, Seattle, United States, July 2022. Association for Computational Linguistics.

Arnav Chavan, Zhuang Liu, Deepak Gupta, Eric Xing, and Zhiqiang Shen. One-for-all: Generalized lora for parameter-efficient fine-tuning. arXiv preprint arXiv: 2306.07967 , 2023.

Banghao Chen, Zhaofeng Zhang, Nicolas Langrené, and Shengxin Zhu. Unleashing the potential of prompt engineering in large language models: a comprehensive review. arXiv preprint arXiv:2310.14735 , 2023.

Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. arXiv preprint arXiv:2205.13535 , 2022.

Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-context large language models. In The Twelfth International Conference on Learning Representations , 2024.

Zhangchi Chen. On nonsingularity of circulant matrices. Linear Algebra and its Applications , 612:162-176, March 2021.

Y. Cheng, F. X. Yu, R. S. Feris, S. Kumar, A. Choudhary, and S. Chang. An exploration of parameter redundancy in deep networks with circulant projections. In 2015 IEEE International Conference on Computer Vision (ICCV) , pages 2857-2865, Los Alamitos, CA, USA, dec 2015. IEEE Computer Society.

M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) , 2014.

Adam Coates, Andrew Ng, and Honglak Lee. An Analysis of Single Layer Networks in Unsupervised Feature Learning. In AISTATS , 2011. https://cs.stanford.edu/~acoates/ papers/coatesleeng_aistats_2011.pdf .

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 248-255, 2009.

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems , 36, 2024.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.

Ali Edalati, Marzieh Tahaei, Ivan Kobyzev, Vahid Partovi Nia, James J Clark, and Mehdi Rezagholizadeh. Krona: Parameter efficient tuning with kronecker adapter. arXiv preprint arXiv:2212.10650 , 2022.

Jindong Gu, Zhen Han, Shuo Chen, Ahmad Beirami, Bailan He, Gengyuan Zhang, Ruotong Liao, Yao Qin, Volker Tresp, and Philip Torr. A systematic survey of prompt engineering on vision-language foundation models. arXiv preprint arXiv:2307.12980 , 2023.

Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning. In International Conference on Learning Representations , 2022.

Xuehai He, Chunyuan Li, Pengchuan Zhang, Jianwei Yang, and Xin Eric Wang. Parameterefficient model adaptation for vision transformers. arXiv preprint arXiv:2203.16329 , 2022.

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 2790-2799. PMLR, 09-15 Jun 2019.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations , 2022.

Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision (ECCV) , 2022.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825 , 2023.

Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. Mora: High-rank updating for parameter-efficient fine-tuning. arXiv preprint arXiv:2405.12130 , 2024.

Shibo Jie and Zhi-Hong Deng. Fact: Factor-tuning for lightweight adaptation on vision transformer. In Proceedings of AAAI Conference on Artificial Intelligence (AAAI) , 2023.

Damjan Kalajdzievski. A rank stabilization scaling factor for fine-tuning with lora. arXiv preprint arXiv:2312.03732 , 2023.

Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks. In Annual Meeting of the Association for Computational Linguistics , 2021.

Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4015-4026, 2023.

Matthias Kissel and Klaus Diepold. Structured matrices and their application in neural networks: A survey. New Generation Computing , 41(3):697-722, Sep 2023.

Alexander Kolesnikov, Alexey Dosovitskiy, Dirk Weissenborn, Georg Heigold, Jakob Uszkoreit, Lucas Beyer, Matthias Minderer, Mostafa Dehghani, Neil Houlsby, Sylvain Gelly, Thomas Unterthiner, and Xiaohua Zhai. An image is worth 16x16 words: Transformers for image recognition at scale. In Ninth International Conference on Learning Representations . ICLR, 2021.

Soroush Abbasi Koohpayegani, Navaneet K L, Parsa Nooralinejad, Soheil Kolouri, and Hamed Pirsiavash. NOLA: Compressing loRA using linear combination of random basis. In The Twelfth International Conference on Learning Representations , 2024.

Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning , pages 3519-3529. PMLR, 2019.

Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). URL http://www. cs. toronto. edu/kriz/cifar. html , 5(4):1, 2010.

George Labahn and Tamir Shalom. Inversion of toeplitz matrices with only two standard equations. Linear Algebra and its Applications , 175:143-158, 1992.

Daniele Lazzaro and Stefania Morigi. Matrix completion for matrices with low-rank displacement. Electronic Transactions on Numerical Analysis , 53:481-499, 2020.

Jaejun Lee, Raphael Tang, and Jimmy Lin. What would elsa do? freezing layers during transformer fine-tuning. arXiv preprint arXiv:1911.03090 , 2019.

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 4582-4597, Online, August 2021. Association for Computational Linguistics.

Yuanzhi Li, Yingyu Liang, and Andrej Risteski. Recovery guarantee of weighted low-rank approximation via alternating minimization. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning , volume 48 of Proceedings of Machine Learning Research , pages 2358-2367, New York, New York, USA, 20-22 Jun 2016. PMLR.

Vladislav Lialin, Vijeta Deshpande, and Anna Rumshisky. Scaling down to scale up: A guide to parameter-efficient fine-tuning. arXiv preprint arXiv: 2303.15647 , 2023.

Shih-yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning , 2024.

I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 8086-8098, 2022.

Yuchen Lu, Zhen Liu, Aristide Baratin, Romain Laroche, Aaron Courville, and Alessandro Sordoni. Using representation expressiveness and learnability to evaluate self-supervised learning methods. Transactions on Machine Learning Research , 2023.

Gen Luo, Minglang Huang, Yiyi Zhou, Xiaoshuai Sun, Guannan Jiang, Zhiyu Wang, and Rongrong Ji. Towards efficient visual adaption via structural re-parameterization. arXiv preprint arXiv:2302.08106 , 2023.

Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. arXiv preprint arXiv: 2106.04647 , 2021.

Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma, Jiawei Han, Scott Yih, and Madian Khabsa. UniPELT: A unified framework for parameter-efficient language model tuning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 6253-6264, 2022.

Nafise Moosavi, Quentin Delfosse, Kristian Kersting, and Iryna Gurevych. Adaptable adapters. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 3742-3753, Seattle, United States, July 2022. Association for Computational Linguistics.

Bernard Mourrain and Victor Y. Pan. Multivariate polynomials, duality, and structured matrices. J. Complex. , 16(1):110-180, 2000.

Henri J Nussbaumer and Henri J Nussbaumer. The fast Fourier transform . Springer, 1982.

Vadim Olshevsky and Mohammad Amin Shokrollahi. Matrix-vector product for confluent cauchy-like matrices with application to confluent rational interpolation. In F. Frances Yao and Eugene M. Luks, editors, Proceedings of the Thirty-Second Annual ACM Symposium on Theory of Computing, May 21-23, 2000, Portland, OR, USA , pages 573-581. ACM, 2000.

Samet Oymak, Zalan Fabian, Mingchen Li, and Mahdi Soltanolkotabi. Generalization guarantees for neural networks via harnessing the low-rank structure of the jacobian. arXiv preprint arXiv:1906.05392 , 2019.

Victor Y. Pan. Structured matrices and polynomials: unified superfast algorithms . SpringerVerlag, Berlin, Heidelberg, 2001.

Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vuli´ c, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. Adapterhub: A framework for adapting transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020): Systems Demonstrations , pages 46-54, Online, 2020. Association for Computational Linguistics.

Phil Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein. The intrinsic dimension of images and its impact on learning. In International Conference on Learning Representations , 2021.

Daniel Povey, Gaofeng Cheng, Yiming Wang, Ke Li, Hainan Xu, Mahsa Yarmohammadi, and Sanjeev Khudanpur. Semi-orthogonal low-rank matrix factorization for deep neural networks. In Proc. Interspeech 2018 , pages 3743-3747, 2018.

Shikai Qiu, Andres Potapczynski, Marc Anton Finzi, Micah Goldblum, and Andrew Gordon Wilson. Compute better spent: Replacing dense layers with structured matrices. In Forty-first International Conference on Machine Learning , 2024.

Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML , 2021.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 21(140):1-67, 2020.

Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. Adapterdrop: On the efficiency of adapters in transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 7930-7946, 2021.

Vikas Sindhwani, Tara Sainath, and Sanjiv Kumar. Structured transforms for small-footprint deep learning. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015.

Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng, et al. UL2: Unifying language learning paradigms. In The Eleventh International Conference on Learning Representations , 2023.

Anna Thomas, Albert Gu, Tri Dao, Atri Rudra, and Christopher Ré. Learning compressed transforms with low displacement rank. Advances in neural information processing systems , 31, 2018.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Lucrezia Valeriani, Diego Doimo, Francesca Cuturello, Alessandro Laio, Alessio Ansuini, and Alberto Cazzaniga. The geometry of hidden representations of large transformer models. arXiv preprint arXiv:2302.00294 , 2023.

Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. DyLoRA: Parameterefficient tuning of pre-trained models using dynamic search-free low-rank adaptation. In Andreas Vlachos and Isabelle Augenstein, editors, Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics , pages 3274-3287, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics.

Grant Van Horn, Elijah Cole, Sara Beery, Kimberly Wilber, Serge Belongie, and Oisin Mac Aodha. Benchmarking representation learning for natural world image collections. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12884-12893, 2021.

Ailong Zheng Victor Y. Pan. Superfast algorithms for cauchy-like matrix computations and extensions. Linear Algebra and its Applications , 310(1-3):83-108, 2000.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP , pages 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics.

Qizhou Wang, Sarah M Erfani, Christopher Leckie, and Michael E Houle. A dimensionalitydriven approach for unsupervised out-of-distribution detection. In Proceedings of the 2021 SIAM International Conference on Data Mining (SDM) , pages 118-126. SIAM, 2021.

Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. AdaMix: Mixture-of-adaptations for parameter-efficient model tuning. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 5744-5760, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Transactions on Machine Learning Research , 2022.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-theart natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 38-45, Online, October 2020. Association for Computational Linguistics.

J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition , pages 3485-3492, June 2010.

Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment. arXiv preprint arXiv:2312.12148 , 2023.

Peng Xu, Xiatian Zhu, and David A Clifton. Multimodal learning with transformers: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(10):12113-12132, 2023.

Z Xu. Multi-atlas labeling beyond the cranial vault-workshop and challenge. Synapse website , 2016.

Bruce X. B. Yu, Jianlong Chang, Lin Liu, Qi Tian, and Changan Chen. Towards a unified view on visual parameter-efficient transfer learning. ArXiv , abs/2210.00788, 2022.

Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages 1-9, 2022.

Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.04867 , 2019.

Kaidong Zhang and Dong Liu. Customized segment anything model for medical image segmentation. arXiv preprint arXiv:2304.13785 , 2023.

Meng Zhang, Fei Liu, and Dongpeng Weng. Speeding-up and compression convolutional neural networks by low-rank decomposition without fine-tuning. J. Real-Time Image Process. , 20(4), may 2023.

Ningyu Zhang, Luoqiu Li, Xiang Chen, Shumin Deng, Zhen Bi, Chuanqi Tan, Fei Huang, and Huajun Chen. Differentiable prompt makes pre-trained language models better few-shot learners. In International Conference on Learning Representations , 2022.

Liang Zhao, Siyu Liao, Yanzhi Wang, Zhe Li, Jian Tang, and Bo Yuan. Theoretical properties for neural networks with weight matrices of low displacement rank. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 4082-4090. PMLR, 06-11 Aug 2017.

Han Zhou, Xingchen Wan, Ivan Vuli´ c, and Anna Korhonen. Autopeft: Automatic configuration search for parameter-efficient fine-tuning. arXiv preprint arXiv:2301.12132 , 2023.

Bojia Zi, Xianbiao Qi, Lingzhi Wang, Jianan Wang, Kam-Fai Wong, and Lei Zhang. Deltalora: Fine-tuning high-rank parameters with the delta of low-rank matrices. arXiv preprint arXiv:2309.02411 , 2023.

A Implementation Details

In this section, we discuss the details of various algorithms and workflows within SURM.

Contents

A.1 Skew-Circulant Matrices

In this section, we introduce another type of structured matrix that is characterized by a linear number of parameters.

Definition A.1 (Skew-Circulant) . Amatrix S = ( s jk ) n -1 j,k =0 is said to be skew-circulant if s jk = s j -k and s -l = -s n -l for 1 ≤ l ≤ n -1 .

This matrix can be represented visually as shown below:

This matrix is parameterized by a linear number of parameters and also enjoy sub-quadratic time complexity matrix-vector multiplications (see Appendix A.3).

A.2 Finding the Smallest Number of Training Parameters for Kronecker Layers

Let W be a d × d matrix that can be written as W = A ⊗ B , where A ∈ R m 1 × n 1 , B ∈ R m 2 × n 2 . We want to minimize the following objective:

We can rewrite the above as : m = d/m and n = d/n . Plugging these back in Eq. 6, we get:

The equality is obtained when √ m 2 n 2 = d/ √ m 2 n 2 , thereby satisfying the constraint m 2 n 2 = m 1 n 1 = d . Essentially this result shows that we can minimize the number of training parameters when the matrices A and B are similarly sized. Furthermore, since both A (and B ) have d training parameters, we can maximize the rank of the matrix if we can make it close to a square matrix (i.e. we choose 2 factors a, b of d , such ab = d and a is as close to b as possible). Note that rank ( A ⊗ B ) = rank ( A ) rank ( B ) . Thus, for our experiments with BERT and ViT models, we take A to be a matrix of size 32 × 24 and B to be of size 24 × 32 . This choice of matrix shapes allows us to substantially reduce the computational complexity of matrix-vector multiplication (see Section A.3).

A.3 Efficient Matrix Vector Multiplication by Structured Matrices

One of our main advantages of using structured matrices is that they allow for sub-quadratic vectormatrix multiplications. Matrix vector multiplication by a circulant matrix can be efficiently done via FFT in O ( n log n ) time. This is done by the following steps : (a) take the FFT of the input vector v and the vector representation of the circulant matrix c , and call them V and C respectively. (b) Take the inverse Fourier transform of the Hadamard (element-wise) product of V and C .

For the sake of convenience, let us define this efficient multiplication operator to be f . The key insight behind this approach is that the circular convolution in the time domain corresponds to element-wise multiplication in the frequency domain after FFT. By leveraging FFT, the time complexity of the multiplication is reduced from O ( n 2 ) to O ( n log n ) .

The same ideas extend to the case of Toeplitz matrices, where one can embed the Toeplitz matrix into a circulant matrix and use FFT as before for efficient matrix-vector multiplication. For ease of reference, let us call the function g that embeds the Toeplitz matrix into a circulant matrix and use the function f as described above to compute the matrix-vector product.

Next, we describe how vector multiplication by skew-circulant matrices can be efficiently performed in O ( n log n ) time. If the skew-circulant matrix S is parameterized by the vector v , then the multiplication is given by

where η = [1 , η, η 2 , · · · η n -1 ] , and η = ( -1) 1 n = exp( iπ/n ) , the root of negative unity.

For the case of a matrix W = AB , where W ∈ R m × n , A ∈ R m × r and B ∈ R r × n , then multiplication by v takes O ( r ( m + n )) and one gets computation gains when r /lessmuch min { m,n } . Finally, for a Hadamard product of matrices A ∈ R r 1 × r 2 , B ∈ R k 1 × k 2 , v ∈ R r 2 k 2 , ( A ⊗ B ) v = vec ( B r ( v ) /latticetop A /latticetop ) , where vec ( · ) is the vectorization operator that takes a matrix M ∈ R m × n and converts it to R mn × 1 column vector by stacking the columns of M on top of each other and r is the PyTorch style reshape operator that reshapes the vector v to a matrix of shape r 2 × k 2 . Choosing max { r i , k i } /lessmuch r i k i , for i = 1 , 2 , one can substantially reduce the computational complexity.

A.4 Increasing Number of Training Parameters

In this section, we explain an elegant way to increase the number of training parameters. We use the sum of product of circulant and skew-circulant matrices of the form

where A i and B i is a circulant and a skew-circulant matrix respectively. Each factor of M has 2 n parameters thus M has 2 nr parameters.

The class of n × n matrices M which can be written via Equation 9 is rich and contains many important classes of matrices.

Theorem A.2 (Expressivity) . The set of matrices M which can be written as in Equation 9 contains:

All n × n Circulant and Skew-Circulant matrices for r ≥ 1

All n × n Toeplitz and Inverses of Toeplitz matrices for r ≥ 2 .

All n × n matrices for r = n .

All linear combinations of the form ∑ p j =1 β i A ( j ) 1 · · · A ( j ) t where r ≥ 2 tp , and A is either a Toeplitz or the inverse of a Toeplitz matrix.

This is Theorem 3.1 in [66]. Efficient multiplication by matrices of this form can be done in sub-quadratic time by simply combining the results from Sec A.3.

Moreover, we note that by choosing a slightly different parameterization of displacement operators, one can obtain low rank matrices and orthogonal polynomial transforms, including the Discrete Fourier and Cosine Transforms (see Proposition 2 in [68]).

Thus our framework encompasses many important classes of matrices including low rank matrices and thus generalizes LoRA.

A.5 Integration of SURMs in Adapters

In this section, we provide additional details about SURMintegration in Adapters.

For simplicity, we follow the Houlsby configuration [25], but our work is also readily applicable in Pfeiffer configuration as well [59]. Recall the definition of Adapter layers:

where σ ( · ) is a non-linear activation function applied point-wise, X ∈ R b × s × n represents input to the layer ( b is the batch size and s is the sequence length), A ∈ R r × n , B ∈ R n × r are two low-rank matrices ( r /lessmuch n ) and Y is the output of the layer. Similar to LoRA, matrix B is initialized randomly, whereas A is initialized as a zero-matrix. For convenience, layer norms and bias terms are not included in the equation.

SURMs can be used in place of low rank A and B . The integration and design choices of various LDRs in this setting mimic that of LoRA.

Circulant Matrices . Similar to the LoRA setting, we apply two circulant matrices C 1 , C 2 , resulting in the following equation of the adapter block:

where f is an operator multiplying input matrix X with the circulant matrix obtained by multiplying two circulant matrices encoded by r 1 and r 2 (Appendix A.3). The vector r 1 is initialized randomly while r 2 and b are initialized as zero vectors. Note that we apply the non-linearity after we multiply X with both the circulant matrices. This may hurt the expressiveness of the network but improves computational complexity. Moreover, we only need to save one vector defining the first row of a circulant matrix and not both: r 1 and r 2 . This results in lower storage costs and faster deployment. This design choice works well in practice as evidenced from the results on the GLUE benchmark (see Table 5).

Toeplitz Matrices . Similar to the case of Toeplitz matrices within LoRA, we use two symmetric Toeplitz matrices T 1 , T 2 , where T 1 and b is initialized as a zero vector and T 2 is initialized randomly. We then define the adapter layer to be:

The position of the non-linear mapping σ is chosen such that we can merge the two trained matrices resulting in smaller storage costs and fast deployment.

Finally, note that the Toeplitz variant is slower than the circulant variant, as it requires two applications of the fast matrix-vector operator, whereas the circulant variant requires only one.

A.6 Computing Approximations Using LDR Matrices

In this section, we show how we can approximate any matrix D ∈ R n × n using Circulant, Toeplitz matrices, and symmetric Toeplitz matrices. We note that each class of structured matrices forms a vector space. Therefore, finding the closest point in the appropriate subspace becomes a convex optimization problem and is given by the orthogonal projection onto the basis vectors of the subspace. More explicitly, if { e 1 , · · · , e n } are a set of orthogonal vectors spanning a subspace W , then the closest vector to v in W is given by

The space of circulant matrices has dim n , so spanned by the orthogonal set { (1 , · · · , · · · 0) , (0 , · · · 1 , · · · 0) , (0 , · · · , · · · 1) } . Using the above formula, one can write down a simplified expression of the circulant matrix as ˆ C := ( ˆ c 1 , · · · ˆ c n ) that approximates D

Note that the same set as before spans the space of symmetric Toeplitz matrices. This yields a compact formula for the approximating Toeplitz matrix:

where M i is the symmetric Toeplitz matrix generated by the i -th element in the set above. Finally the set { ((1 , 0 , · · · , 0) , (0 , · · · , 0)) , · · · ((0 , · · · , 1 , · · · , 0) , (0 , · · · , 0)) , ((0 , · · · , 0) , (0 , · · · , 1 , · · · , 0) } spans all Toeplitz matrices where the first element in each tuple denotes the first row and the second element the first column. Note that since the a 1 1 entry is shared by both first row and column we treat the first vector as n -dimensional vector and the second as n -1 dimensional vector. Thus the dimension of the space is 2 n -1 . Using FFT and the projection formula, one can compute the approximation by a Toeplitz matrix.

A.7 Additional Details on Approximation Errors by LDR

In this section, we present additional details on the various experiments on approximation by LDR matrices presented in Section 4.1.

Random: The first class, with entries taken independently at random from N (0 , 1) , represents a completely unstructured family.

Near-low rank: Each matrix from the second class was chosen from the distribution: GH /latticetop + /epsilon1 R , where G , H ∈ R n × r for r /lessmuch n , R ∈ R n × n , /epsilon1 = 0 . 05 , and the entries of G , H , R are taken independently at random from N (0 , 1) .

Near-low intrinsic rank: Matrices from the third class are constructed as follows. First we sample: t 0 , ..., t n -1 iid ∼ N (0 , 1) . The i -th row of the resulting matrix is of the form: (sin(1 · t i ) , sin(2 · t i ) , ..., sin( n · t i )) + g i , wheare either all g i are zero-vectors or they are taken independently at random from /epsilon1 ∗N (0 , I n ) . Note that even though that matrix is not necessarily low-rank, it is taken from the vicinity of the n -dimensional manifold, since it is fully determined by the sampled tuple ( t 0 , ..., t n -1 ) . Matrices from all the classes are taken from R 100 × 100 .

Optimizing Circulant and Toeplitz Matrices . In general, an optimal approximation (e.g. with respect to the Frobenius norm as a distance) of a given matrix by a matrix W ( G , H ) is not given by the closed-form expression. Thus we will thus construct good-quality approximators via gradientbased optimization (see: Sec. 4.1).

Details on Approximation Experiments in Section 4.2 . Now we provide additional details on the experiments that explicitly compare LDRMs with low-rank matrices. For these experiments, we construct a PSD matrices M ∈ R 50 × 50 with L 2 normalized rows. We fix a parameter budget of n = 50 . The low-rank approximation, in that case, becomes an outer product by a vector v . For the Kronecker product, we choose a factor A ∈ R 10 × 5 . To maintain the parameter budget, the other factor becomes A /latticetop . If ˆ M is the approximating matrix, then we define error = || ˆ M -M || F , where || · || F is the Frobenius norm.

We use the closed-form formula for the optimal circulant and symmetric Toeplitz matrices approximating M and use gradient descent to find the optimal low-rank matrix and Kronecker product of matrices. We use a learning rate of 0 . 1 while computing the optimal low-rank matrix and the Kronecker product of matrices.

A.8 Invertible Toeplitz Matrices

Inverses of Toeplitz matrices can be effectively found [40]. We recall the celebrated result of Gohberg and Semencul.

Theorem A.3. Let A := ( a p -q ) n p,q =1 be a Toeplitz matrix. If the following systems of equations

/negationslash is solvable and x 1 = 0 , then A is invertible.

20

Baseline

15

101

LoRA

Circulant

Toeplitz

10

5

0

100

-5

-10

0

1

15

2

3

-20

4

10-1

-20

-10

0

10

20

0

250

500

750

1000

1250

1500

1750

2000

In our case, we consider only symmetric Toeplitz matrices. Thus the above equation really boils down to solving the first system of equations as the next system can be solved by using the first, i.e. by setting x n -i +1 = y i i = 1 , 2 , · · · n . The first system of equations can be efficiently solved by Gaussian elimination.

B Experiments

In this section, we describe our experimental setup and present additional analysis experiments to evaluate the functioning of SURM. Our code is available at https://github.com/arijitthegame/ structured-matrices-PEFT .

Contents

B.1 Hyperparameters

In this section, we provide the details of the hyperparameters used in our experiments. For GLUE tasks, we use the LORA hyperparameters that are used in the original LoRA paper except we use r = 1 to parameter match our methods as well as α = 1 .

For all the experiments, we use AdamW optimizer [47] with a warmup ratio of 0 . 06 , a linear learning rate scheduler, and a sequence length of 128 . For our methods and the Compacter baseline, we use a batch size of 64 . We report the rest of the hyperparameters in Table 4. The code to run NLP experiments is developed using PyTorch using Huggingface, Adapter-transformer, PEFT libraries, and the original LoRA codebase. For ViT experiments, we use JaX [5] and the open-sourced JAX implementation of ViT.

Additional Details on the Pinwheel Experiment . First, we provide a figure of the pinwheel dataset used to showcase the approximation qualities to LDRMs (see Fig 9 left). We provide additional details on the pinwheel experiment. We tried out 2 settings : (a) simple neural network training for 2000 epochs, (b) the embedding (bottom) layer is frozen and the rest of the network is trained for 2000 epochs. This can be thought of fitting a feature extractor on top of a randomized projection. The setting (b) is presented in the main paper while setting (a) is presented in Appendix B.2. Next, we provide additional details for our text and vision experiments.

NLP Experiments . We train the LoRA-BERT using PEFT library from Huggingface [78]. The hyperparameters used by the original authors are used in this setting. For experiments comparing with the LoRA baseline, we parameter match the LoRA updates with our SURMs, thus the LoRA updates are given by rank 1 matrices. we inject the LoRA modules in query, key, value projection matrices and also show ablations where we remove the adaptation from the key matrix.

For the adapter setting, we apply the GeLU non-linearity. Kronecker-based adapter, though similar to various other methods, was never tested in the BERT-setting and thus we implement it here. And in all cases, we add an (optional) dropout on the representations coming from these adaptive layers. We train the compacter baseline using the adapter-transformers library [59]. For the compacter parameters, we use n = 4 (number of terms in the Tucker decomposition) and the reduction factor to create the low rank matrices to be 16 . All our methods have the same number of training parameters 2 d (excluding bias terms), which gives the reader a holistic overview of how these matrices perform when injected into different PEFT paradigms. All the baseline methods use a batch size of 32 , whereas our methods use a batch size of 64 . AdamW [47] optimizer is used for all experiments.

Image Classification Experiments . For the image experiments, we use Adam optimizer [33] with 20k max iterations per dataset with a batch size of 64 . The learning rate used is 5e-5 except for SVHN where we use a learning rate of 5e-4. The experiments are run on TPUv4 4 × 2 compute resources.

Large Scale Experiments . In this experiment, we investigate if large ranks are needed for learning new tasks. To circumvent the pre-trained knowledge in the Transformers, following [29], we generate random 10K pairs of Universally Unique Identifiers (UUIDs), each pair comprising two UUIDs with 32 hexadecimal values. The task requires the LLM to generate the corresponding UUID based on the input UUID. We use LLaMA-2 7B as base model [69] for this experiment. For the LoRA setting, we apply rank 256 matrices to only the linear layers in the attention layer. For MoRA, we use the same setting as in [29]. However, note that in [29], the authors apply adaptation to all linear layers.

For the SURM methods, we follow the same setting as applying the adaptation to only the attention layers. The number of factors of W ( G , H ) is chosen to be 4 (i.e. same number of parameters as a rank 4 matrix). This experiment also highlights that for effective transfer learning LoRA needs to be applied to all linear layers (which is well-known in the LLM community). In Fig. 8, we observe that LoRA and MoRA struggles to fit the data whereas our method converges.

Image Segmentation Experiments . For this experiment, we use Synapse multi-organ segmentation dataset. 30 abdominal CT scans in the MICCAI 2015 Multi-Atlas Abdomen Labeling Challenge are divided into 18 training cases and 12 test cases. There are 3779 axial contrast-enhanced abdominal CT images in total and the training set contains 2212 axial slices. All the CT volumes contain 85 ∼ 198 slices and each slice includes 512 × 512 pixels with a spatial resolution of ( [0 . 54 ∼ 0 . 54] × [0 . 98 ∼ 0 . 98] × [2 . 5 × 5 . 0] mm 3 ). We use the Segment-Anything-Model (SAM) [34] as the foundation model for this task. There has been a number of works in adapting various PEFT methods to fine tuning SAM. We follow the training details in [86]. More specifically, we adapt the Q , V in ViT-B image encoder in the SAM and normally finetune the small decoder head. Finally, in this small data regime, we use the Circulant variant as it is our most performant variant in this case (see Fig. 6). We report the Dice similarity coefficient (DSC) metric for each of the 8 organ segmentation as well as the average DSC score for all (higher is better). The SAMed model uses a LoRA rank 4 in Q , V . For a fair comparison, we include LoRA rank 1 , matching the exact parameter count of Circulant. We use an A100 40GB GPU for this experiment.

B.2 Additional Experiments

In this section, we provide additional experiments to showcase the efficacy of SURMs.

Experiments on Large Scale Data Next, we conduct additional experiments on the ImageNet-1k dataset [18]. The goal is to show how our methods can scale up to extremely large datasets. We observe that SURM achieves comparable performance to other PEFT methods and even achieving comparable performance to the full fine-tuning results (see Table 6).

We further evaluate the performance of SURM in a large-scale setting using the iNat2021 dataset [72], which contains over 2 . 7 million training images, 100K validation images, and 500K test images, spanning 10 , 000 species (classes). Fine-tuning a ViT model on this dataset achieves an accuracy of 69.98%, while SURM (circulant) achieves 69.01%. Notably, our method requires only 55K

(a)

(b)

(c)

(d)

(e)

(f)

COLA

MRPC

QNLI

65

no bias

bias

95

no bias

bias

95

nobias

bias

60

90

90

MCC

55

85

85

50

45

80

80

40

75

75

circulant

kronecker

circulant

kronecker

circulant

kronecker

RTE

SST2

STSB

96

74

no bias

no bias

95

nobias

72

bias

94

bias

Spearmans correlation

bias

90

70

92

68

90

85

66

64

88

80

62

86

60

75

circulant

kronecker

circulant

kronecker

circulant

kronecker

parameters, compared to 86 M for full fine-tuning, demonstrating its efficiency in parameter usage while maintaining comparable accuracy.

Ablation Experiments . Here, we show the effect of various design choices. Figure 10 illustrates the impact of incorporating the bias term in our adapters. Bias term provides a boost across all tasks and the adapters, the boost being smaller on the Kronecker adapter. Without the bias terms, the sizes of the adapters are around . 04 M, providing an even lightweight but still capable method. Therefore, if there are concerns regarding storage and latency, opting for adapters without bias is a viable option. Moreover, we show the effect on only adapting Q , V instead of Q , K , V as shown in the main paper. Table 7 shows that on GLUE tasks, there is a minimal effect for not adapting the K matrix.

B.3 Comparison of SURM Kronecker Adaptations with Baselines

As mentioned earlier, adaptation using Kronecker product is not new and has been investigated in several works [51, 21, 24]. In both [51] and [24], the authors use the Kronecker decomposition of the weight matrix (in the first case, the weight matrix belongs to an adapter layer and in the second case the weight matrix refers to updates as in the case of LoRA). Write W = ∑ n i =1 A i ⊗ B i . Furthermore, the authors assume that B i is low rank and can be written as B i := u ij v /latticetop ij . The weights A i can also be assumed to low weights or be shared among various layers leading to substantial efficiency gains. Our method is a simplified version of the above where n = 1 . Other main difference between the above methods and ours are : the matrices considered in the above works are square matrices whereas they are almost never square unless the dimension of the transformer is a perfect square and is set up such that the number of parameters are reduced while the rank is as high as possible, contrary to the above. Similar considerations of low rank factors in tensor decomposition are also used in [30]. Our Kronecker adaptation is same as that of [21] in the LoRA setting. In the adapter setting, our implementation follows closely the Houlsby architecture and is a little different than that of [21]. Thus, we implement the Kronecker adaptation in both LoRA and adapter settings and showcase its

0.05

0.05

Cosine Similarity

Similarity

0.04

0.04

Circulant

Circulant

0.03

★

Toeplitz

Kronecker

Cosine

0.03

Toeplitz

Kronecker

--

LORA

LORA

0.02

0.02

1

1

1

0.01

0.01

0.00

6

8

10

12

6

8

10

12

Layer Number

Layer Number

versatility across both vision and language. Moreover, we present this approach as an example of a principled approach to tackle the problem of PEFT.

B.4 Analysis of Weight Matrices in Fine-tuned Models

In this section, we analyze the weights of various fine-tuned models. Even though prior works have found the updates of the weight matrices to have low intrinsic dimension [1] (ID), the updates themselves are of high rank. This is confirmed by looking at the fine-tuned BERT models on various GLUE tasks as well as ViT models fine-tuned on CIFAR10, CIFAR100, and ImageNet. Moreover, we simulate a high-rank LoRA setting on GLUE where we freeze all the weights excepts except for Q , K , V . In that scenario, we manage to replicate the full fine-tuning performance using fewer training epochs than that of LoRA. A quick analysis of the updates reveal that they have full rank.

Many works have delved into intrinsic dimensionality for well-known image classification datasets [60]. These works show that the images have low intrinsic dimensionality compared to the pixel spaces but the dimensionality increases when augmentations like Gaussian noise is added. Recent work [49] studies the intrinsic dimensions of various self-supervised image models. Comparing their results with that of fully supervised ViT models, we observe that the self-supervised models exhibit slightly higher IDs. This is not surprising as the SSL encourages the representations to be spread over an unit hyper sphere. Thus, we believe that various low rank adaptations may fail in situations where the IDs might be high (in case of OOD data) [75].

Encouraged by this analysis, we next investigate the trained weights emerging from our methods. We observe that they have high rank across all vision and text tasks and various fine-tuning strategies. The largest possible rank of the Kronecker matrices considered in this work is 576 and all of our trained matrices are of rank 576 . For rational circulant matrices C , the non-singularity of such matrices is related to divisibility by cyclotomic polynomials. More generally, if we denote by c = ( c 0 , ..c n -1 ) the first column of C , then:

where ω j = e 2 π i j n and i 2 = -1 (for more details see [14]). This fact allows us to efficiently test for the non-singularity of the circulant matrices. In all our cases, we found our matrices to be nonsingular. Regarding Toeplitz matrices, there is a large body of literature that discusses the inversion of such matrices (see Appendix A.8). Using the methods discussed above, we find that the Toeplitz adaptations are invertible, thus full-rank. Therefore, we hypothesize that the high rank compensates for the deficiency of training parameters.

To further explore the differences between the parameters learned by LoRA vs. that learned by SURM methods we performed another set of experiments. We calculate the cosine similarity between the weights learned by the PEFT methods ( ˆ W = W + α ∆ W ) and W (pre-trained weights). A smaller cosine similarity would tell us that SURMs help us in exploring parameters further away from the pre-trained weights ( W ).

We test our hypothesis on the BERT model finetuned on the MRPC dataset by SURMas well as by LoRA. We report the (1-cosine similarity( ˆ W , W )) for both query and key across multiple layers (see Fig 11). We see that LoRA-learnt weights are very similar to the pretrained weights whereas SURMs explore a larger space (as shown by higher dissimilarity). This observation is not too dissimilar to that of [91].

Analysis of trained weight matrices for Pinwheel data . We also want to answer the question: Q: How similar are the representations learned by networks with the SURM layers compared to the full finetuned networks?

We evaluate the CKA similarity [38] between the full fine-tuned network and the network with the LDR layers. CKA is a widely used metric to compare representations coming from different neural networks. We observe that LDR networks have higher CKA similarity with fully finetuned networks than their LoRA counterparts.

B.5 Guidance for Practitioners

To translate our framework into actionable insights, we aim to highlight several key properties of the various classes of SURMs that help us in making the final recommendation. In all our experiments, we found that on average that circulant variant achieves the largest number of best performances across multiple datasets (Figure 1, Table 1, 2, 3). Moreover, in the low data regime, it is clear that the circulant is the most performant variant as well (Figure 6).

The time complexity of LDR matrices is sub-quadratic, in particular, time complexity for both: the circulant and the Toeplitz variant is the same but the Toeplitz one is slower by a factor of 2. The gradients allow for a very simple formula, which is computed in sub-quadratic time (see eq 14 in [15] for the circulant matrix and Proposition 3.6 in [66] for the Toeplitz matrix). Therefore, our general recommendation to practitioners is to use the circulant variant of SURM. It is relatively fast and our most accurate variant.

C Broader Impact & Limitations

Fine-tuning large pre-trained Transformers for downstream tasks requires substantial computational resources. We hope that this work addresses this important problem by reducing the overall computational budget while maintaining high accuracy. We believe that SURMs will make Transformers accessible to researchers and academics world wide and also reduce the carbon footprint associated with training these models. While democratizing powerful Transformers' technologies with those methods, one must still be cautious of the potential harmful biases, inherent to models pre-trained on the internet-scale data. One of the main limitations is the absence of custom kernels for our methods. Despite their theoretical speed advantage, popular methods like LoRA have been extensively optimized by the machine learning community for efficient execution on hardware.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We provide extensive empirical evidence in Section 6 as well additional motivating experiments in Figure 4

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations of our work is detailed in Appendix C.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not contain any new theoretical results but builds on the main theorem in [66].

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide the hyperparameters to replicate the experiments in Appendix B.1.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide access to the code on building the adapter layers using SURM and their integration into pretrained transformers.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide the hyperparameters to replicate the experiments in Appendix B.1 and B.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Given the sheer volume of experiments along with the baselines that we had to replicate, it was too expensive for us to run these experiments for multiple seeds. However, we report error bars for the smaller toy experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide this information in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: All authors have reviewed NeurIPS Code of Ethics and the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The broader impacts of our work is detailed in Appendix C.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We are not releasing any new data or models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited the authors who created the original data and algorithms that are used in this work.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We release the main code detailing the creation of the SURM layers leveraging fast matrix-vector multiplication and their integration in pretrained Transformers.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not use any crowd sourcing and human subjects. The multiorgan segmentation downloaded from synapse and we adhere to the rules of the usage of this data.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We do not IRB approval for the use of the multi-organ segmentation dataset. This can be freely downloaded from the internet once one signs up on synpase. We strictly adhere to the rules of the usage of data.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores the use of structured matrices instead of low-rank matrices for approximating the finetuning updates for Transformer models. The sturcture imposed on the approximation matrices determines the number of trainable parameters. The paper explores use of two structured matrices: Circulant and Toeplitz matrices, and Kronecker product of Matrices for finetuning language models and vision models. The paper shows that structured matrices are better at approximating various class of matrices like random (full rank), near-low-rank and near-low-intrinsic-rank than low-rank matrices, and this translates to better performance when using them for finetuning at a lower parameter cost.\n\nSTRENGTHS:\n- Use of structured matrices for finetuning achieves on-par/better performance than prior works at a lower parameter cost.\n- The toy experiments show that structured matrices can approximate matrices better than low-rank matrices (controlling for the number of parameters). The authors verifiy this through two experiments:\n    - Approximating Symmetric Positive Definite Matrices\n    - Fitting a toy dataset (pinwheel with Gaussian noise) using a neural net with layers composed of structured and low rank matrices.\n\nWEAKNESSES:\n1. **No studies on more difficult tasks**:\n- While I understand that the authors have shown the effectiveness of the proposed method on smaller models, it would be interesting to see how the proposed method performs on larger models for more difficult tasks like language generation (for e.g., insturction tuning) or math/commonsense reasoning which are commonly used to study PEFT methods on large language models.\n2. **Lack of details on choice of hyperparameters**:\n- The paper does not provide sufficient details about how the data is split, if all the methods are compared by training and evaluating on the same data splits.\n- Moreover, the paper does not provide details on how the hyperparameters are chosen, but only the final values are mentioned. (Section E in the appendix.)\n3. **Minor presentation details**:\n- Figure 1: caption says the presented methods are in top left (in green), but appear to be in the top right in the plots, and also do not apprear to be in green.\n\nQUESTIONS:\n1. **Section 4.1**:\n- What is the significance of Section 4.1? What does the comparison between Circulant and Toeplitz matrices with W(G,H) convey? Could the authors provide more explanation about why this comparison is done, especially since the final PEFT algorithm uses only Circulant and Toeplitz matrices?\n- Do all the methods in this study approximate the exact same matrix? Why do the initial errors vary by such a large margin across W(G,H), and Circular and Toeplits matrices?\n- Why do Circulant and Toeplitz matrices use more iterations than W(G,H) (200 vs. 2000)? If the number of iterations is the same (i.e., W(G,H) is optimized for 2000 iterations), do the results change?\n- Why is there no study for approximation of random matrix using Toeplitz Circular matrices?\n- Why is low-rank(I=20) + epsilon not considered for Circular and Toeplitz matrices?\n\n2. **Experiments on image classification using ViTs**:\n- What is the pretraining data and task for the base ViT?\n- How is the data split into train, validation and test splits, and on which split is are the results reported?\n- How and on which split are the hyperparameters tuned? Section E, lines 699-700 mention that the experiments use a learning rate of 5e-5, but SVHN uses 5e-4. How is this chosen?\n\n3. **Experiments on GLUE**:\n- What is the difference bewteen SURM (Circular) and SURM (Circular-LoRA/Adapter) (similarly Toeplitz and Kronecker)?\n\nLIMITATIONS:\nAuthors have addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new technique called SURMs which aim to use structured matrices for PEFT. The technique is tested against many different adapter variants in Vision Adaptation and Natural Language. The unique structure of the matrices allow for efficient computation of the products\n\nSTRENGTHS:\n- The idea of structured matrices compared to low rank ones as avenue of exploration is interesting.\n- Many different adapters are used during evaluation\n- The figures are clear well thought out\n\nWEAKNESSES:\nThe bar for a new variant of LoRA to be substantial contribution to me is fairly high, just due to how crowded the space is.\n\nStructured matrices for low rank adaptation has previously, as noted in the paper Knoecker products have been in previous literature, while Circulant and Toeplitz structures are, to my knowledge, have not. Nevertheless, Structured matrix adaptation has been explored in several prior papers.\n\nThe change in structure of the matrix or rank alone to me is not sufficient unless there is substantial evidence it provides benefit, both empirically and intuitively/theoretically.\n\nThe intuition and/or theory as why we might expect this class of matrices to be a useful, in terms of improved adaptation, is lacking (beyond the FFT-based efficiency). As stated previously, the space is fairly crowded for this line of research and every new modification cannot be considered significant without substantial evidence and careful reasoning behind exploration.\n\nThe empirical evidence for this change also needs to be improved upon. \nPEFT experiments need to largely be conducted in regimes where full-finetuning represents a soft upper bound on the achievable performance. The most common use case for PEFT techniques is improving finetuning compute/memory efficiency, while retaining as much performance as possible. In contrast, for regimes where we don't have sufficient data for full fine-tuning to perform best, it seems that we mainly end up measuring the ability of the technique to reduce overfitting, rather than the techniques ability to retain the learning capacity of the original model at lower compute levels. This is still useful, of course, but there are many other techniques to do this outside of PEFT, and it misses evaluation for the primary use case of PEFT.\n\nAs a reference, I think the [MoRa](https://arxiv.org/pdf/2405.12130) paper does a decent job at this. The original LoRA paper doesn't do a great job with this and oversells, as the community knows from countless experiments with it. To effectively prove out which techniques hold up best its very important to conduct detailed evaluations at scale with models which can achieve SoTA on a particular dataset.\n\nThe following would make the paper much stronger:\n- Stronger intuitive/theoretical justification for why we might expect this class of adaptations to perform better (with some small scale experiments or ablations proving the intuition)\n\n- Experiments with models in NLP and Vision, where we full finetuning generally performs best (the UUID memorization from MoRa is a good example). Imagenet seems like a good candidate assuming techniques have not pertained on that data. Larger NLP datasets seem like another good candidate (extra AR pretraining, adaptation to larger downstream tasks).\n\nFrom my perspective, if I actually wanted to train on the the many of the small vision datasets for example, there are many different alternate fine-tuning techniques or different models I can use to improve performance for data-starved cases, especially since those cited perform worse than other fine-tuning techniques on these datasets. (Cifar-100, DTD, SUN for example, even using the same model arches). \n\nThe paper needs to do a more careful job of picking SoTA finetunings on larger datasets and using PEFT on those.\n\n**Only PEFT can really target the case where I have large amount of fine-tuning data, but not necessarily the compute/memory to target it with a very large model well, so this regime should be emphasized in the experiments.**\n\nQUESTIONS:\nWhat are the ranks of the other adapters used?\n\nWhy do you only test against rank 1 lora in Table 3?\n\nWhat how does the techniques performance change with rank?\n\nWhy might we expect these structures to perform better than other structured matrices?\n\nLIMITATIONS:\nThe paper should address limitations more directly", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
The paper proposed a general framework for parameter-efficient fine-tuning, based on structured unrestricted-rank matrices (SURMs) to substitute LoRA or other parameter-efficient finetuning methods. Three variants of SURMs are included, named Kronecker, Toeplitz, and Circulant, based on the matrix type. The proposed SURM method generally achieves performance comparable with LoRA/adaptors with 50% or less parameters on vision datasets like CIFAR10, SUN397, and DTD. SURM can also be incorporated into adaptors.

STRENGTHS:
1. The authors introduced a novel approach to substitute LoRA. The proposed SURM method provides us with a novel view of how the matrix of the LoRA can be initialized and interacted with.
2. Integration experiments with adaptors show the possibility that this method may also be applied in other modules of models, only if they have low intrinsic ranks.

WEAKNESSES:
1. The tasks include GLUE benchmarks, but the improvements are not convincing. Most of the tasks are in trade-off with other methods. Why not increase the trainable parameter up to the same level as other methods, for example, 0.9, to examine the upper bound performance? This could be part of the ablation studies, showing the scaling relationship between the performance and trainable parameters.
2. Experiments conducted in NLP should contain new architectures, such as LLaMa.

QUESTIONS:
Please see the weaknesses.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

comments: SUMMARY:
This paper explores structured unrestricted-rank matrices (SURM) for parameter-efficient fine-tuning (PEFT) of large-scale Transformer models. This method (SURM) was the first to apply low displacement rank matrices (LDRM) which could support fast matrix-vector multiplication and showed flexibility in finding a balance between compactness and expressiveness. What they used were Circulant, Toeplitz (LDRM), also Kronecker matrix (not LDRM). The authors demonstrated improved accuracy in various data sets while replacing low-rank matrices in LoRA and showed a reduction in the number of parameters in the adapter.

STRENGTHS:
1. Using Circulant and Toeplitz for PEFT seems to be a novel technique and also very effective on many standard benchmarks.

2. Extensive experiments have been provided, which support the authors’ claim.

3. The writing is clear, and comprehensive supplementary material would be very helpful for the readers.

WEAKNESSES:
1. Any reasons why the proposed version w/ the Kronecker product outperforms the previous works, such as Kadaptation?

2. Why is PSD approximation accuracy a good proxy task? is the weight delta during finetuning close to PSD by any chance?

3. As the author mentioned in the limitation section, there are many fancy accelerated implementations of LoRA. How slow is the proposed method compared to those LoRA implementations?

QUESTIONS:
see the weakness section.

LIMITATIONS:
I do not see any serious societal impact of the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13