explain the following like i am a layperson: ""LORD: LOW RANK DECOMPOSITION OF
MONOLINGUAL CODE LLMS FOR ONE-SHOT COMPRESSION
Ayush Kaushal
Universite de Montr ´ eal, Nolano AI ´
<PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>
Tejas Vaidhya
Mila, Universite de Montr ´ eal, Nolano AI ´
<PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>
Irina Rish
Mila, Universite de Montr ´ eal, Nolano AI ´
<PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>
ABSTRACT
Low Rank Decomposition of matrix - splitting a large matrix into a product of
two smaller matrix offers a means for compression that reduces the parameters
of a model without sparsification, and hence delivering more speedup on modern hardware. Moreover, unlike quantization, the compressed linear layers remain
fully differentiable and all the parameters trainable, while being able to leverage
the existing highly efficient kernels over floating point matrices. We study the potential to compress Large Language Models (LLMs) for monolingual Code generation via LOw Rank Decomposition (LoRD) and observe that ranks for the linear
layers in these models can be reduced by upto 39.58% with less than 1% increase
in perplexity. We then use LoRD to compress StarCoder 16B to 13.2B parameter
with no drop and to 12.3B with minimal drop in HumanEval Pass@1 score, in less
than 10 minutes on a single A100. The compressed models speeds up inference
by up to 22.35% with just a single line of change in code over huggingface’s implementation with pytorch backend. LoRD models remain compatible with state
of the art near-lossless quantization method such as SpQR, which allows leveraging further compression gains of quantization. Lastly, QLoRA over LoRD model
further reduces memory requirements by as much as 21.2% over vanilla QLoRA
while offering similar gains from parameter efficient fine tuning. Our work shows
LOw Rank Decomposition (LoRD) as a promising new paradigm for LLM compression. 1
1 INTRODUCTION
Code LLMs have become an integral component of Copilots that boost developer productivity (Peng
et al., 2023) and in LLM based agents (Wang et al., 2023a). These Code LLMs are as large as 34
Billion parameters for the publicly available models Roziere et al. (2023) and more than 175 Billion `
parameter for closed source ones Chen et al. (2021a). There is not only a pressing need for reducing
model size and running models at a lower cost, but also for increasing the inference speed. The latter
is especially significant for Copilot based applications.
Recently, several methods have been proposed to compress and speed up inference of LLMs. Quantization (Frantar et al., 2023; Dettmers et al., 2023b) reduces the number of bits required per weight
parameter of LLM by lowering the precision, and has shown significant model compression as well
as speedups in low-batch decoding phases of LLMs Kim et al. (2023a). Quantization has also been
shown to generalize well to quantized models Shen et al. (2023). Pruning (Sun et al., 2023a; Frantar
& Alistarh, 2023) has offered another means of compression by removing connections from the neural network and hence sparsifying the weight matrices of the neural networks. Distillation Gu et al.
1We will release LoRDCoder at https://huggingface.co/nolanoAI
1
arXiv:2309.14021v1 [cs.CL] 25 Sep 2023
Preprint: Work in Progress
(2023); Agarwal et al. (2023); Jung et al. (2023) method enables one to train a smaller model using
a larger teacher model for supervision. While quantization and pruning methods that do not require
re-training are viable means of compressing the model, distillation involves a significant amount of
compute for retraining a smaller LLM, often from scratch. Here, we consider another compression
paradigm of LOw Rank Decomposition (LoRD) , that does not require expensive retraining as in the
case of distillation and covers up several deficiencies of the quantization and pruning compression
method.
Low Rank Decomposition factorizes a dense matrix of a neural network as a product of two smaller
dense matrices. The LoRD model can leverage the highly optimized floating-point dense matrix
multiplication kernels (NVIDIA, 2007; Blackford et al., 2002) that have been written over modern
hardware. In contrast, quantized models require specialized kernels to be written, often different
for each hardware backend in order to enable fast inference. Moreover, the neural network remaining fully-differentiable and all the parameters remaining trainable even after compression, unlike
quantization. The LoRA Hu et al. (2022) layers of tuned models are also easier to merge back into
floating point matrices compared to the quantized ones.
Pruned models produce sparse matrix weights in the neural network. Matrix multiplication over
sparse matrices is much slower than the resulting dense matrices in LoRD on most GPUs. Dense
matrices, in addition avoid representation format overhead that sparse matrices incur from parameter
reduction 2
and often requires specialized kernels for reducing this overhead Dettmers et al. (2023b).
Dense matrix multiplication is also easier to implement than sparse matrix multiplication, especially
over quantized models.
Several previous works have attempted to apply matrix decomposition methods like SVD, Tucker
or Kronecker decomposition for compression (Ben Noach & Goldberg, 2020; Tahaei et al., 2022;
Edalati et al., 2022). However, these have been limited to small language models like Bert (Devlin
et al., 2019) and GPT2 (Radford et al., 2019), and have shown success only on narrow task-specific
use cases or after retraining, often only with teacher-guided distillation supervision. These works
have observed that weight matrices are not low rank and adapt methods like Singular Value Decomposition for data-aware decomposition of weights (Chen et al., 2021b; Hsu et al., 2022; Yu & Wu,
2023).
We, adapt these approaches for Large Language Models (Billion+ Parameters) over python code, and
show that these models can be low-rank decomposed to compress and speed up inference without
the need for retraining with little to no performance degradation. We study low-rank decomposition
across two families of code LLMs - StarCoder and CodeGen (§2) for varying parameter sizes and
establish the potential for reducing rank of models through decomposition. We then study these
trends across different kinds of linear layers in a transformer block and observe the potential for
upto 39.58% rank reduction with less than 1% change in perplexity.
We propose various considerations for compressing the models and to achieve inference speedup
on GPUs (§3.1). Using these, we achieve compression of the StarCoder 16B model offering 31.67
HumanEval Chen et al. (2021a) Pass@1 score down to 13.2B parameter with similar performance of
31.57 HumanEval and down to 12.3B parameter with 29.22 HumanEval score (§3.2). LoRD models,
offer an inference speedup of as high as 22.35% with just one line of change in huggingface’s (§3.3).
These LoRD models can be further compressed via the near-lossless quantization method of SpQR
Dettmers et al. (2023b) to reduce it’s precision to 8 and 4 bits without any further reduction in
HumanEval performance (§4.1). Finally, these decomposed models also reduce the memory requirements of adapter finetuning by 21.2% over QLoRA (§4.2).
2This overhead in sparse matrix occurs from having to store indices/bitmasks to indicate which values are
present and not. This can be very significant at low levels of sparsity. PyTorch’s sparse formats (CSR, CSC,
COO) all store indices at int64 format, and for moderate levels of sparsity (<50%), the sparse matrix takes up
more space than a dense matrix with zero-ed out values.
2
Preprint: Work in Progress
2 CODE LLMS ARE LOW RANK DECOMPOSABLE
2.1 BACKGROUND
Let an linear layer L of an LLM M with weight W ∈ R
d1×d2 and bias b ∈ R
d1×1
. Let dmin =
minimum(d1, d2) and dmax = maximum(d1, d2)
A Low Rank Decomposition or Low Rank Factorization of a layer L would give us a new layer L˜
with two weight matrices A ∈ R
r×d2 and B ∈ R
d1×r
, and a bias ˜b ∈ R
d1×1
, where r << dmin
such that for a n batch of input vectors X ∈ R
d2×n the batch of output vectors Y ∈ R
d1×n is,
Y = L˜(X) = BAX + ˜b ≈ L(X) = W X + b (1)
Singular Value Decomposition (SVD) offers the best r-rank approximation of matrix W ∈ R
d1×d2
.
First W can be decomposed as W = USV T
, where U ∈ R
d1×d2 and V ∈ R
d2×d2 are orthogonal
matrix and S ∈ R
d1×d2
is a diagonal matrix with entries in decreasing order. Then, by taking top-k
rank, we can decompose W as a product of two low ranked matrices W ≈ BA as follows
W = (U:,:rS:r,:r)
| {z }
B∈Rd1×r
(V:r,:)
| {z }
A∈Rr×d2
(2)
where :a,:b denotes a slice operation over a matrix that gives its first a rows and b columns.
Eigendecomposition is another decomposition method applicable to symmetric matrices. We can
represent the eigendecomposition of a symmetric matrix W ∈ R
d1×d1 as W = QΛQT
. Here
Q ∈ R
d1×d1
is an orthogonal matrix whose columns are the eigenvectors of W, and Λ ∈ R
d1×d1
is a diagonal matrix whose entries are the eigenvalues of W sorted in decreasing order. Similar to
SVD, we can decompose W as a product of two low ranked matrices W ≈ BA by retaining only
the largest r eigenvalues (and corresponding eigenvectors) as follows:
W = (Q:,:rΛ:r,:r)
| {z }
B∈Rd1×r
(Q
T
:r,:
)
| {z }
A∈Rr×d1
(3)
Since Q is orthonormal and the eigenvalues Λ is sorted in descending order, Q:,:rQT
:,:r ≈ I where I
is identity matrix of dimension d1.
While SVD gives the optimal low-rank decomposition of matrix, in terms of Frobenius norm, but
does not take input and output data distribution into account. Approaches like weighted SVD (Hsu
et al., 2022) and SVD over both weight and data (Chen et al., 2021b) have been proposed but
are prohibitively expensive to scale to larger models for their requirement of backpropagation over
calibration dataset. SVD over very large weight matrices is also very computationally expensive.
So, we instead leverage the observation that activations in transformers are low-ranked (Feng et al.,
2022) and adapt the more heuristically driven approach of Atomic Feature Mimicking (AFM) (Yu
& Wu, 2023) that creates low rank matrices conditioned on a small amount of calibration data.
Specifically, consider the eigen-decomposition of Covariance over Y as
E[yyT
] − E[y]E[y]
T = QˆΛˆQˆT
(4)
Here Qˆ is a matrix of its eigenvectors, hence Qˆ
:,:rQˆT
:,:r ≈ I. Using this, we can write the output
vector Y as Y ≈ Qˆ
:,:rQˆT
:,:rY . By writing Y in terms of W, X and b from Equation 1, we have:
Y ≈ Qˆ
:,:rQˆT
:,:rW X + Qˆ
:,:rQˆT
:,:r
b (5)
Comparing to Equation 1, this gives us B = Qˆ
:,:r ∈ R
d1×r
, A = QˆT
:,:rW ∈ R
r×d2 and ˜b =
Qˆ
:,:rQˆT
:,:r
b ≈ b. This approach is also straightforward to adapt for LLMs like LLaMa (Touvron
et al., 2023), Falcon (Penedo et al., 2023), CodeLLaMa (Roziere et al., 2023) which do not have a `
bias term in the linear layer by setting ˜b to zero vector.
3
Preprint: Work in Progress
2.2 EXPERIMENTAL SETTINGS
We take our python calibration dataset from the stack (Kocetkov et al., 2022) and consider the
corresponding subset of the stack smol (Bigcode, 2022) as validation data. We filter out those
sequences which are less than 1024 tokens or 10240 characters in length. We consider CodeGen and
StarCoder model family of models. CodeGen mono models are present across 350M, 2B, 6B and
16B parameters and are CodeGen models that were further trained on only python code. StarCoder
16B is the StarCoderBase 16B model further trained on only python code from the stack dataset’s
train split. We also consider StarCoderBase at 3B and 7B parameter sizes in StarCoder family due
to the lack of their monolingual counterparts. All our experiments were performed on a single A100
GPU in under an hour for each run.
For studying the trends of increase in perplexity for a reduction in rank across difference model
sizes, we set a fixed low-rank r for all the layers. Later we discuss how to achieve compression and
inference speedup via low-rank decomposition in §3
2.3 CHANGE IN PERPLEXITY ACROSS REDUCTION IN RANK
Figure 1a and 1b show the trends of increase in perplexity across reduction in rank of the weight
matrix of CodeGen and StarCoder models. For the largest models in both families, we observe only
about a 1% increase in perplexity for 10% reduction in rank, and upto 35% reduction in rank for less
than 10% increase in perplexity. The smallest model, CodeGen Mono 350M, however, can only be
decomposed to 35% rank reduction for a similar drop in perplexity. We observe that the perplexity
changes much slower for larger models as the % rank reduces, and hence can be compressed mode,
similar to observations in quantization and pruning (Li et al., 2020). It should be noted that for most
models, more than 50% leads to significant output quality degradation.
(a) Perplexity vs % Rank Reduction for CodeGen
Models.
(b) Perplexity vs % Rank Reduction for StarCoder
Models.
Figure 1: Perplexity vs %Reduction in Rank for Different Models.
3 COMPRESSION AND SPEEDUP THROUGH DECOMPOSITION
In this section, we discuss how we adapt the LOw Rank Decomposition (LoRD) for reducing the
size of model and achieving inference speedup without a significant reduction in the output quality
of the model. Following (Kim et al., 2023a), we assume memory bandwidth is the bottleneck for
inference, and thus speedups for decoding are directly proportional to the size of the transformer
model.
3.1 ACHIEVING COMPRESSION AND INFERENCE SPEEDUP
Threshold for size reduction across rank reduction: Consider a weight matrix W ∈ R
d1×d2
of a transformer layer with low rank decomposed A ∈ R
r×d2 and B ∈ R
d1×r
. The number of
parameters before and after decomposition respectively are d1d2 and r(d1 + d2). Therefore, if
4
Preprint: Work in Progress
r > d1d2
(d1+d2)
, (i.e a decomposition with small rank reduction), then the size of the model after
decomposition can even be higher than the original models. Ideally, we would want the rank r <<
d1d2
(d1+d2)
or r << dmin.
Matrix Aspect Ratio and Compression: Let the ratio of the smaller dimension to the larger dimension of the matrix (i.e. the aspect ratio) be α =
dmin
dmax
. For square matrix, α = 1 and for tall or
fat matrices α << 1. We can rewrite, the percentage change in parameters from decomposition, in
terms of percent change in rank %∆r = 100 ∗
dmin−r
dmin
% and aspect ratio as:
100 ∗
r(dmax + dmin) − dmaxdmin
dmaxdmin
= 100α − (1 + α)%∆r (6)
It should be noted that change in parameters from decomposition can either be positive (the number
of parameters increased after decomposition), or negative (the number of parameters decreased after
decomposition). In order to achieve model compression and consequently inference speedups, one
would want a very high negative percentage change in parameters.
Figure 2: Parity Point across various aspect
ratios (α) of the different linear layers in
transformers.
Parity Point for Compression across Rank Reduction: Using Eq. 6, one can observe that little reduction in rank, may lead to increase in model parameters instead of decreasing. For instance, square matrices (α = 1) will have 100% increase (i.e doubling
in size), then %∆r → 0+ and only after the rank
is reduced by more than 50%, will the Parity Point
of the rank reduction be reached, that offers same or
lesser number of a parameter in the decomposed layer
as the original matrix. This parity point for tall or fat
matrices (α → 0+), can be achieved with a very small
percent reduction in rank and can start giving a reduction in model size. For compression to be achieved,
we would want to reduce the rank by an amount to
cross this parity point threshold. However, reducing
the rank by a lot can degrade performance significantly. So we must take the aspect ratio into account,
in order to achieve compression without much reduction in rank (and hence no significant degradation in
output quality)
A transformer model had different aspect ratios across its various linear layers, α = 1.00 for output
projection after attention, α = 0.96 for Multi-query attention (Shazeer, 2019) projections, α = 0.25
for typical MLP projections with intermediate expansion factor of 4 as in the original transformer
and as low as α = 0.12 for the embedding and language model head projection of CodeGen 16B
with 51200 vocab size. Figure 2 plots the % change in the size of the model across % reduction
in rank for matrices with different aspect ratios. For square matrices and near square matrices,
a small rank reduction doubles the size of the linear layer after decomposition, and only after its
parity point of 50% reduction is the size after decomposition, the same as original matrix. By this
extent of rank decomposition, the performance starts to significantly degrade, as seen in §2.3. All the
previous works on smaller models, address this by retraining the model (Yu & Wu, 2023; Chen et al.,
2021b; Hsu et al., 2022; Ben Noach & Goldberg, 2020), often via knowledge distillation supervision
(Hinton et al., 2015; Sanh et al., 2019) on specific narrow tasks. However, retraining is infeasible
for larger models. Thus, we skip matrices with very high aspect ratios such as output projection or
multi-query attention for decomposition. In contrast, the weights in MLP achieve parity at only 20%
rank reduction. While embeddings and LM Head can be compressed through decomposition, as they
have been for smaller transformer models (Baevski & Auli, 2019; Lan et al., 2020), they contribute
only a very small portion of the weight of the model. So, we do not consider decomposing these
matrices. In order to reduce the aspect ratio of matrices, we group layers with the same input vector
to have the same bottleneck matrix after decomposition. Doing so, enables re-use of computation,
and sharing of weights, as well as bringing the aspect ratio down to achieve compression as lower
rank reduction. Candidate layers for grouping include the query, key and value projection matrices
5
Preprint: Work in Progress
in multi-headed attention with aspect ratio reduced to α = 0.33 and the gating layer in SwiGLU
(Shazeer, 2020) with first linear linear of MLP in models like LLaMa (Touvron et al., 2023) with
α = 0.1875.
(a) CodeGen 16B. (b) StarCoder 16B.
Figure 3: Parameter Reduction vs perplexity for decomposition across various layers.
Trends across different layers in a transformer block: In addition to considering the parity point
into account for deciding which layers to decompose, we also additionally study the sensitivity of
each of these layers to low rank decomposition across the large model in the two model families.
Figure 3 shows the increase in perplexity vs reduction in model parameters for the two models. For
both models, decomposing all the linear layers achieves the parity point much later than any one of
these linear layers with low aspect ratio. For CodeGen, the attention weight matrix (query, key and
values projection) offers least increase in perplexity for the biggest drop in parameter count, make
this layer the most suitable candidate to be decomposed. It shows less than 1% increase in perplexity
even after 39.58% rank reduction. We observe the mlp 2 (downscaling mlp) to be a better candidate
for decomposition than mlp 1 (upscaling mlp) across both models. This makes mlp 2 to be a good
candidate for low-rank decomposition over the StarCoder model.
Hardware Considerations: On modern hardware accelerators like GPU and their corresponding
software stack, matrix multiplication kernels are faster if their dimensions are divisible by a high
factor of 2. So, we consider ranks at a reduction of approximately every 10%, rounded off to the
nearest multiple of 128 in our experiments.
3.2 PERFORMANCE OF COMPRESSED MODELS
We consider the largest models of StarCoder and CodeGen family (16B) and perform low-rank decomposition on both with varying ranks. We consider decomposing layers that offers most parameter
reduction (§3.1) with least increase in perplexity - mlp 2 for StarCoder and attention for CodeGen.
We report the Pass@1 and Pass@10 scores over the Human Eval dataset (Chen et al., 2021a) using the code-eval GitHub repo (Bacaj, 2023) in Table 1. We observe that StarCoder models can be
low rank decomposed to 13.2B parameters (50% rank reduction) with no drop in Pass@1 performance and upto 12.3B parameters (62.5% rank reduction) with very little drop. CodeGen models
shows similar trend in drop in Human Eval performance when measured in terms of rank reduction.
However, in terms of parameter reduction count, while showing very little perplexity change with
large reduction in rank (Fig. 3a), shows much more drop in its HumanEval score when measured
in terms of parameter count reduction due to a higher aspect ratio of the matrix being decomposed.
It should be noted that for certain compressed models, the Pass@1 even slightly improves over the
base model. Similar trend of slight improvements from compression across various metrics and
benchmarks has been observed in the case of other compression attempts (Frantar & Alistarh, 2023;
Cerebras, 2022).
6
Preprint: Work in Progress
Starcoder 16B CodeGen 16B Mono
Model Type Rank HumanEval Score Model Type Rank HumanEval Score
Pass @ 1 Pass @ 10 Pass @ 1 Pass @ 10
Base Model 6144 31.67 48.28 Base Model 6144 29.02 46.34
LoRDCoder 14.9B 4480 33.18 48.41 LoRDCoder 15.9B 4480 29.08 46.95
LoRDCoder 14.5B 4096 31.69 45.12 LoRDCoder 15.6B 4096 28.90 46.24
LoRDCoder 13.8B 3584 30.90 47.56 LoRDCoder 15.1B 3584 28.54 45.73
LoRDCoder 13.2B 3072 31.57 45.36 LoRDCoder 14.7B 3072 27.99 43.29
LoRDCoder 12.6B 2560 29.84 42.31 LoRDCoder 14.3B 2560 27.32 45.12
LoRDCoder 12.3B 2304 29.22 40.12 LoRDCoder 14.1B 2304 27.07 41.46
Table 1: Human Eval Score of LoRD across StarCoder and CodeGen.
3.3 SPEEDUP FROM LORD
We next consider accessing the inference speedup (forward pass) of the models over the standard
cuBLAS floating point kernels. We consider the standard Huggingface implementation (Wolf et al.,
2020) of Starcoder with pytorch backend (Paszke et al., 2019) utilizing standard cuBLAS kernels
on A100 GPUs. LoRD decomposed models were implemented by modifying just one line of code
to replace an MLP with an extra linear layer 3
. We benchmark over 1024 tokens and 512 tokens
sequence, averaged across 10 runs with warm up of 3 runs. We plot relative time taken and model
size across reduction in rank in Figure
Figure 4: Time and Model size of StarCoder 16B across ranks.
4.
Inference speedups as high as 22.35% are observed
for decomposed models. The lines in the graph are
generally downward sloping, Therefore reduction in
rank beyond 25% generally implies less inference time
and reduction in model size. However, the underlying
hardware (and pertaining software kernels) also significantly affect the speedup gains. We notice huge gains,
whenever the rank is rounded off to a multiple of a very
high power of 2 (like 4096 and 2560 at 33% and 58%
rank reduction), despite very little reduction in model
size. In contrast, for certain ranks which are multiples
of a lesser power of 2 (like 3584 and 2304 at 41% and
62% rank reduction) are slower than those at slightly
higher ranks. It is worth noting that affect of hardware
inefficient matrix shape is less significant for longer tokens sequence of 1024 because the O(n
2
) attention overhead starts becoming more significant,
especially in the absence of SoTA attention implementation techniques (Rabe & Staats, 2021; Dao
et al., 2022; Dao, 2023) as in the case of Huggingface’s implementations.
4 COMBINING LORD WITH QUANTIZATION AND LORA
4.1 QUANTIZATION
While LoRD enables compression at same precision level, we study whether the decomposed models can be further compressing through quantization. Table 2 shows the HumanEval pass@1 results
for the different LoRDCoder across 8 and 4 bit quantization levels, using the near lossless quantization technique of SpQR (Dettmers et al., 2023b). We observe that the LoRD models can be
combined with quantization for further compression, showing no performance drop for 8-bit and
very little performance drop on 4-bit quantization for most models. Slight increase in HumanEval
after quantization is also observed, similar to Pangu-Coder2 (Shen et al., 2023).
4.2 PARAMETER EFFICIENT TUNING OF LORD MODELS
3
nn.Linear(in, out) -> nn.Sequential(nn.Linear(in, rank), nn.Linear(rank, out))
7
Preprint: Work in Progress
Model Pass@1@FP16 Pass@1@8-bit Pass@1@4-bit
LoRDCoder 14.9B 33.18 33.17 32.01
LoRDCoder 14.5B 31.69 31.58 32.74
LoRDCoder 13.8B 30.90 31.10 30.73
LoRDCoder 13.2B 31.57 31.52 32.01
LoRDCoder 12.6B 29.84 29.87 30.22
LoRDCoder 12.3B 29.22 29.14 29.45
Table 2: Human Eval score of quantized LoRDCoder models.
d x d
d x r’
r’ x d
d x r’
r’ x d
d x r
r x d
d
r’
h
x
d
r’
h
x
r
LoRD
Weights
a) LoRA b) LoRD + LoRA
Pr e t r a i n ed
W e i g h t s
A R
X R
Y R
W R
X R
Y R B R
Figure 5: LoRA vs LoRD + LoRA.
We next test the potential for using
LoRD to further reduce the memory usage over existing parameterefficient techniques. We consider the
code instruction dataset (Chaudhary,
2023) and filter those examples that
pertains to python programming language. We use QLoRA (Dettmers
et al., 2023a), which is an even more
memory efficient version of LoRA
(Hu et al., 2022) storing the weights
in quantized format, for fine-tuning
for 1 epoch. We compare results
from fine-tuning two of the decomposed models LoRDCoder 13.2B and
LoRDCoder 12.3B model to the StarCoder model. We observe a HumanEval pass@1 of 37.80 and 37.62 across LoRDCoder 13.2B
and LoRDCoder 12.3B fine-tuning, competitive to the performance of 37.74 offered by StarCoder
model.
5 RELATED WORK
There is a growing interest in compressing pretrained Large Language Models. Several recent attempts have been dedicated to the quantization of weights of LLMs (Frantar et al., 2023; Lin et al.,
2023; Yuan et al., 2023; Park et al., 2022; Kim et al., 2023b; Chee et al., 2023; Li et al., 2023a) with
tricks such as outlier separation (Dettmers et al., 2022; Dettmers & Zettlemoyer, 2022; Dettmers
et al., 2023c; Wei et al., 2022; Kim et al., 2023a; Lee et al., 2023). Some attempts also quantize
the activations (intermediate representations) in addition to weights to speed up computation time
(Shao et al., 2023; Xiao et al., 2023). The works in quantization that are closest to us is the LowRank Compensation (LoRC) Strategy (Yao et al., 2023; Wu et al., 2023), where the difference of
the quantized matrix to the original matrix is approximated by a product of low-rank matrices. Our
work decomposes the entire matrix for compression.
Pruning neural networks Liang et al. (2021), unlike quantization, reduces the number of parameters
in a model by removing unimportant weights or connections. Several techniques have been proposed
to scale pruning methods for LLMs (Sun et al., 2023a; Frantar & Alistarh, 2023; Ma et al., 2023).
However, pruning as a means of compression is yet to become viable due to no speedups over sparse
matrices without significant performance drop at extreme levels of sparsity or structured sparsity
(Zhu et al., 2023). With low-rank decomposition, we propose an alternate method for reducing
model parameters that offer speedup even at a little reduction in parameter count. Certain works
have also attempted to (Ren & Zhu, 2023; Li et al., 2023b) to split a dense matrix as a sum of lowrank matrices and a sparse matrix. However, these methods require retraining and have been shown
to work only for Language Models of less than a billion parameters.
Low rank decomposition has been proposed for smaller language models like Bert or GPT2 before
using SVD decomposition (Ben Noach & Goldberg, 2020) and Kronecker decompositions (Tahaei
et al., 2022; Edalati et al., 2022). Hsu et al. (2022) modified SVD to be data aware based on approximate second-order gradient information. A better weighted SVD was proposed by (Hua et al.,
2022). Chen et al. (2021b) proposed a data aware decomposition method with a provably optimal
closed-form solution, utilizing a large amount of data points over specific tasks to decompose. Sev8
Preprint: Work in Progress
eral recent works (Yu & Wu, 2023; Feng et al., 2022) have shown that while the weight matrix of
neural networks is not inherently low-rank, the intermediate representations are, thus propose to
decompose based on representations. All these works have focused on small language models and
require re-training. We proposed low-rank decomposition for compressing neural networks without
the need for retraining. The factorization has also been used just for the embedding layers (Baevski
& Auli, 2019; Lan et al., 2020), as they are good candidates due to their very low aspect ratio of
0.015, where a reduction of rank by even 5% would lead to reduction in number of parameters after
decomposition.
There is also a growing interest in fine-tuning large language models Taori et al. (2023); Chiang
et al. (2023); Wang et al. (2023b); Sun et al. (2023b). With the large memory requirements for finetuning full parameters of the LLM, the more parameter-efficient fine-tuning methods like LoRA
(Hu et al., 2022) are getting widely adopted. These methods freeze the original LLM weights, and
attach two low-rank matrices or adapters, in a skip-connection (He et al., 2016) to the linear layers
of the model. These parameter-efficient fine-tuning approaches have seen improvements in lower
activation memory (Zhang et al., 2023) or by keeping non-trainable model weights at 4-bit precision
(Dettmers et al., 2023a). Our work, while focused on compression through low-rank decomposition,
can also enable more efficient fine-tuning, especially in conjunction with existing methods.
6 CONCLUSION
We studied the compression of monolingual code generation models through a novel one-shot compression paradigm of low-rank decomposition. We analyse the change in perplexity with change
in rank across the model families of StarCoder and CodeGen as well as their individual layers and
observe that the rank of these models can be reduced by upto 39.58% with less than 1% change in
perplexity. We then proposed considerations for one-shot compressing these models through LOw
Rank Decomposition (LoRD) in under 10 minutes. Consequently, we compress StarCoder 16B
to 13.2B with no drop in HumanEval pass@1 and very little drop in HumanEval pass@1 to 12.3B
parameters. With a minimal change in code over huggingface’s default inference code of just one
line, we gain speedups of up to 22.35%. The LoRD models are also compatible with near lossless
quantization techniques of SpQR, which offers gains of quantization based compression in addition
to ones from decomposition. The LoRD models also reduce memory requirements by as much as
21.2% over vanilla QLoRA fine-tuning.
7 BROADER IMPACT AND FUTURE WORK
Our work on LoRD, compresses code LLMs which enables them to run on smaller GPUs including
as consumer grade GPUs. This is especially of pressing importance for the next few years when
the shortage of GPU supply is relative to the increasing demand in today’s market. Moreover, faster
inference helps reduce the GPU cycles, enabling lower running costs and lower power consumption
for LLM inference. Our work helps reduce the carbon emissions incurred and moves towards a
greener NLP. Through compression, our work also promotes inference at the edge, and therefore
opening room for applications involving strict privacy requirements. Lower latency will also help
improve the User Experience in applications like CoPilots where lag between suggestions can impact developer’s productivity. Several of these benefits of LoRD such as lower cost and energy
consumption are also applicable for fine-tuning use cases of LLMs.
Our work opens up a new paradigm for compression via Low Rank Decomposition over Large
Language Models in a single shot without the need for retraining. Since, LoRD models can leverage
existing floating point kernels across BLAS and cuBLAS, in contrast to quantization, these are
much easier to implement and reap inference benefits. Our study on hardware considerations for
speedup also opens up the potential for tuning the rank of decomposed models to fit best on the
target hardware and the accompanying GEMM kernels. While our study is limited to monolingual
code LLMs, the low rank decomposition technique is general and not specific to code domain. Thus
exploring its applicability to more general purpose models like LLaMa is a promising direction
for the compression of transformer LLMs beyond quantization. Another interesting unexplored
question is whether the LoRA or QLoRA modules fine-tuned on original models, can be plugged in
as-is for the LoRD models without any performance drop""