CA-LoRA: Adapting Existing LoRA for Compressed LLMs to Enable Efficient Multi-Tasking on Personal Devices
Authors: Weilin Zhao
Yuxiang Huang
Xu Han
Zhiyuan Liu
Zhengyan Zhang
Kuai Li
Chen Chen
TAO YANG
Maosong Sun
Venue: COLM (2024)
Keywords: Model compression, Parameter-efficient tuning, Low-Resource Methods for NLP
TL;DR: Simply applying existing LoRAs on a compressed LLM has a poor performance. We propose Compression-Aware LoRA to address this issue and form an efficient and effective framework for personal devices.

Abstract: Recently, there has been a demand to deploy Large Language Models (LLMs) on personal devices such as laptops and smartphones. These LLMs have different model variants when handling different tasks. However, personal devices have limited resources and require reduced storage overhead. To address this, there are two key methods available: the first is model compression, which compresses LLMs into smaller sizes; the second is LoRA, which can transfer an LLM to other tasks with very few parameters, avoiding the storage of multiple model variants in multi-task scenarios by only preserving LoRAs.
However, our experiments show that directly combining these two methods yields sub-optimal performance. Considering that the open-source community has already contributed many LoRAs to LLMs, we propose to adapt these existing LoRAs from the LLMs to their compressed version and introduce a Compression-Aware LoRA (CA-LoRA) framework.
We incorporate knowledge inheritance and recovery strategies to recover the lost knowledge caused by model compression.
Experiment results demonstrate that CA-LoRA outperforms the vanilla LoRA methods applied to a compressed LLM and achieves comparable performance to the non-compressed LLM with existing LoRA modules. The source code of CA-LoRA is available at https://github.com/thunlp/CA-LoRA.

Published as a conference paper at COLM 2024
CA-LoRA: Adapting Existing LoRA for Compressed LLMs to Enable Efficient Multi-Tasking on Personal Devices
Weilin Zhao?*, Yuxiang Huang"*, Xu Han Zhiyuan Liu Zhengyan Zhang', Kuai Li Chen Chen", Tao Yang', Maosong Sun'
Departmeet of Computer Science and Technology, Institute for Artificial Intelligence, Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, China.
Quan Cheng Laboratory, Shandong, China.
Tencent Machine Learning Platform Department, Tencent Inc, Beijing, China. {zw123, huang-yx22 (rx22)emmail @mails. tsinghua. edu. en, (hanxu2022, 1iuzy) @tsinghua edu,
Abstract
Recently, there has been demand to deploy Large Language Models (LLMs) on personal devices such as laptops and smartphones. These LLMs have different model variants when handling different tasks. However, per- sonal devices have limited resources and require reduced storage overhead. To address this, there are two key methods available: the first is model compression, which compresses LLMs into smaller sizes; the second is LoRA, which can transfer an LLM to other tasks with very few parameters, avoiding the storage of multiple model variants in multi-task scenarios by only preserving LoRAs. However, our experiments show that directly com- bining these two methods yields sub-optimal performance. Considering that the open-source community has already contributed many LoR to LLMs, propose to adapt these existing LoRAs from the LLMs to their compressed version and introduce a Compression-Awar LoRA (CA-LoRA) framework. We incorporate knowledge inheritance and recovery strategies to recover the lost knowledge caused by model compression. Experiment results demonstrate that CA-LoRA outperforms the vanilla LoRA methods applied to a compressed LLM and achieves comparable performance to the non-compressed LLM with existing LoRA modules. The source code of CA-LoRA is available at https:: htlhttss//github.coo gtthtu..ttttom/m/mo
1 Introduction
In recent years, large language models (LLMs) with billions of parameters (Brown et al.
2020: Black et al. 2022 Chowdhery et al. 2023 exhibit more powerful and comprehensive abilities, especially in terms of cognition and embodiment (Lewkowycz et al. 2022 Nakano
eet al. 2021 Driess et al.) 2023). To extend the LLMs to multi-task scenarios (Zhou et al.
2022 Sheng et al.] 2023), parfieter-efficient fine-tuning (PEFT) (Houlsby et 20.9 Hu et al. 2022) has been widely used, where an LLM is frozen as a backbone among different tasks and then tiny tunable PEFT modules are injected learn task-specific knowledge. Compared with conventional full-parameter fine-tuning (FT), where single LLM is tuned into multiple task-specific variants, PEFT tunes much fewer parameters and has lower memory overhead in multi-tasking while achieving comparable performance (Ding et al.,
2023
Among various PEFT methods, LoRA (Hu et al. 2022) is one of the most popular.
Despite the success of LLMs, deploying LLMs in real-world scenarios is an important issue. Online LLM serving well studied (OpenAI (2022 Google 2023), and S-LoRA (Sheng al.
2023) have explored the idea of serving thousands of LoRAs for thousands of tasks around
indicates equal contribution.
indicates corresponding authors.

---

Published as a conference paper at COLM 2024
Compraziiin
FEC
Plocovery Module
LIM
LIM
Thanc
Inhert
LLM
LDPAL
LAAAT
Then
Task
Adapt
CLM
GLM
REG
Training
Daployment
RLM
CLM
CLM
RED
CLM
LLM
Lottk
Lantknn
CLM
Laftis
Thak
Taac
Tank
Trakc
Trsk
(al
Comoress
G-L-RA
S-LoRA
(d) CA-LaRA
Memory!
Storage Friendly
Storage Friendly
Shorage Friendly
Requirs traiinng
LoRVs community
Leveraging ecirting LofAA Apply Quantiztioo, Pruning. MoElication etc.
Only apply Ouamitzation
Figure 1: The training and deployment pattern of previous methods and our CA-LORA.
a single LLM on cloud server. However, how to deploy LLMs on personal devices, such as laptops and smartphones, is still its early stages and faces many challenges (Ggerganov: team 2023), as personal devices have severely limited storage resources compared with cloud servers.
To alleviate the storage constraint, model compression, such as quantization (Lin et al. 2024 pruning (Xia et al. 2024), and MoEfication (Zhang al. 2022b), propose to compress an LLM into a smaller size, which we denote as the compressed LLM (CLM). Q-LoRA (Dettmers) et al. 2023) makes a preliminary attemmp in the scenario of multitasking on personal devices, which proposes to leverage the advantage of LoRA in the multi-task scenario by training LoRA on the compressed LLM. From our experiment, we found two shortages of this approach. Firstly, the LoRA training from scratch the CLM is sub-optimal. Secondly, it only supports quantization, which has a limited model compression ratio and introduces storage overhead personal devices.
Considering that the open-source community has already contributed many LoRAs to LLMs (HuggingFace) we propose adapt those LoRAs from LLM to its compressed version, we call it "Compression-Aware LoRA" (CA-LoRA). We introduce two mechanisms:
(1) LoRA Knowledge Inheritance. A stronger LLM can make learning LoRA modules easier. Meanwhile, the LoRA modules based on the stronger LLM can better grasp task-specific knowledge. Therefore, we propose to adopt the existing LoRA modules learned on the LLM as the initialization to learn the LoRA modules for its compressed version. In this way, the task knowledge of the LLM can be inherited obtain more effective LoRA modules for the CLM, as shown in Figure
(2) Model Knowledge Recovery. Whether LoRA modules trained on an LLM can work well with its compressed version is still an open problem, especially considering that model compression techniques may introduce knowledge loss and performance degradation. To recover the lost knowledge caused by model compression, we propose to inject low-rank non-linear recovery module into the CLM to bridge the knowledge gap that arises from the compression process. We point out that compression techniques may weaken multiple capabilities of the LLM while only a small number of the capabilities it loses are required for specific task. Therefore, the recovery module is also task-specific.
In experiments, CA-LoRA on CLM can recover the performance to the level of existing LoRÁ on LLM. These findings highlight the potential of CA-LoRA effective solution
2

---

Published as a conference paper at COLM 2024
for adapting existing multi-task ability from cloud servers to personal devices, thereby advancing the development personal Al applications.
Related Work
This section mainly introduces PEFT and model compression. More details on LLMs can
refer to surveys (Qiu et al. 2020 Han et al. 2021 [Bommasani et al., 2021 Zhao al. 2023).
2.1 Parameter-Efficient Fine-Tuning
Although LLMs can acquire rich knowledge from massive pre-training data to handle complex tasks in a zero-shot or few-shot manner (Brown et al. 2020 Black et al. 2022), there is still need for adapting LLMs, to better stimulate the knowledge within LLMs to serve downstream tasks. For traditional PLMs, fine-tuning all the parameters of PLMs is the mainstream way to adapt them (Church et al. 2021), yet the parameter inefficiency makes this way costly to adapt LLMs (Ding et al. 2023).
To adapt LLMs to different tasks in more efficient manner, various PEFT methods Lester
et al. 2021 Houlsby et al. 2019 Hu et al. 2022: Li & Liang/ 2021 Ben Zaken et al.) 2022
have been proposed. Specifically, L.MSs are Frozen and some mooee--ndeennden tunable modules are injected into LLMs to help the adaptation process. PEFT modules are usually tiny so that PEFT methods can significantly reduce the cost of adapting LLMs. Among PEFT methods, LoRA is the most popular.
2.2 Model Compression
To improve the efficiency of LLM deployment, is crucial to speed up the computation of LLMs. Model compression is commonly used solution. Here we introduce task-agnostic model compression (Sanh et al., 2019) rather than task-specific compression (Sun et al. 2019), because task-specific compression not memory-friendly in multi-task settings. lask-agnostic compression techniques include Quantization, Pruning, and MoEfication.
Quantization methods represent floating-point number LLM into fixed-point one, from 8-bit (Zffrir et al.] 2019), 4-bit (Frantar et al.] 2023 Lin et al. 2024) to 1-bit (Bai et al., Xu al., 2024b). To avoid the performance degradation caused by quantization, training (QAT) (Stock et al. 2021) has been proposed to use small amount of data to adjust the distribution of model parameters for quantization. Different from quantization methods that compress parameter representations, pruning methods directly discard some parameters. Commonly used pruning methods include structured pruning (Fan et al., 2020; Wang et al. 2020 Zhang et al. 2021: Xia et al. 2022 2024) and unstructured pruning (Han et al. 2015 Chen et al. 2020 Xu al. 2021). MoElication (Zhang et 2022b),, inspired by the mixture-of-expeee
2021
quantization-awal
(MoE) transtormer [epikhin et 2021),
aims to divide LLM parameters into multiple partitions, and each time only tew partitions are used process input data. Typically, make a compressed LLM behave the same as its original version, distillation objectives are often used to align the LLM and CLM, including aligning both output and intermediate states (Hinton al. 2015 Sun et al. 2019 Jiao al.. 2020 Liu et al. 2022 Park et al., 2021). Zhang et al. (2022a) combines the above quantization pruning, MoEfication and distillation into a mixed compression method, achieving a high
compression ratio. More compression methods can refer to surveys (Liang et al. 2021 Xu &
McAuley 2022).
Recently, language models less than 3 billion parameters have been proposed (Bai et al.
2023: Javaheripi & Bubeck 2023 Zhang et al., 2024; Hu et al. 2024 Liu et al.) 2024b), while
Xia et 2024)
Zhang et al. (2022a) state that compressing M is better than small models.
Currently, combining LoRA with model compression is preliminary, and only some works attempt to combine LoRA with quantization (Dettmers et al. 2023 Liu et al. 2024a Xu et al.) 2024a). Recent work (Zhao et al. (024) also attempts to conduct PEFT while compressing, but they are task-specific compression.
3

---

Published as a conference paper at COLM 2024
2.3 Multi-tasking
Maintaining multiple task-specific versions of an LLM in the storage is unacceptably resource-intensive (Zhou et al., 2022). As shown in F full-model finetunes need to be done in vanilla fine-tuning for different tasks, posing significant challenges to the storage. In previous fine-tuning-then-e comp-ssnnng methods, this problem still exists.
Different LoRA modules can share a unified frozen LLM as their backbone, leading to lower computation and storage overhead in multi-task scenarios. PetS (Zhou et al. 2022) and S-LoRA (Sheng et al. 2023) use LoRA for multi-task serving on cloud servers. QLoRA (Dettmers et al. 2023) applies LoRAs on CLM, which is suitable for personal devices.
3 Methodology
3.1 Preliminary
For simplicity, denote an LLM M as Y Y = f(x, where f(.) is the function of the whole transformer architecture, M is the parameters the LLM LLM,, the input, and is the output.
In the FT setting, all parameters of M (i.e., M are tuned as follows
8% M
arg min ((f((x;0x)) 8M
(1)
where (X', Y') the data of the downstream task t, L is the loss function of the task f. 0'M is the final task-specific parameters of the LLM MM.
In the LoRA setting, M is frozen, and additional LoRA modules P are tuned on task-specific data. We denote the parameters of the LoRA modules injected into M as P((M)- As shown in Figure [2] the computation of the transformer architecture is slightly changed due to
the injected LoRA modules and becomes Y = fLORA (X; 8 M,, PM;)) The tuning process is formalized as
P((M) = arg min L(fLORA M, (M, PPIM
(p(M)),
(2)
where 6 ((()) is the final task-specific LoRA modules collaborating with the LLM M.
3.2 Framework
We propose more efficient LoRA framework CA-LoRA, by adapting existing LoRA on an LLM into the compressed L.LM (CLM).
As for the compression method, we choose task-agnostic model compression (Sanh et al., 2019) rather than task-specific compression (Sun et al., (019) because task-specific com- pression is not memory-friendly in multi-task settings. After compressing the LLM M, making M have fewer parameters or lower-bit representations, we denote the compressed C and 0c respectively. Then the computation of
LLM (CLM) and its parameters the compressed model can be described as Y = [X; 0c) without LoRA modules, and Y = fLORA (X; 0g Op(cc) with LoRA modules, where (p(c) indicates the parameters of the LoRA modules injected into the CLM C. CA-LoRA can be formalized as
0 Oic = arg min L(LLORA (x'; Bg, 0 0p(c)), Y*). Opic)
(3)
By combining model compression and LoRA, we can deploy unified LLM to serve multiple tasks, while only maintaining tiny task-specific LoRA modules for each task. It is not difficult to imagine that adopting task-agnostic compression methods may weaken model performance, which will inevitably affect the search for the optimal parameters 0 ('((e) and
the effect of the final model fLORA (X; 0c, 0 p:(c)).
Inspired by the fact that model compression preserves those capabilities LLMs that training smaller model from scratch cannot master, we suppose that the LoRA modules
4

---

Published as a conference paper at COLM 2024
LoHA B
Flecover
Weight Matrix
Compressed
B
A
LoRA
LORA
Fleovvrr
(a) Existing LoRA on LLM
Adapting LoRA to CLM
Figure 2: The overall design of our CA-LoRA.
trained on the non-compressed LLM would contain certain task knowledge that the LoRA modules can hardly learn solely on the CLM. As shown in Figure to better learn LoRA modules for the CLM, we adopt the method of inheriting the LoRA knowledge from those modules trained on the non-compressed LLM. To recover the lost knowledge caused by the compressing process, in addition to the LoRA modules P, we add some knowledge recovery modules R.
3.3 LoRA Knowledge Inheritance
Instead of training LoRA modules based on the CLM from scratch, we propose to use existing LoRA modules of the LLM as their initialization, then adapting them to the CLM. The adaption can lead to learning better LoRA modules on the CLM. Intuitively, it is more effective for teacher to teach students the fundamentals of discipline and then let students adapt their comprehension based on their circumstances rather than letting students learn from scratch. Formally, for task f, we first use existing LoRA module /
P(MM) trained on
the non-compressed LLM M as the initialization of the LoRA module (p(c) the CLM
After that, PC()) is further tuned to 0f P(c) on the data of the task by using Eq. (3)
3.4 Model Knowledge Recovery
We propose to inject the knowledge recovery modules R into the CLM C to recover the lost of common knowledge during compression. As shown in Figure 2 we add a bypass next to each linear layer of the transformer architecture add small amount of change to the output states of those linear layers. To avoid introducing too many parameters, the recovery modules R adopt low-rank structure like LoRA. Since LoRA is low rank while the compression of the model is often not of low rank, we have added additional non-linear activation between to express some non-low-rank structures better. Formally, we denote the recovery module as
R(X) (XD)U,
where D is the down projection matrix, o(.) is the activation function, and U is the up projection matrix. D and U together form the parameter of the recovery module 0g-
To help obtain the optimal parameter 6 of the recovery module for the task f, we design distillation objective. Specifically, we select the existing LoRA modules trained with Eq. and the LLM as the teacher. The whole distillation loss is given as
Cxist M Oc Op(c), 0R)=
(5)
X fLAA (X':0 P(( FLORA (x's (X': 0c, 0
1R)
where X is the input data of the task f. Instead of first learning the recovery modules and then adding the inherited LoRA modules for further adaptation, we simultaneously conduct knowledge recovery and tune LoRA modules. The overall optimization problem

---

Published as a conference paper at COLM 2024
will become
6 Prcy 0
arg min Opic).R
[fLORA (x'; Oc, pp(c), 0R),
(6)
+ + 0 0g Opey, 0x)].
Although CA-LoRA requires the non-compressed model to participate in the training process and results in extra training time, is worth sacrificing little training cost for better task performance.
4 Experiments and Analyses
4.1 The Performance on Typical NLP Tasks
Datasets We evaluate CA-LoRA on 11 typical NLP datasets, more details are in Appendix A
Baselines and Implementation Details We select T5-3b (Raffel et al. 2020) as the LLM and select its CLM from Zhang et al. (022a),, as shown Table Details about compressing T5-3b are in Appendix We compare the following 4 paradigms: (1) T5-3b LoRA: LoRA
modules are attached to the original T5-3b, and only the parameters of LoRA modules
are tunable while the LLM is frozen. (2) T5-base + LoRA:: Tunable LoRA modules are
attached to the frozen T5-base model. (3) CLM + LoRA: LoRA modules are attached to the compressed versions of T5-3b, and then these CLMs are frozen and LoRA modules are tuned on task-specific data. (4) CLM + CA-LoRA: CA-LoRA is applied to the compressed versions of T5-3b, and only LoRA and recovery modules are tuned task-specific data.
We set the bottleneck dimension of the LoRA modules and CA-LoRA recovery modules to 32 in all paradigms. The tunable parameters of LoRA and CA-LoRA modules are about 20M and 120M, respectively. More experimental settings are in our appendix.
Table 1: The models used in the experiments. The notation "T5-3b (X)" represents the T5-3b model with the setting "X". "M", "UP", "SP", and "Q" represent the model compressed with MoEfication, unstructured pruning, structured pruning, and 8-bit quantization, respec- tively. "Q+UP+M" means "Q", "UP" and "M" are combined together to achieve higher compression ratios. "bf16" indicates the model is represented in bfloat16 seating-point rather than 32-bit floating-point format.
Model
Model Size
Ideal Speedup
T5-3b (bf16)
5.61 CB
100%
T5-3b (M)
3.74 GB
150%
T5-3b (UP)
2,81 GB
200%
T5-3b (SP)
2.81 CB
200%
T5-3b (Q)
2.81
200%
T5-3b (Q+UP+M)
600%
ba-base (bf16)
0.44
1400%
The Overall Performance
Figure 3 shows that:
(1) The CLMs cannot perform well as the original LLM, suggesting that task-agnostic com- pression leads to losing some task-specific knowledge. The performance of the compressed model may decrease due to the acceleration. If mechanism makes up the performance gap without affecting the inference speed, applying compression will be more reasonable.
(2) CA-LoRA consistently outperforms vanilla LoRA methods. Such results indicate that task capabilities are effectively migrated through LoRA knowledge inheritance and model knowledge recovery mechanisms.
(3) Quantization and MoEfication have relatively little degradation on the model perfor- mance, and the degradation can be completely restored using CA-LoRA. However, the

---

Published as a conference paper at COLM 2024
Quatization
15 P03
1902
LARA
Acarsry
15-4
Doo Q
CB
COPA
WIC
55T2
VALLI
Q0?
VIVC
ONLI
SQUAD
NoBrication
100
Actureyy tred
50
EXUS
CLU Lalts
Bon
WYC
SGT2
VAUL-
00P
Acoraay (Caact (80)
100
75-0
20-
La
60
10
Unstructured Pruniin
Ascurey Daad Hotch
BCA
CLM CAEEAA
$0010
CB
ATE
COPA
Wic
5512
000
WPPC
ONU
SQUAD
Figure 3: For typical NLP tasks, the results of LoRA and CA-LORA on different CLMs (%).
Table 2: The results of applying CA-LoRA the CLM T5-3b (Q+UP+M) that comes from the mixture of multiple compression methods (%).
Model Size(GB)
BoolQ Acc(%)
COPA Acc(%)
WIC Acc(%)
Method
Acc(%)
Aoc(%)
Acc(%)
15-3b + LoRA
5.61
88.3
100.0
88.6
88.0
74.0
96.1
T5-base + LoRA
0.44
79.5
91.1
80.7
71.0
69.9
93.5
0.94
CLM + CA-LoRA
0.94
87.1
100.0
84.3
86.0
72.0
96.2
MNLI-m Acc(%)
QOP
QQP
MRPC
ONLI Acc(%)
SQuAD EM(%)
SQuAD
Method
Acc(%)
F1(%)
Acc(%)
F1(%)
T5-3b LoRA
90.6
91.3
90.7
89.5
95.4
84.2
92.5
T5-base LoRA
54.8
90.6
89.9
86.5
93.1
79.0
87.8
CLM LoRA
89.0
90.6
89.9
89.7
94.7
79.9
90.6
CLM CA-LORA
89.5
90.5
pruning methods cause more degradation, especially the structured pruning method. Even though, CA-LoRA recovers most of the performance loss caused by pruning methods.
To further evaluate CA-LoRA on higher compression ratio, we use T5-3b (Q+UP+M) in Table [ which is a CLM obtained from mixing quantization, MoEfication, and unstruc- tured pruning, and has close size to T5-base. Table shows the experimental results. Results show that CA-LoRA is compatible and can be easily applied to highly compressed model that uses multiple compression methods. Meanwhile, CA-LoRA demonstrates better performance on CLMs than training small model with close size from scratch.
4.2
The Performance on Instruction Tuning
The above experimental results have proven the strength of CA-LoRA for specific down- stream tasks. In this section, we investigate the effectiveness of CA-LoRA when CA-LoRA is applied more general scenario, instruction tuning.
7

---

Published as a conference paper at COLM 2024
Table 3: The results LoRA and CA-LoRA based on different compressed LLaMA (%) "rL" and "rr" respectively represent the intermediate rank of LoRA and Recovery modules. "+" represents that we couldn't reproduce QA-LoRA and the results are from the original paper.
Methed
FL
Fx
STEM
Human
MMLU Social
Other
Mean
HumanEval
BBH
Drop
Overall Mean
LLaMa-136 13b
0
36.4
44.9
34.1
53.1
47.0
17.7
36.6
35.0
34.1
BMQuanl Zhange et al 2a02a
0
0
34.6
44.5
52.5
53.1
46.1
16.5
37.2
32.3
33,0
NF4
Dethneess 2023
34.1
42.5
51.9
51.1
44.7
38.6
33.7
32.0
QA-LoRA
(u et al. 2024a
0
64
38.3
48.4
54.9
55.2
49.2
BMQuant
16
16
37.6
45.1
54.3
54.1
47.6
17.7
35.8
33.3
33.6
QL (Detmmers
47.7
37.6
34.6
33.2
QLoRA/CA-LORK /CV-LORI
16
16
37.6
44.4
34.7
54.8
47.6
18.3
37.2
33.6
34.2
4.2.1 General Instruction Tuning
Datasets We select Alpaca (Taori et al. 2023) as the instruction tuning dataset and use 5-shot MMLU (Hendrycks al., 2021) the benchmark. We also use HumanEval (Chen et al., 2021
BBH (Suzgun et al. 2023), and Drop (Dua et 2019) for evaluation.
Baselines and Implementation Details Quantization is the compression method for instruction tuning. Therefore, we compress LLaMA-13b (Touvron et al. (202a)) into an 8-bit quantization version for experiments, labeled "BMMuant",, using the quantization method in Zhang et al. (2022a). We also add 4-bit quantization NF4 baseline, which is the quantiza- tion method of QLoRA (Dettmers et al. 2023). Baseline methods are tested without instruc-
tion tuning on Alpaca. To evaluate CA-LoRA we adopt 4 paradigms: QLoRA (Dettmers)
et al. 2023) and QA-LoRA (Xu et al. 2024a) for the previous compressing-thee menssing-then-tuune then-tuning methods and RMQuant+CA-Lokk tuning methods. More details of the LLaMA experiments can be found in Appendix
and QLORA+CA-LORA for our compression-awane
The Overall Performance Table 3 shows that quantization leads to overall performance degradation. BMQuant-CA-LokkA and QLoRA+CA-LORA achieve better overall perfor- mance than other methods, indicating that the design of CA-LoRA is effective. QLoRA+CA- LoRA obtains better performance compared with QLoRA, showing the CA-LoRA can achieve faster inference speed and better performance with fewer trainable parameters.
4.2.2 Task-specific Instruction Tuning
Datasets We select CodeAlpaca-200 Chaudhary 2023) as the instruction tuning dataset and use 0-shot HumanEval (Chen et al. 2021) for evaluation.
Baselines and Implementation Details We compress LLaMa-2-13b (Tovrron et al. 2023b
into an NF4 version, which is the quantization method of QLoRA (Dettmers et al. 2023)
We then compare LoRA, QLoRA (Dettmers et al. 2023) and QLORA-CA-LORA, The inter- mediate rank of the LoRA and Recovery modules is set to 64. We use zero-shot evaluation of LLaMa-2-13b and its NF4 version as baselines.
The Overall Performance Figure shows that QLoRA+CA-LORA achieves better overall performance than baselines, indicating that the design of CA-LoRA is also effective in task-specific instruction tuning.
4.3
Ablation Studies
To more clearly demonstrate the inner mechanisms of CA-LoRA, we conduct ablation studies using the T5-3b (Q+U++M), as in Ta and adopt RTE for evaluation. For CA-LoRA, the intermediate ranks of LoRA and Recovery modules are 8. LoRA modules are injected into linear transformations of Wº and Wk in attention layers. Recovery modules are injected into all linear transformations in all layers.
Table shows how do different mechanisms in CA-LoRA help recover the knowledge loss caused by compression methods:
8

---

Published as a conference paper at COLM 2024
HumanEval Accuracy (16)
1.4
inhertt
17
LoRA
QLORA
L--M3-
NF4
15
# Train Epochs
Figure 4: The performance of LoRA, QLoRA and CA-LoRA HumanEval by instruction- tuning on CodeAlpaca-200k
Table 4: The ablation studies of CA-LoRA on RTE (%). When eliminating inheritance, we keep the LoRA modules injected and train these modules from scratch. When eliminating recovery, we simply remove the recovery modules. When eliminating distillation, the training loss function only consists of the task loss.
Inherit
Recover
Distill
RTE Acc
83.6
85.4
/
82.5
82.5
87.5
85.7
d
86.1
8
88.6
(1) LoRA knowledge inheritance is effective. By initializing the tunable parameters with the existing LoRA modules trained on the non-compressed LLM, the performance of combining the final LoRA modules and the compressed LLM has been significantly improved. This indicates that in the optimization space of the compressed LLM, it is difficult to obtain the optimal task-specific parameters of LoRA modules based on random initialization, but more optimal LoRA modules can be achieved more easily using LoRA knowledge inheritance.
(2) Simply adding recovery modules brings certain level of performance improvements in some circumstances. However, it can achieve further performance improvements by adopting our distillation strategy. This suggests combining recovery modules with the knowledge distillation strategy to enhance LoRA modules is necessary.
More ablations can be found in Appendix
4.4
The Convergence of CA-LoRA
Based on four CLMs of T5-3b (as in Figure 3) we test the convergence speed on BoolQ dataset. From Figure 5) we can find that due to our LoRA knowledge inheritance mecha- nism, CA-LoRA is superior the vanilla LoRA methods terms of convergence speed and final results. Adding the recovery module will not affect the convergence speed. Further- more, when quantization, unstructured pruning, or MoEfication is used, the inheritance mechanism gives better starting point for tuning LoRA modules. While in the case of structured pruning, even though the initial point CA-LoRA does not work well on tasks, it is closer to the optimal point in the optimization space and converges faster.
Moreover, considering the existing LoRA modules based on the LLM may be trained by community users on their own private data and then uploaded to the Internet. When

---

Published as a conference paper at COLM 2024
Quanitzation
Unstructred Pruning
Structured Pruning
MoEfication
90
60
70
LafA
30
20
20
13
15
opech
opoch
opoch
opoch
Figure 5: The convergence of vanilla LoRA, inherited LoRA (CA-LoRA without recovery modules), and CA-LoRA.
adapting these LoRA modules to the CLM, there may not be any task-specific data available for the adaptation process. In this case, we can directly use our LoRA inheritance mechanism without any fine-tuning while still achieving good performance on the CLM, when the compression method is quantization or MoEfication.
Conclusion
We propose an effective LoRA framework for adapting existing LoRA on LLM to the compressed LLM (named CA-LoRA) to deploy language models for multi-tasking on personal devices. Considering task-agnostic compression may cause losing task-specific knowledge, we introduce LoRA knowledge inheritance and model knowledge recovery to recover the lost knowledge. By inheriting the prior task knowledge of the LoRA modules learned on the non-compressed LLM, searching for the optimal LoRA modules for the compressed LLM becomes easier. Moreover, by introducing knowledge recovery modules to recover task-specific capabilities lost in the compression phase, collaborating LoRA modules with the compressed LLM can achieve comparable performance to those LoRA modules based on the non-compressed LLM.
Limitations
Our work focuses on the efficiency and effectiveness, but it requires small amount of extra training time. In this paper, we only choose LoRA as representative of PEFT methods. In fact, our framework can be applied to any other PEFT methods. We focus on algorithmic support for running CA-LORA on personal devices, but not the actual deployment implementation. The compression method adopted in this paper does not change the number of layers the LLM. However, for compression methods that change the hidden dimensions of the model, how transfer the knowledge of LoRA modules the non-compressed LLM remains an open problem for our future work.
Ethic Statement
This paper presents work whose goal is to advance the field of Machine Learning, There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Acknowledgement
This work is supported by the National Key R&D Program of China (No.2022ZD0116311 National Natural Science Foundation of China (No. 62236004), and Institute Guo Qiang at Tsinghua University. Weilin Zhao is supported by Tsinghua University Initiative Scientific Research Program. Yuxiang Huang is supported by Tsinghua University Initiative Scientific Research Program (Student Academic Research Advancement Program).
10

---

Published as a conference paper at COLM 2024
References
Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jing Jin, Xin Jiang, Qun Liu, Michael Lyu, and Irwin King. Binarybert: Pushing the limit of bert quantization. In Proceedings of ACL/IJCNLP, pp. 4334-4348, 2021.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXin:2309.16609 2023.
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. BitFit: Simple pframeter-efficient fine-tuning for transformer-basee masked language-models, In Proceedings of ACL, PP- 1-9, 2022.
Luisa Bentivogli, Bernardo Magnini, Ido Dagan, Hoa Trang Dang, and Danilo Giampiccolo. The fifth PASCAL recognizing textual entailment challenge. In Proceedings of TAC, 2009.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT-NeoX-20B: An open-source autoregressive language model. In Proceedings of ACL Workshop, 2022.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXio:2108.07255 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Proceedings of NeuriPS, volume 33, pp. 1877- 1901, 2020.
Sahil Chaudhary, Code alpaca: An instruction-ffollovingg llama model for code generation. http:///ggtttbbocccmaaaa////////
2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evalu- ating large language models trained on code. arXiv preprint arXiv:2107.0333 2021.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. In Proceedings of NeurlPS, volume 33, pp. 15834-15846, 2020.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gauray Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. JMLR, 2023.
Kenneth Ward Church, Zeyu Chen, and Yanjun Ma. Emerging trends: A gentle introduction to fine-tuning. Natural Language Engineering, 27(6):763-778, 2021.
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no no questions. In Proceedings of NAACL, 2019.
Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: Investigating projection in naturally occurring discourse. In Proceedings of Sinn und Bedeufung, volume 23, pp. 107-124, 2019.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized lims. Xrv preprint ar xrxo:23305.14311 2023.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, pp- 1-16, 2023.
11

---

Published as a conference paper at COLM 2024
William B. Dolan and Chris Brockett Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. PaLM-e: An embodied multimodal language model. In Proceedings of ICML, 2023.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proc. of NAACL, 2019.
Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. In Proceedings of ICLR, 2020.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate quantiza- tion for generative pre-trained transformers. In Proceedings of ICLR, 2023.
Ggerganov. Ggerganov /llmaa.ppp: Llm inference in c/c++. URL https://github. com/
ggerganov/llania.or cpp
Google, Google ai palm 2, 2023. URL
Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural network. In Proceedings of NeurlPS, pp. 1135-1143, 2015.
Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Yuan Yao, Ao Zhang, Liang Zhang, et al. Pre-trained models: Past, present and future. Al Open, 2:225-250 250,, 2021.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In Proceedings of ICLR, 2021.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.02531 2015.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Larous- silhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In Proceedings of ICML, pp. 2790-27999 PMLR, 2019.
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In Proceedings of ICLR, 2022.
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arviv preprint arXio:2404.06335, 2024.
HuggingFace. URL
Mojan Javaheripi and Sébastien Bubeck. Phi-2: The surprising power of small lan-
guage models, Dec 2023. URL https:::: //www//wwwwwww// mi www.ooooioooooooommmmmomooooo.tt wwtmm//nnuu//eserrc//bbb/
phi 1-2-the-uurpriiingg -wwpgg--owe-----ooo wwwsssmmaaaassseaeeeeaaaaaaaaaaa -aap::/sinannnooooooo-aaaaaaaaaa
models/
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and
Qun Liu. TinyBERT: Distilling BERT for natural language understanding. In Findings of EMNLP, pp. 4163 2020.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam M. Shazeer, and Z. Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. In Proceedings of ICLR, 2021.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parimeter-efficient prompt tuning. arXiv preprint ar 21140.2104.08611, 2021.
12

---

Published as a conference paper at COLM 2024
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. In Proceedings of NeurlPS, 2022.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of ACI-IJCNLP,, pp. 4582 2,7244977,222 2021.
Tailin Liang, John Glossner, Lei Wang, Shaobo Shi, and Xiaotong Zhang. Pruning and quantization for deep neural network acceleration: A survey. Neurocompptiing, 461:370- 403, 2021.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangx- uan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for Ilm compression and acceleration. In Proceedings of MLSys, 2024.
Chang Liu, Chongyang Tao, Jiazhan Feng, and Dongyan Zhao. Multi-granularity structural knowledge distillation for language model compression. In Proceedings of ACL, pp. 1001- 1011, 2022.
Peiyu Liu, Zikang Liu, Ze-Feng Gao, Dawei Gao, Wayne Xin Zhao, Yaliang Li, Bolin Ding, and Ji-Rong Wen. Do emergent abilities exist in quantized large language models: An empirical study. In Proceedings of LREC-COLING, 2024a.
Zechun Liu, Changsheng Zhao, Forrest landola, Chen Lai, Yuandong Tian, Igor Fedorov, Yunyang Xiong, Ernie Chang, Yangyang Shi, Raghuraman Krishnamoorthi, Liangzhen Lai, and Vikas Chandra. MobileLLM: Optimizing sub-billion parameter language models for on-device use cases. In Proceedings of ICML, 2024b.
Reicchiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser- assisted question-answeriing with human feedback. arXiv preprint ar 2122:2212.,,,33,, 2021.
TB OpenAI. Chatgpt: Optimizing language models for dialogue. OpenAl, 2022.
Geondo Park, Gyeongman Kim, and Eunho Yang. Distilling linguistic context for language model compression. In Proceedings of EMNLP, pp. 364-378, 2021.
Mohammad Taher Pilehvar and Jose Camacho-Colladee Wic: the word-in-context dataset for evaluating context-sensitive meaning representations. arXiv preprint ar Xio:18880 (tto:800.0911. 09122, 2018.
Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. Pre- trained models for natural language processing: A survey. Science China Technological Sciences, 63110):187--1897 2020.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. JMLR, 21:1-67, 2020.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of EMNLP, 2016.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. Proceedings of AAAI, pp- 90-95, 2011.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXio:1910.01108 2019.
Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, et al. S-lora: Serving thousands of concurrent lora adapters. arXiv preprint ar xix::2311.002888 2023.
13

---

Published as a conference paper at COLM 2024
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of EMNLP, pp. 1631-1642, 2013.
Pierre Stock, Angela Fan, Benjamin Graham, Edouard Grave, Rémi Gribonval, Herve Jegou, and Armand Joulin. Training with quantization noise for extreme model compression. In Proceedings of ICLR, 2021.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for BERT 2019.
model compression. In Proceedings EMNLP-IJCNLP, pp.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. Challeng- ing BIG-bench tasks and whether chain-of-thought can solve them. Findings of the ACL, 2023.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following Ilama model. httpa///tt.//t//////t//////////n
alaca 2023.
MLC team. MLC-LLM, 2023.
www.:lc-ai/mlmll//ccccccoc mmmm
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint ar arXi::2302.13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama Open foundation and fine-tuned chat models. arXiv preprint ArX::2230 099888, 2023b.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of EMNLP Workshop BlackboxNLP, 2018.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general- purpose language understanding systems. In Proceedings of NeurIPS, volume 32, 2019.
Ziheng Wang, Jeremy Wohlwend, and Tao Lei. Structured pruning of large language models. In Proceedings of EMNLP, pp. 6151-6163 2020.
Adina Williams, Nikita Nangia, and Samuel Bowman. broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint ar vrXiv:1704.04422 2017.
Mengzhou Xia, Zexuan Zhong, and Danqi Chen. Structured pruning learns compact and accurate models. In Proceedings of ACL, pp. 1513-1528, 2022.
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared LLaMA: Accelerating language model pre-training via structured pruning. In Proceedings of ICLR, 2024.
Canwen Xu and Julian McAuley. A survey on model compression for natural language processing. arxiv preprint ar arXiv:220..07105, 2022.
Dongkuan Xu, Ian En-Hsu Yen, Jinxi Zhao, and Zhibin Xiao. Rethinking network pruning -under the pre-train and fine-tune paradigm. In Proceedings of NAACL, pp. 2376-2382, 2021.
Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, Xiaopeng Zhang, and Qi Tian. QA-loRA: Quantization-aware low-rank adaptation of large language models. In Proceedings of ICLR, 2024a.
Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, and Wanxiang Che. Onebit: Towards extremely low-bit large language models. arxio preprint ar XrXiu:2402.1122 2024b.
14

---

Published as a conference paper at COLM 2024
Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8bert: Quantized 8bit bert. In Proceedings of EMC2-NIPS, pp. 36-39, IEEE, 2019.
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. Xiviv preprint ar Xio:2401.02383 2024.
Zhengyan Zhang, Fanchao Qi, Zhiyuan Liu, Qun Liu, and Maosong Sun. Know what you don't need: Single-shot meta-pruning for attention heads. AI Open, 2:36-42 42, 2021.
Zhengyan Zhang, Baitao Gong, Yingfa Chen, Xu Han, Guoyang Zeng, Weilin Zhao, Yanxu Chen, Zhiyuan Liu, and Maosong Sun. Bmcook: A task-agnostic compression toolkit for big models. In Proceedings of EMNLP Demonstration, pp. 396-405, 2022a.
Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. MoEfi- cation: Transformer feed-forward layers are mixtures of experts. In Findings of ACL, PP. 877-890, 2022b.
Bowen Zhao, Hannaneh Hajishirzi, and Qingqing Cao. APT: Adaptive pruning and tuning pretrained language models for efficient training and inference. In Proceedings of ICML, 2024.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yinggian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. ar Xiv preprint arXit:2303.18223, 2023.
Zhe Zhou, Xuechao Wei, Jiejing Zhang, and Guangyu Sun. PetS: A unified framework for Parameter-Efficien transformers serving. In Proceedings of ATC, pp. 489-504, 2022.
15

---

Published as a conference paper at COLM 2024
A The Experimental Settings of Typical NLP Tasks
The typical NLP tasks to evaluate our methods including BoolQ Clarr et al. 2019).
CB (De Marneffe al. 2019 RTE (Bentivogli et al. 2009 Wang et al. 2019), COPA Roem
mele et al. 2011), WIC (Pilehvar & Camacho-Collados 2018), SS1-2 (Socher et al.
2013),
MRPC (Dolan & Brockett, 2005), QQP (Wang et al. 2018), MNLI (Williams et al.
2017).
QNLI Rapurkar et al. at 2016), and SQuAD (Rajpurkar et al. 2016
For compressed models, we use the compressed versions of T5-3b released by Zhang et @fal.22022)) including 8-bit quantization, structured pruning, unstructured pruning, and MoEfication. Table 1 shows the models used in our experiments and their ideal inference speedup compared to T5-3b (Zhang et al., (022a). Under this experimental setting, all paradigms are implemented with the open-source toolkit Open (Ding et al. 2023). We select the best learning rate among (le 3, 3, le 4, le 5). The batch size among {8, 16, 32, 64, 128, } The weight decay is le- 2. The distillation coefficient in Eq. 6 is
default to a 0.05.
B The Experimental Settings of Instruction Tuning
For Alpaca-LoRA and QLoRA, we test the released checkpoint. Alpaca-LoRA adds LoRA modules on every Q K, V,O projection and QLoRA adds LoRA modules to every linear transformation. In order to inherit from Alpaca-LoRA in QLoRA+CA-LORA setting, the positions where LoRA modules are added are same as Alpaca-LoRA. For BMQuant+CA- LoRA and QLoRA+CA-LORA, we first apply the compression method to the backbone model, then we apply the LoRA modules and Recovery modules. We inherit from Alpaca- LoRA, then samely distill the modified model with Alpaca-LoRA The positions of adding LoRA modules are the same as Alpaca-LoRA and Recovery modules are added to all linear transformations. The intermediate rank of additional modules are introduced in Table All training settings follow
et al. 2023)
C More Ablation
Is the improvement only caused by increased parameters?
We carry out the study in Table 5 Each setting in this study maintains at least the same number of parameters CA-LORA. A. In CA-LoRA, the tunable parameters consist of LoRA modules and recovery modules. We introduce new settings of LoRA+LORA and Large LoRA. More specifically, in LoRA+LORA settings, we first inject LoRA modules to Wº and W in attention layers, like conventional LoRA, and then inject extra LoRA modules to all linear transformations in both attention and feed ferd-forward layers, replacing the recovery modules in CA-LORA. Both LoRAs have an intermediate rank of 8. We then apply distillation to the modified model. In Large LoRA settings, we only inject LoRA modules with intermediate rank of 32 on Wº and WK. From Table 5 we find that adding more parameters makes marginal improvements, showing that only adding more tunable parameters cannot bridge the knowledge gap caused by model compression.
Table 5: The studies controlling parameter quantity (%).
Method
RTE Acc
LoRA
83.6
LoRA+LoRA
85,4
Large LoRA
81.8
CA-LoRA
88.6
Does adding more parameters slow down model inference?
16

---

Published as a conference paper at COLM 2024
Since CA-LoRA introduces little more parameters than conventional LoRA methods, it has a potential cause of slower inference. Considering the inference speed of compressed models is highly related to implementation, methods with the same backbone on the same platform to avoid uncertainties caused by backbone implementation. We report the average time and corresponding standard deviation of each model call. Table ② and Table 7 show that the inference time of CA- LoRA and LoRA methods are almost the same, proving that inference bottleneck lies in the model itself, while the extra parameters added due to CA-LoRA do not slow down inference.
evaluate CA-LoRA and conventional LoRA
From the ablations above, we prove that each mechanism introduced in CA-LoRA is helpful recover performance degradation of model compression without retarding model inference.
The structural design of CA-LoRA is reasonable, which is beneficial to knowledge inheri- tance and recovery.
Table 6: The average inference time on Alpaca dataset. The average time and corresponding standard deviations represent the time taken for each model call.
Method
Avg. Time (ms)
Var. Time (ms)
QLoRA
226.58
19.23
QLORAICA-LORA
230.87
19.25
Table 7: The average inference time, "CLM" here represents the compressed T5-3b using mixture of compression methods. "# Param" represents the additional parameters com- pared with the backbone model. The average time and corresponding standard deviations represent the time taken for each model call.
Method
# Param
Avg. Time (ms)
CLM+LoRA
10M
(9761+117)
CIMACA-LORA
60M
(9526++70)
17


Reviews:
[{"review_id": "aCJgen9Db8", "reviewer": "Reviewer_Zo74", "summary": "The authors present CA-LoRA, a compression-aware parameter-efficient finetuning method based on LoRA. They train LoRA modules on an uncompressed LLM and then adapt them downstream to be used with a compressed version of the same LLM, with a relatively small number of additional parameters introduced in addition to the LoRA modules. They explore two model families -- T5 and Llama 2 -- across both SuperGLUE and instruction tuning tasks, and see noticeable improvements in nearly all scenarios they tested.\n\nThe experiments are reasonable, the idea is straightforward yet novel, and motivation is sound. The significance of this work lies in their stated motivation: enabling strong, static (or rarely updated) task-specific models to be deployed in resource-constrained environments, such as mobile devices. As far as I am aware, this is a novel method for adapting compressed language models to downstream tasks. They are also clear about limitations: this method requires extra compute to adapt the full sized model before being compressed. Whether this tradeoff is worth it depends on the specific training and deployment scenario.\n\nThe structure of the paper is good, but the clarity could be improved. For example, I am not sure why the extra parameters introduced are called “knowledge recovery modules” – based on my reading, they are essentially extra parameters (+ a nonlinear function) that allow the compressed model + LoRA weights extra freedom during finetuning, but they do not explicitly encode any knowledge from the original LLM. Additionally, I believe that many of the equations written are not vital to understanding the work, and replacing them with prose and/or diagrams would improve the overall readability of the paper.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable suggestion.\n\n### W1: IFT Experiments\n\nConventionally, LoRA has been mainly used to build task-specific models. This has influenced our primary experiments, which are focused on tuning the T5 model for various specific tasks. Recently, with the advancements in the capabilities of LLMs, LoRA has also been employed in instruction tuning. We have recognized this emerging trend and conducted experiments using the Llama model and the instruction tuning dataset, Alpaca. Based on the results presented in Table 3, we believe that CA-LoRA can perform well on IFT, as evidenced by the existing experiments with Llama.\n\nWe appreciate your insight into the limitations of our experiments and acknowledge the need to expand our instruction tuning experiments in the revision, particularly by incorporating more evaluation domains. Additionally, we will add more experimental results to make our existing conclusions more solid.\n\n### Q1: Why do we name the recovery module “knowledge recovery module”\n\nEssentially, you are right that these additional parameters incorporating non-linear functions provide extra learning freedom during the tuning process. In CA-LoRA, LoRA can gain task-specific capabilities through inheritance, while the recovery module mitigates the loss of certain capabilities caused by model compression. Considering that our experiments have not conclusively proven model capabilities as equivalent to knowledge, we intend to refine our terminology to \"recovery modules\" in the revision for better clarity in our paper.\n\n### About Writing\n\nThank you for your valuable feedback on improving the writing and paper structure. We will take your suggestion into consideration and make the following changes in the next camera-ready version:\n\n1. We will move Table 6 from the Appendix to Section 4 to show that CA-LoRA does not affect the inference speed.\n\n2. We will remove unnecessary equations and add more figures to make our method clearer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper demonstrates an interesting and useful strategy for building small but performant task-specific LLMs. They show applicability to real world use cases, and ground their work in an important motivation: deploying strong, task-specific models in resource constrained environments. Additionally, their method is theoretically straightforward, and adds little latency to inference. Overall, the method described seems like a very useful way to adapt compressed LLMs in certain scenarios.", "reasons_to_reject": "The first experiments focus on older models and tasks, which are less relevant for real world scenarios as described in the motivation. SuperGLUE tasks are fairly outdated, and I would prefer to see the instruction tuning experiments expanded and highlighted, because those results are also promising.\n\nThe paper could also be written more clearly. The amount of equations is – in my opinion – unnecessary, and the paper would be more straightforward to read if some of these were changed to prose (and/or diagrams) instead.", "questions_to_authors": "While the T5 experiments show value in CA-LoRA for task-specific finetuning, I believe the strength of the method lies in the instruction tuning experiments; I think they’re the most interesting and applicable to real world scenarios.\n\nIf compute allows, I have a few suggestions for extending the IFT experiments:\n* Adding more evaluation domains, to cover other important properties of instruction-tuned LLMs:\n  * Safety (XSTest, ToxiGen, etc)\n  * Instruction-following ability (AlpacaEval 1 and/or 2, IFEval)\n  * Math (GSM8K)\n* If your compute budget allows, I would recommend adding another baseline:\n  * Llama 2 13B finetuned (with LoRA, most likely), on Alpaca. This would be a fairer baseline vs your method than Llama 2 13B without instruction tuning.\n* Try instruction tuning on a domain-specific IFT set, such as CodeAlpaca-20k, and then evaluating on domain-specific evaluations (e.g. for coding, HumanEval+ and MBPP+). \n\nI don’t think any of these are strictly necessary, but I think they’d help strengthen your argument (especially the evaluations I mentioned in the first bullet point).\n\nAdditionally, if you have the space, I’d recommend moving Table 6 from the appendix to section 4.\n\nOverall, I really like your work!", "extra_scores": {}}, {"review_id": "raQ4Z82DfR", "reviewer": "Reviewer_rQV5", "summary": "The main motivation is on device personalization where having the big LLM on in device memory is expensive. Hence the authors try to understand ways this requirement can be reduced. They propose using some existing compression techniques on the base LLM and then learning Lora modules on top of it. They propose two types of Lora modules one initialized from a task-specific Lora module and another that learns how to recover the information loss that happens due to compression. However, this part if very confusing to me.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable suggestion.\n\n### Q6\n\n1. A single LoRA trained with Eq.6 is evaluated in Tab.4 (Inherit + Distill), whose accuracy is 2.9 lower than our method. This means additional parameters (recovery module) for the recovery objective are necessary.\n\n2. We further clarify that the recovery module with a non-linear activation function $\\sigma$ is not a vanilla LoRA (Fig.2 and Eq.4), and it can represent a higher rank approximation. The compression loss does not have a low-rank pattern. For example, for unstructured pruning with 50\\% sparsity, half of the parameters are eliminated, resulting in a large rank difference. To verify this, we also try \"Inherited LoRA + Another LoRA + Distill\" or \"A single larger LoRA + Distill\" in Appendix C (Table 5) and both of them have poor performance.\n\n3. [1] found that multiple tiny LoRAs are more suitable for multi-task learning compared to a larger LoRA, with more stable training and better performance. Our two objectives can also be considered as a type of multi-task learning and require two separate  modules. \n\n\n### Q1\nLoRA modules are randomly initialized for CLM + LoRA.\n\n### Q2\n\nAs in [2], developers fail to reproduce QA-LoRA, and QA-LoRA is limited to quantized LLMs, yet CA-LoRA can serve more compression methods like pruning or MoEfication. As how we build QLoRA + CA-LoRA (Table 3), adopting CA-LoRA to improve its orthogonal QA-LoRA is easy.\n\n### Q3\n\nThe recovery modules are randomly initialized.\n\n### Q4\n\nWe measure the training cost of CA-LoRA in Table 2 on RTE using $4\\times$ A100. The results show that the training cost of CA-LoRA is about $1.6\\times$ baseline, which is acceptable considering the performance benefit. Note that CLMs based on quantization and MoEfication are fast at decoding but slow at training.\n\n| Method | Time | GPU hours |\n|-|-|-|\n|CA-LoRA | 4197s | 4.66 |\n|CLM + LoRA| 2623s | 2.91 |\n\n### Q5\n\nThanks for pointing this out. Our main contribution is to inherit existing LoRA and bridge the CLM-LLM gap with an extra recovery module. Distillation is a commonly used method to match our purpose. We supplement some of the ablations based on Table 2 here for distillation, and we will include more analysis in our revision.\n\n| Compare to distill verion | BoolQ | CB | COPA | WiC |\n| - | - | - | - | - |\n| CLM+CA-LoRA w/o distill | -0.5 | -1.7 | -2.0 | -2.5 |\n\n[1] Wang et al. Multilora: Democratizing lora for better multi-task learning.\n[2] https://github.com/yuhuixu1993/qa-lora/issues/25", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Problem setting kind-off makes sense.", "reasons_to_reject": "I have some issues with the experimental setting and the method that are listed below.", "questions_to_authors": "1. In CLM + LORA are the LORA modules initialized from the trained checkpoint as in CLM + CA-LORA.\n2. The QA-Lora method is the strongest on MMLU and authors have omitted the results on it saying they took the results from the original paper. I feel like Table-3 should be completed with the results of QA-Lora on other benchmarks. \n3. How are the recovery Lora modules initialized? \n4. Can you provide training time comparisons CA-LORA with further training an inherited lora on top of a CLM? Because it seems like the cost overhead might be too high as distillation requires forward passes on two models. \n5. Looking at Table 4, it seems like distillation is the least useful component, then it would be good to compare with inheret + recover at all the places in other tables? As it is not very clear how this combination performs at other places and if the additional training cost of distillation is worth it. \n6. It does not conceptually make sense to me why do we need two lora modules in parallel to each other as the final output is just a linear combination of the two lora's anyways and both the lora's are trained on the same joint objective. Then why do we even need to recover lora. Can you just not have a single lora and use the objective in eqn-6 to obtain similar results? I feel like this would work equally well as the CA+LORA method.", "extra_scores": {}}, {"review_id": "Ph9Uk6PX0s", "reviewer": "Reviewer_z39k", "summary": "This paper proposes a method to adapt a trained LoRA module for an LLM to create a LoRA module for it's compressed version by initializing the latter with the former before training, and learning an additional a task specific low rank recovery module to recover some of the knowledge lost from the base LM in the compression process. The technique is tested both for task specific adaptation and instruction tuning with T5-3B and adapters trained with the proposed method are shown to improve over standard LoRA for compressed versions of T5-3B obtained by quantization, MoEfication, structured pruning and unstructured pruning.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable advice.\n\n### W1: Restricted to T5\n\nWe appreciate your suggestion to expand our experiments beyond the T5 model. While we do present results for Llama in Table 3, we acknowledge that our experiments currently have a narrow focus. In the revision, we will address this limitation and conduct further experiments on Llama to provide a more comprehensive analysis.\n\n### W2: Does not compare to QLoRA\n\nWe compare CA-LoRA and QLoRA in Table 3. The table shows that CA-LoRA can achieve comparable or even better performance than QLoRA. Owing to the addition of recovery modules, CA-LoRA can be combined with QLoRA and any other \"compression + LoRA\" works, showing greater compatibility and effectiveness. Furthermore, Table 6 shows that additional recovery modules do not reduce the inference speed, showing the efficiency of CA-LoRA.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "There is demand for efficient task specific models in many applications and this method shows promise as being an effective to perform task specific tuning of compressed LMs. The paper includes a result where on instruction tuning, compressed T5-3B + CA-LoRA outperforms T5-base + LoRA.", "reasons_to_reject": "The experiments on this paper are restricted to only one model architecture - T5. This should be acknowledged as a limitation and potential direction for future work.\nIt is also unclear why the paper does not compare to QLoRA.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper combines two different efficiency methods, LORA and compression methods. The results are useful, but the experiments and domains are somewhat narrow.
Decision: Accept